Generalized Linear Models

Common generalized linear models

The generalized linear model includes many model types in its class. The linear model is a special case where the random component (the distribution of the response) is the normal distribution and the link function is the identity function, that is, $g(y)=y$. The linear model (regression and ANOVA), as well as some other special cases, are shown in the table below.

Example GLM with a binary response

Suppose you apply different levels of doses of penicillin to 54 diseased animals. Some animals are cured, some are not. You are interested in how the probability of being cured varies with dose. GLM using maximum likelihood to estimate the probability of a binomial response variable is the same model as binomial logistic regression. In this scenario, let’s explore fitting a GLM to the binary response.

The random component is the binomial distribution. The link function is the logit function. The data are given in the following table.

The GLM models the probability of curing the disease, or P(Response = “Cured”) as a function of ln(dose). The parameter estimates and hypothesis tests for the parameters are shown below.

You can visualize the effect of the predictor ln(dose) on the probability of being cured, P(Response = “Cured”) as seen in the figure below.

As dose increases, so does the probability of being cured.

You can see that the model-fitting mechanics work very similarly for a GLM and for a regression model. The only difference for the analyst is that when you choose a GLM instead of a linear model, you also specify the distribution and the link function. The statistical software implements the model-fitting changes that need to happen behind the scenes.

Example GLM with a Poisson response

Female horseshoe crabs often attract male horseshoe crabs to their nests even if they already have a primary mate in the nest. Data were collected on 173 female horseshoe crabs with a primary mate. The number of other males hanging around the nest (satellites) were counted. Various characteristics of the female were measured, including color, condition of her spine, width of her carapace, and her weight. Since the response variable represents integer counts, the normal distribution assumption of regression doesn’t apply. Instead, a Poisson distribution might be applicable.

The random component is the Poisson distribution; the link function is the log function. In this example, we’ll look at the linear model $\log(\theta) = \beta_0 + \beta_1 x$ where $x$ is the weight of the female crab. That is, the mean of satellites is a linear function of the weight of the female.

The parameter estimates are given in the table below.

The model is visualized in the figure below.

According to this model, as a female horseshoe crab’s weight increases, the number of satellites also increases.

Example GLM with a Poisson response and offset

Sometimes the Poisson counts do not have the same area of opportunity; comparison of counts directly is not fair. An example is counting the number of defects on a book cover when the books are of different sizes. A bigger book is expected to have more defects than a smaller book. You can use an offset parameter, essentially an intercept term in the linear model, to account for varying areas of opportunity.

Instead of modeling $\log(\theta)$ as a linear function of $x$, we model $\log(\theta/N_i)$, where $N_i$ is the area of opportunity for observation $i$. The model is $\log(\theta_i/N_i) = \beta_0 + \beta_1 x_i$ or $\log(\theta_i) = \log(N_i) + \beta_0 + \beta_1 x_i$. The offset is $log(N_i)$.

Suppose you want to understand the relationship between the count of new melanoma cases for white males in different areas of the United States and different age groups.

• Region is Northern or Southern.
• Age Group is < 35, 35-44, 45-54, 55-64, 65-74, and > 75.
• Cases is the number of patients with a new melanoma diagnosis.
• Total is the number of patients in each region and age group.

The total number of patients varies across groups, so the area of opportunity differs among groups. The offset is log(Total).

The data are shown in the table below.

The hypothesis tests for each factor are found below.

The prediction formula for the rate of cases can be visualized in the figures below.

You’ll notice that the rate of cases increases as Age Group increases, but more rapidly for the Southern Region. Not including the offset parameter in the model doesn’t let you calculate the rates correctly.

Statistical details of GLMs

A GLM consists of:

• A systematic component, which relates the predictor variables to the response. This function is the same as the linear model in regression.
• A random component, which uses any distribution in the natural exponential family. Most common distributions are members of the exponential family.
• A link function, which relates the mean response to the systematic component.

The systematic component

The systematic component uses the linear model $\eta_i = \sum_{j=1}^{p} \beta_j x_{ij}$, $i = 1, \ldots, n$. This linear combination is the same as for a linear model.

The random component

The random component can be any distribution from the natural exponential family. This means that the probability density or mass function can be written as $f(y \mid \theta) = a(\theta)b(y)e^{yQ(\theta)}$. That is, it can be written as function of the distribution parameter $\theta$, multiplied by a function of the response, multiplied by an exponential function of the response, multiplied by a function $Q(\theta)$. $Q(\theta)$ is called the natural parameter.

In some cases, the function $a(\theta)$ or $b(y)$ might be constant. The distribution parameter $\theta$ can be a scalar or a vector.

The link function

The link function relates the random component and the systematic component. Let $\mu_i = E(y_i)$. Then the link function is $g(\mu_i) = \eta_i = \sum_{j=1}^{p} \beta_j x_{ij}$. The link function is monotonic and differentiable. It is called the canonical link function if it transforms the mean to the natural parameter $Q(\theta)$.

Examples

In the penicillin example, the random component is the binomial distribution. The probability mass function is $f(y_i \mid \pi_i) = \pi_i^{y_i}(1-\pi_i)^{n-y_i} = (1-\pi_i)^n e^{y_i \log\left(\frac{\pi_i}{1-\pi_i}\right)}$. Here $a(\theta) = (1-\pi)^n$, $b(y) = 1$, and $Q(\theta)=\log\left(\frac{\pi}{1-\pi}\right)$. The canonical link function is the log function.

In the crab satellites example, the random component is $f(y \mid \theta) = \frac{e^{-\theta}\theta^y}{y!},\ y=0,1,2,\ldots$. Here $a(\theta) = e^{-\theta}$, $b(y) = \frac{1}{y!}$, and $Q(\theta) = \log(\theta)$. The canonical link function is the logit function.

The melanoma example uses a Poisson distribution and log link function, just as in the crab satellites example, so the mathematical details of the random component and the link function are the same. In the melanoma example, the model also includes an offset, to account for differing areas of opportunity.