用R做GLM的Summary相关指标解释——以Poission regression为例

Residual

In ordinary least-squares, the residual associated with the $i$-th observation. The term residual comes from the residual sum of squares (RSS), which is defined as

$$\begin{gathered} r_i=y_i-f(x_i) \\ RSS=\sum _{i=1}^n{r}_i^2 \end{gathered}$$

where the residual $r_i$ is defined as the difference between predicted values $f(x_i)$ from the observed value $y_i$.

For GLMs, there are several ways for specifying residuals. We will cover four types of residuals: response residuals, working residuals, Pearson residuals, and, deviance residuals.

Response residuals

For type = "response", the conventional residual on the response level is computed, that is,
$$r_i=y_i-\hat{f}(x_i)$$

# type = "response"
res.response1 <- residuals(model.pois, type = "response")
res.response2 <- expected - estimates
all.equal(res.response1, res.response2)

## [1] TRUE

Working residuals

For type = "working", the residuals are normalized by the estimates $\hat{f}(x_i)$:

$$r_i=\frac{y_i-\hat{f}(x_i)}{\hat{f}(x_i)}$$

# type = "working"
res.working1 <- residuals(model.pois, type="working")
res.working2 <- (expected - estimates) / estimates
all.equal(res.working1, res.working2)

## [1] TRUE

Pearson residuals

For type = "pearson", the Pearson residuals are computed. They are obtained by normalizing the residuals by the square root of the estimate:

$$r_i=\frac{y_i-\hat{f}(x_i)}{\sqrt{f(x_i)}}$$

# type = "pearson"
res.pearson1 <- residuals(model.pois, type="pearson")
res.pearson2 <- (expected - estimates) / sqrt(estimates)
all.equal(res.pearson1, res.pearson2)

## [1] TRUE

Deviance residuals

Deviance residuals are defined by the deviance.

Deviance

The deviance of a model is given by

$$D(y,\hat{\mu })=2(\log (p(y\mid {\hat{\theta }}_s))-\log (p(y\mid {\hat{\theta }}_0)))$$

where

• $y$ is the outcome.
• $\hat{\mu }$ is the estimate of the model.
• ${\hat{\theta }}_s$ and ${\hat{\theta }}_0$ are the parameters of the fitted saturated and proposed models, respectively. A saturated model has as many parameters as it has training points, that is, $p=n$. Thus, it has a perfect fit. The proposed model can be the any other model.
• p(y|θ) is the likelihood of data given the model.

The deviance indicates the extent to which the likelihood of the saturated model exceeds the likelihood of the proposed model. If the proposed model has a good fit, the deviance will be small. If the proposed model has a bad fit, the deviance will be high. For example, for the Poisson model, the deviance is
$$D=2\cdot \sum _{i=1}^n\left[y_i\cdot \log (\frac{y_i}{{\hat{\mu }}_i})-(y_i-{\hat{\mu }}_i)\right]$$

具体推导过程参考： Poission regression deviance 的推导

1. 首先假设 $x$只有一个位置参数，$Y$ 服从Poission distribution，并且 $Y$ 的均值是一个关于 $x$ 的函数，即
   $$\mathbb{E}(Y|x)=\lambda (x)$$

2. 对于Poission regression用 $\log$ 作为link function，即$$\log (\lambda (x))={\beta }_0+{\beta }_{1}x$$

3. 那么Likelihood function就是$$L({\beta }_0,{\beta }_1;y_i)=\prod _{i=1}^n\frac{e^{-\lambda (x_i)}[\lambda (x_i{)}^{y_i}]}{y_i!}=\prod _{i=1}^n\frac{e^{e^{-{\beta }_0+{\beta }_1{x}_i}}[e^{{\beta }_0+{\beta }_1{x}_i}{]}^{y_i}]}{y_i!}$$

Log likelihood function就是：
$$l({\beta }_0,{\beta }_1;y_i)=\sum _{i=1}^n{y}_i({\beta }_0+{\beta }_1{x}_i)-\sum _{i=1}^n{e}^{{\beta }_0+{\beta }_1{x}_i}-\sum _{i=1}^n\log (y_i!)(1)$$

写成用${\hat{\mu }}_i$表示均值预测值的形式，即
$$l(\hat{\mu })=\sum _{i=1}^n{y}_i\log ({\hat{\mu }}_i)-\sum _{i=1}^n{\hat{\mu }}_i-\sum _{i=1}^n\log (y_i!)(1^{\prime })$$

4. 用 ${\mu }_1,{\mu }_2,\cdots ,{\mu }_n$代替"saturated model"的参数（saturated model：每个数据点都有一个自己的参数构成的模型）。

那么，saturated model 的 log likelihood function 即为：
$$l(\mu )=\sum _{i=1}^n{y}_i\log ({\mu }_i)-\sum _{i=1}^n{\mu }_i-\sum _{i=1}^n\log (y_i!)(2)$$

欲求（2）式中${\mu }_i$的最优估计，即对（2）求偏导数，并令其等于0，即：
$$\frac{\partial }{\partial {\mu }_i}l(\mu )=\frac{y_i}{{\mu }_i}-1=0$$
解得，’‘saturated model’’ 的参数估计值为：
$${\hat{\mu }}_i=y_i$$
将上面的结果带入到（2）式中，即得“saturated model”的likelihood function：
$$l(\hat{\mu })=\sum _{i=1}^n{y}_i\log (y_i)-\sum _{i=1}^n{y}_i-\sum _{i=1}^n\log (y_i!)(3)$$

5. 由Deviance的定义：$D(y,\hat{\mu })=2(\log (p(y\mid {\hat{\theta }}_s))-\log (p(y\mid {\hat{\theta }}_0)))$，即用$2\cdot [(3)-(1^{\prime })]，得$：
   $$D(y,\hat{\mu })=2\cdot \left[\sum _{i=1}^n{y}_i\log \frac{y_i}{{\hat{\mu }}_i}-\sum _{i=1}^n(y_i-{\hat{\mu }}_i)\right]$$

注意：这仅仅对于分类模型，有时也用 (D)表示Deviance, (D∗)表示Scaled Deviance，即D除以上一个尺度参数。

即
$$D^{\ast }=D\cdot \varphi$$
例如，Poission分布的尺度参数就为1，Nomal distribution的尺度参数为${\sigma }^2$.

所以，Nomal distribution的Deviance推出来是
$$D(y,\mu )=(y-\mu {)}^2$$
Scaled Deviance是
$$D^{\ast }(y,\mu )=\frac{(y-\mu {)}^2}{{\sigma }^2}$$
详情https://stats.stackexchange.com/questions/315788/deviance-in-simple-linear-regression

Deviance residuals

In R, the deviance residuals represent the contributions of individual samples to the deviance D. More specifically, they are defined as the signed square roots of the unit deviances.

How does such a deviance look like in practice? For example, for the Poisson distribution, the deviance residuals are defined as:（这里我个人有个小疑问暂时没解决，根号下的数不一定是正的呀）

$$r_i=sign(y-{\hat{\mu }}_i)\cdot \sqrt{2\cdot \left[y_i\cdot \log \left(\frac{y_i}{{\hat{\mu }}_i}\right)-(y_i-{\hat{\mu }}_i)\right]}$$

# type = "deviance"
res.dev1 <- residuals(model.pois, type = "deviance")
res.dev2 <- residuals(model.pois)
poisson.dev <- function (y, mu) 
    # unit deviance
    2 * (y * log(ifelse(y == 0, 1, y/mu)) - (y - mu))
res.dev3 <- sqrt(poisson.dev(expected, estimates)) * 
        ifelse(expected > estimates, 1, -1)
all.equal(res.dev1, res.dev2, res.dev3)

## [1] TRUE

Deviance residuals举例

We can obtain the deviance residuals of our model using the residuals function:

summary(residuals(model.pois))

如果有以下输出

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
## -4.0451 -2.4138 -0.8169 -0.1840  1.4265  8.7946

Since the median deviance residual is close to zero, this means that our model is not biased in one direction (i.e. the out come is neither over- nor underestimated).

Tips:

Note that, for ordinary least-squares models, the deviance residual is identical to the conventional residual.

推导见：
https://bookdown.org/egarpor/PM-UC3M/glm-deviance.html#eq:dev
或者
https://stats.stackexchange.com/questions/315788/deviance-in-simple-linear-regression

Residual 小结

名称	Response residuals(conventional residuals)	Working residuals	Pearson residuals	Deviance residuals
公式	$$r_i=y_i-\hat{f}(x_i)$$	$$r_i=\frac{y_i-\hat{f}(x_i)}{\hat{f}(x_i)}$$	$$r_i=\frac{y_i-\hat{f}(x_i)}{\sqrt{f(x_i)}}$$	$$r_i=sign(y-{\hat{\mu }}_i)\cdot \sqrt{2\cdot \left[y_i\cdot \log \left(\frac{y_i}{{\hat{\mu }}_i}\right)-(y_i-{\hat{\mu }}_i)\right]}$$
意义	常规	Normalize by the estimates	Normalize the residuals by the square root of the estimate	represent the contributions of individual samples to the deviance $D$

Null Deviance and Residual Deviance

图解Null model, proposed model and saturated model——以Nomal regression为例

这部分参考了视频：https://www.youtube.com/watch?v=9T0wlKdew6I&t=0s

我们假设只有均值这一种未知参数（即方差已知）。

如图，左上角的就是Null model，即这些数据只用一个参数去刻画；右上角的就是Saturated Model，即每个数据都有一个参数刻画它，模型很饱和；下方的是Proposed Model，即我们使用的模型，需要去评估的模型，它可以是任何介于Null Model和Saturated Model之间的模型。

Since we have already introduced the deviance, understanding the null and residual deviance is not a challenge anymore. Let us repeat the definition of the deviance once again:

$$D(y,\mu )=2(\log (p(y\mid {\theta }^s))-\log (p(y\mid {\theta }^0)))$$

The null and residual deviance differ in ${\theta }_0$:

• Null deviance: ${\theta }_0$ refers to the null model (i.e. an intercept-only model)
• Residual deviance: ${\theta }_0$ refers to the trained model

How can we interpret these two quantities?

• Null deviance: A low null deviance implies that the data can be modeled well merely using the intercept. If the null deviance is low, you should consider using few features for modeling the data.
• Residual deviance: A low residual deviance implies that the model you have trained is appropriate. Congratulations!

图解：

Confusing one: Deviance Residuals VS Residual Deviance

名称	Deviance Residuals	Residual Deviance
公式	$$r_i=sign(y-{\hat{\mu }}_i)\cdot \sqrt{2\cdot \left[y_i\cdot \log \left(\frac{y_i}{{\hat{\mu }}_i}\right)-(y_i-{\hat{\mu }}_i)\right]}$$	$ResidualDeviance=\sum [(DevianceResidual{)}_i{]}^2$ $$D(y,\hat{\mu })=2(\log (p(y\mid {\hat{\theta }}_s))-\log (p(y\mid {\hat{\theta }}_0)))$$
区别	Deviance Residuals是Residual的一种，重点关注每个样本点，区别于上文提到的另外三种Residual	Residual Deviance是Deviance的一种，重点关注模型整体，是‘saturated model’与‘proposed model’的log likelihood差的二倍，区别于Null Deviance（ ‘saturated model’ 与 ‘null model’ 的log likelihood差的二倍）

Other outputs of the summary function

Here, I deal with the other outputs of the GLM summary fuction: the dispersion parameter, the AIC, and the statement about Fisher scoring iterations.

Dispersion parameter

Dispersion (variability/scatter/spread) simply indicates whether a distribution is wide or narrow. The GLM function can use a dispersion parameter to model the variability.

However, for likelihood-based model, the dispersion parameter is always fixed to 1. It is adjusted only for methods that are based on quasi-likelihood estimation such as when family = "quasipoisson" or family = "quasibinomial". These methods are particularly suited for dealing with overdispersion.

AIC

The Akaike information criterion (AIC) is an information-theoretic measure that describes the quality of a model. It is defined as

$$AIC=2p-2\ln (\hat{L})$$

where $p$ is the number of model parameters and $\hat{L}$ is the maximum of the likelihood function. A model with a low AIC is characterized by low complexity (minimizes $p$) and a good fit (maximizes $\hat{L}$).

参考文献

1. 主要参考：Interpreting Generalized Linear Models

2. 次要参考：Notes for Predictive Modeling 5.5

3. youtube视频：Deviance Residuals

4. BOOK：Beyond Multiple Linear Regression——Applied Generalized Linear Models and Multilevel Models in R 第四章

5. Poission regression deviance 的推导

6. wikipedia：Deviance

7. https://stats.stackexchange.com/questions/5135/interpretation-of-rs-lm-output

8. https://stats.stackexchange.com/questions/315788/deviance-in-simple-linear-regression

9. https://stats.stackexchange.com/questions/108995/interpreting-residual-and-null-deviance-in-glm-r

10. Data science in insurance: an R intro