对线性回归的补充——正规方程法

1. 引言

  在单变量线性回归和多变量线性回归中，参数的更新都使用了梯度下降算法进行迭代，但是线性回归的参数最优值可以直接得到解析解。

2. 单变量线性回归的解析解

  模型：$f(x)=wx+b$
  优化目标：$(w^{\ast },b^{\ast })=\underset{w^{\ast },b^{\ast }}{\mathrm{argmin}}\sum _{i=1}^m[y_i-f(x_i){]}^2=\underset{w^{\ast },b^{\ast }}{\mathrm{argmin}}\sum _{i=1}^m[y_i-w{x}_i-b{]}^2$
  因为$E(w,b)=\sum _{i=1}^m[y_i-w{x}_i-b{]}^2$是凸函数，当它关于$w$和$b$的导数为0时，得到$w$和$b$的最优解。
  $w$和$b$的最优解计算过程如下：
$$\{\begin{array}{rl}\frac{\partial E(w,b)}{\partial w}& =2\sum _{i=1}^m-x_i(y_i-w{x}_i-b)\\ & =2\sum _{i=1}^{m}w{x}_i^2-2\sum _{i=1}^m{x}_i(y_i-b)\\ & =0(1)\\ \frac{\partial E(w,b)}{\partial b}& =2\sum _{i=1}^m-(y_i-w{x}_i-b)\\ & =2mb-2\sum _{i=1}^m(y_i-w{x}_i)\\ & =0(2)\end{array}$$
  把(2)中解出来的$b=\frac{1}{m}\sum _{i=1}^m(y_i-w{x}_i)$代入(1):
$$\begin{array}{r}w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m[x_i{y}_i-\frac{x_i}{m}\sum _{i=1}^m(y_i-w{x}_i)]=0\\ w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m{x}_i{y}_i+\sum _{i=1}^m{y}_i\sum _{i=1}^m\frac{x_i}{m}-\frac{w}{m}(\sum _{i=1}^m{x}_i{)}^2=0\\ w(\sum _{i=1}^m{x}_i^2-\frac{1}{m}(\sum _{i=1}^m{x}_i{)}^2)=\sum _{i=1}^m{y}_i(x_i-\sum _{i=1}^m\frac{x_i}{m})\\ w=\frac{\sum _{i=1}^m{y}_i(x_i-\overline{x})}{\sum _{i=1}^m{x}_i^2-\frac{1}{m}(\sum _{i=1}^m{x}_i{)}^2}\end{array}$$
  最后$w$和$b$的最优解为
$$\{\begin{array}{rl}w& =\frac{\sum _{i=1}^m{y}_i(x_i-\overline{x})}{\sum _{i=1}^m{x}_i^2-\frac{1}{m}(\sum _{i=1}^m{x}_i{)}^2}\\ b& =\frac{1}{m}\sum _{i=1}^m(y_i-w{x}_i)\end{array}$$

3. 多变量线性回归的解析解

  多变量线性回归的模型为$f(x)=wx+b$。令$\hat{w}=(w;b)$，$\hat{x}=(x;1)$，将$m\times d$的数据集D表示成$m\times (d+1)$的矩阵
X = [ x 11 x 12 ⋯ x 1 d 1 x 21 x 22 ⋯ x 2 d 1 ⋮ ⋮ ⋱ ⋮ ⋮ x m 1 x m 2 ⋯ x m d 1 ] . \Large X= \begin{bmatrix} x_{11}& x_{12}& \cdots &x_{1d} &1\\ x_{21}&x_{22} & \cdots & x_{2d}&1\\ \vdots & \vdots & \ddots & \vdots & \vdots\\ x_{m1}&x_{m2} & \cdots & x_{md} & 1 \end{bmatrix}. X= ​x11​x21​⋮xm1​​x12​x22​⋮xm2​​⋯⋯⋱⋯​x1d​x2d​⋮xmd​​11⋮1​ ​.
  模型：$f(x)=\hat{w}\hat{x}$
  优化目标：${\hat{w}}^{\ast }=\underset{\hat{w}}{\mathrm{argmin}}(y-X\hat{w}{)}^T(y-X\hat{w})$
  $\hat{w}$最优值的解析解如下：
$$\begin{array}{rl}& \frac{\partial (y-X\hat{w}{)}^T(y-X\hat{w})}{\partial \hat{w}}=0\\ & \frac{\partial (y^T-{\hat{w}}^T{X}^T)(y-X\hat{w})}{\partial \hat{w}}=0\\ & \frac{\partial (y^{T}y-{\hat{w}}^T{X}^{T}y-y^{T}X\hat{w}+{\hat{w}}^T{X}^{T}X\hat{w})}{\partial \hat{w}}=0\\ & \frac{\partial y^{T}y}{\partial \hat{w}}-\frac{\partial {\hat{w}}^T{X}^{T}y}{\partial \hat{w}}-\frac{\partial y^{T}X\hat{w}}{\partial \hat{w}}+\frac{\partial {\hat{w}}^T{X}^{T}X\hat{w}}{\partial \hat{w}}=0\\ & 0-X^{T}y-X^{T}y+(X^{T}X+(X^{T}X{)}^T)\hat{w}=0\\ & X^{T}X\hat{w}=X^{T}y\\ & \\ & 如果X^{T}X可逆，则\hat{w}=(X^{T}X{)}^{-1}{X}^{T}y。\\ & 如果X^{T}X不可逆，则\hat{w}存在多个解。\end{array}$$
$\frac{\partial {\hat{w}}^T{X}^{T}y}{\partial \hat{w}}=\frac{\partial y^{T}X\hat{w}}{\partial \hat{w}}=X^{T}y$和$\frac{\partial {\hat{w}}^T{X}^{T}X\hat{w}}{\partial \hat{w}}=2X^{T}X\hat{w}$的计算见4.2.1和4.2.2。

参考

周志华 机器学习