西瓜书——线性模型笔记

线性模型

Lei_ZM
2019-09-10

---

---

1. 一元线性回归

求解偏置$b$和权重$w$推导思路

1. 由最小二乘法导出损失函数$E(w,b)$

2. 证明损失函数

3. 分别对损失函数$E(w,b)$关于$b$和$w$求一阶偏导数

4. 令各自的一阶偏导数等于0解出$b$和$w$

1.1. 由最小二乘法导出损失函数$E(w,b)$

$$\begin{array}{rl}{E}_{(w,b)}& =\sum _{i=1}^m{\left(y_i-f\left(x_i\right)\right)}^2\\ & =\sum _{i=1}^m{\left(y_i-\left(w{x}_i+b\right)\right)}^2\\ & =\sum _{i=1}^m{\left(y_i-w{x}_i-b\right)}^2\end{array}\text{\hspace{1em}(西瓜书式3.4)}$$

---

1.2. 证明损失函数

1.2.1. 二元函数判断凹凸性：

设$f(x,y)$在区域$D$上具有二阶连续偏导数，记$A=f_{xx}^{\prime \prime }(x,y)$，$B=f_{xy}^{\prime \prime }(x,y)$，$C=f_{yy}^{\prime \prime }(x,y)$。则：

1. 在$D$上恒有$A>0$，且$AC-B^2\ge 0$时，$f(x,y)$在区域$D$上是凸函数
2. 在$D$上恒有$A<0$，且$AC-B^2\ge 0$时，$f(x,y)$在区域$D$上是凹函数

1.2.2. 二元凹凸函数求最值：

设$f(x,y)$是在开区域$D$内具有连续偏导数的凸（或者凹）函数，$(x_0,y_0)\in D$，且$f_x^{\prime }(x_0,y_0)=0$，$f_y^{\prime }(x_0,y_0)=0$，则$f(x_0,y_0)$必为$f(x,y)$在$D$内的最小值（或最大值）。

1.2.3. 证明

证明损失函数$E(w,b)$是关于$w$和$b$的凸函数——求$A=f_{xx}^{\prime \prime }(x,y)$：

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial w}& =\frac{\partial }{\partial w}\left[\sum _{i=1}^m{\left(y_i-\left(w{x}_i+b\right)\right)}^2\right]\\ & =\sum _{i=1}^m\frac{\partial }{\partial w}{\left(y_i-w{x}_i-b\right)}^2\\ & =\sum _{i=1}^{m}2\cdot \left(y_i-w{x}_i-b\right)\cdot \left(-x_i\right)\\ & =2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)\end{array}\text{\hspace{1em}(西瓜书式3.5)}$$

故有：

$$\begin{array}{rl}\frac{{\partial }^2{E}_{(w,b)}}{\partial w^2}& =\frac{\partial }{\partial w}\left(\frac{\partial E_{(w,b)}}{\partial w}\right)\\ & =\frac{\partial }{\partial w}\left[2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)\right]\\ & =\frac{\partial }{\partial w}\left[2w\sum _{i=1}^m{x}_i^2\right]\\ & =2\sum _{i=1}^m{x}_i^2\end{array}$$

此式即为$A=f_{xx}^{\prime \prime }(x,y)$。

证明损失函数$E(w,b)$是关于$w$和$b$的凸函数——求$B=f_{xy}^{\prime \prime }(x,y)$：

$$\begin{array}{rl}\frac{{\partial }^2{E}_{(w,b)}}{\partial w\partial b}& =\frac{\partial }{\partial b}\left(\frac{\partial E_{(w,b)}}{\partial w}\right)\\ & =\frac{\partial }{\partial b}\left[2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)\right]\\ & =\frac{\partial }{\partial b}\left[-2\sum _{i=1}^m\left(y_i-b\right)x_i\right]\\ & =\frac{\partial }{\partial b}\left(-2\sum _{i=1}^m{y}_i{x}_i+2\sum _{i=1}^{m}b{x}_i\right)\\ & =\frac{\partial }{\partial b}\left(2\sum _{i=1}^{m}b{x}_i\right)\\ & =2\sum _{i=1}^m{x}_i\end{array}$$

此式即为$B=f_{xy}^{\prime \prime }(x,y)$。

证明损失函数$E(w,b)$是关于$w$和$b$的凸函数——求$C=f_{yy}^{\prime \prime }(x,y)$：

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial b}& =\frac{\partial }{\partial b}\left[\sum _{i=1}^m{\left(y_i-\left(w{x}_i+b\right)\right)}^2\right]\\ & =\sum _{i=1}^m\frac{\partial }{\partial b}{\left(y_i-w{x}_i-b\right)}^2\\ & =\sum _{i=1}^{m}2\cdot \left(y_i-w{x}_i-b\right)\cdot (-1)\\ & =2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)\end{array}\text{\hspace{1em}(西瓜书式3.6)}$$

故有：

$$\begin{array}{rl}\frac{{\partial }^2{E}_{(w,b)}}{\partial b^2}& =\frac{\partial }{\partial b}\left(\frac{\partial E_{(w,b)}}{\partial b}\right)\\ & =\frac{\partial }{\partial b}\left[2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)\right]\\ & =\frac{\partial }{\partial b}(2mb)\\ & =2m\end{array}$$

此式即为$C=f_{yy}^{\prime \prime }(x,y)$。

综上所述，有：

$$\{\begin{array}{rl}& A=f_{xx}^{\prime \prime }(x,y)=2\sum _{i=1}^m{x}_i^2\\ & B=f_{xy}^{\prime \prime }(x,y)=2\sum _{i=1}^m{x}_i\\ & C=f_{yy}^{\prime \prime }(x,y)=2m\end{array}$$

所以：

$$\begin{array}{rl}AC-B^2& =2m\cdot 2\sum _{i=1}^m{x}_i^2-{\left(2\sum _{i=1}^m{x}_i\right)}^2\\ & =4m\sum _{i=1}^m{x}_i^2-4{\left(\sum _{i=1}^m{x}_i\right)}^2\\ & =4m\sum _{i=1}^m{x}_i^2-4\cdot m\cdot \frac{1}{m}\cdot {\left(\sum _{i=1}^m{x}_i\right)}^2\\ & =4m\sum _{i=1}^m{x}_i^2-4m\cdot \bar{x}\cdot \sum _{i=1}^m{x}_i\\ & =4m\left(\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m{x}_i\bar{x}\right)\\ & =4m\sum _{i=1}^m\left(x_i^2-x_i\bar{x}\right)\\ & =4m\sum _{i=1}^m\left(x_i^2-x_i\bar{x}-x_i\bar{x}+x_i\bar{x}\right)\\ & \sum _{i=1}^m{x}_i\bar{x}=\bar{x}\sum _{i=1}^m{x}_i=\bar{x}\cdot m\cdot \frac{1}{m}\cdot \sum _{i=1}^m{x}_i=m{\bar{x}}^2=\sum _{i=1}^m{\bar{x}}^2\\ & =4m\sum _{i=1}^m\left(x_i^2-x_i\bar{x}-x_i\bar{x}+{\bar{x}}^2\right)\\ & =4m\sum _{i=1}^m{\left(x_i-\bar{x}\right)}^2\end{array}$$

故有：

$$AC-B^2=4m\sum _{i=1}^m{\left(x_i-\bar{x}\right)}^2\ge 0$$

也即损失函数$E(w,b)$是关于$w$和$b$的凸函数，得证！

---

1.3. 分别对损失函数$E(w,b)$关于$b$和$w$求一阶偏导数

损失函数$E(w,b)$关于$b$求一阶偏导数：

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial b}& =\frac{\partial }{\partial b}\left[\sum _{i=1}^m{\left(y_i-\left(w{x}_i+b\right)\right)}^2\right]\\ & =\sum _{i=1}^m\frac{\partial }{\partial b}{\left(y_i-w{x}_i-b\right)}^2\\ & =\sum _{i=1}^{m}2\cdot \left(y_i-w{x}_i-b\right)\cdot (-1)\\ & =2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)\end{array}\text{\hspace{1em}(西瓜书式3.6)}$$

损失函数$E(w,b)$关于$w$求一阶偏导数：

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial w}& =\frac{\partial }{\partial w}\left[\sum _{i=1}^m{\left(y_i-\left(w{x}_i+b\right)\right)}^2\right]\\ & =\sum _{i=1}^m\frac{\partial }{\partial w}{\left(y_i-w{x}_i-b\right)}^2\\ & =\sum _{i=1}^{m}2\cdot \left(y_i-w{x}_i-b\right)\cdot \left(-x_i\right)\\ & =2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)\end{array}\text{\hspace{1em}(西瓜书式3.5)}$$

---

1.4. 令各自的一阶偏导数等于0解出$b$和$w$

令损失函数$E(w,b)$关于$b$的一阶偏导数等于0解出$b$：

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial b}& =2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)=0\\ & \Rightarrow mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)=0\\ & \begin{array}{rl}\Rightarrow b& =\frac{1}{m}\sum _{i=1}^m\left(y_i-w{x}_i\right)\\ & =\frac{1}{m}\sum _{i=1}^m{y}_i-w\frac{1}{m}\sum _{i=1}^m{x}_i\\ & =\bar{y}-w\bar{x}\end{array}\end{array}\text{\hspace{1em}(西瓜书式3.8)}$$

令损失函数$E(w,b)$关于$w$的一阶偏导数等于0解出$w$：

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial w}& =2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)=0\\ & \Rightarrow w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i=0\\ & \Rightarrow w\sum _{i=1}^m{x}_i^2=\sum _{i=1}^m{y}_i{x}_i-\sum _{i=1}^{m}b{x}_i\\ & b=\bar{y}-w\bar{x}\\ & \Rightarrow w\sum _{i=1}^m{x}_i^2=\sum _{i=1}^m{y}_i{x}_i-\sum _{i=1}^m(\bar{y}-w\bar{x})x_i\\ & \Rightarrow w\sum _{i=1}^m{x}_i^2=\sum _{i=1}^m{y}_i{x}_i-\bar{y}\sum _{i=1}^m{x}_i+w\bar{x}\sum _{i=1}^m{x}_i\\ & \Rightarrow w\sum _{i=1}^m{x}_i^2-w\bar{x}\sum _{i=1}^m{x}_i=\sum _{i=1}^m{y}_i{x}_i-\bar{y}\sum _{i=1}^m{x}_i\\ & \Rightarrow w\left(\sum _{i=1}^m{x}_i^2-\bar{x}\sum _{i=1}^m{x}_i\right)=\sum _{i=1}^m{y}_i{x}_i-\bar{y}\sum _{i=1}^m{x}_i\\ & \begin{array}{rl}\Rightarrow w& =\frac{{\sum }_{i=1}^m{y}_i{x}_i-\bar{y}{\sum }_{i=1}^m{x}_i}{{\sum }_{i=1}^m{x}_i^2-\bar{x}{\sum }_{i=1}^m{x}_i}\\ & \bar{y}\sum _{i=1}^m{x}_i=\frac{1}{m}\sum _{i=1}^m{y}_i\sum _{i=1}^m{x}_i=\bar{x}\sum _{i=1}^m{y}_i\\ & \bar{x}\sum _{i=1}^m{x}_i=\frac{1}{m}\sum _{i=1}^m{x}_i\sum _{i=1}^m{x}_i=\frac{1}{m}{\left(\sum _{i=1}^m{x}_i\right)}^2\\ & =\frac{{\sum }_{i=1}^m{y}_i{x}_i-\bar{x}{\sum }_{i=1}^m{y}_i}{{\sum }_{i=1}^m{x}_i^2-\frac{1}{m}{\left({\sum }_{i=1}^m{x}_i\right)}^2}\\ & =\frac{{\sum }_{i=1}^m{y}_i\left(x_i-\bar{x}\right)}{{\sum }_{i=1}^m{x}_i^2-\frac{1}{m}{\left({\sum }_{i=1}^m{x}_i\right)}^2}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎‪‫‫‌‬‎‮‌‌‫⁠‌\end{array}‏‌‎‬‎‪‍‭‎\end{array}\text{\hspace{1em}(西瓜书式3.7)}$$

将$w$向量化，有：

$$\begin{array}{rl}w& =\frac{{\sum }_{i=1}^m{y}_i\left(x_i-\bar{x}\right)}{{\sum }_{i=1}^m{x}_i^2-\frac{1}{m}{\left({\sum }_{i=1}^m{x}_i\right)}^2}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎\\ & \frac{1}{m}{\left(\sum _{i=1}^m{x}_i\right)}^2=\left(\frac{1}{m}\sum _{i=1}^m{x}_i\right)\sum _{i=1}^m{x}_i=\bar{x}\sum _{i=1}^m{x}_i=\sum _{i=1}^m{x}_i\bar{x}\\ & =\frac{{\sum }_{i=1}^m\left(y_i{x}_i-y_i\bar{x}\right)}{{\sum }_{i=1}^m\left(x_i^2-x_i\bar{x}\right)}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎\\ & =\frac{{\sum }_{i=1}^m\left(y_i{x}_i-y_i\bar{x}-y_i\bar{x}-y_i\bar{x}\right)}{{\sum }_{i=1}^m\left(x_i^2-x_i\bar{x}-x_i\bar{x}-x_i\bar{x}\right)}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎\\ & \sum _{i=1}^m{y}_i\bar{x}=\bar{x}\sum _{i=1}^m{y}_i=\frac{1}{m}\sum _{i=1}^m{x}_i\sum _{i=1}^m{y}_i=\sum _{i=1}^m{x}_i\cdot \frac{1}{m}\cdot \sum _{i=1}^m{y}_i=\sum _{i=1}^m{x}_i\bar{y}\\ & \sum _{i=1}^m{y}_i\bar{x}=\bar{x}\sum _{i=1}^m{y}_i=\bar{x}\cdot m\cdot \frac{1}{m}\cdot \sum _{i=1}^m{y}_i=m\bar{x}\bar{y}=\sum _{i=1}^m\bar{x}\bar{y}\\ & \sum _{i=1}^m{x}_i\bar{x}=\bar{x}\sum _{i=1}^m{x}_i=\bar{x}\cdot m\cdot \frac{1}{m}\cdot \sum _{i=1}^m{x}_i=m{\bar{x}}^2=\sum _{i=1}^m{\bar{x}}^2\\ & =\frac{{\sum }_{i=1}^m\left(y_i{x}_i-y_i\bar{x}-x_i\bar{y}-\bar{x}\bar{y}\right)}{{\sum }_{i=1}^m\left(x_i^2-x_i\bar{x}-x_i\bar{x}-{\bar{x}}^2\right)}‎‬‎‪‍‭‎‏‌‎‬‎‪‍‭‎\\ & =\frac{{\sum }_{i=1}^m\left(x_i-\bar{x}\right)\left(y_i-\bar{y}\right)}{{\sum }_{i=1}^m{\left(x_i-\bar{x}\right)}^2}‎\\ & x={\left(x_1,x_2,\cdots ,x_m\right)}^T\\ & y={\left(y_1,y_2,\cdots ,y_m\right)}^T\\ & x_d={\left(x_1-\bar{x},x_2-\bar{x},\cdots ,x_m-\bar{x}\right)}^T\\ & y_d={\left(y_1-\bar{y},y_2-\bar{y},\cdots ,y_m-\bar{y}\right)}^T\\ & =\frac{x_d^T{y}_d}{x_d^T{x}_d}\end{array}$$

---

---

2. 二元线性回归

求解权重$\hat{w}$的公式推导推导思路：

1. 由最小二乘法导出损失函数$E_{\hat{w}}$

2. 证明损失函数$E_{\hat{w}}$是关于$\hat{w}$的凸函数

3. 对损失函数$E_{\hat{w}}$关于$\hat{w}$求一阶偏导数

4. 令各自的一阶偏导数等于0解出${\hat{w}}^{\ast }$

2.1. 将$w$和$b$组合成$\hat{w}$

$$\begin{array}{rl}f\left({\mathbf{x}}_i\right)& ={\mathbf{w}}^T{\mathbf{x}}_i+b\\ & =\left(\begin{array}{cccc}{w}_1& w_2& \dots & w_d\end{array}\right)\left(\begin{array}{c}{x}_{i1}\\ x_{i2}\\ ⋮\\ x_{id}\end{array}\right)+b\\ & =w_1{x}_{i1}+w_2{x}_{i2}+\dots +w_d{x}_{id}+b\\ & w_{d+1}=b\\ & =w_1{x}_{i1}+w_2{x}_{i2}+\dots +w_d{x}_{id}+w_{d+1}\cdot 1\\ & =\left(\begin{array}{ccccc}{w}_1& w_2& \dots & w_d& w_{d+1}\end{array}\right)\left(\begin{array}{c}{x}_{i1}\\ x_{i2}\\ ⋮\\ x_{id}\\ 1\end{array}\right)\\ & ={\hat{w}}^T{\hat{x}}_i\end{array}$$

---

2.2. 由最小二乘法导出损失函数$E_{\hat{w}}$

$$\begin{array}{rl}{E}_{\hat{\mathbf{w}}}& =\sum _{i=1}^m{\left(y_i-f\left({\hat{\mathbf{x}}}_i\right)\right)}^2\\ & =\sum ^m{\left(y_i-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i\right)}^2\\ & \begin{array}{rl}& \mathbf{X}=\left(\begin{array}{ccccc}{x}_{11}& x_{12}& \dots & x_{1d}& 1\\ x_{21}& x_{22}& \dots & x_{2d}& 1\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ x_{m1}& x_{m2}& \dots & x_{md}& 1\end{array}\right)=\left(\begin{array}{cc}{\mathbf{x}}_1^{\mathrm{T}}& 1\\ {\mathbf{x}}_2^{\mathrm{T}}& 1\\ ⋮& ⋮\\ {\mathbf{x}}_m^{\mathrm{T}}& 1\end{array}\right)=\left(\begin{array}{c}{\hat{\mathbf{x}}}_1^T\\ {\hat{\mathbf{x}}}_2^T\\ ⋮\\ {\hat{\mathbf{x}}}_m^T\end{array}\right)\\ & \mathbf{y}={\left(y_1,y_2,\cdots ,y_m\right)}^T\end{array}\\ & ={\left(y_1-{\hat{\mathbf{w}}}^T{\hat{x}}_1\right)}^2+{\left(y_2-{\hat{\mathbf{w}}}^T{\hat{x}}_2\right)}^2+\cdots +{\left(y_m-{\hat{\mathbf{w}}}^T{\hat{x}}_m\right)}^2\\ & =\left(\begin{array}{cccc}{y}_1-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_1& y_2-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_2& \cdots & y_m-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_m\end{array}\right)\left(\begin{array}{c}{y}_1-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_1\\ y_2-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_2\\ ⋮\\ y_m-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_m\end{array}\right)\\ & (\begin{array}{c}{y}_1-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_1\\ y_2-{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_2\\ ⋮\\ y_m\end{array}\\ & \\ & \end{array}$$