多元线性回归方程正规方程解（Normal Equation）公式推导详细过程

多元线性方程公式

定义多元线性方程的损失函数如下：
$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m({\hat{y}}^{(i)}-y^{(i)}{)}^2(1)$$

其中，${\hat{y}}^{(i)}$ 为：
$${\hat{y}}^{(i)}={\theta }_0+{\theta }_1{X}_1^{(i)}+{\theta }_2{X}_2^{(i)}+\cdots +{\theta }_n{X}_n^{(i)}(2)$$

其中，$m$ 为样本个数，$n$ 为特征数量

定义向量：
$$\begin{array}{rl}\theta & =({\theta }_0,{\theta }_1,\cdots ,{\theta }_n{)}^T\\ & \\ X^{(i)}& =(X_0^{(i)},X_1^{(i)},X_2^{(i)},\cdots ,X_n^{(i)}{)}^T，其中i=(1,2,\cdots ,m)，X_0^{(i)}\equiv 1\\ & \\ X_j& =(X_j^{(1)},X_j^{(2)},\cdots ,X_j^{(m)}{)}^T，其中j=(0,1,2,\cdots ,n)\\ & \\ y& =(y^{(1)},y^{(2)},\cdots ,y^{(m)})\end{array}$$

定义矩阵：

$$\begin{array}{r}X=(X^{(1)},X^{(2)},\cdots ,X^{(m)}{)}^T=(X_0,X_1,X_2,\cdots ,X_n)={\left(\begin{array}{cccc}{X}_0^{(1)}& X_1^{(1)}& X_2^{(1)}& \cdots X_n^{(1)}\\ & & & \\ X_0^{(2)}& X_1^{(2)}& X_2^{(2)}& \cdots X_n^{(2)}\\ & & & \\ \cdots & & & \cdots \\ & & & \\ X_0^{(n)}& X_1^{(m)}& X_2^{(m)}& \cdots X_n^{(m)}\end{array}\right)}_{m\times (n+1)}\end{array}$$

当 $\theta$ 取下值时，损失函数最小：
$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$

公式推导

温馨提示：公式推导过程不难，但很绕，请耐心…

将(2)式代入(1)式得：

$$\begin{array}{rl}J(\theta )& =\frac{1}{2m}\sum _{i=1}^m({\theta }_0+{\theta }_1{X}_1^{(i)}+{\theta }_2{X}_2^{(i)}+\cdots +{\theta }_n{X}_n^{(i)}-y^{(i)}{)}^2\\ & \\ & =\frac{1}{2m}\sum _{i=1}^m({\theta }_0{X}_0^{(i)}+{\theta }_1{X}_1^{(i)}+{\theta }_2{X}_2^{(i)}+\cdots +{\theta }_n{X}_n^{(i)}-y^{(i)}{)}^2（补个X_0^{(i)}）\\ & \\ & =\frac{1}{2m}[({\theta }_0{X}_0^{(1)}+{\theta }_1{X}_1^{(1)}+{\theta }_2{X}_2^{(1)}+\cdots +{\theta }_n{X}_n^{(1)}-y^{(1)}{)}^2\\ & \\ & +({\theta }_0{X}_0^{(2)}+{\theta }_1{X}_1^{(2)}+{\theta }_2{X}_2^{(2)}+\cdots +{\theta }_n{X}_n^{(2)}-y^{(2)}{)}^2\\ & \\ & +\cdots \\ & \\ & +({\theta }_0{X}_0^{(m)}+{\theta }_1{X}_1^{(2)}+{\theta }_2{X}_2^{(m)}+\cdots +{\theta }_n{X}_n^{(m)}-y^{(m)}{)}^2]\end{array}$$

现在对 $\theta$ 求偏导，下面只对 ${\theta }_1$ 求偏导，其他的依次类推：

$$\begin{array}{rl}\frac{\partial }{\partial {\theta }_1}J(\theta )& =\frac{1}{m}[X_1^{(1)}({\theta }_0{X}_0^{(1)}+{\theta }_1{X}_1^{(1)}+{\theta }_2{X}_2^{(1)}+\cdots +{\theta }_n{X}_n^{(1)}-y^{(1)})\\ & \\ & +X_1^{(2)}({\theta }_0{X}_0^{(2)}+{\theta }_1{X}_1^{(2)}+{\theta }_2{X}_2^{(2)}+\cdots +{\theta }_n{X}_n^{(2)}-y^{(2)})\\ & \\ & +\cdots \\ & \\ & +X_1^{(m)}({\theta }_0{X}_0^{(m)}+{\theta }_1{X}_1^{(2)}+{\theta }_2{X}_2^{(m)}+\cdots +{\theta }_n{X}_n^{(m)}-y^{(m)})]\\ & \\ & \\ & =\frac{1}{m}[({\theta }_0{X}_0^{(1)}{X}_1^{(1)}+{\theta }_1{X}_1^{(1)}{X}_1^{(1)}+{\theta }_2{X}_2^{(1)}{X}_1^{(1)}+\cdots +{\theta }_n{X}_n^{(1)}{X}_1^{(1)}-y^{(1)}{X}_1^{(1)})\\ & \\ & +({\theta }_0{X}_0^{(2)}{X}_1^{(2)}+{\theta }_1{X}_1^{(2)}{X}_1^{(2)}+{\theta }_2{X}_2^{(2)}{X}_1^{(2)}+\cdots +{\theta }_n{X}_n^{(2)}{X}_1^{(2)}-y^{(2)}{X}_1^{(2)})\\ & \\ & +\cdots \\ & \\ & +({\theta }_0{X}_0^{(m)}{X}_1^{(m)}+{\theta }_1{X}_1^{(2)}{X}_1^{(m)}+{\theta }_2{X}_2^{(m)}{X}_1^{(m)}+\cdots +{\theta }_n{X}_n^{(m)}-y^{(m)}{X}_1^{(m)})]（把X_1^{(i)}乘进去）\\ & \\ & \\ & =\frac{1}{m}[(X_0^{(1)}{X}_1^{(1)}+X_0^{(2)}{X}_1^{(2)}+\cdots +X_0^{(m)}{X}_1^{(m)})\cdot {\theta }_0\\ & \\ & +(X_1^{(1)}{X}_1^{(1)}+X_1^{(2)}{X}_1^{(2)}+\cdots +X_1^{(m)}{X}_1^{(m)})\cdot {\theta }_1\\ & \\ & +(X_2^{(1)}{X}_1^{(1)}+X_2^{(2)}{X}_1^{(2)}+\cdots +X_2^{(m)}{X}_1^{(m)})\cdot {\theta }_2\\ & \\ & +\cdots \\ & \\ & +(X_n^{(1)}{X}_1^{(1)}+X_n^{(2)}{X}_1^{(2)}+\cdots +X_n^{(m)}{X}_1^{(m)})\cdot {\theta }_n\\ & \\ & -(y^{(1)}{X}_1^{(1)}+y^{(2)}{X}_1^{(2)})+\cdots +y^{(m)}{X}_1^{(m)}]（将\theta 提出来）\\ & \\ & \\ & =\frac{1}{m}[{\theta }_0\cdot \sum _{i=1}^m{X}_0^{(i)}{X}_1^{(i)}+{\theta }_1\cdot \sum _{i=1}^m{X}_1^{(i)}{X}_1^{(i)}+{\theta }_2\cdot \sum _{i=1}^m{X}_2^{(i)}{X}_1^{(i)}+\cdots +{\theta }_n\cdot \sum _{i=1}^m{X}_n^{(i)}{X}_1^{(i)}-\sum _{i=1}^m{y}^{(i)}{X}_1^{(i)}]\end{array}$$

我们先对 ${\sum }_{i=1}^m{X}_a^{(i)}{X}_b^{(i)}$ 做下研究：
$$\sum _{i=1}^m{X}_a^{(i)}{X}_b^{(i)}=(X_a^{(1)},X_a^{(2)},\cdots ,X_a^{(n)})\cdot \left(\begin{array}{c}{X}_b^{(1)}\\ \\ X_b^{(2)}\\ \\ \cdots \\ \\ X_b^{(n)}\end{array}\right)=X_a^T\cdot X_b=X_b^T\cdot X_a(3)$$

将(3)式代入 $\frac{\partial }{\partial {\theta }_1}J(\theta )$ 得

$$\begin{array}{rl}\frac{\partial }{\partial {\theta }_1}J(\theta )& =\frac{1}{m}[{\theta }_0\cdot \sum _{i=1}^m{X}_0^{(i)}{X}_1^{(i)}+{\theta }_1\cdot \sum _{i=1}^m{X}_1^{(i)}{X}_1^{(i)}+{\theta }_2\cdot \sum _{i=1}^m{X}_2^{(i)}{X}_1^{(i)}+\cdots +{\theta }_n\cdot \sum _{i=1}^m{X}_n^{(i)}{X}_1^{(i)}-\sum _{i=1}^m{y}^{(i)}{X}_1^{(i)}]\\ & \\ & =\frac{1}{m}(X_0^T\cdot X_1\cdot {\theta }_0+X_1^T\cdot X_1\cdot {\theta }_1+X_2^T\cdot X_1\cdot {\theta }_2+\cdots +X_n^T\cdot X_1\cdot {\theta }_n-X_1^T\cdot y)\\ & \end{array}$$

令 $\frac{\partial }{\partial {\theta }_1}J(\theta )=0$ ，得到如下等式：

$$(X_0^T{X}_1,X_1^T{X}_1,X_2^T{X}_1\cdots ,X_n^T{X}_1)\cdot \left(\begin{array}{c}{\theta }_0\\ \\ {\theta }_1\\ \\ \cdots \\ \\ {\theta }_n\end{array}\right)=X_1^T\cdot y$$

与 $\frac{\partial }{\partial {\theta }_1}J(\theta )=0$ 同理，对 ${\theta }_0,{\theta }_2,{\theta }_3,\cdots ,{\theta }_n$ 做偏导等零，最终得：

$$\left(\begin{array}{c}{X}_0^T{X}_0,X_1^T{X}_0,X_2^T{X}_0\cdots ,X_n^T{X}_0\\ \\ X_0^T{X}_1,X_1^T{X}_1,X_2^T{X}_1\cdots ,X_n^T{X}_1\\ \\ X_0^T{X}_2,X_1^T{X}_2,X_2^T{X}_2\cdots ,X_n^T{X}_2\\ \\ \cdots \\ \\ X_0^T{X}_n,X_1^T{X}_n,X_2^T{X}_n\cdots ,X_n^T{X}_n\end{array}\right)\cdot \left(\begin{array}{c}{\theta }_0\\ \\ {\theta }_1\\ \\ {\theta }_2\\ \\ \cdots \\ \\ {\theta }_n\end{array}\right)=\left(\begin{array}{c}{X}_0^T\cdot y\\ \\ X_1^T\cdot y\\ \\ X_2^T\cdot y\\ \\ \cdots \\ \\ X_n^T\cdot y\end{array}\right)$$

左右两边进行处理，得（由(3)式子知 $X_a^T\cdot X_b=X_b^T\cdot X_a$，这里交换了一下）：
$$\left(\begin{array}{c}{X}_0^T\\ \\ X_1^T\\ \\ X_2^T\\ \\ \cdots \\ \\ X_n^T\end{array}\right)\cdot (X_0,X_1,X_2,\cdots ,X_n)\cdot \left(\begin{array}{c}{\theta }_0\\ \\ {\theta }_1\\ \\ {\theta }_2\\ \\ \cdots \\ \\ {\theta }_n\end{array}\right)=\left(\begin{array}{c}{X}_0^T\cdot y\\ \\ X_1^T\cdot y\\ \\ X_2^T\cdot y\\ \\ \cdots \\ \\ X_n^T\cdot y\end{array}\right)$$

将上式继续变换，得：
$$X^T\cdot X\cdot \theta =X^T\cdot y$$

两边同时乘 $(X^T\cdot X{)}^{-1}$ 得最终结果：
$$\theta =(X^T\cdot X{)}^{-1}\cdot X^T\cdot y$$

---

参考资料

考研必备数学公式大全: https://blog.csdn.net/zhaohongfei_358/article/details/106039576

机器学习纸上谈兵之线性回归: https://blog.csdn.net/zhaohongfei_358/article/details/117967229