深度学习-线性回归推导

线性表示

$$f(x)={\theta }_0+{\theta }_1{x}_1+\cdots +{\theta }_n{x}_n$$

记作
$$f(x)=\left[\begin{array}{cccc}{\theta }_0& {\theta }_1& \cdots & {\theta }_n\end{array}\right]\times \left[\begin{array}{c}1\\ x_1\\ ⋮\\ x_n\end{array}\right]={\theta }^T\mathbf{x}$$
考虑误差
$$f(x)={\theta }^T\mathbf{x}+{ϵ}^{}$$
对于每一组数据
$$y_i={\theta }^T{x}_i+{ϵ}_i$$
其中误差
$${ϵ}_i=y_i-{\theta }^T{x}_1$$

高斯曲线

$$f(x)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$$

• $\mu$：期望
• $\sigma$：标准差
• ${\sigma }^2$：方差

一般认为误差$ϵ$服从均值为零$\mu =0$的高斯分布
$$f(x)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{ϵ}^2}{2{\sigma }^2}}$$
将$(5)$带入
$$f(x)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(y_i-{\theta }^T{x}_i)}^2}{2{\sigma }^2}}$$
就统计而言，这个描述了数值的分布，对后续预测进行指导，那就是概率$P$了。

同时，强调$\theta$的作用关系，也就是条件概率，我们这样进行表示
$$P(y_i|x_i;\theta )=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(y_i-{\theta }^T{x}_i)}^2}{2{\sigma }^2}}$$

似然函数

$$L(\theta )=\stackrel{n}{\prod _{i=1}}p(y_i|x_i;\theta )=\stackrel{n}{\prod _{i=1}}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(y_i-{\theta }^T{x}_i)}^2}{2{\sigma }^2}}$$

为了简化计算，我们进行对数操作
$$\begin{gathered} \log (L(\theta ))=\stackrel{m}{\sum _{i=1}}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(y_i-{\theta }^T{x}_i)}^2}{2{\sigma }^2}} \\ =m\log \frac{1}{\sqrt{2\pi }\sigma }-\frac{1}{{\sigma }^2}\cdot \frac{1}{2}\stackrel{m}{\sum _{i=1}}{\left(y_i-{\theta }^T{x}_i\right)}^2 \end{gathered}$$
为了求得最大概率，也就是求解这个式子的极值，抛开固定的值，不定项为
$$\begin{gathered} J(\theta )=\frac{1}{2}\stackrel{m}{\sum _{i=1}}{\left(f_{\theta (x_i)}-y_i\right)}^2 \\ =\frac{1}{2}(X\theta -y{)}^T(X\theta -y) \end{gathered}$$
求导推算
$$\begin{gathered} {▽}_{\theta }J(\theta )={▽}_{\theta }(\frac{1}{2}(X\theta -y{)}^T(X\theta -y)) \\ ={▽}_{\theta }(\frac{1}{2}({\theta }^T{X}^T-y)(X\theta -y)) \\ ={▽}_{\theta }(\frac{1}{2}({\theta }^T{X}^{T}X\theta -{\theta }^T{X}^{T}y-y^{T}X\theta +y^{T}y)) \\ =\frac{1}{2}(X^{T}X\theta -X^{T}y-(y^{T}X{)}^T) \\ =X^{T}X\theta -X^{T}y \end{gathered}$$
令导数为零，求解$\theta$
$$\begin{gathered} X^{T}X\theta -X^{T}y=0 \\ \Rightarrow X^{T}X\theta =X^{T}y \\ \Rightarrow \theta =(X^{T}X{)}^{-1}{X}^{T}y \end{gathered}$$