线性回归推导（三）--梯度下降法及纯python实现

1、梯度下降法

假设函数：
$$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_n{x}_n=\sum _{i=0}^n{\theta }_i{x}_i(x_0=1)$$
代价函数：
$$\begin{array}{rl}J(\theta )& =\frac{1}{2}\sum _{i=1}^m[h_{\theta }(x^{(i)})-y^{(i)}{]}^2\\ & =\frac{1}{2}\sum _{i=1}^m[{\theta }^T{x}^{(i)}-y^{(i)}{]}^2\end{array}$$
梯度下降公式：
$$\theta :=\theta -\alpha \frac{\partial J(\theta )}{\partial \theta }$$
则$\theta$的推导过程为
$$\begin{array}{rl}\frac{\partial J(\theta )}{\partial \theta }& =\frac{1}{2}\frac{\partial }{\partial \theta }\sum _{i=1}^m[h_{\theta }(x^{(i)})-y^{(i)}{]}^2\\ & =\frac{1}{2}\frac{\partial }{\partial \theta }[({\theta }^T{x}^{(1)}-y^{(1)}{)}^2+({\theta }^T{x}^{(2)}-y^{(2)}{)}^2+...+({\theta }^T{x}^{(m)}-y^{(m)}{)}^2]\\ & =\frac{1}{2}\cdot 2\sum _{i=1}^m[({\theta }^T{x}^{(i)}-y^{(i)})\cdot \frac{\partial }{\partial \theta }({\theta }^T{x}^{(i)}-y^{(i)})]\\ & =\sum _{i=1}^m[({\theta }^T{x}^{(i)}-y^{(i)})\cdot \frac{\partial }{\partial \theta }({\theta }^T{x}^{(i)})]\\ & =\sum _{i=1}^m[({\theta }^T{x}^{(i)}-y^{(i)})\cdot x^{(i)}]\\ & =\sum _{i=1}^m[(h_{\theta }(x^{(i)})-y^{(i)})\cdot x^{(i)}](error\ast feature)\end{array}$$
故
$$\theta :=\theta -\alpha \sum _{i=1}^m[(h_{\theta }(x^{(i)})-y^{(i)})\cdot x^{(i)}]$$

2、python实现

代码如下

import numpy as np
import matplotlib.pyplot as plt
import time


# 加载数据
def load_data():
    X = [2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013]
    X_p = np.array(X)
    Y = [2.000, 2.500, 2.900, 3.147, 4.515, 4.903, 5.365, 5.704, 6.853, 7.971, 8.561, 10.000, 11.280, 12.900]
    Y_p = np.array(Y)
    return X_p, Y_p


# 梯度下降法求解线性回归
def gradient_descent(X, Y, learn_rate = 0.15):
    theta0, theta1 = 0, 0
    loss, temploss = 1, 0
    y, iter_  = 0, 0
    lenth = len(X)
    loss_list = []
    iteration = []

    # 均值方差归一化
    mean = np.mean(X)
    variance = np.std(X)
    X_new = (X - mean)/variance

    # 梯度下降，以loss变化小于1e-10为迭代停止条件
    while abs(loss-temploss)>1e-10:
        iter_ += 1
        temploss = loss
        loss=0
        gradient0, gradient1 = 0, 0
        # 计算梯度
        for i in range(lenth):
            gradient0 += theta0 + theta1 * X_new[i] - Y[i]
            gradient1 += (theta0 + theta1 * X_new[i] - Y[i]) * X_new[i]
        # 参数更新
        theta0 = theta0 - learn_rate * gradient0
        theta1 = theta1 - learn_rate * gradient1
        # 动态学习率
        learn_rate = learn_rate * 0.99
        # 计算损失
        for i in range(lenth):
            loss += (theta0 + theta1 * X_new[i] - Y[i]) ** 2
        loss_list.append(loss)
        iteration.append(iter_)
        # print(loss)
        y = X_new * theta1 + theta0
    # 预测2014年
    x = (2014 - mean)/variance
    print("the housing price in 2014 is %f"%(x * theta1 + theta0))


if __name__ == "__main__":
    X, Y = load_data()
    print("\n--------------gradient descent----------------")
    gradient_descent(X, Y)

拟合过程如下

(自己学习机器学习的笔记，如有错误望提醒修正）