笔记：ML-LHY-1 Regression Gradient Descent

回归问题

损失函数

$$\mathrm{L}(w,b)=\sum _{n=1}^N{\left({\hat{y}}_n-\left(b+w\cdot x_n\right)\right)}^2$$
更一般的有($W=(w_1,w_2,...,w_i)$)：
$$\mathrm{L}(W,b)=\sum _{n=1}^N{\left({\hat{y}}_n-\left(b+\sum _{i=1}^K{w}_i\cdot x_n\right)\right)}^2$$
N为训练数据量，K为优化类别数目
而对于多次则有(类别设为1种)：
$$\mathrm{L}(W,b)=\sum _{n=1}^N{\left({\hat{y}}_n-\left(b+\sum _{i=1}^K\sum _{j=1}^M{w}_{ij}\cdot x_n^j\right)\right)}^2$$
M为模型次数，次数过高会导致对权重更敏感，所以添加正则化项：
$$\mathrm{L}(W,b)=\sum _{n=1}^N{\left({\hat{y}}_n-\left(b+\sum _{i=1}^K\sum _{j=1}^M{w}_{ij}\cdot x_n^j\right)\right)}^2+\lambda \sum _{i=1}^K\sum _{j=1}^M{\left(w_{ij}\right)}^2$$
需要注意的是正则化项不需要添加偏置项，其目的只为了是函数更平滑，偏置项bias只会上下移动

梯度下降

按最简单形式，即一元(不包括偏置项)一次函数：
$$\mathrm{L}(w,b)=\sum _{n=1}^N{\left({\hat{y}}_n-\left(b+w\cdot x_n\right)\right)}^2$$

$$\frac{\partial L}{\partial w}=\sum _{n=1}^{10}2\left({\hat{y}}_n-\left(b+w\cdot x_n\right)\right)\left(-x_n\right)$$

$$\frac{\partial L}{\partial b}=\sum _{n=1}^{10}2\left({\hat{y}}_n-\left(b+w\cdot x_n\right)\right)$$

---

$$\begin{gathered} \text{ Compute }{{\frac{\partial L}{\partial w}|}_{w=w^0,b=b^0,}\frac{\partial L}{\partial b}|}_{w=w^0,b=b^0} \\ {w}^1\leftarrow w^0-{\eta \frac{\partial L}{\partial w}|}_{w=w^0,b=b^0}{b}^1\leftarrow b^0-{\eta \frac{\partial L}{\partial b}|}_{w=w^0,b=b^0} \end{gathered}$$

---

$$\begin{array}{rl}& \text{ Compute }{{\frac{\partial L}{\partial w}|}_{w=w^1,b=b^1,}\frac{\partial L}{\partial b}|}_{w=w^1,b=b^1}\\ & w^2\leftarrow w^1-{\eta \frac{\partial L}{\partial w}|}_{w=w^1,b=b^1}{b}^2\leftarrow b^1-{\eta \frac{\partial L}{\partial b}|}_{w=w^1,b=b^1}\end{array}$$

---

对应上面，更一般的形式：
$$\nabla L(W)=[\frac{\partial L}{\partial w_0},\frac{\partial L}{\partial w_1},...,\frac{\partial L}{\partial w_i}{]}^T$$
$$W^k=W^{k-1}+\alpha \nabla L\left(W^{k-1}\right)$$

以上参考李宏毅老师视频和ppt，仅作为学习笔记交流使用

代码

import numpy as np
import matplotlib.pyplot as plt

if $y=2+0.5x$

def load_dataset(n):
    k = 0.5
    b = 20
    noise = np.random.rand(n)
    X = [x for x in range(n)]
    y = [(k * X[i]  + b + noise[i]) for i in range(n)]
    return np.array(X).T, np.array(y).T

x, y = load_dataset(20)
plt.ylim(0, 50)
plt.xlim(0, 20)
plt.scatter(x, y)

假设损失函数

$loss(w)=\frac{1}{2m}{\sum }_{i=1}^m(w_1+w_2{x}_i-y_i{)}^2$

先对$w_1$求偏导

$w_1=w_1-\alpha \frac{1}{m}{\sum }_{i=1}^m(w_1+w_2{x}_i-y_i)$

对$w_2$求偏导

$w_2=w_2-\alpha \frac{1}{m}{\sum }_{i=1}^m(w_1+w_2{x}_i-y_i)x_i$

程序计算时，求每次的

$\frac{1}{m}{\sum }_{i=1}^m(w_1+w_2{x}_i-y_i)$

def calc_loss(x,y,w1,w2):
    J = 0
    for i in range(len(x)):
        mse = (w1 + x[i]*w2 -y[i])**2
        J += mse
    return J / (2*len(x))

loss = 10000000000
min_loss = 0.0001
w1 = 0;
w2 = 0;
m = len(x)
alpha = 0.1 # 学习率
max_itc = 100000
itc = 0
loss = calc_loss(x, y , w1, w2)
loss_pre = loss  + min_loss+ 1
loss_array = [loss]
while abs(loss - loss_pre) > min_loss and itc < max_itc:
    # g1
    g1 = 0
    for i in range(m):
        g1 = g1 + w1 + w2 * x[i]  - y[i]
    g1 = g1 / m
    w1_ = w1 - alpha * g1
#     print(w1_)
    
     # g2
    g2 = 0
    for i in range(m):
        g2 = g2 = (w1  + w2 * x[i]  - y[i]) * x[i]
    g2 = g2 / m
    w2_ = w2 - alpha * g2
    
    w1 = w1_
    w2 = w2_
    
    #loss
    loss_pre = loss
    loss = calc_loss(x, y , w1, w2)
    loss_array.append(loss)
#     print(loss)
    itc += 1

# loss_array

plt.plot(range(len(loss_array)), loss_array)

w1, w2

(20.683619269764357, 0.4669721730697502)