Adam优化算法理解与实现

引入

  用torch的时候，有个老几出现频率忒高，让不了解它的我蠢蠢欲动，这究竟是何方神圣？

1 Adam介绍

  最开始的GD很慢，每次只爱一个样本。SGD想要改变，又是随机，又是批量。后来加了自适应，何苦学习率开始基情满满，越往后越敷衍。
  RMSprop则不同了，生是让学习率一个劲往上
  Adam则是集大成者

  更新过程如下：
$$\begin{array}{rl}{v}_t& ={\beta }_1{v}_{t-1}+(1+{\beta }_1)gra{d}_t\\ s_t& ={\beta }_2{s}_{t-1}+(1-{\beta }_2)gra{d}_t^2\end{array}$$这里$v$和$s$是不同的动量，前者用于记录上一次的梯度，后者则保留RMSprop的特长。
  $v$和$s$偏导计算如下：
$$\begin{array}{rl}{v}_t^{\prime }& =\frac{v_t}{1-{\beta }_1^t}\\ s_t^{\prime }& =\frac{s_t}{1-{\beta }_2^t}\end{array}$$  最终的更新如下：
$$\begin{array}{rl}gra{d}_t^{\prime }& =\frac{lr\ast v_t^{\prime }}{\sqrt{s_t^{\prime }}+ϵ}\\ {\theta }_t& ={\theta }_{t-1}-gra{d}_t^{\prime }\end{array}$$

2 具体实现

  使用如下例子：
$$\begin{array}{rl}f(x)& =ax+b\\ (y-f(x){)}^2& =(y-(ax+b){)}^2\\ \frac{dy}{da}& =-2x(y-(ax+b))\\ \frac{dy}{db}& =-2(y-(ax+b))\end{array}$$

import numpy as np
import matplotlib.pyplot as plt
from sympy import symbols, diff


def get_data():

    ret_x = np.linspace(-1, 1, 100)
    return ret_x, [(lambda x: 2 * x + 3)(x) for x in ret_x]


def grad():

    x, y, a, b = symbols(["x", "y", "a", "b"])
    loss = (y - (a * x + b))**2
    return diff(loss, a), diff(loss, b)


def test2(n_iter=50, lr=0.1, batch_size=20, beta1=0.9, beta2=0.999, epsilon=1e-6, shuffle=True):
    x, y = get_data()
    ga, gb = grad()
    n = len(x)
    idx = np.random.permutation(n)
    s, v = 0, 0
    a, b = 0, 0
    move_a, move_b = [a], [b]
    move_lr_a, move_lr_b = [lr], [lr]
    t = 1
    for _ in range(n_iter):
        if shuffle:
            np.random.shuffle(idx)
        batch_idxes = [idx[k: k + batch_size] for k in range(0, n, batch_size)]
        for idxes in batch_idxes:
            sum_ga, sum_gb = 0, 0
            for j in idxes:
                sum_ga += ga.subs({
     "x": x[j], "y": y[j], "a": a, "b": b})
                sum_gb += gb.subs({
     "x": x[j], "y": y[j], "a": a, "b": b})
            sum_ga /= batch_size
            sum_gb /= batch_size
            g = np.array([sum_ga, sum_gb])

            v = beta1 * v + (1 - beta1) * g
            s = beta2 * s + (1 - beta2) * g * g

            v_norm = v / (1 - np.power(beta1, t))
            s_norm = s / (1 - np.power(beta2, t))
            t += 1

            lr_a, lr_b = lr * v_norm[0], lr * v_norm[1]
            move_lr_a.append(lr_a)
            move_lr_b.append(lr_b)

            g_a_norm = lr_a / (np.sqrt(float(s_norm[0])) + epsilon)
            g_b_norm = lr_b / (np.sqrt(float(s_norm[1])) + epsilon)

            a -= g_a_norm
            b -= g_b_norm
            move_a.append(a)
            move_b.append(b)

    plt.subplot(211)
    plt.plot(move_a)
    plt.plot(move_b)
    plt.legend(["a", "b"])
    plt.subplot(212)
    plt.plot(move_lr_a)
    plt.plot(move_lr_b)
    plt.legend(["a", "b"])
    plt.show()


if __name__ == '__main__':
    test2()