优化梯度下降算法

Optimization problem

speed up the training of your neural network

Normalizing inputs

1. subtract mean

$$\begin{gathered} \mu =\frac{1}{m}\sum _{i=1}^m{x}^{(i)} \\ x:=x-\mu \end{gathered}$$

2. normalize variance

$$\begin{gathered} {\sigma }^2=\frac{1}{m}\sum _{i=1}^m(x^{(i)}{)}^2 \\ x/=\sigma \end{gathered}$$

vanishing/exploding gradients

$$\begin{gathered} y=w^{[l]}{w}^{[l-1]}...w^{[2]}{w}^{[1]}x \\ {w}^{[l]}>I\to (w^{[l]}{)}^L\to \infty \\ {w}^{[l]}<I\to (w^{[l]}{)}^L\to 0 \end{gathered}$$

weight initialize

$$\begin{gathered} var(w)=\frac{1}{n^{(l-1)}} \\ {w}^{[l]}=np.random.randn(shape)\ast np.sqrt(\frac{1}{n^{(l-1)}}) \end{gathered}$$

gradient check

Numerical approximation

$$\begin{gathered} f(\theta )={\theta }^3 \\ {f}^{\prime }(\theta )=\frac{f(\theta +\epsilon )-f(\theta -\epsilon )}{2\epsilon } \end{gathered}$$

grad check

$$\begin{gathered} d{\theta }_{approx}[i]=\frac{J({\theta }_1,...{\theta }_i+\epsilon ...)-J({\theta }_1,...{\theta }_i-\epsilon ...)}{2\epsilon }=d\theta [i] \\ check:\frac{\Vert d{\theta }_{approx}-d\theta {\Vert }_2}{\Vert d{\theta }_{approx}{\Vert }_2+\Vert d\theta {\Vert }_2}<10^{-7} \end{gathered}$$

Optimize algorithm

mini-bach gradient

$$\begin{gathered} [x^{(1)}...x^{(m)}]\to [x^{\{1\}}...x^{\{m/u\}}] \\ (anepoch:Forwardpropon{x}^{\{t\}}: \\ {z}^{[l]}=w^{[l]}{X}^{\{t\}}+b^{[l]} \\ {A}^{[l]}=g^{[l]}(z^{[l]}) \\ {J}^{\{t\}}=\frac{1}{1000}\sum _{i=1}^l\mathcal{L}({\hat{y}}^{(i)},y^{(i)})+\frac{\lambda }{2\ast size}\sum _l\Vert w^{[l]}{\Vert }_F^2 \\ Backwardprop \end{gathered}$$

mini-batch size

size = m -> Batch gradient descent <- small train set (<2000)

size = 1 -> stochastic gradient descent

typical mini-batch size (62,128,256…)

exponential weighted averages

$$
v_\theta = 0\
\theta_t\rightarrow v_\theta:=\beta v_{\theta-1}+(1-\beta)\theta_\theta\

$$

Bias correction

$$\frac{1}{1-\beta }\to \frac{v_t}{1-{\beta }^t}$$

Momentum

$$\begin{gathered} V_{dw}=\beta V_{dw}+(1-\beta )dw \\ {V}_{db}=\beta V_{db}+(1-\beta )db \\ w:=w-\alpha V_{dw} \end{gathered}$$

RMSprop

$$\begin{gathered} S_{dw}={\beta }_2{S}_{dw}+(1-{\beta }_2)d{w}^2 \\ {S}_{db}={\beta }_2{S}_{db}+(1-{\beta }_2)d{b}^2 \\ w:=w-\alpha \frac{dw}{{\sqrt{S}}_{dw}+\epsilon } \end{gathered}$$

Adam algorithm

$$\begin{gathered} V_{dw}=0,S_{dw}=0 \\ {V}_{dw}={\beta }_1{V}_{dw}+(1-{\beta }_1)dw \\ {V}_{db}={\beta }_1{V}_{db}+(1-{\beta }_1)db \\ {S}_{dw}={\beta }_2{S}_{dw}+(1-{\beta }_2)d{w}^2 \\ {S}_{db}={\beta }_2{S}_{db}+(1-{\beta }_2)d{b}^2 \\ {V}_{dw}^{correct}=\frac{v_{dw}}{1-{\beta }_1^t} \\ {S}_{dw}^{correct}=\frac{s_{dw}}{1-{\beta }_2^t} \\ W:=W-\alpha \frac{V_{dw}^{correct}}{\sqrt{S_{dw}^{correct}}+\epsilon } \\ {\beta }_1:0.9,{\beta }_2:0.999 \end{gathered}$$

Learning rate decay

$$\begin{gathered} \alpha =\frac{1}{1+decayRate\ast epochNumber}{\alpha }_0 \\ \alpha =\frac{k}{\sqrt{epochNum}}{\alpha }_0 \end{gathered}$$

Local optima