6. 类神经网络训练不起来怎么办（三）Learning rate的影响

类神经网络训练不起来怎么办（三）

Loss不再下降不一定是卡在Critical point

不同的参数需要不同的Learning rate

• 平稳需要大一点的Learning rate
• 陡峭需要小一点的Learning rate

${\theta }_i^{t+1}\leftarrow {\theta }_i^t-\eta g_i^t$改成${\theta }_i^{t+1}\leftarrow {\theta }_i^t-\frac{\eta }{{\sigma }_i^t}{g}_i^t$

Root Mean Squre

${\sigma }_i^t=\sqrt{\frac{1}{t+1}{\sum }_{i=0}^t(g_i^t{)}^2}$ Used in Adagrad

RMSProp

${\sigma }_i^t=\sqrt{\alpha ({\sigma }_i^{t-1})+(1-\alpha )g_i^t}$

Adam: RMSProp+Mumentum

Learning rate scheduling

Learning rate decay,Warm up