Coursera | Andrew Ng (02-week2-2.10)—局部最优的问题

该系列仅在原课程基础上部分知识点添加个人学习笔记，或相关推导补充等。如有错误，还请批评指教。在学习了 Andrew Ng 课程的基础上，为了更方便的查阅复习，将其整理成文字。因本人一直在学习英语，所以该系列以英文为主，同时也建议读者以英文为主，中文辅助，以便后期进阶时，为学习相关领域的学术论文做铺垫。- ZJ

Coursera 课程 |deeplearning.ai |网易云课堂

---

转载请注明作者和出处：ZJ 微信公众号-「SelfImprovementLab」

知乎：https://zhuanlan.zhihu.com/c_147249273

CSDN：http://blog.csdn.net/JUNJUN_ZHAO/article/details/79146876

---

2.10 The problem of local optima (局部最优的问题)

(字幕来源：网易云课堂)

In the early days of deep learning,people used to worry a lot about the optimization algorithm getting stuck in bad local optima.But as this theory of deep learning has advanced,our understanding of local optima is also changing.Let me show you how we now think about local optima and problems in the optimization problem in deep learning.This was a picture people used to have in mind when they worried about local optima.Maybe you are trying to optimize some set of parameters,we call them W1 and W2 ,and the height in the surface is the cost function.In this picture,it looks like there are a lot of local optima in all those places.And it’d be easy for gradient descent,or one of the other algorithms to get stuck in a local optimumrather than find its way to a global optimum.It turns out that if you are plotting a figurelike this in two dimensions,then it’s easy to create plots like thiswith a lot of different local optima.And these very low dimensional plotsused to guide their intuition.But this intuition isn’t actually correct.

在深度学习研究早期，人们总是担心优化算法，会困在极差的局部最优，不过随着深度学习理论不断发展，我们对局部最优的理解也发生了改变，我向你展示一下现在我们怎么看局部最优，以及深度学习中的优化问题，这是曾经人们在想到局部最优时，脑海里会出现的图，也许你想优化一些参数，我们把它们称之为 W1 和 W2 ，平面的高度就是损失函数，在图中，似乎各处都分布着局部最优，梯度下降法，或者某个算法可能困在一个局部最优中，而不会抵达全局最优，如果你要作图计算一个数字，比如说这两个维度，就容易出现，有多个不同局部最优的图，而这些低维的图，曾经影响了我们的理解，但是这些理解并不正确。

It turns out if you create a neural network,most points of zero gradients are not local optima like points like this.Instead most points of zero gradient in a cost function are saddle points.So, that’s a point where this t is maybe W1 , W2 ,and the height is the value of the cost function J.But informally, a function of very high dimensional space,if the gradient is zero,then in each directionit can either be a convex light function or a concave light function.And if you are in, say, a 20,000 dimensional space,then for it to be a local optima,all 20,000 directions need to look like this.And so the chance of that happening is maybe very small,maybe two to the minus 20,000.Instead you’re much more likely to get some directions where the curve bends up like so,as well as some directions where the curve function is bending downrather than have them all bend upwards.So that’s why in very high-dimensional spaces you’re actually much more likely to run into a saddle point like that shown on the right,then the local optimum.

事实上 如果你要创建一个神经网络，通常梯度为零的点，并不是这个图中的局部最优点，实际上成本函数的零梯度点，通常是鞍点，也就是在这个点，这里是 W1 W2 ，高度即成本函数 J 的值，但是一个具有高维空间的函数，如果梯度为0，那么在每个方向，它可能是凸函数，也可能是凹函数，如果你在 2万维空间中，那么要想得到局部最优，所有的 2万个方向都需要是这样，但发生的机率也许很小，也许是 2(−20000) ，你更有可能遇到，有些方向的曲线会这样向上弯曲，另一些方向曲线向下湾，而不是所有的都向上弯曲，因此在高维度空间，你更有可能碰到鞍点，就像右边的这种，而不会碰到局部最优。

As for why the surface is called a saddle point,if you can picture,maybe this is a sort of saddle you put on a horse, right?Maybe this is a horse.This is a head of a horse,this is the eye of a horse.Well, not a good drawing of a horse but you get the idea.Then you, the rider,will sit here in the saddle.So that’s why this point here,where the derivative is zero,that point is called a saddle point.There’s really the point on this saddlewhere you would sit, I guess,and that happens to have derivative zero.And so, one of the lessons we learned in history of deep learningis that a lot of our intuitions about low-dimensional spaces,like what you can plot on the left,they really don’t transfer to the very high-dimensional spacesthat any other algorithms are operating over.Because if you have 20,000 parameters,then J as your function over 20,000 dimensional vector,then you’re much more likely to see saddle points than local optimum.

至于为什么把一个平面叫作鞍点，你想象一下，这就像是放在，马背部的马鞍一样，如果这是马，这是马的头，这就是马的眼睛，画得不好 请多包涵，然后你是骑马的人，要坐在马鞍上，因此这里的这个，导数为 0 的点，这个点叫做鞍点，我想那确实是，你坐在马鞍上的那一点，而这里导数为0，所以我们从，深度学习历史中学到的一课就是，我们对低维度空间的大部分直觉，比如你可以画出左边的图，并不能应用到高维度空间中，适用于其他算法，因为如果你有 2万个参数，那么 J 函数有2万个维度向量，你更有可能遇到鞍点，而不是局部最优点。

If local optima aren’t a problem,then what is a problem?It turns out that plateaus can really slow down learning and a plateau is a region where the derivative is close to zero for a long time.So if you’re here,then gradient descents will move down the surface,and because the gradient is zero or near zero,the surface is quite flat.You can actually take a very long time, you know,to slowly find your way to maybe this point on the plateau.And then because of a random perturbation of left or right,maybe then finally I’m going to switch pen colors for clarity.Your algorithm can then find its way off the plateau.Let it take this very long slope offbefore it’s found its way hereand they could get off this plateau.So the takeaways from this video are,first, you’re actually pretty unlikely to get stuck in bad local optima,so long as you’re training a reasonably large neural network,so a lot of parameters,and the cost function J is defined over a relatively high dimensional space.But second, that plateaus are a problem and you can actually make learning pretty slow.And this is where algorithms like momentum or RmsProp or Adamcan really help your learning algorithm as well.And these are scenarios where more sophisticated observation algorithms,such as Adam,can actually speed up the rateat which you could move down the plateau and then get off the plateau.

如果局部最优不是问题，那么问题是什么？结果是平稳段会减缓学习，平稳段是一块区域，其中导数长时间接近于 0，如果你在此处，梯度会从平面从上向下下降，因为梯度等于或接近 0，平面很水平，你得花上很长时间，慢慢抵达平稳段的这个点，因为左边或右边的随机扰动，我换个笔墨颜色 大家看得清楚一些，然后你的算法能够走出平稳段，我们可以沿着这段长坡走，直到这里，然后走出平稳段，所以此次视频的要点是，首先 你不太可能，困在极差的局部最优中，条件是你在训练较大的神经网络，存在大量参数，并且成本函数 J，被定义在较高的维度空间，第二点 平稳段是一个问题，这样使得学习十分缓慢，这也是像 Momentum，或是 RmsProp Adam 这样的算法，能够加速学习算法的地方，在这些情况下，更成熟的观察算法，如 Adam 算法，能够加快速度，让你尽早往下走出平稳段。

So because your network is solving optimizations problems over such high dimensional spaces, to be honest,I don’t think anyone has great intuitions about what these spaces really look like,and our understanding of them is still evolving.But I hope this gives you some better intuition about the challenges that the optimization algorithms may face.So that’s congratulations on coming to the end of this week’s content.Please take a look at this week’s quiz as well as the exercise.I hope you enjoy practicing some of these ideas of this week exercise,and I look forward to seeing you at the start of next week’s videos.

因为你的网络要解决优化问题，说实话 要面临如此之高的维度空间，我觉得没有人有那么好的直觉，知道这些空间长什么样，而且我们对它们的理解还在不断发展，不过我希望这一点能够让你更好地理解，优化算法所面临的问题，恭喜你学会了，本周所有的内容，麻烦你做一下本周的小测以及练习，希望你能学以致用并乐在其中，期待你观看下周的视频。

---

重点总结：

局部最优问题

在低纬度的情形下，我们可能会想象到一个Cost function 如左图所示，存在一些局部最小值点，在初始化参数的时候，如果初始值选取的不得当，会存在陷入局部最优点的可能性。

但是，如果我们建立一个神经网络，通常梯度为零的点，并不是如左图中的局部最优点，而是右图中的鞍点（叫鞍点是因为其形状像马鞍的形状）。

在一个具有高维度空间的函数中，如果梯度为 0，那么在每个方向，Cost function 可能是凸函数，也有可能是凹函数。但如果参数维度为 2万维，想要得到局部最优解，那么所有维度均需要是凹函数，其概率为 2−20000 ，可能性非常的小。也就是说，在低纬度中的局部最优点的情况，并不适用于高纬度，我们在梯度为 0 的点更有可能是鞍点。

在高纬度的情况下：

• 几乎不可能陷入局部最小值点；
• 处于鞍点的停滞区会减缓学习过程，利用如 Adam 等算法进行改善。

参考文献：

[1]. 大树先生.吴恩达Coursera深度学习课程 DeepLearning.ai 提炼笔记（2-2）– 优化算法

---

PS: 欢迎扫码关注公众号：「SelfImprovementLab」！专注「深度学习」，「机器学习」，「人工智能」。以及 「早起」，「阅读」，「运动」，「英语 」「其他」不定期建群 打卡互助活动。