PyTorch深度学习实践——3.梯度下降＆随机梯度下降
PyTorch深度学习实践——3.梯度下降＆随机梯度下降
课程链接：《PyTorch深度学习实践》3.梯度下降算法
梯度下降（Gradient Descent）算法：
w按梯度下降方向移动，这样一定次数的移动后就会移动到最优解。
（a为学习因子，影响每次移动的步长，越小越精确但时间复杂度也会变高）
通过求导，可以求出具体的表达式，根据表达式就可以写出代码。
梯度下降 Gradient_Descent
import matplotlib.pyplot as plt
# 自定义数据集
# prepare the training set
x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]
w = 1.0 # 猜测初始权重
# initial guess of weight
#定义权重W
# 定义模型（前馈forward）
# define the model linear model y = w*x
#定义前馈计算 y_hat = X * W
def forward(x):
return x * w # y_hat = x * w
# 计算 MSE（Mean Square Error）
#define the cost function MSE
# 定义损失函数
#定义成本函数 cost
def cost(xs, ys):
# '输入：x_data,y_data;输出：损失cost = (y_hat - y)的平方累加再求均值'
cost = 0 # 统计损失值
for x, y in zip(xs, ys):
y_pred = forward(x) # 计算 y_hat = x * w
# 计算预测的y值
cost += (y_pred - y) ** 2 # 计算每一组数据的损失值 cost = (y_hat - y) ^ 2，并求和
# 计算损失函数
return cost / len(xs) # 计算平均损失
# 除以样本数量求均值
# 返回平均损失值
# 计算梯度
# define the gradient function gd
# 定义梯度函数
def gradient(xs, ys):
#'求梯度：cost对w求导 '
grad = 0 # 累加后求均值
for x, y in zip(xs, ys):
grad += 2 * x * (x * w - y) # 梯度: 损失 loss 的导数
# 对损失函数求导得到梯度，针对x和y取不同值有一个累加过
return grad / len(xs) # 乘 1/N = 除以长度
# 返回整个验证集中所产生的所有梯度迭代值
# 保存图像中x轴（权重）和y轴（损失）的值
w_list = [] #保存epoch w
cost_val_list = [] #保存每个epoch对应的cost
print('Predict (before training)', 4, forward(4)) # 训练前 x = 4 时的 y_hat 值 (y_hat = 4 * w)
# 对x=4时y的值进行预测
# 训练权重w的过程
epoch_list =[] #保存epoch
# 开始迭代
for epoch in range(100): # 进行100轮的训练
cost_val = cost(x_data, y_data) # 计算平均成本（损失值），为输出和绘图
# 损失验证值和梯度验证值，其中损失验证值是为了画图
#第一步：先算当前步的损失值（cost）
grad_val = gradient(x_data, y_data) # 求梯度
#第二步：计算梯度（gradient）
w -= 0.01 * grad_val # 更新（训练）Update: 权重w = w - 学习率a * 梯度grad_val
# 0.01 learning rate
w_list.append(w) # 记录每次训练变化的权重
cost_val_list.append(cost_val) # 记录不同权重时的损失
print('Epoch:', epoch, 'w=', w, 'loss=', cost_val) # 输出每轮训练的日志
print("Epoch:",epoch,"w ={:.3f}".format(w),"loss ={:.3f}".format(cost_val))
#对输出进行格式调整
epoch_list.append(epoch) # 保存0~99，100个轮数
print('Predict (after training)', 4, forward(4)) # 训练后 x = 4 时的 y_hat 值 (y_hat = 4 * w)
# 画出每一轮训练的损失值cost图像
plt.plot(epoch_list, cost_val_list) # x轴为Epoch、y轴为每一轮的损失值
# == plt.plot(list(range(100)), cost_val_list)
plt.ylabel('Cost')
plt.xlabel('Epoch')
plt.grid(ls='--') # 生成网格
plt.show()
输出结果：
Predict (before training) 4 4.0
Epoch: 0 w= 1.0933333333333333 loss= 4.666666666666667
Epoch: 1 w= 1.1779555555555554 loss= 3.8362074074074086
Epoch: 2 w= 1.2546797037037036 loss= 3.1535329869958857
Epoch: 3 w= 1.3242429313580246 loss= 2.592344272332262
Epoch: 4 w= 1.3873135910979424 loss= 2.1310222071581117
Epoch: 5 w= 1.4444976559288012 loss= 1.7517949663820642
Epoch: 6 w= 1.4963445413754464 loss= 1.440053319920117
Epoch: 7 w= 1.5433523841804047 loss= 1.1837878313441108
Epoch: 8 w= 1.5859728283235668 loss= 0.9731262101573632
Epoch: 9 w= 1.6246153643467005 loss= 0.7999529948031382
Epoch: 10 w= 1.659651263674342 loss= 0.6575969151946154
Epoch: 11 w= 1.6914171457314033 loss= 0.5405738908195378
Epoch: 12 w= 1.7202182121298057 loss= 0.44437576375991855
Epoch: 13 w= 1.7463311789976905 loss= 0.365296627844598
Epoch: 14 w= 1.7700069356245727 loss= 0.3002900634939416
Epoch: 15 w= 1.7914729549662791 loss= 0.2468517784170642
Epoch: 16 w= 1.8109354791694263 loss= 0.2029231330489788
Epoch: 17 w= 1.8285815011136133 loss= 0.16681183417217407
Epoch: 18 w= 1.8445805610096762 loss= 0.1371267415488235
Epoch: 19 w= 1.8590863753154396 loss= 0.11272427607497944
Epoch: 20 w= 1.872238313619332 loss= 0.09266436490145864
Epoch: 21 w= 1.8841627376815275 loss= 0.07617422636521683
Epoch: 22 w= 1.8949742154979183 loss= 0.06261859959338009
Epoch: 23 w= 1.904776622051446 loss= 0.051475271914629306
Epoch: 24 w= 1.9136641373266443 loss= 0.04231496130368814
Epoch: 25 w= 1.9217221511761575 loss= 0.03478477885657844
Epoch: 26 w= 1.9290280837330496 loss= 0.02859463421027894
Epoch: 27 w= 1.9356521292512983 loss= 0.023506060193480772
Epoch: 28 w= 1.9416579305211772 loss= 0.01932302619282764
Epoch: 29 w= 1.9471031903392007 loss= 0.015884386331668398
Epoch: 30 w= 1.952040225907542 loss= 0.01305767153735723
Epoch: 31 w= 1.9565164714895047 loss= 0.010733986344664803
Epoch: 32 w= 1.9605749341504843 loss= 0.008823813841374291
Epoch: 33 w= 1.9642546069631057 loss= 0.007253567147113681
Epoch: 34 w= 1.9675908436465492 loss= 0.005962754575689583
Epoch: 35 w= 1.970615698239538 loss= 0.004901649272531298
Epoch: 36 w= 1.9733582330705144 loss= 0.004029373553099482
Epoch: 37 w= 1.975844797983933 loss= 0.0033123241439168096
Epoch: 38 w= 1.9780992835054327 loss= 0.0027228776607060357
Epoch: 39 w= 1.980143350378259 loss= 0.002238326453885249
Epoch: 40 w= 1.9819966376762883 loss= 0.001840003826269386
Epoch: 41 w= 1.983676951493168 loss= 0.0015125649231412608
Epoch: 42 w= 1.9852004360204722 loss= 0.0012433955919298103
Epoch: 43 w= 1.9865817286585614 loss= 0.0010221264385926248
Epoch: 44 w= 1.987834100650429 loss= 0.0008402333603648631
Epoch: 45 w= 1.9889695845897222 loss= 0.0006907091659248264
Epoch: 46 w= 1.9899990900280147 loss= 0.0005677936325753796
Epoch: 47 w= 1.9909325082920666 loss= 0.0004667516012495216
Epoch: 48 w= 1.9917788075181404 loss= 0.000383690560742734
Epoch: 49 w= 1.9925461188164473 loss= 0.00031541069384432885
Epoch: 50 w= 1.9932418143935788 loss= 0.0002592816085930997
Epoch: 51 w= 1.9938725783835114 loss= 0.0002131410058905752
Epoch: 52 w= 1.994444471067717 loss= 0.00017521137977565514
Epoch: 53 w= 1.9949629871013967 loss= 0.0001440315413480261
Epoch: 54 w= 1.9954331083052663 loss= 0.0001184003283899171
Epoch: 55 w= 1.9958593515301082 loss= 9.733033217332803e-05
Epoch: 56 w= 1.9962458120539648 loss= 8.000985883901657e-05
Epoch: 57 w= 1.9965962029289281 loss= 6.57716599593935e-05
Epoch: 58 w= 1.9969138906555615 loss= 5.406722767150764e-05
Epoch: 59 w= 1.997201927527709 loss= 4.444566413387458e-05
Epoch: 60 w= 1.9974630809584561 loss= 3.65363112808981e-05
Epoch: 61 w= 1.9976998600690001 loss= 3.0034471708953996e-05
Epoch: 62 w= 1.9979145397958935 loss= 2.4689670610172655e-05
Epoch: 63 w= 1.9981091827482769 loss= 2.0296006560253656e-05
Epoch: 64 w= 1.9982856590251044 loss= 1.6684219437262796e-05
Epoch: 65 w= 1.9984456641827613 loss= 1.3715169898293847e-05
Epoch: 66 w= 1.9985907355257035 loss= 1.1274479219506377e-05
Epoch: 67 w= 1.9987222668766378 loss= 9.268123006398985e-06
Epoch: 68 w= 1.9988415219681517 loss= 7.61880902783969e-06
Epoch: 69 w= 1.9989496465844576 loss= 6.262999634617916e-06
Epoch: 70 w= 1.9990476795699081 loss= 5.1484640551938914e-06
Epoch: 71 w= 1.9991365628100501 loss= 4.232266273994499e-06
Epoch: 72 w= 1.999217150281112 loss= 3.479110977946351e-06
Epoch: 73 w= 1.999290216254875 loss= 2.859983851026929e-06
Epoch: 74 w= 1.9993564627377531 loss= 2.3510338359374262e-06
Epoch: 75 w= 1.9994165262155628 loss= 1.932654303533636e-06
Epoch: 76 w= 1.999470983768777 loss= 1.5887277332523938e-06
Epoch: 77 w= 1.9995203586170245 loss= 1.3060048068548734e-06
Epoch: 78 w= 1.9995651251461022 loss= 1.0735939958924364e-06
Epoch: 79 w= 1.9996057134657994 loss= 8.825419799121559e-07
Epoch: 80 w= 1.9996425135423248 loss= 7.254887315754342e-07
Epoch: 81 w= 1.999675878945041 loss= 5.963839812987369e-07
Epoch: 82 w= 1.999706130243504 loss= 4.902541385825727e-07
Epoch: 83 w= 1.9997335580874436 loss= 4.0301069098738336e-07
Epoch: 84 w= 1.9997584259992822 loss= 3.312926995781724e-07
Epoch: 85 w= 1.9997809729060159 loss= 2.723373231729343e-07
Epoch: 86 w= 1.9998014154347876 loss= 2.2387338352920307e-07
Epoch: 87 w= 1.9998199499942075 loss= 1.8403387118941732e-07
Epoch: 88 w= 1.9998367546614149 loss= 1.5128402140063082e-07
Epoch: 89 w= 1.9998519908930161 loss= 1.2436218932547864e-07
Epoch: 90 w= 1.9998658050763347 loss= 1.0223124683409346e-07
Epoch: 91 w= 1.9998783299358769 loss= 8.403862850836479e-08
Epoch: 92 w= 1.9998896858085284 loss= 6.908348768398496e-08
Epoch: 93 w= 1.9998999817997325 loss= 5.678969725349543e-08
Epoch: 94 w= 1.9999093168317574 loss= 4.66836551287917e-08
Epoch: 95 w= 1.9999177805941268 loss= 3.8376039345125727e-08
Epoch: 96 w= 1.9999254544053418 loss= 3.154680994333735e-08
Epoch: 97 w= 1.9999324119941766 loss= 2.593287985380858e-08
Epoch: 98 w= 1.9999387202080534 loss= 2.131797981222471e-08
Epoch: 99 w= 1.9999444396553017 loss= 1.752432687141379e-08
Predict (after training) 4 7.999777758621207
(结果应该是收敛的，如果不收敛可能是a值过大)
随机梯度下降法在神经网络中被证明是有效的。效率较低(时间复杂度较高)，学习性能较好。
梯度求导
import torch
A = torch.arange(4., requires_grad=True) # 将A的梯度存储到grad这里,[0,1,2,3]，将A设置为可导
B = 2 * torch.dot(A, A) # 将A与A内积，再乘以2；即2乘以x的平方
B.backward() # 利用反向传播函数进行梯度求导，应为4乘以x
print(A)
print(B)
print(A.grad) # 对A求导
B = A.sum()
A.grad.zero_() # 为什么这个不添加的话，A.grad的输出是([1,5,9,13])
B.backward()
print(B)
print(A.grad) # 查看对A求导得结果
x = torch.tensor([1., 2., 3.], requires_grad=True)
y = torch.pow(x, 2)
gradient = torch.tensor([1.0, 1.0, 0.004])# 与上述X变量的格式一一对应，将[1,2,3]的权重分别设置为1,1,0.004
y.backward(gradient)
print(x.grad)
随机梯度下降（Stochastic Gradient Descent )
有N个样本，每一次迭代中随机选择一个样本来求梯度值进行权重更新。不再累加求平均值。
优点：效率低，性能好
缺点：时间复杂度高
类似梯度下降，但是这里用的是随机某个样本（而不是整体）的梯度。
这样的好处是由于单个样本一般有噪声，具有随机性，可能帮助走出鞍点从而进入最优解。
坏处是计算依赖上次结果，多个样本x无法并行，时间复杂度高。因此会有一个中间的方法，Mini-Batch(或称Batch)。将若干个样本点分成一组，每次用一组来更新w。
随机梯度下降法相对于梯度下降法的改进在于，不再取所有x、y值计算出的损失函数的平均值，而是随机取一组x和y计算损失函数。此方法的作用在于：数据中大多含有随机噪声，则利用这种随机噪声避免鞍点时无法继续迭代的情况。
随机梯度下降法和梯度下降法的主要区别在于：
1、损失函数由cost()更改为loss()。cost是计算所有训练数据的损失，loss是计算一个训练函数的损失。对应于源代码则是少了两个for循环。
2、梯度函数gradient()由计算所有训练数据的梯度更改为计算一个训练数据的梯度。
3、本算法中的随机梯度主要是指，每次拿一个训练数据来训练，然后更新梯度参数。本算法中梯度总共更新100(epoch)x3 = 300次。梯度下降法中梯度总共更新100(epoch)次。
随机梯度下降 Stochastic_Gradient_Descent
import matplotlib.pyplot as plt
x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]
w = 1.0 # 设置初始权重
#定义前馈计算 y_hat = X * W
def forward(x): # 定义前馈函数
return x * w
# calculate loss function
#mse,单个样本
# 定义损失函数，此时不需要将x和y取所有值，故不需要循环，只需要随机取一个x和y即可
#定义成本函数 cost
def loss(x, y): # 只求一个样本的损失
y_pred = forward(x) # 求 y_hat
#随机抽取一个样本进行正向传播，以该样本的梯度代替所有样本的梯度进行梯度下降
return (y_pred - y) ** 2 # 求损失 loss = (y_hat - y) ^ 2
# 此时不是平均损失，只是随机损失
# define the gradient function sgd
#梯度,单个样本
# 定义梯度函数
def gradient(x, y):
return 2 * x * (x * w - y) # 梯度: 损失 loss 的导数
# 此时也不是平均梯度，只是随机梯度
w_list = [] #保存
l_list = [] #保存
print('Predict (before training)', 4, forward(4))
epoch_list = [] #保存epoch
for epoch in range(100): # 训练100轮（x轴）
l = 0
for x, y in zip(x_data, y_data): # x和y在范围中取值（数据多的话应该要体现随机性，现在数据太少），此处并未体现出随机，而是把每一对x和y的值都取到了，此处的x和y并不是自由搭配，而是一一对应关系
grad = gradient(x, y) # 对每一个样本求梯度 (损失loss对权重w求导数(求梯度): dloss/dw)
# update weight by every grad of sample of training set
w -= 0.01 * grad # 更新权重w
# 核心算式，此步骤计算最后要输出的w
print("\tgrad:", x, y, grad)
l = loss(x, y) # 记录当前损失（y轴）
w_list.append(w)
l_list.append(l)
epoch_list.append(epoch) # 这两个语句只能放在epoch的循环中，而不能放在x，y的循环中因为若放在x，y的循环中，则epoch的值不变时，l的值却在不断变化，在图像上形成很多竖直线
print('progress:', epoch, 'w=', w, 'loss=', l) # 此时的l和w会固定取最后一次循环中的l和w，相当于固定x=3,y=6，并不随机
# 对输出进行格式调整
print("Progress Epoch:", epoch, "w ={:.3f}".format(w), "loss ={:.3f}".format(l))
print('Predict (after training)', 4, forward(4))
# plt.plot(w_list, l_list)
plt.plot(epoch_list, l_list)
plt.ylabel('Cost')
plt.xlabel('Epoch')
plt.grid(ls='--') # 生成网格
plt.show()
此代码并没有体现随机性，利用zip函数把每一对x和y值都计算了一次（若是输入数据很多则计算量极大），此外，计算出的l和w每次都利用x和y循环中最后一次的结果，即固定取x=3,y=6，与随机性没有任何关系，同时造成前两对x和y值的计算完全无效，属于浪费资源。
输出结果：
Predict (before training) 4 4.0
grad: 1.0 2.0 -2.0
grad: 2.0 4.0 -7.84
grad: 3.0 6.0 -16.2288
progress: 0 w= 1.260688 loss= 4.919240100095999
grad: 1.0 2.0 -1.478624
grad: 2.0 4.0 -5.796206079999999
grad: 3.0 6.0 -11.998146585599997
progress: 1 w= 1.453417766656 loss= 2.688769240265834
grad: 1.0 2.0 -1.093164466688
grad: 2.0 4.0 -4.285204709416961
grad: 3.0 6.0 -8.87037374849311
progress: 2 w= 1.5959051959019805 loss= 1.4696334962911515
grad: 1.0 2.0 -0.8081896081960389
grad: 2.0 4.0 -3.1681032641284723
grad: 3.0 6.0 -6.557973756745939
progress: 3 w= 1.701247862192685 loss= 0.8032755585999681
grad: 1.0 2.0 -0.59750427561463
grad: 2.0 4.0 -2.3422167604093502
grad: 3.0 6.0 -4.848388694047353
progress: 4 w= 1.7791289594933983 loss= 0.43905614881022015
grad: 1.0 2.0 -0.44174208101320334
grad: 2.0 4.0 -1.7316289575717576
grad: 3.0 6.0 -3.584471942173538
progress: 5 w= 1.836707389300983 loss= 0.2399802903801062
grad: 1.0 2.0 -0.3265852213980338
grad: 2.0 4.0 -1.2802140678802925
grad: 3.0 6.0 -2.650043120512205
progress: 6 w= 1.8792758133988885 loss= 0.1311689630744999
grad: 1.0 2.0 -0.241448373202223
grad: 2.0 4.0 -0.946477622952715
grad: 3.0 6.0 -1.9592086795121197
progress: 7 w= 1.910747160155559 loss= 0.07169462478267678
grad: 1.0 2.0 -0.17850567968888198
grad: 2.0 4.0 -0.6997422643804168
grad: 3.0 6.0 -1.4484664872674653
progress: 8 w= 1.9340143044689266 loss= 0.03918700813247573
grad: 1.0 2.0 -0.13197139106214673
grad: 2.0 4.0 -0.5173278529636143
grad: 3.0 6.0 -1.0708686556346834
progress: 9 w= 1.9512159834655312 loss= 0.021418922423117836
grad: 1.0 2.0 -0.09756803306893769
grad: 2.0 4.0 -0.38246668963023644
grad: 3.0 6.0 -0.7917060475345892
progress: 10 w= 1.9639333911678687 loss= 0.01170720245384975
grad: 1.0 2.0 -0.07213321766426262
grad: 2.0 4.0 -0.2827622132439096
grad: 3.0 6.0 -0.5853177814148953
progress: 11 w= 1.9733355232910992 loss= 0.006398948863435593
grad: 1.0 2.0 -0.05332895341780164
grad: 2.0 4.0 -0.2090494973977819
grad: 3.0 6.0 -0.4327324596134101
progress: 12 w= 1.9802866323953892 loss= 0.003497551760830656
grad: 1.0 2.0 -0.039426735209221686
grad: 2.0 4.0 -0.15455280202014876
grad: 3.0 6.0 -0.3199243001817109
progress: 13 w= 1.9854256707695 loss= 0.001911699652671057
grad: 1.0 2.0 -0.02914865846100012
grad: 2.0 4.0 -0.11426274116712065
grad: 3.0 6.0 -0.2365238742159388
progress: 14 w= 1.9892250235079405 loss= 0.0010449010656399273
grad: 1.0 2.0 -0.021549952984118992
grad: 2.0 4.0 -0.08447581569774698
grad: 3.0 6.0 -0.17486493849433593
progress: 15 w= 1.9920339305797026 loss= 0.0005711243580809696
grad: 1.0 2.0 -0.015932138840594856
grad: 2.0 4.0 -0.062453984255132156
grad: 3.0 6.0 -0.12927974740812687
progress: 16 w= 1.994110589284741 loss= 0.0003121664271570621
grad: 1.0 2.0 -0.011778821430517894
grad: 2.0 4.0 -0.046172980007630926
grad: 3.0 6.0 -0.09557806861579543
progress: 17 w= 1.9956458879852805 loss= 0.0001706246229305199
grad: 1.0 2.0 -0.008708224029438938
grad: 2.0 4.0 -0.03413623819540135
grad: 3.0 6.0 -0.07066201306448505
progress: 18 w= 1.9967809527381737 loss= 9.326038746484765e-05
grad: 1.0 2.0 -0.006438094523652627
grad: 2.0 4.0 -0.02523733053271826
grad: 3.0 6.0 -0.052241274202728505
progress: 19 w= 1.9976201197307648 loss= 5.097447086306101e-05
grad: 1.0 2.0 -0.004759760538470381
grad: 2.0 4.0 -0.01865826131080439
grad: 3.0 6.0 -0.03862260091336722
progress: 20 w= 1.998240525958391 loss= 2.7861740127856012e-05
grad: 1.0 2.0 -0.0035189480832178432
grad: 2.0 4.0 -0.01379427648621423
grad: 3.0 6.0 -0.028554152326460525
progress: 21 w= 1.99869919972735 loss= 1.5228732143933469e-05
grad: 1.0 2.0 -0.002601600545300009
grad: 2.0 4.0 -0.01019827413757568
grad: 3.0 6.0 -0.021110427464781978
progress: 22 w= 1.9990383027488265 loss= 8.323754426231206e-06
grad: 1.0 2.0 -0.001923394502346909
grad: 2.0 4.0 -0.007539706449199102
grad: 3.0 6.0 -0.01560719234984198
progress: 23 w= 1.9992890056818404 loss= 4.549616284094891e-06
grad: 1.0 2.0 -0.0014219886363191492
grad: 2.0 4.0 -0.005574195454370212
grad: 3.0 6.0 -0.011538584590544687
progress: 24 w= 1.999474353368653 loss= 2.486739429417538e-06
grad: 1.0 2.0 -0.0010512932626940419
grad: 2.0 4.0 -0.004121069589761106
grad: 3.0 6.0 -0.008530614050808794
progress: 25 w= 1.9996113831376856 loss= 1.3592075910762856e-06
grad: 1.0 2.0 -0.0007772337246287897
grad: 2.0 4.0 -0.0030467562005451754
grad: 3.0 6.0 -0.006306785335127074
progress: 26 w= 1.9997126908902887 loss= 7.429187207079447e-07
grad: 1.0 2.0 -0.0005746182194226179
grad: 2.0 4.0 -0.002252503420136165
grad: 3.0 6.0 -0.00466268207967957
progress: 27 w= 1.9997875889274812 loss= 4.060661735575354e-07
grad: 1.0 2.0 -0.0004248221450375844
grad: 2.0 4.0 -0.0016653028085471533
grad: 3.0 6.0 -0.0034471768136938863
progress: 28 w= 1.9998429619451539 loss= 2.2194855602869353e-07
grad: 1.0 2.0 -0.00031407610969225175
grad: 2.0 4.0 -0.0012311783499932005
grad: 3.0 6.0 -0.0025485391844828342
progress: 29 w= 1.9998838998815958 loss= 1.213131374411496e-07
grad: 1.0 2.0 -0.00023220023680847746
grad: 2.0 4.0 -0.0009102249282886277
grad: 3.0 6.0 -0.0018841656015560204
progress: 30 w= 1.9999141657892625 loss= 6.630760559646474e-08
grad: 1.0 2.0 -0.00017166842147497974
grad: 2.0 4.0 -0.0006729402121816719
grad: 3.0 6.0 -0.0013929862392156878
progress: 31 w= 1.9999365417379913 loss= 3.624255915449335e-08
grad: 1.0 2.0 -0.0001269165240174175
grad: 2.0 4.0 -0.0004975127741477792
grad: 3.0 6.0 -0.0010298514424817995
progress: 32 w= 1.9999530845453979 loss= 1.9809538924707548e-08
grad: 1.0 2.0 -9.383090920422887e-05
grad: 2.0 4.0 -0.00036781716408107457
grad: 3.0 6.0 -0.0007613815296476645
progress: 33 w= 1.9999653148414271 loss= 1.0827542027017377e-08
grad: 1.0 2.0 -6.937031714571162e-05
grad: 2.0 4.0 -0.0002719316432120422
grad: 3.0 6.0 -0.0005628985014531906
progress: 34 w= 1.999974356846045 loss= 5.9181421028034105e-09
grad: 1.0 2.0 -5.1286307909848006e-05
grad: 2.0 4.0 -0.00020104232700646207
grad: 3.0 6.0 -0.0004161576169003922
progress: 35 w= 1.9999810417085633 loss= 3.2347513278475087e-09
grad: 1.0 2.0 -3.7916582873442906e-05
grad: 2.0 4.0 -0.0001486330048638962
grad: 3.0 6.0 -0.0003076703200690645
progress: 36 w= 1.9999859839076413 loss= 1.7680576050779005e-09
grad: 1.0 2.0 -2.8032184717474706e-05
grad: 2.0 4.0 -0.0001098861640933535
grad: 3.0 6.0 -0.00022746435967313516
progress: 37 w= 1.9999896377347262 loss= 9.6638887447731e-10
grad: 1.0 2.0 -2.0724530547688857e-05
grad: 2.0 4.0 -8.124015974608767e-05
grad: 3.0 6.0 -0.00016816713067413502
progress: 38 w= 1.999992339052936 loss= 5.282109892545845e-10
grad: 1.0 2.0 -1.5321894128117464e-05
grad: 2.0 4.0 -6.006182498197177e-05
grad: 3.0 6.0 -0.00012432797771566584
progress: 39 w= 1.9999943361699042 loss= 2.887107421958329e-10
grad: 1.0 2.0 -1.1327660191629008e-05
grad: 2.0 4.0 -4.4404427951505454e-05
grad: 3.0 6.0 -9.191716585732479e-05
progress: 40 w= 1.9999958126624442 loss= 1.5780416225633037e-10
grad: 1.0 2.0 -8.37467511161094e-06
grad: 2.0 4.0 -3.282872643772805e-05
grad: 3.0 6.0 -6.795546372551087e-05
progress: 41 w= 1.999996904251097 loss= 8.625295142578772e-11
grad: 1.0 2.0 -6.191497806007362e-06
grad: 2.0 4.0 -2.4270671399762023e-05
grad: 3.0 6.0 -5.0240289795056015e-05
progress: 42 w= 1.999997711275687 loss= 4.71443308235547e-11
grad: 1.0 2.0 -4.5774486259198e-06
grad: 2.0 4.0 -1.794359861406747e-05
grad: 3.0 6.0 -3.714324913239864e-05
progress: 43 w= 1.9999983079186507 loss= 2.5768253628059826e-11
grad: 1.0 2.0 -3.3841626985164908e-06
grad: 2.0 4.0 -1.326591777761621e-05
grad: 3.0 6.0 -2.7460449796734565e-05
progress: 44 w= 1.9999987490239537 loss= 1.4084469615916932e-11
grad: 1.0 2.0 -2.5019520926150562e-06
grad: 2.0 4.0 -9.807652203264183e-06
grad: 3.0 6.0 -2.0301840059744336e-05
progress: 45 w= 1.9999990751383971 loss= 7.698320862431846e-12
grad: 1.0 2.0 -1.8497232057157476e-06
grad: 2.0 4.0 -7.250914967116273e-06
grad: 3.0 6.0 -1.5009393983689279e-05
progress: 46 w= 1.9999993162387186 loss= 4.20776540913866e-12
grad: 1.0 2.0 -1.3675225627451937e-06
grad: 2.0 4.0 -5.3606884460322135e-06
grad: 3.0 6.0 -1.109662508014253e-05
progress: 47 w= 1.9999994944870796 loss= 2.299889814334344e-12
grad: 1.0 2.0 -1.0110258408246864e-06
grad: 2.0 4.0 -3.963221296032771e-06
grad: 3.0 6.0 -8.20386808086937e-06
progress: 48 w= 1.9999996262682318 loss= 1.2570789110540446e-12
grad: 1.0 2.0 -7.474635363990956e-07
grad: 2.0 4.0 -2.930057062755509e-06
grad: 3.0 6.0 -6.065218119744031e-06
progress: 49 w= 1.999999723695619 loss= 6.870969979249939e-13
grad: 1.0 2.0 -5.526087618612507e-07
grad: 2.0 4.0 -2.166226346744793e-06
grad: 3.0 6.0 -4.484088535150477e-06
progress: 50 w= 1.9999997957248556 loss= 3.7555501141274804e-13
grad: 1.0 2.0 -4.08550288710785e-07
grad: 2.0 4.0 -1.6015171322436572e-06
grad: 3.0 6.0 -3.3151404608133817e-06
progress: 51 w= 1.9999998489769344 loss= 2.052716967104274e-13
grad: 1.0 2.0 -3.020461312175371e-07
grad: 2.0 4.0 -1.1840208351543424e-06
grad: 3.0 6.0 -2.4509231284497446e-06
progress: 52 w= 1.9999998883468353 loss= 1.1219786256679713e-13
grad: 1.0 2.0 -2.2330632942768602e-07
grad: 2.0 4.0 -8.753608113920563e-07
grad: 3.0 6.0 -1.811996877876254e-06
progress: 53 w= 1.9999999174534755 loss= 6.132535848018759e-14
grad: 1.0 2.0 -1.6509304900935717e-07
grad: 2.0 4.0 -6.471647520100987e-07
grad: 3.0 6.0 -1.3396310407642886e-06
progress: 54 w= 1.999999938972364 loss= 3.351935118167793e-14
grad: 1.0 2.0 -1.220552721115098e-07
grad: 2.0 4.0 -4.784566662863199e-07
grad: 3.0 6.0 -9.904052991061008e-07
progress: 55 w= 1.9999999548815364 loss= 1.8321081844499955e-14
grad: 1.0 2.0 -9.023692726373156e-08
grad: 2.0 4.0 -3.5372875473171916e-07
grad: 3.0 6.0 -7.322185204827747e-07
progress: 56 w= 1.9999999666433785 loss= 1.0013977760018664e-14
grad: 1.0 2.0 -6.671324292994996e-08
grad: 2.0 4.0 -2.615159129248923e-07
grad: 3.0 6.0 -5.413379398078177e-07
progress: 57 w= 1.9999999753390494 loss= 5.473462367088053e-15
grad: 1.0 2.0 -4.932190122985958e-08
grad: 2.0 4.0 -1.9334185274999527e-07
grad: 3.0 6.0 -4.002176350326181e-07
progress: 58 w= 1.9999999817678633 loss= 2.991697274308627e-15
grad: 1.0 2.0 -3.6464273378555845e-08
grad: 2.0 4.0 -1.429399514307761e-07
grad: 3.0 6.0 -2.9588569994132286e-07
progress: 59 w= 1.9999999865207625 loss= 1.6352086111474931e-15
grad: 1.0 2.0 -2.6958475007887728e-08
grad: 2.0 4.0 -1.0567722164012139e-07
grad: 3.0 6.0 -2.1875184863517916e-07
progress: 60 w= 1.999999990034638 loss= 8.937759877335403e-16
grad: 1.0 2.0 -1.993072418216002e-08
grad: 2.0 4.0 -7.812843882959442e-08
grad: 3.0 6.0 -1.617258700292723e-07
progress: 61 w= 1.9999999926324883 loss= 4.885220495987371e-16
grad: 1.0 2.0 -1.473502342363986e-08
grad: 2.0 4.0 -5.7761292637792394e-08
grad: 3.0 6.0 -1.195658771990793e-07
progress: 62 w= 1.99999999455311 loss= 2.670175009618106e-16
grad: 1.0 2.0 -1.0893780100218464e-08
grad: 2.0 4.0 -4.270361841918202e-08
grad: 3.0 6.0 -8.839649012770678e-08
progress: 63 w= 1.9999999959730488 loss= 1.4594702493172377e-16
grad: 1.0 2.0 -8.05390243385773e-09
grad: 2.0 4.0 -3.1571296688071016e-08
grad: 3.0 6.0 -6.53525820126788e-08
progress: 64 w= 1.9999999970228268 loss= 7.977204100704301e-17
grad: 1.0 2.0 -5.9543463493128e-09
grad: 2.0 4.0 -2.334103754719763e-08
grad: 3.0 6.0 -4.8315948575350376e-08
progress: 65 w= 1.9999999977989402 loss= 4.360197735196887e-17
grad: 1.0 2.0 -4.402119557767037e-09
grad: 2.0 4.0 -1.725630838222969e-08
grad: 3.0 6.0 -3.5720557178819945e-08
progress: 66 w= 1.9999999983727301 loss= 2.3832065197304227e-17
grad: 1.0 2.0 -3.254539748809293e-09
grad: 2.0 4.0 -1.2757796596929438e-08
grad: 3.0 6.0 -2.6408640607655798e-08
progress: 67 w= 1.9999999987969397 loss= 1.3026183953845832e-17
grad: 1.0 2.0 -2.406120636067044e-09
grad: 2.0 4.0 -9.431992964437086e-09
grad: 3.0 6.0 -1.9524227568012975e-08
progress: 68 w= 1.999999999110563 loss= 7.11988308874388e-18
grad: 1.0 2.0 -1.7788739370416806e-09
grad: 2.0 4.0 -6.97318647269185e-09
grad: 3.0 6.0 -1.4434496264925656e-08
progress: 69 w= 1.9999999993424284 loss= 3.89160224698574e-18
grad: 1.0 2.0 -1.3151431055291596e-09
grad: 2.0 4.0 -5.155360582875801e-09
grad: 3.0 6.0 -1.067159693945996e-08
progress: 70 w= 1.9999999995138495 loss= 2.1270797208746147e-18
grad: 1.0 2.0 -9.72300906454393e-10
grad: 2.0 4.0 -3.811418736177075e-09
grad: 3.0 6.0 -7.88963561149103e-09
progress: 71 w= 1.9999999996405833 loss= 1.1626238773828175e-18
grad: 1.0 2.0 -7.18833437218791e-10
grad: 2.0 4.0 -2.8178277489132597e-09
grad: 3.0 6.0 -5.832902161273523e-09
progress: 72 w= 1.999999999734279 loss= 6.354692062078993e-19
grad: 1.0 2.0 -5.314420015167798e-10
grad: 2.0 4.0 -2.0832526814729135e-09
grad: 3.0 6.0 -4.31233715403323e-09
progress: 73 w= 1.9999999998035491 loss= 3.4733644793346653e-19
grad: 1.0 2.0 -3.92901711165905e-10
grad: 2.0 4.0 -1.5401742103904326e-09
grad: 3.0 6.0 -3.188159070077745e-09
progress: 74 w= 1.9999999998547615 loss= 1.8984796531526204e-19
grad: 1.0 2.0 -2.9047697580608656e-10
grad: 2.0 4.0 -1.1386696030513122e-09
grad: 3.0 6.0 -2.3570478902001923e-09
progress: 75 w= 1.9999999998926234 loss= 1.0376765851119951e-19
grad: 1.0 2.0 -2.1475310418850313e-10
grad: 2.0 4.0 -8.418314934033333e-10
grad: 3.0 6.0 -1.7425900722400911e-09
progress: 76 w= 1.9999999999206153 loss= 5.671751114309842e-20
grad: 1.0 2.0 -1.5876944203796484e-10
grad: 2.0 4.0 -6.223768167501476e-10
grad: 3.0 6.0 -1.2883241140571045e-09
progress: 77 w= 1.9999999999413098 loss= 3.100089617511693e-20
grad: 1.0 2.0 -1.17380327679939e-10
grad: 2.0 4.0 -4.601314884666863e-10
grad: 3.0 6.0 -9.524754318590567e-10
progress: 78 w= 1.9999999999566096 loss= 1.6944600977692705e-20
grad: 1.0 2.0 -8.678080476443029e-11
grad: 2.0 4.0 -3.4018121652934497e-10
grad: 3.0 6.0 -7.041780492045291e-10
progress: 79 w= 1.9999999999679208 loss= 9.2616919156479e-21
grad: 1.0 2.0 -6.415845632545825e-11
grad: 2.0 4.0 -2.5150193039280566e-10
grad: 3.0 6.0 -5.206075570640678e-10
progress: 80 w= 1.9999999999762834 loss= 5.062350511130293e-21
grad: 1.0 2.0 -4.743316850408519e-11
grad: 2.0 4.0 -1.8593837580738182e-10
grad: 3.0 6.0 -3.8489211817704927e-10
progress: 81 w= 1.999999999982466 loss= 2.7669155644059242e-21
grad: 1.0 2.0 -3.5067948545020045e-11
grad: 2.0 4.0 -1.3746692673066718e-10
grad: 3.0 6.0 -2.845563784603655e-10
progress: 82 w= 1.9999999999870368 loss= 1.5124150106147723e-21
grad: 1.0 2.0 -2.5926372160256506e-11
grad: 2.0 4.0 -1.0163070385260653e-10
grad: 3.0 6.0 -2.1037571684701106e-10
progress: 83 w= 1.999999999990416 loss= 8.26683933105326e-22
grad: 1.0 2.0 -1.9167778475548403e-11
grad: 2.0 4.0 -7.51381179497912e-11
grad: 3.0 6.0 -1.5553425214420713e-10
progress: 84 w= 1.9999999999929146 loss= 4.518126871054872e-22
grad: 1.0 2.0 -1.4170886686315498e-11
grad: 2.0 4.0 -5.555023108172463e-11
grad: 3.0 6.0 -1.1499068364173581e-10
progress: 85 w= 1.9999999999947617 loss= 2.469467919185614e-22
grad: 1.0 2.0 -1.0476508549572827e-11
grad: 2.0 4.0 -4.106759377009439e-11
grad: 3.0 6.0 -8.500933290633839e-11
progress: 86 w= 1.9999999999961273 loss= 1.349840097651456e-22
grad: 1.0 2.0 -7.745359908994942e-12
grad: 2.0 4.0 -3.036149109902908e-11
grad: 3.0 6.0 -6.285105769165966e-11
progress: 87 w= 1.999999999997137 loss= 7.376551550022107e-23
grad: 1.0 2.0 -5.726086271806707e-12
grad: 2.0 4.0 -2.2446045022661565e-11
grad: 3.0 6.0 -4.646416584819235e-11
progress: 88 w= 1.9999999999978835 loss= 4.031726170507742e-23
grad: 1.0 2.0 -4.233058348290797e-12
grad: 2.0 4.0 -1.659294923683774e-11
grad: 3.0 6.0 -3.4351188560322043e-11
progress: 89 w= 1.9999999999984353 loss= 2.2033851437431755e-23
grad: 1.0 2.0 -3.1294966618133913e-12
grad: 2.0 4.0 -1.226752033289813e-11
grad: 3.0 6.0 -2.539835008974478e-11
progress: 90 w= 1.9999999999988431 loss= 1.2047849775995315e-23
grad: 1.0 2.0 -2.3137047833188262e-12
grad: 2.0 4.0 -9.070078021977679e-12
grad: 3.0 6.0 -1.8779644506139448e-11
progress: 91 w= 1.9999999999991447 loss= 6.5840863393251405e-24
grad: 1.0 2.0 -1.7106316363424412e-12
grad: 2.0 4.0 -6.7057470687359455e-12
grad: 3.0 6.0 -1.3882228699912957e-11
progress: 92 w= 1.9999999999993676 loss= 3.5991747246272455e-24
grad: 1.0 2.0 -1.2647660696529783e-12
grad: 2.0 4.0 -4.957811938766099e-12
grad: 3.0 6.0 -1.0263789818054647e-11
progress: 93 w= 1.9999999999995324 loss= 1.969312363793734e-24
grad: 1.0 2.0 -9.352518759442319e-13
grad: 2.0 4.0 -3.666400516522117e-12
grad: 3.0 6.0 -7.58859641791787e-12
progress: 94 w= 1.9999999999996543 loss= 1.0761829795642296e-24
grad: 1.0 2.0 -6.914468997365475e-13
grad: 2.0 4.0 -2.7107205369247822e-12
grad: 3.0 6.0 -5.611511255665391e-12
progress: 95 w= 1.9999999999997444 loss= 5.875191475205477e-25
grad: 1.0 2.0 -5.111466805374221e-13
grad: 2.0 4.0 -2.0037305148434825e-12
grad: 3.0 6.0 -4.1460168631601846e-12
progress: 96 w= 1.999999999999811 loss= 3.2110109830478153e-25
grad: 1.0 2.0 -3.779199175824033e-13
grad: 2.0 4.0 -1.4814816040598089e-12
grad: 3.0 6.0 -3.064215547965432e-12
progress: 97 w= 1.9999999999998603 loss= 1.757455879087579e-25
grad: 1.0 2.0 -2.793321129956894e-13
grad: 2.0 4.0 -1.0942358130705543e-12
grad: 3.0 6.0 -2.2648549702353193e-12
progress: 98 w= 1.9999999999998967 loss= 9.608404711682446e-26
grad: 1.0 2.0 -2.0650148258027912e-13
grad: 2.0 4.0 -8.100187187665142e-13
grad: 3.0 6.0 -1.6786572132332367e-12
progress: 99 w= 1.9999999999999236 loss= 5.250973729513143e-26
Predict (after training) 4 7.9999999999996945
1、梯度、偏微分以及梯度的区别和联系
(1)导数是指一元函数对于自变量求导得到的数值，它是一个标量，反映了函数的变化趋势；
(2)偏微分是多元函数对各个自变量求导得到的，它反映的是多元函数在各个自变量方向上的变化趋势，也是标量；
(3)梯度是一个矢量，是有大小和方向的，其方向是指多元函数增大的方向，而大小是指增长的趋势快慢。
2、在寻找函数的最小值的时候可以利用梯度下降法来进行寻找，一般会出现以下两个问题局部最优解和铵点(不同自变量的变化趋势相反，一个处于极小，一个处于极大)
3、初始状态、学习率和动量(如何逃出局部最优解)是全部寻优的三个重要影响因素
4、常见函数的梯度计算基本和一元函数导数是一致的
5、常见的激活函数主要有三种：
(1)sigmoid函数
(2)tanh函数
(3)Relu函数
6、常用的损失函数-loss函数
1)MSE-回归问题
(2)Cross entropy loss-分类问题
7、对于MSE均方差的loss函数，计算时可以利用norm函数来进行计算，调用的格式如下所示：
MSE=torch.norm(y-y_pre,2).pow(2)
也可以直接调用API：
F.mse_loss(y,y_pre)来进行求解
8、对于梯度的求取在pytorch里面主要有两种方式：
(1)torch.autograd.grad(loss,[w0,w1,...],retain_graph=True)
(2)loss.backward(retain_graph=True)
#其中使得retain_graph=True的目的是使得其不发生变化
9、感知机及其梯度推导——完整的神经网络的梯度计算的过程
(1)单输出模型的梯度计算
(2)多输出模型的梯度计算
10、链式法则
链式法则主要是用来求取多隐含层和多输出的神经网络的梯度计算关系，它可以方便地进行计算，高效准确