刘二大人 PyTorch深度学习实践 笔记 P4 反向传播

P4 反向传播

目的： 在图上面进行梯度的传播，帮助建立更具有弹性的模型结构

1、P3回顾

2、反向传播

如果是非常复杂的网络，无法直接计算。
但是如果把网络看作图，通过图传播梯度，就能把梯度计算出来，即反向传播。


矩阵计算书籍 Matricx cookbook https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

发现每一层都可以变成 y = w * x + b ，为了增加网络复杂程度，添加一个激活函数

链式求导法则

3、计算过程

1. 创建计算图 前馈运算 得到z
2. 计算局部梯度
3. 反向 得到Loss对z的偏导
4. 使用链式求导计算 Loss对x的偏导 Loss对y的偏导
   
   具体例子：
   
   得到相应的梯度之后，就可以进行权重的更新了
   

4、作业

作业4-1：计算y=x*w的梯度，理解forward和back ward过程

更新模型，增加一个偏置量

作业4-2：代码实现4-1

import torch
import matplotlib.pyplot as plt

x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

# torch.Tensor()是一种类，只会直接引用生成 torch.FloatTensor()类型的数据
# torch.tensor()是python函数，其中数据可以是标量、向量、矩阵、高低维等
# 会对数据做拷贝，生成相应的torch.LongTensor、torch.FloatTensor和torch.DoubleTensor等

# tensor包含两个部分 data grad
w = torch.tensor([1.0])
w.requires_grad = True # 需要计算梯度

# 前馈函数
def forward(x):
	return x * w # w为张量，数乘后结果也为张量

# 计算损失
def loss(x, y):
	y_pred = forward(x)
	return (y_pred - y) ** 2

# 取标量 不创建计算图
# 张量.item() 相当于把数据拿出来，为标量，不进行梯度计算
# 张量.data 相当于对数据进行修改，为张量，不进行梯度计算
print('Predict (before training)', 4, forward(4).item())

epoch_list = []
mse_list = []
for epoch in range(100):
	for x, y in zip(x_data, y_data):
		l = loss(x, y)
		l.backward() # 反向传播
		print('\tgrad:', x, y, w.grad.item())
		w.data = w.data - 0.01 * w.grad.data

		w.grad.data.zero_() # 更新过权重值之后，需要将梯度清0，以便下一次创建图计算

	epoch_list.append(epoch)
	mse_list.append(l.item() / len(x_data))
	print('progress:', epoch, l.item())

print('predict (after training)', 4, forward(4).item())

plt.plot(epoch_list, mse_list)
plt.xlabel('Epoch')
plt.ylabel('Cost')
plt.show()

输出：

Predict (before training) 4 4.0
	grad: 1.0 2.0 -2.0
	grad: 2.0 4.0 -7.840000152587891
	grad: 3.0 6.0 -16.228801727294922
progress: 0 7.315943717956543
	grad: 1.0 2.0 -1.478623867034912
	grad: 2.0 4.0 -5.796205520629883
	grad: 3.0 6.0 -11.998146057128906
progress: 1 3.9987640380859375
	grad: 1.0 2.0 -1.0931644439697266
	grad: 2.0 4.0 -4.285204887390137
	grad: 3.0 6.0 -8.870372772216797
progress: 2 2.1856532096862793
	grad: 1.0 2.0 -0.8081896305084229
	grad: 2.0 4.0 -3.1681032180786133
	grad: 3.0 6.0 -6.557973861694336
progress: 3 1.1946394443511963
	grad: 1.0 2.0 -0.5975041389465332
	grad: 2.0 4.0 -2.3422164916992188
	grad: 3.0 6.0 -4.848389625549316
progress: 4 0.6529689431190491
	grad: 1.0 2.0 -0.4417421817779541
	grad: 2.0 4.0 -1.7316293716430664
	grad: 3.0 6.0 -3.58447265625
progress: 5 0.35690122842788696
	grad: 1.0 2.0 -0.3265852928161621
	grad: 2.0 4.0 -1.2802143096923828
	grad: 3.0 6.0 -2.650045394897461
progress: 6 0.195076122879982
	grad: 1.0 2.0 -0.24144840240478516
	grad: 2.0 4.0 -0.9464778900146484
	grad: 3.0 6.0 -1.9592113494873047
progress: 7 0.10662525147199631
	grad: 1.0 2.0 -0.17850565910339355
	grad: 2.0 4.0 -0.699742317199707
	grad: 3.0 6.0 -1.4484672546386719
progress: 8 0.0582793727517128
	grad: 1.0 2.0 -0.1319713592529297
	grad: 2.0 4.0 -0.5173273086547852
	grad: 3.0 6.0 -1.070866584777832
progress: 9 0.03185431286692619
	grad: 1.0 2.0 -0.09756779670715332
	grad: 2.0 4.0 -0.3824653625488281
	grad: 3.0 6.0 -0.7917022705078125
progress: 10 0.017410902306437492
	grad: 1.0 2.0 -0.07213282585144043
	grad: 2.0 4.0 -0.2827606201171875
	grad: 3.0 6.0 -0.5853137969970703
progress: 11 0.009516451507806778
	grad: 1.0 2.0 -0.053328514099121094
	grad: 2.0 4.0 -0.2090473175048828
	grad: 3.0 6.0 -0.43272972106933594
progress: 12 0.005201528314501047
	grad: 1.0 2.0 -0.039426326751708984
	grad: 2.0 4.0 -0.15455150604248047
	grad: 3.0 6.0 -0.3199195861816406
progress: 13 0.0028430151287466288
	grad: 1.0 2.0 -0.029148340225219727
	grad: 2.0 4.0 -0.11426162719726562
	grad: 3.0 6.0 -0.23652076721191406
progress: 14 0.0015539465239271522
	grad: 1.0 2.0 -0.021549701690673828
	grad: 2.0 4.0 -0.08447456359863281
	grad: 3.0 6.0 -0.17486286163330078
progress: 15 0.0008493617060594261
	grad: 1.0 2.0 -0.01593184471130371
	grad: 2.0 4.0 -0.062453269958496094
	grad: 3.0 6.0 -0.12927818298339844
progress: 16 0.00046424579340964556
	grad: 1.0 2.0 -0.011778593063354492
	grad: 2.0 4.0 -0.046172142028808594
	grad: 3.0 6.0 -0.09557533264160156
progress: 17 0.0002537401160225272
	grad: 1.0 2.0 -0.00870823860168457
	grad: 2.0 4.0 -0.03413581848144531
	grad: 3.0 6.0 -0.07066154479980469
progress: 18 0.00013869594840798527
	grad: 1.0 2.0 -0.006437778472900391
	grad: 2.0 4.0 -0.025236129760742188
	grad: 3.0 6.0 -0.052239418029785156
progress: 19 7.580435340059921e-05
	grad: 1.0 2.0 -0.004759550094604492
	grad: 2.0 4.0 -0.018657684326171875
	grad: 3.0 6.0 -0.038620948791503906
progress: 20 4.143271507928148e-05
	grad: 1.0 2.0 -0.003518819808959961
	grad: 2.0 4.0 -0.0137939453125
	grad: 3.0 6.0 -0.028553009033203125
progress: 21 2.264650902361609e-05
	grad: 1.0 2.0 -0.00260162353515625
	grad: 2.0 4.0 -0.010198593139648438
	grad: 3.0 6.0 -0.021108627319335938
progress: 22 1.2377059647405986e-05
	grad: 1.0 2.0 -0.0019233226776123047
	grad: 2.0 4.0 -0.0075397491455078125
	grad: 3.0 6.0 -0.0156097412109375
progress: 23 6.768445018678904e-06
	grad: 1.0 2.0 -0.0014221668243408203
	grad: 2.0 4.0 -0.0055751800537109375
	grad: 3.0 6.0 -0.011541366577148438
progress: 24 3.7000872907810844e-06
	grad: 1.0 2.0 -0.0010514259338378906
	grad: 2.0 4.0 -0.0041217803955078125
	grad: 3.0 6.0 -0.008531570434570312
progress: 25 2.021880391112063e-06
	grad: 1.0 2.0 -0.0007772445678710938
	grad: 2.0 4.0 -0.0030469894409179688
	grad: 3.0 6.0 -0.006305694580078125
progress: 26 1.1044940038118511e-06
	grad: 1.0 2.0 -0.0005745887756347656
	grad: 2.0 4.0 -0.0022525787353515625
	grad: 3.0 6.0 -0.0046634674072265625
progress: 27 6.041091182851233e-07
	grad: 1.0 2.0 -0.0004248619079589844
	grad: 2.0 4.0 -0.0016651153564453125
	grad: 3.0 6.0 -0.003444671630859375
progress: 28 3.296045179013163e-07
	grad: 1.0 2.0 -0.0003139972686767578
	grad: 2.0 4.0 -0.0012311935424804688
	grad: 3.0 6.0 -0.0025491714477539062
progress: 29 1.805076408345485e-07
	grad: 1.0 2.0 -0.00023221969604492188
	grad: 2.0 4.0 -0.0009107589721679688
	grad: 3.0 6.0 -0.0018854141235351562
progress: 30 9.874406714516226e-08
	grad: 1.0 2.0 -0.00017189979553222656
	grad: 2.0 4.0 -0.0006742477416992188
	grad: 3.0 6.0 -0.00139617919921875
progress: 31 5.4147676564753056e-08
	grad: 1.0 2.0 -0.0001270771026611328
	grad: 2.0 4.0 -0.0004978179931640625
	grad: 3.0 6.0 -0.00102996826171875
progress: 32 2.9467628337442875e-08
	grad: 1.0 2.0 -9.393692016601562e-05
	grad: 2.0 4.0 -0.0003681182861328125
	grad: 3.0 6.0 -0.0007610321044921875
progress: 33 1.6088051779661328e-08
	grad: 1.0 2.0 -6.937980651855469e-05
	grad: 2.0 4.0 -0.00027179718017578125
	grad: 3.0 6.0 -0.000560760498046875
progress: 34 8.734787115827203e-09
	grad: 1.0 2.0 -5.125999450683594e-05
	grad: 2.0 4.0 -0.00020122528076171875
	grad: 3.0 6.0 -0.0004177093505859375
progress: 35 4.8466972657479346e-09
	grad: 1.0 2.0 -3.790855407714844e-05
	grad: 2.0 4.0 -0.000148773193359375
	grad: 3.0 6.0 -0.000308990478515625
progress: 36 2.6520865503698587e-09
	grad: 1.0 2.0 -2.8133392333984375e-05
	grad: 2.0 4.0 -0.000110626220703125
	grad: 3.0 6.0 -0.0002288818359375
progress: 37 1.4551915228366852e-09
	grad: 1.0 2.0 -2.09808349609375e-05
	grad: 2.0 4.0 -8.20159912109375e-05
	grad: 3.0 6.0 -0.00016880035400390625
progress: 38 7.914877642178908e-10
	grad: 1.0 2.0 -1.5497207641601562e-05
	grad: 2.0 4.0 -6.103515625e-05
	grad: 3.0 6.0 -0.000125885009765625
progress: 39 4.4019543565809727e-10
	grad: 1.0 2.0 -1.1444091796875e-05
	grad: 2.0 4.0 -4.482269287109375e-05
	grad: 3.0 6.0 -9.1552734375e-05
progress: 40 2.3283064365386963e-10
	grad: 1.0 2.0 -8.344650268554688e-06
	grad: 2.0 4.0 -3.24249267578125e-05
	grad: 3.0 6.0 -6.580352783203125e-05
progress: 41 1.2028067430946976e-10
	grad: 1.0 2.0 -5.9604644775390625e-06
	grad: 2.0 4.0 -2.288818359375e-05
	grad: 3.0 6.0 -4.57763671875e-05
progress: 42 5.820766091346741e-11
	grad: 1.0 2.0 -4.291534423828125e-06
	grad: 2.0 4.0 -1.71661376953125e-05
	grad: 3.0 6.0 -3.719329833984375e-05
progress: 43 3.842615114990622e-11
	grad: 1.0 2.0 -3.337860107421875e-06
	grad: 2.0 4.0 -1.33514404296875e-05
	grad: 3.0 6.0 -2.86102294921875e-05
progress: 44 2.2737367544323206e-11
	grad: 1.0 2.0 -2.6226043701171875e-06
	grad: 2.0 4.0 -1.049041748046875e-05
	grad: 3.0 6.0 -2.288818359375e-05
progress: 45 1.4551915228366852e-11
	grad: 1.0 2.0 -1.9073486328125e-06
	grad: 2.0 4.0 -7.62939453125e-06
	grad: 3.0 6.0 -1.430511474609375e-05
progress: 46 5.6843418860808015e-12
	grad: 1.0 2.0 -1.430511474609375e-06
	grad: 2.0 4.0 -5.7220458984375e-06
	grad: 3.0 6.0 -1.1444091796875e-05
progress: 47 3.637978807091713e-12
	grad: 1.0 2.0 -1.1920928955078125e-06
	grad: 2.0 4.0 -4.76837158203125e-06
	grad: 3.0 6.0 -1.1444091796875e-05
progress: 48 3.637978807091713e-12
	grad: 1.0 2.0 -9.5367431640625e-07
	grad: 2.0 4.0 -3.814697265625e-06
	grad: 3.0 6.0 -8.58306884765625e-06
progress: 49 2.0463630789890885e-12
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 50 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 51 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 52 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 53 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 54 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 55 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 56 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 57 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 58 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 59 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 60 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 61 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 62 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 63 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 64 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 65 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 66 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 67 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 68 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 69 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 70 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 71 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 72 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 73 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 74 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 75 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 76 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 77 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 78 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 79 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 80 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 81 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 82 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 83 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 84 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 85 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 86 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 87 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 88 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 89 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 90 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 91 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 92 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 93 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 94 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 95 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 96 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 97 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 98 9.094947017729282e-13
	grad: 1.0 2.0 -7.152557373046875e-07
	grad: 2.0 4.0 -2.86102294921875e-06
	grad: 3.0 6.0 -5.7220458984375e-06
progress: 99 9.094947017729282e-13
predict (after training) 4 7.999998569488525

作业4-3：计算y = w1x^2 + w2x + b 的 梯度

作业4-4：代码实现4-3

import torch
import matplotlib.pyplot as plt

def forward(x):
	return x ** 2 * w1 + x * w2 + b

def loss(x, y):
	return (forward(x) - y) ** 2

x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w1 = torch.tensor([1.0])
w2 = torch.tensor([2.0])
b = torch.tensor([3.0])
w1.requires_grad = True
w2.requires_grad = True
b.requires_grad = True
r = 0.01

print('predict(before training):', 4, forward(4).item())

epoch_list = []
loss_list = []

for epoch in range(100):
	for x, y in zip(x_data, y_data):
		l = loss(x, y)
		l.backward()
		print('\tgrad:', x, y, w1.grad.item(), w2.grad.item(), b.grad.item())
		w1.data -= r * w1.grad.data
		w2.data -= r * w2.grad.data
		b.data -= r * b.grad.data

		w1.grad.data.zero_()
		w2.grad.data.zero_()
		b.grad.data.zero_()

	epoch_list.append(epoch)
	loss_list.append(l.item())
	print('progress:', epoch, l.item())

print('predict(after training):', 4, forward(4).item())

plt.plot(epoch_list, loss_list)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.show()

输出：

predict(before training): 4 27.0
	grad: 1.0 2.0 8.0 8.0 8.0
	grad: 2.0 4.0 51.52000427246094 25.76000213623047 12.880001068115234
	grad: 3.0 6.0 97.58880615234375 32.52960205078125 10.84320068359375
progress: 0 29.39375114440918
	grad: 1.0 2.0 2.8975682258605957 2.8975682258605957 2.8975682258605957
	grad: 2.0 4.0 -9.041646957397461 -4.5208234786987305 -2.2604117393493652
	grad: 3.0 6.0 -69.30754089355469 -23.10251235961914 -7.7008376121521
progress: 1 14.825724601745605
	grad: 1.0 2.0 5.042389869689941 5.042389869689941 5.042389869689941
	grad: 2.0 4.0 18.422988891601562 9.211494445800781 4.605747222900391
	grad: 3.0 6.0 9.384512901306152 3.128170967102051 1.0427236557006836
progress: 2 0.2718181610107422
	grad: 1.0 2.0 3.8239336013793945 3.8239336013793945 3.8239336013793945
	grad: 2.0 4.0 4.956966400146484 2.478483200073242 1.239241600036621
	grad: 3.0 6.0 -26.23502540588379 -8.74500846862793 -2.9150028228759766
progress: 3 2.124310255050659
	grad: 1.0 2.0 4.1789045333862305 4.1789045333862305 4.1789045333862305
	grad: 2.0 4.0 10.562461853027344 5.281230926513672 2.640615463256836
	grad: 3.0 6.0 -8.704278945922852 -2.901426315307617 -0.9671421051025391
progress: 4 0.2338409572839737
	grad: 1.0 2.0 3.809941291809082 3.809941291809082 3.809941291809082
	grad: 2.0 4.0 7.3196258544921875 3.6598129272460938 1.8299064636230469
	grad: 3.0 6.0 -15.941230773925781 -5.313743591308594 -1.7712478637695312
progress: 5 0.7843297719955444
	grad: 1.0 2.0 3.785682201385498 3.785682201385498 3.785682201385498
	grad: 2.0 4.0 8.218498229980469 4.109249114990234 2.054624557495117
	grad: 3.0 6.0 -11.68948745727539 -3.896495819091797 -1.2988319396972656
progress: 6 0.4217410981655121
	grad: 1.0 2.0 3.6085901260375977 3.6085901260375977 3.6085901260375977
	grad: 2.0 4.0 7.213901519775391 3.6069507598876953 1.8034753799438477
	grad: 3.0 6.0 -12.817611694335938 -4.2725372314453125 -1.4241790771484375
progress: 7 0.5070714950561523
	grad: 1.0 2.0 3.5098752975463867 3.5098752975463867 3.5098752975463867
	grad: 2.0 4.0 7.117706298828125 3.5588531494140625 1.7794265747070312
	grad: 3.0 6.0 -11.475434303283691 -3.8251447677612305 -1.2750482559204102
progress: 8 0.4064370095729828
	grad: 1.0 2.0 3.3816747665405273 3.3816747665405273 3.3816747665405273
	grad: 2.0 4.0 6.620697021484375 3.3103485107421875 1.6551742553710938
	grad: 3.0 6.0 -11.314312934875488 -3.771437644958496 -1.257145881652832
progress: 9 0.39510393142700195
	grad: 1.0 2.0 3.2739076614379883 3.2739076614379883 3.2739076614379883
	grad: 2.0 4.0 6.331199645996094 3.165599822998047 1.5827999114990234
	grad: 3.0 6.0 -10.634078979492188 -3.5446929931640625 -1.1815643310546875
progress: 10 0.34902358055114746
	grad: 1.0 2.0 3.163087844848633 3.163087844848633 3.163087844848633
	grad: 2.0 4.0 5.965343475341797 2.9826717376708984 1.4913358688354492
	grad: 3.0 6.0 -10.224632263183594 -3.4082107543945312 -1.1360702514648438
progress: 11 0.32266390323638916
	grad: 1.0 2.0 3.0598936080932617 3.0598936080932617 3.0598936080932617
	grad: 2.0 4.0 5.654441833496094 2.827220916748047 1.4136104583740234
	grad: 3.0 6.0 -9.717287063598633 -3.239095687866211 -1.0796985626220703
progress: 12 0.2914372384548187
	grad: 1.0 2.0 2.9591164588928223 2.9591164588928223 2.9591164588928223
	grad: 2.0 4.0 5.336635589599609 2.6683177947998047 1.3341588973999023
	grad: 3.0 6.0 -9.282554626464844 -3.0941848754882812 -1.0313949584960938
progress: 13 0.26594388484954834
	grad: 1.0 2.0 2.862949848175049 2.862949848175049 2.862949848175049
	grad: 2.0 4.0 5.0399932861328125 2.5199966430664062 1.2599983215332031
	grad: 3.0 6.0 -8.839994430541992 -2.946664810180664 -0.9822216033935547
progress: 14 0.24118982255458832
	grad: 1.0 2.0 2.7701501846313477 2.7701501846313477 2.7701501846313477
	grad: 2.0 4.0 4.750751495361328 2.375375747680664 1.187687873840332
	grad: 3.0 6.0 -8.426067352294922 -2.8086891174316406 -0.9362297058105469
progress: 15 0.21913151443004608
	grad: 1.0 2.0 2.681084632873535 2.681084632873535 2.681084632873535
	grad: 2.0 4.0 4.4746551513671875 2.2373275756835938 1.1186637878417969
	grad: 3.0 6.0 -8.022817611694336 -2.6742725372314453 -0.8914241790771484
progress: 16 0.1986592710018158
	grad: 1.0 2.0 2.595376968383789 2.595376968383789 2.595376968383789
	grad: 2.0 4.0 4.2083892822265625 2.1041946411132812 1.0520973205566406
	grad: 3.0 6.0 -7.637712478637695 -2.5459041595458984 -0.8486347198486328
progress: 17 0.1800452172756195
	grad: 1.0 2.0 2.513005256652832 2.513005256652832 2.513005256652832
	grad: 2.0 4.0 3.9528884887695312 1.9764442443847656 0.9882221221923828
	grad: 3.0 6.0 -7.266348838806152 -2.422116279602051 -0.8073720932006836
progress: 18 0.1629624217748642
	grad: 1.0 2.0 2.433791160583496 2.433791160583496 2.433791160583496
	grad: 2.0 4.0 3.707111358642578 1.853555679321289 0.9267778396606445
	grad: 3.0 6.0 -6.909919738769531 -2.3033065795898438 -0.7677688598632812
progress: 19 0.14736725389957428
	grad: 1.0 2.0 2.3576345443725586 2.3576345443725586 2.3576345443725586
	grad: 2.0 4.0 3.470977783203125 1.7354888916015625 0.8677444458007812
	grad: 3.0 6.0 -6.56707763671875 -2.18902587890625 -0.72967529296875
progress: 20 0.1331065148115158
	grad: 1.0 2.0 2.2844080924987793 2.2844080924987793 2.2844080924987793
	grad: 2.0 4.0 3.2439804077148438 1.6219902038574219 0.8109951019287109
	grad: 3.0 6.0 -6.237607955932617 -2.079202651977539 -0.6930675506591797
progress: 21 0.12008565664291382
	grad: 1.0 2.0 2.2140016555786133 2.2140016555786133 2.2140016555786133
	grad: 2.0 4.0 3.02581787109375 1.512908935546875 0.7564544677734375
	grad: 3.0 6.0 -5.920875549316406 -1.9736251831054688 -0.6578750610351562
progress: 22 0.10819990187883377
	grad: 1.0 2.0 2.146306037902832 2.146306037902832 2.146306037902832
	grad: 2.0 4.0 2.8161354064941406 1.4080677032470703 0.7040338516235352
	grad: 3.0 6.0 -5.61646842956543 -1.8721561431884766 -0.6240520477294922
progress: 23 0.09736023843288422
	grad: 1.0 2.0 2.0812158584594727 2.0812158584594727 2.0812158584594727
	grad: 2.0 4.0 2.6146163940429688 1.3073081970214844 0.6536540985107422
	grad: 3.0 6.0 -5.3238115310668945 -1.7746038436889648 -0.5915346145629883
progress: 24 0.08747830241918564
	grad: 1.0 2.0 2.0186309814453125 2.0186309814453125 2.0186309814453125
	grad: 2.0 4.0 2.4209213256835938 1.2104606628417969 0.6052303314208984
	grad: 3.0 6.0 -5.0425615310668945 -1.6808538436889648 -0.5602846145629883
progress: 25 0.07847971469163895
	grad: 1.0 2.0 1.9584546089172363 1.9584546089172363 1.9584546089172363
	grad: 2.0 4.0 2.2347793579101562 1.1173896789550781 0.5586948394775391
	grad: 3.0 6.0 -4.772186279296875 -1.590728759765625 -0.530242919921875
progress: 26 0.07028938829898834
	grad: 1.0 2.0 1.9005932807922363 1.9005932807922363 1.9005932807922363
	grad: 2.0 4.0 2.055877685546875 1.0279388427734375 0.5139694213867188
	grad: 3.0 6.0 -4.512325286865234 -1.5041084289550781 -0.5013694763183594
progress: 27 0.06284283846616745
	grad: 1.0 2.0 1.8449583053588867 1.8449583053588867 1.8449583053588867
	grad: 2.0 4.0 1.883941650390625 0.9419708251953125 0.47098541259765625
	grad: 3.0 6.0 -4.262515068054199 -1.4208383560180664 -0.47361278533935547
progress: 28 0.0560772679746151
	grad: 1.0 2.0 1.7914619445800781 1.7914619445800781 1.7914619445800781
	grad: 2.0 4.0 1.7186965942382812 0.8593482971191406 0.4296741485595703
	grad: 3.0 6.0 -4.022438049316406 -1.3408126831054688 -0.44693756103515625
progress: 29 0.04993829503655434
	grad: 1.0 2.0 1.7400236129760742 1.7400236129760742 1.7400236129760742
	grad: 2.0 4.0 1.5598983764648438 0.7799491882324219 0.38997459411621094
	grad: 3.0 6.0 -3.791656494140625 -1.263885498046875 -0.421295166015625
progress: 30 0.04437240585684776
	grad: 1.0 2.0 1.6905627250671387 1.6905627250671387 1.6905627250671387
	grad: 2.0 4.0 1.4072799682617188 0.7036399841308594 0.3518199920654297
	grad: 3.0 6.0 -3.569835662841797 -1.1899452209472656 -0.3966484069824219
progress: 31 0.03933249041438103
	grad: 1.0 2.0 1.6430025100708008 1.6430025100708008 1.6430025100708008
	grad: 2.0 4.0 1.2606124877929688 0.6303062438964844 0.3151531219482422
	grad: 3.0 6.0 -3.356649398803711 -1.1188831329345703 -0.37296104431152344
progress: 32 0.034774985164403915
	grad: 1.0 2.0 1.5972709655761719 1.5972709655761719 1.5972709655761719
	grad: 2.0 4.0 1.1196670532226562 0.5598335266113281 0.27991676330566406
	grad: 3.0 6.0 -3.1517200469970703 -1.0505733489990234 -0.3501911163330078
progress: 33 0.030658453702926636
	grad: 1.0 2.0 1.5532960891723633 1.5532960891723633 1.5532960891723633
	grad: 2.0 4.0 0.9842185974121094 0.4921092987060547 0.24605464935302734
	grad: 3.0 6.0 -2.95477294921875 -0.98492431640625 -0.32830810546875
progress: 34 0.026946552097797394
	grad: 1.0 2.0 1.5110101699829102 1.5110101699829102 1.5110101699829102
	grad: 2.0 4.0 0.8540611267089844 0.4270305633544922 0.2135152816772461
	grad: 3.0 6.0 -2.7654476165771484 -0.9218158721923828 -0.30727195739746094
progress: 35 0.02360401302576065
	grad: 1.0 2.0 1.4703483581542969 1.4703483581542969 1.4703483581542969
	grad: 2.0 4.0 0.728973388671875 0.3644866943359375 0.18224334716796875
	grad: 3.0 6.0 -2.5835208892822266 -0.8611736297607422 -0.28705787658691406
progress: 36 0.02060055546462536
	grad: 1.0 2.0 1.4312481880187988 1.4312481880187988 1.4312481880187988
	grad: 2.0 4.0 0.6087875366210938 0.3043937683105469 0.15219688415527344
	grad: 3.0 6.0 -2.408632278442383 -0.8028774261474609 -0.2676258087158203
progress: 37 0.017905892804265022
	grad: 1.0 2.0 1.3936481475830078 1.3936481475830078 1.3936481475830078
	grad: 2.0 4.0 0.4932861328125 0.24664306640625 0.123321533203125
	grad: 3.0 6.0 -2.2405757904052734 -0.7468585968017578 -0.24895286560058594
progress: 38 0.01549438200891018
	grad: 1.0 2.0 1.3574919700622559 1.3574919700622559 1.3574919700622559
	grad: 2.0 4.0 0.3823051452636719 0.19115257263183594 0.09557628631591797
	grad: 3.0 6.0 -2.079008102416992 -0.6930027008056641 -0.2310009002685547
progress: 39 0.013340353965759277
	grad: 1.0 2.0 1.3227224349975586 1.3227224349975586 1.3227224349975586
	grad: 2.0 4.0 0.27565765380859375 0.13782882690429688 0.06891441345214844
	grad: 3.0 6.0 -1.9237918853759766 -0.6412639617919922 -0.21375465393066406
progress: 40 0.011422762647271156
	grad: 1.0 2.0 1.2892866134643555 1.2892866134643555 1.2892866134643555
	grad: 2.0 4.0 0.17319488525390625 0.08659744262695312 0.04329872131347656
	grad: 3.0 6.0 -1.7745494842529297 -0.5915164947509766 -0.1971721649169922
progress: 41 0.009719215333461761
	grad: 1.0 2.0 1.2571325302124023 1.2571325302124023 1.2571325302124023
	grad: 2.0 4.0 0.07473373413085938 0.03736686706542969 0.018683433532714844
	grad: 3.0 6.0 -1.631169319152832 -0.5437231063842773 -0.18124103546142578
progress: 42 0.008212078362703323
	grad: 1.0 2.0 1.2262120246887207 1.2262120246887207 1.2262120246887207
	grad: 2.0 4.0 -0.019863128662109375 -0.009931564331054688 -0.004965782165527344
	grad: 3.0 6.0 -1.4933252334594727 -0.4977750778198242 -0.1659250259399414
progress: 43 0.0068827783688902855
	grad: 1.0 2.0 1.1964750289916992 1.1964750289916992 1.1964750289916992
	grad: 2.0 4.0 -0.1107635498046875 -0.05538177490234375 -0.027690887451171875
	grad: 3.0 6.0 -1.3608970642089844 -0.4536323547363281 -0.15121078491210938
progress: 44 0.005716175306588411
	grad: 1.0 2.0 1.1678781509399414 1.1678781509399414 1.1678781509399414
	grad: 2.0 4.0 -0.198089599609375 -0.0990447998046875 -0.04952239990234375
	grad: 3.0 6.0 -1.2336015701293945 -0.41120052337646484 -0.13706684112548828
progress: 45 0.0046968297101557255
	grad: 1.0 2.0 1.140376091003418 1.140376091003418 1.140376091003418
	grad: 2.0 4.0 -0.2819938659667969 -0.14099693298339844 -0.07049846649169922
	grad: 3.0 6.0 -1.11126708984375 -0.37042236328125 -0.12347412109375
progress: 46 0.0038114646449685097
	grad: 1.0 2.0 1.1139264106750488 1.1139264106750488 1.1139264106750488
	grad: 2.0 4.0 -0.36260414123535156 -0.18130207061767578 -0.09065103530883789
	grad: 3.0 6.0 -0.99371337890625 -0.33123779296875 -0.11041259765625
progress: 47 0.0030477354303002357
	grad: 1.0 2.0 1.088489055633545 1.088489055633545 1.088489055633545
	grad: 2.0 4.0 -0.4400444030761719 -0.22002220153808594 -0.11001110076904297
	grad: 3.0 6.0 -0.8807430267333984 -0.2935810089111328 -0.09786033630371094
progress: 48 0.002394161419942975
	grad: 1.0 2.0 1.0640249252319336 1.0640249252319336 1.0640249252319336
	grad: 2.0 4.0 -0.5144424438476562 -0.2572212219238281 -0.12861061096191406
	grad: 3.0 6.0 -0.7721672058105469 -0.2573890686035156 -0.08579635620117188
progress: 49 0.0018402537098154426
	grad: 1.0 2.0 1.0404958724975586 1.0404958724975586 1.0404958724975586
	grad: 2.0 4.0 -0.5859127044677734 -0.2929563522338867 -0.14647817611694336
	grad: 3.0 6.0 -0.6678228378295898 -0.22260761260986328 -0.0742025375366211
progress: 50 0.0013765041949227452
	grad: 1.0 2.0 1.0178661346435547 1.0178661346435547 1.0178661346435547
	grad: 2.0 4.0 -0.6545791625976562 -0.3272895812988281 -0.16364479064941406
	grad: 3.0 6.0 -0.5675897598266602 -0.18919658660888672 -0.0630655288696289
progress: 51 0.0009943152545019984
	grad: 1.0 2.0 0.9961013793945312 0.9961013793945312 0.9961013793945312
	grad: 2.0 4.0 -0.7205257415771484 -0.3602628707885742 -0.1801314353942871
	grad: 3.0 6.0 -0.4712362289428711 -0.15707874298095703 -0.052359580993652344
progress: 52 0.0006853814120404422
	grad: 1.0 2.0 0.9751672744750977 0.9751672744750977 0.9751672744750977
	grad: 2.0 4.0 -0.7838821411132812 -0.3919410705566406 -0.1959705352783203
	grad: 3.0 6.0 -0.37866783142089844 -0.1262226104736328 -0.04207420349121094
progress: 53 0.00044255965622141957
	grad: 1.0 2.0 0.9550323486328125 0.9550323486328125 0.9550323486328125
	grad: 2.0 4.0 -0.8447341918945312 -0.4223670959472656 -0.2111835479736328
	grad: 3.0 6.0 -0.2897043228149414 -0.09656810760498047 -0.032189369201660156
progress: 54 0.000259038875810802
	grad: 1.0 2.0 0.9356646537780762 0.9356646537780762 0.9356646537780762
	grad: 2.0 4.0 -0.9031887054443359 -0.45159435272216797 -0.22579717636108398
	grad: 3.0 6.0 -0.2042255401611328 -0.06807518005371094 -0.022691726684570312
progress: 55 0.0001287286140723154
	grad: 1.0 2.0 0.9170365333557129 0.9170365333557129 0.9170365333557129
	grad: 2.0 4.0 -0.9593276977539062 -0.4796638488769531 -0.23983192443847656
	grad: 3.0 6.0 -0.12209415435791016 -0.04069805145263672 -0.013566017150878906
progress: 56 4.6009205107111484e-05
	grad: 1.0 2.0 0.8991179466247559 0.8991179466247559 0.8991179466247559
	grad: 2.0 4.0 -1.0132503509521484 -0.5066251754760742 -0.2533125877380371
	grad: 3.0 6.0 -0.043181419372558594 -0.014393806457519531 -0.004797935485839844
progress: 57 5.755046004196629e-06
	grad: 1.0 2.0 0.8818821907043457 0.8818821907043457 0.8818821907043457
	grad: 2.0 4.0 -1.065032958984375 -0.5325164794921875 -0.26625823974609375
	grad: 3.0 6.0 0.03263282775878906 0.010877609252929688 0.0036258697509765625
progress: 58 3.28673286276171e-06
	grad: 1.0 2.0 0.8653020858764648 0.8653020858764648 0.8653020858764648
	grad: 2.0 4.0 -1.1147651672363281 -0.5573825836181641 -0.27869129180908203
	grad: 3.0 6.0 0.10549449920654297 0.035164833068847656 0.011721611022949219
progress: 59 3.4349042834946886e-05
	grad: 1.0 2.0 0.8493533134460449 0.8493533134460449 0.8493533134460449
	grad: 2.0 4.0 -1.1625251770019531 -0.5812625885009766 -0.2906312942504883
	grad: 3.0 6.0 0.17549800872802734 0.05849933624267578 0.019499778747558594
progress: 60 9.506034257356077e-05
	grad: 1.0 2.0 0.834010124206543 0.834010124206543 0.834010124206543
	grad: 2.0 4.0 -1.2083892822265625 -0.6041946411132812 -0.3020973205566406
	grad: 3.0 6.0 0.24274635314941406 0.08091545104980469 0.026971817016601562
progress: 61 0.00018186973466072232
	grad: 1.0 2.0 0.8192505836486816 0.8192505836486816 0.8192505836486816
	grad: 2.0 4.0 -1.2524280548095703 -0.6262140274047852 -0.3131070137023926
	grad: 3.0 6.0 0.30735111236572266 0.10245037078857422 0.034150123596191406
progress: 62 0.0002915577497333288
	grad: 1.0 2.0 0.805051326751709 0.805051326751709 0.805051326751709
	grad: 2.0 4.0 -1.2947158813476562 -0.6473579406738281 -0.32367897033691406
	grad: 3.0 6.0 0.36942386627197266 0.12314128875732422 0.041047096252441406
progress: 63 0.0004212160420138389
	grad: 1.0 2.0 0.7913913726806641 0.7913913726806641 0.7913913726806641
	grad: 2.0 4.0 -1.3353118896484375 -0.6676559448242188 -0.3338279724121094
	grad: 3.0 6.0 0.4290504455566406 0.14301681518554688 0.047672271728515625
progress: 64 0.0005681613693013787
	grad: 1.0 2.0 0.7782487869262695 0.7782487869262695 0.7782487869262695
	grad: 2.0 4.0 -1.3742942810058594 -0.6871471405029297 -0.34357357025146484
	grad: 3.0 6.0 0.48633384704589844 0.1621112823486328 0.05403709411621094
progress: 65 0.0007300018914975226
	grad: 1.0 2.0 0.7656049728393555 0.7656049728393555 0.7656049728393555
	grad: 2.0 4.0 -1.4117164611816406 -0.7058582305908203 -0.35292911529541016
	grad: 3.0 6.0 0.5413684844970703 0.18045616149902344 0.06015205383300781
progress: 66 0.0009045674232766032
	grad: 1.0 2.0 0.7534389495849609 0.7534389495849609 0.7534389495849609
	grad: 2.0 4.0 -1.4476470947265625 -0.7238235473632812 -0.3619117736816406
	grad: 3.0 6.0 0.5942230224609375 0.1980743408203125 0.0660247802734375
progress: 67 0.0010898178443312645
	grad: 1.0 2.0 0.7417335510253906 0.7417335510253906 0.7417335510253906
	grad: 2.0 4.0 -1.4821281433105469 -0.7410640716552734 -0.3705320358276367
	grad: 3.0 6.0 0.6450004577636719 0.21500015258789062 0.07166671752929688
progress: 68 0.0012840295676141977
	grad: 1.0 2.0 0.7304706573486328 0.7304706573486328 0.7304706573486328
	grad: 2.0 4.0 -1.5152320861816406 -0.7576160430908203 -0.37880802154541016
	grad: 3.0 6.0 0.6937780380249023 0.23125934600830078 0.0770864486694336
progress: 69 0.001485580112785101
	grad: 1.0 2.0 0.7196331024169922 0.7196331024169922 0.7196331024169922
	grad: 2.0 4.0 -1.547006607055664 -0.773503303527832 -0.386751651763916
	grad: 3.0 6.0 0.7406158447265625 0.2468719482421875 0.0822906494140625
progress: 70 0.0016929376870393753
	grad: 1.0 2.0 0.709205150604248 0.709205150604248 0.709205150604248
	grad: 2.0 4.0 -1.5774974822998047 -0.7887487411499023 -0.39437437057495117
	grad: 3.0 6.0 0.7856168746948242 0.2618722915649414 0.08729076385498047
progress: 71 0.0019049193942919374
	grad: 1.0 2.0 0.6991701126098633 0.6991701126098633 0.6991701126098633
	grad: 2.0 4.0 -1.6067638397216797 -0.8033819198608398 -0.4016909599304199
	grad: 3.0 6.0 0.8288154602050781 0.2762718200683594 0.09209060668945312
progress: 72 0.0021201700437813997
	grad: 1.0 2.0 0.6895127296447754 0.6895127296447754 0.6895127296447754
	grad: 2.0 4.0 -1.6348438262939453 -0.8174219131469727 -0.40871095657348633
	grad: 3.0 6.0 0.8703403472900391 0.2901134490966797 0.09670448303222656
progress: 73 0.002337939338758588
	grad: 1.0 2.0 0.6802182197570801 0.6802182197570801 0.6802182197570801
	grad: 2.0 4.0 -1.661794662475586 -0.830897331237793 -0.4154486656188965
	grad: 3.0 6.0 0.9101743698120117 0.3033914566040039 0.10113048553466797
progress: 74 0.0025568436831235886
	grad: 1.0 2.0 0.6712737083435059 0.6712737083435059 0.6712737083435059
	grad: 2.0 4.0 -1.687643051147461 -0.8438215255737305 -0.42191076278686523
	grad: 3.0 6.0 0.9484634399414062 0.31615447998046875 0.10538482666015625
progress: 75 0.0027764905244112015
	grad: 1.0 2.0 0.6626648902893066 0.6626648902893066 0.6626648902893066
	grad: 2.0 4.0 -1.7124481201171875 -0.8562240600585938 -0.4281120300292969
	grad: 3.0 6.0 0.985224723815918 0.32840824127197266 0.10946941375732422
progress: 76 0.0029958882369101048
	grad: 1.0 2.0 0.6543788909912109 0.6543788909912109 0.6543788909912109
	grad: 2.0 4.0 -1.7362480163574219 -0.8681240081787109 -0.43406200408935547
	grad: 3.0 6.0 1.0205097198486328 0.34016990661621094 0.11338996887207031
progress: 77 0.00321432133205235
	grad: 1.0 2.0 0.6464033126831055 0.6464033126831055 0.6464033126831055
	grad: 2.0 4.0 -1.7590675354003906 -0.8795337677001953 -0.43976688385009766
	grad: 3.0 6.0 1.054387092590332 0.35146236419677734 0.11715412139892578
progress: 78 0.003431271994486451
	grad: 1.0 2.0 0.6387262344360352 0.6387262344360352 0.6387262344360352
	grad: 2.0 4.0 -1.7809600830078125 -0.8904800415039062 -0.4452400207519531
	grad: 3.0 6.0 1.0869426727294922 0.36231422424316406 0.12077140808105469
progress: 79 0.0036464333534240723
	grad: 1.0 2.0 0.6313357353210449 0.6313357353210449 0.6313357353210449
	grad: 2.0 4.0 -1.8019561767578125 -0.9009780883789062 -0.4504890441894531
	grad: 3.0 6.0 1.1181678771972656 0.3727226257324219 0.12424087524414062
progress: 80 0.003858948824927211
	grad: 1.0 2.0 0.6242218017578125 0.6242218017578125 0.6242218017578125
	grad: 2.0 4.0 -1.8220863342285156 -0.9110431671142578 -0.4555215835571289
	grad: 3.0 6.0 1.1481657028198242 0.3827219009399414 0.12757396697998047
progress: 81 0.004068779293447733
	grad: 1.0 2.0 0.6173720359802246 0.6173720359802246 0.6173720359802246
	grad: 2.0 4.0 -1.8413944244384766 -0.9206972122192383 -0.46034860610961914
	grad: 3.0 6.0 1.1769447326660156 0.3923149108886719 0.13077163696289062
progress: 82 0.004275305196642876
	grad: 1.0 2.0 0.6107778549194336 0.6107778549194336 0.6107778549194336
	grad: 2.0 4.0 -1.8599014282226562 -0.9299507141113281 -0.46497535705566406
	grad: 3.0 6.0 1.2045907974243164 0.40153026580810547 0.13384342193603516
progress: 83 0.004478515591472387
	grad: 1.0 2.0 0.6044282913208008 0.6044282913208008 0.6044282913208008
	grad: 2.0 4.0 -1.8776435852050781 -0.9388217926025391 -0.46941089630126953
	grad: 3.0 6.0 1.2311210632324219 0.4103736877441406 0.13679122924804688
progress: 84 0.004677960183471441
	grad: 1.0 2.0 0.5983142852783203 0.5983142852783203 0.5983142852783203
	grad: 2.0 4.0 -1.8946495056152344 -0.9473247528076172 -0.4736623764038086
	grad: 3.0 6.0 1.2565698623657227 0.4188566207885742 0.1396188735961914
progress: 85 0.004873357247561216
	grad: 1.0 2.0 0.5924272537231445 0.5924272537231445 0.5924272537231445
	grad: 2.0 4.0 -1.9109458923339844 -0.9554729461669922 -0.4777364730834961
	grad: 3.0 6.0 1.281005859375 0.427001953125 0.142333984375
progress: 86 0.005064740777015686
	grad: 1.0 2.0 0.5867581367492676 0.5867581367492676 0.5867581367492676
	grad: 2.0 4.0 -1.9265632629394531 -0.9632816314697266 -0.4816408157348633
	grad: 3.0 6.0 1.3044633865356445 0.43482112884521484 0.14494037628173828
progress: 87 0.005251928232610226
	grad: 1.0 2.0 0.5812978744506836 0.5812978744506836 0.5812978744506836
	grad: 2.0 4.0 -1.9415245056152344 -0.9707622528076172 -0.4853811264038086
	grad: 3.0 6.0 1.3269596099853516 0.4423198699951172 0.14743995666503906
progress: 88 0.00543463509529829
	grad: 1.0 2.0 0.5760388374328613 0.5760388374328613 0.5760388374328613
	grad: 2.0 4.0 -1.955862045288086 -0.977931022644043 -0.4889655113220215
	grad: 3.0 6.0 1.3485546112060547 0.44951820373535156 0.1498394012451172
progress: 89 0.005612961482256651
	grad: 1.0 2.0 0.5709733963012695 0.5709733963012695 0.5709733963012695
	grad: 2.0 4.0 -1.9695930480957031 -0.9847965240478516 -0.4923982620239258
	grad: 3.0 6.0 1.3692569732666016 0.4564189910888672 0.15213966369628906
progress: 90 0.005786619149148464
	grad: 1.0 2.0 0.5660943984985352 0.5660943984985352 0.5660943984985352
	grad: 2.0 4.0 -1.9827346801757812 -0.9913673400878906 -0.4956836700439453
	grad: 3.0 6.0 1.3891525268554688 0.46305084228515625 0.15435028076171875
progress: 91 0.00595600251108408
	grad: 1.0 2.0 0.5613937377929688 0.5613937377929688 0.5613937377929688
	grad: 2.0 4.0 -1.995330810546875 -0.9976654052734375 -0.49883270263671875
	grad: 3.0 6.0 1.4081897735595703 0.46939659118652344 0.1564655303955078
progress: 92 0.0061203655786812305
	grad: 1.0 2.0 0.5568656921386719 0.5568656921386719 0.5568656921386719
	grad: 2.0 4.0 -2.0073795318603516 -1.0036897659301758 -0.5018448829650879
	grad: 3.0 6.0 1.4265060424804688 0.47550201416015625 0.15850067138671875
progress: 93 0.006280615925788879
	grad: 1.0 2.0 0.5525016784667969 0.5525016784667969 0.5525016784667969
	grad: 2.0 4.0 -2.0189208984375 -1.00946044921875 -0.504730224609375
	grad: 3.0 6.0 1.4440240859985352 0.4813413619995117 0.1604471206665039
progress: 94 0.006435819435864687
	grad: 1.0 2.0 0.5482978820800781 0.5482978820800781 0.5482978820800781
	grad: 2.0 4.0 -2.0299606323242188 -1.0149803161621094 -0.5074901580810547
	grad: 3.0 6.0 1.460855484008789 0.4869518280029297 0.16231727600097656
progress: 95 0.006586724426597357
	grad: 1.0 2.0 0.544245719909668 0.544245719909668 0.544245719909668
	grad: 2.0 4.0 -2.04052734375 -1.020263671875 -0.5101318359375
	grad: 3.0 6.0 1.4769744873046875 0.4923248291015625 0.1641082763671875
progress: 96 0.006732881534844637
	grad: 1.0 2.0 0.5403413772583008 0.5403413772583008 0.5403413772583008
	grad: 2.0 4.0 -2.050628662109375 -1.0253143310546875 -0.5126571655273438
	grad: 3.0 6.0 1.4924583435058594 0.4974861145019531 0.16582870483398438
progress: 97 0.006874789949506521
	grad: 1.0 2.0 0.5365777015686035 0.5365777015686035 0.5365777015686035
	grad: 2.0 4.0 -2.0602989196777344 -1.0301494598388672 -0.5150747299194336
	grad: 3.0 6.0 1.5072813034057617 0.5024271011352539 0.16747570037841797
progress: 98 0.007012027781456709
	grad: 1.0 2.0 0.5329499244689941 0.5329499244689941 0.5329499244689941
	grad: 2.0 4.0 -2.0695419311523438 -1.0347709655761719 -0.5173854827880859
	grad: 3.0 6.0 1.5214862823486328 0.5071620941162109 0.1690540313720703
progress: 99 0.007144816219806671
predict(after training): 4 8.724994659423828