深度学习之-玄学调参

``## 深度学习之-玄学调参
**在code动手学深度学习的Resnet代码时，出现了如下错误：RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 512, 3, 3]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
**
相关代码如下：

#ResNET
class Residual(nn.Module):
    def __init__(self,input_channels,num_channels,use_1_1conv=False,strides=1):
        super().__init__()
        self.conv1 = nn.Conv2d(input_channels,num_channels,kernel_size=3,padding=1,stride=strides)
        self.conv2 = nn.Conv2d(num_channels,num_channels,kernel_size=3,padding=1)
        if use_1_1conv:
            self.conv3 = nn.Conv2d(input_channels,num_channels,kernel_size=1,stride=strides)
        else:
            self.conv3 = None
        self.bn1 = nn.BatchNorm2d(num_channels)
        self.bn2 = nn.BatchNorm2d(num_channels)
    def forward(self,x):
        y = F.relu(self.bn1(self.conv1(x)))
        y = F.relu(self.bn2(self.conv2(y)))
        if self.conv3:
            x = F.relu(self.conv3(x))
        y += x
        return F.relu(y)

resnet_b1 = nn.Sequential(
    nn.Conv2d(1,64,kernel_size=7,padding=3,stride=2),nn.BatchNorm2d(64),nn.ReLU(),
    nn.MaxPool2d(kernel_size=3,stride=2,padding=1)
)
def resnet_block(input_channels,num_channels,num_residuals,first_block=False):
    blk = []
    for i in range(num_residuals):
        if i == 0 and not first_block:
            blk.append(Residual(input_channels,num_channels,use_1_1conv=True,strides=2))
        else:
            blk.append(Residual(num_channels,num_channels))
    return blk
resnet_b2 = nn.Sequential(*resnet_block(64,64,2,first_block=True))
resnet_b3 = nn.Sequential(*resnet_block(64,128,2))
resnet_b4 = nn.Sequential(*resnet_block(128,256,2))
resnet_b5 = nn.Sequential(*resnet_block(256,512,2))
#resnet-18
resnet_net = nn.Sequential(resnet_b1,resnet_b2,resnet_b3,resnet_b4,resnet_b5,
                           nn.AdaptiveAvgPool2d((1,1)),
                           nn.Flatten(),
                           nn.Linear(512,10)
                           )

lr,num_epochs,batch_size = 0.05,10,256
train_iter,test_iter = d2l.load_data_fashion_mnist(batch_size,resize=96)
train_ch6(resnet_net,train_iter,test_iter,num_epochs,lr,d2l.try_gpu())

然后就报错了。。。。。。
尝试了一推，譬如：
1.变量名不能用一样的，不行，pass
2.梯度更新放在最后面，optimizer.step(),但是这个训练代码训练其他的googlenet等网络都没有问题，pass
3.问题应该在resnet本身。试了很多，发现时残差本身的合并步骤出了错，即第一个模块的最后一步：

	y += x

+=导致了inplace操作，修改成如下即可：

	y = x + y

附上结果

loss:0.025,train_acc:0.992,test_acc:0.869
1318.1269680345883 example/sec on cuda:0

源代码抄书的，出了bug，真就瞎调参，噗嗤！
参考：

https://zh.d2l.ai/