动手学深度学习14- pytorch Dropout 实现与原理
- 方法
- 从零开始实现
- 定义模型参数
- 网络
- 评估函数
- 优化方法
- 定义损失函数
- 数据提取与训练评估
- pytorch简洁实现
- 小结
针对深度学习中的过拟合问题，通常使用丢弃法(dropout),丢弃法有很多的变体，本文提高的丢弃法特指倒置丢弃法(inverted dorpout)。
方法
在会议多层感知机的图3.3描述了一个单隐藏层的多层感知机。其中输入个数为4，隐藏单元个数为5，且隐藏单元$h_(1,2,3,4,5)$的计算表达式为 \(h_{i} = \varphi(x_{1}w_{1i}+x_{2}w_{2i}+x_{1}w_{3i}+x_{1}w_{4i}+b_{i})\) 这个里的$\varphi$是激活函数，\(x_{1},x_{2},x_{3},x_{4}是输入，隐层单元i的权重参数为w_{1i}，w_{2i}，w_{3i}，w_{4i}，偏置参数为b_{i}\)，当对该隐藏层使用丢弃法是，该层的隐藏单元将有一定概率的被丢弃掉。设丢弃的概率为P,那么有p的概率hi会被清零，有1-p的概率hi会除以1-p做拉伸。丢弃概率是丢弃法的超参数。具体来说，设随机变量$\xi为0和1的概率分别是p和1-p。使用丢弃法时，我们使用计算新的隐藏单元h_^{'}$ \(h_{i}^{'} = \frac{\xi_{i}}{1-p}h_{i}\) 由于$E(\xi_) = 1-p $,因此 \(E(\xi_{i}^{'}) = \frac{E(\xi_{i})}{1-p}h_{i} = h_{i}\) 丢弃法不改变其输入的期望值。
如图3.5所示，其中，h2和h5被清零，这时输出值的计算不再依赖h2和h5，在反向传播时，与这两个隐藏单元相关的权重的梯度均为0。由于训练汇总隐藏层的神经元丢弃是随机的，即h1,h2,h3,h4,h5都可能被清零，输出层计算无法过度依赖h1,h2,h3..h5中的任一个，从而在训练模型时起到的正则化的作用，可以用来应付过拟合。在测试模型时，我们为了拿到更加确定的结果，一般不适用丢弃法。
从零开始实现
%matplotlib inline
import torch
import torch.nn as nn
import numpy as np
import sys
sys.path.append('..')
import d2lzh_pytorch as d2l
def dropout(X,drop_prob):
X = X.float()
assert 0<=drop_prob<=1
keep_prob = 1-drop_prob
if keep_prob==0:
return torch.torch.zeros_like(X)
mask = (torch.rand(X.shape)X = torch.arange(16).view(2,8)
dropout(X,0)
tensor([[ 0., 1., 2., 3., 4., 5., 6., 7.],
[ 8., 9., 10., 11., 12., 13., 14., 15.]])
dropout(X,0.5)
tensor([[ 0., 0., 0., 0., 0., 10., 0., 14.],
[16., 18., 0., 0., 0., 26., 28., 30.]])
dropout(X,1)
tensor([[0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0.]])
定义模型参数
num_inputs,num_outputs, num_hidden1,num_hidden2 = 784,10,256,256
W1 = torch.tensor(np.random.normal(0,0.01,size=(num_inputs,num_hidden1)),dtype =torch.float32,requires_grad=True )
b1 = torch.zeros(num_hidden1,requires_grad=True)
W2 = torch.tensor(np.random.normal(0,0.01,size=(num_hidden1,num_hidden2)),dtype =torch.float32,requires_grad=True )
b2 = torch.zeros(num_hidden2,requires_grad=True)
W3 = torch.tensor(np.random.normal(0,0.01,size=(num_hidden2,num_outputs)),dtype =torch.float32,requires_grad=True )
b3 = torch.zeros(num_outputs,requires_grad=True)
params = [W1,b1,W2,b2,W3,b3]
网络
drop_prob1,drop_prob2 = 0.2,0.5
def net(X,is_training=True):
X = X.view(-1,num_inputs)
H1 = (torch.matmul(X,W1)+b1).relu()
if is_training:
H1 = dropout(H1,drop_prob1)
H2 = (torch.matmul(H1,W2)+b2).relu()
if is_training:
H2 = dropout(H2,drop_prob2)
return torch.matmul(H2,W3)+b3
评估函数
def evaluate_accuracy(data_iter,net):
acc_sum ,n = 0.0,0
for X,y in data_iter:
if isinstance(net,torch.nn.Module): #如果是torch.nn里简洁的实现的模型
net.eval() # 评估模式，这时会关闭Dropout
acc_sum+=(net(X).argmax(dim=1)==y).float().sum().item()
net.train() # 改回训练模式
else: # 自己定义的模型
if ('is_training' in net.__code__.co_varnames): # 如果有训练这个参数
# 将is_training 设置为False
acc_sum +=(net(X,is_training=False).argmax(dim=1)==y).float().sum().item()
else:
acc_sum+=(net(X),argmax(dim=1)==y).float().sum().item()
n+= y.shape[0]
return acc_sum/n
优化方法
def sgd(params,lr,batch_size):
for param in params:
# param.data -=lr* param.grad/batch_size
param.data-= lr* param.grad # 计算loss使用的是pytorch的交叉熵
# 这个梯度可以不用除以batch_size,pytorch 在计算loss的时候已经除过一次了，
定义损失函数
loss = torch.nn.CrossEntropyLoss()
数据提取与训练评估
num_epochs,lr,batch_size=15,0.3,256
batch_size = 256
train_iter,test_iter = d2l.get_fahsion_mnist(batch_size)
def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size,
params=None, lr=None, optimizer=None):
for epoch in range(num_epochs):
train_l_sum, train_acc_sum, n = 0.0, 0.0, 0
for X, y in train_iter:
y_hat = net(X)
l = loss(y_hat, y).sum()
# 梯度清零
if optimizer is not None:
optimizer.zero_grad()
elif params is not None and params[0].grad is not None:
for param in params:
param.grad.data.zero_()
l.backward()
if optimizer is None:
sgd(params, lr, batch_size)
else:
optimizer.step() # “softmax回归的简洁实现”一节将用到
train_l_sum += l.item()
train_acc_sum += (y_hat.argmax(dim=1) == y).sum().item()
n += y.shape[0]
test_acc = evaluate_accuracy(test_iter, net)
print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'
% (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))
train_ch3(net,train_iter,test_iter,loss,num_epochs,batch_size,params,lr)
epoch 1, loss 0.0049, train acc 0.513, test acc 0.693
epoch 2, loss 0.0024, train acc 0.776, test acc 0.781
epoch 3, loss 0.0020, train acc 0.818, test acc 0.780
epoch 4, loss 0.0018, train acc 0.835, test acc 0.846
epoch 5, loss 0.0017, train acc 0.846, test acc 0.843
epoch 6, loss 0.0016, train acc 0.855, test acc 0.843
epoch 7, loss 0.0015, train acc 0.861, test acc 0.843
epoch 8, loss 0.0015, train acc 0.863, test acc 0.855
epoch 9, loss 0.0014, train acc 0.870, test acc 0.861
epoch 10, loss 0.0014, train acc 0.872, test acc 0.845
epoch 11, loss 0.0013, train acc 0.874, test acc 0.853
epoch 12, loss 0.0013, train acc 0.878, test acc 0.848
epoch 13, loss 0.0013, train acc 0.880, test acc 0.859
epoch 14, loss 0.0013, train acc 0.882, test acc 0.858
epoch 15, loss 0.0012, train acc 0.885, test acc 0.863
pytorch简洁实现
net = nn.Sequential(
d2l.FlattenLayer(),
nn.Linear(num_inputs,num_hidden1),
nn.ReLU(),
nn.Dropout(drop_prob1),
nn.Linear(num_hidden1,num_hidden2),
nn.ReLU(),
nn.Dropout(drop_prob2),
nn.Linear(num_hidden2,num_outputs)
)
for param in net.parameters():
nn.init.normal_(param,mean=0,std=0.01)
optimizer = torch.optim.SGD(net.parameters(),lr=0.3)
train_ch3(net,train_iter,test_iter,loss,num_epochs,batch_size,None,None,optimizer)
epoch 1, loss 0.0048, train acc 0.525, test acc 0.725
epoch 2, loss 0.0024, train acc 0.779, test acc 0.787
epoch 3, loss 0.0020, train acc 0.818, test acc 0.771
epoch 4, loss 0.0018, train acc 0.836, test acc 0.834
epoch 5, loss 0.0017, train acc 0.847, test acc 0.848
epoch 6, loss 0.0016, train acc 0.855, test acc 0.855
epoch 7, loss 0.0015, train acc 0.859, test acc 0.850
epoch 8, loss 0.0014, train acc 0.863, test acc 0.853
epoch 9, loss 0.0014, train acc 0.868, test acc 0.848
epoch 10, loss 0.0014, train acc 0.872, test acc 0.837
epoch 11, loss 0.0013, train acc 0.876, test acc 0.849
epoch 12, loss 0.0013, train acc 0.879, test acc 0.872
epoch 13, loss 0.0013, train acc 0.880, test acc 0.847
epoch 14, loss 0.0013, train acc 0.883, test acc 0.862
epoch 15, loss 0.0012, train acc 0.886, test acc 0.865
小结
- 可以使用Dropout应对过拟合
- 丢弃法只能在训练模型时使用