pytorch两种常用的学习率衰减方法

阶梯式衰减

torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)

每个训练step_size个epoch，lr会自动乘以gamma

LR = 0.01
optimizer = Adam(model.parameters(),lr = LR)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer,step_size=5,gamma = 0.9)
for epoch in range(100):
    net.fit(x,y) # 注意要先训练再更新学习率
    scheduler.step()  # 更新学习率

 指数衰减

torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma, last_epoch=-1)

每个epoch中lr都乘以gamma

LR = 0.01
optimizer = Adam(model.parameters(),lr = LR)
scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.99)
for epoch in range(100):
    net.fit(x,y) # 注意要先训练再更新学习率
    scheduler.step()  # 更新学习率

 其他学习率衰减方法可以看https://www.jianshu.com/p/9643cba47655