pytorch-网络层初始化记录

为节省时间，特此记录有关pytorch-权重初始化和L2_weight设置的东西。

初始化：

pytorch 官网有关自然语言处理-权重初始化

def init_weights(m: nn.Module):
    for name, param in m.named_parameters():
        if 'weight' in name:
            nn.init.normal_(param.data, mean=0, std=0.01)
        else:
            nn.init.constant_(param.data, 0)


model.apply(init_weights)

l2-weight正则化项：有的只对权重进行正则化，不对偏执进行正则化。当时查了很久，觉得靠谱的版本特此记录：

参考：

https://discuss.pytorch.org/t/l2-regularization-with-only-weight-parameters/56853/2

https://blog.csdn.net/loseinvain/article/details/81708474#commentsedit

    weight_p, bias_p = [], []
    for name, p in model.named_parameters():
        if 'bias' in name:
            bias_p += [p]
        else:
            weight_p += [p]

    optimizer = optim.Adam(
        [{'params': bias_p, 'weight_decay': 0},
         {'params': weight_p, 'weight_decay': 1e-2}],
        lr=TRAIN_LR,
        betas=(0.5, 0.999)
    )