211014记录：pytorch模型中初始化公式（共13种）总结torch.nn.init
torch官方网址：
https://pytorch.org/docs/stable/nn.init.html
1、torch.nn.init.uniform_(tensor, a=0.0, b=1.0)#均匀分布U(a,b)
2、 torch.nn.init.normal_(tensor, mean=0.0, std=1.0)
3、torch.nn.init.constant_(tensor, val)
4、 torch.nn.init.ones_(tensor)
5、 torch.nn.init.zeros_(tensor)
6、torch.nn.init.eye_(tensor)
7、 torch.nn.init.dirac_(tensor, groups=1)
import torch
from torch import nn
w = torch.empty(3, 5)
w
tensor([[0.8851, 0.1706, 1.0505, 1.1845, 0.8852],
[0.2051, 0.8593, 0.3408, 0.0350, 0.4771],
[0.0308, 0.6101, 0.9387, 1.2147, 0.2101]])
nn.init.constant_(w, 0.3)
tensor([[0.3000, 0.3000, 0.3000, 0.3000, 0.3000],
[0.3000, 0.3000, 0.3000, 0.3000, 0.3000],
[0.3000, 0.3000, 0.3000, 0.3000, 0.3000]])
nn.init.ones_(w)
tensor([[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.]])
nn.init.eye_(w)
tensor([[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.],
[0., 0., 1., 0., 0.]])
y = torch.empty(3, 16,5,5)
nn.init.dirac_(y)#Dirac 三角函数
tensor([[[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]])
8、torch.nn.init.xavier_uniform_(tensor, gain=1.0)
9、torch.nn.init.xavier_normal_(tensor, gain=1.0)
具体讲解可参考：https://blog.csdn.net/dss_dssssd/article/details/83959474
10、torch.nn.init.orthogonal_(tensor, gain=1)
11、torch.nn.init.kaiming_uniform_(tensor, a=0, mode=‘fan_in’, nonlinearity=‘leaky_relu’)
12、torch.nn.init.kaiming_normal_(tensor, a=0, mode=‘fan_in’, nonlinearity=‘leaky_relu’)
13、torch.nn.init.sparse_(tensor, sparsity, std=0.01)
w
tensor([[-0.6032, 0.2567, -0.2297, 0.3286, -0.1441],
[-0.6870, 0.0125, 0.4179, -0.0441, -0.7955],
[ 0.0219, -0.0436, 0.2819, -0.3877, 0.0829]])
nn.init.orthogonal_(w)
#10、torch.nn.init.orthogonal_(tensor, gain=1)
tensor([[-0.1015, 0.8846, -0.3718, 0.1489, -0.2163],
[ 0.5792, 0.3852, 0.3320, -0.5103, 0.3816],
[ 0.1322, 0.0781, 0.0669, 0.7398, 0.6516]])
nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')
#11、torch.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')
#sampled from U(−bound,bound)
tensor([[ 0.5894, 0.3218, 0.9177, -0.7206, 0.5082],
[-0.5999, 0.7737, -0.2971, -0.9128, -0.1794],
[ 0.7544, -0.1546, 0.4230, -0.6117, 0.9302]])
nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')
#12、torch.nn.init.kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')
#have values sampled from N（0，std2）
tensor([[ 0.5359, 0.0348, 0.2314, 0.9119, -0.1461],
[ 1.1514, 1.2837, -0.7474, -0.1975, -0.7097],
[-0.5272, 0.9738, 0.6795, 0.4681, 0.3370]])
nn.init.sparse_(w, sparsity=0.1)
#13、torch.nn.init.sparse_(tensor, sparsity, std=0.01)
#N(0,0.01)
tensor([[ 0.0088, -0.0029, 0.0025, -0.0073, 0.0000],
[-0.0055, 0.0000, 0.0000, 0.0014, -0.0087],
[ 0.0000, 0.0080, 0.0054, 0.0000, 0.0009]])