nn.Module之激活函数

最常用的ReLU函数

relu = nn.ReLU(inplace = True)
input = t.randn(2, 3)
print(input)
output = relu(input)
print(output)

out：
tensor([[-0.1881, -0.0310, -1.4321],
        [-0.8111,  0.1544,  0.2727]])
tensor([[ 0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.1544,  0.2727]])  # 小于0的截断

ReLU函数的inplace参数，如果设为True，它会把输出直接覆盖到输入中，节省内存/显存，但是一般不要使用inplace操作。

对于前馈神经网络，每次写forward函数会有些麻烦，因此有两种简化方式，ModuleList和Sequential
Sequential是一个特殊的Module，它包含几个子Module，向前传播时会将输入一层接一层的传递下去。

"""
Sequential 的三种写法
"""
net1 = nn.Sequential()
net1.add_module('conv',nn.Conv2d(3,3,3))
net1.add_module('batchnorm', nn.BatchNorm2d(3))
net1.add_module('activation_layer',nn.ReLU())

net2 = nn.Sequential(
        nn.Conv2d(3,3,3),
        nn.BatchNorm2d(3),
        nn.ReLU()
        )

from collections import OrderedDict
net3 = nn.Sequential(OrderedDict([
        ('conv1', nn.Conv2d(3,3,3)),
        ('bn1',nn.BatchNorm2d(3)),
        ('relu1',nn.ReLU())
        ]))
print('net1',net1)
print('net2',net2)
print('net3',net3)

结果

net1 Sequential(
  (conv): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))
  (batchnorm): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (activation_layer): ReLU()
)
net2 Sequential(
  (0): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))
  (1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU()
)
net3 Sequential(
  (conv1): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))
  (bn1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu1): ReLU()
)

输出

In：
input = t.rand(1, 3, 4, 4)
output = net1(input)
output = net2(input)
output = net3(input)

 Out：
 tensor([[[[ 0.0000,  0.9335],
          [ 0.0000,  0.0000]],

         [[ 0.0000,  0.6925],
          [ 0.0000,  0.0000]],

         [[ 0.5271,  0.0000],
          [ 0.5408,  0.2767]]]])