ResNeXt
ResNeXt的PyTorch实现：https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/resnext.py
以ResNeXt101_64x4d为例
(224, 224, 3)
→【self.features】→(7, 7, 2048)
→【self.avg_pool, ks=7, s=1】→(1, 1, 2048)
→【reshape】→(2048,)→【self.last_linear】→(1000,)
下面着重研究self.features的结构，源代码：https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/resnext_features/resnext101_64x4d_features.py
(224, 224, 3)
→【Conv, Cout=64, ks=7, s=2, p=3】→【bn, relu】→(112, 112, 64)
→【max pool, ks=3, s=2, p=1】→(56, 56, 64)
→【Sequence_1】→(56, 56, 256)
→【Sequence_2】→(28, 28, 512)
→【Sequence_3】→(14, 14, 1024)
→【Sequence_4】→()
研究第1个Sequence的结构：(56, 56, 64)→【Sequence_1】→(56, 56, 256)
(56, 56, 64)
→【Sequence_1-Block1】→(56, 56, 256)
→【Sequence_1-Block2】→(56, 56, 256)
→【Sequence_1-Block3】→(56, 56, 256)
第1个Block，(56, 56, 64)→【Sequence_1-Block1】→(56, 56, 256)，不改变空间上的维度，将通道数从64增加到256，代码如下
nn.Sequential(#Sequential,
LambdaMap(lambda x: x, #ConcatTable,
nn.Sequential(#Sequential,
nn.Sequential(#Sequential,
nn.Conv2d(64, 256, (1, 1), (1, 1), (0, 0), 1, 1, bias = False),
nn.BatchNorm2d(256),
nn.ReLU(),
nn.Conv2d(256, 256, (3, 3), (1, 1), (1, 1), 1, 64, bias = False),
nn.BatchNorm2d(256),
nn.ReLU(),
),
nn.Conv2d(256, 256, (1, 1), (1, 1), (0, 0), 1, 1, bias = False),
nn.BatchNorm2d(256),
),
nn.Sequential(#Sequential,
nn.Conv2d(64, 256, (1, 1), (1, 1), (0, 0), 1, 1, bias = False),
nn.BatchNorm2d(256),
),
),
LambdaReduce(lambda x, y: x + y), #CAddTable,
nn.ReLU(),
)
上述代码中，LambdaMap产生分支，LambdaReduce合并分支
(56, 56, 64)
分支1：→【Conv, Cout=256, ks=1,】→(56, 56, 256)→【bn, relu】→(56, 56, 256)
→【Conv, Cout=256, ks=3, p=1，group=64】→(56, 56, 256)→【bn, relu】
→【Conv, Cout=256, ks=1】→(56, 56, 256)→【bn】→(56, 56, 256)
分支2(short cut)：→【Conv, Cout=256, ks=1】→(56, 56, 256)→【bn】→(56, 56, 256)
合并add：→(56, 56, 256)→【relu】→(56, 56, 256)
第2个Block，(56, 56, 256)→【Sequence_1-Block2】→(56, 56, 256)，相当于Identity Block，代码如下
nn.Sequential(#Sequential,
LambdaMap(lambda x: x, #ConcatTable,
nn.Sequential(#Sequential,
nn.Sequential(#Sequential,
nn.Conv2d(256, 256, (1, 1), (1, 1), (0, 0), 1, 1, bias = False),
nn.BatchNorm2d(256),
nn.ReLU(),
nn.Conv2d(256, 256, (3, 3), (1, 1), (1, 1), 1, 64, bias = False),
nn.BatchNorm2d(256),
nn.ReLU(),
),
nn.Conv2d(256, 256, (1, 1), (1, 1), (0, 0), 1, 1, bias = False),
nn.BatchNorm2d(256),
),
Lambda(lambda x: x), #Identity,
),
LambdaReduce(lambda x, y: x + y), #CAddTable,
nn.ReLU(),
)
(56, 56, 256)
分支1：→【Conv, Cout=256, ks=1,】→(56, 56, 256)→【bn, relu】→(56, 56, 256)
→【Conv, Cout=256, ks=3, p=1，group=64】→(56, 56, 256)→【bn, relu】
→【Conv, Cout=256, ks=1】→(56, 56, 256)→【bn】→(56, 56, 256)
分支2(short cut)：→【Identity】→(56, 56, 256)
合并add：→(56, 56, 256)→【relu】→(56, 56, 256)
【附】参数group
默认情况下groups=1，表示常规的卷积，例如：(7, 7, 6)→【Conv, Cout=12, ks=3, groups=1】→(5, 5, 12)，weight为(12, 3, 3, 6)
若指定groups=3，(7, 7, 6)→【Conv, Cout=12, ks=3, groups=3】→(5, 5, 12)，output的size不变，但是weight变为(12, 3, 3, 2)，显然计算的方式发生了改变
groups=3表示input, output的通道数都被分成了3组，具体来说
input (7, 7, 6)被分为3组，每组包含6/3=2个通道，size为(7, 7, 2)
即input[:, :, 0:2], input[:, :, 2:4], input[:, :, 4:6]
12个filter被分为3组，每组包含12/3=4个filter，size为(4, 3, 3, 2)（注意最后一维是2，与每一组input的通道数相等）
即weight[0:4], weight[4:8], weight[8:12]
每组的input和filter分别进行卷积运算，得到3组output，每组output的size为(5, 5, 4)
将它们在通道维度上拼接起来，最终output的size为(5, 5, 12)
以下是一段验证代码
import torch
import torch.nn as nn
import torch.nn.functional as F
x = torch.rand(1, 6, 7, 7)
layer = nn.Conv2d(6, 12, kernel_size=3, groups=3, bias=False)
output = layer(x)
output2 = []
w = layer.weight
for i in range(layer.groups):
c = layer.in_channels // layer.groups
x_slice = x[:, i*c:(i+1)*c]
c = layer.out_channels // layer.groups
w_slice = w[i*c:(i+1)*c]
output2.append( F.conv2d(x_slice, w_slice) )
output2 = torch.cat(output2, dim=1)
print( (output - output2).sum().item() )