【vision transformer】DETR原理及代码详解（四）
本节是 DETR流程及 构建backbone和position embedding 相关部分的代码解析
一、DETR代码流程：
STEP 1: Create model and criterion #构建模型和标准
STEP 2: Create train and val dataloader #构建训练和测试数据集
STEP 3: Define lr_scheduler #定义学习策略
STEP 4: Define optimizer #定义优化器
STEP 5: Load pretrained model or load resume model and optimizer states 加载预训练模型
STEP 6: Validation
STEP 7: Start training and validation
STEP 8：Model save
STEP 1: Create model and criterion #构建模型和标准
model, criterion, postprocessors = build_detr(config)def build_detr(config):
""" build detr model from config"""
# 1. build backbone with position embedding
backbone = build_backbone(config)
# 2. build transformer (encoders and decoders)
transformer = build_transformer(config)
# 3. build DETR model
aux_loss = not config.EVAL # set true during training
detr = DETR(backbone=backbone,
transformer=transformer,
num_classes=config.MODEL.NUM_CLASSES,
num_queries=config.MODEL.NUM_QUERIES,
aux_loss=aux_loss)
# 4. build matcher
matcher = build_matcher()
# 5. setup aux_loss
weight_dict = {'loss_ce': 1., 'loss_bbox': 5., 'loss_giou': 2.}
if aux_loss:
aux_weight_dict = {}
for i in range(config.MODEL.TRANS.NUM_DECODERS-1):
aux_weight_dict.update({k + f'_{i}': v for k, v in weight_dict.items()})
weight_dict.update(aux_weight_dict)
# 6. build criterion
losses = ['labels', 'boxes', 'cardinality']
criterion = SetCriterion(config.MODEL.NUM_CLASSES,
matcher=matcher,
weight_dict=weight_dict,
eos_coef=0.1,
losses=losses)
# 7. build postprocessers
postprocessors = {'bbox': PostProcess()}
return detr, criterion, postprocessorsbuild_detr 函数涉及build_backbone ，build_transformer，build_DETR model，build_match，
设置 aux_loss，aux_loss代表是否要对Transformer中Decoder的每层输出都计算loss。
构建 postprossers：PostProcess()这个函数是将output转换位coco api的格式。
二、Build backbone with position embedding
（1）backbone
用的是常见的resnet网络, backbone调用resnet50中把主干网络提取出来。
pretrained=paddle.distributed.get_rank(),表示在分布式当前进程中使用预训练权重。
paddle.distributed分布式训练，get_rank()获取当前进程值。
norm_layer=FrozenBatchNorm2D 归一化层，类似BN，将统计量（均值与方差）和可学习的仿射参数固定住。在实现的时候，需要将以上4个量注册到buffer，以便阻止梯度反向传播而更新它们，同时又能够记录在模型的state_dict中。
replace_stride_with_dilation=[False, False, dilation], 替换步长用膨胀率，如果为None，设置默认值为[False, False, False]
class Backbone(BackboneBase):
"""Get resnet backbone from resnet.py with multiple settings and return BackboneBase instance"""
def __init__(self, name, train_backbone, return_interm_layers, dilation, backbone_lr):
backbone = getattr(resnet, name)(pretrained=paddle.distributed.get_rank() == 0,
norm_layer=FrozenBatchNorm2D,
replace_stride_with_dilation=[False, False, dilation],
backbone_lr=backbone_lr)
num_channels = 512 if name in ('resnet18', 'resnet34') else 2048
super().__init__(backbone, train_backbone, num_channels, return_interm_layers)（2）BackboneBase
在BackboneBase中，若 return_interm_layers = true，则需要记录resnet每一层的输出。
self.body = IntermediateLayerGetter(backbone, return_layers=return_layers)
获取网络中间几层的结果IntermediateLayerGetter（），它的输出是一个dict，对应了每层的输出，key是用户自定义的赋予输出特征的名字。
思想：先创建一个model ,然后把它传入IntermediateLayerGetter中，并传入一个字典，传入字典的key是model的直接的层，传入字典的value是返回字典中的key，返回字典的value对应的是model运行的中间结果。
在定义它的时候注明作用的模型和要返回的layer，得到new_m。使用时喂输入变量，返回的就是对应的layer。
mask = F.interpolate(m, size=x.shape[-2:])[0] #[batch_size, feat_h, fea_w]
将mask插值到与输出特征图尺寸一致。
NestedTensor()函数,包括tensor和mask两个成员，tensor就是输入的图像。mask跟tensor同高宽但是单通道。
BackboneBase 的前向方法中输入是NestedTensor 这个类的实例，其实质就是将图像张量和对应的mask封装到一起。
class BackboneBase(nn.Layer):
"""Backbone Base class for NestedTensor input and multiple outputs
This class handles the NestedTensor as input, run inference through backbone
and return multiple output tensors(NestedTensors) from selected layers
"""
def __init__(self,
backbone: nn.Layer,
train_backbone: bool,
num_channels: int,
return_interm_layers: bool):
super().__init__()
for name, parameter in backbone.named_parameters():
if not train_backbone or 'layer2' not in name and 'layer3' not in name and 'layer4' not in name:
parameter.stop_gradient = True
if return_interm_layers:
return_layers = {'layer0': '0', 'layer2': '1', 'layer3':'2', 'layer4':'3'}
else:
return_layers = {'layer4': '0'}
self.body = IntermediateLayerGetter(backbone, return_layers=return_layers)
self.num_channels = num_channels
def forward(self, tensor_list):
#Inference through resnet backbone, which takes the paddle.Tensor as input
#tensor_list contains .tensor(paddle.Tensor) and .mask(paddle.Tensor) for batch inputs
xs = self.body(tensor_list.tensors)
out = {}
for name, x in xs.items():
# x.shape: [batch_size, feat_dim, feat_h, feat_w]
m = tensor_list.mask # [batch_size, orig_h, orig_w]
assert m is not None
m = m.unsqueeze(0).astype('float32') # [1, batch_size, orig_h, orig_w]
mask = F.interpolate(m, size=x.shape[-2:])[0] #[batch_size, feat_h, fea_w]
# 将mask插值到与输出特征图尺寸一致
out[name] = NestedTensor(x, mask)
return out(3)backbone 与 position embedding
DETR将位置编码模块与backbone 通过Joiner（）合在一起作为modle，在backbone 输出特征图的同时对其进行位置编码，以供后续Transformer 使用。
joiner是一个小型库，用于以类似sql连接的方式合并数据组。此处Joiner 就是将backbone 和position encoding 集成到一个模块nn.Module，使得前向过程中更方便地使用两者的功能。
Joiner 是nn.Sequectial 的子类，通过初始化，使得self[0]是backbone，self[1]是position-encoding。前向过程就是对backbone 的每层输出都进行位置编码，最终返回backbone的输出及对应的位置编码结果。
class Joiner(nn.Sequential):
""" Joiner layers(nn.Sequential) for backbone and pos_embed
Arguments:
backbone: nn.Layer, backbone layer (resnet)
position_embedding: nn.Layer, position_embedding(learned, or sine)
"""
def __init__(self, backbone, position_embedding):
super().__init__(backbone, position_embedding)
def forward(self, x):
# feature from resnet backbone inference
xs = self[0](x)
out = []
pos = []
# for each backbone output, apply position embedding
for name, xx in xs.items():
out.append(xx)
pos.append(self[1](xx).astype(xx.tensors.dtype))
return out, pos
（4）build_backbone
def build_backbone(config):
""" build resnet backbone and position embedding according to config """
#对backbone输出的特征图进行位置编码，用于后续Transformer 部分
assert config.MODEL.BACKBONE in ['resnet50', 'resnet101'], "backbone name is not supported!"
backbone_name = config.MODEL.BACKBONE
dilation = False
train_backbone = not config.EVAL
#是否需要训练backbone （即是否采用预训练backbone）
return_interm_layers = False #TODO: impl case True for segmentation
#是否需要记录backbone的每层输出
backbone_lr = config.MODEL.BACKBONE_LR
position_embedding = build_position_encoding(config.MODEL.TRANS.EMBED_DIM)
backbone = Backbone(backbone_name, train_backbone, return_interm_layers, dilation, backbone_lr)
model = Joiner(backbone, position_embedding)
#将backbone 与 position embedding 集成在一个model中
model.num_channels = backbone.num_channels
return model(5) position_embedding
常见的Position Embedding一般有两种，一个基于sine函数的位置编码，一个是可以学习位置编码。
DERT中分别对这两种编码方式，都写成了继承Module的layer，PositionEmbeddingSine，PositionEmbeddingLearned。
class PositionEmbeddingSine(nn.Layer):
# 基于sin函数的位置编码-
def __init__(self, num_position_features=64, temp=10000, norm=False, scale=None):
super().__init__()
self.num_position_features = num_position_features
self.temp = temp
self.norm = norm
if scale is not None and norm is False:
raise ValueError('norm should be true if scale is passed')
self.scale = 2 * math.pi if scale is None else scale
def forward(self, tensor_list):
x = tensor_list.tensors
mask = tensor_list.mask
not_mask = (mask < 0.5).astype('float32')
y_embed = not_mask.cumsum(1, dtype='float32')
x_embed = not_mask.cumsum(2, dtype='float32')
if self.norm:
eps = 1e-6
y_embed = y_embed / (y_embed[:, -1:, :] + eps) * self.scale
x_embed = x_embed / (x_embed[:, :, -1:] + eps) * self.scale
dim_t = paddle.arange(self.num_position_features, dtype='int32')
dim_t = self.temp ** (2 * (dim_t // 2) / self.num_position_features)
pos_y = y_embed.unsqueeze(-1) / dim_t
pos_x = x_embed.unsqueeze(-1) / dim_t
pos_y = paddle.stack((pos_y[:, :, :, 0::2].sin(),
pos_y[:, :, :, 1::2].cos()), axis=4).flatten(3)
pos_x = paddle.stack((pos_x[:, :, :, 0::2].sin(),
pos_x[:, :, :, 1::2].cos()), axis=4).flatten(3)
pos = paddle.concat((pos_y, pos_x), axis=3).transpose([0, 3, 1, 2])
return pos
class PositionEmbeddingLearned(nn.Layer):
#基于学习方式的位置编码
def __init__(self, num_position_features=256):
super().__init__()
w_attr_1 = self.init_weights()
self.row_embed = nn.Embedding(50, num_position_features, weight_attr=w_attr_1)
w_attr_2 = self.init_weights()
self.col_embed = nn.Embedding(50, num_position_features, weight_attr=w_attr_2)
def init_weights(self):
return paddle.ParamAttr(initializer=nn.initializer.Uniform(low=0., high=1.))
def forward(self, tensor_list):
x = tensor_list.tensors #[batch, 2048(R50 feature), H, W]
h, w = x.shape[-2:]
i = paddle.arange(w)
j = paddle.arange(h)
x_embed = self.col_embed(i)
y_embed = self.row_embed(j)
pos = paddle.concat([
x_embed.unsqueeze(0).expand((h, x_embed.shape[0], x_embed.shape[1])),
y_embed.unsqueeze(1).expand((y_embed.shape[0], w, y_embed.shape[1])),
], axis=-1)
pos = pos.transpose([2, 0, 1]) #[dim, h, w]
pos = pos.unsqueeze(0) #[1, dim, h, w]
pos = pos.expand([x.shape[0]] + pose.shape[1::]) #[batch, dim, h, w]
return pos下一节介绍 build transformer， build dert部分。