faster rcnn pytorch 复现系列（三）： Anchor_target_layer.py
注：此篇文章的基础知识可见链接：
pytoch faster rcnn复现系列（一） RPN层输入输出维度理解
faster rcnn pytorch 复现系列（二）：generate_anchors源码解析
RPN 预备编程知识（一） : Numpy VS Pytorch （ Anchor_target_layer ）
RPN 预备编程知识（二） : 函数篇章 bbox_transform.py
faster rcnn中 softmax，softmax loss和cross entropy的讲解
首先附上，参考的两篇caffe中faster rcnn的 Anchor_target_layer.py源码分析链接：
Faster rcnn代码理解（3）
r-cnn学习（六）：RPN及AnchorTargetLayer学习
然后写下pytorch中、Anchor_target_layer.py的解析，pytorch允许batchsize>1：
目录
0. 变量名说明
1. 导入相关包
2. __init__初始化
3. 通过ravel、meshgrid合用，得到平移坐标，构造通过原始anchor+平移，得到所有anchors。
4. 去除过界的anchor留下，inds_inside+anchors
5. 通过overlaps制作labels，找到anchor对应的gt最大索引，和gt对应的anchor最大索引
6. 降采样（正样本或者负样本太多）
7. bbox_targets、bbox_inside_weights、bbox_inside_weights
8. 所有anchors赋上label、bbox_targets、bbox_inside_weights、bbox_outside_weights属性
9. _unmap见该函数：
10. 不需要反向传播
11. 手绘流程图
12. caffe 版本
补充. cnn 大局观
y^=wx+b，x是输入，w是特征权值向量。
w：往往我们说的3*3*256的卷积就是w了，
y^：通过卷积生成的（1,36，H，W）的就是y^，
y* ：gt对应的
L(y^,y*): y^和y*，做损失函数，通过优化器，迭代更新3*3*256的w卷积，学习到复杂的特征（就是w），来使得loss减少，预测的接近于真实的。
比如faster rcnn 中smooth l1 loss 就使得rpn_box_pred(预测的偏移)接近于rpn_box_target(真实的偏移),降低loss的方法，就是不断学习调整各个w，以学习复杂特征。（在这个过程中利用gt在anchor_target_layer中，先删除边界框外的anchor，然后获得边界框内的rpn_box_inside，然后重新弄一个新的整个anchor的，统一赋值0，然后讲框内的rpn_box_inside 再利用之前的赋值，来确定哪一个是整个anchor中，正样本对应的anchor，只对他进行smooth l1 loss，也就是回归正anchor），
在这个过程中rpn_box_pred是直接在共享特征层进行3*3*256的卷积后，进行1*1*36的全卷积w（这个就是要学的w）生成y^（1,36，H，W），来存储预测的anchor。
而rpn_box_target则是通过寻找原始anchor最近的gt的方法，找到每个anchor对应的gt的偏移，用数学表达为：y*（1,36，H，W）的向量。其中（1,36，H，W）可以理解为(1,4,9*H,W)也就是每个anchor的4个偏移。
首先：让我学习到：
看源码
- 首先要弄清楚论文中的原理，
- 搞清楚这个py的输入和输出
- 分清楚流程，需要哪些步骤，求取中间值，然后构造中间函数【输入+输出】和中间变量。
0. 变量名说明
学习总结别人的命名。
- shift_x和shift_y：分别对应x和y轴上的偏移量。
- _num_anchors： anchor的3*3 9种数量，【最后的名词类似于num短小这种放在最前面】
- gt_boxes：gt的boxes【保证不头重脚轻，这种时候长名词放在后面】
- shifts： 偏移量，只要原始anchor+shifts，就可以实现逐行算anchor，得到所有anchor
- inds_inside：inside anchors的index
- overlaps ： （B，M，N）表示B个batchsize，M表示内部anchor总数，N表示gt的个数。IOU
- max_overlaps：每个内部anchor，找iou最大的gt （B，M）
- argmax_overlaps：每个内部anchor，找iou最大的gt 的索引
- gt_max_overlaps：每个内部gt，找iou最大的anchor （B，N）两种赋正样本的规则
- offset：因为batchsize导致的计算bbox_targets需要输入(B*K,5)的gt_boxes，索引经过了reshape，索引会发生便宜。
- bbox_targets：（1，4*9，H，W）把他看成（1，4, 9*h，w）每个anchor的偏移，边框回归的输入 M表示内部anchor的数量
- bbox_inside_weights：label是1，它就是1。其他为0，只就算正样本的损失。#(B,M) 与caffe版本的不同， # use a single value instead of 4 values for easy index，pi*
- bbox_outside_weights：#(B,M) 样本权重归一化后正负样本都是1/256.其他背景为0
1. 导入相关包
from .generate_anchors .表示同一文件路径下。
import torch
import torch.nn as nn
import numpy as np
import numpy.random as npr
from model.utils.config import cfg
from .generate_anchors import generate_anchors
from .bbox_transform import clip_boxes, bbox_overlaps_batch, bbox_transform_batch
import pdb
DEBUG = False
try:
long # Python 2
except NameError:
long = int # Python 32. __init__初始化
传入特征图与原图的倍数差，scales，ratios，初始化这三个参数，并generate_anchors生成tensor形式的初始anchor
class _AnchorTargetLayer(nn.Module):
"""
给所有的anchors赋对应的gt目标，制造anchor二分类的labels和bbox的回归用的targets
targets 包括:dx dy dw dh
""" # 16 8 16 32 0.5 1 2
def __init__(self, feat_stride, scales, ratios):
super(_AnchorTargetLayer, self).__init__()
self._feat_stride = feat_stride
self._scales = scales
anchor_scales = scales
#!这里从np转为了float_tensor，方便运用torch中函数
#传入的scales和ratios是元组形式，需要np.array转换
self._anchors =torch.from_numpy(generate_anchors(scales=np.array(anchor_scales), ratios=np.array(ratios))).float() #float_tensor
self._num_anchors = self._anchors.size(0)
# 是否允许包括少部分边界外的anchor
self._allowed_border = 0 # default is 03. 通过ravel、meshgrid合用，得到平移坐标，构造通过原始anchor+平移，得到所有anchors。
shift_x和shift_y分别对应x和y轴上的偏移量，用在之前说过的用generate_anchors（）函数生成的最左上角的anchors上，对其进行偏移，从而获得所有图像上的anchors；all_anchors用来存储所有这些anchors，total_anchors用来存储这些anchors的数量K×A,其中，K是输入图像的num,A是一幅图像上anchor的num；之后作者还对这些anchors进行了筛选，超出图像边界的anchors都将其丢弃～继续：
def forward(self, input):
"""
对于公共特征图(H, W)，在每一个位置i，生成9个anchors
超出边界的去掉
"""
#input_>torch.Tensor
rpn_cls_score = input[0]#map存有特征图的W H
gt_boxes = input[1]#boxes labels[1,5]
im_info = input[2]#(im_info[0][1])原图w(im_info[0][0])原图h
num_boxes = input[3]
# map of shape (..., H, W) torch里的size和shape一样
height, width = rpn_cls_score.size(2), rpn_cls_score.size(3)
#输如多少个gt
batch_size = gt_boxes.size(0)
feat_height, feat_width = rpn_cls_score.size(2), rpn_cls_score.size(3)
#见RPN编程预备知识
shift_x = np.arange(0, feat_width) * self._feat_stride
shift_y = np.arange(0, feat_height) * self._feat_stride
shift_x, shift_y = np.meshgrid(shift_x, shift_y)
shifts = torch.from_numpy(np.vstack((shift_x.ravel(), shift_y.ravel(),
shift_x.ravel(), shift_y.ravel())).transpose())
# tensor.contiguous() → Tensor
#返回一个内存连续的有相同数据的tensor，如果原tensor内存连续则返回原tensor
#shifts.view need 内存连续，可以改成reshape函数，就不需要这一步了。
#type_as(tesnor)将tensor投射为参数给定tensor类型并返回。 如果tensor已经是正确的类型则不会执行操作。等效于：self.type(tensor.type())
#shifts int_>float
shifts = shifts.contiguous().type_as(rpn_cls_score).float()
# add A anchors (1, A, 4) to
# cell K shifts (K, 1, 4) to get
# shift anchors (K, A, 4)
# reshape to (K*A, 4) shifted anchors
A = self._num_anchors
K = shifts.size(0)
self._anchors = self._anchors.type_as(gt_boxes) #int_tensor_>float_tensor
#board
all_anchors = self._anchors.view(1, A, 4) + shifts.view(K, 1, 4)
all_anchors = all_anchors.view(K * A, 4)
total_anchors = int(K * A)4. 去除过界的anchor留下，inds_inside+anchors，没有anchors[np.where()[0],:]方便
total_anchors = int(K * A)
#没有过界的anchors索引 xmin>=0 ymin>=0 xmax= -self._allowed_border) &
(all_anchors[:, 1] >= -self._allowed_border) &
(all_anchors[:, 2] < long(im_info[0][1]) + self._allowed_border) &
(all_anchors[:, 3] < long(im_info[0][0]) + self._allowed_border))
#inds_inside ：没有过界的anchors索引
inds_inside = torch.nonzero(keep).view(-1)
#anchors：没有过界的anchors
# keep only inside anchors
anchors = all_anchors[inds_inside, :] 5. 通过overlaps制作labels，找到anchor对应的gt最大索引，和gt对应的anchor最大索引
这一部分主要就是获得这些anchors和对应gt的最大重叠率的情况，以及正样本的划分标准：
a.对于每一个gt，重叠率最大的那个anchor为fg；
b,对于每一个anchor，与gts中最大重叠率大于0.7的为fg；
cfg.TRAIN.RPN_CLOBBER_POSITIVE则涉及到一种情况，即如果最大重叠率小于cfg.TRAIN.RPN_NEGATIVE_OVERLAP=0.3,则到底正还是负，这里的cfg.TRAIN.RPN_CLOBBER_POSITIVE默认是False；
# label: 1 is positive, 0 is negative, -1 is dont care
labels = gt_boxes.new(batch_size, inds_inside.size(0)).fill_(-1) #labels (B*M)
bbox_inside_weights = gt_boxes.new(batch_size, inds_inside.size(0)).zero_()
bbox_outside_weights = gt_boxes.new(batch_size, inds_inside.size(0)).zero_()
#返回（B，M，N）表示B个batchsize，M表示内部anchor总数，N表示gt的个数。
overlaps = bbox_overlaps_batch(anchors, gt_boxes)
#axis=2，意味着从（B，M，N）的 N 个gts中找出最大的iou 的 gt
#返回（B，M）
max_overlaps, argmax_overlaps = torch.max(overlaps, 2)
#axis=1，意味着从（B，M，N）的 M 个anchor中找出最大iou 的 anchor
#返回（B，N）
gt_max_overlaps, _ = torch.max(overlaps, 1)
if not cfg.TRAIN.RPN_CLOBBER_POSITIVES:
labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 0
#如果gt与所有anchors相交最大的那一个iou = 0,
gt_max_overlaps[gt_max_overlaps==0] = 1e-5
#gt_max_overlaps.view(batch_size,1,-1)的维度：（B，1，N）
#overlaps 的维度：（B，M，N）
#在这里我认为没有必要expand_as，因为执行运算会自动广播啊？
#eq 相等返回1，不相等返回0,不像np中的==返回true，false
#torch.sum（...，2）返回（B，M）
keep = torch.sum(overlaps.eq(gt_max_overlaps.view(batch_size,1,-1).expand_as(overlaps)), 2)#keep（B，M）
if torch.sum(keep) > 0:
#找出与gt相交最大且iou不为0的那个anchor，作为正样本
labels[keep>0] = 1
# fg label: above threshold IOU
#如果一个anchor与gts相交的最大值>0.7，赋予正样本
labels[max_overlaps >= cfg.TRAIN.RPN_POSITIVE_OVERLAP] = 1
#默认FALse
if cfg.TRAIN.RPN_CLOBBER_POSITIVES:
##如果最大的anchor的iou都小于0.3，那么这个anchor是背景
labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 06. 降采样（正样本或者负样本太多）
这一部分是说，如果我们得到的正样本或者负样本太多的话，那么就选取一定数量的，丢弃一定数量的anchors，应该是为了加速（这里的选取方法也很直接，就是随机选取），继续：
#前景要求的数量
num_fg = int(cfg.TRAIN.RPN_FG_FRACTION * cfg.TRAIN.RPN_BATCHSIZE)
sum_fg = torch.sum((labels == 1).int(), 1)#（B,）
sum_bg = torch.sum((labels == 0).int(), 1)#（B,）
#如果多了，随机删除 感觉不严谨
for i in range(batch_size):
# subsample positive labels if we have too many
if sum_fg[i] > num_fg:
fg_inds = torch.nonzero(labels[i] == 1).view(-1)
# torch.randperm seems has a bug on multi-gpu setting that cause the segfault.
# See https://github.com/pytorch/pytorch/issues/1868 for more details.
# use numpy instead.
#rand_num = torch.randperm(fg_inds.size(0)).type_as(gt_boxes).long()
rand_num = torch.from_numpy(np.random.permutation(fg_inds.size(0))).type_as(gt_boxes).long()
disable_inds = fg_inds[rand_num[:fg_inds.size(0)-num_fg]]
labels[i][disable_inds] = -1
# num_bg = cfg.TRAIN.RPN_BATCHSIZE - sum_fg[i]
num_bg = cfg.TRAIN.RPN_BATCHSIZE - torch.sum((labels == 1).int(), 1)[i]
#RPN_BATCHSIZE=256 RPN_FG_FRACTION=0.5
#正样本如果最开始就是不够128个（256*0.5），就不够，负样本会大于128
# subsample negative labels if we have too many
if sum_bg[i] > num_bg:
bg_inds = torch.nonzero(labels[i] == 0).view(-1)
#rand_num = torch.randperm(bg_inds.size(0)).type_as(gt_boxes).long()
rand_num = torch.from_numpy(np.random.permutation(bg_inds.size(0))).type_as(gt_boxes).long()
disable_inds = bg_inds[rand_num[:bg_inds.size(0)-num_bg]]
labels[i][disable_inds] = -17. bbox_targets、bbox_inside_weights、bbox_inside_weights
bbox_targets是一个anchor与他最近的gt【ancho对应的gt最大索引】之前的回归四要素。tx,ty,tw,th
这一部分是生成bbox_targets、bbox_inside_weights、bbox_inside_weights；其中对于bbox_targets，它这里是调用了_compute_targets()函数，见bbox_transform_batch：.
而对于后两个bbox_inside_weights和bbox_outside_weights，函数中定义的是bbox_inside_weights初始化为n×4的0数组，然后其中正样本的坐标的权值均为1；而bbox_outside_weights同样的初始化，其中正样本和负样本都被赋值1/num(anchors的数量)
def _compute_targets_batch(ex_rois, gt_rois):
"""Compute bounding-box regression targets for an image."""
return bbox_transform_batch(ex_rois, gt_rois[:, :, :4])
# gt_boxes (B,K,5)
offset = torch.arange(0, batch_size)*gt_boxes.size(1)#offset(batch_size,)
# argmax_overlaps（B，M） + （B，1） ！！索引需要
argmax_overlaps = argmax_overlaps + offset.view(batch_size, 1).type_as(argmax_overlaps)
# gt_boxes.view(-1,5) (B*K,5)所以需要offset
# bbox_targets (batch_size, -1, 5)
bbox_targets = _compute_targets_batch(anchors, gt_boxes.view(-1,5)[argmax_overlaps.view(-1), :].view(batch_size, -1, 5))
# use a single value instead of 4 values for easy index.
#求出的偏移，是要看这个anchor离哪一个gt最近【那如果都不近呢，返回的应该是第一个gt的偏移】
bbox_inside_weights[labels==1] = cfg.TRAIN.RPN_BBOX_INSIDE_WEIGHTS[0]
#默认RPN_POSITIVE_WEIGHT=-1
if cfg.TRAIN.RPN_POSITIVE_WEIGHT < 0:
#num_examples = torch.sum(labels[i] >= 0)
# #样本权重归一化
num_examples = torch.sum(labels[i] >= 0).item()#正负的样本总数目
positive_weights = 1.0 / num_examples#正样本权重 1/总样本，感觉可以直接256上啊
negative_weights = 1.0 / num_examples
else:
assert ((cfg.TRAIN.RPN_POSITIVE_WEIGHT > 0) &
(cfg.TRAIN.RPN_POSITIVE_WEIGHT < 1))
bbox_outside_weights[labels == 1] = positive_weights
bbox_outside_weights[labels == 0] = negative_weights8. 所有anchors赋上label、bbox_targets、bbox_inside_weights、bbox_outside_weights属性
至于为什么先view然后permute，可以参考reshape和transpose的区别
#利用inds_inside索引，建立一个所有anchors数组，按照labels中正负样本打1 0 标签，其余的fill-1
labels = _unmap(labels, total_anchors, inds_inside, batch_size, fill=-1)
bbox_targets = _unmap(bbox_targets, total_anchors, inds_inside, batch_size, fill=0)
#(B,H*W)
bbox_inside_weights = _unmap(bbox_inside_weights, total_anchors, inds_inside, batch_size, fill=0)
bbox_outside_weights = _unmap(bbox_outside_weights, total_anchors, inds_inside, batch_size, fill=0)
outputs = []
#B,9*W*H
labels = labels.view(batch_size, height, width, A).permute(0,3,1,2).contiguous()#B,9,h,w
labels = labels.view(batch_size, 1, A * height, width)#B 1 9*H W
outputs.append(labels)
bbox_targets = bbox_targets.view(batch_size, height, width, A*4).permute(0,3,1,2).contiguous()#B,9*4,H,W
outputs.append(bbox_targets)
anchors_count = bbox_inside_weights.size(1)
bbox_inside_weights = bbox_inside_weights.view(batch_size,anchors_count,1).expand(batch_size, anchors_count, 4)#
bbox_inside_weights = bbox_inside_weights.contiguous().view(batch_size, height, width, 4*A)\
.permute(0,3,1,2).contiguous()##B,W,9*4,H
outputs.append(bbox_inside_weights)
bbox_outside_weights = bbox_outside_weights.view(batch_size,anchors_count,1).expand(batch_size, anchors_count, 4)
bbox_outside_weights = bbox_outside_weights.contiguous().view(batch_size, height, width, 4*A)\
.permute(0,3,1,2).contiguous()
outputs.append(bbox_outside_weights)
return outputs9. _unmap见该函数：
def _unmap(data, count, inds, batch_size, fill=0):
""" Unmap a subset of item (data) back to the original set of items (of
size count) """
if data.dim() == 2:
ret = torch.Tensor(batch_size, count).fill_(fill).type_as(data)
ret[:, inds] = data
else:
ret = torch.Tensor(batch_size, count, data.size(2)).fill_(fill).type_as(data)
ret[:, inds,:] = data
return ret
之后会把这些属性信息经过reshape封装进该网络层即top[0]、top[1]、top[2]、top[3]中；之后：
10. 不需要反向传播
def backward(self, top, propagate_down, bottom):
"""This layer does not propagate gradients."""
pass
def reshape(self, bottom, top):
"""Reshaping happens during the call to forward."""
pass由于该层不需要反向传播，所以backward函数也不需要写了，在前向传播中已经reshape了，就不用再写reshape函数了～
11. 手绘流程图
12. caffe版本的源码解析
# --------------------------------------------------------
# Faster R-CNN
# Copyright (c) 2015 Microsoft
# Licensed under The MIT License [see LICENSE for details]
# Written by Ross Girshick and Sean Bell
# --------------------------------------------------------
import os
import caffe
import yaml
from fast_rcnn.config import cfg
import numpy as np
import numpy.random as npr
from generate_anchors import generate_anchors
from utils.cython_bbox import bbox_overlaps
from fast_rcnn.bbox_transform import bbox_transform
DEBUG = False
class AnchorTargetLayer(caffe.Layer):
"""
Assign anchors to ground-truth targets. Produces anchor classification
labels and bounding-box regression targets.
"""
def setup(self, bottom, top):
layer_params = yaml.load(self.param_str_)
anchor_scales = layer_params.get('scales', (8, 16, 32))
self._anchors = generate_anchors(scales=np.array(anchor_scales))
self._num_anchors = self._anchors.shape[0]
self._feat_stride = layer_params['feat_stride']
if DEBUG:
print 'anchors:'
print self._anchors
print 'anchor shapes:'
print np.hstack((
self._anchors[:, 2::4] - self._anchors[:, 0::4],
self._anchors[:, 3::4] - self._anchors[:, 1::4],
))
self._counts = cfg.EPS
self._sums = np.zeros((1, 4))
self._squared_sums = np.zeros((1, 4))
self._fg_sum = 0
self._bg_sum = 0
self._count = 0
# allow boxes to sit over the edge by a small amount
self._allowed_border = layer_params.get('allowed_border', 0)
height, width = bottom[0].data.shape[-2:]
if DEBUG:
print 'AnchorTargetLayer: height', height, 'width', width
A = self._num_anchors
# labels
top[0].reshape(1, 1, A * height, width)
# bbox_targets
top[1].reshape(1, A * 4, height, width)
# bbox_inside_weights
top[2].reshape(1, A * 4, height, width)
# bbox_outside_weights
top[3].reshape(1, A * 4, height, width)
def forward(self, bottom, top):
# Algorithm:
#
# for each (H, W) location i
# generate 9 anchor boxes centered on cell i
# apply predicted bbox deltas at cell i to each of the 9 anchors
# filter out-of-image anchors
# measure GT overlap
assert bottom[0].data.shape[0] == 1, \
'Only single item batches are supported'
# map of shape (..., H, W)
height, width = bottom[0].data.shape[-2:]
# GT boxes (x1, y1, x2, y2, label)
gt_boxes = bottom[1].data
# im_info
im_info = bottom[2].data[0, :]
if DEBUG:
print ''
print 'im_size: ({}, {})'.format(im_info[0], im_info[1])
print 'scale: {}'.format(im_info[2])
print 'height, width: ({}, {})'.format(height, width)
print 'rpn: gt_boxes.shape', gt_boxes.shape
print 'rpn: gt_boxes', gt_boxes
# 1. Generate proposals from bbox deltas and shifted anchors
shift_x = np.arange(0, width) * self._feat_stride
shift_y = np.arange(0, height) * self._feat_stride
shift_x, shift_y = np.meshgrid(shift_x, shift_y)
shifts = np.vstack((shift_x.ravel(), shift_y.ravel(),
shift_x.ravel(), shift_y.ravel())).transpose()
# add A anchors (1, A, 4) to
# cell K shifts (K, 1, 4) to get
# shift anchors (K, A, 4)
# reshape to (K*A, 4) shifted anchors
A = self._num_anchors
K = shifts.shape[0]
all_anchors = (self._anchors.reshape((1, A, 4)) +
shifts.reshape((1, K, 4)).transpose((1, 0, 2)))
all_anchors = all_anchors.reshape((K * A, 4))
total_anchors = int(K * A)
# only keep anchors inside the image
#注意np.where返回的是行+列，【0】表示行号
inds_inside = np.where(
(all_anchors[:, 0] >= -self._allowed_border) &
(all_anchors[:, 1] >= -self._allowed_border) &
(all_anchors[:, 2] < im_info[1] + self._allowed_border) & # width
(all_anchors[:, 3] < im_info[0] + self._allowed_border) # height
)[0]
if DEBUG:
print 'total_anchors', total_anchors
print 'inds_inside', len(inds_inside)
# keep only inside anchors
#返回内部anchor
anchors = all_anchors[inds_inside, :]
if DEBUG:
print 'anchors.shape', anchors.shape
# label: 1 is positive, 0 is negative, -1 is dont care
#创建一个(n,)的接近于0的数组
labels = np.empty((len(inds_inside), ), dtype=np.float32)
#fill全部变成-1
labels.fill(-1)
# overlaps between the anchors and the gt boxes
# overlaps (ex, gt) k表示gt的数量 N=len(inds_inside)，内部ANCHOR的数量
# overlaps: (N, K) ndarray of overlap between boxes and query_boxes
#overlaps(m,n):m行，列，第m个anchor与第n个box的iou
overlaps = bbox_overlaps(
np.ascontiguousarray(anchors, dtype=np.float),
np.ascontiguousarray(gt_boxes, dtype=np.float))
#找到某个anchor与所有gt_box最大的overlaps【np.array】，按行找，返回列索引
#(axis=1)返回（原行，） #(axis=0 返回（原列，）
argmax_overlaps = overlaps.argmax(axis=1)#N行1列 argmax返回【np.array】
#最大的overlaps值 overlaps【行索引，列索引】找出每一行对应的最大值！！！
max_overlaps = overlaps[np.arange(len(inds_inside)), argmax_overlaps]#（N，）
#按列取最大index，找出K个gt_box与哪一行的anchor的iou最大，返回的是行索引
gt_argmax_overlaps = overlaps.argmax(axis=0)#找到的是每一个的第一个最大值gt_argmax_overlaps（k，）
#写上行索引，然后np.arange（len（gt_box）），对应的iou的大小
gt_max_overlaps = overlaps[gt_argmax_overlaps,#(K， )
np.arange(overlaps.shape[1])]#(K， )
#overlaps (N, K) gt_max_overlaps(K， )，数组广播！！！！！！1行k列填充成N行k列
#找到所有的最大值，可能有相同大小的！！！
#overlaps == gt_max_overlaps是 (N, K)的true false
gt_argmax_overlaps = np.where(overlaps == gt_max_overlaps)[0]#【0】行【1】列
#RPN_CLOBBER_POSITIVES默认是FALSE
if not cfg.TRAIN.RPN_CLOBBER_POSITIVES:
# assign bg labels first so that positive labels can clobber them
#如果最大的anchor的iou都小于0.3，那么这个anchor是背景
labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 0#max_overlaps （N，）广播
#与每个gt相交最大的，可能不止一个，标签设置为1
# fg label: for each gt, anchor with highest overlap
labels[gt_argmax_overlaps] = 1
#iou>0.7设置为前景
# fg label: above threshold IOU
labels[max_overlaps >= cfg.TRAIN.RPN_POSITIVE_OVERLAP] = 1
if cfg.TRAIN.RPN_CLOBBER_POSITIVES:#默认是false，如果是true的话，如果同时满足正负样本条件，弄成负样本。
# assign bg labels last so that negative labels can clobber positives
labels[max_overlaps < cfg.TRAIN.RPN_NEGATIVE_OVERLAP] = 0
# subsample positive labels if we have too many
num_fg = int(cfg.TRAIN.RPN_FG_FRACTION * cfg.TRAIN.RPN_BATCHSIZE)
fg_inds = np.where(labels == 1)[0]#找到所有正样本的行号
#如果多了，随机删除 感觉不严谨
if len(fg_inds) > num_fg:
disable_inds = npr.choice(
fg_inds, size=(len(fg_inds) - num_fg), replace=False)
labels[disable_inds] = -1
# subsample negative labels if we have too many，
#RPN_BATCHSIZE=256 RPN_FG_FRACTION=0.5
#正样本如果最开始就是不够128个（256*0.5），就不够，负样本会大于128
num_bg = cfg.TRAIN.RPN_BATCHSIZE - np.sum(labels == 1)#正样本比较少的情况
bg_inds = np.where(labels == 0)[0]
if len(bg_inds) > num_bg:
disable_inds = npr.choice(
bg_inds, size=(len(bg_inds) - num_bg), replace=False)
labels[disable_inds] = -1#随机去除背景，当做不关系的对象
#print "was %s inds, disabling %s, now %s inds" % (
#len(bg_inds), len(disable_inds), np.sum(labels == 0))
bbox_targets = np.zeros((len(inds_inside), 4), dtype=np.float32)
#求出的偏移，是要看这个anchor离哪一个gt最近【那如果都不近呢，返回的应该是第一个gt的偏移】
bbox_targets = _compute_targets(anchors, gt_boxes[argmax_overlaps, :])#argmax_overlaps所有anchor对应的最大iou的gtbox的索引
bbox_inside_weights = np.zeros((len(inds_inside), 4), dtype=np.float32)
# ！！特别注意下类似的切片 == 的赋值方式的学习#前景的权重为1.0
bbox_inside_weights[labels == 1, :] = np.array(cfg.TRAIN.RPN_BBOX_INSIDE_WEIGHTS)#1.0 1.0 1.0 1.0
bbox_outside_weights = np.zeros((len(inds_inside), 4), dtype=np.float32)
#默认RPN_POSITIVE_WEIGHT=-1
if cfg.TRAIN.RPN_POSITIVE_WEIGHT < 0:
#样本权重归一化
# uniform weighting of examples (given non-uniform sampling)
num_examples = np.sum(labels >= 0)#正负的样本总数目
positive_weights = np.ones((1, 4)) * 1.0 / num_examples#正样本权重 1/总样本，感觉可以直接256上啊
negative_weights = np.ones((1, 4)) * 1.0 / num_examples#负样本权重 1/总样本
#反而觉得RPN_POSITIVE_WEIGHT=0.5更合适些。。
else:
assert ((cfg.TRAIN.RPN_POSITIVE_WEIGHT > 0) &
(cfg.TRAIN.RPN_POSITIVE_WEIGHT < 1))
positive_weights = (cfg.TRAIN.RPN_POSITIVE_WEIGHT /
np.sum(labels == 1))#自己设置正负样本的权重，
negative_weights = ((1.0 - cfg.TRAIN.RPN_POSITIVE_WEIGHT) /
np.sum(labels == 0))
bbox_outside_weights[labels == 1, :] = positive_weights
bbox_outside_weights[labels == 0, :] = negative_weights
if DEBUG:
self._sums += bbox_targets[labels == 1, :].sum(axis=0)
self._squared_sums += (bbox_targets[labels == 1, :] ** 2).sum(axis=0)
self._counts += np.sum(labels == 1)
means = self._sums / self._counts
stds = np.sqrt(self._squared_sums / self._counts - means ** 2)
print 'means:'
print means
print 'stdevs:'
print stds
# map up to original set of anchors
#total_anchors所有anchor的数量
#利用inds_inside索引，建立一个所有anchors数组，按照labels中正负样本打1 0 标签，其余的fill-1
labels = _unmap(labels, total_anchors, inds_inside, fill=-1)
bbox_targets = _unmap(bbox_targets, total_anchors, inds_inside, fill=0)
bbox_inside_weights = _unmap(bbox_inside_weights, total_anchors, inds_inside, fill=0)
bbox_outside_weights = _unmap(bbox_outside_weights, total_anchors, inds_inside, fill=0)
if DEBUG:
print 'rpn: max max_overlap', np.max(max_overlaps)
print 'rpn: num_positive', np.sum(labels == 1)
print 'rpn: num_negative', np.sum(labels == 0)
self._fg_sum += np.sum(labels == 1)
self._bg_sum += np.sum(labels == 0)
self._count += 1
print 'rpn: num_positive avg', self._fg_sum / self._count
print 'rpn: num_negative avg', self._bg_sum / self._count
#变形赋值top[0]
# labels
labels = labels.reshape((1, height, width, A)).transpose(0, 3, 1, 2)
labels = labels.reshape((1, 1, A * height, width))
top[0].reshape(*labels.shape)
top[0].data[...] = labels
# bbox_targets
bbox_targets = bbox_targets \
.reshape((1, height, width, A * 4)).transpose(0, 3, 1, 2)
top[1].reshape(*bbox_targets.shape)
top[1].data[...] = bbox_targets
# bbox_inside_weights
bbox_inside_weights = bbox_inside_weights \
.reshape((1, height, width, A * 4)).transpose(0, 3, 1, 2)
assert bbox_inside_weights.shape[2] == height
assert bbox_inside_weights.shape[3] == width
top[2].reshape(*bbox_inside_weights.shape)
top[2].data[...] = bbox_inside_weights
# bbox_outside_weights
bbox_outside_weights = bbox_outside_weights \
.reshape((1, height, width, A * 4)).transpose(0, 3, 1, 2)
assert bbox_outside_weights.shape[2] == height
assert bbox_outside_weights.shape[3] == width
top[3].reshape(*bbox_outside_weights.shape)
top[3].data[...] = bbox_outside_weights
def backward(self, top, propagate_down, bottom):
"""This layer does not propagate gradients."""
pass
def reshape(self, bottom, top):
"""Reshaping happens during the call to forward."""
pass
def _unmap(data, count, inds, fill=0):
""" Unmap a subset of item (data) back to the original set of items (of
size count) """
if len(data.shape) == 1:
ret = np.empty((count, ), dtype=np.float32)
ret.fill(fill)
ret[inds] = data
else:
ret = np.empty((count, ) + data.shape[1:], dtype=np.float32)
ret.fill(fill)
ret[inds, :] = data
return ret
def _compute_targets(ex_rois, gt_rois):
"""Compute bounding-box regression targets for an image."""
assert ex_rois.shape[0] == gt_rois.shape[0]#都是N行
assert ex_rois.shape[1] == 4
assert gt_rois.shape[1] == 5# # GT boxes (x1, y1, x2, y2, label)
return bbox_transform(ex_rois, gt_rois[:, :4]).astype(np.float32, copy=False)