Faster RCNN 推理 从头写 java (三) RPN to ROIs

目录:

• 1. 图片预处理
• 2. RPN网络预测
• 3. RPN to ROIs
• 4. Classifier 网络预测
• 5. Classifier网络输出对 ROIs过滤与修正
• 6. NMS (非最大值抑制)
• 7. 坐标转换为原始图片维度

一: 输入输出

输入:

• cls: RPN网络的输出, shape为 [1, 37, 50, 49]
• reg: RPN网络的输出, shape为 [1, 37, 50, 196]

输出:

• R: ROIs, shape为 [300, 4]

二: 流程

• 遍历所有的anchor, 7个anchorSize, 7个anchorRatio, 共 49个anchor
• 计算每个anchor在每个feature map pixel上的矩形 (x, y, w, h)
• 使用RPN网络的reg输出来修正每个anchor的矩形
• 限制anchor矩形范围值
• 非最大值抑制，找出300个ROIs

三: code by code

RPN的输出除以std_scaling = 4
原因是: 训练时最终计算出regr后，乘了个std_scaling

regr.divi(std_scaling);

获取行与列的size.

int rows = (int)cls.shape()[1];
int cols = (int)cls.shape()[2];

每个anchor(共49个) 经过RPN 的输出 regr 修正后的ROIs

A.shape = [4, 37, 50, 49]
INDArray A = Nd4j.zeros(4, cls.shape()[1], cls.shape()[2], cls.shape()[3]);

遍历每个anchor
一共49个layer.

int curr_layer = 0;
 
for (int i = 0; i < anchor_sizes.length; i++)
{
    for (int j = 0; j < anchor_ratios.length; j++)
    {
        int anchor_size = anchor_sizes[i];
        int[] anchor_ratio = anchor_ratios[j];

计算出每个anchor在feature map 维度上的宽和高.
anchor_x: anchor 的宽
anchor_y: anchor 的高
rpn_stride: 16 是VGG16模型 输入图片size 到 feature map size 的缩小比例.

int anchor_x = (anchor_size * anchor_ratio[0]) / rpn_stride;
int anchor_y = (anchor_size * anchor_ratio[1]) / rpn_stride;

获取当前anchor在RPN输出regr上的回归值.
current_regr 是当前anchor的回归值, shape = [37, 50, 4], 并将其permute成 [4, 37, 50]

INDArray current_regr = regr.get(NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.interval(4 * curr_layer, 4 * curr_layer + 4));
current_regr = current_regr.permute(2, 0, 1);

构建MeshGrid, 用来生成anchor在每个pixel上的坐标.
X: 是x坐标
0, 1, 2, ....., 47, 48, 49
共37列
Y: 是y坐标
0,
1,
2,
.....
34,
35,
36
共50行.

INDArray[] meshgrid = Nd4j.meshgrid(Nd4j.arange(cols).castTo(DataType.INT), Nd4j.arange(rows).castTo(DataType.INT));
INDArray X = meshgrid[0];
INDArray Y = meshgrid[1];

计算出当前anchor 在feature map 上的矩形 [x, y, w, h]

X.subi(anchor_x / 2);
Y.subi(anchor_y / 2);

// 设置每个anchor在所有像素点上的x, y, w, h

A.put(new INDArrayIndex[]{NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, X);
A.put(new INDArrayIndex[]{NDArrayIndex.point(1), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, Y);
A.put(new INDArrayIndex[]{NDArrayIndex.point(2), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, anchor_x);
A.put(new INDArrayIndex[]{NDArrayIndex.point(3), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, anchor_y);

使用RPN 的输出regr 来修正每个anchor对应的矩形.
该逻辑是与训练时生成RPN网络标注数据执行了相反的操作.

INDArray applyRegr = apply_regr_np(A.get(NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)), current_regr);
A.put(new INDArrayIndex[]{NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, applyRegr);

将坐标从[x, y, w, h] 转换为 [x1, y1, x2, y2]
限制的anchor矩形坐标
x1 >= 0
y1 >= 0
x2 <= cols - 1
y2 <= rows - 1

INDArrayIndex[] x_indices = new INDArrayIndex[]{NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArrayIndex[] y_indices = new INDArrayIndex[]{NDArrayIndex.point(1), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArrayIndex[] w_indices = new INDArrayIndex[]{NDArrayIndex.point(2), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArrayIndex[] h_indices = new INDArrayIndex[]{NDArrayIndex.point(3), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArray x = A.get(x_indices[0], x_indices[1], x_indices[2], x_indices[3]);
INDArray y = A.get(y_indices[0], y_indices[1], y_indices[2], y_indices[3]);
INDArray w = A.get(w_indices[0], w_indices[1], w_indices[2], w_indices[3]);
INDArray h = A.get(h_indices[0], h_indices[1], h_indices[2], h_indices[3]);

限制宽度和高度最小值为1.
格式为: x, y, w, h

A.put(w_indices, Transforms.max(w, 1.0));
A.put(h_indices, Transforms.max(h, 1.0));

转换为 x1, y1, x2, y2

A.put(w_indices, x.add(w));
A.put(h_indices, y.add(h));

将 [x1, y1], [x2, y2] 约束在 [0, cols - 1], [0, rows - 1] 范围内.
w_indice/h_indice 其实表示的是 x2, y1 的 indices.

A.put(x_indices, Transforms.max(x, 0.0));
A.put(y_indices, Transforms.max(y, 0.0));
A.put(w_indices, Transforms.min(w, cols - 1));
A.put(h_indices, Transforms.min(h, rows - 1));
INDArray applyRegr = apply_regr_np(A.get(NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)), current_regr);
A.put(new INDArrayIndex[]{NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, applyRegr);

将 坐标的shape (4, 37, 50, 49) 转换为 (90650, 4)
将 概率的shape (1, 37, 50, 49) 转换为 (90650,)

INDArray all_boxes = A.permute(0, 3, 1, 2).reshape(4, -1).permute(1, 0);
INDArray all_probs = cls.permute(0, 3, 1, 2).reshape(-1);

删除坐标中 （x1 > x2 || y1 > y2）的无效坐标.

INDArray x1 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(0));
INDArray y1 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(1));
INDArray x2 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(2));
INDArray y2 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(3));
 
INDArrayIndex validIdx = validIndex_restrain_x1x2y1y2(x1, x2, y1, y2);
 
INDArray valid_box = all_boxes.get(validIdx, NDArrayIndex.all());
INDArray valid_probs = all_probs.get(validIdx);

执行NMS(非最大值抑制)
overlap_thresh 为 0.7
maxBoxes 为 300

return non_max_suppression_fast(valid_box, valid_probs, 0.7f, 300);