pointpillars点云算法TensorRT环境加速系列二

简述

　　上一篇博文撰写了关于pointpillars算法的pytorch模型如何转换为onnx模型中间件，具体参考此链接：pointpillars点云算法TensorRT环境加速系列一以此来方便TensorRT进行加载解析优化模型。接下来，我们在完成第一步模型成功从pytorch模型转换成为onnx之后，需要验证onnx模型转换之后的精度与原始的pytorch模型精度差多少。

Attention: 环境与代码与第一篇一致

　或者你可以参考我的github上ReadMe，里面有具体的步骤关于如何测试onnx模型的精度对比，传送门点击此链接。

Compare pfe.onnx with pytorch-model pfe-layer outputs

　　首先，我们对第一个onnx模型pfe.onnx与原始pytorch模型对于pfe层的网络输出结果进行对比，需要将输入对应一致进行检查：

　　以下为onnx模型预测pfe.onnx模型输出的主要代码部分： 简单说下：输入分别为pillar_x、pillar_y、pillar_z、pillar_i、num_points_per_pillar、x_sub_shaped、y_sub_shaped、mask总共8个张量作为输入，同时需要主要每个张量的device设置，我这边通过打印原始的输出device进行手动添加以此来对应上。

def onnx_model_predict(config_path=None, model_dir=None):
    import onnxruntime
    from second.pytorch.models.pointpillars import PillarFeatureNet, PointPillarsScatter

    # check the pfe onnx model IR input paramters as follows 
    pillar_x = torch.ones([1, 1, 12000, 100], dtype=torch.float32, device="cuda:0")
    pillar_y = torch.ones([1, 1, 12000, 100], dtype=torch.float32, device="cuda:0")
    pillar_z = torch.ones([1, 1, 12000, 100], dtype=torch.float32, device="cuda:0")
    pillar_i = torch.ones([1, 1, 12000, 100], dtype=torch.float32, device="cuda:0")
    num_points_per_pillar = torch.ones([1, 12000], dtype=torch.float32, device="cuda:0")
    x_sub_shaped = torch.ones([1, 1, 12000, 100], dtype=torch.float32, device="cuda:0")
    y_sub_shaped = torch.ones([1, 1, 12000, 100], dtype=torch.float32, device="cuda:0")
    mask = torch.ones([1, 1, 12000, 100], dtype=torch.float32, device="cuda:0")
    
    pfe_session = onnxruntime.InferenceSession("pfe.onnx")

    # Compute ONNX Runtime output prediction
    pfe_inputs = {pfe_session.get_inputs()[0].name: (pillar_x.data.cpu().numpy()),
                  pfe_session.get_inputs()[1].name: (pillar_y.data.cpu().numpy()),
                  pfe_session.get_inputs()[2].name: (pillar_z.data.cpu().numpy()),
                  pfe_session.get_inputs()[3].name: (pillar_i.data.cpu().numpy()),
                  pfe_session.get_inputs()[4].name: (num_points_per_pillar.data.cpu().numpy()),
                  pfe_session.get_inputs()[5].name: (x_sub_shaped.data.cpu().numpy()),
                  pfe_session.get_inputs()[6].name: (y_sub_shaped.data.cpu().numpy()),
                  pfe_session.get_inputs()[7].name: (mask.data.cpu().numpy())}

    pfe_outs = pfe_session.run(None, pfe_inputs)
    print('-------------------------- PFE ONNX Outputs ----------------------------')
    print(pfe_outs) # also you could save it to file for comparing
    print('-------------------------- PFE ONNX Ending ----------------------------')

　　看过主要的预测pfe.onnx代码之后，我们来对比一下结果：左边为原始pytorch模型关于pfe-layer的张量输出结果，右边为pfe.onnx模型的输出结果，可以看到输出误差在小数点后三位级别。

Compare rpn.onnx with pytorch-model rpn-layer outputs

　　接下来，我们来主要对比一下rpn.onnx模型的输出与pytorch模型进过rpn网络的输出进行对比。由于rpn网络之前并不是直接接入pfe网络的输出，中间进过pillarscatter网络，所以针对验证输入输出不可以单独抛开pfe网络层。当然，你也可以直接读取pfe层的输出，通过pillarscatter网络后，接入rpn网络来查看输出进行对比。

　　以下为预测rpn.onnx主要部分代码，包括rpn网络输入连接pillarscatter网络的输出，这样也连接上pfe网络的输出，具体代码如下：

def onnx_model_predict(config_path=None, model_dir=None):
    import onnxruntime
    from second.pytorch.models.pointpillars import PillarFeatureNet, PointPillarsScatter
    # check the rpn onnx model IR input paramters as follows 
    pillar_x = torch.ones([1, 1, 9918, 100], dtype=torch.float32, device="cuda:0")
    pillar_y = torch.ones([1, 1, 9918, 100], dtype=torch.float32, device="cuda:0")
    pillar_z = torch.ones([1, 1, 9918, 100], dtype=torch.float32, device="cuda:0")
    pillar_i = torch.ones([1, 1, 9918, 100], dtype=torch.float32, device="cuda:0")
    num_points_per_pillar = torch.ones([1, 9918], dtype=torch.float32, device="cuda:0")
    x_sub_shaped = torch.ones([1, 1, 9918, 100], dtype=torch.float32, device="cuda:0")
    y_sub_shaped = torch.ones([1, 1, 9918, 100], dtype=torch.float32, device="cuda:0")
    mask = torch.ones([1, 1, 9918, 100], dtype=torch.float32, device="cuda:0")

    pfe_session = onnxruntime.InferenceSession("pfe.onnx")
    # Compute ONNX Runtime output prediction
    pfe_inputs = {pfe_session.get_inputs()[0].name: (pillar_x.data.cpu().numpy()),
                  pfe_session.get_inputs()[1].name: (pillar_y.data.cpu().numpy()),
                  pfe_session.get_inputs()[2].name: (pillar_z.data.cpu().numpy()),
                  pfe_session.get_inputs()[3].name: (pillar_i.data.cpu().numpy()),
                  pfe_session.get_inputs()[4].name: (num_points_per_pillar.data.cpu().numpy()),
                  pfe_session.get_inputs()[5].name: (x_sub_shaped.data.cpu().numpy()),
                  pfe_session.get_inputs()[6].name: (y_sub_shaped.data.cpu().numpy()),
                  pfe_session.get_inputs()[7].name: (mask.data.cpu().numpy())}

    pfe_outs = pfe_session.run(None, pfe_inputs)
    # print('-------------------------- PFE ONNX Outputs ----------------------------')
    # print(pfe_outs) # also you could save it to file for comparing
    # print('-------------------------- PFE ONNX Ending ----------------------------')
    ##########################Middle-Features-Extractor#########################
    # numpy --> tensor
    pfe_outs = np.array(pfe_outs)
    voxel_features_tensor = torch.from_numpy(pfe_outs)

    voxel_features = voxel_features_tensor.squeeze()
    # voxel_features = np.array(pfe_outs).squeeze()
    voxel_features = voxel_features.permute(1, 0)

    if isinstance(config_path, str):
        config = pipeline_pb2.TrainEvalPipelineConfig()
        with open(config_path, "r") as f:
            proto_str = f.read()
            text_format.Merge(proto_str, config)
    else:
        config = config_path
    model_cfg = config.model.second
    vfe_num_filters = list(model_cfg.voxel_feature_extractor.num_filters)
    voxel_generator = voxel_builder.build(model_cfg.voxel_generator)
    grid_size = voxel_generator.grid_size
    output_shape = [1] + grid_size[::-1].tolist() + [vfe_num_filters[-1]]
    num_input_features = vfe_num_filters[-1]
    batch_size = 2
    mid_feature_extractor = PointPillarsScatter(output_shape,
                                                num_input_features,
                                                batch_size)

    device = torch.device("cuda:0")
    coors_numpy = np.loadtxt('./onnx_predict_outputs/coors.txt', dtype=np.int32)
    coors = torch.from_numpy(coors_numpy)
    coors = coors.to(device).cuda() # CPU Tensor --> GPU Tensor

    voxel_features = voxel_features.to(device).cuda()
    rpn_input_features = mid_feature_extractor(voxel_features, coors)

    #################################RPN-Feature-Extractor########################################
    # rpn_input_features = torch.ones([1, 64, 496, 432], dtype=torch.float32, device='cuda:0')
    rpn_session = onnxruntime.InferenceSession("rpn.onnx")
    # compute RPN ONNX Runtime output prediction
    rpn_inputs = {rpn_session.get_inputs()[0].name: (rpn_input_features.data.cpu().numpy())}

    rpn_outs = rpn_session.run(None, rpn_inputs)
    print('---------------------- RPN ONNX Outputs ----------------------')
    print(rpn_outs)
    print('---------------------- RPN ONNX Ending ----------------------')

　　当然，rpn-layer层对比rpn.onnx操作对比输出的精度差，如下可以查看，误差系数在小数点3位之后：

小结

　　到此，验证pointpillars算法pytorch模型转换成为pfe.onnx与rpn.onnx精度对比基本完成。具体的代码将会在我的github上面进行更新，目前结果来看模型转换成功，下一篇将会将其使用TensorRT进行加速来对比一下经过TensorRT优化加速之后，效率提升了多少，敬请期待。

参考文献

https://github.com/SmallMunich/nutonomy_pointpillars

https://blog.csdn.net/Small_Munich/article/details/101559424

https://arxiv.org/abs/1812.05784