TensorRT 从7.2升级到8.5，改写plugin以适配新版本

前言

TensorRT是NVIDIA推出的一款高效深度学习模型推理框架，其包括了深度学习推理优化器和运行时，能够让深度学习推理应用拥有低时延和高吞吐的优点。

TensorRT的版本迭代速度非常快，很多之前写的plugin在版本升级后可能就没法直接使用，这时就需要将代码升级，以适配新版TRT。

本文仅记录TRT从7.2升级到8.5时，改写plugin过程中需要注意的内容。

阅读官方的Change Log

• TensorRT now declares API’s with the noexcept keyword to clarify that exceptions must not cross the library boundary. All TensorRT classes that an application inherits from (such as IGpuAllocator, IPluginV2, etc…) must guarantee that methods called by TensorRT do not throw uncaught exceptions, or the behavior is undefined.
• All API’s have been marked as noexcept where appropriate. The IErrorRecorder interface has been fully integrated into the API for error reporting. The Logger is only used as a fallback when the ErrorRecorder is not provided by the user.
• Callback changes are now marked noexcept, therefore, implementations must also be marked noexcept. TensorRT has never catered to exceptions thrown by callbacks, but this is now captured in the API.
• Methods that take parameters of type void** where the array of pointers is unmodifiable are now changed to take type void*const*.
• IPlugin and IPluginFactory interfaces were deprecated in TensorRT 6.0 and have been removed in TensorRT 8.0. We recommend that you write new plugins or refactor existing ones to target the IPluginV2DynamicExt and IPluginV2IOExt interfaces. For more information, refer to the Migrating Plugins From TensorRT 6.x Or 7.x To TensorRT 8.x.x section.
• We removed samplePlugin since it was meant to demonstrate the IPluginExt interface, which is no longer supported in TensorRT 8.0.
• Interface functions that provided a destroy function are deprecated in TensorRT 8.0. The destructors will be exposed publicly in order for the delete operator to work as expected on these classes.
• nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_PRECISION is deprecated.
• nvinfer1::IResizeLayer::setAlignCorners and nvinfer1::IResizeLayer::getAlignCorners are deprecated. Use nvinfer1::IResizeLayer::setCoordinateTransformation, nvinfer1::IResizeLayer::setSelectorForSinglePixel and nvinfer1::IResizeLayer::setNearestRounding instead.
• Destructors for classes with destroy() methods were previously protected. They are now public, enabling use of smart pointers for these classes. The destroy() methods are deprecated.

改动总结

对升级代码过程中的变化做个总结，主要改动如下：

__author__ = 'TracelessLe'
__website__ = 'https://blog.csdn.net/TracelessLe'

通用变化：
1、添加noexcept关键字
2、BaseCreator->nvinfer1::pluginInternal::BaseCreator
3、ASSERT->PLUGIN_ASSERT
4、头文件导入有变化，如#include "bboxUtils.h"->#include "common/bboxUtils.h"，#include "gatherNMSOutputs.h"->#include "batchedNMSPluginV2/gatherNMSOutputs.h"
5、const void* const*->void const* const*，void**->void* const*
6、PluginFormat::kNCHW->PluginFormat::kLINEAR，TensorFormat::kNCHW->PluginFormat::kLINEAR
7、getPluginName增减
8、部分构造和析构函数需要注释，如BatchedNMSPluginV2Creator();和~BatchedNMSPluginV2Creator() override = default;
9、read->nvinfer1::plugin::read，write->nvinfer1::plugin::write
10、部分protected:内容需要注释，如// using nvinfer1::IPluginV2DynamicExt::canBroadcastInputAcrossBatch;

特殊算子变化：
1、nms新增参数int scoreBits, bool caffeSemantics
2、对于新增加的算子，需要在plugin/common/kernel.h头文件中增加定义

Binding使用变化：
1、新版TRT Python binding使用，builder与config的使用差异
2、升级TRT的同时PyTorch可能也做了升级，需要注意新版PyTorch的torch.onnx.export没有enable_onnx_checker参数，增加try/except

版权说明

本文为原创文章，独家发布在blog.csdn.net/TracelessLe。未经个人允许不得转载。如需帮助请email至[email protected]或扫描个人介绍栏二维码咨询。

参考资料

[1] TensorRT SDK | NVIDIA Developer
[2] Release Notes :: NVIDIA Deep Learning TensorRT Documentation
[3] ONNX转TensorRT加速模型推理_TracelessLe的博客-CSDN博客