用C++部署yolov5+deepsort+tensorrt实现目标跟踪
一、参考资料
Jetson 系列——基于yolov5和deepsort的多目标头部识别，跟踪，使用tensorrt和c++加速
二、相关介绍
2.1 重要说明
该项目能部署在Jetson系列的产品，也能部署在X86 服务器中。
2.2 项目结构
.
├── assets
│ └── yolosort.gif
├── build # 编译的文件夹
│ ├── CMakeCache.txt
│ ├── CMakeFiles
│ ├── cmake_install.cmake
│ ├── libdeepsort.so
│ ├── libyolov5_trt.so
│ ├── Makefile
│ └── yolosort
├── CMakeLists_deepsort-tensorrt_win10.txt
├── CMakeLists.txt
├── CMakeLists_yolov5-deepsort-tensorrt_win10.txt
├── deepsort # deepsort源码
│ ├── include
│ └── src
├── resources # 存放engine引擎的文件夹
├── include
│ └── manager.hpp
├── LICENSE
├── README.md
├── requirementes-gpu.txt
├── src
│ ├── main.cpp # 入口函数
│ └── manager.cpp
└── yolo # yolo源码
├── include
└── src
2.3 主项目
yolov5-deepsort-tensorrt
2.4 生成 onnx 文件
deep_sort_pytorch
2.5 生成 deepsort.engine
deepsort-tensorrt
2.6 生成 yolov5s.engine
TensorRT实现yolov5推理加速（一）
tensorrtx/yolov5
三、实验环境
3.1 系统环境
由于博主在PC主机上测试部分代码，以下实验环境为PC主机的环境，以后有条件，我再测试一下Jetson TX2的效果。
Environment
Operating System + Version: Ubuntu + 16.04
GPU Type: GeForce GTX1650,4GB
Nvidia Driver Version: 470.63.01
CUDA Version: 10.2.300
CUDNN Version: 7.6.5
Python Version (if applicable): 3.6.14
virtualenv：20.13.0
gcc：7.5.0
g++：7.5.0
3.2 requirements-gpu.txt
absl-py==1.0.0
appdirs==1.4.4
backcall==0.2.0
cached-property==1.5.2
cachetools==4.2.4
certifi==2021.10.8
chardet==4.0.0
charset-normalizer==2.0.10
cycler==0.11.0
Cython==0.29.26
dataclasses==0.8
decorator==4.4.2
distlib==0.3.4
easydict==1.9
filelock==3.4.1
flake8==4.0.1
future==0.18.2
gdown==3.10.1
google-auth==2.3.3
google-auth-oauthlib==0.4.6
graphsurgeon @ file:///home/yichao/360Downloads/TensorRT-7.1.3.4/graphsurgeon/graphsurgeon-0.4.5-py2.py3-none-any.whl
grpcio==1.43.0
h5py==3.1.0
idna==3.3
imageio==2.13.5
importlib-metadata==4.2.0
importlib-resources==5.4.0
imutil==0.2.5
ipdb==0.13.9
ipython==7.16.3
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.17.2
kiwisolver==1.3.1
Mako==1.1.6
Markdown==3.3.5
MarkupSafe==2.0.1
matplotlib==3.3.4
mccabe==0.6.1
networkx==2.5.1
numpy==1.19.5
oauthlib==3.1.1
opencv-python==4.5.5.62
pandas==1.1.5
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.4.0
platformdirs==2.4.0
prompt-toolkit==3.0.24
protobuf==3.19.3
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.8.0
pycuda==2020.1
pyflakes==2.4.0
Pygments==2.11.2
pyparsing==3.0.6
PySocks==1.7.1
python-dateutil==2.8.2
pytools==2021.2.9
pytz==2021.3
PyWavelets==1.1.1
PyYAML==6.0
requests==2.27.1
requests-oauthlib==1.3.0
rsa==4.8
scikit-image==0.17.2
scipy==1.5.4
seaborn==0.11.2
six==1.16.0
tb-nightly==2.8.0a20220117
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorrt @ file:///home/yichao/360Downloads/TensorRT-7.1.3.4/python/tensorrt-7.1.3.4-cp36-none-linux_x86_64.whl
tifffile==2020.9.3
toml==0.10.2
torch @ file:///home/yichao/%E4%B8%8B%E8%BD%BD/torch-1.9.0%2Bcu102-cp36-cp36m-linux_x86_64.whl
torchvision @ file:///home/yichao/%E4%B8%8B%E8%BD%BD/torchvision-0.10.0%2Bcu102-cp36-cp36m-linux_x86_64.whl
tqdm==4.62.3
traitlets==4.3.3
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
Werkzeug==2.0.2
yacs==0.1.8
yapf==0.32.0
zipp==3.6.0
四、生成 onnx 文件
4.1 下载github代码
git clone https://github.com/RichardoMrMu/deepsort-tensorrt.git
4.2 下载预训练模型
ckpt.t7，46MB
下载完成后，将ckpt.t7放到 deep_sort_pytorch/deep_sort/deep/checkpoint/ckpt.t7 路径。
4.3 生成 onnx 文件
# 拷贝文件
cp {deepsort-tensorrt}/exportOnnx.py {deep_sort_pytorch}/
cd deep_sort_pytorch
# 生成onnx文件
python3 exportOnnx.py
# 拷贝onnx文件到对应目录
mv {deep_sort_pytorch}/deepsort.onnx {deepsort-tensorrt}/resources
(venv) yichao@yichao:~/下载/deep_sort_pytorch$ time python exportOnnx.py
==> Exporting model to ONNX format at 'deepsort.onnx'
/home/yichao/MyDocuments/Yolov5_DeepSort_Pytorch/venv/lib/python3.6/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
real 0m26.103s
user 0m7.749s
sys 0m2.463s
deepsort.onnx，44.7MB
4.3.1 显卡占用情况
Fri Jan 21 15:03:09 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01 Driver Version: 470.63.01 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 27% 30C P0 17W / 75W | 1419MiB / 3903MiB | 2% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1481 G /usr/lib/xorg/Xorg 264MiB |
| 0 N/A N/A 10705 C python 1151MiB |
+-----------------------------------------------------------------------------+
五、生成 deepsort.engine
5.1 拷贝onnx文件到对应目录
mv {deep_sort_pytorch}/deepsort.onnx {deepsort-tensorrt}/resources
5.2 生成 CMakeLists.txt 文件
cd {deepsort-tensorrt}
# cmake
mkdir build
cd build
cmake ..
5.3 修改 CMakeLists.txt 文件
deepsort-tensorrt/CMakeLists.txt 添加tensorRT库。
# tensorRT
include_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/include/)
link_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/lib/)
deepsort-tensorrt/CMakeLists.txt 添加Eigen库。
include_directories(/usr/include/eigen3)
5.4 编译
make -j8
5.5 生成engine引擎
deepsort.onnx 的路径：/home/yichao/下载/deepsort-tensorrt/resources/deepsort.onnx。
deepsort.engine 的路径：/home/yichao/下载/deepsort-tensorrt/resources/deepsort.engine。
./onnx2engine ../resources/deepsort.onnx ../resources/deepsort.engine
(venv) yichao@yichao:~/下载/deepsort-tensorrt/build$ time ./onnx2engine ../resources/deepsort.onnx ../resources/deepsort.engine
----------------------------------------------------------------
Input filename: ../resources/deepsort.onnx
ONNX IR version: 0.0.6
Opset version: 10
Producer name: pytorch
Producer version: 1.9
Domain:
Model version: 0
Doc string:
----------------------------------------------------------------
[01/21/2022-15:10:32] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
==============
| SUCCESS! |
==============
real 1m58.582s
user 0m27.711s
sys 0m3.474s
deepsort.engine，24.6MB
5.5.1 显存占用情况
Fri Jan 21 15:11:04 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01 Driver Version: 470.63.01 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 27% 34C P0 44W / 75W | 1255MiB / 3903MiB | 100% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1481 G /usr/lib/xorg/Xorg 264MiB |
| 0 N/A N/A 11074 C ./onnx2engine 987MiB |
+-----------------------------------------------------------------------------+
5.6 测试引擎是否生成
./demo ../resource/deepsort.engine ../resources/track.txt
六、主项目
6.1 下载github代码
# 下载github代码
git clone https://github.com/RichardoMrMu/yolov5-deepsort-tensorrt.git
6.2 拷贝engine到对应目录
yolov5-deepsort-tensorrt/engine/yolov5s.engine
yolov5-deepsort-tensorrt/engine/deepsort.engine
6.3 修改配置
yolov5-deepsort-tensorrt/src/main.cpp，修改engine的路径。
// before you cmake and make, please change ./src/main.cpp char* yolo_engine = ""; char* sort_engine = ""; to your own path
char* yolo_engine = "yolov5-deepsort-tensorrt/engine/yolov5s.engine";
char* sort_engine = "yolov5-deepsort-tensorrt/engine/deepsort.engine";
// 设置摄像头索引
frame = capture.open(0);
// 显示检测结果
// 在 yolov5-deepsort-tensorrt/src/manager.cpp 文件中，取消注释
showDetection(frame,det); # 取消注释
6.4 生成 CMakeLists.txt 文件
cd yolov5-deepsort-tensorrt
# cmake
mkdir build
cd build
cmake ..
6.5 修改 CMakeLists.txt 文件
deepsort-tensorrt/CMakeLists.txt 添加tensorRT库。
# tensorRT
include_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/include)
link_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/lib/)
deepsort-tensorrt/CMakeLists.txt 添加Eigen库。
include_directories(/usr/include/eigen3)
6.6 编译
make -j8
6.7 运行
# 运行
./yolosort
(venv) yichao@yichao:~/下载/yolov5-deepsort-tensorrt/build$ ./yolosort
yolov5_trt_create ...
yolov5_trt_create cuda engine...
yolov5_trt_create buffer ...
yolov5_trt_create stream ...
yolov5_trt_create done ...
create yolov5-trt , instance = 0x25a10a0
delay_proress:3ms, delay_infer:7ms
delay_infer:33ms
delay_proress:6ms, delay_infer:6ms
delay_infer:21ms
delay_proress:3ms, delay_infer:6ms
delay_infer:16ms
delay_proress:12ms, delay_infer:7ms
delay_infer:29ms
delay_proress:4ms, delay_infer:7ms
delay_infer:19ms
delay_proress:12ms, delay_infer:7ms
delay_infer:30ms
delay_proress:10ms, delay_infer:7ms
delay_infer:26ms
delay_proress:5ms, delay_infer:7ms
delay_infer:18ms
...
delay_proress：yolov5预处理时间；
delay_infer：yolov5目标检测时间；
delay_infer：总共的时间。
6.7.1 显存占用情况
Fri Jan 21 14:37:18 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01 Driver Version: 470.63.01 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 27% 31C P0 18W / 75W | 1082MiB / 3903MiB | 16% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1481 G /usr/lib/xorg/Xorg 264MiB |
| 0 N/A N/A 9950 C ./yolosort 814MiB |
+-----------------------------------------------------------------------------+
七、可能存在的问题
Q1：yaml.load()函数缺少参数错误
运行 python exportOnnx.py 出错。
(venv) yichao@yichao:~/下载/deep_sort_pytorch$ python exportOnnx.py
Traceback (most recent call last):
File "exportOnnx.py", line 22, in <module>
cfg.merge_from_file(args.config_deepsort)
File "/home/yichao/下载/deep_sort_pytorch/utils/parser.py", line 23, in merge_from_file
self.update(yaml.load(fo.read()))
TypeError: load() missing 1 required positional argument: 'Loader'
错误原因：
yaml.load() 函数的参数不对，缺少 Loader 对象。
yaml.load(CLIN,Loader=yaml.FullLoader) # 加上Loader=yaml.FullLoader 避免警告。
注：yaml版本5.1之后弃用，YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated.
解决办法：
修改 deep_sort_pytorch/utils/parser.py 中的代码。
cfg_dict.update(yaml.load(fo.read()))
改成
cfg_dict.update(yaml.load(fo, Loader=yaml.FullLoader))
Q2：缺少 NInfer.h 文件
In file included from /home/yichao/下载/deepsort-tensorrt/include/deepsort.h:6:0,
from /home/yichao/下载/deepsort-tensorrt/src/deepsort.cpp:1:
/home/yichao/下载/deepsort-tensorrt/include/featuretensor.h:7:10: fatal error: NvInfer.h: 没有那个文件或目录
#include 错误原因：
NvInfer.h 头文件属于 TensorRT 下的一个专有头文件，在编译C++ 代码时需要找到它。
解决办法：
deepsort-tensorrt/CMakeLists.txt，增加tensorRT的依赖库
# tensorRT
include_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/include)
link_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/lib/)
Q3：缺少 Eigen/Core 文件
/home/yichao/360Downloads/TensorRT-7.1.3.4/include/NvInfer.h:3339:22: note: declared here
class TRT_DEPRECATED IOutputDimensionsFormula
^~~~~~~~~~~~~~~~~~~~~~~~
/home/yichao/360Downloads/TensorRT-7.1.3.4/include/NvInfer.h:5565:81: warning: ‘IPluginLayer’ is deprecated [-Wdeprecated-declarations]
ITensor* const* inputs, int nbInputs, IPluginExt& plugin) TRTNOEXCEPT = 0;
^
/home/yichao/360Downloads/TensorRT-7.1.3.4/include/NvInfer.h:3373:22: note: declared here
class TRT_DEPRECATED IPluginLayer : public ILayer
^~~~~~~~~~~~
CMakeFiles/Makefile2:86: recipe for target 'CMakeFiles/deepsort.dir/all' failed
make[1]: *** [CMakeFiles/deepsort.dir/all] Error 2
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2
解决办法：重新make
make -j8
/home/yichao/下载/deepsort-tensorrt/include/datatype.h:28:10: fatal error: Eigen/Core: 没有那个文件或目录
#include 错误原因：
编译找不到Eigen
解决办法：
1. 安装eigen3，默认安装在 /usr/include/eigen3
sudo apt-get install libeigen3-dev
2. deepsort-tensorrt/CMakeLists.txt，增加eigen3的依赖库
include_directories(/usr/include/eigen3)
# 如果还不成功，尝试创建软链接
sudo ln -s /usr/include/eigen3/Eigen /usr/include/Eigen
3. 重新编译
make clean
make -j8
Q4：打不开摄像头
(venv) yichao@yichao:~/下载/yolov5-deepsort-tensorrt/build$ ./yolosort
yolov5_trt_create ...
yolov5_trt_create cuda engine...
yolov5_trt_create buffer ...
yolov5_trt_create stream ...
yolov5_trt_create done ...
create yolov5-trt , instance = 0x295e000
GStreamer: Error opening bin: no element "0"
can not open
(venv) yichao@yichao:~/下载/yolov5-deepsort-tensorrt/build$ ./yolosort
yolov5_trt_create ...
yolov5_trt_create cuda engine...
yolov5_trt_create buffer ...
yolov5_trt_create stream ...
yolov5_trt_create done ...
create yolov5-trt , instance = 0x186d000
GStreamer: Error opening bin: empty pipeline not allowed
can not open
错误原因：
在 yolov5-deepsort-tensorrt/src/main.cpp 文件中，未设置摄像头索引或者设置摄像头失败。
frame = capture.open(""); // 未设置索引
frame = capture.open("0"); // 设置失败，索引为int类型
解决办法：
在 yolov5-deepsort-tensorrt/src/main.cpp 文件中，设置摄像头索引。
frame = capture.open(0);