【工程项目】训练yolov5模型并转换为rknn部署到RK3588S开发板

一. 部署概述

环境：Ubuntu20.04、python3.8

芯片：RK3568

二. 开发板刷系统

进入官网，下载必要文件

这里我选择下载ubuntu系统镜像。

1. 安装驱动

进入DriverAssitant_v5.1.1文件夹，开始安装驱动。

2. 安装系统

进入RKDevTool_Release_v2.93文件夹，启动开发工具。

插上开发板电源，并通过typec接口与电脑连接

当开发工具显示检测到ADB设备后，选择下载好的镜像。

点击Upgrade开始烧录系统。

三. 准备模型

接下来的操作在另一台linux系统上进行！
接下来的操作在另一台linux系统上进行！！
接下来的操作在另一台linux系统上进行！！！

重要的事情说三遍。

1.配置基本环境

安装anaconda3，如果没安装过的话，可以参考我这篇博客。【记录】使用ssh在Ubuntu Server上配置环境

2. 下载yolov5

yolov5（V6.0版本） github

创建虚拟环境conda

conda create -n yolov5 python=3.8
conda activate yolov5
conda install -n yolov5 conda=23.7.3 conda-build=3.26.1
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt -i https://pypi.douban.com/simple
pip install numpy==1.20.3 -i https://pypi.douban.com/simple
pip install onnx==1.12.0 -i https://pypi.douban.com/simple

3. 配置RKNN环境

建议采用通过Docker镜像安装的方式，后续不用担心因环境搭建引起的问题。其中包含的项目代码和docker镜像在github。

rknn-toolkit2-1.6.0-cp38-docker.tar.gz
rknn-toolkit2-master.zip

安装docker

sudo apt-get install docker.io

# 加载镜像
docker load --input rknn-toolkit2-1.6.0-cp38-docker.tar.gz 

feef05f055c9: Loading layer [==================================================>]  75.13MB/75.13MB
27a0fcbed699: Loading layer [==================================================>]  3.584kB/3.584kB
f62852363a2c: Loading layer [==================================================>]    424MB/424MB
d3193fc26692: Loading layer [==================================================>]  4.608kB/4.608kB
85943b0adcca: Loading layer [==================================================>]  9.397MB/9.397MB
0bec62724c1a: Loading layer [==================================================>]  9.303MB/9.303MB
e71db98f482d: Loading layer [==================================================>]  262.1MB/262.1MB
bde01abfb33a: Loading layer [==================================================>]  4.498MB/4.498MB
da9eed9f1e11: Loading layer [==================================================>]  5.228GB/5.228GB
85893de9b3b8: Loading layer [==================================================>]  106.7MB/106.7MB
0c9ec6e0b723: Loading layer [==================================================>]  106.7MB/106.7MB
d16b85c303bc: Loading layer [==================================================>]  106.7MB/106.7MB
Loaded image: rknn-toolkit2:1.6.0-cp38

# 检查镜像
docker images

REPOSITORY      TAG          IMAGE ID       CREATED         SIZE
rknn-toolkit2   1.6.0-cp38   afab63ce3679   7 months ago    6.29GB
hello-world     latest       feb5d9fea6a5   19 months ago   13.3kB

# 运行镜像并将examples映射到镜像空间。根据自己路径修改命令中的路径。
docker run -t -i --privileged -v /dev/bus/usb:/dev/bus/usb -v/Projects/zxc/rknn-toolkit2-master/rknn-toolkit2/examples//:/examples rknn-toolkit2:1.6.0-cp38 /bin/bash

bin  boot  dev  etc  examples  home  lib  lib32  lib64  libx32  media  mnt  opt  packages  proc  root  run  sbin  srv  sys  tmp  usr  var

# 运行demo
python test.py 

W __init__: rknn-toolkit2 version: 1.4.0-22dcfef4
--> Config model
W config: 'target_platform' is None, use rk3566 as default, Please set according to the actual platform!
done
--> Loading model
2023-04-20 15:21:12.334160: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:
2023-04-20 15:21:12.334291: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
done
--> Building model
I base_optimize ...
I base_optimize done.
I …………………………………………
D RKNN: [15:21:23.735] ----------------------------------------
D RKNN: [15:21:23.735] Total Weight Memory Size: 4365632
D RKNN: [15:21:23.735] Total Internal Memory Size: 1756160
D RKNN: [15:21:23.735] Predict Internal Memory RW Amount: 10331296
D RKNN: [15:21:23.735] Predict Weight Memory RW Amount: 4365552
D RKNN: [15:21:23.735] ----------------------------------------
D RKNN: [15:21:23.735] <<<<<<<< end: N4rknn21RKNNMemStatisticsPassE
I rknn buiding done
done
--> Export rknn model
done
--> Init runtime environment
W init_runtime: Target is None, use simulator!
done
--> Running model
Analysing : 100%|█████████████████████████████████████████████████| 60/60 [00:00<00:00, 1236.81it/s]
Preparing : 100%|██████████████████████████████████████████████████| 60/60 [00:00<00:00, 448.14it/s]
mobilenet_v1
-----TOP 5-----
[156]: 0.9345703125
[155]: 0.0570068359375
[205]: 0.00429534912109375
[284]: 0.003116607666015625
[285]: 0.00017178058624267578

done

4. 测试rk官方提供的yolov5s.onnx

进入目录/rknn-toolkit2-1.4.0/examples/onnx/yolov5，执行

python test.py

class: person, score: 0.8223356008529663
box coordinate left,top,right,down: [473.26745200157166, 231.93780636787415, 562.1268351078033, 519.7597033977509]
class: person, score: 0.817978024482727
box coordinate left,top,right,down: [211.9896697998047, 245.0290389060974, 283.70787048339844, 513.9374527931213]
class: person, score: 0.7971192598342896
box coordinate left,top,right,down: [115.24964022636414, 232.44154334068298, 207.7837154865265, 546.1097872257233]
class: person, score: 0.4627230763435364
box coordinate left,top,right,down: [79.09242534637451, 339.18042743206024, 121.60038471221924, 514.234916806221]
class: bus , score: 0.7545359134674072
box coordinate left,top,right,down: [86.41703361272812, 134.41848754882812, 558.1083570122719, 460.4184875488281]

执行完在此路径下可以看到生成了一张result.jpg，打开可以看到预测结果图。

5.转换yolov5s-Pytorch模型并测试推理

5.1 pt转onnx

注意！
注意！！
注意！！！

接下来的步骤很关键，我在网上找的博客提供的方案都不能正常推理。

转换步骤：

• 修改models/yolo.py，修改class Detect(nn.Module):的forward函数，注意！！！仅在转换时修改，在训练时改回原状态！再训练时不要忘记哦！

	# def forward(self, x):
    #     z = []  # inference output
    #     for i in range(self.nl):
    #         x[i] = self.m[i](x[i])  # conv
    #         bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
    #         x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
    #
    #         if not self.training:  # inference
    #             if self.grid[i].shape[2:4] != x[i].shape[2:4] or self.onnx_dynamic:
    #                 self.grid[i] = self._make_grid(nx, ny).to(x[i].device)
    #
    #             y = x[i].sigmoid()
    #             if self.inplace:
    #                 y[..., 0:2] = (y[..., 0:2] * 2. - 0.5 + self.grid[i]) * self.stride[i]  # xy
    #                 y[..., 2:4] = (y[..., 2:4] * 2) ** 2 * self.anchor_grid[i]  # wh
    #             else:  # for YOLOv5 on AWS Inferentia https://github.com/ultralytics/yolov5/pull/2953
    #                 xy = (y[..., 0:2] * 2. - 0.5 + self.grid[i]) * self.stride[i]  # xy
    #                 wh = (y[..., 2:4] * 2) ** 2 * self.anchor_grid[i].view(1, self.na, 1, 1, 2)  # wh
    #                 y = torch.cat((xy, wh, y[..., 4:]), -1)
    #             z.append(y.view(bs, -1, self.no))
    #
    #     return x if self.training else (torch.cat(z, 1), x)

    # 很多博主的修改是这一版本，但是这一版本会导致最终部署在RK3588S开发板上是出现测试结果出错
    # 出现重复框选，选框混乱，置信度高于1的错误。
    # def forward(self, x):
    #     z = []  # inference output
    #     for i in range(self.nl):
    #         x[i] = self.m[i](x[i])  # conv
    #     return x
    
    def forward(self, x):
        z = []  # inference output
        for i in range(self.nl):
            if os.getenv('RKNN_model_hack', '0') != '0':
                x[i] = torch.sigmoid(self.m[i](x[i]))  # conv

        return x

• 将export.py文件中的run函数下的语句：

shape = tuple((y[0] if isinstance(y, tuple) else y).shape)  # model output shape

改为：

# 这里同样和别人不同，记得更新成我的这个版本
shape = tuple(y[0].shape)  # model output shape

并且在代码开头添加：

import os
os.environ['RKNN_model_hack'] = 'npu_2'

运行指令：

# 可以将模型路径修改为自己训练保存的模型路径
python export.py --weights best.pt --img 640 --batch 1 --include onnx --opset 12

• 然后在主文件夹下出现了一个best.onnx文件，在Netron中查看模型是否正确
• 点击左上角菜单->Properties…
• 查看右侧OUTPUTS是否出现三个输出节点，是则ONNX模型转换成功。
• 如果转换好的best.onnx模型不是三个输出节点，则不用尝试下一步，会各种报错。

5.2 onnx转rknn并在PC上仿真测试

修改examples/onnx/yolov5/test.py文件

• 修改onnx路径
• 修改rknn保存路径
• 修改img测试图片路径
• 修改类别数

修改完成后运行脚本，获得推理坐标值和rknn模型文件，推理结果如下：

5.3 用自己数据集训练的模型板端推理

1. 进入/rknpu2/examples/rknn_yolov5_demo_v5/convert_rknn_demo/yolov5目录，修改onnx2rknn.py：

• 修改platform
• 修改模型路径
• 修改输入图片大小

2. 修改dataset.txt：

3. 运行脚本，将onnx转为rknn；

python onnx2rknn.py

4. 复制rknn模型到/rknpu2/examples/rknn_yolov5_demo_c4/model/RK3588

5. 进入 /rknpu2/examples/rknn_yolov5_demo_c4/model目录，修改coco_80_labels_list.txt：

6. 修改/rknpu2/examples/rknn_yolov5_demo_c4/include/postprocess.h，修改类别数和置信度阈值：

7.进入/rknpu2/examples/rknn_yolov5_demo目录，运行build-linux_RK356X.sh脚本编译程序，编译成功会生成一个install/和build/文件夹：

bash ./build-linux_RK356X.sh

编译成功会生成一个install/和build/文件夹，

将install文件夹下的文件全部复制到开发板中，赋予rknn_yolov5_demo可执行权限，进入开发板中运行程序测试推理：

[root@RK356X:/mnt/rknn_yolov5_demo_Linux_c4]# chmod a+x rknn_yolov5_demo
[root@RK356X:/mnt/rknn_yolov5_demo_Linux_c4]# ./rknn_yolov5_demo ./model/RK356X/best-sim.rknn ./model/per_car.jpg
post process config: box_conf_threshold = 0.25, nms_threshold = 0.25
Read ./model/per_car.jpg ...
img width = 640, img height = 640
Loading mode...
sdk version: 1.4.0 (a10f100eb@2022-09-09T09:07:14) driver version: 0.4.2
model input num: 1, output num: 3
  index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
  index=0, name=output, n_dims=4, dims=[1, 27, 80, 80], n_elems=172800, size=172800, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=64, scale=0.122568
  index=1, name=320, n_dims=4, dims=[1, 27, 40, 40], n_elems=43200, size=43200, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=38, scale=0.110903
  index=2, name=321, n_dims=4, dims=[1, 27, 20, 20], n_elems=10800, size=10800, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=36, scale=0.099831
model is NHWC input fmt
model input height=640, width=640, channel=3
once run use 86.341000 ms
loadLabelName ./model/coco_80_labels_list.txt
person @ (61 190 284 520) 0.907933
car @ (381 370 604 446) 0.897453
car @ (343 366 412 394) 0.873785
car @ (404 367 429 388) 0.628984
car @ (425 361 494 388) 0.365345
loop count = 10 , average run  96.655700 ms

推理结果如下：