在RK3588上使用Gstreamer做推拉流并推理记录
在RK3588上使用Gstreamer做推拉流并推理记录
基础环境
| 角色 | 版本 |
|---|---|
| rknn-toolkit2 | 1.2.0 |
| python | 3.7.3 |
1. 视频流使用结构
视频流拉取：
video/rtsp/camera ! decoder ! videoconvert ! appsink
视频流推流或显示：
appsrc ! display
推理结构：
appsink——rknn.inference(model)——appsrc
2. 环境安装
2.1 Gstreamer安装
apt-get update
apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
apt-get install libunwind8-dev
pkg-config --cflags --libs gstreamer-1.0
# 查看效果
dpkg -l | grep gstreamer
测试，在终端输入以下正常运行则安装成功
gst-launch-1.0 filesrc location=./test.mp4 ! decodebin ! videoconvert ! ximagesink
2.2 opencv安装
不能使用python3-opencv，通过pip安装的opencv无法支持gstreamer，必须使用源码编译安装
git clone https://github.com/opencv/opencv.git
unzip opencv.zip # 解压zip压缩包
cd opencv # 切换到源码包目录
mkdir build && cd build # 创建build目录并切换进去
# 其中需要配置python目录以及对gstreamer支持
cmake -D WITH_GSTREAMER=ON -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_opencv_python2=OFF -D BUILD_opencv_python3=ON -D PYTHON3_PACKAGES_PATH=/usr/local/python3.7/lib/python3.7/site-packages/ -D PYTHON3_LIBRARY=/usr/local/python3.7/lib/libpython3.7m.so -D OPENCV_GENERATE_PKGCONFIG=YES ..
确保opencv开启对gstreamer与python支持的api。
Video I/O:
DC1394: YES (2.2.5)
FFMPEG: YES
avcodec: YES (58.54.100)
avformat: YES (58.29.100)
avutil: YES (56.31.100)
swscale: YES (5.5.100)
avresample: YES (4.0.0)
GStreamer: YES (1.16.2) # 识别到gstreamer
v4l/v4l2: YES (linux/videodev2.h)
Python 3:
Interpreter: /usr/local/bin/python3 (ver 3.7.3)
Libraries: /usr/local/python3.7/lib/libpython3.7m.so (ver 3.7.3)
numpy: /usr/local/python3.7/lib/python3.7/site-packages/numpy/core/include (ver 1.21.6)
install path: /usr/local/python3.7/lib/python3.7/site-packages//cv2/python-3.7
编译安装
make -j4 # 编译
make install # 安装opencv库到/usr/local下
cd /etc/ld.so.conf.d/ # 切换目录
touch opencv.conf # 新建opencv配置文件
echo /usr/local/lib/ > opencv.conf # 填写opencv编译后库所在的路径
sudo ldconfig # 使配置文件生效
最后需要把cv2模块的代码目录中的cv2.so拷贝到python的site-packages目录中,替换原本python3-opencv的cv2.so。
注：# 如果查不到opencv的版本，需要将opencv4.pc 重命名为opencv.pc
运行以下程序正常表示安装成功
root@ztl:/# python3
Python 3.7.3 (default, Jul 6 2022, 07:24:11)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.getBuildInformation())
3. 使用案例
拉流：
rtsp拉取：in_gst = ('rtspsrc location={} latency=0 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw,format=(string)BGR ! appsink drop=true').format(url)
mp4视频拉取解码：in_gst = ('filesrc location={} ! decodebin ! videoconvert ! video/x-raw,format=BGR! appsink drop=true caps=video/x-raw,format=BGR').format(url)
推流：
本地显示：out_gst = 'appsrc is-live=true caps=video/x-raw,format=(string)BGR,framerate=(fraction)30/1 ! videoconvert ! ximagesink sync=false'
rtmp推流：后续更新
推理：
# 视频帧数据处理
cap_in = cv2.VideoCapture(in_gst, cv2.CAP_GSTREAMER)
cap_out = cv2.VideoWriter(out_gst, cv2.VideoWriter_fourcc(*'H264'), fps, (640, 320), True)
# 逐帧推理
output = model.inference(image)
# 视频流输出
cap_out = cv2.write(image)