FPV Camera(RPI3+V2.1) | wfb_ng Release 23.01 | Intel(R) Core(TM)2 Duo CPU T6570 @ 2.10GHz 测试
@[TOC](FPV Camera(RPI3+V2.1) | wfb_ng Release 23.01 | Intel® Core™2 Duo CPU T6570 @ 2.10GHz 测试)
1. 简介
最近搭建了基于树莓派和wfb_ng通信的FPV摄像头方案，该方案能解决：
- 视频数据传输
- 控制链路通信
- 链路状态检测
- 链路安全加密
对该方案的一些简单学习和研读，基本规格&特性如下：
- 1280x720@60FPS latency ~50ms
- 支持MAVLink OSD显示 (兼容Ardupilot/PX4)
- 支持Smart FEC
- 支持双向MAVLink电传通道
- 支持IP-over-WFB IPv4 tunnel
- 基于libsodium支持数据加密
- 支持链路维护检测
注：本文不做其他扩展，主要基于当前测试框架下对测试数据做出解释，并提出后续完善的硬件方案和预测。
2. 硬件准备
天空端#1
- Rapberry Pi Zero W
- Rapberry Pi v2.1 imx219 3280x2464
天空端#2
- Rapberry Pi 3B+
- Rapberry Pi v2.1 imx219 3280x2464
地面端
- Lenovo Laptop SL410
- Intel® Core™2 Duo CPU T6570 @ 2.10GHz
- [AMD/ATI] RV710/M92 [Mobility Radeon HD 4530/4570/545v]
注：后来发现gst-launch-1.0播放H264的时候，并没有使用显卡GPU，而从一些参考信息中看该显卡至少是支持的，具体原因不清楚，有知道的朋友也请指点下。
3. 软件准备
请参考wfb-ng Release 23.01镜像无头烧录&配置(2)准备天空端和地面端软件。
鉴于测试，所有脚本都采用显式命令行调用方式执行，以确保每个步骤执行是否有意外发生，方便定位问题。
3.1 天空端命令
启动天空端wfb_ng
$ sudo systemctl start wifibroadcast@drone
天空端wfb_ng状态查询
$ sudo systemctl status wifibroadcast@drone
天空端wfb_ng日志查询
sudo journalctl -xu wifibroadcast@drone
停止天空端wfb_ng
$ sudo systemctl stop wifibroadcast@drone
视频采集命令(CTRL + C 退出)
$ raspivid -n -ex fixedfps -w 1280 -h 720 -b 4000000 -fps 30 -t 0 -o - \
| gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=35 ! udpsink sync=false host=127.0.0.1 port=5602
or
$ gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=1280,height=720,framerate=30/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
3.2 地面端命令
启动地面端wfb_ng
$ sudo systemctl start wifibroadcast@gs
地面端wfb_ng状态查询
$ sudo systemctl stop wifibroadcast@gs
地面端wfb_ng日志查询
sudo journalctl -xu wifibroadcast@gs
停止地面端wfb_ng
$ sudo systemctl stop wifibroadcast@gs
地面端维测命令(CTRL + C 退出)
$ wfb-cli gs
视频播放命令(CTRL + C 退出)
$ gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' \
! rtph264depay ! avdec_h264 ! clockoverlay valignment=bottom ! autovideosink fps-update-interval=1000 sync=false
or
$ gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' \
! rtph264depay ! avdec_h264 ! clockoverlay valignment=bottom ! xvimagesink sync=false
4. 测试
默认地面端采用以下方式播放天空端视频：
$ gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' \
! rtph264depay ! avdec_h264 ! clockoverlay valignment=bottom ! autovideosink fps-update-interval=1000 sync=false
4.1 使用 天空端#1 + 地面端 测试
4.1.1 raspivid 1280x720 720P 30FPS
raspivid -n -ex fixedfps -w 1280 -h 720 -b 4000000 -fps 30 -t 0 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=35 ! udpsink sync=false host=127.0.0.1 port=5602
raspivid1280x720@fps30
4.1.2 raspivid 1280x720 720P 60FPS
raspivid -n -ex fixedfps -w 1280 -h 720 -b 4000000 -fps 60 -t 0 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=35 ! udpsink sync=false host=127.0.0.1 port=5602
raspivid1280x720@fps60
4.1.3 gst-launch-1.0 1280x720 720P 30FPS
gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=1280,height=720,framerate=30/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
RPI-zero-gst1280x720@fps30
4.1.4 gst-launch-1.0 1280x720 720P 60FPS
gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=1280,height=720,framerate=60/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
**测试失败，需要重启树莓派才能启动摄像头。**请参考：Performance test issue based on Release 23.01 image #289
$ gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_lin e_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_ header=1" \
! video/x-h264,profile=high,width=1280,height=720,framerate=60/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-laten cy ! udpsink host=127.0.0.1 port=5602 sync=false
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:07.5 / 99:99:99.
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.640766] Internal error: Oops - undefined instruction: 0 [#1] ARM
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.660055] Process v4l2src0:src (pid: 786, stack limit = 0xcb2e4637)
0:00:07.5 / 99:99:99.
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.660360] Stack: (0xc32cbe10 to 0xc32cc000)
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.660558] be00: c32cbe3b c0099ba4 c0c2f028 00000000
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.660898] be20: 00000000 00000000 00000000 00000000 c990d0a0 b3c0c878 00004000 00000003
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.661152] be40: 00000001 00000000 00002003 00000001 00000000 00000000 00000000 00000000
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.661376] be60: 00000000 00000000 00000000 00000000 00000000 00000000 00000001 002a3000
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.661614] be80: 000e1000 00000000 00000000 c020f9a4 c32cbed8 00000080 00000001 00000000
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.672070] bea0: c32cbde4 00000010 00010000 00000000 00000000 c32cbdbc 00000000 3c38b2dd
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.689153] bec0: 00000000 bf26cd50 c990d0a1 00000000 c990d0a0 c0c2f028 b3c0c878 00000009
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.700053] bee0: c32cbefc c32cbef0 bf26cd6c bf26c7b4 c32cbf14 c32cbf00 bf264148 bf26cd5c
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.718157] bf00: c044560f c990d0a1 c32cbfa4 c32cbf18 c02244d4 bf264108 00000010 00000000
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.728048] bf20: 00000517 c04e0794 c32cbf74 c32cbf38 c008072c c04e0780 00000517 00000000
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.738462] bf40: 25eee4e0 00000000 00000193 3c38b2dd c32ca000 b4df8b48 c0c2f028 b6f15880
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.754826] bf60: 00000193 c00083e4 c32cbfa4 c32cbf78 c0091c74 3c38b2dd 00000517 b3c0c850
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.766579] bf80: 00000001 b3c0c858 00000036 c00083e4 c32ca000 00000000 00000000 c32cbfa8
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.785670] bfa0: c0008260 c02243cc b3c0c850 00000001 00000009 c044560f b3c0c878 b6b53d40
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.794324] bfc0: b3c0c850 00000001 b3c0c858 00000036 004dd790 b5840000 004f4a6c 004f4b94
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.805115] bfe0: b58405d8 b4df839c b57efa84 b6b53d4c 60000010 00000009 00000000 00000000
0:00:07.5 / 99:99:99.
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.820897] Backtrace:
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.830017] [<bf26c7a8>] (video_usercopy [videodev]) from [<bf26cd6c>] (video_ioctl2+0x1c/0x24 [videodev])
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.849377] r10:00000009 r9:b3c0c878 r8:c0c2f028 r7:c990d0a0 r6:00000000 r5:c990d0a1
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.859025] r4:bf26cd50
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.867712] [<bf26cd50>] (video_ioctl2 [videodev]) from [<bf264148>] (v4l2_ioctl+0x4c/0x64 [videodev])
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.880672] [<bf2640fc>] (v4l2_ioctl [videodev]) from [<c02244d4>] (sys_ioctl+0x114/0x9b0)
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.892259] r5:c990d0a1 r4:c044560f
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.903704] [<c02243c0>] (sys_ioctl) from [<c0008260>] (ret_fast_syscall+0x0/0x1c)
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.917238] Exception stack(0xc32cbfa8 to 0xc32cbff0)
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.929780] bfa0: b3c0c850 00000001 00000009 c044560f b3c0c878 b6b53d40
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.939761] bfc0: b3c0c850 00000001 b3c0c858 00000036 004dd790 b5840000 004f4a6c 004f4b94
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.954747] bfe0: b58405d8 b4df839c b57efa84 b6b53d4c
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.968891] r10:00000000 r9:c32ca000 r8:c00083e4 r7:00000036 r6:b3c0c858 r5:00000001
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.985942] r4:b3c0c850
Message from syslogd@piCamera at Mar 9 09:26:11 ...
kernel:[ 1303.993852] Code: bfa396d4 00000018 0001712a 00000056 (00051cf2)
^Chandling interrupt..
Interrupt: Stopping pipeline ...
Execution ended after 0:00:47.241435745
Setting pipeline to NULL ...
4.2 使用 天空端#2 + 地面端 测试
4.2.1 gst-launch-1.0 1280x720 720P 30FPS
gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=1280,height=720,framerate=30/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
RPI3B-gst1280x720@fps30
4.2.2 gst-launch-1.0 1280x720 720P 45FPS
gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=1280,height=720,framerate=45/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
RPI3B-gst1280x720@fps45
4.2.3 gst-launch-1.0 1280x720 720P 60FPS
gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=1280,height=720,framerate=60/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
RPI3B-gst1280x720@fps60
4.1.4 gst-launch-1.0 800x600 SVGA 60FPS
$ gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=800,height=600,framerate=60/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
RPI3B-gst800x600@fps60
4.1.5 gst-launch-1.0 640x480 VGA 60FPS
$ gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=640,height=480,framerate=60/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
RPI3B-gst640x480@fps60
4.1.6 gst-launch-1.0 480x320 HVGA 60FPS
gst-launch-1.0 v4l2src do-timestamp=true io-mode=mmap device=/dev/video0 \
extra-controls="controls,image_stabilization=1,h264_profile=4,h264_i_frame_period=60,h264_level=11,power_line_frequency=1,exposure_metering_mong=1,exposure_dynamic_framerate=0,scene_mode=0,video_bitrate_mode=1,video_bitrate=2000000,repeat_sequence_header=1" \
! video/x-h264,profile=high,width=480,height=320,framerate=60/1,stream-format=byte-stream \
! h264parse disable-passthrough=true ! rtph264pay config-interval=1 pt=96 mtu=1400 aggregate-mode=zero-latency ! udpsink host=127.0.0.1 port=5602 sync=false
RPI3B-gst480x320@fps60
5. 测试总结
本次测试，基本上没有发生丢包，也没有显著的FEC纠错报文出现，整体收到WiFi信号干扰可以忽略不计。
5.1 天空端#1(RPI Zero W)
- raspiivid视频性能不如gst-launch-1.0
- gst-launch-1.0 720P 60FPS出现内核异常，视频卡主
- RPI v2.1摄像头可手动调节焦距（测试视频没有调好）
wfb_ng作者svpcom当下推荐：RPI3 or jetson with MIPI cameras
注：RPI Zero W出现内核驱动异常，可能是88XXau_wfb.ko在ARM1176JZF-S上执行时出现了问题，有兴趣的朋友可跟下这个问题。
5.2 天空端#2(RPI 3b+)延时分析
5.2.1 480x320 HVGA 60FPS x 3张清晰延时，统计平均 44ms，最小43ms，最大44ms
-
第一张：75 - 31 = 44 ms
-
第二张：852 - 808 = 44ms
- 第三张：457 - 414 = 43 ms
5.2.2 640x480 VGA 60FPS x 3张清晰延时，统计平均 57ms，最小43ms，最大87ms
- 第一张：304 - 261 = 43ms
- 第二张：687 - 644 = 43ms
- 第三张：863 - 776 = 87ms
5.2.3 800x600 SVGA 60FPS x 3张清晰延时，统计平均 86ms，最小86ms，最大86ms
- 第一张：200 - 114 = 86ms
- 第二张：614 - 528 = 86ms
- 第三张：994 - 908= 86 ms
5.2.3 1280x720 720P 45FPS x 3张清晰延时，统计平均 100ms，最小86ms，最大130ms
- 第一张：1028 - 898= 130 ms
- 第二张：934 - 848 = 86 ms
- 第三张：816 - 686= 130 ms
5.2.5 1280x720 720P 60FPS - 视频卡顿
鉴于该模式下，整体视频出现卡顿，延时测试取消。
**天空端CPU占用率：7.9 + 7.9 + 1.0 x 2 + 0.7 x 4 ~= 20 **
**地面端CPU占用率：69.6 + 20.6 + 18.6 + 15.7 + 7.8 + 2.9 + 2.0 x 2 + 1.0 x 3 ~= 142.2 **
5.3 视频延时组成
视频链路：摄像头==》编码==》传输==》解码==》显示，大体上可以分为以下几个延时：
- 摄像头数据采集延时(camera)
- 编码器编码延时(H264 codec)
- 无线网络延时(wfb_ng)： ~ 5ms
- 解码器解码延时(H264 decoder)
- 显示器刷新延时(monitor refresh rate)
上述时延并非全同步的情况下顺序发生，因为编解码码过程有IBP帧算法问题考虑，这里我们暂时不深入研究，对算法有兴趣的朋友可以看下补充资料，然后扩展，自行研究。
5.4 视频延时测试结果
基于以下前提：
- 视频流畅
- 静态背景
- 丢包为零
- FEC纠正忽略不计
- 地面端软解
- 480x320 HVGA 60FPS x 3张清晰延时，统计平均 44ms，最小43ms，最大44ms
- 640x480 VGA 60FPS x 3张清晰延时，统计平均 57ms，最小43ms，最大87ms
- 800x600 SVGA 60FPS x 3张清晰延时，统计平均 86ms，最小86ms，最大86ms
- 1280x720 720P 45FPS x 3张清晰延时，统计平均 100ms，最小86ms，最大130ms
得出以下结论：
- 天空端CPU占用率低，主要采用H264编码器编码
- 地面端CPU占用率高，主要采用CPU软解H264
- 降低分辨率，减少编解码延时，可以显著提升时延
- 与wfb_ng开源作者svpcom测试 720P ~ 50ms相比，主要问题可能是解码
后续工作重点：
- 视频时延计算方法，需要了解H264/H265编解码和IPB帧；
- RPI3摄像头编码部分CPU占用率低，考虑地面端采用同类型芯片进行解码测试；
- 背景画面大幅度变化(飞机飞行画面对比);
- gstream参数优化 ;
6. 参考资料
【1】FPV Camera(RPI3+V2.1) | wfb_ng Release 23.01 | Ubuntu 20.04 xfce软解测试
【2】wfb-ng Release 23.01镜像无头烧录&配置(2)
【3】wfb-ng Release 23.01镜像无头烧录&配置(1)
【4】wfb-ng 锁定WiFi接口
【5】Raspbian镜像无头烧录
【6】ATI Mobility Radeon HD 4530
【7】RadeonDriver for ubuntu
【8】Test RPI ZERO W(8812AU) + laptop (8812AU) with Release 23.01
【9】Performance test issue based on Release 23.01 image #289
7. 补充资料
Understanding Video Codecs -Why Some Footage is So Hard to Edit-
H.265 (HEVC) vs H.264 (AVC) Compression: Explained!
H264 vs H265 Explained