基于NXP iMX8M Mini处理器测试DPDK
By Toradex秦海
1). 简介
DPDK (Data Plane Development Kit) 软件是一组用户空间库和驱动程序，可加速在所有主要 CPU 架构上运行的网络数据包处理工作负载，以便提升整个网络数据服务的QoS。其最早由 Intel 大约 2010年创建，后由6WIND公司发展为开源社区版本后，极大的拓展了DPDK的影响力，现在DPDI已经是是 Linux 基金会旗下的开源项目之一，其支持涵盖了主流的CPU架构和繁多的NIC驱动以及虚拟化平台支持。
DPDK使用了轮询 (polling) 而不是中断来处理数据包。在收到数据包时，经 DPDK 重载的网卡驱动不会通过中断通知 CPU ，而是直接将数据包存入内存，交付应用层软件通过 DPDK 提供的接口来直接处理，这样节省了大量的 CPU 中断时间和内存拷贝时间。
本文所要测试的 NXP iMX8M Mini ARM SoC平台集成的 FEC NIC 在 DPDK 21.11 LTS发布版本提供了相关支持， 具体的 DPDK ENETFEC Poll Mode 驱动描述以及功能支持情况请见如下 dpdk.org 文档，本文就对其进行简单部署测试。
20. ENETFEC Poll Mode Driver — Data Plane Development Kit 23.03.0 documentation
本文所演示的平台来自于Toradex Verdin iMX8M Mini 嵌入式平台，基于 NXP iMX8M Mini 系列 ARM 处理器，核心为 Cortex-A53，包含一个支持 DPDK PMD 驱动的以太网接口。
2). 硬件准备
a). DUT-A : Verdin iMX8MM ARM 核心版配合 Dahlia Carrier Board 载板，并连接调试串口，后续测试中作为DPDK Packet Generator 设备。
b). DUT-B : Verdin iMX8MM ARM 核心版配合 Verdin Development Board 载板，并连接调试串口，后续测试中作为DPDK Packet Generator 设备。
c). Dahlia 载板 X11千兆网口和 Verdin Development Board X25 千兆网口通过一根网线直通连接。
3). 软件准备
a). Toradex Verdin iMXMM 模块标准 Ycoto Linux BSP 中没有包含 DPDK 相关库和驱动，需要通过 Ycoto Project/Openembedded 编译环境来进行添加编译，生成新的测试 Linux BSP Image。
b). 参考 Ycoto 编译环境来下载配置相关编译环境，使用最新的针对 Ycoto Linux BSP 6/Linux Kernel 5.15.x 版本的 branch
---------------------------------------
$ mkdir ${HOME}/oe-core
$ cd ${HOME}/oe-core
$ repo init -u git://git.toradex.com/toradex-manifest.git -b kirkstone-6.x.y -m tdxref/default.xml
$ repo sync
---------------------------------------
c). 由于默认的 oe-core/layers/meta-freescale/recipes-extended/dpdk/ 提供的 DPDK 19.11版本还未支持 iMX8MM，需要参考 NXP i.MX_Yocto_Project_User's_Guide 文档，下载 NXP imx-6.1.1-1.0.0 branch 代码来获取 DPDK 21.11 版本的 BB 描述文件。
---------------------------------------
$ mkdir ${HOME}/imx-yocto-bsp
$ cd ${HOME}/imx-yocto-bsp
$ repo init -u https://github.com/nxp-imx/imx-manifest -b imx-linux-langdale -m imx-6.1.1-1.0.0.xml
$ repo sync
---------------------------------------
d). 将 NXP imx-6.1.1-1.0.0 branch 包含的 DPDK 21.11 BB 文件复制到 Toradex Ycoto 环境
---------------------------------------
$ cd ${HOME}/oe-core
$ mkdir -p layers/meta-toradex-nxp/recipes-extended/dpdk
$ cp ${HOME}/imx-ycoto-bsp_6.1.1-1.0.0/sources/meta-imx/meta-sdk/recipes-extended/dpdk/dpdk* layers/meta-toradex-nxp/recipes-extended/dpdk/
$ $ ls layers/meta-toradex-nxp/recipes-extended/dpdk/
dpdk_21.11.bb dpdk.inc
---------------------------------------
e). 修改配置文件后编译测试 Ycoto Linux BSP Image 文件
---------------------------------------
### Add below to ${HOME}/oe-core/build/conf/local.conf ###
# add dpdk package
IMAGE_INSTALL:append = " dpdk dpdk-examples dpdk-tools"
# accept freescale license
ACCEPT_FSL_EULA = "1"
### Compile BSP Image ###
$ MACHINE=verdin-imx8mm bitbake tdx-reference-multimedia-image
---------------------------------------
f). 参考这里的说明将编译生成的 Ycoto Linux Image 通过 Toradex Easy Installer 更新到两个测试平台的 Verdin iMX8MM 模块
g). 由于默认 Linux BSP 网口驱动还是 NXP 标准 Linux FEC Kernel驱动，需要通过修改 Linux Kernel Device Tree 来适配为 DPDK PMD 驱动。
./ 参考这里的说明下载 Linux Kernel 源代码， 对应 Branch 为 toradex_5.15-2.1.x-imx
./ 参考如下 Patch 文件修改 Device Tree 源代码，并重新编译生成所需的 Device Tree Binary 文件 imx8mm-verdin-wifi-dev.dtb
---------------------------------------
diff --git a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts
index ef952021832e..b2d60695ecf9 100644
--- a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts
+++ b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts
@@ -16,3 +16,7 @@
"toradex,verdin-imx8mm",
"fsl,imx8mm";
};
+
+&fec1 {
+ compatible = "fsl,imx8mm-fec-uio";
+};
---------------------------------------
./ 将新的 imx8mm-verdin-wifi-dev.dtb 替换两个平台的 Verdin iMX8MM Linux /boot 下默认使用的 Device Tree 文件并重新启动，查看原本的 eth0 接口已经没有 probe 了
---------------------------------------
root@verdin-imx8mm-07276322:~# dmesg |grep eth
[ 0.000000] psci: probing for conduit method from DT.
[ 1.716513] fec-uio 30be0000.ethernet: UIO device full name imx-fec-uio initialized
[ 8.382278] using random self ethernet address
[ 8.387105] using random host ethernet address
root@verdin-imx8mm-07276322:~# ifconfig -a
can0: flags=128
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 10 (UNSPEC)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 78
lo: flags=73
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 188 bytes 13742 (13.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 188 bytes 13742 (13.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
...
---------------------------------------
4). DPDK 示例应用测试
a). 首先配置 Huge Pages，可以通过设置下面 U-boot 环境变量来配置，这样就无需每次启动都配置一次
---------------------------------------
### configure huge paces ###
$ fw_setenv tdxargs ‘default_hugepagesz=2m hugepagesz=2m hugepages=448’
$ reboot
### check huge pages status ###
root@verdin-imx8mm-07276322:~# cat /proc/meminfo |grep -i '^hugepages'
HugePages_Total: 448
HugePages_Free: 448
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
---------------------------------------
b). 示例测试1 ： dpdk-testpmd 示例应用测试
./ dpdk-testpmd 应用配合的参数说明可以参考这里说明
./ DUT-B 通过 dpdk-testpmd 示例应用 Rx 模式作为 Packet Receiver
---------------------------------------
### DUT-B configure to Rx Mode ###
root@verdin-imx8mm-07275765:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=rxonly
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: lsx_pciep:SoC(0xcc000000) not supported
EAL: Scan for (lsx_pciep) bus failed.
EAL: Selected IOVA mode 'PA'
EAL: No available 32768 kB hugepages reported
EAL: No available 64 kB hugepages reported
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set rxonly packet forwarding mode
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p
arameters along with --numa.
testpmd: create a new mbuf pool
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 01:01:01:01:01:01
Checking link statuses...
Done
### start forward mode ###
testpmd> start
rxonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m
ode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
rxonly packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=1
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
---------------------------------------
./ DUT-A 通过 dpdk-testpmd 示例应用 Tx 模式作为 Packet Generator
---------------------------------------
### DUT-A configure to Tx Mode ###
root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=txonly
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: lsx_pciep:SoC(0xcc000000) not supported
EAL: Scan for (lsx_pciep) bus failed.
EAL: Selected IOVA mode 'PA'
EAL: No available 32768 kB hugepages reported
EAL: No available 64 kB hugepages reported
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set txonly packet forwarding mode
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p
arameters along with --numa.
testpmd: create a new mbuf pool
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 01:01:01:01:01:01
Checking link statuses...
Done
### start forward mode ###
testpmd> start
txonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m
ode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
txonly packet forwarding packets/burst=32
packet len=64 - nb packet segments=1
nb forwarding cores=1 - nb forwarding ports=1
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
---------------------------------------
./ 测试一段时间后按顺序分别停止 DUT-A 和 DUT-B，查看统计结果，显示收发一致，工作正常
---------------------------------------
### Tx statistic on DUT-A ###
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 23614688 TX-dropped: 0 TX-total: 23614688
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 23614688 TX-dropped: 0 TX-total: 23614688
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
### Rx statistic on DUT-B ###
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 23614688 RX-dropped: 0 RX-total: 23614688
TX-packets: 0 TX-dropped: 0 TX-total: 0
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 23614688 RX-dropped: 0 RX-total: 23614688
TX-packets: 0 TX-dropped: 0 TX-total: 0
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------
c). 示例测试2 ： dpdk-testpmd Tx Mode 配合 dpdk-l2fwd 示例应用测试
./ dpdk-l2fwd 应用配合的参数说明可以参考这里说明
./ DUT-B 通过 dpdk-l2fwd 配置为 L2 forward 模式
---------------------------------------
root@verdin-imx8mm-07275765:~# /usr/share/dpdk/examples/dpdk-l2fwd -c 0x1 -n 1 --vdev 'net_enetfec' -- -p 0x1 -P
---------------------------------------
// 注意使用 -P 参数来接受全部 MAC Address的包，否则不属于本机的会被自动丢弃
./ DUT-A 如示例 1 通过 dpdk-testpmd 示例应用 Tx 模式作为 Packet Generator 并 start 开始发包测试
---------------------------------------
root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --txd=1 --burst=1 --forward-mode=txonly
...
testpmd> start
...
---------------------------------------
// 注意，使用 --txd 和 --burst 来控制发包的速率，太快则 dpdk-l2fwd 无法及时处理
./ 测试一段时间后停止 DUT-A 发包，查看统计结果，显示收发一致，同时 DUT-B L2 IO 转发也一致，工作正常
---------------------------------------
### Tx statistic on DUT-A ###
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 13839386 TX-dropped: 41509766 TX-total: 55349152
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 13839386 TX-dropped: 83019532 TX-total: 96858918
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
### IO statistic on DUT-B ###
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 13839386
Packets received: 13839386
Packets dropped: 0
Aggregate statistics ===============================
Total packets sent: 13839386
Total packets received: 13839386
Total packets dropped: 0
====================================================
---------------------------------------
d). 示例测试3 ： dpdk-testpmd IO Mode 示例应用测试
./ DUT-B 通过 dpdk-testpmd 配置为 IO forward 模式并 start 开始准备转发从 DUT-A 发送的包
---------------------------------------
root@verdin-imx8mm-07275765:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io
...
testpmd> start
...
---------------------------------------
./ DUT-A 通过 dpdk-testpmd 配置为 IO forward 模式并通过 start tx_first开始进行先发一个包然后再 IO 收包的测试
---------------------------------------
root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io
...
testpmd> start tx_first
...
---------------------------------------
./ 测试一段时间后按顺序分别停止 DUT-A 和 DUT-B，查看统计结果，DUT-A 因为先发一组 Burst 包（ Burst 数值默认是32），因此 Tx 比 Rx 多32；同时 DUT-B IO 转发一致且和 DUT-A Tx 一致，工作正常
---------------------------------------
### IO statistic on DUT-A ###
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 10381092 RX-dropped: 0 RX-total: 10381092
TX-packets: 10381124 TX-dropped: 0 TX-total: 10381124
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 10381092 RX-dropped: 0 RX-total: 10381092
TX-packets: 10381124 TX-dropped: 0 TX-total: 10381124
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
### IO statistic on DUT-B ###
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 10381124 RX-dropped: 0 RX-total: 10381124
TX-packets: 10381124 TX-dropped: 0 TX-total: 10381124
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 10381124 RX-dropped: 0 RX-total: 10381124
TX-packets: 10381124 TX-dropped: 0 TX-total: 10381124
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------
5). 总结
本文基于NXP iMX8M Mini系列处理器简单演示了 DPDK 软件和驱动的部署以及示例应用测试，按照 NXP 官方说明， iMX8M Plus平台 ETH1 应该也支持 DPDK，只是目前在 toradex_5.15-2.0.x-imx branch 下驱动支持还存在一定问题，后续更高版本 branch 可能能够解决。
参考文档
NXP i.MX_Linux_Reference_Manual Chapter 10 Data Plane Development Kit (DPDK)
20. ENETFEC Poll Mode Driver — Data Plane Development Kit 23.03.0 documentation