【DPDK】DPDK-20.11.3在CentOS8.4上编译运行
1、安装前提条件
# uname -a
Kernel version >= 2.6.34
# ldd --version
glibc >=2.7
2、编译安装DPDK
2.1、编译工具安装meson和ninja
pip3 install meson ninja
2.2、numa库安装
yum install -y numactl numactl-devel
2.3、gcc工具套件
dnf install -y gcc-toolset-9
2.4、编译DPDK
#源码下载地址 http://core.dpdk.org/download/
tar xJf dpdk-20.11.3.tar.xz
cd dpdk-stable-20.11.3/
#指定同时编译例子 (https://blog.csdn.net/force_eagle/article/details/110270131)
#更复杂的配置见上面链接
meson -Dexamples=all build
# 编译安装
cd build
ninja install
编译（部分）如下:
Message:
=================
Libraries Enabled
=================
libs:
kvargs, telemetry, eal, ring, rcu, mempool, mbuf, net,
meter, ethdev, pci, cmdline, metrics, hash, timer, acl,
bbdev, bitratestats, cfgfile, compressdev, cryptodev, distributor, efd, eventdev,
gro, gso, ip_frag, jobstats, kni, latencystats, lpm, member,
power, pdump, rawdev, regexdev, rib, reorder, sched, security,
stack, vhost, ipsec, fib, port, table, pipeline, flow_classify,
bpf, graph, node,
Message:
===============
Drivers Enabled
===============
common:
cpt, dpaax, iavf, octeontx, octeontx2, sfc_efx, qat,
bus:
dpaa, fslmc, ifpga, pci, vdev, vmbus,
mempool:
bucket, dpaa, dpaa2, octeontx, octeontx2, ring, stack,
net:
af_packet, ark, atlantic, avp, axgbe, bond, bnx2x, bnxt,
cxgbe, dpaa, dpaa2, e1000, ena, enetc, enic, failsafe,
fm10k, i40e, hinic, hns3, iavf, ice, igc, ixgbe,
kni, liquidio, memif, netvsc, nfp, null, octeontx, octeontx2,
pfe, qede, ring, sfc, softnic, tap, thunderx, txgbe,
vdev_netvsc, vhost, virtio, vmxnet3,
raw:
dpaa2_cmdif, dpaa2_qdma, ioat, ntb, octeontx2_dma, octeontx2_ep, skeleton,
crypto:
bcmfs, caam_jr, dpaa_sec, dpaa2_sec, nitrox, null, octeontx, octeontx2,
scheduler, virtio,
compress:
octeontx, zlib,
regex:
octeontx2,
vdpa:
ifc,
event:
dlb, dlb2, dpaa, dpaa2, octeontx2, opdl, skeleton, sw,
dsw, octeontx,
baseband:
null, turbo_sw, fpga_lte_fec, fpga_5gnr_fec, acc100,
Message:
=================
Content Skipped
=================
libs:
drivers:
common/mvep: missing dependency, "libmusdk"
common/mlx5: missing dependency, "mlx5"
crypto/qat: missing dependency, libcrypto
net/af_xdp: missing dependency, "libbpf"
net/ipn3ke: missing dependency, "libfdt"
net/mlx4: missing dependency, "mlx4"
net/mlx5: missing internal dependency, "common_mlx5"
net/mvneta: missing dependency, "libmusdk"
net/mvpp2: missing dependency, "libmusdk"
net/nfb: missing dependency, "libnfb"
net/pcap: missing dependency, "libpcap"
net/szedata2: missing dependency, "libsze2"
raw/ifpga: missing dependency, "libfdt"
crypto/aesni_gcm: missing dependency, "libIPSec_MB"
crypto/aesni_mb: missing dependency, "libIPSec_MB"
crypto/armv8: missing dependency, "libAArch64crypto"
crypto/ccp: missing dependency, "libcrypto"
crypto/kasumi: missing dependency, "libIPSec_MB"
crypto/mvsam: missing dependency, "libmusdk"
crypto/openssl: missing dependency, "libcrypto"
crypto/snow3g: missing dependency, "libIPSec_MB"
crypto/zuc: missing dependency, "libIPSec_MB"
compress/isal: missing dependency, "libisal"
regex/mlx5: missing internal dependency, "common_mlx5"
vdpa/mlx5: missing internal dependency, "common_mlx5"
Build targets in project: 1043
DPDK 20.11.3
User defined options
examples: all
Found ninja-1.10.2.git.kitware.jobserver-1 at /usr/local/bin/ninja
安装（部分）如下:
Installing /home/dpdk-stable-20.11.3/drivers/event/dlb/rte_pmd_dlb.h to /usr/local/include
Installing /home/dpdk-stable-20.11.3/drivers/event/dlb2/rte_pmd_dlb2.h to /usr/local/include
Installing /home/dpdk-stable-20.11.3/drivers/baseband/fpga_5gnr_fec/rte_pmd_fpga_5gnr_fec.h to /usr/local/include
Installing /home/dpdk-stable-20.11.3/drivers/baseband/acc100/rte_acc100_cfg.h to /usr/local/include
Installing /home/dpdk-stable-20.11.3/usertools/dpdk-devbind.py to /usr/local/bin
Installing /home/dpdk-stable-20.11.3/usertools/dpdk-pmdinfo.py to /usr/local/bin
Installing /home/dpdk-stable-20.11.3/usertools/dpdk-telemetry.py to /usr/local/bin
Installing /home/dpdk-stable-20.11.3/usertools/dpdk-hugepages.py to /usr/local/bin
Installing /home/dpdk-stable-20.11.3/build/rte_build_config.h to /usr/local/include
Installing /home/dpdk-stable-20.11.3/build/meson-private/libdpdk-libs.pc to /usr/local/lib64/pkgconfig
Installing /home/dpdk-stable-20.11.3/build/meson-private/libdpdk.pc to /usr/local/lib64/pkgconfig
Running custom install script '/bin/sh /home/dpdk-stable-20.11.3/config/../buildtools/symlink-drivers-solibs.sh lib64 dpdk/pmds-21.0
3、相关配置
3.1、配置大页内存
大页内存，需要libhugetlbfs库的支持,如果设置2MB的据说不需要,我还是安装真实环境来操作:
先用命令yum list installed | grep huge，查看是否安装有巨页内存库libhugetlbfs。
如果没有安装巨页内存所需的库：
yum install libhugetlbfs
安装之后，修改grub配置文件，配置巨页内存
vim /etc/default/grub
#在GRUB_CMDLINE_LINUX="...所在行末尾添加 transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024 如下所示
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024 "
大页内存每页大小为2M，一共设置1024页面，即2GB，根据自己主机内存情况进行设置，如果内存多的，CPU又支持1GB的大页内存，可以每页设置1GB的大页内存。
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=1 "
重新生成 grub.cfg
[root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
[root@localhost ~]# reboot
验证大页内存
[root@localhost ~]# cat /proc/meminfo |grep -i HugePages
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
HugePages_Total: 1024
HugePages_Free: 1024
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
配置了1024个大页内存页面,每个大小为2048KB即2MB,一共大页内存大小为2GB。
大页内存挂载
大页内存除了指定动态库,一般要挂载到特定目录使用,命令如下:
[root@localhost ~]# mkdir -p /mnt/huge
[root@localhost ~]# mount -t hugetlbfs nodev /mnt/huge
设置系统启动后自动挂载:
vim /etc/fstab
.
.
.
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/cl-root / xfs defaults 0 0
UUID=6c792e6c-71dc-4235-a530-dcc7b181fd6f /boot xfs defaults 0 0
/dev/mapper/cl-swap none swap defaults 0 0
#添加以下内容
#如果是1GB 则nodev /mnt/huge hugetlbfs pagesize=1GB 0 0
nodev /mnt/huge hugetlbfs defaults 0 0
#重启系统
reboot
3.2、修改内存锁限制
[root@localhost ~]# vim /etc/security/limits.conf
# -
#
#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4
#增加如下两行
* soft memlock 4194304
* hard memlock 4194304
# End of file
3.3、开启vfio支持
- VFIO是一个可以安全地把设备I/O、中断、DMA等暴露到用户空间（userspace），从而可以在用户空间完成设备驱动的框架。用户空间直接设备访问，虚拟机设备分配可以获得更高的IO性能。
- 依赖于IOMMU. vfio-pci
- 相比于UIO，VFIO更为强健和安全
修改grub配置，开启vfio支持，检查GRUB配置:
#内核需要配置支持Intel® VT-x、VT-d,内核通过如下命令查看:
[root@localhost ~]# cat /proc/cmdline | grep iommu=pt
[root@localhost ~]# cat /proc/cmdline | grep intel_iommu=on
没发现则需要添加：
vim /etc/default/grub
#在下面行中添加:iommu=pt intel_iommu=on
GRUB_CMDLINE_LINUX=
#现在变成了:
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet iommu=pt intel_iommu=on transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024 "
# 更新配置和重启
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot
验证下:
[root@localhost ~]# cat /proc/cmdline | grep intel_iommu=on
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-240.el8.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet iommu=pt intel_iommu=on transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024
[root@localhost ~]# cat /proc/cmdline | grep iommu=pt
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-240.el8.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet iommu=pt intel_iommu=on transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024
3.4、加载网卡驱动
[root@localhost ~]# modprobe vfio
[root@localhost ~]# modprobe vfio-pci
卸载网卡驱动
[root@localhost ~]# rmmod vfio-pci
查看
[root@localhost ~]# lsmod | grep vfio_pci
vfio_pci 61440 0
vfio_virqfd 16384 1 vfio_pci
vfio 36864 2 vfio_iommu_type1,vfio_pci
irqbypass 16384 2 vfio_pci,kvm
注意虚拟机环境需要noniommu_mode，配置enable_unsafe_noiommu_mode选项
[root@localhost ~]# echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
#允许在没有IOMMU的情况下使用VFIO。这种模式适用于与基于UIO的驱动程序相同的所有意图和目的，并具有所有相同的优点与限制。
查看并记录网卡设备的bus-info值
[root@localhost ~]# ethtool -i ens37
driver: e1000
version: 7.3.21-k8-NAPI
firmware-version:
expansion-rom-version:
bus-info: 0000:02:05.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
[root@localhost ~]# ethtool -i ens38
driver: e1000
version: 7.3.21-k8-NAPI
firmware-version:
expansion-rom-version:
bus-info: 0000:02:06.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
绑定网卡，到DPDK目录下/home/dpdk-stable-20.11.3/usertools，使用脚本./dpdk-devbind.py绑定网卡
[root@localhost ~]# ifconfig ens37 down
[root@localhost ~]# ifconfig ens38 down
[root@localhost ~]# ./dpdk-devbind.py -b vfio-pci 0000:02:05.0
[root@localhost ~]# ./dpdk-devbind.py -b vfio-pci 0000:02:06.0
查看网卡状态
[root@localhost usertools]# ./dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' drv=vfio-pci unused=e1000
0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' drv=vfio-pci unused=e1000
Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=vfio-pci *Active*
No 'Baseband' devices detected
==============================
4、运行实例
4.1、执行dpdk-helloword实例
[root@localhost examples]# ./dpdk-helloworld
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Invalid NUMA socket, default to 0
EAL: Invalid NUMA socket, default to 0
EAL: using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(4)
EAL: Probe PCI driver: net_e1000_em (8086:100f) device: 0000:02:05.0 (socket 0)
EAL: Invalid NUMA socket, default to 0
EAL: Ignore mapping IO port bar(4)
EAL: Probe PCI driver: net_e1000_em (8086:100f) device: 0000:02:06.0 (socket 0)
EAL: No legacy callbacks, legacy socket not created
hello from core 1
hello from core 2
hello from core 3
hello from core 0
4.2、执行dpdk-l2fwd实例
[root@localhost examples]# ./dpdk-l2fwd -c 0x3 -n 4 -- -q 1 -p 0x3
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Invalid NUMA socket, default to 0
EAL: Invalid NUMA socket, default to 0
EAL: using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(4)
EAL: Probe PCI driver: net_e1000_em (8086:100f) device: 0000:02:05.0 (socket 0)
EAL: Invalid NUMA socket, default to 0
EAL: Ignore mapping IO port bar(4)
EAL: Probe PCI driver: net_e1000_em (8086:100f) device: 0000:02:06.0 (socket 0)
EAL: No legacy callbacks, legacy socket not created
MAC updating enabled
Lcore 0: RX port 0 TX port 1
Lcore 1: RX port 1 TX port 0
Initializing port 0... EAL: Error enabling MSI-X interrupts for fd 21
done:
Port 0, MAC address: 00:0C:29:BE:75:B2
Initializing port 1... EAL: Error enabling MSI-X interrupts for fd 25
done:
Port 1, MAC address: 00:0C:29:BE:75:BC
Checking link statusdone
Port 0 Link up at 1 Gbps FDX Autoneg
Port 1 Link up at 1 Gbps FDX Autoneg
L2FWD: entering main loop on lcore 1
L2FWD: -- lcoreid=1 portid=1
L2FWD: entering main loop on lcore 0
L2FWD: -- lcoreid=0 portid=0
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 123
Packets received: 110
Packets dropped: 0
Statistics for port 1 ------------------------------
Packets sent: 110
Packets received: 123
Packets dropped: 0
Aggregate statistics ===============================
Total packets sent: 233
Total packets received: 233
Total packets dropped: 0
====================================================
表示，分配给2个逻辑内核，每个内核分别有1个收发队列，而一共分配了2个端口。注意这里面需要两个网卡,队列相互转发。
Lcore 0: RX port 0 TX port 1
Lcore 1: RX port 1 TX port 0
示意图如下,即为二层转发,-q为一个内核可以处理几个队列,-p后面的0x3 对应两个网卡的掩码。
4.3、执行dpdk-testpmd实例
[root@localhost app]# ./dpdk-testpmd -w 0000:02:02.0 -c 0x3 -- --total-num-mbufs=2048 -ai
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
Option -w, --pci-whitelist is deprecated, use -a, --allow option instead
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
Auto-start selected
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=2048, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
Start automatic packet forwarding
io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support enabled, MP allocation mode: native
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=0
testpmd> start
Packet forwarding already started
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 249 RX-dropped: 0 RX-total: 249
TX-packets: 249 TX-dropped: 0 TX-total: 249 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Done.
testpmd>
参考命令
#查看NUMA节点对应的CPU
[root@localhost ~]# lscpu | grep NUMA
NUMA node(s): 2
NUMA node0 CPU(s): 0-9,20-29
NUMA node1 CPU(s): 10-19,30-39
#查看巨页内存分配是否成功
[root@localhost ~]# cat /proc/meminfo |grep -i HugePages
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
HugePages_Total: 1024
HugePages_Free: 1024
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
#查看网卡对应的bus-info值
[root@localhost ~]# ethtool -i ens37 | grep bus-info
bus-info: 0000:02:05.0
#查看网卡对应的传输速率(千兆or万兆)
[root@localhost ~]# ethtool ens33 | grep Speed
Speed: 1000Mb/s
[root@localhost ~]# ethtool en5s0 | grep Speed
Speed: 10000Mb/s
#查看网卡对应的NUMA节点
[root@localhost ~]# ethtool -i ens37 | grep bus-info
bus-info: 0000:02:05.0
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:02\:05.0/numa_node
0
如果是网卡ens37用来收包，可以得到对应的NUMA节点为0，然后根据NUMA对应的CPU节点可以使用CPU 0-9,20-29来绑定dpdk的收包线程，这样可以避免跨NUMA节点处理，提高程序性能。（其中CPU0预留不使用）
如有侵权请告知删除~
参考网站
- DPDK-20.11.1版本在Centos8上安装和测试
- 云网络性能测试流程
- GitHub - FDio/tldk at dev-next-socket