Macos+VMware Fusion+DPDK环境搭建

---

前言

记录一下，在mac系统中，使用虚拟机vmware fusion+centos7.9，安装dpdk-22.11.2

参考文章：《DPDK-20.11.1版本在Centos8上安装和测试》

---

一、安装系统

镜像地址：https://mirrors.aliyun.com/centos/7.9.2009/isos/x86_64/ 可后退路径，按需下载版本

虚拟机连网参考：《Mac中VMWare Fusion安装CentOS7虚拟机无法上网的问题》

我选择的是CentOS-7-x86_64-Minimal-2009.iso，分配 2核 4G(按需)，设置2个网络适配器，一个NAT模式用作ssh，一个桥接模式用作dpdk（可添加多个）

二、安装DPDK

1.安装依赖

yum update -y

# 库
yum install -y epel-release

# 工具
yum install -y wget git vim net-tools

# dpdk
# 官网：http://core.dpdk.org/download/
wget http://fast.dpdk.org/rel/dpdk-22.11.2.tar.xz

# 依赖
yum install -y meson ninja-build numactl numactl-devel gcc libhugetlbfs pciutils
pip3 install pyelftools

2.编译DPDK

tar xvf dpdk-22.11.2.tar.xz
cd dpdk-stable-22.11.2/
#指定同时编译例子 (https://blog.csdn.net/force_eagle/article/details/110270131)
#更复杂的配置见上面链接
meson -Dexamples=all build
# 编译安装
cd build
ninja install

3.配置内存大页

vim /etc/default/grub

# 在 GRUB_CMDLINE_LINUX 引号内，添加如下内容
transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024

# 重新生成 grub.cfg
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

大页内存每页大小为2M，一共设置1024页面，即2GB，根据自己虚拟机内存分配情况进行设置，建议小于总内存，不然可能溢出
如果内存多的，CPU又支持1GB的大页内存，可以每页设置1GB的大页内存

# 重启后验证
[root@localhost ~]# cat /proc/meminfo |grep -i HugePages
AnonHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

大页内存除了指定动态库,一般要挂载到特定目录使用,命令如下:

mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge

设置系统启动后自动挂载

vim /etc/fstab
#添加以下内容
#如果是1GB 则nodev /mnt/huge hugetlbfs pagesize=1GB 0 0
nodev /mnt/huge hugetlbfs  defaults 0 0

#重启系统
reboot

4.运行测试程序

[root@localhost ~]# cd dpdk-stable-22.11.2/build/examples/
[root@localhost examples]# ./dpdk-helloworld
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
TELEMETRY: No legacy callbacks, legacy socket not created
hello from core 1
hello from core 0

5.加载驱动

VFIO跟着参考文章走了一遍，同样的报错：Not all devices in IOMMU group bound to VFIO or unbound，暂未解决，感兴趣可参考官方文档：http://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio

UIO是DPDK支持的另一种驱动模式

#安装依赖库
yum install -y elfutils-libelf-devel kernel-devel-$(uname -r)
#下载
git clone https://github.com/atsgen/dpdk-kmod.git
# 安装
cd dpdk-kmod/scripts
sh install.sh 
#加载驱动
modprobe igb_uio
modprobe uio

6.绑定网卡

[root@localhost ~]# cd dpdk-stable-22.11.2/usertools/
[root@localhost usertools]# ./dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio *Active*
0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens36 drv=e1000 unused=igb_uio *Active*

ens33是我用来ssh的NAT网卡
ens36是将要用来绑定DPDK的，显示是Active状态，先down掉，第一列为PCI地址
可以绑定多个网卡，在虚拟机设置中配置多个网络适配器即可

# 如果你用了ifconfig ens36 down，后续绑定时显示设备不存在，重启重新加载驱动再用ip命令尝试
ip link set ens36 down

[root@localhost usertools]# ./dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio *Active*
0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens36 drv=e1000 unused=igb_uio 

[root@localhost usertools]# ./dpdk-devbind.py  --bind=igb_uio 0000:02:04.0
[root@localhost usertools]# ./dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' drv=igb_uio unused=e1000

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio *Active*

7.解决报错

再次运行用例发现如下报错：EAL: Error reading from file descriptor 19: Input/output error

[root@localhost examples]# ./dpdk-helloworld
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probe PCI driver: net_e1000_em (8086:100f) device: 0000:02:04.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Error reading from file descriptor 19: Input/output error
hello from core 1
hello from core 0
EAL: Error disabling interrupts for fd 19 (Input/output error)

解决方法参考：《Eal:Error reading from file descriptor 33: Input/output error》

《dpdk EAL: Error reading from file descriptor 23: Input/output error》

修改一行代码后，reboot，然后从5.#安装开始重新走流程

cd dpdk-kmod/scripts
sh install.sh 
modprobe igb_uio
modprobe uio
ip link set ens36 down

cd dpdk-stable-22.11.2/usertools/
./dpdk-devbind.py  --bind=igb_uio 0000:02:04.0

测试examples

[root@localhost examples]#./dpdk-l2fwd -c 0x3  -n 4 -- -q 1 -p 0x1

Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent:                       47
Packets received:                   47
Packets dropped:                     0
Aggregate statistics ===============================
Total packets sent:                 47
Total packets received:             47
Total packets dropped:               0
====================================================
^C

Signal 2 received, preparing to exit...
Closing port 0... Done
Bye...

---