ubuntu-18.04的DPDK-19.11安装过程
DPDK安装过程
- DPDK安装
-
- 安装环境
- 安装前准备（下载、解压、安装依赖包）
- 利用`usertools/dpdk-setup.sh`配置环境
- 利用`usertools/dpdk-devbind.py`绑定网卡设备
- HugePage的配置
- `examples\helloworld`例程的编译运行
DPDK安装
DPDK (数据平面开发套件, Data Plane Development Kit)在ubuntu环境下的安装过程。
安装环境
VirtualBox虚拟机中安装DPDK19.11，之前曾装双系统，但只有一个网卡可以连接网络，DPDK绑定网卡后会接管网卡接到的所有数据帧，遂放弃真机，选择虚拟机进行环境配置。
在VirtualBox中设置两个NAT网卡，
- 操作系统：Ubuntu 18.04.3 LTS 64位
- 网卡型号：Intel Pro/1000 MT 桌面(82540EM) * 2
安装前准备（下载、解压、安装依赖包）
浏览 Home-DPDK官网 ，下载 DPDK 19.11.0 (LTS)压缩包，放到/home/user/目录下。根据Documentation里的Quick Start Guide，在Terminal里敲击以下命令：
- 解压源压缩包
tar xf dpdk.tar.gz
cd dpdk
- libpcap依赖安装
sudo apt-get install libpcap-dev
- Enable pcap (libpcap headers are required).
make config T=x86_64-native-linuxapp-gcc
sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config
- 安装numa依赖（numa.h库）
apt-get install libnuma-dev
- 编译
make
利用usertools/dpdk-setup.sh配置环境
- 利用官方已经写好的bash脚本，跳出选项界面
cd usertools
./dpdk-setup.sh
- 选择自己对应的DPDK环境。（我选择的是 [38] x86_64-native-linuxapp-gcc ）
------------------------------------------------------------------------------
----------------------------------------------------------
Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] arm64-armada-linuxapp-gcc
[2] arm64-armada-linux-gcc
[3] arm64-armv8a-linuxapp-clang
[4] arm64-armv8a-linuxapp-gcc
[5] arm64-armv8a-linux-clang
[6] arm64-armv8a-linux-gcc
[7] arm64-bluefield-linuxapp-gcc
[8] arm64-bluefield-linux-gcc
[9] arm64-dpaa-linuxapp-gcc
[10] arm64-dpaa-linux-gcc
[11] arm64-emag-linuxapp-gcc
[12] arm64-emag-linux-gcc
[13] arm64-n1sdp-linuxapp-gcc
[14] arm64-n1sdp-linux-gcc
[15] arm64-octeontx2-linuxapp-gcc
[16] arm64-octeontx2-linux-gcc
[17] arm64-stingray-linuxapp-gcc
[18] arm64-stingray-linux-gcc
[19] arm64-thunderx2-linuxapp-gcc
[20] arm64-thunderx2-linux-gcc
[21] arm64-thunderx-linuxapp-gcc
[22] arm64-thunderx-linux-gcc
[23] arm64-xgene1-linuxapp-gcc
[24] arm64-xgene1-linux-gcc
[25] arm-armv7a-linuxapp-gcc
[26] arm-armv7a-linux-gcc
[27] i686-native-linuxapp-gcc
[28] i686-native-linuxapp-icc
[29] i686-native-linux-gcc
[30] i686-native-linux-icc
[31] ppc_64-power8-linuxapp-gcc
[32] ppc_64-power8-linux-gcc
[33] x86_64-native-bsdapp-clang
[34] x86_64-native-bsdapp-gcc
[35] x86_64-native-freebsd-clang
[36] x86_64-native-freebsd-gcc
[37] x86_64-native-linuxapp-clang
[38] x86_64-native-linuxapp-gcc
[39] x86_64-native-linuxapp-icc
[40] x86_64-native-linux-clang
[41] x86_64-native-linux-gcc
[42] x86_64-native-linux-icc
[43] x86_x32-native-linuxapp-gcc
[44] x86_x32-native-linux-gcc
----------------------------------------------------------
Option: 38
- 设置linux的环境
在此处可以进行IGB UIO\VFIO\KNI 模块的加载、HugePage的映射设置、网卡绑定等功能，如Terminal的菜单所示，此处不一一列举，需要按照自己的要求进行选择。将在之后再介绍网卡绑定
----------------------------------------------------------
Step 2: Setup linux environment
----------------------------------------------------------
[45] Insert IGB UIO module
[46] Insert VFIO module
[47] Insert KNI module
[48] Setup hugepage mappings for non-NUMA systems
[49] Setup hugepage mappings for NUMA systems
[50] Display current Ethernet/Baseband/Crypto device settings
[51] Bind Ethernet/Baseband/Crypto device to IGB UIO module
[52] Bind Ethernet/Baseband/Crypto device to VFIO module
[53] Setup VFIO permissions
----------------------------------------------------------
Option:
利用usertools/dpdk-devbind.py绑定网卡设备
- 首先利用ifconfig查看当前的网卡配置信息，如果没有安装net-tools，需要
sudo apt-get install net-tools
如安装完net-tools，可以直接敲击ifconfig查看当前的网卡配置
ifconfig
结果如下图所示，enp0s8的包数量通过较少。
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::63c3:f3fe:13df:8486 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:60:56:7d txqueuelen 1000 (以太网)
RX packets 596354 bytes 656782296 (656.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 183059 bytes 21797442 (21.7 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.3.15 netmask 255.255.255.0 broadcast 10.0.3.255
inet6 fe80::6096:f17d:d1ca:a488 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:ea:05:4f txqueuelen 1000 (以太网)
RX packets 468 bytes 43340 (43.3 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 632 bytes 56957 (56.9 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (本地环回)
RX packets 12923 bytes 1369263 (1.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12923 bytes 1369263 (1.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
- 利用官方工具
usertools/dpdk-devbind.py查看当前的网卡状态
./dpdk-devbind.py --status
可以看到当前我的两个网卡enp0s3和enp0s8都处于活动状态，且目前仍采用linux内核驱动。
Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller 100e' if=enp0s3 drv=e1000 unused= *Active*
0000:00:08.0 '82540EM Gigabit Ethernet Controller 100e' if=enp0s8 drv=e1000 unused= *Active*
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
- ifconfig将enp0s8网卡解绑
ifconfig enp0s8 down
此时再利用ifconfig查询网卡配置信息可发现，enp0s8已经没了，利用官方的python文件usertools/dpdk-devbind.py查看网卡状态信息，enp0s8也不再处于活跃状态，可以进行绑定。
- 用官方工具
usertools/dpdk-devbind.py绑定设备
加载uio_pci_generic功能：
sudo modprobe uio_pci_generic
将enp0s8，也就是00:08.0绑定到uio_pci_generic下：
./dpdk-devbind.py --bind=uio_pci_generic 00:08.0
或者
./dpdk-devbind.py --bind=uio_pci_generic enp0s8
此时再通过敲击命令./dpdk-devbind.py --status查看网卡设备状态：
Network devices using DPDK-compatible driver
============================================
0000:00:08.0 '82540EM Gigabit Ethernet Controller 100e' drv=uio_pci_generic unused=e1000
Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller 100e' if=enp0s3 drv=e1000
unused=uio_pci_generic *Active*
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
可见已经绑定完毕。
那如何解绑网卡enp0s8呢？根据网卡的设备状态，drv表示驱动名，仍利用dpdk-devbind.py进行原模式重新绑定即可。
#首先解除绑定该设备
#./dpdk-devbind.py -u
./dpdk-devbind.py -b e1000 0000:00:08.0
# 执行完毕后我的Ubuntu-GNOME桌面就检测到了网络连接信息，打开即可
HugePage的配置
虚拟机属于non-numa system，这里我用HugePage的配置采用官方工具usertools/dpdk-setup.sh进行配置。
----------------------------------------------------------
Step 2: Setup linux environment
----------------------------------------------------------
[45] Insert IGB UIO module
[46] Insert VFIO module
[47] Insert KNI module
[48] Setup hugepage mappings for non-NUMA systems
[49] Setup hugepage mappings for NUMA systems
[50] Display current Ethernet/Baseband/Crypto device settings
[51] Bind Ethernet/Baseband/Crypto device to IGB UIO module
[52] Bind Ethernet/Baseband/Crypto device to VFIO module
[53] Setup VFIO permissions
----------------------------------------------------------
Option:48 #（my input）
Removing currently reserved hugepages #由于之前配置过，所以需要删除当前保存的hugepage信息
Unmounting /mnt/huge and removing directory
Input the number of 2048kB hugepages
Example: to have 128MB of hugepages available in a 2MB huge page system,
enter '64' to reserve 64 * 2MB pages
Number of pages: 64 #（my input）
Reserving hugepages
Creating /mnt/huge and mounting as hugetlbfs
Press enter to continue ...
examples\helloworld例程的编译运行
目前除了例程所需要的$RTE_SDK和$RTE_TARGET，基本都已配置好。
cd ..
cd examples/helloworld
export RTE_SDK=/home/lianpeng/dpdk-19.11（此处为DPDK的解压目录，改成自己的）
export RTE_TARGET=x86_64-native-linuxapp-gcc（改成自己的DPDK环境配置）
make
其目录下的Makefile运行并输出以下log，编译成功：
CC main.o
LD helloworld
INSTALL-APP helloworld
INSTALL-MAP helloworld.map
编译成功的可执行文件已放在./build目录下：
cd build
./helloworld
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
# 该错误，网上的说法是 VirtualBox的网卡设置采用了NAT模式
# 如果改成桥接可能就不会出现该问题
EAL: probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:100e net_e1000_em
hello from core 1
hello from core 2
hello from core 3
hello from core 0
基本上就安装完成了。