X710-DA4有四个10GE口
两台DELL 730(支持PCIE3.0)各插一张X710-DA4网卡
一台用于发包，一台用于l3fwd
ubuntu-1(pktgen-dpdk,ubuntu17.10)
68:05:CA:32:02:F0
68:05:CA:32:02:F1
68:05:CA:32:02:F2
68:05:CA:32:02:F3
ubuntu-2(l3fwd dpdk17.11,ubuntu17.10)
68:05:CA:32:03:18
68:05:CA:32:03:19
68:05:CA:32:03:1A
68:05:CA:32:03:1B
端口对连
68:05:CA:32:02:F0----68:05:CA:32:03:18
68:05:CA:32:02:F1----68:05:CA:32:03:19
68:05:CA:32:02:F2----68:05:CA:32:03:1A
68:05:CA:32:02:F3----68:05:CA:32:03:1B
驱动与固件版本
root@ubuntu-2:~# ethtool -i enp5s0f0
driver: i40e
version: 2.4.6
firmware-version: 6.01 0x80003494 1.1747.0
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
查看网卡所在的cpu socket号((或者用 lstopo-no-graphics，见文章最后))
root@ubuntu-2:~# cat /sys/class/net/enp5s0f0/device/numa_node
0
root@ubuntu-2:~# cat /sys/class/net/enp5s0f1/device/numa_node
0
root@ubuntu-2:~# cat /sys/class/net/enp5s0f2/device/numa_node
0
root@ubuntu-2:~# cat /sys/class/net/enp5s0f3/device/numa_node
0
查看cpu core和lcpu与cpu socket的对应关系
root@ubuntu-2:~# cat start-l3fwd.sh
export DPDK_DIR=/root/dpdk/dpdk-17.11
export DPDK_TARGET=x86_64-native-linuxapp-gcc
export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
mkdir -p /dev/hugepages
mount -t hugetlbfs hugetlbfs /dev/hugepages
modprobe uio
insmod $DPDK_BUILD/kmod/igb_uio.ko
export RTE_SDK=$DPDK_DIR
export RTE_TARGET=$DPDK_TARGET
cd /root/dpdk/
####X710-DA4(将lcpu分配在网卡所在的socket为高性能，否则为低性能,以下用两个l3fwd分别进行了对比测试,先用lcpu=3,5,6,9,15,17,19,21测试低性能，再用lcpu=2,4,6,8,14,16,18,20测试高性能)
$DPDK_DIR/usertools/dpdk-devbind.py -b igb_uio 0000:05:00.0 0000:05:00.1 0000:05:00.2 0000:05:00.3
./l3fwd -l 3,5,7,9,15,17,19,21 -n 4 --proc-type auto --socket-mem 2048,2048 --huge-dir /dev/hugepages -- -p 0xf -L --config="(0,0,3),(0,1,15),(1,0,5),(1,1,17),(2,0,7),(2,1,19),(3,0,9),(3,1,21)"
#./l3fwd -l 2,4,6,8,14,16,18,20 -n 4 --proc-type auto --socket-mem 2048,2048 --huge-dir /dev/hugepages -- -p 0xf -L --config="(0,0,2),(0,1,14),(1,0,4),(1,1,16),(2,0,6),(2,1,18),(3,0,8),(3,1,20)"
执行l3fwd
注意路由及目标MAC
root@ubuntu-2:~# sh start-l3fwd.sh
EAL: Detected 24 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
PMD: Global register is changed during enable FDIR flexible payload
PMD: Global register is changed during support QinQ parser
PMD: Global register is changed during configure hash input set
PMD: Global register is changed during configure fdir mask
PMD: Global register is changed during configure hash mask
PMD: Global register is changed during support QinQ cloud filter
PMD: Global register is changed during support TPID configuration
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
PMD: Global register is changed during enable FDIR flexible payload
PMD: Global register is changed during support QinQ parser
PMD: Global register is changed during configure hash input set
PMD: Global register is changed during configure fdir mask
PMD: Global register is changed during configure hash mask
PMD: Global register is changed during support QinQ cloud filter
PMD: Global register is changed during support TPID configuration
EAL: PCI device 0000:05:00.2 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
PMD: Global register is changed during enable FDIR flexible payload
PMD: Global register is changed during support QinQ parser
PMD: Global register is changed during configure hash input set
PMD: Global register is changed during configure fdir mask
PMD: Global register is changed during configure hash mask
PMD: Global register is changed during support QinQ cloud filter
PMD: Global register is changed during support TPID configuration
EAL: PCI device 0000:05:00.3 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
PMD: Global register is changed during enable FDIR flexible payload
PMD: Global register is changed during support QinQ parser
PMD: Global register is changed during configure hash input set
PMD: Global register is changed during configure fdir mask
PMD: Global register is changed during configure hash mask
PMD: Global register is changed during support QinQ cloud filter
PMD: Global register is changed during support TPID configuration
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:07:00.1 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:84:00.0 on NUMA socket 1
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:84:00.1 on NUMA socket 1
EAL: probe driver: 8086:10fb net_ixgbe
L3FWD: Longest-prefix match selected
Initializing port 0 ... Creating queues: nb_rxq=2 nb_txq=8...Address:68:05:CA:32:03:18, Destination:02:00:00:00:00:00, Allocated mbuf pool on socket 1
LPM: Adding route 0x01010100 / 24 (0)
LPM: Adding route 0x02010100 / 24 (1)
LPM: Adding route 0x03010100 / 24 (2)
LPM: Adding route 0x04010100 / 24 (3)
LPM: Adding route IPV6 / 48 (0)
LPM: Adding route IPV6 / 48 (1)
LPM: Adding route IPV6 / 48 (2)
LPM: Adding route IPV6 / 48 (3)
txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1
Initializing port 1 ... Creating queues: nb_rxq=2 nb_txq=8...Address:68:05:CA:32:03:19, Destination:02:00:00:00:00:01, txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1
Initializing port 2 ... Creating queues: nb_rxq=2 nb_txq=8... Address:68:05:CA:32:03:1A, Destination:02:00:00:00:00:02, txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1
Initializing port 3 ... Creating queues: nb_rxq=2 nb_txq=8...Address:68:05:CA:32:03:1B, Destination:02:00:00:00:00:03, txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1
Initializing rx queues on lcore 3 ... rxq=0,0,1
Initializing rx queues on lcore 5 ... rxq=1,0,1
Initializing rx queues on lcore 7 ... rxq=2,0,1
Initializing rx queues on lcore 9 ... rxq=3,0,1
Initializing rx queues on lcore 15 ... rxq=0,1,1
Initializing rx queues on lcore 17 ... rxq=1,1,1
Initializing rx queues on lcore 19 ... rxq=2,1,1
Initializing rx queues on lcore 21 ... rxq=3,1,1
Checking link statusdone
Port0 Link Up. Speed 10000 Mbps -full-duplex
Port1 Link Up. Speed 10000 Mbps -full-duplex
Port2 Link Up. Speed 10000 Mbps -full-duplex
Port3 Link Up. Speed 10000 Mbps -full-duplex
L3FWD: entering main loop on lcore 5
L3FWD: -- lcoreid=5 portid=1 rxqueueid=0
L3FWD: entering main loop on lcore 7
L3FWD: -- lcoreid=7 portid=2 rxqueueid=0
L3FWD: entering main loop on lcore 15
L3FWD: entering main loop on lcore 21
L3FWD: -- lcoreid=21 portid=3 rxqueueid=1
L3FWD: entering main loop on lcore 9
L3FWD: -- lcoreid=9 portid=3 rxqueueid=0
L3FWD: entering main loop on lcore 19
L3FWD: -- lcoreid=19 portid=2 rxqueueid=1
L3FWD: -- lcoreid=15 portid=0 rxqueueid=1
L3FWD: entering main loop on lcore 17
L3FWD: -- lcoreid=17 portid=1 rxqueueid=1
L3FWD: entering main loop on lcore 3
L3FWD: -- lcoreid=3 portid=0 rxqueueid=0
说明:
LPM: Adding route 0x01010100 / 24 (0)
LPM: Adding route 0x02010100 / 24 (1)
LPM: Adding route 0x03010100 / 24 (2)
LPM: Adding route 0x04010100 / 24 (3)
使用的LPM最长匹配模式
数据包经端口0发送出去,可以到达1.1.1.0/24网段,目标mac为02:00:00:00:00:00
数据包经端口1发送出去,可以到达2.1.1.0/24网段,目标mac为02:00:00:00:00:01
数据包经端口2发送出去,可以到达3.1.1.0/24网段,目标mac为02:00:00:00:00:02
数据包经端口3发送出去,可以到达4.1.1.0/24网段,目标mac为02:00:00:00:00:03
发包方
root@ubuntu-1:~# cat start-pktgen-dpdk-x710.sh
export DPDK_DIR=/root/dpdk/dpdk-17.11
export DPDK_TARGET=x86_64-native-linuxapp-gcc
export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
mkdir -p /dev/hugepages
mount -t hugetlbfs hugetlbfs /dev/hugepages
modprobe uio
insmod $DPDK_BUILD/kmod/igb_uio.ko
export RTE_SDK=$DPDK_DIR
export RTE_TARGET=$DPDK_TARGET
cd /root/pktgen-3.4.9/
$DPDK_DIR/usertools/dpdk-devbind.py -b igb_uio 0000:05:00.0 0000:05:00.1 0000:05:00.2 0000:05:00.3
./pktgen -l 0,2-9 -n 4 --proc-type auto --socket-mem 2048,2048 --huge-dir /dev/hugepages -- -P -T -m '[2:3].0,[4:5].1,[6:7].2,[8:9].3'
准备发包
root@ubuntu-1:~# sh start-pktgen-dpdk-x710.sh
发包配置
set 0 src ip 1.1.1.241/24
set 0 src mac 02:00:00:00:00:00
set 0 dst ip 2.1.1.242
set 0 dst mac 68:05:CA:32:03:18
set 1 src ip 2.1.1.242/24
set 1 src mac 02:00:00:00:00:01
set 1 dst ip 1.1.1.241
set 1 dst mac 68:05:CA:32:03:19
set 2 src ip 3.1.1.241/24
set 2 src mac 02:00:00:00:00:02
set 2 dst ip 4.1.1.242
set 2 dst mac 68:05:CA:32:03:1A
set 3 src ip 4.1.1.242/24
set 3 src mac 02:00:00:00:00:03
set 3 dst ip 3.1.1.241
set 3 dst mac 68:05:CA:32:03:1B
#启动端口发包(全双向)
start 0
start 1
start 2
start 3
路由表的设计参见intel
https://software.intel.com/en-us/articles/introduction-to-the-dpdk-sample-applications
低性能(lcpu=3,5,7,9,15,17,19,21)----10.55Mpps
高性能(lcpu=2,4,6,8,14,16,18,20)----37.3Mpps
结论:
l3fwd转发，lcpu的绑定一定要在网卡所在的socket上,否则会导致低性能
用lstopo可以查看那个pci设备和那个scoket直连
apt-get install hwloc
root@ubuntu-1:~# lstopo-no-graphics
Machine (63GB total)
NUMANode L#0 (P#0 31GB)
Package L#0 + L3 L#0 (15MB)
L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#12)
L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
PU L#2 (P#2)
PU L#3 (P#14)
L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
PU L#4 (P#4)
PU L#5 (P#16)
L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
PU L#6 (P#6)
PU L#7 (P#18)
L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
PU L#8 (P#8)
PU L#9 (P#20)
L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
PU L#10 (P#10)
PU L#11 (P#22)
HostBridge L#0
PCIBridge
PCI 1000:005d
Block(Disk) L#0 "sda"
Block(Disk) L#1 "sdb"
PCIBridge
PCI 8086:1572
Net L#2 "enp5s0f0"
PCI 8086:1572
Net L#3 "enp5s0f1"
PCI 8086:1572
Net L#4 "enp5s0f2"
PCI 8086:1572
Net L#5 "enp5s0f3"
PCIBridge
PCI 14e4:168a
Net L#6 "eno1"
PCI 14e4:168a
Net L#7 "eno2"
PCI 14e4:168a
Net L#8 "eno3"
PCI 14e4:168a
Net L#9 "eno4"
PCIBridge
PCI 8086:1572
Net L#10 "enp7s0f0"
PCI 8086:1572
Net L#11 "enp7s0f1"
PCI 8086:8d62
PCIBridge
PCIBridge
PCIBridge
PCIBridge
PCI 102b:0534
GPU L#12 "card0"
GPU L#13 "controlD64"
PCI 8086:8d02
NUMANode L#1 (P#1 31GB)
Package L#1 + L3 L#1 (15MB)
L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
PU L#12 (P#1)
PU L#13 (P#13)
L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
PU L#14 (P#3)
PU L#15 (P#15)
L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8
PU L#16 (P#5)
PU L#17 (P#17)
L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9
PU L#18 (P#7)
PU L#19 (P#19)
L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
PU L#20 (P#9)
PU L#21 (P#21)
L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
PU L#22 (P#11)
PU L#23 (P#23)
HostBridge L#9
PCIBridge
PCI 8086:10fb
Net L#14 "enp132s0f0"
PCI 8086:10fb
Net L#15 "enp132s0f1"
4 x { PCI 8086:10ed }