iperf3跑满100G网卡实测记录
环境准备
拓扑
两台服务器 100G网卡对插直连。
ubuntu20.04, 系统设置默认状态
MTU 1500
网卡
Mellonax ConnectX5 100G
root@vnet:~# lspci | grep Mellanox
98:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
98:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
设置为以太网模式
#mlxconfig -d /dev/mst/mt4119_pciconf0 set LINK_TYPE_P1=2
root@vnet:~# mlxconfig -d /dev/mst/mt4119_pciconf0 q |grep LINK_TYPE_P
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
测试
服务端
三个窗口分别开3个iperf3 server
iperf3 -s -p 5200
iperf3 -s -p 5201
iperf3 -s -p 5202
客户端
三个窗口分别同时开3个iperf3 client，跑10分钟
iperf3 -c 10.0.0.1 -p 5200 -t 600
iperf3 -c 10.0.0.1 -p 5201 -t 600
iperf3 -c 10.0.0.1 -p 5202 -t 600
iperf3结果
三个iperf3 Client, 合并速率大约 92GB/s
第一个
root@vnet:~# iperf3 -c 10.0.0.1 -p 5200 -t 600
Connecting to host 10.0.0.1, port 5200
[ 5] local 10.0.0.2 port 39104 connected to 10.0.0.1 port 5200
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 5.70 GBytes 49.0 Gbits/sec 639 2.82 MBytes
[ 5] 1.00-2.00 sec 3.82 GBytes 32.8 Gbits/sec 0 2.82 MBytes
[ 5] 2.00-3.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.82 MBytes
[ 5] 3.00-4.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.82 MBytes
[ 5] 4.00-5.00 sec 3.70 GBytes 31.8 Gbits/sec 0 2.83 MBytes
[ 5] 5.00-6.00 sec 3.71 GBytes 31.8 Gbits/sec 0 2.83 MBytes
[ 5] 6.00-7.00 sec 3.66 GBytes 31.4 Gbits/sec 0 2.83 MBytes
[ 5] 7.00-8.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.85 MBytes
[ 5] 8.00-9.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.85 MBytes
[ 5] 9.00-10.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.89 MBytes
...
[ 5] 594.00-595.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.81 MBytes
[ 5] 595.00-596.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.81 MBytes
[ 5] 596.00-597.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.95 MBytes
[ 5] 597.00-598.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.95 MBytes
[ 5] 598.00-599.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.95 MBytes
[ 5] 599.00-600.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.95 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-600.00 sec 2.14 TBytes 31.4 Gbits/sec 7061 sender
[ 5] 0.00-600.00 sec 2.14 TBytes 31.4 Gbits/sec receiver
第二个
root@vnet:~# iperf3 -c 10.0.0.1 -p 5201 -t 600
Connecting to host 10.0.0.1, port 5201
[ 5] local 10.0.0.2 port 42866 connected to 10.0.0.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 4.76 GBytes 40.9 Gbits/sec 3 2.72 MBytes
[ 5] 1.00-2.00 sec 3.71 GBytes 31.9 Gbits/sec 0 2.72 MBytes
[ 5] 2.00-3.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.72 MBytes
[ 5] 3.00-4.00 sec 3.61 GBytes 31.0 Gbits/sec 0 2.72 MBytes
[ 5] 4.00-5.00 sec 3.55 GBytes 30.5 Gbits/sec 0 2.72 MBytes
[ 5] 5.00-6.00 sec 3.59 GBytes 30.8 Gbits/sec 0 2.72 MBytes
[ 5] 6.00-7.00 sec 3.65 GBytes 31.3 Gbits/sec 0 3.21 MBytes
[ 5] 7.00-8.00 sec 3.65 GBytes 31.3 Gbits/sec 0 3.21 MBytes
[ 5] 8.00-9.00 sec 3.65 GBytes 31.3 Gbits/sec 0 3.21 MBytes
[ 5] 9.00-10.00 sec 3.65 GBytes 31.3 Gbits/sec 0 3.21 MBytes
[ 5] 10.00-11.00 sec 3.65 GBytes 31.3 Gbits/sec 0 3.21 MBytes
...
[ 5] 594.00-595.00 sec 3.65 GBytes 31.4 Gbits/sec 0 4.76 MBytes
[ 5] 595.00-596.00 sec 3.65 GBytes 31.4 Gbits/sec 0 4.76 MBytes
[ 5] 596.00-597.00 sec 3.65 GBytes 31.4 Gbits/sec 0 4.76 MBytes
[ 5] 597.00-598.00 sec 3.65 GBytes 31.4 Gbits/sec 0 4.76 MBytes
[ 5] 598.00-599.00 sec 3.65 GBytes 31.4 Gbits/sec 0 4.76 MBytes
[ 5] 599.00-600.00 sec 4.24 GBytes 36.4 Gbits/sec 0 4.76 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-600.00 sec 2.14 TBytes 31.4 Gbits/sec 270 sender
[ 5] 0.00-600.00 sec 2.14 TBytes 31.4 Gbits/sec receiver
第三个
root@vnet:~# iperf3 -c 10.0.0.1 -p 5202 -t 600
Connecting to host 10.0.0.1, port 5202
[ 5] local 10.0.0.2 port 33730 connected to 10.0.0.1 port 5202
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.30 GBytes 28.4 Gbits/sec 0 1.76 MBytes
[ 5] 1.00-2.00 sec 3.65 GBytes 31.4 Gbits/sec 0 1.85 MBytes
[ 5] 2.00-3.00 sec 3.65 GBytes 31.4 Gbits/sec 0 1.85 MBytes
[ 5] 3.00-4.00 sec 3.70 GBytes 31.8 Gbits/sec 0 1.85 MBytes
[ 5] 4.00-5.00 sec 3.71 GBytes 31.8 Gbits/sec 0 1.85 MBytes
[ 5] 5.00-6.00 sec 3.66 GBytes 31.5 Gbits/sec 0 1.85 MBytes
[ 5] 6.00-7.00 sec 3.66 GBytes 31.4 Gbits/sec 0 1.85 MBytes
[ 5] 7.00-8.00 sec 3.66 GBytes 31.4 Gbits/sec 0 1.97 MBytes
[ 5] 8.00-9.00 sec 3.66 GBytes 31.4 Gbits/sec 0 1.97 MBytes
[ 5] 9.00-10.00 sec 3.66 GBytes 31.4 Gbits/sec 0 1.97 MBytes
[ 5] 10.00-11.00 sec 3.66 GBytes 31.4 Gbits/sec 50 1.53 MBytes
...
[ 5] 590.00-591.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 591.00-592.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 592.00-593.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 593.00-594.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 594.00-595.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 595.00-596.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 596.00-597.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 597.00-598.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 598.00-599.00 sec 3.65 GBytes 31.4 Gbits/sec 0 2.07 MBytes
[ 5] 599.00-600.00 sec 6.34 GBytes 54.4 Gbits/sec 715 824 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-600.00 sec 2.15 TBytes 31.5 Gbits/sec 12151 sender
[ 5] 0.00-600.00 sec 2.15 TBytes 31.5 Gbits/sec receiver
接口速率
其中发送速率12016406，基本稳定在1200000 KB/s以上
| Trans KB/s | Trans Kbps | Trans Mbps | Trans Gbps |
|---|---|---|---|
| 12016406 | 96131248 | 96131.248 | 96.131248 |
Kpbs 与 KB/s换算关系是 x8。
Kbps Mbps Gbps换算倍数是 1000。
CPU占用
客户端
top - 07:48:53 up 5 min, 6 users, load average: 0.61, 0.24, 0.10
Tasks: 765 total, 2 running, 763 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 1.5 sy, 0.0 ni, 98.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 515430.8 total, 250782.5 free, 263932.7 used, 715.5 buff/cache
MiB Swap: 8192.0 total, 8192.0 free, 0.0 used. 249515.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3394 root 20 0 5900 3184 2760 R 34.9 0.0 0:06.79 iperf3
3393 root 20 0 5900 3100 2676 S 33.9 0.0 0:07.63 iperf3
3392 root 20 0 5900 3188 2764 S 33.2 0.0 0:07.17 iperf3
服务端
Tasks: 622 total, 3 running, 249 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 2.3 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 0.6 si, 0.0 st
KiB Mem : 52781030+total, 48375638+free, 43817464 used, 236448 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 48168425+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3216 root 20 0 9752 2316 2064 R 67.4 0.0 0:32.56 iperf3
3143 root 20 0 9884 2408 2148 S 59.5 0.0 7:45.59 iperf3
3176 root 20 0 9884 2352 2096 R 55.8 0.0 6:41.01 iperf3