基于DPDK的MellanoxCX4-40G网卡性能测试

1、硬件环境

a)      服务器：Sugon W560-G20 workstation

b)      处理器：Intel Xeno E5 -2660 V3处理器（2.60GHz） 20 CPU cores*2 NUMA nodes

c)      内存：128G，2*32GB DIMMs * 2NUMA nodes 2133MHz

d)      主板：S7070A2NR-B，PCIe_4x16（CPU0），PCIe_2x16（CPU1）

e)      网卡：Mellanox ConnectX-4 Lx 40GbE网卡（model No:CX416A）

2、软件环境

a)      操作系统：RedHat Linux 7.1

(default_hugepage_size=4G 2048*2M)

b)      BIOS:AMI v7.102.R01 2016/3/30

(Power Profile PERFORMANCE; C-states OFF; P-states OFF;TurboBoost ON; HyperThreading OFF; Virt OFF; VTd OFF; SR-IOV OFF; SMI OFFIONonPostedPrefetching OFF;)

c)      内核版本：3.10.0-229.el7.x86_64

d)      网卡固件版本：14.18.2000

e)      网卡驱动版本：MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.1-x86_64

f)       DPDK版本：MellanoxDPDK 16.11_3.0

(启用mlx5 PMD,CONFIG_RTE_LIBRTE_MLX5_PMD=y)

g)      Ans版本（2017-11-23）：基于dpdk17.05.2

 

3、测试环境

测试分两种情形，一种是针对数据包进行简单的收发测试，另一种是在对数据包进行三层协议转发条件下进行测试。

 

图1 DPDKTestpmd性能测试环境

如上图所示，在测试环境A中，TestPMD应用程序把同一块网卡的两个以太网端口连成环回模式。这样可以在没有外部流量发生器的情况下检查网络设备的接收和传输功能。发送的数据包为UDP数据包。

图2 DPDK下使用ANS进行转发测试

在测试环境B中，Port3绑定pktgen，port1绑定ans。Port1和port3中间通过交换机相连，ans从Port1发送UDP数据包到port3。

4、测试工具

使用Mellanox DPDK 16.11_3.0内提供的测试工具Testpmd

基于DPDK 的含协议栈的应用ans及发包程序dpdk-udp

基于DPDK的发包程序pktgen-3.4.2

网卡收发报文速率统计脚本eth_stat.sh

5、测试结果

表1 测试环境A下测试结果

序号	Pkt_data_len	DEF_PKT_BURST	nbcore	nb_rxq/txq	RTE_TEST_RX_DESC_DEFAULT	RTE_TEST_TX_DESC_DEFAULT	Gbps
1	64	32	1	1	128	512	4.1
2	64	32	2	1	128	512	4.2
3	64	32	3	1	128	512	4.2
4	64	64	2	1	128	512	4.6
5	64	64	4	1	128	512	5.7
6	64	128	1	1	128	512	5.0
7	64	128	2	1	128	512	5.0
8	64	128	3	1	128	512	5.0
9	64	256	1	1	128	512	4.9
10	64	256	2	1	128	512	5.0
11	64	256	3	1	128	512	5.0
12	64	512	1	1	128	512	4.9
13	64	512	2	1	128	512	5.0
14	64	512	3	1	128	512	5.0
15	64	512	4	1	128	512	6.3
16	64	32	1	2	128	512	4.5
17	64	32	2	2	128	512	4.7
18	64	32	3	2	128	512	4.7
19	64	64	2	2	128	512	5.2
20	64	64	4	2	128	512	6.3
21	64	128	4	2	128	512	6.3
22	64	128	3	3	128	512	6.3
23	64	256	4	3	128	512	6.3
24	64	256	4	4	128	512	6.3
25	128	32	1	1	128	512	7.1
26	128	32	2	1	128	512	7.3
27	128	32	3	1	128	512	7.3
28	128	32	4	1	128	512	7.4
29	128	64	4	1	128	512	9.4
30	128	128	2	1	128	512	8.7
31	128	256	4	1	128	512	10.3
32	128	64	4	2	128	512	10.3
33	128	256	4	4	128	512	10.3
34	256	32	2	2	128	512	14.7
35	256	256	4	2	128	512	18.2
36	256	256	4	4	128	512	18.2
37	512	128	1	2	128	512	29.1
38	512	128	2	2	128	512	29.6
39	1024	128	2	2	128	512	39.7
40	1024	256	2	2	128	512	39.7
41	1024	256	1	2	128	512	39.1
42	1024	128	2	4	128	512	39.4
43	1024	256	2	4	128	512	39.8
44	1024	256	4	4	128	512	42.2

 

上表中，各字段含义如下：

Pkt_data_len：报文长度，单位字节

DEF_PKT_BURST：一次发送报文的数量

Nbcore：处理报文的CPU核数目

nb_rxq/txq：报文接收队列，发送队列数目

RTE_TEST_RX_DESC_DEFAULT：收包队列长度

RTE_TEST_TX_DESC_DEFAULT：发包队列长度

Gbps：网卡数据包通过率 Gbit/sec

 

表2 测试环境B下测试结果

序号	Pkt_data_len(不含协议头)	Socket fd数目	Gbps(port1->port3)
1	32	1	5.5
2	32	1(app不绑核)	2.7
3	32	2(两个app分别绑定core2,core3)	5.5
4	64	1	7.3
5	64	2(两个app分别绑定core2,core3)	7.3
6	128	1	10.9
7	128	2(两个app分别绑定core3,core4; ans绑定core1和core2)	11
8	256	1	17.9
9	512	1	31.8
10	1024	1	35.0
11	2048	1	40.0
12	4096	1	40.0

注：上表中如果没有特别说明ans默认绑定core1， dpdk_udp默认绑定core2

6、分析总结

针对测试环境A，表1中列出了除硬件和软件环境外影响网卡收发包性能的6个因素。分析如下：

1、  数据包大小对网卡速率影响最大，当数据包大小达到1024字节后，速率能接近线速40Gbps。

2、  在数据包大小为64字节时，一次发送报文的数量越多，通过率越大。

3、  处理报文的CPU核数目越多，通过率越大。

4、  在处理报文的CPU核数目较多的情况下增加报文接收发送队列数目能显著提高通过率。

5、  收包队列长度和发包队列长度对网卡速率影响较小，其中收包队列长度为128，发包队列长度为512能满足大部分场景的应用。

针对测试环境B，对表2的分析如下：

1、  数据包大小对速率影响最大，UDP数据包达到2048字节（有效数据字段，不包括协议字段）时达到线速。

2、  APP绑核情况下能提高速率，在32字节小包情况下速率能提升一倍。

3、  采用两个以上app从同一个port不同socket端口进行发数时，与单个app相比对总速率提升没有影响。

4、  Ans绑定1个核和绑定多个核对速率提升影响不大。

7、测试脚本

测试环境A

Test pmd

shell:
./testpmd -c 0xfff -l 1-9 -n 4 -- -i 

start tx_only


change place:

1 testpmd.h #define TXONLY_DEF_PACKET_LEN 64

2 testpmd.c queueid_t nb_rxq=1; queueid_t nb_txq=1

3 set nbcore 

4 set burst

5 testpmd.c #define RTE_TEST_RX_DESC_DEFAULT #define RTE_TEST_TX_DESC_DEFAULT

测试环境B

shell

pktgen

./usr/dpdk/MLNX_DPDK_16.11_3.0/examples/pktgen-3.4.2/app/app/x86_64-native-linuxapp-gcc/pktgen -l 7,8-9 -n 4 --proc-type auto --log-level 7 --socket-mem 2048,2048 --file-prefix pg -b 80:00.0 -b 80:00.1 -- -T -P --crc-strip -m 8.1 -m 9.3 -f themes/black-yellow.themes

set 3 proto udp

set 3 src ip 10.1.0.10/24

set 3 sport 9999

enable 3 garp


ans

./ans -c 0xf -n 4 -- -p 0x2 --config="(1,0,1)"

./ans -c 0x6 -n 4 -- -p 0x2 --config="(1,0,1),(1,1,2)"

dpdk_udp

./dpdk_udp 2

两个DPDK_UDP应用程序代码如下，分别使用port1的不同的socket

/*-
 *   BSD LICENSE
 *
 *   Copyright(c) 2015-2017 Ansyun . All rights reserved.
 *   All rights reserved.
 *
 *   Redistribution and use in source and binary forms, with or without
 *   modification, are permitted provided that the following conditions
 *   are met:
 *
 *     * Redistributions of source code must retain the above copyright
 *       notice, this list of conditions and the following disclaimer.
 *     * Redistributions in binary form must reproduce the above copyright
 *       notice, this list of conditions and the following disclaimer in
 *       the documentation and/or other materials provided with the
 *       distribution.
 *     * Neither the name of Ansyun  nor the names of its
 *       contributors may be used to endorse or promote products derived
 *       from this software without specific prior written permission.
 *
 *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#ifndef __linux__
  #ifdef __FreeBSD__
    #include 
  #else
    #include 
  #endif
#endif

#include 
#include 
#include 
#include 

#include "anssock_intf.h"
#include "ans_errno.h"


struct epoll_event events[20];

int main(int argc, char *argv[])
{
    int ret;
    int i = 0 ;
    int fd, fd1,recvfd = 0;
    int epfd;
    int data_num =0;
    char send_data[65536];
    struct timeval start, end;
    struct sockaddr_in addr_in,addr_in1;  
    struct sockaddr_in remote_addr;  
    struct epoll_event event;
    char recv_buf[2038];
    int recv_len; 
    int core = -1;

    if(argc >= 2)
    {
        core = atoi(argv[1]);
        printf("affinity to core %d \n", core);

        /*initialize thread bind cpu*/
        cpu_set_t cpus;

        CPU_ZERO(&cpus);
        CPU_SET((unsigned)core, &cpus);
        sched_setaffinity(0, sizeof(cpus), &cpus);  
    }
    else
    {
        printf("no affinity by default \n");
    }

    ret = anssock_init(NULL);

    /* create epoll socket */
    epfd = anssock_epoll_create(0);

    fd = anssock_socket(AF_INET, SOCK_DGRAM, 0);
    

    memset(&addr_in, 0, sizeof(addr_in));      
    addr_in.sin_family = AF_INET;  
    addr_in.sin_port   = htons(4444);  
    addr_in.sin_addr.s_addr = inet_addr("10.1.0.5"); 

    ret =  anssock_bind(fd, (struct sockaddr *)&addr_in, sizeof(addr_in) );
    


    memset(&remote_addr, 0, sizeof(remote_addr));      
    remote_addr.sin_family = AF_INET;  
    remote_addr.sin_port   = htons(9999);  
    remote_addr.sin_addr.s_addr = inet_addr("10.1.0.10");


    memset(send_data, 0, sizeof(send_data));
    
    while(1)
    {
      anssock_sendto(fd, send_data, 128, 0, (struct sockaddr *)&remote_addr,  sizeof(remote_addr));
    }
    anssock_close(fd);
    anssock_close(epfd);

    return 0;
}

/*-
 *   BSD LICENSE
 *
 *   Copyright(c) 2015-2017 Ansyun . All rights reserved.
 *   All rights reserved.
 *
 *   Redistribution and use in source and binary forms, with or without
 *   modification, are permitted provided that the following conditions
 *   are met:
 *
 *     * Redistributions of source code must retain the above copyright
 *       notice, this list of conditions and the following disclaimer.
 *     * Redistributions in binary form must reproduce the above copyright
 *       notice, this list of conditions and the following disclaimer in
 *       the documentation and/or other materials provided with the
 *       distribution.
 *     * Neither the name of Ansyun  nor the names of its
 *       contributors may be used to endorse or promote products derived
 *       from this software without specific prior written permission.
 *
 *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#ifndef __linux__
  #ifdef __FreeBSD__
    #include 
  #else
    #include 
  #endif
#endif

#include 
#include 
#include 
#include 

#include "anssock_intf.h"
#include "ans_errno.h"


struct epoll_event events[20];

int main(int argc, char *argv[])
{
    int ret;
    int i = 0 ;
    int fd, fd1,recvfd = 0;
    int epfd;
    int data_num =0;
    char send_data[1024];
    struct timeval start, end;
    struct sockaddr_in addr_in,addr_in1;  
    struct sockaddr_in remote_addr;  
    struct epoll_event event;
    char recv_buf[2038];
    int recv_len; 
    int core = -1;

    if(argc >= 2)
    {
        core = atoi(argv[1]);
        printf("affinity to core %d \n", core);

        /*initialize thread bind cpu*/
        cpu_set_t cpus;

        CPU_ZERO(&cpus);
        CPU_SET((unsigned)core, &cpus);
        sched_setaffinity(0, sizeof(cpus), &cpus);  
    }
    else
    {
        printf("no affinity by default \n");
    }

    ret = anssock_init(NULL);

    /* create epoll socket */
    epfd = anssock_epoll_create(0);

    fd = anssock_socket(AF_INET, SOCK_DGRAM, 0);
    

    memset(&addr_in, 0, sizeof(addr_in));      
    addr_in.sin_family = AF_INET;  
    addr_in.sin_port   = htons(8888);  
    addr_in.sin_addr.s_addr = inet_addr("10.1.0.2"); 

    ret =  anssock_bind(fd, (struct sockaddr *)&addr_in, sizeof(addr_in) );
    
    

    memset(&remote_addr, 0, sizeof(remote_addr));      
    remote_addr.sin_family = AF_INET;  
    remote_addr.sin_port   = htons(9999);  
    remote_addr.sin_addr.s_addr = inet_addr("10.1.0.10"); 

    memset(send_data, 0, sizeof(send_data));
    
    while(1)
    {
      anssock_sendto(fd, send_data,  128, 0, (struct sockaddr *)&remote_addr,  sizeof(remote_addr));
    }
    anssock_close(fd);
    anssock_close(epfd);

    return 0;
}