Netperf使用总结

什么是Netperf

• Netperf是由惠普公司开发的一种网络性能测量工具，主要针对基于TCP或UDP的传输。
• Netperf根据应用的不同，可以进行不同模式的网络性能测试，即批量数据传输（bulk data transfer）模式和请求/应答（request/reponse）模式。
• Netperf测试结果所反映的是一个系统能够以多快的速度向另外一个系统发送数据，以及另
  外一个系统能够以多块的速度接收数据
• https://github.com/HewlettPackard/netperf
• https://manpages.org/netperf
• netserver(1) - Linux man page

基础用法

• 两台机器，一台服务端 运行 netserver，另一端 客户端，运行 netperf
• Netperf 缺省情况下进行 TCP 批量传输，即 -t TCP_STREAM
• 关闭防火墙

[root@localhost ~]# netserver
Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_UNSPEC

[root@localhost netperf]# netperf -H 127.0.0.1 -l 1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

131072  16384  16384    1.00     24765.25   

不指定端口的话 12865 为 netserver 默认端口

显示结果

• Protocol : 传输协议类型 TCP、UDP
• Elapsed : 测试的时间
• Throughput ： 吞吐量
• Throughput Units ： 吞吐量的单位
• Minimum/Maximum/Mean/50th/90th/99th Percentile Latency : 各种时延信息
• 其他更多结果参考 9.3.1 Omni Output Selectors

详细使用方法

命令格式：

 netserver [-4] [-6] [-d] [-h] [-L name,family] [-p portnum] [-v verbosity] [-V]
 netperf  -- 

global 是全局设置选项，test-specific 是针对于这个测试的特定配置

一些规律：很多小写的命令表示发送/本地，大写的命令表示远程/接收

help

[root@localhost netperf]# netserver -h

Usage: netserver [options] 

Options:
    -h                Display this text
    -D                Do not daemonize
    -d                Increase debugging output
    -f                Do not spawn chilren for each test, run serially
    -L name,family    Use name to pick listen address and family for family
    -N                No debugging output, even if netperf asks
    -p portnum        Listen for connect requests on portnum.
    -4                Do IPv4
    -6                Do IPv6
    -v verbosity      Specify the verbosity level
    -V                Display version information and exit
    -Z passphrase     Expect passphrase as the first thing received

[root@localhost ~]# netperf -h

Usage: netperf [global options] -- [test options]

Global options:
    -a send,recv      Set the local send,recv buffer alignment
    -A send,recv      Set the remote send,recv buffer alignment
    -B brandstr       Specify a string to be emitted with brief output
    -c [cpu_rate]     Report local CPU usage
    -C [cpu_rate]     Report remote CPU usage
    -d                Increase debugging output
    -D time,[units] * Display interim results at least every time interval
                      using units as the initial guess for units per second
                      A negative value for time will make heavy use of the
                      system's timestamping functionality
    -f G|M|K|g|m|k    Set the output units
    -F lfill[,rfill]* Pre-fill buffers with data from specified file
    -h                Display this text
    -H name|ip,fam *  Specify the target machine and/or local ip and family
    -i max,min        Specify the max and min number of iterations (15,1)
    -I lvl[,intvl]    Specify confidence level (95 or 99) (99)
                      and confidence interval in percentage (10)
    -j                Keep additional timing statistics
    -l testlen        Specify test duration (>0 secs) (<0 bytes|trans)
    -L name|ip,fam *  Specify the local ip|name and address family
    -o send,recv      Set the local send,recv buffer offsets
    -O send,recv      Set the remote send,recv buffer offset
    -n numcpu         Set the number of processors for CPU util
    -N                Establish no control connection, do 'send' side only
    -p port,lport*    Specify netserver port number and/or local port
    -P 0|1            Don't/Do display test headers
    -r                Allow confidence to be hit on result only
    -s seconds        Wait seconds between test setup and test start
    -S                Set SO_KEEPALIVE on the data connection
    -t testname       Specify test to perform
    -T lcpu,rcpu      Request netperf/netserver be bound to local/remote cpu
    -v verbosity      Specify the verbosity level
    -W send,recv      Set the number of send,recv buffers
    -v level          Set the verbosity level (default 1, min 0)
    -V                Display the netperf version and exit
    -y local,remote   Set the socket priority
    -Y local,remote   Set the IP_TOS. Use hexadecimal.
    -Z passphrase     Set and pass to netserver a passphrase

[root@localhost netperf]# netperf -t TCP_RR -- -h

Usage: netperf [global options] -- [test options] 

OMNI and Migrated BSD Sockets Test Options:
    -b number         Send number requests at start of _RR tests
    -c                Explicitly declare this a connection test such as
                      TCP_CRR or TCP_CC
    -C                Set TCP_CORK when available
    -d direction      Explicitly set test direction based on bitwise OR
                      of 0x2 for transmit and 0x4 for receive. Default:
                      based on test type
    -D [L][,R]        Set TCP_NODELAY locally and/or remotely (TCP_*)
    -h                Display this text
    -H name[/mask],fam  Use name (or IP) and family as target of data connection
                      A mask value will cause randomization of the IP used
    -k [file]         Generate keyval output optionally based on file
                      Use filename of '?' to get the list of choices
    -K loc[,rem]      Set the local and/or remote congestion control
                      algorithm to use on those platforms where it can
                      be set.
    -L name[/mask],fam  Use name (or IP) and family as source of data connection
                      A mask value will cause randomization of the IP used
    -m local,remote   Set the send size for _STREAM/_MAERTS tests
    -M local,remote   Set the recv size for _STREAM/_MAERTS tests
    -n                Use the connected socket for UDP locally
    -N                Use the connected socket for UDP remotely
    -o [file]         Generate CSV output optionally based on file
                      Use filename of '?' to get the list of choices
    -O [file]         Generate classic-style output based on file
                      Use filename of '?' to get the list of choices
    -p min[,max]      Set the min/max port numbers for TCP_CRR, TCP_TRR
    -P local[,remote] Set the local/remote port for the data socket
    -r req,[rsp]      Set request/response sizes (TCP_RR, UDP_RR)
    -R 0/1            Allow routing of traffic on data connection.
                      Default: 0 (off) for UDP_STREAM, 1 (on) otherwise
    -s send[,recv]    Set local socket send/recv buffer sizes
    -S send[,recv]    Set remote socket send/recv buffer sizes
    -t type           Explicitly set socket type. Default is implicit
                      based on other settings
    -T protocol       Explicitly set data connection protocol. Default is
                      implicit based on other settings
    -u uuid           Use the supplied string as the UUID for this test.
    -4                Use AF_INET (eg IPv4) on both ends of the data conn
    -6                Use AF_INET6 (eg IPv6) on both ends of the data conn

For those options taking two parms, at least one must be specified;
specifying one value without a comma will set both parms to that
value, specifying a value with a leading comma will set just the second
parm, a value with a trailing comma will set just the first. To set
each parm to unique values, specify both and separate them with a
comma.

global options

H

指定server的ip

-t testname

• TCP_STREAM, TCP_MAERTS, TCP_SENDFILE, TCP_RR, TCP_CRR, TCP_CC
• UDP_STREAM, UDP_RR
• XTI_TCP_STREAM, XTI_TCP_RR, XTI_TCP_CRR, XTI_TCP_CC
• XTI_UDP_STREAM, XTI_UDP_RR
• SCTP_STREAM, SCTP_RR
• DLCO_STREAM, DLCO_RR, DLCL_STREAM, DLCL_RR
• LOC_CPU, REM_CPU
• OMNI

可分为两类：
一类是单项的传输性能测试， STEAM 后缀的，一类是 往返测试 RR后缀的
常用的：
UDP_STREAM
UDP_RR

-s

在建立链接和传输数据之间sleep一段时间，用于多次运行netperf，不希望之前的netperf对后面的netperf 有影响的场景。

-p

-p remote_port,local_port

注意，指定的端口，不是实际发包用的端口，而是客户端服务端传输基础配置等信息的端口,netstat 时该端口是 ESTABLISHED 状态

The first value of the optionspec passed-in with this option tells netperf the port number at which it should expect the remote netserver to be listening for control connections. The second value of the optionspec will request netperf to bind to that local port number before establishing the control connection

-P

-P 0  # 关闭横幅，默认1

cC

-c # local cpu utilization
-C # remote cpu utilization

当"oOk"中包含cpu参数 时，应该加此命令，否则cpu利用率显示为0.

-T

This option controls the CPU, and probably by extension memory, affinity of netperf and/or netserver.

设置服务端和客户端的cpu亲和性

test options

oOk

-o : 横向显示结果，逗号分隔
-O ： 横向显示结果
-k ： 纵向显示结果

r && mM

r主要用于RR场景

-r request size,response size

request size : Remote Recv Size / Request Size Bytes
response size : Remote Send Size / Response Size Bytes

mM主要用于 STREAM场景

-m 32768/32k  # 传给send的buf大小
-M 32768/32k  # 传给 recv 的大小

k/m/g/K/M/G : 10^3/10^6/10^9/2^10/2^20/2^30 # 单位
m : Local Send Size
M : Remote Recv Size

RR测试是本地发射和远端的接收 或 本地接收和远端发射相等吗？为什么只有两个参数，如果相等，为什么在RR实测中 Local Send Size,Local Recv Size 的值和r配置无关？

## 这4个size用于 non-request/response，也就是STREAM
LOCAL_SEND_SIZE
This will display the size of the buffers netperf passed in any “send” calls it made on the data connection for a non-request/response test. Units: Bytes.
LOCAL_RECV_SIZE
This will display the size of the buffers netperf passed in any “receive” calls it made on the data connection for a non-request/response test. Units: Bytes.
REMOTE_SEND_SIZE
This will display the size of the buffers netserver passed in any “send” calls it made on the data connection for a non-request/response test. Units: Bytes.
REMOTE_RECV_SIZE
This will display the size of the buffers netserver passed in any “receive” calls it made on the data connection for a non-request/response test. Units: Bytes.

## 这2个size用于 request/response，也就是RR
REQUEST_SIZE
This will display the size of the requests netperf sent in a request-response test. Units: Bytes.
RESPONSE_SIZE
This will display the size of the responses netserver sent in a request-response test. Units: Bytes.

L

指定本地IP地址测试

-P

--- -P remote_port,local_port
#Set the local and/or remote port numbers for the data connection.

我理解-p是用于控制连接的端口，而-P是真正数据传输的测试的端口

eg：

-p 12866,11111 -- -P 12869,11112
[root@localhost netperf]# netstat -antup | grep net
Active Internet connections (servers and established)
tcp        0      0 127.0.0.1:11111         127.0.0.1:12866         ESTABLISHED 2905792/netperf     
tcp6       0      0 :::12865                :::*                    LISTEN      497555/netserver    
tcp6       0      0 :::12866                :::*                    LISTEN      2532430/netserver   
tcp6       0      0 127.0.0.1:12866         127.0.0.1:11111         ESTABLISHED 2905794/netserver   
udp        0      0 0.0.0.0:12869           0.0.0.0:*                           2905792/netperf     
udp     4352      0 0.0.0.0:11112           0.0.0.0:*                           2905794/netserver

sS

-s : This option sets the local (netperf) send and receive socket buffer sizes for the data connection to the value(s) specified.
-S :This option sets the remote (netserver) send and/or receive socket buffer sizes for the data connection to the value(s) specified.

常用辅助命令

netstat -lnpt
kill -9 pid of netperf
fillwall-cmd --state # 防火墙状态
systemctl stop fillwalld.service # 关闭防火墙

其他

https://github.com/HewlettPackard/netperf/blob/master/doc/netperf.html 基本讲了所有使用方法，还有一些测试例子https://github.com/HewlettPackard/netperf/blob/master/doc/examples/tcp_rr_script

文档中描述的的一些英文词汇理解：

control connections： 指用于一些控制信息传输的连接
data connection: 指用于测试数据传输的那部分连接

参考资料

山外笔记-工具框架】Netperf网络性能测试工具详解教程

netperf的安装、性能测试、参数、启动报错的坑、实例

性能测试工具iPerf和Netperf使用介绍

网络性能测试最佳实践

netperf常用命令使用方法