性能调优之网络篇 - 用ss命令替代netstat
netstat命令大家肯定已经很熟悉了，但是在2001年的时候netstat 1.42版本之后就没更新了，之后取代的工具是ss命令，是iproute2 package的一员。
# rpm -ql iproute | grep ss /usr/sbin/ss /usr/share/man/man8/ss.8.gz
ss可以显示跟netstat类似的信息，但是速度却比netstat快很多，netstat是基于/proc/net/目录下查询相关信息输出的，用strace跟踪一下netstat查询tcp的连接，会看到他open的是/proc/net/tcp的信息。
# strace -eopen -f netstat -antl | grep tcp
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en_US/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/net/tcp", O_RDONLY) = 3
open("/proc/net/tcp6", O_RDONLY) = 3
但是同样跟踪一下ss，你会发现他调用了cache, libc 和slabinfo：
# strace -e open -f ss -antl
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/proc/slabinfo", O_RDONLY) = 3
ss快的秘密就在于它利用的是TCP协议的tcp_diag模块，而且是从内核直接读取信息。感兴趣的朋友可以去读一下他的源码：https://github.com/shemminger/iproute2/blob/master/misc/ss.c 。
分析完各自的原理之后咱们开始上手学习一下工作中遇到的能用ss的场景。
常用命令：
ss -t -a #print tcp info, -u, the udp info
# ss -t -a State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:http *:* LISTEN 0 5 127.0.0.1:domain *:* LISTEN 0 5 172.31.23.95:domain *:* LISTEN 0 128 *:https *:* LISTEN 0 3 *:1723 *:*
ss -ltp #show process name and pid
# ss -ltp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:http *:* users:(("nginx",7695,6),("nginx",7694,6),("nginx",7693,6),("nginx",7692,6),("nginx",1144,6))
LISTEN 0 5 127.0.0.1:domain *:* users:(("dnsmasq",1186,7))
LISTEN 0 5 172.31.23.95:domain *:* users:(("dnsmasq",1186,5))
LISTEN 0 128 *:https *:* users:(("nginx",7695,9),("nginx",7694,9),("nginx",7693,9),("nginx",7692,9),("nginx",1144,9))
LISTEN 0 3 *:1723 *:* users:(("pptpd",1386,6))
LISTEN 0 128 *:8000 *:* users:(("nginx",7695,8),("nginx",7694,8),("nginx",7693,8),("nginx",7692,8),("nginx",1144,8))
LISTEN 0 128 *:8388 *:* users:(("ssserver",1572,4))
LISTEN 0 50 127.0.0.1:mysql *:* users:(("mysqld",1418,10))
LISTEN 0 128 *:7788 *:* users:(("sshd",1166,3))
LISTEN 0 128 :::http :::* users:(("nginx",7695,7),("nginx",7694,7),("nginx",7693,7),("nginx",7692,7),("nginx",1144,7))
LISTEN 0 5 ::1:domain :::* users:(("dnsmasq",1186,11))
LISTEN 0 5 fe80::408:35ff:fe6b:31f9:domain :::* users:(("dnsmasq",1186,9))
LISTEN 0 128 :::munin :::* users:(("munin-node",901,5))
LISTEN 0 128 :::7788 :::* users:(("sshd",1166,4))
ss -s # print summary statistics
这个让我想起来网上有个面试题，就是统计当前系统的各个状态的连接数，一般的做法都是用netstat和awk，
netstat -an | awk '/^tcp/ {++s[$NF]} END{for (i in s) print i, s[i]}'
但是有了ss -s很简洁的四个字符就可以显示出来想要的全部信息：
# ss -s Total: 145 (kernel 186) TCP: 29 (estab 1, closed 14, orphaned 0, synrecv 0, timewait 14/0), ports 0 Transport Total IP IPv6 * 186 - - RAW 0 0 0 UDP 8 5 3 TCP 15 10 5 INET 23 15 8 FRAG 0 0 0
ss -ant -o # print the time info:
# ss -ant -o State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:80 *:* LISTEN 0 5 127.0.0.1:53 *:* LISTEN 0 5 172.31.23.95:53 *:* LISTEN 0 128 *:443 *:* LISTEN 0 3 *:1723 *:* LISTEN 0 128 *:8000 *:* LISTEN 0 128 *:8388 *:* LISTEN 0 50 127.0.0.1:3306 *:* LISTEN 0 128 *:7788 *:* TIME-WAIT 0 0 172.31.23.95:37018 111.161.68.235:80 timer:(timewait,6.184ms,0) TIME-WAIT 0 0 172.31.23.95:37029 111.161.68.235:80 timer:(timewait,44sec,0) TIME-WAIT 0 0 172.31.23.95:37024 111.161.68.235:80 timer:(timewait,27sec,0) ESTAB 0 0 172.31.23.95:7788 124.193.167.1:3560 timer:(keepalive,9min44sec,0) TIME-WAIT 0 0 172.31.23.95:37021 111.161.68.235:80 timer:(timewait,16sec,0) TIME-WAIT 0 0 172.31.23.95:37023 111.161.68.235:80 timer:(timewait,23sec,0) TIME-WAIT 0 0 172.31.23.95:37020 111.161.68.235:80 timer:(timewait,13sec,0) TIME-WAIT 0 0 172.31.23.95:37033 111.161.68.235:80 timer:(timewait,59sec,0) TIME-WAIT 0 0 172.31.23.95:37028 111.161.68.235:80 timer:(timewait,40sec,0) TIME-WAIT 0 0 172.31.23.95:37031 111.161.68.235:80 timer:(timewait,50sec,0) TIME-WAIT 0 0 172.31.23.95:37025 111.161.68.235:80 timer:(timewait,30sec,0) TIME-WAIT 0 0 172.31.23.95:37030 111.161.68.235:80 timer:(timewait,47sec,0) TIME-WAIT 0 0 172.31.23.95:37019 111.161.68.235:80 timer:(timewait,9.588ms,0) TIME-WAIT 0 0 172.31.23.95:37026 111.161.68.235:80 timer:(timewait,33sec,0) TIME-WAIT 0 0 172.31.23.95:37032 111.161.68.235:80 timer:(timewait,56sec,0) TIME-WAIT 0 0 172.31.23.95:37027 111.161.68.235:80 timer:(timewait,37sec,0) TIME-WAIT 0 0 172.31.23.95:37022 111.161.68.235:80 timer:(timewait,20sec,0)
ss -lt -4[-6] # print IPv4 or IPv6 socket connections
# ss -lt6 State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 :::http :::* LISTEN 0 5 ::1:domain :::* LISTEN 0 5 fe80::408:35ff:fe6b:31f9:domain :::* LISTEN 0 128 :::munin :::* LISTEN 0 128 :::7788 :::* # root at shanker in /etc/nginx/sites-available [2:54:49] # ss -lt4 State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:http *:* LISTEN 0 5 127.0.0.1:domain *:* LISTEN 0 5 172.31.23.95:domain *:* LISTEN 0 128 *:https *:* LISTEN 0 3 *:1723 *:* LISTEN 0 128 *:8000 *:* LISTEN 0 128 *:8388 *:* LISTEN 0 50 127.0.0.1:mysql *:* LISTEN 0 128 *:7788 *:*
ss [ options ] [ state-filer ] [ address-filter ] # filter connections by tcp state
# ss -ant4 state listening Recv-Q Send-Q Local Address:Port Peer Address:Port 0 128 *:80 *:* 0 5 127.0.0.1:53 *:* 0 5 172.31.23.95:53 *:* 0 128 *:443 *:* 0 3 *:1723 *:* 0 128 *:8000 *:* 0 128 *:8388 *:* 0 50 127.0.0.1:3306 *:* 0 128 *:7788 *:* # root at shanker in /etc/nginx/sites-available [2:56:21] # ss -ant4 state established Recv-Q Send-Q Local Address:Port Peer Address:Port 0 436 172.31.23.95:7788 124.193.167.1:3560
state can be used by following:
1. established 2. syn-sent 3. syn-recv 4. fin-wait-1 5. fin-wait-2 6. time-wait 7. closed 8. close-wait 9. last-ack 10. closing 11. all - All of the above states 12. connected - All the states except for listen and closed 13. synchronized - All the connected states except for syn-sent 14. bucket - Show states, which are maintained as minisockets, i.e. time-wait and syn-recv. 15. big - Opposite to bucket state.
ss的高级用法，过滤地址和端口号，有点类似于tcpdump的用法：
过滤目标端口是80的或者源端口是1723的连接，dst后面要跟空格然后加“：”：
# ss -ant dst :80 or src :1723 State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 3 *:1723 *:* TIME-WAIT 0 0 172.31.23.95:37269 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37263 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37267 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37262 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37271 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37270 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37265 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37268 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37272 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37264 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37261 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37266 111.161.68.235:80
或者可以这样写ss -ant dport = :80 or sport = : 1723, dport后面必须有空格，“=”左右必须有空格，“:80"不能有空格：
# ss -ant dport = :80 or sport = :1723 State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 3 *:1723 *:* TIME-WAIT 0 0 172.31.23.95:37344 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37347 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37353 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37354 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37356 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37346 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37343 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37352 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37351 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37349 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37355 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37348 111.161.68.235:80 SYN-SENT 0 1 172.31.23.95:37358 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37357 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37350 111.161.68.235:80 TIME-WAIT 0 0 172.31.23.95:37345 111.161.68.235:80
地址筛选：
目标地址是111.161.68.235的连接
ss -ant dst 111.161.68.235
端口大小筛选：
源端口大于1024的端口：
ss sport gt 1024
后话：
前几日看tcp slow start 和congestion avoidance的资料，一直 在纠结这个mss，cwnd大小的问题，后来发现用ss -i命令就可以查看到该连接对应的cwnd，mss，rto，rtt的值大小，因为从内核2.6.32（not sure）之后cwnd的值是动态调整的，默认是10个mss大小，我截屏的连接都是连接到aws的一台ec2上的结果：
# ss -tai State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:http *:* rto:1000 mss:536 cwnd:10 LISTEN 0 5 127.0.0.1:domain *:* rto:1000 mss:536 cwnd:10 LISTEN 0 5 172.31.23.95:domain *:* rto:1000 mss:536 cwnd:10 LISTEN 0 128 *:https *:* rto:1000 mss:536 cwnd:10 LISTEN 0 3 *:1723 *:* rto:1000 mss:536 cwnd:10 LISTEN 0 128 *:8000 *:* rto:1000 mss:536 cwnd:10 LISTEN 0 128 *:8388 *:* rto:1000 mss:536 cwnd:10 LISTEN 0 50 127.0.0.1:mysql *:* rto:1000 mss:536 cwnd:10 LISTEN 0 128 *:7788 *:* rto:1000 mss:536 cwnd:10 TIME-WAIT 0 0 172.31.23.95:37489 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37482 111.161.68.235:http ESTAB 0 436 172.31.23.95:7788 124.193.167.1:3560 cubic wscale:2,7 rto:768 rtt:318.5/94 ato:40 mss:1380 cwnd:4 ssthresh:3 send 138.6Kbps unacked:4 retrans:0/40 rcv_rtt:248991 rcv_space:71960 TIME-WAIT 0 0 172.31.23.95:37491 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37480 111.161.68.235:http ESTAB 0 278 172.31.23.95:37493 111.161.68.235:http cubic wscale:7,7 rto:14592 rtt:304/152 mss:1448 cwnd:1 ssthresh:7 send 38.1Kbps unacked:1 retrans:1/4 lost:1 rcv_space:26883 TIME-WAIT 0 0 172.31.23.95:37487 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37490 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37481 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37488 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37483 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37492 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37484 111.161.68.235:http TIME-WAIT 0 0 172.31.23.95:37486 111.161.68.235:http LISTEN 0 128 :::http :::* rto:1000 mss:536 cwnd:10 LISTEN 0 5 ::1:domain :::* rto:1000 mss:536 cwnd:10 LISTEN 0 5 fe80::408:35ff:fe6b:31f9:domain :::* rto:1000 mss:536 cwnd:10 LISTEN 0 128 :::munin :::* rto:1000 mss:536 cwnd:10 LISTEN 0 128 :::7788 :::* rto:1000 mss:536 cwnd:10 TIME-WAIT 0 0 ::ffff:127.0.0.1:munin ::ffff:127.0.0.1:38260
ss命令帮助
# ss -h
Usage: ss [ OPTIONS ]
ss [ OPTIONS ] [ FILTER ]
-h, --help this message
-V, --version output version information
-n, --numeric don't resolve service names
-r, --resolve resolve host names
-a, --all display all sockets
-l, --listening display listening sockets
-o, --options show timer information
-e, --extended show detailed socket information
-m, --memory show socket memory usage
-p, --processes show process using socket
-i, --info show internal TCP information
-s, --summary show socket usage summary
-b, --bpf show bpf filter socket information
-4, --ipv4 display only IP version 4 sockets
-6, --ipv6 display only IP version 6 sockets
-0, --packet display PACKET sockets
-t, --tcp display only TCP sockets
-u, --udp display only UDP sockets
-d, --dccp display only DCCP sockets
-w, --raw display only RAW sockets
-x, --unix display only Unix domain sockets
-f, --family=FAMILY display sockets of type FAMILY
-A, --query=QUERY, --socket=QUERY
QUERY := {all|inet|tcp|udp|raw|unix|packet|netlink}[,QUERY]
-D, --diag=FILE Dump raw information about TCP sockets to FILE
-F, --filter=FILE read filter information from FILE
FILTER := [ state TCP-STATE ] [ EXPRESSION ]
Enjoy yourself.
参考连接：
http://www.cyberciti.biz/files/ss.html
http://www.binarytides.com/linux-ss-command/
http://stackoverflow.com/questions/11763376/difference-between-netstat-and-ss-in-linux
https://www.ttlsa.com/linux-command/ss-replace-netstat/