使用 docker 后出现的网络异常现象
硬件:
[root@sh-storage-128204 ~]# dmidecode -t system
dmidecode 2.12
SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: To be filled by O.E.M.
Product Name: Tecal RH2288H V2-24S
Version: V100R002
Serial Number: 2102310QPE10E9000146
UUID: 4A190814-D21D-B211-8DC0-000000821800
Wake-up Type: Power Switch
SKU Number: Type1Sku0
Family: Type1Family
环境:
[root@sh-storage-128204 ~]# docker info
Containers: 4
Images: 153
Storage Driver: devicemapper
Pool Name: docker-8:17-1075308946-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.478 GB
Data Space Total: 107.4 GB
Data Space Available: 103.9 GB
Metadata Space Used: 6.689 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.141 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 24
Total Memory: 125.7 GiB
Name: sh-storage-128204.sh.vclound.com
ID: QV4X:VHQE:EMOI:4TBJ:FZ6K:3N3C:A64Y:PRYR:X2QZ:HHUB:OTND:ZSFF
用法:
用于搭建 docker private registry
架构:
nginx <- 本地 docker
|
|---------------+-------------------|
registry registry registry <- 本地 docker
|---------------+-------------------|
|
ceph 集群 (rados) <- 另外一个集群问题:
搭建了 docker, 启动成功后, 发现上传镜像时偶尔会出现连接卡住问题
检测与 mon 连接发现:
[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200
Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:43301 > 10.198.128.200:6789 S ttl=48 id=11619 iplen=44 seq=2271879518 win=1024 <mss 1460>
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:43302 > 10.198.128.200:6789 S ttl=45 id=9421 iplen=44 seq=2271945055 win=1024 <mss 1460>
Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)
Host is up (0.000091s latency).
PORT STATE SERVICE
6789/tcp filtered ibm-db2-admin
MAC Address: 90:E2:BA:85:21:28 (Intel Corporate)
Nmap done: 1 IP address (1 host up) scanned in 0.29 seconds
[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200
Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:48544 > 10.198.128.200:6789 S ttl=54 id=17389 iplen=44 seq=1275261731 win=1024 <mss 1460>
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:48545 > 10.198.128.200:6789 S ttl=53 id=40168 iplen=44 seq=1275327266 win=1024 <mss 1460>
Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)
Host is up (0.000092s latency).
PORT STATE SERVICE
6789/tcp filtered ibm-db2-admin
MAC Address: 90:E2:BA:85:21:28 (Intel Corporate)
Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds
[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200
Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST
Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)
Host is up (0.00039s latency).
PORT STATE SERVICE
6789/tcp open ibm-db2-admin
MAC Address: 90:E2:BA:85:21:28 (Intel Corporate)
Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds与 mon 连接时, 会经常出现 6789/tcp filtered ibm-db2-admin 异常现象
另外, 参考系统日志
会出现下面异常 (过滤了大部分, 只显示重要部分信息
/var/log/messages
Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet
Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet
Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet
....
Jan 20 09:49:44 sh-storage-128204 kernel: net_ratelimit: 137 callbacks suppressed
Jan 20 09:49:49 sh-storage-128204 kernel: net_ratelimit: 166 callbacks suppressed
Jan 20 09:49:54 sh-storage-128204 kernel: net_ratelimit: 64 callbacks suppressed
Jan 20 09:50:01 sh-storage-128204 kernel: net_ratelimit: 132 callbacks suppressedping 也会出现异常现场
[root@sh-storage-128204 ~]# ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
ping: sendmsg: Operation not permitted原因: 当前的网络连接数过大, 导致内核无法响应
解决方法:
修改 /et/sysctl.conf 进行内核参数优化
[root@sh-storage-128204 ~]# sysctl -p
vm.swappiness = 10
net.ipv4.ip_forward = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 8192
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.netfilter.nf_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 54000
net.netfilter.nf_conntrack_generic_timeout = 120