奇怪的ping, sendmsg: Operation not permitted

今天zabbix的一台机器总是报ping错误，结果ping了下，发现了个很奇怪的问题。

 
 

64 bytes from 192.168.0.98: icmp_seq=108 ttl=64 time=0.088 ms
ping: sendmsg: Operation not permitted
64 bytes from 192.168.0.98: icmp_seq=110 ttl=64 time=0.109 ms
64 bytes from 192.168.0.98: icmp_seq=111 ttl=64 time=0.148 ms
64 bytes from 192.168.0.98: icmp_seq=112 ttl=64 time=0.102 ms
64 bytes from 192.168.0.98: icmp_seq=113 ttl=64 time=0.078 ms
64 bytes from 192.168.0.98: icmp_seq=114 ttl=64 time=0.117 ms
64 bytes from 192.168.0.98: icmp_seq=115 ttl=64 time=0.122 ms
64 bytes from 192.168.0.98: icmp_seq=116 ttl=64 time=0.093 ms
64 bytes from 192.168.0.98: icmp_seq=117 ttl=64 time=0.110 ms
64 bytes from 192.168.0.98: icmp_seq=118 ttl=64 time=0.138 ms
64 bytes from 192.168.0.98: icmp_seq=119 ttl=64 time=0.057 ms
64 bytes from 192.168.0.98: icmp_seq=120 ttl=64 time=0.091 ms
64 bytes from 192.168.0.98: icmp_seq=121 ttl=64 time=0.084 ms
64 bytes from 192.168.0.98: icmp_seq=122 ttl=64 time=0.091 ms
64 bytes from 192.168.0.98: icmp_seq=123 ttl=64 time=0.088 ms
64 bytes from 192.168.0.98: icmp_seq=124 ttl=64 time=0.103 ms
ping: sendmsg: Operation not permitted
64 bytes from 192.168.0.98: icmp_seq=126 ttl=64 time=0.068 ms
64 bytes from 192.168.0.98: icmp_seq=127 ttl=64 time=0.084 ms
ping: sendmsg: Operation not permitted

网上说这种是因为iptables引起的，不过我查了iptable并没有问题

被ping机器的iptable

 

[root@DB_98 ~]# iptables -v -L
Chain INPUT (policy DROP 319K packets, 38M bytes)
 pkts bytes target     prot opt in     out     source               destination
 312M   51G ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
13279  946K ACCEPT     all  --  lo     any     anywhere             anywhere
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere            udp dpt:http
  200  9756 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:http
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere            udp dpt:ssh
 1129 64668 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:ssh
    0     0 ACCEPT     tcp  --  any    any     DB_98                anywhere            tcp dpt:smtp
    0     0 ACCEPT     udp  --  any    any     DB_98                anywhere            udp dpt:smtp
1729K  146M ACCEPT     icmp --  any    any     anywhere             anywhere
67630 4058K ACCEPT     tcp  --  any    any     localhost            anywhere            tcp dpt:5666
    0     0 ACCEPT     udp  --  any    any     localhost            anywhere            udp dpt:5666
8728K  524M ACCEPT     tcp  --  any    any     localhost/24         anywhere            tcp dpt:mysql
    0     0 ACCEPT     udp  --  any    any     localhost/24         anywhere            udp dpt:mysql

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 281M packets, 589G bytes)
 pkts bytes target     prot opt in     out     source               destination

主动ping机器的iptable

 

[root@bj_89 zabbix]# iptables -v -L
Chain INPUT (policy DROP 295K packets, 35M bytes)
 pkts bytes target     prot opt in     out     source               destination
  16G 3477G ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
44610 2677K ACCEPT     all  --  lo     any     anywhere             anywhere
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere            udp dpt:http
  181  8760 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:http
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere            udp dpt:ssh
 1129 64620 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:ssh
    0     0 ACCEPT     tcp  --  any    any     app3                 anywhere            tcp dpt:smtp
    0     0 ACCEPT     udp  --  any    any     app3                 anywhere            udp dpt:smtp
2795K  235M ACCEPT     icmp --  any    any     anywhere             anywhere
54310 3259K ACCEPT     tcp  --  any    any     localhost            anywhere            tcp dpt:nrpe
    0     0 ACCEPT     udp  --  any    any     localhost            anywhere            udp dpt:5666
 859M   52G ACCEPT     tcp  --  any    any     192.168.0.0/24       anywhere            tcp dpt:7749
    0     0 ACCEPT     udp  --  any    any     localhost/24         anywhere            udp dpt:7749
 701M   42G ACCEPT     tcp  --  any    any     localhost/24         anywhere            tcp dpt:osm-appsrvr
    0     0 ACCEPT     udp  --  any    any     localhost/24         anywhere            udp dpt:osm-appsrvr

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 13G packets, 8534G bytes)
 pkts bytes target     prot opt in     out     source               destination

似乎看不出问题，两台机器的CPU和LOAD很低，两台机器的ESTABLISHED才1500左右，因为是在线服务器，所以这值也正常。

 

经高手指点

通过看系统日志，发现的确有问题

Jul 13 09:40:21 bj_89 kernel: printk: 162 messages suppressed.
Jul 13 09:40:21 bj_89 kernel: ip_conntrack: table full, dropping packet.
Jul 13 09:40:27 bj_89 kernel: printk: 54 messages suppressed.
Jul 13 09:40:27 bj_89 kernel: ip_conntrack: table full, dropping packet.
Jul 13 09:40:31 bj_89 kernel: printk: 297 messages suppressed.
Jul 13 09:40:31 bj_89 kernel: ip_conntrack: table full, dropping packet.
Jul 13 09:40:36 bj_89 kernel: printk: 431 messages suppressed.
Jul 13 09:40:36 bj_89 kernel: ip_conntrack: table full, dropping packet.
Jul 13 09:40:41 bj_89 kernel: printk: 384 messages suppressed.
Jul 13 09:40:41 bj_89 kernel: ip_conntrack: table full, dropping packet.

 

[root@bj_89 zabbix]# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
65536

MD 都这么大了，还不够用，于是加大了。

echo '655360' >> /proc/sys/net/ipv4/netfilter/ip_conntrack_max

同时修改/etc/sysctl.conf对应的值

 

 

.至于为啥会这样，还是没弄清楚。。。以后再研究