NGINX日志增大过快的排错过程
前些天服务器调整,重新配置了NGINX和keepalived,服务正常运行,可是没过几天发现NGINX的日志文件急剧变大,
1,查看了日志文件/var/log/目录下面的所有日志文件大小都正常,只有messages和messages-201808{19,26}两个文件过大,
2.查看日志内容:tail -n 100 /var/log/messages,可见是keepalived出了问题和内核相关的问题
[root@liwai5 ipvsadm-1.26]# tail -n 100 /var/log/messages
Aug 28 16:31:01 liwai5 modprobe: FATAL: Error inserting ip_vs (/lib/modules/2.6.32-573.12.1.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Aug 28 16:31:01 liwai5 kernel: ip_vs: Unknown symbol icmpv6_send
Aug 28 16:31:01 liwai5 Keepalived_healthcheckers[29998]: IPVS: Can't initialize ipvs: Protocol not available
Aug 28 16:31:01 liwai5 Keepalived[17362]: Healthcheck child process(29998) died: Respawning
Aug 28 16:31:01 liwai5 Keepalived[17362]: Starting Healthcheck child process, pid=30001
Aug 28 16:31:01 liwai5 kernel: ip_vs: Unknown symbol ip6_local_out
Aug 28 16:31:01 liwai5 kernel: ip_vs: Unknown symbol ip6_route_me_harder
Aug 28 16:31:01 liwai5 kernel: ip_vs: Unknown symbol ipv6_dev_get_saddr
Aug 28 16:31:01 liwai5 kernel: ip_vs: Unknown symbol ip6_route_output
Aug 28 16:31:01 liwai5 modprobe: FATAL: Error inserting ip_vs (/lib/modules/2.6.32-573.12.1.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko): Unknown symbol in module, or unknown parameter (see dmesg)
3.查看dmesg,看出和ip_vs相关,
keepalived[30069]: segfault at 0 ip 000000000040ea38 sp 00007fffe31c51c0 error 4 in keepalived[400000+49000]
ip_vs: Unknown symbol ip6_local_out
ip_vs: Unknown symbol ip6_route_me_harder
ip_vs: Unknown symbol ipv6_dev_get_saddr
ip_vs: Unknown symbol ip6_route_output
ip_vs: Unknown symbol icmpv6_send
keepalived[30072]: segfault at 0 ip 000000000040ea38 sp 00007fffe31c51c0 error 4 in keepalived[400000+49000]
ip_vs: Unknown symbol ip6_local_out
ip_vs: Unknown symbol ip6_route_me_harder
ip_vs: Unknown symbol ipv6_dev_get_saddr
ip_vs: Unknown symbol ip6_route_output
ip_vs: Unknown symbol icmpv6_send
keepalived[30075]: segfault at 0 ip 000000000040ea38 sp 00007fffe31c51c0 error 4 in keepalived[400000+49000]
4,可见是keepalived出了问题和ip_vs相关的内容,发现我的主机没有安装ip_vs模块
modprode ipvsadm 没有正常的返回结果,说明没有加载成功,应该是没有安装ipvsadm
5, 安装ipvsadm,yum -y install ipvsadm
6,根据提示,没有开启ipv6的可能很大,查看IP信息,果然没有开启ipv6
1: lo:
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: em1:
link/ether **:2b:2b:69:8a:b6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.14/24 brd 192.168.1.255 scope global em1
7,开启CentOS6的ipv6的步骤:
一、开启IPV6命令：sysctl -w net.ipv6.conf.all.disable_ipv6=0
[root@VM_centos ~]# sysctl -w net.ipv6.conf.all.disable_ipv6=0
error: "net.ipv6.conf.all.disable_ipv6" is an unknown key -->内核参数没加载，则会报这个错(注意：centos6.2/7.0/7.1版本的默认是加载的，直接运行这个命令便可以成功，其它centos版本都得重启主机来加载，往下看...)
二、原来还要在配置文件中/etc/modprobe.d/ipv6.conf 修改内核参数，options ipv6 disable=0，设置为0表示打开这个参数。
[root@VM_centos ~]# cat /etc/modprobe.d/ipv6.conf
# Anaconda disabling ipv6
options ipv6 disable=0
三、设置了还不行再操作第一步，还是报错，原来必须要重启云主机刚修改的内核参数才会加载生效。重启云主机后，再执行：sysctl -w net.ipv6.conf.all.disable_ipv6=0 就OK了。（最好把这条命令写在/etc/rc.local配置文件里，开机自动执行，不然重启后，又会默认关闭了。）
查看下设置生效的方法：
[root@VM_centos ~]# sysctl -a|grep ipv6|grep all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
再执行下ifconfig看下网卡，就会发现多了一个IPV6的地址。如第一段的开启效果。
注意：上述方法适用于centos6.2、centos6.3、centos6.4、centos6.5 centos6.6
centos5.8/5.11略有不同，IPV6内核参数文件在/etc/modprobe.conf中修改。
8,重启一下iptables
iptables-save > /etc/sysconfig/iptables
service iptables restart
9,
再查看ip a
[root@t14 log]# ip a
1: lo:
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1:
link/ether **:2b:2b:69:8a:b6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.14/24 brd 192.168.1.255 scope global em1
inet6 fe80::862b:2bff:fe69:8ab6/64 scope link
valid_lft forever preferred_lft forever
10,查看lsmod |grep ip_vs,查看/var/log/messages正常,排查结束.
[root@t14 opt]# lsmod |grep ip_vs
ip_vs 126534 0
ipv6 335525 71 ip_vs
libcrc32c 1246 1 ip_vs有结果,
tail /var/log/messages 返回的结果正常,不再是是上面显示的重复的日志输出了.cha