硬件配置:
阿里云服务器(CPU:4核 内存:4GB 数据盘:450G 带宽:5Mbps).
系统配置:
iluckysi@ILUCKYSI-PC:/etc# cat issue
Ubuntu 12.04.5 LTS \n \l
异常信息:
cpu连续n天使用率为100%.
查找原因:
查看系统cpu和memory使用情况.
iluckysi@ILUCKYSI-PC:/etc# top
top - 11:27:43 up 303 days, 3:31, 2 users, load average: 5.06, 5.03, 5.05
Tasks: 180 total, 6 running, 174 sleeping, 0 stopped, 0 zombie
Cpu(s): 69.8%us, 30.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 4051260k total, 3592692k used, 458568k free, 227332k buffers
Swap: 2187260k total, 70940k used, 2116320k free, 2351156k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19608 daemon 20 0 29948 2768 1060 R 97 0.1 160766:24 perl
26598 daemon 20 0 23740 1964 708 R 85 0.0 168071:12 perl
4020 daemon 20 0 832 8 0 R 80 0.0 237454:42 pdflush
19612 daemon 20 0 29948 2524 864 R 72 0.1 177923:31 perl
31348 daemon 20 0 23740 728 400 R 62 0.0 167897:47 perl
25444 root 20 0 3321m 550m 12m S 2 13.9 45:24.95 java
11739 daemon 20 0 30216 344 192 S 1 0.0 24:40.73 perl
然后按c,查看COMMAND对应的实际进程.
iluckysi@ILUCKYSI-PC:/etc# top
top - 11:29:20 up 303 days, 3:32, 2 users, load average: 5.01, 5.02, 5.05
Tasks: 185 total, 6 running, 179 sleeping, 0 stopped, 0 zombie
Cpu(s): 65.3%us, 34.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.2%si, 0.0%st
Mem: 4051260k total, 3594584k used, 456676k free, 227332k buffers
Swap: 2187260k total, 70940k used, 2116320k free, 2351272k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4020 daemon 20 0 832 8 0 R 99 0.0 237456:05 [pdflush]
31348 daemon 20 0 23740 728 400 R 78 0.0 167898:59 klogd -x
19608 daemon 20 0 29948 2768 1060 R 76 0.1 160767:42 /usr/sbin/sshd -D
19612 daemon 20 0 29948 2524 864 R 74 0.1 177924:48 /usr/sbin/acpid
26598 daemon 20 0 23740 1964 708 R 68 0.0 168072:28 klogd -x
25444 root 20 0 3321m 550m 12m S 2 13.9 45:25.68 /usr/lib/jdk/jdk1.7.0_71/jre/bin/java -Djava.util.logging.config.file=/usr/lib/tomcat/apache-tomcat-7.0.32/conf/logging.prope
398 root 20 0 679m 12m 8224 S 1 0.3 405:19.89 /usr/local/aegis/aegis_client/aegis_00_73/AliYunDun
12731 root 20 0 882m 9000 6284 S 1 0.2 124:16.60 /usr/local/aegis/alihids/AliHids
更多关于top指令的使用方法,请上网查询......
异常分析:
查看cpu使用率过高的这几个进程关联的操作:ls -al:查看某个进程打开的文件的权限.
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/4020/fd/
total 0
dr-x------ 2 daemon daemon 0 Jun 12 16:45 .
dr-xr-xr-x 7 daemon daemon 0 May 1 09:56 ..
lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[68038509]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> /dev/null (deleted)
l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> /dev/null (deleted)
lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> /var/tmp/.fontUnix (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/31348/fd/
total 0
dr-x------ 2 daemon daemon 0 Jun 12 16:45 .
dr-xr-xr-x 7 daemon daemon 0 Jan 23 16:02 ..
lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[634666134]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> pipe:[634666138]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 Jun 12 16:45 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 Jun 12 16:45 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> pipe:[634666141]
lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> socket:[637677664]
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/19608/fd/
total 0
dr-x------ 2 daemon daemon 0 May 6 00:44 .
dr-xr-xr-x 7 daemon daemon 0 May 1 09:56 ..
lr-x------ 1 daemon daemon 64 May 6 00:44 0 -> pipe:[1849120182]
l-wx------ 1 daemon daemon 64 May 6 00:44 1 -> pipe:[1849120186]
l-wx------ 1 daemon daemon 64 May 6 00:44 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 May 6 00:44 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 May 6 00:44 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 May 6 00:44 2 -> pipe:[1849120188]
lrwx------ 1 daemon daemon 64 May 6 00:44 3 -> socket:[1335075729]
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/19612/fd/
total 0
dr-x------ 2 daemon daemon 0 May 6 00:44 .
dr-xr-xr-x 7 daemon daemon 0 May 1 09:56 ..
lr-x------ 1 daemon daemon 64 May 6 00:44 0 -> pipe:[1849120182]
l-wx------ 1 daemon daemon 64 May 6 00:44 1 -> pipe:[1849120186]
l-wx------ 1 daemon daemon 64 May 6 00:44 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 May 6 00:44 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 May 6 00:44 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 May 6 00:44 2 -> pipe:[1849120188]
lrwx------ 1 daemon daemon 64 May 6 00:44 3 -> socket:[2163495078]
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/26598/fd/
total 0
dr-x------ 2 daemon daemon 0 Jun 12 16:45 .
dr-xr-xr-x 7 daemon daemon 0 Jan 23 16:02 ..
lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[3453697476]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> pipe:[3453697477]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 Jun 12 16:45 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 Jun 12 16:45 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> pipe:[3453697478]
lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> socket:[525581618]
iluckysi@ILUCKYSI-PC:/var/tmp#
同时可以使用如下指令进行分析: lsof -p pid:查看正在运行中的进程打开了哪些文件,目录和套接字,是系统监测工具之一.
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 4020
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
pdflush 4020 daemon cwd DIR 3,1 12288 1835016 /usr/sbin
pdflush 4020 daemon rtd DIR 3,1 4096 2 /
pdflush 4020 daemon txt REG 3,1 562008 264410 /var/tmp/pdflush (deleted)
pdflush 4020 daemon 0r FIFO 0,8 0t0 68038509 pipe
pdflush 4020 daemon 1w CHR 1,3 0t0 4782 /dev/null (deleted)
pdflush 4020 daemon 2w CHR 1,3 0t0 4782 /dev/null (deleted)
pdflush 4020 daemon 3u REG 3,1 0 263762 /var/tmp/.fontUnix (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 31348
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 31348 daemon cwd DIR 3,1 4096 2097153 /tmp
perl 31348 daemon rtd DIR 3,1 4096 2 /
perl 31348 daemon txt REG 3,1 10352 1841631 /usr/bin/perl
perl 31348 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl 31348 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl 31348 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl 31348 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl 31348 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl 31348 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl 31348 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl 31348 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1
perl 31348 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl 31348 daemon 0r FIFO 0,8 0t0 634666134 pipe
perl 31348 daemon 1w FIFO 0,8 0t0 634666138 pipe
perl 31348 daemon 2w FIFO 0,8 0t0 634666141 pipe
perl 31348 daemon 3u IPv4 637677664 0t0 TCP 110.76.39.140:44833->209.92.176.14:http (ESTABLISHED)
perl 31348 daemon 11w REG 3,1 138468379 661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl 31348 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl 31348 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 19608
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 19608 daemon cwd DIR 3,1 4096 2097153 /tmp
perl 19608 daemon rtd DIR 3,1 4096 2 /
perl 19608 daemon txt REG 3,1 10352 1841631 /usr/bin/perl
perl 19608 daemon mem REG 3,1 105288 1314564 /lib/x86_64-linux-gnu/libresolv-2.15.so
perl 19608 daemon mem REG 3,1 31104 1314576 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
perl 19608 daemon mem REG 3,1 52120 1314561 /lib/x86_64-linux-gnu/libnss_files-2.15.so
perl 19608 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl 19608 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl 19608 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl 19608 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl 19608 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl 19608 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl 19608 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl 19608 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1
perl 19608 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl 19608 daemon 0r FIFO 0,8 0t0 1849120182 pipe
perl 19608 daemon 1w FIFO 0,8 0t0 1849120186 pipe
perl 19608 daemon 2w FIFO 0,8 0t0 1849120188 pipe
perl 19608 daemon 3u IPv4 1335075729 0t0 TCP 110.76.39.140:53110->119.68.205.1:smtp (ESTABLISHED)
perl 19608 daemon 11w REG 3,1 138471419 661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl 19608 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl 19608 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 19612
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 19612 daemon cwd DIR 3,1 4096 2097153 /tmp
perl 19612 daemon rtd DIR 3,1 4096 2 /
perl 19612 daemon txt REG 3,1 10352 1841631 /usr/bin/perl
perl 19612 daemon mem REG 3,1 105288 1314564 /lib/x86_64-linux-gnu/libresolv-2.15.so
perl 19612 daemon mem REG 3,1 31104 1314576 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
perl 19612 daemon mem REG 3,1 52120 1314561 /lib/x86_64-linux-gnu/libnss_files-2.15.so
perl 19612 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl 19612 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl 19612 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl 19612 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl 19612 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl 19612 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl 19612 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl 19612 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1
perl 19612 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl 19612 daemon 0r FIFO 0,8 0t0 1849120182 pipe
perl 19612 daemon 1w FIFO 0,8 0t0 1849120186 pipe
perl 19612 daemon 2w FIFO 0,8 0t0 1849120188 pipe
perl 19612 daemon 3u IPv4 2163495078 0t0 TCP 110.76.39.140:43416->119.68.205.1:smtp (ESTABLISHED)
perl 19612 daemon 11w REG 3,1 138471723 661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl 19612 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl 19612 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 26598
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 26598 daemon cwd DIR 3,1 4096 2097153 /tmp
perl 26598 daemon rtd DIR 3,1 4096 2 /
perl 26598 daemon txt REG 3,1 10352 1841631 /usr/bin/perl
perl 26598 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl 26598 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl 26598 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl 26598 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl 26598 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl 26598 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl 26598 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl 26598 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1
perl 26598 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl 26598 daemon 0r FIFO 0,8 0t0 3453697476 pipe
perl 26598 daemon 1w FIFO 0,8 0t0 3453697477 pipe
perl 26598 daemon 2w FIFO 0,8 0t0 3453697478 pipe
perl 26598 daemon 3u IPv4 525581618 0t0 TCP 110.76.39.140:34176->210.253.114.69:81 (ESTABLISHED)
perl 26598 daemon 11w REG 3,1 138472179 661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl 26598 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl 26598 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp#
由上面的分析得知,除了4020进程,其余进程都和httpd有关系.
并且除了4020进程,其余所有的进程都对外有一个TCP连接,查看连接的ip,都是来自国外.
深入分析:
在上面的输出中,我们看到了.fontUnix,查询apache的error.log,看到如下记录.
[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] Connecting to 85.236.52.116:80...
[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] connected.
[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] HTTP request sent, awaiting response...
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 200 OK
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] Length: 1018 [text/plain]
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] Saving to: `/var/tmp/.font-unix.sh'
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102]
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 0K 100% 199M=0s
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102]
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 2014-11-22 22:31:33 (199 MB/s) - `/var/tmp/.font-unix.sh' saved [1018/1018]\
解决方法:
将上面分析出的的有问题的进程强制杀死.
iluckysi@ILUCKYSI-PC:~# kill -9 4020
iluckysi@ILUCKYSI-PC:~# kill -9 31348
iluckysi@ILUCKYSI-PC:~# kill -9 26598
iluckysi@ILUCKYSI-PC:~# kill -9 19612
iluckysi@ILUCKYSI-PC:/tmp# kill -9 19608
此时到阿里云平台查看服务器cpu使用率,发现cpu使用率降下来了.
最终方案:
阿里云Linux Bash严重漏洞修复紧急通知:http://bbs.aliyun.com/read/176977.html
升级系统版本,升级bash版本.
