docker容器监控&自动恢复

关于实现对docker容器监控以及自动恢复，这里介绍两种实现方案。

方案1：

实现思路：
找到（根据正则表达式）所有待监控的docker容器，此处筛选逻辑根据docker运行状态找到已停止（Exit）类型的容器

代码清单1：

#!/bin/bash

# 获取时间信息，格式为："年-月-日T时:分:秒"
ctime1=`date +%Y-%m-%dT%k:%M:%S`

echo "--------------------------------------------------- Script start at : $ctime1 "

# 正则表达式匹配所有已biz-aaa/biz-bbb/biz-ccc开头，且以":v9"结尾的镜像名，结果中排除容器"container_to_be_excluded"，获取位于索引为18位置上的docker名称
#set1=($(sudo docker ps -a | grep -E "biz-(aaa|bbb|ccc).*:v9" | grep -v ' container_to_be_excluded' | awk '{print $18}'))

# 正则表达式匹配所有已biz-aaa/biz-bbb/biz-ccc开头，且以":v9"结尾的镜像名，且进一步过滤出已停止的容器（status = "Exit"）,此时其docker名位于索引为16的位置处
set1=($(sudo docker ps -a | grep -E "biz-(aaa|bbb|ccc).*:v9" | grep "Exit" | awk '{print $16}'))

# 获取数组长度
set1_size=${#set1[@]}

# A).判断数组长度是否为0，注意数字的判断请使用:
##  -eq : 相等
##  -ne : 不同
# B).如果判断字符串，则为：
##  =   : 相等
##  !=  : 不同
if [[ $set1_size -eq 0 ]]; then
    echo ">>>> All business docker processes are health!"
else 
    echo ">>>> Some business docker processes are not health, will be restarted!"
    sleep 60;
	
	# 迭代数组，可以使用${set1[@]}，或${set1[?]}
	# for循环如果一行搞定，可以这样使用： for i in ${set1[@]};do ${CODE_LOGICAL}; done
    for i in ${set1[@]};do 
        echo ">>>>>>>> Docker instance - $i will be restarted right now"
        echo ">>>>>>>> Will execute command : docker restart $i"
        docker restart $i

        # Wait some time for resource to be released
        sleep 30
        
        # Re-check if docker process is up
        set2=($(sudo docker ps -a | grep $i | grep "Exit" | awk '{print $16}'))
        set2_size=${#set2[@]}
        if (( set2_size != 0 )); then 
            docker restart $i
        fi
    done
    
fi


ctime2=`date +%Y-%m-%dT%k:%M:%S`
echo "--------------------------------------------------- Script end   at : $ctime2 "

方案2：

实现思路：
找到（根据正则表达式）所有待监控的docker容器，"docker ps -a"匹配出所有需要监控的容器列表，"docker ps"匹配出当前存活的容器列表，差集即为有问题需要重启的容器

代码清单2：

#!/bin/bash


set1=($(sudo docker ps -a | grep -E "biz-(aaa|bbb|ccc).*:v9" | grep -v ' container_to_be_excluded' | awk '{print $18}'))
set2=($(sudo docker ps | grep -E "biz-(aaa|bbb|ccc).*:v9" | grep -v ' container_to_be_excluded' | awk '{print $18}'))

set1_size=${#set1[@]}
set2_size=${#set2[@]}

# Sort sets
printf "%s\n" "${set1[@]}" | sort > set1_sorted.txt
printf "%s\n" "${set2[@]}" | sort > set2_sorted.txt

difference=($(comm -23 set1_sorted.txt set2_sorted.txt))

if [[ $set1_size -eq $set2_size ]]; then
    echo ">>>> All business docker processes are health!"
else 
    echo ">>>> Some business docker processes are not health, will be restarted!"
    for i in ${difference[@]};do 
        echo ">>>>>>>> Docker instance - $i will be restarted right now"
        docker restart $i
        #echo "-------------------------" $i
    done
    
fi

rm -f set1_sorted.txt set2_sorted.txt

配置crontab调度，实现简单定时调度

crontab -e : 编辑crontab
查看已配置的crontab：crontab -l

*/10 * * * * /aaa/bbb/my_docker_ha.sh >> /aaa/bbb/logs/my_docker_ha.log 2>&1