docker，consul，mesos集群日常维护

docker启动失败
root@jjh35:/home/qboxserver/boots-docker/log# cat docker.log-0102144556
time="2019-01-02T14:45:56.182146778+08:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING
[!]"
time="2019-01-02T14:45:56.185443427+08:00" level=info msg="libcontainerd: new containerd process, pid: 24887"
time="2019-01-02T14:45:57.228799012+08:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
time="2019-01-02T14:45:57.228909333+08:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"

解决办法：删除原来docker目录，启动自动重建
cd /disk1
mv docker /tmp/

然后启动即可
root@jjh630:/disk1# supervisorctl start dockerd
dockerd: started

dockerd启动失败，报错日志：
time="2019-03-01T14:52:09.091818748+08:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2019-03-01T14:52:09.093160163+08:00" level=info msg="libcontainerd: new containerd process, pid: 52587"
time="2019-03-01T14:52:10.164273335+08:00" level=fatal msg="Error starting daemon: error initializing graphdriver: \"/disk1/docker\" contains several valid graphdrivers: aufs, overlay2; Please cleanup or explicitly choose storage driver (-s )"
解决办法：
root@cd-dora-jp-6:/disk1# cd /disk1
root@cd-dora-jp-6:/disk1# rm -rf docker

删除docker目录失败，报错：
rm: cannot remove 'docker/aufs': Device or resource busy
文件占用，卸载挂载点：
root@cd-dora-jp-6:/disk1# cat /proc/mounts|grep docker
/dev/sdc1 /disk1/docker/overlay2 ext4 rw,noatime,data=ordered 0 0
/dev/sdc1 /disk1/docker/aufs ext4 rw,noatime,data=ordered 0 0

root@cd-dora-jp-6:/disk1# cat /proc/mounts|grep docker |awk '{print $2}'|while read line;do umount -l $line;done
root@cd-dora-jp-6:/disk1# df -h | egrep 'shm|overlay' | grep -v tmpfs | awk '{print $6}' | xargs umount

然后删除docker目录：
root@cd-dora-jp-6:/disk1# rm -rf docker
再启动dockerd：
root@cd-dora-jp-6:/disk1# supervisorctl start dockerd
dockerd: started

consul启动失败：
root@cd-dora-jp-6:/home/qboxserver/consul/log# cat consul.log-0301154443
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul client: Failed to start lan serf: Failed to create memberlist: Failed to start TCP listener. Err: listen tcp 10.24.34.241:8301: bind: cannot assign requested address
解决办法：bind_addr 改为本机内网 ip地址
vim /home/qboxserver/consul/current/conf/consul.conf
"bind_addr": "192.168.1.68",

mesos启动失败：ip改为本机ip地址
cd /home/qboxserver/mesos-agent/current/conf/mesos-agent
vim ip
192.168.1.68
保存退出启动

mesos清理脏数据方法：
root@xsgpu92:/disk1/mesos# curl -vX PUT "localhost:8500/v1/agent/service/deregister/boots:image.0c9dc975-f7a9-11e8-b21d-00e0ed50e991.1543919320179542264:HTTP"* Trying 127.0.0.1...

• Connected to localhost (127.0.0.1) port 8500 (#0)

  PUT /v1/agent/service/deregister/boots:image.0c9dc975-f7a9-11e8-b21d-00e0ed50e991.1543919320179542264:HTTP HTTP/1.1
  Host: localhost:8500
  User-Agent: curl/7.47.0
  Accept: /

  < HTTP/1.1 200 OK
  < Date: Wed, 02 Jan 2019 08:53:16 GMT
  < Content-Length: 0
  < Content-Type: text/plain; charset=utf-8