etcd备份还原方案,这种比较高级。
使用docker,自动化处理。
如果单节点备份,ETCD_ENDPOINTS一个即可。
如果多节点恢复,依次执行恢复脚本即可。
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: backup
namespace: kube-system
spec:
# activeDeadlineSeconds: 100
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
# Same image as in /etc/kubernetes/manifests/etcd.yaml
image: harbor.xxx.cn/3rd_part/k8s.gcr.io/etcd:3.3.10
env:
- name: ETCDCTL_API
value: "3"
- name: ETCD_ENDPOINTS
value: "https://1.1:2379,https://1.2:2379,https://1.3:2379"
command: ["/bin/sh"]
args: ["-c", "etcdctl --endpoints=${ETCD_ENDPOINTS} --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-ha-snapshot-$(date +%Y-%m-%d_%H:%M:%S_%Z).db"]
volumeMounts:
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
readOnly: true
- mountPath: /backup
name: backup
restartPolicy: OnFailure
nodeSelector:
node-role.kubernetes.io/master: ""
tolerations:
- key: "node-role.kubernetes.io/master"
effect: "NoSchedule"
hostNetwork: true
volumes:
- name: etcd-certs
hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
- name: backup
hostPath:
path: /tmp/etcd_backup/
type: DirectoryOrCreate
主要参考url:
https://www.cnblogs.com/chenqionghe/p/10622859.html
相当于把c,etcd单节点数据恢复操作三次即可。但要注意,在作数据恢复之前,千万一定肯定要停掉整个etcd集群的服务!!!!切记切记切记!!!!并且要平时或是找一个类似环境演练,作到胆大心细!!!
如果已有备份数据,在只有etcd数据损坏的下,可根据以下步骤进行恢复。
1, 将/etc/kubernetes/manifests/ kube-apiserver.yaml文件里的镜像版本更改,停止kube-api server服务。
2, 将/etc/kubernetes/manifests/ etcd.yaml文件里的镜像版本更改,停止etcd server服务。
3, 运行如下命令,将损坏的数据文件移至其它地方。
mv /var/lib/etcd/* /tmp/
4, 运行以下命令,以临时docker运行的方式,将数据从备份里恢复到/var/lib/etcd/。
docker run --rm \
-v '/tmp:/backup' \
-v '/var/lib/etcd:/var/lib/etcd' \
--env ETCDCTL_API=3 \
'k8s.gcr.io/etcd-amd64:3.1.12' \
/bin/sh -c "etcdctl snapshot restore '/backup/etcd-snapshot-xxx_UTC.db' ; mv /default.etcd/member/ /var/lib/etcd/"
[上面的命令中,假定我们已将待还原数据放置于/tmp/目录下]
5, 改回/etc/kubernetes/manifests/kube-apiserver.yaml文件里的镜像版本,恢复etcd server服务。
6, 改回/etc/kubernetes/manifests/etcd.yaml文件里的镜像版本,恢复kube-api server服务。