对k8s demo集群etcd的备份与恢复
之前的文章对k8s demo集群进行了CICD，以及监控配置，对于日常运维来说，对k8s集群的控制数据的备份也是必不可少的，所以本文对这个k8s demo集群的etcd进行了备份与恢复的测试演练。
1，配置etcdctl
在/etc/profile增加如下配置
export ETCDCTL_API=3
alias etcdctl='etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key '
查看etcd的状态数据
[root@VM-12-8-centos ~]# source /etc/profie
#查看状态
[root@VM-12-8-centos ~]# etcdctl endpoint status -w table
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://[127.0.0.1]:2379 | fc2c7cf28a83253f | 3.3.10 | 5.8 MB | true | 5 | 22421398 |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
#健康检查
[root@VM-12-8-centos ~]# etcdctl endpoint health -w table
https://[127.0.0.1]:2379 is healthy: successfully committed proposal: took = 1.073883ms
#etcd集群成员列表
[root@VM-12-8-centos ~]# etcdctl member list -w table
+------------------+---------+----------------+------------------------+------------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+----------------+------------------------+------------------------+
| fc2c7cf28a83253f | started | vm-12-8-centos | https://10.0.12.8:2380 | https://10.0.12.8:2379 |
+------------------+---------+----------------+------------------------+------------------------+查看etcd的相关配置文件，查看etcd的数据目录
[root@VM-12-8-centos manifests]# pwd
/etc/kubernetes/manifests
[root@VM-12-8-centos manifests]# cat etcd.yaml
......
- --data-dir=/var/lib/etcd
......
检查etcd的数据备份目录
[root@VM-12-8-centos etcd]# pwd
/var/lib/etcd
[root@VM-12-8-centos etcd]# tree member
member
├── snap
│ ├── 0000000000000005-0000000001555f8e.snap
│ ├── 0000000000000005-000000000155869f.snap
│ ├── 0000000000000005-000000000155adb0.snap
│ ├── 0000000000000005-000000000155d4c1.snap
│ ├── 0000000000000005-000000000155fbd2.snap
│ └── db
└── wal
├── 000000000000010e-00000000015074f9.wal
├── 000000000000010f-000000000151acf4.wal
├── 0000000000000110-000000000152e4f4.wal
├── 0000000000000111-0000000001541cef.wal
├── 0000000000000112-00000000015554ef.wal
└── 0.tmp2，etcd数据备份
建立一个etcd数据备份的目录/data/etcd_bak，再进行备份
[root@VM-12-8-centos etcd_bak]# etcdctl snapshot save /data/etcd_bak/etcd-snapshot-`date +%Y%m%d`.db
Snapshot saved at /data/etcd_bak/etcd-snapshot-20230401.db
[root@VM-12-8-centos etcd_bak]# ll
total 5656
-rw-r--r-- 1 root root 5787680 Apr 1 15:35 etcd-snapshot-20230401.db以上备份命令可以放在shell脚本中，配置crontab，每天凌晨的时候定时备份
3，模拟etcd数据丢失
将/var/lib/etcd/member改名，模拟etcd数据丢失
mv /var/lib/etcd/member /var/lib/etcd/member.bk
检查确认
[root@VM-12-8-centos etcd]# etcdctl member list -w table
Error: context deadline exceeded
[root@VM-12-8-centos etcd]# etcdctl endpoint health -w table
127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
[root@VM-12-8-centos etcd]# kubectl get po
No resources found.可以看到将member目录改名后，很快etcdctl查看etcd状态，用kubectl查看集群已经不正常了，获取不到相关数据
4，etcd备份恢复
需要注意的是备份恢复需要遵守以下的顺序
停止kube-apiserver --> 停止ETCD --> 恢复数据 --> 启动ETCD --> 启动kube-apiserve
由于我的k8s集群是采用kubeadm的方式安装，etcd和apiserver不属于系统服务，所以重启这2个进程采取静态pod重启的方式，即在/etc/kubernetes/manifests目录里移走apiserver，etcd的yml文件，再kill掉apiserver和etcd，待做了etcd数据恢复后，再分别移入etcd的yml文件，和apiserver的yml文件的方式来分别重启这2个进程
etcd备份数据恢复
[root@VM-12-8-centos etcd_bak]# etcdctl snapshot restore etcd-snapshot-20230401.db --data-dir=/var/lib/etcd
2023-04-01 16:04:03.430332 I | mvcc: restore compact to 19663311
2023-04-01 16:04:03.441067 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32启动etcd和apiserver
[root@VM-12-8-centos kubernetes]# mv etcd.yaml manifests/
[root@VM-12-8-centos etcd_bak]# p etcd
root 19873 19853 1 16:04 ? 00:00:00 etcd --advertise-client-urls=https://10.0.12.8:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://10.0.12.8:2380 --initial-cluster=vm-12-8-centos=https://10.0.12.8:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://10.0.12.8:2379 --listen-peer-urls=https://10.0.12.8:2380 --name=vm-12-8-centos --peer-cert-file=/etc/kubernetes/pki/etc/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
[root@VM-12-8-centos kubernetes]# mv kube-apiserver.yaml manifests/
[root@VM-12-8-centos etcd_bak]# p apiserver
root 20878 20858 99 16:05 ? 00:00:03 kube-apiserver --advertise-address=10.0.12.8 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.1.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key5，检查集群状态
[root@VM-12-8-centos etcd_bak]# etcdctl member list -w table
+------------------+---------+----------------+-----------------------+------------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+----------------+-----------------------+------------------------+
| 8e9e05c52164694d | started | vm-12-8-centos | http://localhost:2380 | https://10.0.12.8:2379 |
+------------------+---------+----------------+-----------------------+------------------------+
[root@VM-12-8-centos etcd_bak]# etcdctl endpoint health -w table
https://[127.0.0.1]:2379 is healthy: successfully committed proposal: took = 737.506µs
[root@VM-12-8-centos etcd_bak]# etcdctl endpoint status -w table
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://[127.0.0.1]:2379 | 8e9e05c52164694d | 3.3.10 | 5.8 MB | true | 2 | 611 |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
[root@VM-12-8-centos etcd_bak]# kubectl get ns
NAME STATUS AGE
default Active 156d
kube-node-lease Active 156d
kube-public Active 156d
kube-system Active 156d
kube-users Active 142d
monitoring Active 138d
[root@VM-12-8-centos etcd_bak]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-bccdc95cf-kwgjx 1/1 Running 0 156d
coredns-bccdc95cf-s52jg 1/1 Running 0 156d
etcd-vm-12-8-centos 1/1 Running 0 156d
kube-apiserver-vm-12-8-centos 1/1 Running 0 151d
kube-controller-manager-vm-12-8-centos 1/1 Running 7 137d