ETCD数据备份与恢复验证

ETCD数据备份与恢复验证

一、单机

说明：执行etcd备份数据的恢复的机器必须和原先etcd所在机器一致

1、单机备份

etcdctl --endpoints="https://10.25.72.62:2379" \
        --cert=/etc/etcd/ssl/etcd.pem \
        --key=/etc/etcd/ssl/etcd-key.pem \
        --cacert=/etc/kubernetes/ssl/ca.pem \
        snapshot save snapshot.db

2、单机数据恢复

[root@SZD-L0105331 ~]# systemctl stop etcd   # 停止etcd服务
[root@SZD-L0105331 ~]# export ETCDCTL_API=3; # 使用ETCDCTL API 3
[root@SZD-L0105331 ~]# etcdctl  snapshot restore snapshot.db \
        --name=SZD-L0105331 \
        --endpoints=https://10.25.72.62:2379 \
        --cacert=/etc/kubernetes/ssl/ca.pem \
        --cert=/etc/etcd/ssl/etcd.pem \
        --key=/etc/etcd/ssl/etcd-key.pem  \
        --initial-cluster=SZD-L0105331=https://10.25.72.62:2380  \
        --initial-advertise-peer-urls=https://10.25.72.62:2380 \
        --initial-cluster-token=etcd-cluster-0 \
        --data-dir=/var/lib/etcd4   # 注意--data-dir参数,以下步骤需要使用

[root@SZD-L0105331 ~]# # 修改etcd启动参数--data-dir指向上一步的数据回复目录，一般在/etc/etcd/etcd文件中

[root@SZD-L0105331 ~]# cat /etc/etcd/etcd  # 完整参数文件如下
ETCD_OPTIONS="--name=SZD-L0105331 \
   --client-cert-auth=true \
   --cert-file=/etc/etcd/ssl/etcd.pem \
   --key-file=/etc/etcd/ssl/etcd-key.pem \
   --peer-cert-file=/etc/etcd/ssl/etcd.pem \
   --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
   --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
   --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
   --initial-advertise-peer-urls=https://10.25.72.62:2380 \
   --listen-peer-urls=https://10.25.72.62:2380 \
   --listen-client-urls=https://10.25.72.62:2379,https://127.0.0.1:2379 \
   --advertise-client-urls=https://10.25.72.62:2379 \
   --initial-cluster-token=etcd-cluster-0 \
   --initial-cluster=SZD-L0105331=https://10.25.72.62:2380 \
   --initial-cluster-state=new \
   --data-dir=/var/lib/etcd"

[root@SZD-L0105331 ~]# systemctl start etcd # 启动etcd服务

二、集群

1、模拟写入数据到Etcd集群

# 使用API 3写入数据库
[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem put /name/1 zxg
# 读取数据
[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem get  /name/1
2018-08-16 20:47:13.017586 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
/name/1
zxg

# 使用API 2写入数据库
[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=2 etcdctl --cert-file /root/etcd-cluster/etcd.pem --ca-file /root/etcd-cluster/ca.pem --key-file /root/etcd-cluster/etcd-key.pem --endpoints="https://10.25.73.25:2379,https://10.25.73.150:2379,https://10.25.84.251:2379" set /name1 zxg1
2018-08-16 18:27:35.060559 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
zxg1
# 读取数据
[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=2 etcdctl --cert-file /root/etcd-cluster/etcd.pem --ca-file /root/etcd-cluster/ca.pem --key-file /root/etcd-cluster/etcd-key.pem --endpoints="https://10.25.73.25:2379,https://10.25.73.150:2379,https://10.25.84.251:2379" get /name1
2018-08-16 18:27:47.884908 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
zxg1

2、备份etcd数据

[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem  snapshot save mysnapshot.db
2018-08-16 20:50:47.206334 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Snapshot saved at mysnapshot.db
[root@SZD-L0097856 etcd-cluster]#

3、停止etcd集群

• 停止方法：分别在3台etcd的宿主机上执行以下命令停止etcd服务

systemctl stop etcd

• 停掉Leader 10.25.73.25， 查看集群状况，重新选举出了leader，集群可正常使用

[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem  endpoint status --write-out=table
2018-08-16 18:31:50.686352 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Failed to get the status of endpoint https://10.25.73.25:2379 (context deadline exceeded)
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://10.25.84.251:2379 | 2a53eb67dfa37cc0 | 3.1.10  | 25 kB   | false     |        14 |         19 |
| https://10.25.73.150:2379 | c267a0ca02c6bff7 | 3.1.10  | 25 kB   | true      |        14 |         19 |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
[root@SZD-L0097856 etcd-cluster]#

• 停掉10.25.73.150，查看集群状况，集群已经无法正常使用，说明3节点的Etcd容错为1

[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem  endpoint status --write-out=table
2018-08-16 18:33:44.526382 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Failed to get the status of endpoint https://10.25.73.25:2379 (context deadline exceeded)
Failed to get the status of endpoint https://10.25.73.150:2379 (context deadline exceeded)
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://10.25.84.251:2379 | 2a53eb67dfa37cc0 | 3.1.10  | 25 kB   | false     |        14 |         19 |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+

• 删除etcd数据，方法：登录etcd所在主机执行：(注意：危险操作，请谨慎操作，确保在有数据备份并且确定Etcd集群无法正常工作后操作)

rm -rf /var/lib/etcd

4、使用备份数据进行恢复

• 恢复10.25.84.251节点数据到/var/lib/etcd

[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --name=SZD-L0097856 --endpoints="https://10.25.84.251:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem --initial-cluster-token=etcd-cluster-0 --initial-advertise-peer-urls=https://10.25.84.251:2380 --initial-cluster=SZD-L0101798=https://10.25.73.25:2380,SZD-L0103739=https://10.25.73.150:2380,SZD-L0097856=https://10.25.84.251:2380 --data-dir=/varlib/etcd snapshot restore mysnapshot.db
2018-08-16 19:52:33.409406 I | etcdserver/membership: added member 2a53eb67dfa37cc0 [https://10.25.84.251:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:52:33.409481 I | etcdserver/membership: added member 304bcbfa92e84c75 [https://10.25.73.25:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:52:33.409499 I | etcdserver/membership: added member c267a0ca02c6bff7 [https://10.25.73.150:2380] to cluster 4741a9e2cf17e1fa
[root@SZD-L0097856 etcd-cluster]#

• 恢复10.25.73.25节点数据到/var/lib/etcd

[root@SZD-L0101798 etcd-cluster]# ETCDCTL_API=3 etcdctl --name=SZD-L0101798 --endpoints="https://10.25.73.25:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem --initial-cluster-token=etcd-cluster-0 --initial-advertise-peer-urls=https://10.25.73.25:2380 --initial-cluster=SZD-L0101798=https://10.25.73.25:2380,SZD-L0103739=https://10.25.73.150:2380,SZD-L0097856=https://10.25.84.251:2380 --data-dir=/varlib/etcd snapshot restore mysnapshot.db
2018-08-16 19:55:30.735518 I | etcdserver/membership: added member 2a53eb67dfa37cc0 [https://10.25.84.251:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:55:30.735601 I | etcdserver/membership: added member 304bcbfa92e84c75 [https://10.25.73.25:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:55:30.735617 I | etcdserver/membership: added member c267a0ca02c6bff7 [https://10.25.73.150:2380] to cluster 4741a9e2cf17e1fa
[root@SZD-L0101798 etcd-cluster]#

• 恢复10.25.73.150节点数据到/var/lib/etcd

[root@SZD-L0103739 etcd-cluster]# ETCDCTL_API=3 etcdctl --name=SZD-L0103739 --endpoints="https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem --initial-cluster-token=etcd-cluster-0 --initial-advertise-peer-urls=https://10.25.73.150:2380 --initial-cluster=SZD-L0101798=https://10.25.73.25:2380,SZD-L0103739=https://10.25.73.150:2380,SZD-L0097856=https://10.25.84.251:2380 --data-dir=/varlib/etcd snapshot restore mysnapshot.db
2018-08-16 19:58:21.892719 I | etcdserver/membership: added member 2a53eb67dfa37cc0 [https://10.25.84.251:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:58:21.892796 I | etcdserver/membership: added member 304bcbfa92e84c75 [https://10.25.73.25:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:58:21.892812 I | etcdserver/membership: added member c267a0ca02c6bff7 [https://10.25.73.150:2380] to cluster 4741a9e2cf17e1fa
[root@SZD-L0103739 etcd-cluster]#

5、启动Etcd服务

• 分别在etcd所在主机执行如下命令：

systemctl start etcd

6、验证数据完整性

• 经过验证，使用ETCDCTL_API=2 存放的数据会丢失，使用ETCDCTL_API=3存放的数据能正常恢复

附录：

• 查询etcd API3的键

[root@SZD-L0101798 ~]#  ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem get / --prefix --keys-only
2018-08-16 20:38:06.954368 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
/name/1

/name/2

/name/3

/name/4