etcd3储存的备份与恢复
etcd3 集群安装方法：
参考：https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/03-部署高可用Etcd集群.md
使用api 3 备份与恢复：
export ETCDCTL_API=3
1. 备份数据
etcdctl --endpoints=192.168.5.7:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/flanneld/ssl/flanneld.pem --key=/etc/flanneld/ssl/flanneld-key.pem snapshot save etcd-data-20171002-2.snapshot2.把备份后的数据传输到各个etcd节点，并恢复到新的目录：
etcdctl snapshot restore ./etcd-data-20171002-2.snapshot --endpoints=192.168.5.7:2379 --name=etcd-host0 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/flanneld/ssl/flanneld.pem --key=/etc/flanneld/ssl/flanneld-key.pem --initial-advertise-peer-urls=https://192.168.5.7:2380 --initial-cluster-token=etcd-cluster-0 --initial-cluster=etcd-host0=https://192.168.5.7:2380,etcd-host1=https://192.168.5.8:2380,etcd-host2=https://192.168.5.86:2380 --data-dir=/var/lib/etcd4
etcdctl snapshot restore ./etcd-data-20171002-2.snapshot --endpoints=192.168.5.8:2379 --name=etcd-host1 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/flanneld/ssl/flanneld.pem --key=/etc/flanneld/ssl/flanneld-key.pem --initial-advertise-peer-urls=https://192.168.5.8:2380 --initial-cluster-token=etcd-cluster-0 --
initial-cluster=etcd-host0=https://192.168.5.7:2380,etcd-host1=https://192.168.5.8:2380,etcd-host2=https://192.168.5.86:2380 --data-dir=/var/lib/etcd4
etcdctl snapshot restore ./etcd-data-20171002-2.snapshot --endpoints=192.168.5.86:2379 --name=etcd-host2 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/flanneld/ssl/flanneld.pem --key=/etc/flanneld/ssl/flanneld-key.pem --initial-advertise-peer-urls=https://192.168.5.86:2380 --initial-cluster-token=etcd-cluster-0 --initial-cluster=etcd-host0=https://192.168.5.7:2380,etcd-host1=https://192.168.5.8:2380,etcd-host2=https://192.168.5.86:2380 --data-dir=/var/lib/etcd43.检查etcd集群状态：
etcdctl --endpoints=192.168.5.7:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/flanneld/ssl/flanneld.pem --key=/etc/flanneld/ssl/flanneld-key.pem member list
etcdctl --endpoints=192.168.5.7:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/flanneld/ssl/flanneld.pem --key=/etc/flanneld/ssl/flanneld-key.pem get / --prefix --keys-only4.恢复k8s集群：
由于flannels 7.1 目前只支持 etcd api 2 ,使用api 3 备份后的数据，里面如果有api 2 写入的数据，使用api3恢复不了这些数据，所有需要重新写入一次:
$ export ETCDCTL_API=2
$ etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/flanneld/ssl/flanneld.pem \
--key-file=/etc/flanneld/ssl/flanneld-key.pem \
set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
以下是api v3的测试，不支持。只是测试用：
$ etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--cacert=/etc/kubernetes/ssl/ca.pem \
--cert=/etc/flanneld/ssl/flanneld.pem \
--key=/etc/flanneld/ssl/flanneld-key.pem \
put ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
5.老k8s集群有旧的数据：例如，已经mount的虚拟目录，kubelet，kube-proxy 的旧数据目录(/var/lib/kubelt,kube-proxy)，需要删除，免得出现莫名的问题。
6.注意系统的时钟，一定要同步；
7.修改etcd的启动配置服务文件（/etc/systemd/system/etcd.service），把数据存储目录改为： --data-dir=/var/lib/etcd4
8. 启动k8s相关服务：
master：
systemctl start kube-apiserver
systemctl start kube-scheduler
systemctl start kube-controller-manager
worker：
systemctl start docker
systemctl start flanneld
systemctl start kubelet
systemctl start kube-proxy
9.检查数据：
# kubectl get ep kube-controller-manager -n kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"etcd-host0","leaseDurationSeconds":15,"acquireTime":"2017-10-05T09:37:15Z","renewTime":"2017-10-05T09:37:19Z","leaderTransitions":1}'
creationTimestamp: 2017-10-05T09:34:07Z
name: kube-controller-manager
namespace: kube-system
resourceVersion: "2378"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: 5589c015-a9b0-11e7-bffe-080027622af8
subsets: []
参考：
http://dockone.io/article/8464
https://skyao.gitbooks.io/learning-etcd3/content/documentation/op-guide/maintenance.html
https://skyao.gitbooks.io/learning-etcd3/content/documentation/op-guide/recovery.html
http://www.jianshu.com/p/c60c08effaaa
http://blog.csdn.net/u010278923/article/details/71727682