使用Ansible playbooks快速构建etcd集群

本篇文章记录一下使用ansible-playbooks进行快速构建一个可用的etcd集群。在阅读并实践本文章之前，请确保您有一个可用的ansible环境。

一、集群规划

注意:本文档基于centos7+的操作系统上进行构建，在rhel7+系列也可用使用，其他发行版本可酌情参考

etcd_name	节点	端口
etcd1	10.0.0.77	2379/2380
etcd2	10.0.0.78	2379/2380
etcd3	10.0.0.79	2379/2380

以上节点详情信息用来规划集群规模以及名称角色，配置文件中ETCD_INITIAL_CLUSTER选项中的etcd_name和地址需要和主机规划中一致

二、etcd集群部署

环境前提：已经配置过ansible相关的环境，并且可以免密登录

1.etcd集群相关的配置文件

$ tree . 
.
├──etcd-install.yml   #ansible-playbook 脚本
├──etcd.conf.j2       #etcd主配置文件
├──etcd.service       #systemd服务启动文件
├──host               #etcd集群主机列表

# ansible-playbooks 脚本
$ cat etcd-install.yml
---
- hosts: "{{ host }}"
  remote_user: root
  vars:
    #hostip: "{{ ansible_all_ipv4_addresses[0] }}"
    hostip: "{{ inventory_hostname }}"
    node_name: "{{ nodename }}"
    ipv6: "{{ ansible_all_ipv6_addresses[0].split(':')[-1] }}"
    packagedir: /tmp/
    serverdir: /export/servers/
    download: https://github.com/coreos/etcd/releases/
    etcd_version: etcd-v3.1.5-linux-amd64
  tasks:
  - name: "test ping"
    ping:
  - name: "init env"
    shell: "mkdir -p /export/servers /etc/etcd/ /export/Data/etcd/ "


  - name: "wget {{ etcd_version }}"
    get_url:
     url: "{{ item.url }}"
     dest: "{{ item.dest }}"
     mode: 0644
     owner: root
     group: root
    with_items:
        - { url: "{{ download }}{{ etcd_version }}.tar.gz" , dest: "{{ packagedir }}" }

  - name: "untar the {{ etcd_version }}"
    unarchive:
     src: "{{ item.src }}"
     dest: "{{ item.dest }}"
     owner: root
     group: root
     remote_src: yes
    with_items:
      - { src: "{{ packagedir }}{{ etcd_version }}.tar.gz", dest: "{{ serverdir }}" }

  - name: "copy the etcd execfile"
    shell: "cp -rp {{ serverdir }}{{ etcd_version }}/etcd* /usr/bin/"

  - name: "init the es configuration!"
    template:
      src: "{{ item.src }}"
      dest: "{{ item.dest }}"
      mode: 0755
      owner: root
      group: root
    with_items:
      #dest后面的引号与大括号中间一定不能有空格,还有后面引号的空格也要取消掉,不然文件会有空格
      - { src: "etcd.conf.j2", dest: "/etc/etcd/etcd.conf" }
      - { src: "etcd.service", dest: "/etc/systemd/system/" }

# etcd主配置文件模板，这里采用jinja2模板引擎
$ cat etcd.conf.j2
# [member]
BASE_DIR="/export"
# 由于etcd_name变量比较特殊，需要和ETCD_INITIAL_CLUSTER中的值一一对应，暂时写为etcd1,配置下发后需要将该值替换成对应节点的名称
ETCD_NAME="{{ node_name }}"
ETCD_DATA_DIR="/export/Data/etcd"
ETCD_LISTEN_PEER_URLS="http://{{ hostip }}:2380"
ETCD_LISTEN_CLIENT_URLS="http://{{ hostip }}:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://{{ hostip }}:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://{{ hostip }}:2379"

#[initial-cluster]
# 需要按照etcd集群规划节点来填写,在host文件内需要指定name和ip的对应关系
ETCD_INITIAL_CLUSTER="etcd1=http://10.0.0.77:2380,etcd2=http://10.0.0.78:2380,etcd3=http://10.0.0.79:2380"


# systemd 启动脚本
$ cat etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
# 数据目录，需要提前进行创建
WorkingDirectory=/export/Data/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
  --name ${ETCD_NAME} \
  --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
  --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
  --initial-cluster ${ETCD_INITIAL_CLUSTER} \
  --initial-cluster-state new \
  --data-dir=${ETCD_DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

$ cat hosts
[etcd]
10.0.0.77  nodename=etcd1
10.0.0.78  nodename=etcd2
10.0.0.79  nodename=etcd3

2.使用ansible-playbooks进行部署etcd集群

$ ansible-playbook -i host -e host=etcd etcd-install.yml 
# 如果有多个环境变量也可以将变量信息写入json串中
--extra-vars "host=etcd user=root"
--extra-vars "{'host':'etcd', 'user':'root'}"
--extra-vars "@vars.json"

等待正确执行完成之后，三个节点的etcd就已经安装好了。

3.登录每个节点进行修改etcd_name

修改每个节点上的/etc/etcd/etcd.conf文件:
注意：需要将各个节点ip和etcd_name对应起来(etcd1->77 etcd2->78 etcd->79)

77节点上的配置如下(其他两个节点修改对应的etcd_name)：

~~ETCD_NAME="etcd1"~~
~~ETCD_INITIAL_CLUSTER="etcd1=http://10.0.0.77:2380,etcd2=http://10.0.0.78:2380,etcd3=http://10.0.0.79:2380"~~

4.检测配置文件并进行启动集群

$ ansible -i host all -m shell -a 'systemctl daemon-reload && systemctl start etcd && systemctl enable etcd'

注意：由于etcd采用集群模式，在启动第一个实例的时候会一直监听其他实例，这个时候需要尽快启动其他实例，以完成集群发现，否则第一个实例会在一定时间内挂掉

5.集群状态监测

$ etcdctl cluster-health
member 4d946fabd2e1eb9f is healthy: got healthy result from http://10.0.0.79:2379
member 805150a1f1a44604 is healthy: got healthy result from http://10.0.0.77:2379
member 81ddc2f4095fe8a1 is healthy: got healthy result from http://10.0.0.78:2379
cluster is healthy

$ etcdctl member list
4d946fabd2e1eb9f: name=etcd3 peerURLs=http://10.0.0.79:2380 clientURLs=http://10.0.0.79:2379 isLeader=true
805150a1f1a44604: name=etcd1 peerURLs=http://10.0.0.77:2380 clientURLs=http://10.0.0.77:2379 isLeader=false
81ddc2f4095fe8a1: name=etcd2 peerURLs=http://10.0.0.78:2380 clientURLs=http://10.0.0.78:2379 isLeader=false

如此，可以发现我们的etcd集群已经迅速运行起来了！

三、etcd集群的基本使用

注意1:由于是集群模式，所以可用在任何一个节点进行写入操作，用户也可用使用etcd的http接口进行使用
注意2:etcd v2和v3的API是完全不同的两个存储实现，所以用户在使用的时候需要注意在etcd v2版本中是采用set存数据的，v3版本中采用put方式

指定API版本
$ export ETCDCTL_API=3
$ etcdctl  put foo "Hello World!"
$ etcdctl get foo
以json格式输出
$ etcdctl --write-out="json" get foo 

通过前缀获取数据

$ etcdctl --endpoints=$ENDPOINTS put web1 value1
$ etcdctl --endpoints=$ENDPOINTS put web2 value2
$ etcdctl --endpoints=$ENDPOINTS put web3 value3

$ etcdctl --endpoints=$ENDPOINTS get web --prefix
web1
value1
web2
value2
web3
value3

删除数据

$ etcdctl --endpoints=$ENDPOINTS put key myvalue
$ etcdctl --endpoints=$ENDPOINTS del key

$ etcdctl --endpoints=$ENDPOINTS put k1 value1
$ etcdctl --endpoints=$ENDPOINTS put k2 value2
$ etcdctl --endpoints=$ENDPOINTS del k --prefix

集群状态信息

v3 的API才有这个功能

$ etcdctl --write-out=table  --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 endpoint status
+-------------------+------------------+---------+---------+-----------+-----------+------------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-------------------+------------------+---------+---------+-----------+-----------+------------+
| 10.0.0.77:2379 | 805150a1f1a44604 | 3.1.5   | 1.2 MB  | false     |       639 |     188421 |
| 10.0.0.78:2379 | 81ddc2f4095fe8a1 | 3.1.5   | 1.2 MB  | false     |       639 |     188421 |
| 10.0.0.79:2379 | 4d946fabd2e1eb9f | 3.1.5   | 1.2 MB  | true      |       639 |     188421 |
+-------------------+------------------+---------+---------+-----------+-----------+------------+

原文地址