etcd入门之安全配置

1 基于TLS的通信安全配置

etcd通过简单的配置就能支持基于https的安全通信。https通信需要准备相应的证书文件。本文通过cfssl工具生成需要的证书文件

1.1 cfssl工具安装

cfssl提供了一系列与证书管理相关的工具，根据需要下载相应的二进制文件即可。几个常用的工具包括：
cfssl_1.6.1_linux_amd64、cfssljson_1.6.1_linux_amd64、cfssl-certinfo_1.6.1_linux_amd64

# cp cfssl_1.6.1_linux_amd64 /usr/bin/cfssl
# cp cfssljson_1.6.1_linux_amd64 /usr/bin/cfssljson
# cp cfssl-certinfo_1.6.1_linux_amd64 /usr/bin/cfssl-certinfo
# chmod a+x /usr/bin/cfssl && chmod a+x /usr/bin/cfssljson && chmod a+x /usr/bin/cfssl-certinfo

1.2 生成CA（根）证书

生成一个自签名的CA证书，作为整个集群的根证书。其他的所有证书均由该根证书签发

创建一个根证书的CSR（Certificate Signing Request）文件ca-csr.json，这样需要生成CA证书时相应的配置可以重复使用。

$ cd ssl
$ mkdir ca && cd ca

$ vim ca-csr.json
{
    "CN": "Self-Signed-CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "China",
            "ST": "Guangdong",
            "L": "Shenzhen",
            "O": "sys",
            "OU": "sys"
        }
    ]
}

$ ls
ca-csr.json

CSR文件提供了需要认证的信息。CN（Common Name）是证书公共名字，一般是证书申请者的标识，如网站的话一般是网站域名。这里因为是自签名证书，自己取一个标识名字就可以了。key字段是生成的密钥的算法及相关配置。names字段指定了申请者的更详细的信息。C（Country）：国家；ST（State）：省（州）；L（Location）：城市；O（Organization）：公司等组织；OU（Organization Unit）：公司等组织内的子单位。因为是根证书CSR，所有不需要提供hosts字段。

证书中的CN（Common Name）和O（Organization Name）是两个需要特别关注的字段。CN字段对于 SSL 证书，一般为网站域名；而对于代码签名证书则为申请单位名称；而对于客户端证书则为证书申请者的姓名；在k8s系统中，kube-apiserver 从证书中提取CN字段作为请求的用户名（User Name）；浏览器使用该字段验证网站是否合法；O字段对于 SSL 证书，一般为网站域名；而对于代码签名证书则为申请单位名称；而对于客户端单位证书则为证书申请者所在单位名称；在k8s系统中，kube-apiserver 从证书中提取该字段作为请求用户所属的组（Group）。

执行命令生成根证书：

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2022/08/17 21:19:59 [INFO] generating a new CA key and certificate from CSR
2022/08/17 21:19:59 [INFO] generate received request
2022/08/17 21:19:59 [INFO] received CSR
2022/08/17 21:19:59 [INFO] generating key: rsa-2048
2022/08/17 21:20:00 [INFO] encoded CSR
2022/08/17 21:20:00 [INFO] signed certificate with serial number 46204290406799778272816736364379807158759070127

$ ls
ca.csr  ca-csr.json  ca-key.pem  ca.pem

命令执行完后，生成了三个文件：ca-key.pem、ca.pem、ca.csr。ca-key.pem是CA证书私钥，为其他申请者签发证书时，CA使用这个私钥加密申请的证书认证信息。ca.pem是CA证书，里面包含了签名的CA公钥，需要对证书信息进行验证的使用者使用该证书中的公钥解密认证信息，从而对证书进行认证、ca.csr是编码后的CSR信息。

解析一下证书认证内容，可以发现就是上面CSR中填写的信息

$ cfssl-certinfo -cert ca.pem
{
  "subject": {
    "common_name": "Self-Signed-CA",
    "country": "China",
    "organization": "sys",
    "organizational_unit": "sys",
    "locality": "Shenzhen",
    "province": "Guangdong",
    "names": [
      "China",
      "Guangdong",
      "Shenzhen",
      "sys",
      "sys",
      "Self-Signed-CA"
    ]
  },
  "issuer": {
    "common_name": "Self-Signed-CA",
    "country": "China",
    "organization": "sys",
    "organizational_unit": "sys",
    "locality": "Shenzhen",
    "province": "Guangdong",
    "names": [
      "China",
      "Guangdong",
      "Shenzhen",
      "sys",
      "sys",
      "Self-Signed-CA"
    ]
  },
  "serial_number": "46204290406799778272816736364379807158759070127",
  "not_before": "2022-08-17T13:15:00Z",
  "not_after": "2027-08-16T13:15:00Z",
  "sigalg": "SHA256WithRSA",
  "authority_key_id": "A:13:B:5B:AE:FB:D0:CA:1E:BF:49:6:D2:71:FD:29:9E:8E:3D:BE",
  "subject_key_id": "A:13:B:5B:AE:FB:D0:CA:1E:BF:49:6:D2:71:FD:29:9E:8E:3D:BE",
  "pem": "-----BEGIN CERTIFICATE-----\nMIIDzDCCArSgAwIBAgIUCBffPeRx9Y4faKqYFL0Nj3ssGa8wDQYJKoZIhvcNAQEL\nBQAwbDEOMAwGA1UEBhMFQ2hpbmExEjAQBgNVBAgTCUd1YW5nZG9uZzERMA8GA1UE\nBxMIU2hlbnpoZW4xDDAKBgNVBAoTA3N5czEMMAoGA1UECxMDc3lzMRcwFQYDVQQD\nEw5TZWxmLVNpZ25lZC1DQTAeFw0yMjA4MTcxMzE1MDBaFw0yNzA4MTYxMzE1MDBa\nMGwxDjAMBgNVBAYTBUNoaW5hMRIwEAYDVQQIEwlHdWFuZ2RvbmcxETAPBgNVBAcT\nCFNoZW56aGVuMQwwCgYDVQQKEwNzeXMxDDAKBgNVBAsTA3N5czEXMBUGA1UEAxMO\nU2VsZi1TaWduZWQtQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs\nSBNyzo/mQcKoE2LJ9H/Q0Ynt5ZAmqFRfJs0Zvi0g/podzihgwH4BqW86pwhTbRTn\nnVKkPX5kFuf3/EBtWTZHYCIiA/54oKITYaCvEg6OmZmGWfxbFWCoNz+7zSSkr7K2\nqfbfR8wuRj0mbAgY+lrypypdSP1jzdu/cygLywtKzV7KEDC9X3+fUo9B5NYYZrDo\nysTPnSaoUAmRMSxvUCyL28dAmP1WaGFVcz1n/CPk//ENoHifOf/qRd/OeV+8WI35\nhtkArI3plCrnU4XfRqwu9/lBdzXgy4SAk7QkEEfnF6+GkOxUFawQV5PDfvBMCUAN\nG2eWKV77nE/UK7NJxi9zAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB\nAf8ECDAGAQH/AgECMB0GA1UdDgQWBBQKEwtbrvvQyh6/SQbScf0pno49vjAfBgNV\nHSMEGDAWgBQKEwtbrvvQyh6/SQbScf0pno49vjANBgkqhkiG9w0BAQsFAAOCAQEA\npaJsh/i68SDSsGU2oC73O7nxl9L/MawjU/ckDgrG01pCg6mRO3KWot+xcajGf6Ur\nrFwAba7o8Oh7CaRNovPkidOPSh24aK3b0C+gpmv88gXjOrum7FvWMRrImF6E1YiU\n+n+WvSJNA7Geok5KjKfpghghQREcc5Vipdtgtofr4NFIQGWw1dhF1h7MoktH3sII\n8Q6z8w1mjS/iZ8W38nfYNk0MK6Q2KosUcTAwYHaYfCrQTW7fg6rbDX1vs3GQpjlp\nIQtDBjR/6ndsxc4ZJoaKvPaGLBVBXNjtgH+EksKCWIYdeU9RO56watPbecLvoS8E\njVIGXeZBuza45mDLF030Aw==\n-----END CERTIFICATE-----\n"
}

1.3 etcd服务生成相关证书

1.3.1 创建证书配置

由于证书生成过程中一些公共配置会反复用到，所以可以配置到一个公共配置文件中，待后续使用

$ cd ssl
$ vim cert-config.json
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "server": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

配置文件中expiry字段规定了生成的证书的有效时间，上面默认是1年。配置文件中还定义了三个profile，使用时可以指定相应的profile，从而使用相应profile中的配置。usage字段配置了生成的证书可以用在哪些方面：比如签名（signing）、加密（key encipherment）、客户端对服务端进行认证（server auth）、服务端对客户端进行认证（client auth）。

1.3.2 生成etcd集群间通信使用的证书

创建相应的CSR文件，hosts字段填写集群节点相互用来通信的ip地址和dns名字等。建立tls连接时会检查实际报文的源ip地址等是否在证书的hosts字段中

$ cd ssl
$ mkdir etcd-peer && cd etcd-peer
$ vim etcd-peer-csr.json

{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "10.131.21.11",
      "10.131.21.12",
      "10.131.21.13",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shenzhen",
            "L": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

$ ls
etcd-peer-csr.json

# ca.pem和ca.pem是上面生成的CA根证书和密钥，配置信息使用cert-config.json中的peer profile。生成的文件名为etcd-peer
$ cfssl gencert -ca ../ca/ca.pem -ca-key ../ca/ca-key.pem -config ../cert-config.json -profile peer etcd-peer-csr.json | cfssljson -bare etcd-peer -
2022/08/17 21:26:54 [INFO] generate received request
2022/08/17 21:26:54 [INFO] received CSR
2022/08/17 21:26:54 [INFO] generating key: rsa-2048
2022/08/17 21:26:54 [INFO] encoded CSR
2022/08/17 21:26:54 [INFO] signed certificate with serial number 322003155427865617191473591973409438172195496664
2022/08/17 21:26:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

$ ls
etcd-peer.csr  etcd-peer-csr.json  etcd-peer-key.pem  etcd-peer.pem

1.3.3 生成etcd节点与client通信使用的证书

跟生成peer证书类似，只是CSR中的hosts只需要包括节点本身用于通信的ip地址即可。由于CN和O字段的特殊用途，根据需要设置为对应的值，我这里没有改。profile使用server profile。

$ cd ssl
$ mkdir etcd-server && cd etcd-server
$ vim etcd-server-csr.json

{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "10.131.21.11"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shenzhen",
            "L": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

$ ls
etcd-server-csr.json

$ cfssl gencert -ca ../ca/ca.pem -ca-key ../ca/ca-key.pem -config ../cert-config.json -profile server etcd-server-csr.json | cfssljson -bare etcd-server -

$ ls
etcd-server.csr  etcd-server-csr.json  etcd-server-key.pem  etcd-server.pem

1.3.4 为客户端生成证书

假设客户端节点的ip地址为10.131.21.14，用与上面类似的方式生成证书。hosts字段包括自己用于通信的ip，由于CN和O字段的特殊用途，根据需要设置为对应的值，我这里没有改。profile使用client profile。

$ cd ssl
$ mkdir etcd-client && cd etcd-client
$ vim etcd-client-csr.json

{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "10.131.21.14"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shenzhen",
            "L": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

$ ls
etcd-client-csr.json

$ cfssl gencert -ca ../ca/ca.pem -ca-key ../ca/ca-key.pem -config ../cert-config.json -profile client etcd-client-csr.json | cfssljson -bare etcd-client -

$ ls
etcd-client.csr  etcd-client-csr.json  etcd-client-key.pem  etcd-client.pem

1.3.5 生成的证书目录结构

$ tree ssl
ssl
├── ca
│   ├── ca.csr
│   ├── ca-csr.json
│   ├── ca-key.pem
│   └── ca.pem
├── cert-config.json
├── etcd-client
│   ├── etcd-client.csr
│   ├── etcd-client-csr.json
│   ├── etcd-client-key.pem
│   └── etcd-client.pem
├── etcd-peer
│   ├── etcd-peer.csr
│   ├── etcd-peer-csr.json
│   ├── etcd-peer-key.pem
│   └── etcd-peer.pem
└── etcd-server
    ├── etcd-server.csr
    ├── etcd-server-csr.json
    ├── etcd-server-key.pem
    └── etcd-server.pem

在etcd节点的部署工作目录下创建一个ssl目录，并将上面ca、etcd-peer、etcd-server目录下的所有证书相关文件拷贝到相应节点的ssl目录下。
例如，节点10.131.21.11的ssl目录下内容为：

$ cd etcd
$ ls ssl
ca.pem etcd-peer-key.pem etcd-peer.pem etcd-server-key.pem etcd-server.pem

1.4 启动集群

假设集群有三个节点，服务ip分别为10.131.21.11、10.131.21.12、10.131.21.13，为这三个节点分别创建相应的配置文件

1.4.1 节点配置

节点1（10.131.21.11）配置文件内容：

$ cat conf/etcd.yaml

# 节点名字，集群内唯一
name: etcd-cnlab0

# 数据存储目录
data-dir: ./data

# 日志文件配置
# logger: zap
log-outputs: ['./log/etcd.log']
# log-level: info

# 客户端连接相关配置
# 服务监听端点
listen-client-urls: https://0.0.0.0:2379
# 向客户端发布的服务端点
advertise-client-urls: https://10.131.21.11:2379

# 集群相关配置
# 监听集群其他节点连接的端点
listen-peer-urls: https://0.0.0.0:2380
# 向集群其他节点发布的服务端点
initial-advertise-peer-urls: https://10.131.21.11:2380
# 集群成员的名字以及服务端点列表，名字与每个节点配置的name字段值对应
initial-cluster: etcd-cnlab0=https://10.131.21.11:2380,etcd-cnlab1=https://10.131.21.12:2380,etcd-cnlab2=https://10.131.21.13:2380
# 集群标识token，可以认为是集群名字
initial-cluster-token: etcd-cnlab
# 创建一个新的集群。如果data-dir目录下的数据属于另外一个集群，则无法启动
inital-cluster-state: new

# 配置证书
# 用于客户端认证etcd的证书
client-transport-security:
  cert-file: ./ssl/etcd-server.pem
  key-file: ./ssl/etcd-server-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true
# 用于集群节点间相互认证的证书
peer-transport-security:
  cert-file: ./ssl/etcd-peer.pem
  key-file: ./ssl/etcd-peer-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true

节点2（10.131.21.12）配置文件内容：

$ cat conf/etcd.yaml

name: etcd-cnlab1
data-dir: ./data
listen-client-urls: https://0.0.0.0:2379
advertise-client-urls: https://10.131.21.12:2379
listen-peer-urls: https://0.0.0.0:2380

initial-advertise-peer-urls: https://10.131.21.12:2380
initial-cluster: etcd-cnlab0=https://10.131.21.11:2380,etcd-cnlab1=https://10.131.21.12:2380,etcd-cnlab2=https://10.131.21.13:2380
initial-cluster-token: etcd-cnlab
inital-cluster-state: new

client-transport-security:
  cert-file: ./ssl/etcd-server.pem
  key-file: ./ssl/etcd-server-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true
peer-transport-security:
  cert-file: ./ssl/etcd-peer.pem
  key-file: ./ssl/etcd-peer-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true

节点3（10.131.21.13）配置文件内容：

$ cat conf/etcd.yaml

name: etcd-cnlab2
data-dir: ./data
listen-client-urls: https://0.0.0.0:2379
advertise-client-urls: https://10.131.21.13:2379
listen-peer-urls: https://0.0.0.0:2380

initial-advertise-peer-urls: https://10.131.21.13:2380
initial-cluster: etcd-cnlab0=https://10.131.21.11:2380,etcd-cnlab1=https://10.131.21.12:2380,etcd-cnlab2=https://10.131.21.13:2380
initial-cluster-token: etcd-cnlab
inital-cluster-state: new

client-transport-security:
  cert-file: ./ssl/etcd-server.pem
  key-file: ./ssl/etcd-server-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true
peer-transport-security:
  cert-file: ./ssl/etcd-peer.pem
  key-file: ./ssl/etcd-peer-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true

1.4.2 启动三个节点

$ ./bin/etcd --config-file=conf/etcd.yaml

1.5 客户端连接etcd

在客户端节点10.131.20.14上连接集群节点10.131.20.11，并执行命令

$ ./bin/etcdctl --endpoints=https://10.131.20.11:2379 --cert=./ssl/etcd.pem --key=./ssl/etcd-key.pem --cacert=./ssl/ca.pem member list
5adcdda313f6e536, started, etcd-cnlab1, https://10.131.20.12:2380, https://10.131.20.12:2379, false
886dc7c192c66fe4, started, etcd-cnlab2, https://10.131.20.13:2380, https://10.131.20.13:2379, false
fa92ba5824e127ae, started, etcd-cnlab0, https://10.131.20.11:2380, https://10.131.20.11:2379, false

2 基于角色的权限认证

不管是否开启TLS，etcd都可以通过用户名（user）和角色（role）对用户的访问权限进行控制。

默认情况下，etcd集群没有开启基于角色的权限认证。开启权限认证，一定存在一个root用户和root角色，root角色自然拥有所有权限。所以开启权限认证，需要先创建root用户和root角色，并将root用户绑定到root角色。

在任意一个集群节点上执行下列命令，创建root用户和root角色，并绑定。

# 查询已配置的用户和角色，默认没有
$ ./bin/etcdctl user list
$ ./bin/etcdctl role list

# 创建root用户和root角色
$ ./bin/etcdctl user add root
$ ./bin/etcdctl role add root

$ ./bin/etcdctl user list
root
$ ./bin/etcdctl role list
root

# 绑定角色
$ ./bin/etcdctl user grant-role root root

在任意一个集群节点上执行下列命令，开始权限认证：

$ ./bin/etcdctl auth enable

现在开始，需要使用用户名和密码才能访问etcd集群了

$ ./bin/etcdctl --user=root:etcd123 role list
root

创建非root用户和角色

$ ./bin/etcdctl --user=root:etcd123 user add u_sys

$ ./bin/etcdctl --user=root:etcd123 role add r_sys

# key的读权限
$ ./bin/etcdctl role grant-permission r_sys read /foo

# 整个目录的写权限
$ ./bin/etcdctl role grant-permission r_sys write /sys/*

# 读写权限
$ ./bin/etcdctl role grant-permission r_sys readwrite /abc

# 查看角色的权限
$ ./bin/etcdctl role get r_sys
Role r_sys
KV Read:
    /abc
    /foo
KV Write:
    /abc
    [/sys/block, /sys/bus)

# 绑定用户和角色，使用户获得相应的权限
$ ./bin/etcdctl user grant-role u_sys r_sys

# 撤销角色的权限
$ ./bin/etcdctl role revoke-permission r_sys /foo

删除命令

$ ./bin/etcdctl role delete r_sys
$ ./bin/etcdctl user delete u_sys