(3)二进制文件方式部署Kubernetes高可用集群----------创建TLS证书和秘钥
目录
2 创建TLS证书和秘钥
2.1 生成的CA证书和密钥文件
2.2 使用证书的组件
2.3 二进制源码包安装cfssl
2.3.1 二进制源码包下载
2.3.2 默认安装路径
2.3.3 自定义安装路径
2.4 创建CA证书和私钥
2.4.1 生成CA配置文件
2.4.2 编辑CA配置文件(ca-config.json)
2.4.2 编辑CA签名请求文件(ca-config.json)
2.4.3 生成CA证书和私钥
2.5 创建kubernets证书和私钥
2.5.1 创建kuberne证书签名请求文件(kubernetes-csr.json)
2.5.2 生成kubernetes证书和私钥
2.6 创建admin证书和私钥
2.6.1 创建admin证书签名请求文件(admin-csr.json)
2.6.2 生成kube-proxy客户端证书和私钥
2.7 创建kube-proxy证书和私钥
2.7.1 创建kube-proxy证书签名请求文件(kube-proxy-csr.json)
2.7.2 生成kube-proxy客户端证书和私钥
2.8 生成的证书列表
2.9 检验证书
2.9.1 使用openssl命令
2.9.2 使用cfss-certinfo命令
2.10 分发证书
2.11 需要TLS证书认证的组件
kubernetes系统的各组件需要使用TLS证书对通信进行加密，每个k8s集群都需要有独立的CA证书体系。这里，我们的集群部署采用CloudFlare的PKI工具集cfssl来生成Certificate Authority(CA)证书和秘钥文件(CA 是自签名的证书，用来签名后续创建的其它 TLS 证书。)。
2 创建TLS证书和秘钥
2.1 生成的CA证书和密钥文件
※ca-key.pem
※ca.pem
※kubernetes-key.pem
※kubernetes.pem
※kube-proxy.pem
※kube-proxy-key.pem
※admin.pem
※admin-key.pem
2.2 使用证书的组件
※etcd：使用ca.pem、kubernetes-key.pem、kubernetes.pem(etcd对外提供服务、节点间通信（etcd peer）使用同一套证书)
※kube-apiserver：使用ca.pem、kubernetes-key.pem、kubernetes.pem
※kubelet：使用ca.pem
※kube-proxy：使用ca.pem、kube-proxy-key.pem、kube-proxy.pem
※kubectl：使用ca.pem、admin-key.pem、admin.pem
kube-controller-manager、kube-scheduler当前需要和kube-apiserver部署在同一台机器上并且使用非安全端口通信，因此不需要证书。注意证书只需要创建一次即可，以后向集群中添加新节点时，只要将/opt/kubernetes/ssl目录下的证书拷贝到新的节点上即可。
2.3 二进制源码包安装cfssl
这里直接采用这里cfssl的二进制源码包安装，不同之处实际上就是两种方式下的环境变量配置。
2.3.1 二进制源码包下载
| wget -P /root/kubernetes/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget -P /root/kubernetes/cfssl https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget -P /root/kubernetes/cfssl https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 |
2.3.2 默认安装路径
默认路径：/usr/local/bin
文件拷贝：
| chmod 777 /root/kubernetes/cfssl -R cp ./cfssl/cfssl_linux-amd64 /usr/local/bin/cfssl cp ./cfssl/cfssljson_linux-amd64 /usr/local/bin/cfssljson cp ./cfssl/cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo |
临时配置环境变量：“[root@master01 /]# export PATH=/usr/local/bin/:$PATH”
2.3.3 自定义安装路径
路径：/opt/kubernetes/bin/cfssl
文件拷贝：
| chmod 777 /root/kubernetes/cfssl -R cp ./cfssl/cfssl_linux-amd64 /opt/kubernetes/bin/cfssl cp ./cfssl/cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljson cp ./cfssl/cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo |
如果临时环境变量不生效，就修改profile配置文件。
| [root@master01 /]# vi /etc/profile 永久环境变量配置： export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/kubernetes/bin |
2.4 创建CA证书和私钥
2.4.1 生成CA配置文件
保存路径：/root/kubernetes/cajson【路径看个人喜好自行设置即可】
| [root@master01 cajson]# cfssl print-defaults config > ca-config.json [root@master01 cajson]# cfssl print-defaults csr > ca-csr.json |
2.4.2 编辑CA配置文件(ca-config.json)
ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile。修改后内容如下：
| { "signing": { "default": { "expiry": "43800h" }, "profiles": { "kubernetes": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } |
字段说明：
| signing：表示该证书可用于签名其它证书；生成的ca.pem证书中CA=TRUE; server auth：表示client可以用该CA证书对server提供的证书进行验证； client auth：表示server可以用该CA证书对client提供的证书进行验证； |
2.4.2 编辑CA签名请求文件(ca-config.json)
修改后内容如下：
| { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ChengDu", "L": "ChengDu", "O": "k8s", "OU": "System" } ] } |
字段说明：
| "CN"：Common Name，kube-apiserver从证书中提取该字段作为请求用户名(User Name);浏览器使用该字段验证网站是否合法。 “O”：Organization，kube-apiserver从证书中提取该字段作为请求用户所属的组(Group)； |
2.4.3 生成CA证书和私钥
| [root@master01 cajson]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca [root@master01 cajson]# ls ca* ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem |
2.5 创建kubernets证书和私钥
2.5.1 创建kuberne证书签名请求文件(kubernetes-csr.json)
修改后内容：
| { "CN": "kubernetes", "hosts": [ "127.0.0.1", "master01", "192.168.0.143", "master02", "192.168.0.144", "master03", "192.168.0.145", "192.168.0.140" "10.10.10.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ChengDu", "L": " ChengDu ", "O": "k8s", "OU": "System" } ] }
|
2.5.2 生成kubernetes证书和私钥
| [root@master01 cajson]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes [root@master01 cajson]# ls kubernetes* kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem |
2.6 创建admin证书和私钥
2.6.1 创建admin证书签名请求文件(admin-csr.json)
修改后内容：
| { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ChengDu", "L": "ChengDu", "O": "system:masters", "OU": "System" } ] }
|
2.6.2 生成kube-proxy客户端证书和私钥
|
[root@master01 cajson]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin [root@master01 cajson]# ls admin* admin.csr admin-csr.json admin-key.pem admin.pem
|
2.7 创建kube-proxy证书和私钥
2.7.1 创建kube-proxy证书签名请求文件(kube-proxy-csr.json)
修改后的内容：
| { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ChengDu", "L": "ChengDu", "O": "k8s", "OU": "System" } ] }
|
2.7.2 生成kube-proxy客户端证书和私钥
| [root@master01 cajson]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy [root@master01 cajson]# ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem |
2.8 生成的证书列表
| -rw-------. 1 root root 1679 May 26 10:44 admin-key.pem -rw-r--r--. 1 root root 1403 May 26 10:44 admin.pem -rw-------. 1 root root 1679 May 26 09:31 ca-key.pem -rw-r--r--. 1 root root 1363 May 26 09:31 ca.pem -rw-------. 1 root root 1675 May 26 10:52 kube-proxy-key.pem -rw-r--r--. 1 root root 1403 May 26 10:52 kube-proxy.pem -rw-------. 1 root root 1679 May 26 09:31 kubernetes-key.pem -rw-r--r--. 1 root root 1635 May 26 09:31 kubernetes.pem |
当然，如果感觉上边的证书和私钥生成麻烦，可以把生成命令编辑到脚本中，一次生成.
#!/bin/sh
echo '生成证书和私钥'
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
echo '证书和私钥生成成功'2.9 检验证书
以kubernetes证书为例
2.9.1 使用openssl命令
[root@master01 cajson]# openssl x509 -noout -text -in kubernetes.pem
| Certificate: Data: Version: 3 (0x2) Serial Number: 5b:72:5d:00:64:c5:12:39:ae:d4:da:bb:e0:ec:70:8e:54:6d:96:2c Signature Algorithm: sha256WithRSAEncryption Issuer: C=CN, ST=ChengDu, L=ChengDu, O=k8s, OU=System, CN=kubernetes Validity Not Before: May 28 14:04:00 2019 GMT Not After : May 27 14:04:00 2020 GMT Subject: C=CN, ST=ChengDu, L=ChengDu, O=k8s, OU=System, CN=kubernetes Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:c0:f0:54:bc:02:99:5a:a5:96:51:7e:b9:94:75: 65:07:a6:18:dd:85:3a:07:2d:49:2d:f2:79:e6:42: 39:75:cf:41:de:42:5e:6b:20:c3:a0:0e:20:02:2b: cd:82:8f:10:9c:8e:ad:fc:24:d1:59:40:3c:b6:92: 09:34:8d:2b:6a:d1:4c:e6:02:b1:f7:58:ee:80:0b: 96:6d:90:53:52:34:66:41:48:dc:a3:6e:93:93:10: eb:ed:98:77:cf:3e:be:45:20:61:39:3d:01:f2:a0: 9e:91:67:b2:aa:a9:b5:6a:54:16:dc:3e:f1:c6:b7: 3d:08:01:58:b4:bd:a0:56:da:6c:d3:65:b2:d4:09: 35:dd:b1:eb:ad:e3:1d:97:1e:f5:50:bd:8e:6c:a6: 88:a2:8c:13:c5:2a:ac:cd:4b:3c:f1:ef:4f:33:6d: 43:06:c5:6d:d5:cf:1a:e8:9a:12:39:17:5e:9a:98: 1f:c7:94:33:5f:4f:fc:8b:93:83:e1:d1:19:35:b3: db:0a:aa:71:9a:45:54:10:70:b7:95:27:a5:38:88: 6b:f9:21:d1:99:19:4e:0b:59:4e:47:e9:bc:d8:79: f8:de:35:b5:94:71:cf:84:d3:3d:77:07:ea:28:c2: 5e:99:26:4f:89:4b:63:39:84:4e:bf:dd:5b:1f:c2: 61:ff Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: 3E:A9:A4:2D:ED:CA:02:15:CB:F7:FD:3C:0D:D4:52:3F:F0:18:5C:6C X509v3 Authority Key Identifier: keyid:93:B1:C8:5B:46:33:FD:44:3B:6D:3E:09:B7:BE:FC:4C:A3:59:A8:69
X509v3 Subject Alternative Name: DNS:master01, DNS:master02, DNS:master03, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.72.143, IP Address:192.168.72.144, IP Address:192.168.72.145 Signature Algorithm: sha256WithRSAEncryption 5e:21:e4:48:51:30:98:7e:80:c5:0a:48:ca:63:c7:9d:cd:2b: 60:f5:32:ec:00:27:71:a0:c5:60:36:bd:a0:f5:0a:4c:69:f6: bc:b9:60:ad:5f:ff:8d:6a:16:a7:45:97:d8:6f:3b:3d:42:13: 49:a2:1f:55:59:c2:2b:67:e3:4b:c9:e9:28:93:85:9c:4c:80: 58:aa:36:5b:b2:c4:84:37:73:7c:11:d4:fa:89:92:44:8f:3d: a6:24:f4:85:59:53:79:16:c7:86:56:ae:26:b5:a1:0d:4d:6e: 09:de:a9:93:c6:af:6a:d3:06:8a:17:d5:7e:a3:2a:aa:d4:da: 4b:98:54:89:7e:e6:dd:29:ba:62:73:ae:5e:38:90:0d:09:0a: 61:22:80:4a:a1:35:8c:13:96:e4:8e:7e:15:3c:ed:87:c6:b2: a4:49:cd:18:8b:8c:68:56:6b:96:fb:e1:69:4d:0e:05:aa:fe: eb:01:2f:73:c5:55:8f:10:03:ab:0a:2e:5d:05:56:4a:dc:a2: 01:cb:ad:ba:a6:24:ce:3f:a3:68:f6:47:31:2c:da:ae:24:6b: 45:c0:49:38:e1:50:79:06:a0:ff:fc:1b:29:f3:95:13:a1:b8: 60:4b:5a:d5:b5:11:15:f4:ba:b9:12:07:66:fa:6a:92:73:c4: ab:1d:6b:73
|
2.9.2 使用cfss-certinfo命令
[root@master01 cajson]# cfssl-certinfo -cert kubernetes.pem
| { "subject": { "common_name": "kubernetes", "country": "CN", "organization": "k8s", "organizational_unit": "System", "locality": "ChengDu", "province": "ChengDu", "names": [ "CN", "ChengDu", "ChengDu", "k8s", "System", "kubernetes" ] }, "issuer": { "common_name": "kubernetes", "country": "CN", "organization": "k8s", "organizational_unit": "System", "locality": "ChengDu", "province": "ChengDu", "names": [ "CN", "ChengDu", "ChengDu", "k8s", "System", "kubernetes" ] }, "serial_number": "522068546674139593159398095086002169705095075372", "sans": [ "master01", "master02", "master03", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local", "127.0.0.1", "192.168.72.143", "192.168.72.144", "192.168.72.145" ], "not_before": "2019-05-28T14:04:00Z", "not_after": "2020-05-27T14:04:00Z", "sigalg": "SHA256WithRSA", "authority_key_id": "93:B1:C8:5B:46:33:FD:44:3B:6D:3E:9:B7:BE:FC:4C:A3:59:A8:69", "subject_key_id": "3E:A9:A4:2D:ED:CA:2:15:CB:F7:FD:3C:D:D4:52:3F:F0:18:5C:6C", "pem": "-----BEGIN CERTIFICATE-----\nMIIEnTCCA4WgAwIBAgIUW3JdAGTFEjmu1Nq74OxwjlRtliwwDQYJKoZIhvcNAQEL\nBQAwZTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0NoZW5nRHUxEDAOBgNVBAcTB0No\nZW5nRHUxDDAKBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwpr\ndWJlcm5ldGVzMB4XDTE5MDUyODE0MDQwMFoXDTIwMDUyNzE0MDQwMFowZTELMAkG\nA1UEBhMCQ04xEDAOBgNVBAgTB0NoZW5nRHUxEDAOBgNVBAcTB0NoZW5nRHUxDDAK\nBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwprdWJlcm5ldGVz\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwPBUvAKZWqWWUX65lHVl\nB6YY3YU6By1JLfJ55kI5dc9B3kJeayDDoA4gAivNgo8QnI6t/CTRWUA8tpIJNI0r\natFM5gKx91jugAuWbZBTUjRmQUjco26TkxDr7Zh3zz6+RSBhOT0B8qCekWeyqqm1\nalQW3D7xxrc9CAFYtL2gVtps02Wy1Ak13bHrreMdlx71UL2ObKaIoowTxSqszUs8\n8e9PM21DBsVt1c8a6JoSORdempgfx5QzX0/8i5OD4dEZNbPbCqpxmkVUEHC3lSel\nOIhr+SHRmRlOC1lOR+m82Hn43jW1lHHPhNM9dwfqKMJemSZPiUtjOYROv91bH8Jh\n/wIDAQABo4IBQzCCAT8wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUF\nBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQ+qaQt7coCFcv3\n/TwN1FI/8BhcbDAfBgNVHSMEGDAWgBSTschbRjP9RDttPgm3vvxMo1moaTCBvwYD\nVR0RBIG3MIG0gghtYXN0ZXIwMYIIbWFzdGVyMDKCCG1hc3RlcjAzggprdWJlcm5l\ndGVzghJrdWJlcm5ldGVzLmRlZmF1bHSCFmt1YmVybmV0ZXMuZGVmYXVsdC5zdmOC\nHmt1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3RlcoIka3ViZXJuZXRlcy5kZWZh\ndWx0LnN2Yy5jbHVzdGVyLmxvY2FshwR/AAABhwTAqEiShwTAqEiQhwTAqEiRMA0G\nCSqGSIb3DQEBCwUAA4IBAQBeIeRIUTCYfoDFCkjKY8edzStg9TLsACdxoMVgNr2g\n9QpMafa8uWCtX/+NahanRZfYbzs9QhNJoh9VWcIrZ+NLyekok4WcTIBYqjZbssSE\nN3N8EdT6iZJEjz2mJPSFWVN5FseGVq4mtaENTW4J3qmTxq9q0waKF9V+oyqq1NpL\nmFSJfubdKbpic65eOJANCQphIoBKoTWME5bkjn4VPO2HxrKkSc0Yi4xoVmuW++Fp\nTQ4Fqv7rAS9zxVWPEAOrCi5dBVZK3KIBy626piTOP6No9kcxLNquJGtFwEk44VB5\nBqD//Bsp85UTobhgS1rVtREV9Lq5Egdm+mqSc8SrHWtz\n-----END CERTIFICATE-----\n" } |
2.10 分发证书
将生成的证书和秘钥文件(后缀名为.pem)拷贝到所有机器的/opt/kubernetes/ssl 目录下备用
| 本机复用节点： [root@master01 cajson]# cp *.pem /opt/kubernetes/ssl/ 分给其他节点： [root@master01 cajson]# scp *.pem 192.168.0.144: /opt/kubernetes/ssl/ [root@master01 cajson]# scp *.pem 192.168.0.145:/opt/kubernetes/ssl/ 注意： 集群中的node节点复用master节点的证书 |
2.11 需要TLS证书认证的组件
Kube-scheduler、kube-controller-manager一般和kube-apiserver部署在同一台机器上，它们使用非安全端口和kube-apiserver通信，非安全端口默认为http的8080，可以使用--insure-port指定，监听非安全端口的地址默认为127.0.0.1，可以使用--insure-bind-address指定。
Kubelet、kube-proxy、kubectl部署在其它Node节点上，如果通过安全端口访问kube-apiserver，则必须通过TLS证书认证，再通过RBAC授权。安全端口默认为https的6443，可以使用--secure-port指定，监听安全端口的地址默认为0.0.0.0(监听所有接口)，可以使用—bind-address指定。