k8s集群部署v1.15实践11:work节点配置kubelet

work节点配置kubelet

注:

所有work节点都要部署kubelet.

kublet 运行在每个 worker 节点上,接收 kube-apiserver 发送的请求,管理 Pod 容器,执行交互式命令,如 exec、run、logs 等.

kublet 启动时自动向 kube-apiserver 注册节点信息,内置的 cadvisor 统计和监控节点的资源使用情况.

为确保安全,本文档只开启接收 https 请求的安全端口,对请求进行认证和授权,拒绝未授权的访问(如 apiserver、heapster).

1.把kubelet的二进制执行文件分发到节点

分发

[root@k8s-node1 bin]# cp kubelet kube-proxy /opt/k8s/bin/
[root@k8s-node1 bin]# scp kubelet kube-proxy root@k8s-node2:/opt/k8s/bin/
kubelet                                                                                                      100%  114MB  57.1MB/s   00:02    
kube-proxy                                                                                                   100%   35MB  35.4MB/s   00:00    
[root@k8s-node1 bin]# scp kubelet kube-proxy root@k8s-node3:/opt/k8s/bin/
kubelet                                                                                                      100%  114MB  57.1MB/s   00:02    
kube-proxy                                                                                                   100%   35MB  38.2MB/s   00:00    
[root@k8s-node1 bin]#

修改下权限,节点都执行

 chown -R k8s /opt/k8s/bin

2.kubelet bootstrap基础知识

参考文档:Kubernetes - kubelet bootstrap 流程

1.kubelet bootstrap是什么

要让 api server 信任 worker,worker 得需要先过 master 认证鉴权这一关.k8s 以插件化的方式支持很多种认证方式,比如 token 文件,x509证书,service account 等等.其中就包含一个名为“Bootstrap Token Authentication”的认证方式,基本上 k8s 的默认安装就默认支持这种认证方式.这种方式最初就是设计用来给添加 worker node 用的.使用 Bootstrap Token Authentication 时,只需告诉 kubelet 一个特殊的 token,kubelet 自然就能通过 api server 的认证.

首先k8s集群中需要创建token.

有了 token,kubelet 就能通过 api server 的认证,连接建立,至此是不是万事大吉了.NO!

也许你已经注意到了,token 定义有 expiration 字段,表示这个 token 是有有效期的,过了有效期，token 失效,kubelet 就无法使用 token 跟 api server 通信,所以这个 token 只能作为 kubelet 初始化时跟 api server 的临时通信,而并非持久方案.

kubelet 最终还是需要使用证书跟 api server 通信,证书从何而来?如果让 k8s 的安装人员为每一个 worker node 生成并维护一份证书,工作量太大太繁琐,这也是设计 bootstrap token 要解决的问题之一,即:kubelet 使用低权限的 bootstrap token 跟 api server 建立连接后,要能够自动向 api server 申请自己的证书,并且 api server 要能够自动审批证书.

是的,很多人都会想到,k8s 支持 cert sign API.在 k8s 的证书认证中,要么由管理员手动为用户生成证书,要么是使用更为服务化的方式,由需要证书的用户自己向 k8s 申请.k8s 管理员只需在后台审批即可.但手动的为每一个用户审批,并没有带来很大的便利性,所以 k8s 还支持自动审批,只不过目前自动审批仅仅针对 kubelet.

k8s提供clusterrole让 bootstrap token 所代表的用户(username：system:bootstrap:，group：system:bootstrappers)申请的 csr 能够被自动审批.

bootstrap token 是 kubelet 引导的关键,如果其他人知道了 bootstrap token 那就意味着可以访问 k8s 的某些资源,所以要么给 bootstrap token 设定一个很短的有效期,要么 kubelet 引导结束后就手动删除,防止 bootstrap token 被再次使用.

kubelet 有了自己跟 api server 通信的证书,剩下的一个任务就是处理证书过期的问题.既然证书的生成都是自动化的,如果证书的 renew 需要手动那就太 low 了,所以跟生成证书的原理相似,k8s 也提供一个 clusterrole 可以赋予 kubelet 用户或组让 renew 证书的请求也能够被自动审批.

综合:配置work节点配置kubelet service,创建的东西见下:

---

创建了 bootstrap token

---

给 bootstrap token 代表的用户 system:bootstrap:abcdef 赋予 clusterole certificatesigningrequests.certificates.k8s.io/nodeclient 和 system:node-bootstrapper,让该用户可以访问 csr API 以及自动审批其创建的 csr

---

给新的 work node 代表的用户 system:node:test-node 赋予 clusterrole system:certificates.k8s.io:certificatesigningrequests:selfnodeclient,让它发送的证书 renew 的请求能被自动审批

---

3.配置kubelet bootstrap kubeconfig文件

在master节点创建使用的kubeconfig文件

注意1:不能缺少第一步,必须生成唯一的BOOTSTRAP_TOKEN.

[root@k8s-node1 kubelet]# export BOOTSTRAP_TOKEN=$(kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:k8s-node1 --kubeconfig ~/.kube/config)
[root@k8s-node1 kubelet]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.174.127:8443 --kubeconfig=kubelet-bootstrap-k8s-node1.kubeconfig
Cluster "kubernetes" set.
[root@k8s-node1 kubelet]# kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap-k8s-node1.kubeconfig
User "kubelet-bootstrap" set.
[root@k8s-node1 kubelet]# kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap-k8s-node1.kubeconfig
Context "default" created.
[root@k8s-node1 kubelet]# kubectl config use-context default --kubeconfig=kubelet-bootstrap-k8s-node1.kubeconfig
Switched to context "default".

[root@k8s-node1 kubelet]# ls
kubelet-bootstrap-k8s-node1.kubeconfig
[root@k8s-node1 kubelet]# cat kubelet-bootstrap-k8s-node1.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURzRENDQXBpZ0F3SUJBZ0lVUkppc3E0aEc2OGFVTlNFREJTSjdEVHR6WGlnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1hqRUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWxOYU1Rc3dDUVlEVlFRSEV3SlRXakVNTUFvRwpBMVVFQ2hNRGF6aHpNUkl3RUFZRFZRUUxFd2swVUdGeVlXUnBaMjB4RXpBUkJnTlZCQU1UQ210MVltVnlibVYwClpYTXdIaGNOTVRreE1ETXdNRFF4TmpBd1doY05NalF4TURJNE1EUXhOakF3V2pCZU1Rc3dDUVlEVlFRR0V3SkQKVGpFTE1Ba0dBMVVFQ0JNQ1Uxb3hDekFKQmdOVkJBY1RBbE5hTVF3d0NnWURWUVFLRXdOck9ITXhFakFRQmdOVgpCQXNUQ1RSUVlYSmhaR2xuYlRFVE1CRUdBMVVFQXhNS2EzVmlaWEp1WlhSbGN6Q0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFQYzF6YklnejFyYzlYL2liakFGc2lIdFFGL3JaSkZHaTU2dXhxcW8KOFpOMGNzKzZidkVMUG8vOFBzUUVLN2RZWnAzMGZTeWRTdVNETUNCNnFWaEs5TzVLamNvZXIvSENkazU2eCtVcwprZGc2VGpKOVJKRnhXdVdjbTBGUkpQVjVyUkhKOG56bHhXTmFhTXl2Y0NYRkYzN1N3K1BZa1NmRW5rTU5tbEE3ClFjRnJMbE5VeVVYN25CaVJXWXBCb0xJUEZqMUVLbFYyT3YxNU9qRzFZSEhkQTdWakd1VENYeTlwRUVSUHBjK2QKNHhsekVqNUJ3Rm1QczJZTGNWWlZMM1FQajJvUU82eGhWK25ybURJZWRpU3J4Y3ZVeTIwbFpoT3lxTFFFdDdQSApKUzZoRSt0T2dabUVPOU9kbm16QWNHa3FTZ0hyWVhTQ3pvUDZNNVFIdmFQY2FUY0NBd0VBQWFObU1HUXdEZ1lEClZSMFBBUUgvQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUl3SFFZRFZSME9CQllFRkM4cGNmeWcKbDFEQWh1bUk2ZkZXMjZ6R3FUWUdNQjhHQTFVZEl3UVlNQmFBRkM4cGNmeWdsMURBaHVtSTZmRlcyNnpHcVRZRwpNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUJQQ3BOV3lIS25KeERkWFpuUE0yR3piTkQ4K1JZWUJ5VVR3ZncrCmppbi9ZYkk3T3dWbVhJNHlJSlR6VkcxYi9HS0NxRWg4Q3U0QkR0b2FWV0Zhd2U4aGtNbmVVME1KaWV3SjBTdFEKTXlUaGpRZzE1MDd3RTVJUVdGNzNGU3luM0dHQTJFK0xQRGV0ZjdZdktxUGY3aFRlcnVneFFlZlFNRkdXVSszawoxcVJWN3I0N0NMbG0yakhXeWhlVm40Zkl5YXpVZVpvdnhFamhXSGZ3VzRmNEdYT0wvNDQ2Tmx2MUk5TWMvNmN5Cmp1TnhQc202RjVjQmlnR1YxY3IwSFEzL0ZUb1p1T1hZRWJBZVp3SHpHZWZUQ2RGTzJud29RRGFOMXpnRGxmZjEKRXFERERtM0JFRGt2Um1CZ0NwcTdnQmt0UTlSUFJCMXlDNjVFbGxaVG5Zcm1CN2NtCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.174.127:8443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: oe8cld.p3mvpb7on8s07sxy
[root@k8s-node1 kubelet]# 

分发 bootstrap kubeconfig 文件到 worker 节点

[root@k8s-node1 kubelet]# cp kubelet-bootstrap-k8s-node1.kubeconfig  /etc/kubernetes/kubelet-bootstrap.kubeconfig
[root@k8s-node1 kubelet]# scp kubelet-bootstrap-k8s-node1.kubeconfig   root@k8s-node2:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-node1.kubeconfig                                                                       100% 2132     2.3MB/s   00:00    
[root@k8s-node1 kubelet]# scp kubelet-bootstrap-k8s-node1.kubeconfig   root@k8s-node3:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-node1.kubeconfig                                                                       100% 2132     2.4MB/s   00:00    
[root@k8s-node1 kubelet]#

*4.创建和分发kubelet参数配置文件

默认见下

[root@k8s-node1 kubelet]# cat kubelet.config.json.template 
{
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1",
"authentication": {
"x509": {
"clientCAFile": "/etc/kubernetes/cert/ca.pem"
},
"webhook": {
"enabled": true,
"cacheTTL": "2m0s"
},
"anonymous": {
"enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
"address": "##NODE_IP##",
"port": 10250,
"readOnlyPort": 0,
"cgroupDriver": "cgroupfs",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"featureGates": {
"RotateKubeletClientCertificate": true,
"RotateKubeletServerCertificate": true
},
"clusterDomain": "${CLUSTER_DNS_DOMAIN}",
"clusterDNS": ["${CLUSTER_DNS_SVC_IP}"]
}

address：API 监听地址,不能为 127.0.0.1,否则 kube-apiserver、heapster 等不能调用 kubelet 的 API.

readOnlyPort=0:关闭只读端口(默认 10255),等效为未指定.

authentication.anonymous.enabled:设置为 false,不允许匿名访问 10250 端口.

authentication.x509.clientCAFile:指定签名客户端证书的 CA 证书,开启 HTTP 证书认证.

authentication.webhook.enabled=true:开启 HTTPs bearer token 认证.对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端),将被拒绝,提示 Unauthorized.

authroization.mode=Webhook:kubelet 使用 SubjectAcce***eview API 查询kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC).

featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate:自动 rotate 证书,证书的有效期取决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数.

需要 root 账户运行.

分发

[root@k8s-node1 kubelet]# cp kubelet.config.json.template /etc/kubernetes/kubelet.config.json
[root@k8s-node1 kubelet]# scp kubelet.config.json.template root@k8s-node2:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template                                                                                 100%  704   715.4KB/s   00:00    
[root@k8s-node1 kubelet]# scp kubelet.config.json.template root@k8s-node3:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template                                                                                 100%  704   669.0KB/s   00:00    
[root@k8s-node1 kubelet]# 

配置文件里的变量修改

NODE_IP

NODE_IP改成每个节点对应的IP

[root@k8s-node1 kubelet]#  echo ${CLUSTER_DNS_DOMAIN}
cluster.local.
[root@k8s-node1 kubelet]# echo ${CLUSTER_DNS_SVC_IP}
10.254.0.2
[root@k8s-node1 kubelet]# 

这两个变量改成真实地址

5.创建和分发 kubelet systemd unit 文件

[root@k8s-node1 kubelet]# cat kubelet.service.template 
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/k8s/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/cert \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.config.json \
  --hostname-override=##nodename## \
  --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \
  --allow-privileged=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
[root@k8s-node1 kubelet]#

WorkingDirectory=/var/lib/kubelet ##目录默认没有，手动去创建

--hostname-override=##nodename## ##nodename修改成在节点的名字

分发

[root@k8s-node1 kubelet]# cp kubelet.service.template /etc/systemd/system/kubelet.service
[root@k8s-node1 kubelet]# scp kubelet.service.template root@k8s-node2:/etc/systemd/system/kubelet.service
kubelet.service.template                                                                                     100%  753   773.8KB/s   00:00    
[root@k8s-node1 kubelet]# scp kubelet.service.template root@k8s-node3:/etc/systemd/system/kubelet.service
kubelet.service.template                                                                                     100%  753   898.7KB/s   00:00    
[root@k8s-node1 kubelet]#

创建目录

[root@k8s-node1 kubelet]# mkdir -p /var/lib/kubelet && chown -R k8s /var/lib/kubelet
[root@k8s-node1 kubelet]# ssh k8s-node2 "mkdir -p /var/lib/kubelet && chown -R k8s /var/lib/kubelet"
[root@k8s-node1 kubelet]# ssh k8s-node3 "mkdir -p /var/lib/kubelet && chown -R k8s /var/lib/kubelet"

修改hostname,每个节点名字对应修改下即可使用,用条sed命令,参考见下：

sed -i 's/##nodename##/k8s-node1/' /etc/systemd/system/kubelet.service

6.Bootstrap Token Auth 和授予权限

kublet 启动时查找配置的 --kubeletconfig 文件是否存在,如果不存在则使用 --bootstrapkubeconfig向 kube-apiserver 发送证书签名请求 (CSR).kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证（事先使用 kubeadm 创建
的 token),认证通过后将请求的 user 设置为 system:bootstrap:group 设置为system:bootstrappers,这一过程称为 Bootstrap Token Auth

[root@k8s-node1 kubelet]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
[root@k8s-node1 kubelet]#

7.启动服务

systemctl enable kubelet  && systemctl start kubelet

1.15前的版本可以正常启动,1.15后的版本会启动报错,报错见下:

Nov 05 03:47:51 k8s-node3 kubelet[58295]: F1105 03:47:51.615082   58295 server.go:156] unknown flag: --allow-privileged

把systemd unit文件里的--allow-privileged去掉即可

[root@k8s-node1 kubelet]# systemctl daemon-reload && systemctl restart kubelet
[root@k8s-node1 kubelet]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-11-05 03:52:41 EST; 1s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 59846 (kubelet)
    Tasks: 7
   Memory: 15.6M
   CGroup: /system.slice/kubelet.service
           └─59846 /opt/k8s/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --cert-dir=/etc/kubernetes/cert ...
Nov 05 03:52:41 k8s-node1 kubelet[59846]: I1105 03:52:41.728647   59846 feature_gate.go:216] feature gates: &{map[RotateKubeletClien...e:true]}
Nov 05 03:52:41 k8s-node1 kubelet[59846]: I1105 03:52:41.731842   59846 feature_gate.go:216] feature gates: &{map[RotateKubeletClien...e:true]}
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.129232   59846 mount_linux.go:175] Detected OS with systemd
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.129321   59846 server.go:425] Version: v1.15.5
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.129366   59846 feature_gate.go:216] feature gates: &{map[RotateKubeletClien...e:true]}
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.129409   59846 feature_gate.go:216] feature gates: &{map[RotateKubeletClien...e:true]}
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.129556   59846 plugins.go:103] No cloud provider specified.
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.129576   59846 server.go:541] No cloud provider specified: "" from the config file: ""
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.129658   59846 bootstrap.go:117] Using bootstrap kubeconfig to generate TLS...fig file
Nov 05 03:52:42 k8s-node1 kubelet[59846]: I1105 03:52:42.131845   59846 bootstrap.go:148] No valid private key and/or certificate fo... new one
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-node1 kubelet]# 

8.kubelet启动的csr情况

kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求,当这个CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 --kubeletconfig 文件.

注意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --cluster-signing-key-file 参数,才会为 TLS Bootstrap 创建证书和私钥.

[root@k8s-node1 kubelet]# kubectl get csr
NAME                                                   AGE    REQUESTOR                 CONDITION
node-csr-9vLGgTIIPzxdkYKc11PM4yEiJq6qio9hvkNmX44g11I   80s    system:bootstrap:oe8cld   Pending
node-csr-BUMOL8xAhk5y0ZcozWKKnsfRaP5RSIjwpsxx7L3Izgs   77s    system:bootstrap:oe8cld   Pending
node-csr-zECMBpfqGy7C7J0VKtrCgTVAplHkKdsqt8-bCkbGl10   4m7s   system:bootstrap:oe8cld   Pending
[root@k8s-node1 kubelet]# 

approve kubelet CSR 请求可以手动或自动 approve CSR 请求.推荐使用自动的方式,因为从 v1.8 版本开始,可以自动轮转approve csr 后生成的证书.

9.创建自动 approve CSR请求

创建三个 ClusterRoleBinding,分别用于自动 approve client、renew client、renewserver 证书

[root@k8s-node1 kubelet]# cat csr-crb.yaml 
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: auto-approve-csrs-for-group
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  apiGroup: rbac.authorization.k8s.io
---
# To let a node of the group "system:nodes" renew its own credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-client-cert-renewal
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]
---
# To let a node of the group "system:nodes" renew its own server credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-server-cert-renewal
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: approve-node-server-renewal-csr
  apiGroup: rbac.authorization.k8s.io
[root@k8s-node1 kubelet]#

[root@k8s-node1 kubelet]# kubectl apply -f csr-crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/auto-approve-csrs-for-group created
clusterrolebinding.rbac.authorization.k8s.io/node-client-cert-renewal created
clusterrole.rbac.authorization.k8s.io/approve-node-server-renewal-csr created
clusterrolebinding.rbac.authorization.k8s.io/node-server-cert-renewal created

等待一段时间(1-10 分钟),节点的 CSR 都被自动 approve.

[root@k8s-node1 kubelet]# kubectl get nodes
NAME        STATUS   ROLES    AGE    VERSION
k8s-node1   Ready       3m6s   v1.15.5
k8s-node2   Ready       19s    v1.15.5
k8s-node3   Ready       16s    v1.15.5
[root@k8s-node1 kubelet]# kubectl get csr
NAME                                                   AGE   REQUESTOR                 CONDITION
node-csr-9vLGgTIIPzxdkYKc11PM4yEiJq6qio9hvkNmX44g11I   14m   system:bootstrap:oe8cld   Approved,Issued
node-csr-BUMOL8xAhk5y0ZcozWKKnsfRaP5RSIjwpsxx7L3Izgs   13m   system:bootstrap:oe8cld   Approved,Issued
node-csr-zECMBpfqGy7C7J0VKtrCgTVAplHkKdsqt8-bCkbGl10   16m   system:bootstrap:oe8cld   Approved,Issued
[root@k8s-node1 kubelet]#