kubernetes-1.11.0集群部署之master集群 (二)
Master
master 的组件有以下:
1. API Server
Api server提供HTTP/HTTPS RESTful API,既kubernetes API,API server 是kubernetes Cluster的前端接口，各种客户端工具(CLI或UI)以及kubernetes其他组件可以通过它管理cluster各种资源
2. Scheduler(kube-scheduler)
scheduler 负责决定Pod放在哪个node 上运行.scheduler 在调度时会充分考虑cluster 的拓扑结构，当前各个节点的负载，以及应用对高可用、性能、数据亲和性的需求.
3. Controller Manager(kube-controller-manager)
Controller Manager 负责管理CLuster 各种资源，保障资源处于预期的状态，Controller Manager 由多种controller 组成，包括replication controller、endpoints controller 、namespace controller、serviceaccounts controller 等。
不同的controller 管理不同的资源，例如,replication controller管理Deployment、SatefulSet、DaemonSet 的生命周期,namespace controller 管理namespace 资源
4. etcd
etcd负责保存kubernetes Cluster的配置信息和各种资源的状态信息.当数据发生变化时，etcd快速通知kubernetes 相关组件
5.Pod
Pod 要能够相互通信，kubernetes Cluster 必须部署Pod 网络，网络组件可选择fannel calico
(kubectl 安装在所有需要进行操作的机器上）
6.master 需要安装的组建
kube-apiserver，kube-controller-manager，kube-scheduler，kubectl，kubelet，kubeadm安装组件
将之前编译好的二进制文件,kube-apiserver，kube-controller-manager，kube-scheduler，kubectl，kubelet，kubeadm 拷贝到/usr/local/bin/ 所有master 进行同步
创建admin 证书
kubectl 与 kube-apiserver 的安全端口通信，需要为安全通信提供TLS 证书和秘钥
vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
生成admin 证书和秘钥
cd /opt/ssl/
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
证书拷贝到master 的机器上，154，156.159
scp ssl/admin* [email protected]:/etc/kubernetes/ssl/
...........生成kubernetes配置文件
#生成证书相关配置文件存储于/root/.kube 目录中
配置kubernetes 集群
在10.39.10.154 这台机器上操作
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://127.0.0.1:6443
配置客户端认证
kubectl config set-credentials admin --client-certificate=/etc/kubernetes/ssl/admin.pem --embed-certs=true --client-key=/etc/kubernetes/ssl/admin-key.pem
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin
kubectl config use-context kubernetes创建kubernetes 证书
vim kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.39.10.154",
"10.39.10.156",
"10.39.10.159",
"10.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
# 这里 hosts 字段中 三个 IP 分别为 127.0.0.1 本机，10.39.10.154 和 10.39.10.156，10.39.10.159 为 Master 的IP，多个Master需要写多个。 10.254.0.1 为 kubernetes SVC 的 IP， 一般是 部署网络的第一个IP , 如: 10.254.0.1 ， 在启动完成后，我们使用 kubectl get svc ， 就可以查看到
生成kubernetes 证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
拷贝kubernetes*.pem /etc/kubernetes/ssl/
scp kubernetes*.pem [email protected]:/etc/kubernetes/ssl/
........配置kube-apiserver
[root@devops ssl]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
da4090c6baadef99e577a9ac5da6f684
创建encryption-config.yaml 配置
cat > encryption-config.yaml <[https://kubernetes.io/docs/tasks/debug-application-cluster/audit/]()
cat >> audit-policy.yaml <启动kube-apiserver
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver 配置kube-controller-manager
vim /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--address=0.0.0.0 \
--master=http://127.0.0.1:8080 \
--allocate-node-cidrs=true \
--service-cluster-ip-range=10.254.0.0/18 \
--cluster-cidr=10.254.64.0/18 \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--feature-gates=RotateKubeletServerCertificate=true \
--controllers=*,tokencleaner,bootstrapsigner \
--experimental-cluster-signing-duration=86700h0m0s \
--cluster-name=kubernetes \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--leader-elect=true \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--pod-eviction-timeout=5m0s \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target启动kube-controller-manager
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager
systemctl status kube-controller-manager配置kube-scheduler
vi /etc/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-scheduler \
--address=0.0.0.0 \
--master=http://127.0.0.1:8080 \
--leader-elect=true \
--v=1
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target 启动kube-scheduler
systemctl daemon-reload
systemctl enable kube-scheduler
systemctl start kube-scheduler
systemctl status kube-scheduler 验证master 节点
每台master 上执行kubectl get componentstatuses 
配置 kubelet 认证
kubelet 授权 kube-apiserver 的一些操作 exec run logs 等
# RBAC 只需创建一次就可以
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes创建 bootstrap kubeconfig 文件
#注意: token 生效时间为 1day , 超过时间未创建自动失效，需要重新创建 token
# 创建 集群所有 kubelet 的 token
在master 上执行
kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:kubernetes-master-154 --kubeconfig ~/.kube/config
kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:kubernetes-master-156 --kubeconfig ~/.kube/config
kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:kubernetes-master-159 --kubeconfig ~/.kube/config
# 查看生成的token
kubeadm token list --kubeconfig ~/.kube/config 
以下为了区分会先生成node名称加bootstrap.kubeconfig
生成kubernetes-master-154 bootstrap.kubeconfig
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://127.0.0.1:6443 --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig
#配置客户端认证
kubectl config set-credentials kubelet-bootstrap --token=aaa8j5.4nvwg82imbrzb7r2 --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig
#配置关联
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig
#配置默认关键
kubectl config use-context default --kubeconfig=kubernetes-master-154-bootstrap.kubeconfig
拷贝生成的kubernetes-master-154-bootstrap.kubeconfig 文件
mv kubernetes-master-154-bootstrap.kubeconfig /etc/kubernetes/bootstrap.kubeconfig
生成kubernetes-master-156 bootstrap.kubeconfig
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://127.0.0.1:6443 --kubeconfig=kubernetes-master-156-bootstrap.kubeconfig
#配置客户端认证
kubectl config set-credentials kubelet-bootstrap --token=rz2col.l8x1x9dg5kg7jjw6 --kubeconfig=kubernetes-master-156-bootstrap.kubeconfig
#配置关联
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=kubernetes-master-156-bootstrap.kubeconfig
#配置默认关联
kubectl config use-context default --kubeconfig=kubernetes-master-156-bootstrap.kubeconfig
#拷贝生成的kubernetes-master-156-bootstrap.kubeconfig
mv kubernetes-master-156-bootstrap.kubeconfig /etc/kubernetes/bootstrap.kubeconfig
# 生成159的bootstrap.kubeconfig
#配置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=kubernetes-master-159-bootstrap.kubeconfig
#配置客户端认证
kubectl config set-credentials kubelet-bootstrap \
--token=9ocdef.pjd1s7twtro2ho8a \
--kubeconfig=kubernetes-master-159-bootstrap.kubeconfig
#配置关联
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=kubernetes-master-159-bootstrap.kubeconfig
#配置默认关联
kubectl config use-context default --kubeconfig=kubernetes-master-159-bootstrap.kubeconfig
#拷贝生成
mv kubernetes-master-159-bootstrap.kubeconfig /etc/kubernetes/bootstrap.kubeconfig 配置bootstrap RBAC 权限
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers创建自动批准相关CSR请求ClusterRole
vim /etc/kubernetes/tls-instructs-csr.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
#导入yaml 文件
kubectl apply -f /etc/kubernetes/tls-instructs-csr.yaml
#查看
kubectl describe ClusterRole/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
#将ClusterRole 绑定到适当的用户组
#自动批准 system:bootstrappers 组用户 TLS bootstrapping 首次申请证书的 CSR 请求
kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --group=system:bootstrappers
#自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求
kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
#自动批准 system:nodes 组用户更新 kubelet 10250 api 端口证书的 CSR 请求
kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes创建 kubelet.service 文件
关于kubectl get node 中的ROLES 的标签
单 Master 打标签 kubectl label node kubernetes-master-154 node-role.kubernetes.io/master=''
这里需要将 单Master 更改为 NoSchedule
更新标签命令为 kubectl taint nodes kubernetes-64 node-role.kubernetes.io/master=:NoSchedule
既 Master 又是 node 打标签 kubectl label node kubernetes-65 node-role.kubernetes.io/master=””
单 Node 打标签 kubectl label node kubernetes-66 node-role.kubernetes.io/node=””
关于删除 label 可使用 - 号相连 如: kubectl label nodes kubernetes-65 node-role.kubernetes.io/node-
#创建kubelet.service
vi /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
--hostname-override=kubernetes-master-154 \
--pod-infra-container-image=jicki/pause-amd64:3.1 \
--bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--config=/etc/kubernetes/kubelet.config.json \
--cert-dir=/etc/kubernetes/ssl \
--logtostderr=true \
--v=2
[Install]
WantedBy=multi-user.target
# 创建 kubelet config 配置文件
vi /etc/kubernetes/kubelet.config.json
{
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1",
"authentication": {
"x509": {
"clientCAFile": "/etc/kubernetes/ssl/ca.pem"
},
"webhook": {
"enabled": true,
"cacheTTL": "2m0s"
},
"anonymous": {
"enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
"address": "10.39.10.154",
"port": 10250,
"readOnlyPort": 0,
"cgroupDriver": "cgroupfs",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"RotateCertificates": true,
"featureGates": {
"RotateKubeletClientCertificate": true,
"RotateKubeletServerCertificate": true
},
"MaxPods": "512",
"failSwapOn": false,
"containerLogMaxSize": "10Mi",
"containerLogMaxFiles": 5,
"clusterDomain": "cluster.local.",
"clusterDNS": ["10.254.0.2"]
} 启动kubelet
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet总结一下
master 安装的有
systemctl status kubelet
systemctl status kube-scheduler
systemctl status kube-controller-manager
systemctl status kube-apiserver