Kubernetes 集群部署之Master部署（三）

目录

Master 组件

kube-apiserver

kube-controller-manager

kube-scheduler

1、基础环境准备

2、ETC集群部署

3、部署Master

切换目录

生成kube-apiserver证书

生成证书

使用自签ca证书办法kube-apiserver证书

生成证书

4、安装docker（所有节点）

下载软件包

 解压软件包

 编辑启动文件

创建配置文件

启动服务

5、准备软件包

下载

创建软件包目录

解压软件包

拷贝文件到指定目录

拷贝证书

5、部署kube-apiserver

创建配置文件

参数说明

创建token文件

创建systemd文件来管理apiserver

启动并设置开机自启动

授权kubelet-bootstrap用户允许请求证书

6、部署kube-controller-manager

创建配置文件

参数说明

创建systemd文件来管理controller-manager

启动并设置开机自启动

6、kube-scheduler部署

创建配置文件

参数说明

创建systemd管理scheduler

启动并设置开机自启动

---

Master 组件

Master组件可以在集群中任何节点上运行。但是为了简单起见，通常在一台VM/机器上启动所有Master组件，并且不会在此VM/机器上运行用户容器

• kube-apiserver

kube-apiserver用于暴露Kubernetes API。任何的资源请求/调用操作都是通过kube-apiserver提供的接口进行。

• kube-controller-manager

kube-controller-manager运行管理控制器，它们是集群中处理常规任务的后台线程。逻辑上，每个控制器是一个单独的进程，但为了降低复杂性，它们都被编译成单个二进制文件，并在单个进程中运行。

这些控制器包括：

1. 节点（Node）控制器。
2. 副本（Replication）控制器：负责维护系统中每个副本中的pod。
3. 端点（Endpoints）控制器：填充Endpoints对象（即连接Services＆Pods）。
4. Service Account和Token控制器：为新的Namespace 创建默认帐户访问API Token

• kube-scheduler

kube-scheduler 监视新创建没有分配到Node的Pod，为Pod选择一个Node。

1、基础环境准备

Kubernetes 集群部署之基础环境准备_集群部署基本的环境_abel_dwh的博客-CSDN博客

2、ETC集群部署

Kubernetes 集群部署之ETCD集群部署_kubernetes etcd集群_abel_dwh的博客-CSDN博客

3、部署Master

• 切换目录

[root@master ssl]# cd /root/TLS/k8s/

• 生成kube-apiserver证书

[root@master k8s]# cat ca-config.json 
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}

[root@master k8s]# cat ca-csr.json 
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}

• 生成证书

[root@master k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/05/10 18:13:10 [INFO] generating a new CA key and certificate from CSR
2021/05/10 18:13:10 [INFO] generate received request
2021/05/10 18:13:10 [INFO] received CSR
2021/05/10 18:13:10 [INFO] generating key: rsa-2048
2021/05/10 18:13:10 [INFO] encoded CSR
2021/05/10 18:13:10 [INFO] signed certificate with serial number 27631474529929329151854966158487610252489583230


[root@master k8s]# ls *.pem
ca-key.pem  ca.pem

• 使用自签ca证书办法kube-apiserver证书

[root@master k8s]# cat server-csr.json 
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.44.128",
"192.168.44.129",
"192.168.44.130",
"192.168.44.131",  #132-134为预留IP地址
"192.168.44.132",
"192.168.44.133",
"192.168.44.134",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}

• 生成证书

[root@master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json --profile=kubernetes server-csr.json | cfssljson -bare server                    
2021/05/10 18:19:22 [INFO] generate received request
2021/05/10 18:19:22 [INFO] received CSR
2021/05/10 18:19:22 [INFO] generating key: rsa-2048
2021/05/10 18:19:23 [INFO] encoded CSR
2021/05/10 18:19:23 [INFO] signed certificate with serial number 607707363525984056077203895987947070954749588362
2021/05/10 18:19:23 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@master k8s]# ls server*.pem
server-key.pem  server.pem

4、安装docker（所有节点）

• 下载软件包

[root@master ~]# wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

•  解压软件包

[root@master ~]# tar zxvf docker-19.03.9.tgz
[root@master ~]# mv docker/* /usr/bin

•  编辑启动文件

[root@master ~]# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP 
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target

• 创建配置文件

[root@master ~]# mkdir /etc/docker
[root@master ~]# cat /etc/docker/daemon.json 
{
"registry-mirrors": ["https://e3bi90pi.mirror.aliyuncs.com"]
}

• 启动服务

[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl start docker
[root@master ~]# systemctl enable docker

5、准备软件包

• 下载

wget https://dl.k8s.io/v1.18.18/kubernetes-server-linux-amd64.tar.gz

• 创建软件包目录

[root@master ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

• 解压软件包

[root@master ~]# tar -xvf kubernetes-server-linux-amd64.tar.gz 
kubernetes/
kubernetes/addons/
kubernetes/LICENSES
kubernetes/kubernetes-src.tar.gz
kubernetes/server/
kubernetes/server/bin/
kubernetes/server/bin/kube-scheduler.tar
kubernetes/server/bin/kube-controller-manager.docker_tag
kubernetes/server/bin/kube-proxy.docker_tag
kubernetes/server/bin/kube-proxy.tar
kubernetes/server/bin/kube-proxy
kubernetes/server/bin/kube-scheduler
kubernetes/server/bin/kube-apiserver
kubernetes/server/bin/kubeadm
kubernetes/server/bin/mounter
kubernetes/server/bin/kube-apiserver.tar
kubernetes/server/bin/apiextensions-apiserver
kubernetes/server/bin/kube-apiserver.docker_tag
kubernetes/server/bin/kube-controller-manager
kubernetes/server/bin/kubelet
kubernetes/server/bin/kubectl
kubernetes/server/bin/kube-controller-manager.tar
kubernetes/server/bin/kube-scheduler.docker_tag

• 拷贝文件到指定目录

[root@master ~]# cd kubernetes/server/bin
[root@master bin]# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
[root@master bin]# cp kubectl /usr/bin/

• 拷贝证书

[root@master ~]# cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

5、部署kube-apiserver

• 创建配置文件

[root@master bin]# cat /opt/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://192.168.44.129:2379,https://192.168.44.130:2379,https://192.168.44.128:2379 \
--bind-address=192.168.44.128 \
--secure-port=6443 \
--advertise-address=192.168.44.128 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

• 参数说明

--logtostderr：启用日志
---v：日志等级
--log-dir：日志目录
--etcd-servers：etcd集群地址
--bind-address：监听地址
--secure-port：https安全端口
--advertise-address：集群通告地址
--allow-privileged：启用授权
--service-cluster-ip-range：Service虚拟IP地址段
--enable-admission-plugins：准入控制模块
--authorization-mode：认证授权，启用RBAC授权和节点自管理
--enable-bootstrap-token-auth：启用TLS bootstrap机制
--token-auth-file：bootstrap token文件
--service-node-port-range：Service nodeport类型默认分配端口范围
--kubelet-client-xxx：apiserver访问kubelet客户端证书
--tls-xxx-file：apiserver https证书
--etcd-xxxfile：连接Etcd集群证书
--audit-log-xxx：审计日志

• 创建token文件

备注：格式：token，用户名，UID，用户组

[root@master bin]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
267137427e1cb7519d63974aa7598091
[root@master bin]# vim /opt/kubernetes/cfg/token.csv
[root@master bin]# cat /opt/kubernetes/cfg/token.csv
267137427e1cb7519d63974aa7598091,kubelet-bootstrap,10001,"system:node-bootstrapper"

• 创建systemd文件来管理apiserver

[root@master bin]# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

• 启动并设置开机自启动

[root@master kubernetes]#  systemctl daemon-reload
[root@master kubernetes]#  systemctl start kube-apiserver
[root@master kubernetes]#  systemctl enable kube-apiserver


[root@master kubernetes]# kubectl get cs                          
NAME                 STATUS      MESSAGE                                                                                     ERROR
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused   
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused   
etcd-0               Healthy     {"health":"true"}                                                                           
etcd-1               Healthy     {"health":"true"}                                                                           
etcd-2               Healthy     {"health":"true"} 

• 授权kubelet-bootstrap用户允许请求证书

[root@master kubernetes]# kubectl create clusterrolebinding kubelet-bootstrap \
> --clusterrole=system:node-bootstrapper \
> --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

6、部署kube-controller-manager

• 创建配置文件

[root@master kubernetes]# cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
> KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
> --v=2 \\
> --log-dir=/opt/kubernetes/logs \\
> --leader-elect=true \\
> --master=127.0.0.1:8080 \\
> --bind-address=127.0.0.1 \\
> --allocate-node-cidrs=true \\
> --cluster-cidr=10.244.0.0/16 \\
> --service-cluster-ip-range=10.0.0.0/24 \\
> --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
> --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
> --root-ca-file=/opt/kubernetes/ssl/ca.pem \\
> --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
> --experimental-cluster-signing-duration=87600h0m0s"
> EOF

• 参数说明

--master：通过本地非安全本地端口8080连接apiserver。
--leader-elect：当该组件启动多个时，自动选举（HA）
--cluster-signing-cert-file/--cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

• 创建systemd文件来管理controller-manager

[root@master ~]# cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
> [Unit]
> Description=Kubernetes Controller Manager
> Documentation=https://github.com/kubernetes/kubernetes
> [Service]
> EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
> ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
> Restart=on-failure
> [Install]
> WantedBy=multi-user.target
> EOF

• 启动并设置开机自启动

[root@master kubernetes]# systemctl daemon-reload
[root@master kubernetes]# systemctl start kube-controller-manager
[root@master kubernetes]# systemctl enable kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
[root@master ~]# kubectl get cs
NAME                 STATUS      MESSAGE                                                                                     ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Healthy     ok                                                                                          
etcd-1               Healthy     {"health":"true"}                                                                           
etcd-2               Healthy     {"health":"true"}                                                                           
etcd-0               Healthy     {"health":"true"}  

6、kube-scheduler部署

• 创建配置文件

[root@master kubernetes]# cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
> KUBE_SCHEDULER_OPTS="--logtostderr=false \\
> --v=2 \\
> --log-dir=/opt/kubernetes/logs \\
> --leader-elect \\
> --master=127.0.0.1:8080 \\
> --bind-address=127.0.0.1"
> EOF

• 参数说明

--master：通过本地非安全本地端口8080连接apiserver。
--leader-elect：当该组件启动多个时，自动选举（HA）

• 创建systemd管理scheduler

[root@master kubernetes]# cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
> [Unit]
> Description=Kubernetes Scheduler
> Documentation=https://github.com/kubernetes/kubernetes
> [Service]
> EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
> ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
> Restart=on-failure
> [Install]
> WantedBy=multi-user.target
> EOF

• 启动并设置开机自启动

[root@master kubernetes]# systemctl daemon-reload
[root@master kubernetes]# systemctl start kube-scheduler
[root@master kubernetes]# systemctl enable kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@master kubernetes]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}