01 kubernetes二进制部署
kubernetes二进制部署文档-集群部分
文章目录
-
- kubernetes二进制部署文档-集群部分
- 一、系统规划
-
-
- 1.1 系统组件分布
- 1.2 部署拓扑
- 1.3 系统环境
-
- 二、初始化系统环境
-
-
- 2.1 升级系统内核
- 2.2内核优化
- 2.3 启用Ipvs内核模块
- 2.4 安装常用工具
- 2.5 安装NTP时间服务
- 2.6 关闭Swap分区
- 2.7 关闭系统防火墙和selinux
- 2.8 设置集群内互为免密登陆
-
- 三、部署容器环境
-
-
- 3.1 部署docker
- 3.2 检查docker环境
-
- 四、部署Kubernetes
-
-
- 4.1 准备集群根证书
-
- 4.1.1 安装cfssl
- 4.1.2 创建生成CA证书签名请求（csr）的json配置文件
- 4.1.3 生成CA证书、私钥和csr证书签名请求
- 4.1.4 创建基于根证书的config配置文件
- 4.1.6 创建metrics-server证书
-
- 4.1.6.1 创建 metrics-server 使用的证书
- 4.1.6.2 生成 metrics-server 证书和私钥
- 4.1.6.3 分发证书
- 4.1.7 创建Node节点kubeconfig文件
-
- 4.1.7.1 创建TLS Bootstrapping Token、 kubelet kubeconfig、kube-proxy kubeconfig
- 4.1.7.2 生成证书
- 4.1.7.3 分发证书
- 4.2 部署Etcd集群
-
- 4.2.1 基础设置
- 4.2.2 分发证书
- 4.2.3 编写etcd配置文件脚本 etcd-server-startup.sh
- 4.2.4 执行etcd.sh 生成配置脚本
- 4.2.4 检查etcd01运行情况
- 4.2.5 分发etcd-server-startup.sh脚本至master02、master03上
- 4.2.6 启动etcd02并检查运行状态
- 4.2.7 启动etcd03并检查运行状态
- 4.2.8 检查etcd集群状态
- 4.3 部署APISERVER
-
- 4.3.1 下载kubernetes二进制安装包并解压
- 4.3.4 配置master组件并运行
-
- 4.3.4.1 创建 kube-apiserver 配置文件脚本
- 4.3.4.2 创建生成 kube-controller-manager 配置文件脚本
- 4.3.4.3 创建生成 kube-scheduler 配置文件脚本
- 4.3.5 复制执行文件
- 4.3.7 添加权限并分发至master02和master03
- 4.3.8 运行配置文件
-
- 4.3.8.1 运行k8smaster01
- 4.3.8.2 运行k8smaster02
- 4.3.8.3 运行k8smaster03
- 4.3.9 验证集群健康状态
- 4.4 部署kubelet
-
- 4.4.1 配置kubelet证书自动续期和创建Node授权用户
-
- 4.4.1.1 创建 `Node节点` 授权用户 `kubelet-bootstrap`
- 4.4.1.2 创建自动批准相关 CSR 请求的 ClusterRole
- 4.4.1.3 自动批准 kubelet-bootstrap 用户 TLS bootstrapping 首次申请证书的 CSR 请求
- 4.4.1.4 自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求
- 4.4.1.5 自动批准 system:nodes 组用户更新 kubelet 10250 api 端口证书的 CSR 请求
- 4.4.2 各节点准备基础镜像
- 4.4.3 创建`kubelet`配置脚本
- 4.4.4 master节点启用`kubelet`
- 4.4.5 worker节点启用`kubelet`
- 4.5 部署kube-proxy
-
- 4.5.6 创建 `kube-proxy` 配置脚本
- 4.5.7 master节点启动`kube-proxy`
- 4.5.8 worker节点启动`kube-proxy`
- 4.6 检测状态
- 4.7 其他补充设置
-
- 4.7.1 master节点添加taint
- 4.7.2命令补全
-
- 五、部署网络组件
-
-
- 5.1 部署calico网络 BGP模式
-
- 5.1.1 修改部分配置
- 5.1.2 应用calico文件
- 5.2 部署coredns
-
- 5.2.1 编辑coredns配置文件
- 5.2.2 应用部署coredns
-
- 六、部署traefik
-
-
- 6.1 编辑traefik资源配置文件
-
- 6.1.1 编辑svc
- 6.1.2 编辑rbac
- 6.1.3 编辑ingress-traefik
- 6.1.4 编辑deployment
- 6.2 应用配置文件
- 6.3 检查状态
- 6.4 添加域名解析及反向代理
- 6.6 测试traefikdashboard
-
- 七、部署Kubernetes Dashboard
-
-
- 7.1 下载Dashboard的yaml文件
- 7.2 创建Dashboard的命令空间和存储
- 7.3 应用recommended
- 7.4 创建认证
-
- 7.4.1 创建登陆认证和获取token
- 7.4.2 创建自签证书
- 7.4.3 创建secret对象来引用证书文件
- 7.4.4 创建dashboard的ingressrrouter
- 7.4.5 配置nginx反代代理
-
- 7.4.5.1 分发pem和key文件至nginx
- 7.4.5.2 修改nginx.conf
- 7.5 配置DNS解析
- 7.6 登陆测试
- 7.7 配置dashboard分权
-
- 7.7.1 创建ServiceAccount 、ClusterRole和ClusterRoleBinding
-
- 后续参考：
提供者：MappleZF
版本：1.0.0
一、系统规划
1.1 系统组件分布
| 主机名 | IP | 组件 |
|---|---|---|
| lbvip.host.com | 192.168.13.100 | 虚拟VIP地址，由PCS+Haproxy提供 |
| k8smaster01.host.com | 192.168.13.101 | etcd kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy docker calico Ipvs Ceph: mon mgr mds osd |
| k8smaster02.host.com | 192.168.13.102 | etcd kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy docker calico Ipvs Ceph:mon mgr mds ods |
| k8smaster03.host.com | 192.168.13.103 | etcd kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy docker calico Ipvs Ceph: mon mgr mds |
| k8sworker01.host.com | 192.168.13.105 | kubelet kube-proxy docker calico Ipvs osd |
| k8sworker02.host.com | 192.168.13.106 | kubelet kube-proxy docker calico Ipvs osd |
| k8sworker03.host.com | 192.168.13.107 | kubelet kube-proxy docker calico Ipvs osd |
| k8sworker04.host.com | 192.168.13.108 | kubelet kube-proxy docker calico Ipvs osd |
| k8sworker05.host.com | 192.168.13.109 | kubelet kube-proxy docker calico Ipvs osd |
| lb01.host.com | 192.168.13.97 | NFS PCS pacemaker corosync Haproxy nginx |
| lb02.host.com | 192.168.13.98 | harbor PCS pacemaker corosync Haproxy nginx |
| lb03.host.com | 192.168.13.99 | DNS PCS pacemaker corosync Haproxy nginx |
1.2 部署拓扑
集群基础组件分布图
集群辅助组件分布图
集群网络访问示意图
1.3 系统环境
基础系统： CentOS Linux release 7.8.2003 (Core Linux 5.7.10-1.el7.elrepo.x86_64）
Kubernetes 版本： v1.19.0
后端存储： Ceph集群+NFS混合
网络链路： k8s业务网络 + Ceph存储数据网络
网络插件： calico组件，采用BGP模式，类似于flannel的host-gw模式；IPVS
负载均衡及反向代理：
Haproxy 负责k8s集群外的TCP四层流量负载转发和反向代理；
Nginx 负责k8s集群外的TCP七层HTTPS和HTTP流量负载转发和反向代理；
Traefik 负责k8s集群内部的流量负载转发和反向代理。
注：流量特别大的情况下，可尝试在Haproxy之前架设LVS负载均衡集群或物理负载均衡设备，将流量调度至其他k8s集群。
二、初始化系统环境
初始化系统环境，所有主机上操作，且基本相同。
2.1 升级系统内核
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum makecache && yum update -y
rpm -Uvh https://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml -y
grubby --default-kernel
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /boot/grub2/grub.cfg
reboot
yum update -y && yum makecache
注：如果内核升级失败，需要到BIOS里面将securt boot 设置为disable
2.2内核优化
cat >> /etc/sysctl.d/k8s.conf << EOF
net.ipv4.tcp_fin_timeout = 10
net.ipv4.ip_forward = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_keepalive_time = 600
net.ipv4.ip_local_port_range = 4000 65000
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_max_tw_buckets = 1048576
net.ipv4.route.gc_timeout = 100
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 16384
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.optmem_max=16777216
net.netfilter.nf_conntrack_max=2097152
net.nf_conntrack_max=2097152
net.netfilter.nf_conntrack_tcp_timeout_fin_wait=30
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
net.netfilter.nf_conntrack_tcp_timeout_close_wait=15
net.netfilter.nf_conntrack_tcp_timeout_established=300
net.ipv4.tcp_max_orphans = 524288
fs.file-max=2097152
fs.nr_open=2097152
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
sysctl --system
cat >> /etc/systemd/system.conf << EOF
DefaultLimitNOFILE=1048576
DefaultLimitNPROC=1048576
EOF
cat /etc/systemd/system.conf
cat >> /etc/security/limits.conf << EOF
* soft nofile 1048576
* hard nofile 1048576
* soft nproc 1048576
* hard nproc 1048576
EOF
cat /etc/security/limits.conf
reboot
最好重启检查
2.3 启用Ipvs内核模块
#创建内核模块载入相关的脚本文件/etc/sysconfig/modules/ipvs.modules，设定自动载入的内核模块。文件内容如下：
每个节点全要开启
# 新建脚本
vim /etc/sysconfig/modules/ipvs.modules
#脚本内容如下
#!/bin/bash
ipvs_modules_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for i in $(ls $ipvs_modules_dir | grep -o "^[^.]*");
do
/sbin/modinfo -F filename $i &> /dev/null
if [ $? -eq 0 ]; then
/sbin/modprobe $i
fi
done
#修改文件权限，并手动为当前系统加载内核模块：
chmod +x /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
#检测加载情况
lsmod | grep ip_vs
2.4 安装常用工具
yum install -y nfs-utils curl yum-utils device-mapper-persistent-data lvm2 net-tools
yum install -y conntrack-tools wget sysstat vim-enhanced bash-completion psmisc traceroute iproute* tree
yum install -y libseccomp libtool-ltdl ipvsadm ipset
2.5 安装NTP时间服务
vim /etc/chrony.conf
修改
server 192.168.20.4 iburst
server time4.aliyun.com iburst
systemctl enable chronyd
systemctl start chronyd
chronyc sources
2.6 关闭Swap分区
swapoff -a
vim /etc/fstab文件中注释掉 swap一行
2.7 关闭系统防火墙和selinux
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
2.8 设置集群内互为免密登陆
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub k8smaster01
...省略
三、部署容器环境
3.1 部署docker
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum -y install docker-ce
mkdir -p /data/docker /etc/docker
修改/usr/lib/systemd/system/docker.service文件，在“ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock”一行之后新增一行如下内容：
ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
cat < /etc/docker/daemon.json
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"registry-mirrors": ["https://kk0rm6dj.mirror.aliyuncs.com","https://registry.docker-cn.com"],
"insecure-registries": ["harbor.iot.com"],
"bip": "10.244.101.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
EOF
注意：https://kk0rm6dj.mirror.aliyuncs.com阿里云的镜像加速地址最好自己申请（免费的）
systemctl enable docker
systemctl daemon-reload
systemctl restart docker
注： big的地址根据主机IP进行相应的变更
3.2 检查docker环境
[[email protected]:/root]# docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.12
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.8.6-2.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 30.84GiB
Name: k8smaster01.host.com
ID: FZAG:XAY6:AYKI:7SUR:62AP:ANDH:JM33:RG5O:Q2XJ:NHGJ:ROAO:6LEM
Docker Root Dir: /data/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
harbor.iot.com
registry.lowaniot.com
127.0.0.0/8
Registry Mirrors:
https://*********.mirror.aliyuncs.com/
https://registry.docker-cn.com/
Live Restore Enabled: true
四、部署Kubernetes
4.1 准备集群根证书
kubernetes各组件使用证书如下：
etcd：使用 ca.pem、etcd-key.pem、etcd.pem；(etcd对外提供服务、节点间通信（etcd peer）使用同一套证书)
kube-apiserver：使用 ca.pem、ca-key.pem、kube-apiserver-key.pem、kube-apiserver.pem；
kubelet：使用 ca.pem ca-key.pem；
kube-proxy：使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem；
kubectl：使用 ca.pem、admin-key.pem、admin.pem；
kube-controller-manager：使用 ca-key.pem、ca.pem、kube-controller-manager.pem、kube-controller-manager-key.pem；
kube-scheduler：使用ca-key.pem、ca.pem、kube-scheduler-key.pem、kube-scheduler.pem；
安装证书生成工具 cfssl，并生成相关证书
关于cfssl工具：
cfssl：证书签发的主要工具
cfssl-json：将cfssl生成的整数（json格式）变为文件承载式证书
cfssl-certinfo：验证证书的信息
4.1.1 安装cfssl
//创建目录用于存放 SSL 证书
[[email protected]:/opt/src]# mkdir /data/ssl -p
// 下载生成证书命令
[[email protected]:/opt/src]# wget -c https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[[email protected]:/opt/src]# wget -c https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[[email protected]:/opt/src]# wget -c https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
// 添加执行权限并查看
[[email protected]:/opt/src]# chmod +x cfssl-certinfo_linux-amd64 cfssljson_linux-amd64 cfssl_linux-amd64
[[email protected]:/opt/src]# ll
total 18808
-rwxr-xr-x. 1 root root 6595195 Mar 30 2016 cfssl-certinfo_linux-amd64
-rwxr-xr-x. 1 root root 2277873 Mar 30 2016 cfssljson_linux-amd64
-rwxr-xr-x. 1 root root 10376657 Mar 30 2016 cfssl_linux-amd64
// 移动到 /usr/local/bin 目录下
[[email protected]:/opt/src]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
[[email protected]:/opt/src]# mv cfssljson_linux-amd64 /usr/local/bin/cfssl-json
[[email protected]:/opt/src]# mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
// 进入证书目录
[[email protected]:/opt/src]# cd /data/ssl/
PS : cfssl-certinfo -cert serverca
4.1.2 创建生成CA证书签名请求（csr）的json配置文件
自签证书会有个根证书ca（需权威机构签发/可自签 ）
CN：浏览器使用该字段验证网站是否合法，一般写的是域名，非常重要
C：国家
ST：州/省
L：地区/城市
O：组织名称/公司名称
OU：组织单位名称，公司部门
[[email protected]:/data/ssl]#
cat > /data/ssl/ca-csr.json <{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
],
"ca": {
"expiry": "175200h"
}
}
EOF
4.1.3 生成CA证书、私钥和csr证书签名请求
会生成运行CA所需求的文件ca-key.pem(私钥)和ca.pem(证书)，还会生成ca.csr(证书签名请求)，用于交叉签名或重新签名。
[[email protected]:/data/ssl]# cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
2020/09/09 17:53:11 [INFO] generating a new CA key and certificate from CSR
2020/09/09 17:53:11 [INFO] generate received request
2020/09/09 17:53:11 [INFO] received CSR
2020/09/09 17:53:11 [INFO] generating key: rsa-2048
2020/09/09 17:53:11 [INFO] encoded CSR
2020/09/09 17:53:11 [INFO] signed certificate with serial number 633952028496783804989414964353657856875757870668
[[email protected]:/data/ssl]# ls
ca.csr ca-csr.json ca-key.pem ca.pem
[[email protected]:/data/ssl]# ll
total 16
-rw-r--r-- 1 root root 1001 Sep 9 17:53 ca.csr
-rw-r--r-- 1 root root 307 Sep 9 17:52 ca-csr.json
-rw------- 1 root root 1679 Sep 9 17:53 ca-key.pem
-rw-r--r-- 1 root root 1359 Sep 9 17:53 ca.pem
4.1.4 创建基于根证书的config配置文件
解析：
ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile;
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE;
server auth：表示client可以用该 CA 对server提供的证书进行验证;
client auth：表示server可以用该CA对client提供的证书进行验证;
expiry: 表示证书过期时间，这里设置10年，可以适当减少
[[email protected]:/data/ssl]#
cat > /data/ssl/ca-config.json <{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"kubernetes": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
[[email protected]:/data/ssl]#
cat > server-csr.json <{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.13.100",
"192.168.13.101",
"192.168.13.102",
"192.168.13.103",
"192.168.13.104",
"192.168.13.105",
"192.168.13.106",
"192.168.13.107",
"192.168.13.108",
"192.168.13.109",
"192.168.13.110",
"10.10.0.1",
"lbvip.host.com",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
[[email protected]:/data/ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssl-json -bare server
2020/09/09 18:03:25 [INFO] generate received request
2020/09/09 18:03:25 [INFO] received CSR
2020/09/09 18:03:25 [INFO] generating key: rsa-2048
2020/09/09 18:03:25 [INFO] encoded CSR
2020/09/09 18:03:25 [INFO] signed certificate with serial number 520236944854379069364132464186673377003193743849
2020/09/09 18:03:25 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[[email protected]:/data/ssl]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server.csr server-csr.json server-key.pem server.pem
[[email protected]:/data/ssl]#
cat > admin-csr.json <{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
[[email protected]:/data/ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssl-json -bare admin
2020/09/09 18:08:34 [INFO] generate received request
2020/09/09 18:08:34 [INFO] received CSR
2020/09/09 18:08:34 [INFO] generating key: rsa-2048
2020/09/09 18:08:34 [INFO] encoded CSR
2020/09/09 18:08:34 [INFO] signed certificate with serial number 662043138212200112498221957779202406541973930898
2020/09/09 18:08:34 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[[email protected]:/data/ssl]# ls
admin.csr admin-csr.json admin-key.pem admin.pem ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server.csr server-csr.json server-key.pem server.pem
[[email protected]:/data/ssl]#
cat > kube-proxy-csr.json <{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
[[email protected]:/data/ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssl-json -bare kube-proxy
2020/09/09 18:11:10 [INFO] generate received request
2020/09/09 18:11:10 [INFO] received CSR
2020/09/09 18:11:10 [INFO] generating key: rsa-2048
2020/09/09 18:11:10 [INFO] encoded CSR
2020/09/09 18:11:10 [INFO] signed certificate with serial number 721336273441573081460131627771975553540104946357
2020/09/09 18:11:10 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[[email protected]:/data/ssl]# ls
admin.csr admin-key.pem ca-config.json ca-csr.json ca.pem kube-proxy-csr.json kube-proxy.pem server-csr.json server.pem
admin-csr.json admin.pem ca.csr ca-key.pem kube-proxy.csr kube-proxy-key.pem server.csr server-key.pem
4.1.6 创建metrics-server证书
4.1.6.1 创建 metrics-server 使用的证书
# 注意: "CN": "system:metrics-server" 一定是这个，因为后面授权时用到这个名称，否则会报禁止匿名访问
[[email protected]:/data/ssl]#
cat > metrics-server-csr.json <{
"CN": "system:metrics-server",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "system"
}
]
}
EOF
4.1.6.2 生成 metrics-server 证书和私钥
[[email protected]:/data/ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes metrics-server-csr.json | cfssl-json -bare metrics-server2020/09/09 18:33:24 [INFO] generate received request
2020/09/09 18:33:24 [INFO] received CSR
2020/09/09 18:33:24 [INFO] generating key: rsa-2048
2020/09/09 18:33:24 [INFO] encoded CSR
2020/09/09 18:33:24 [INFO] signed certificate with serial number 611188048814430694712004119513191220396672578396
2020/09/09 18:33:24 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[[email protected]:/data/ssl]# ls
admin.csr admin.pem ca-csr.json kube-proxy.csr kube-proxy.pem metrics-server-key.pem server-csr.json
admin-csr.json ca-config.json ca-key.pem kube-proxy-csr.json metrics-server.csr metrics-server.pem server-key.pem
admin-key.pem ca.csr ca.pem kube-proxy-key.pem metrics-server-csr.json server.csr server.pem
4.1.6.3 分发证书
[[email protected]:/data/ssl]# cp metrics-server-key.pem metrics-server.pem /opt/kubernetes/ssl/
[[email protected]:/data/ssl]# scp metrics-server-key.pem metrics-server.pem k8smaster02:/opt/kubernetes/ssl/
[[email protected]:/data/ssl]# scp metrics-server-key.pem metrics-server.pem k8smaster03:/opt/kubernetes/ssl/
4.1.7 创建Node节点kubeconfig文件
4.1.7.1 创建TLS Bootstrapping Token、 kubelet kubeconfig、kube-proxy kubeconfig
[[email protected]:/data/ssl]# vim kubeconfig.sh
# 创建 TLS Bootstrapping Token
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
cat > token.csv <{BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
#----------------------
# 创建kubelet bootstrapping kubeconfig
export KUBE_APISERVER="https://lbvip.host.com:7443"
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------
# 创建kube-proxy kubeconfig文件
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=./kube-proxy.pem \
--client-key=./kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
4.1.7.2 生成证书
[[email protected]:/data/ssl]# bash kubeconfig.sh
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".
[[email protected]:/data/ssl]# ls
admin.csr admin.pem ca.csr ca.pem kube-proxy-csr.json kube-proxy.pem metrics-server-key.pem server-csr.json token.csv
admin-csr.json bootstrap.kubeconfig ca-csr.json kubeconfig.sh kube-proxy-key.pem metrics-server.csr metrics-server.pem server-key.pem
admin-key.pem ca-config.json ca-key.pem kube-proxy.csr kube-proxy.kubeconfig metrics-server-csr.json server.csr server.pem
4.1.7.3 分发证书
[[email protected]:/data/ssl]# cp *kubeconfig /opt/kubernetes/cfg
[[email protected]:/data/ssl]# scp *kubeconfig k8smaster02:/opt/kubernetes/cfg
[[email protected]:/data/ssl]# scp *kubeconfig k8smaster03:/opt/kubernetes/cfg
4.2 部署Etcd集群
k8smaster01上操作，把执行文件copy到master02 master03
摘要：etcd部署在master三台节点上做高可用，etcd集群采用raft算法选举Leader， 由于Raft算法在做决策时需要多数节点的投票，所以etcd一般部署集群推荐奇数个节点，推荐的数量为3、5或者7个节点构成一个集群。
二进制包下载地址：(https://github.com/etcd-io/etcd/releases)
4.2.1 基础设置
k8smaster01上操作
# 创建存储etcd数据目录
[[email protected]:/root]# mkdir -p /data/etcd/
# 创建etcd日志目录
[[email protected]:/root]# mkdir -p /data/logs/etcd-server
# 创建kubernetes 集群配置目录
[[email protected]:/root]# mkdir -p /opt/kubernetes/{bin,cfg,ssl}
# 下载二进制etcd包，并把执行文件放到 /opt/kubernetes/bin/ 目录
[[email protected]:/root]# cd /opt/src/
[[email protected]:/opt/src]# wget -c https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz
[[email protected]:/opt/src]# tar -xzf etcd-v3.4.13-linux-amd64.tar.gz -C /data/etcd
[[email protected]:/opt/src]# cd /data/etcd/etcd-v3.4.13-linux-amd64/
[[email protected]:/data/etcd/etcd-v3.4.13-linux-amd64]# cp -a etcd etcdctl /opt/kubernetes/bin/
[[email protected]:/data/etcd/etcd-v3.4.13-linux-amd64]# cd
# 把 /opt/kubernetes/bin 目录加入到 PATH
[[email protected]:/root]# echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile
[[email protected]:/root]# source /etc/profile
k8smaster02和k8smaster03上操作
mkdir -p /data/etcd/ /data/logs/etcd-server
mkdir -p /opt/kubernetes/{bin,cfg,ssl}
echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile
source /etc/profile
在各worker节点上执行
mkdir -p /data/worker
mkdir -p /opt/kubernetes/{bin,cfg,ssl}
echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile
source /etc/profile
4.2.2 分发证书
# 进入 K8S 集群证书目录
[[email protected]:/root]# cd /data/ssl/
[[email protected]:/data/ssl]# ls
admin.csr admin-key.pem ca-config.json ca-csr.json ca.pem kube-proxy-csr.json kube-proxy.pem server-csr.json server.pem
admin-csr.json admin.pem ca.csr ca-key.pem kube-proxy.csr kube-proxy-key.pem server.csr server-key.pem
# 把证书 copy 到 k8s-master1 机器 /opt/kubernetes/ssl/ 目录
[[email protected]:/data/ssl]# cp ca*pem server*pem /opt/kubernetes/ssl/
[[email protected]:/data/ssl]# cd /opt/kubernetes/ssl/ && ls
ca-key.pem ca.pem server-key.pem server.pem
其中 etcd-peer-key.pem 是私钥 权限为600
[[email protected]:/opt/kubernetes/ssl]# cd ..
# 把etcd执行文件与证书 copy 到 k8smaster02 k8smaster03
# 把ca.pem证书copy到worker各节点
[[email protected]:/opt/kubernetes]# scp -r /opt/kubernetes/* k8smaster02:/opt/kubernetes/
[[email protected]:/opt/kubernetes]# scp -r /opt/kubernetes/* k8smaster03:/opt/kubernetes/
4.2.3 编写etcd配置文件脚本 etcd-server-startup.sh
k8smaster01上操作
[[email protected]:/opt/kubernetes]# cd /data/etcd/
[[email protected]:/data/etcd]# vim etcd-server-startup.sh
#!/bin/bash
ETCD_NAME=${1:-"etcd01"}
ETCD_IP=${2:-"127.0.0.1"}
ETCD_CLUSTER=${3:-"etcd01=https://127.0.0.1:2379"}
cat </opt/kubernetes/cfg/etcd.yml
name: ${ETCD_NAME}
data-dir: /var/lib/etcd/default.etcd
listen-peer-urls: https://${ETCD_IP}:2380
listen-client-urls: https://${ETCD_IP}:2379,https://127.0.0.1:2379
advertise-client-urls: https://${ETCD_IP}:2379
initial-advertise-peer-urls: https://${ETCD_IP}:2380
initial-cluster: ${ETCD_CLUSTER}
initial-cluster-token: etcd-cluster
initial-cluster-state: new
client-transport-security:
cert-file: /opt/kubernetes/ssl/server.pem
key-file: /opt/kubernetes/ssl/server-key.pem
client-cert-auth: false
trusted-ca-file: /opt/kubernetes/ssl/ca.pem
auto-tls: false
peer-transport-security:
cert-file: /opt/kubernetes/ssl/server.pem
key-file: /opt/kubernetes/ssl/server-key.pem
client-cert-auth: false
trusted-ca-file: /opt/kubernetes/ssl/ca.pem
auto-tls: false
debug: false
logger: zap
log-outputs: [stderr]
EOF
cat </usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://github.com/etcd-io/etcd
Conflicts=etcd.service
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
LimitNOFILE=65536
Restart=on-failure
RestartSec=5s
TimeoutStartSec=0
ExecStart=/opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd
4.2.4 执行etcd.sh 生成配置脚本
[[email protected]:/data/etcd]# chmod +x etcd-server-startup.sh
[[email protected]:/data/etcd]# ./etcd-server-startup.sh etcd01 192.168.13.101 etcd01=https://192.168.13.101:2380,etcd02=https://192.168.13.102:2380,etcd03=https://192.168.13.103:2380
注： etcd与peer三个主机内部通讯是走2380端口，与etcd外部通讯是走2379端口
注： 此处etcd启动时卡启动状态，但是服务已经起来了.需要等etcd02和etcd03启动后，状态才正常。
4.2.4 检查etcd01运行情况
[[email protected]:/data/etcd]# netstat -ntplu | grep etcd
tcp 0 0 192.168.13.101:2379 0.0.0.0:* LISTEN 25927/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 25927/etcd
tcp 0 0 192.168.13.101:2380 0.0.0.0:* LISTEN 25927/etcd
[[email protected]:/data/etcd]# ps -ef | grep etcd
root 25878 6156 0 15:33 pts/3 00:00:00 /bin/bash ./etcd-server-startup.sh etcd01 192.168.13.101 etcd01=https://192.168.13.101:2380,etcd02=https://192.168.13.102:2380,etcd03=https://192.168.13.103:2380
root 25921 25878 0 15:33 pts/3 00:00:00 systemctl restart etcd
root 25927 1 3 15:33 ? 00:00:16 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
root 25999 3705 0 15:41 pts/0 00:00:00 grep --color=auto etcd
4.2.5 分发etcd-server-startup.sh脚本至master02、master03上
[[email protected]:/data/etcd]# scp /data/etcd/etcd-server-startup.sh k8smaster02:/data/etcd/
[[email protected]:/data/etcd]# scp /data/etcd/etcd-server-startup.sh k8smaster03:/data/etcd/
4.2.6 启动etcd02并检查运行状态
[[email protected]:/root]# cd /data/etcd
[[email protected]:/data/etcd]# ./etcd-server-startup.sh etcd02 192.168.13.102 etcd01=https://192.168.13.101:2380,etcd02=https://192.168.13.102:2380,etcd03=https://192.168.13.103:2380
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
[[email protected]:/data/etcd]# systemctl status etcd -l
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-09-09 12:48:49 CST; 35s ago
Docs: https://github.com/etcd-io/etcd
Main PID: 24630 (etcd)
Tasks: 29
Memory: 17.6M
CGroup: /system.slice/etcd.service
└─24630 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
Sep 09 12:48:54 k8smaster02.host.com etcd[24630]: {"level":"warn","ts":"2020-09-09T12:48:54.257+0800","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"8ebc349f826f93fe","rtt":"0s","error":"dial tcp 192.168.13.103:2380: connect: connection refused"}
Sep 09 12:48:55 k8smaster02.host.com etcd[24630]: {"level":"info","ts":"2020-09-09T12:48:55.144+0800","caller":"rafthttp/stream.go:250","msg":"set message encoder","from":"a1f9be88a19d2f3c","to":"a1f9be88a19d2f3c","stream-type":"stream MsgApp v2"}
Sep 09 12:48:55 k8smaster02.host.com etcd[24630]: {"level":"info","ts":"2020-09-09T12:48:55.144+0800","caller":"rafthttp/peer_status.go:51","msg":"peer became active","peer-id":"8ebc349f826f93fe"}
Sep 09 12:48:55 k8smaster02.host.com etcd[24630]: {"level":"warn","ts":"2020-09-09T12:48:55.144+0800","caller":"rafthttp/stream.go:277","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"a1f9be88a19d2f3c","remote-peer-id":"8ebc349f826f93fe"}
Sep 09 12:48:55 k8smaster02.host.com etcd[24630]: {"level":"info","ts":"2020-09-09T12:48:55.144+0800","caller":"rafthttp/stream.go:250","msg":"set message encoder","from":"a1f9be88a19d2f3c","to":"a1f9be88a19d2f3c","stream-type":"stream Message"}
Sep 09 12:48:55 k8smaster02.host.com etcd[24630]: {"level":"warn","ts":"2020-09-09T12:48:55.144+0800","caller":"rafthttp/stream.go:277","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"a1f9be88a19d2f3c","remote-peer-id":"8ebc349f826f93fe"}
Sep 09 12:48:55 k8smaster02.host.com etcd[24630]: {"level":"info","ts":"2020-09-09T12:48:55.159+0800","caller":"rafthttp/stream.go:425","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"a1f9be88a19d2f3c","remote-peer-id":"8ebc349f826f93fe"}
Sep 09 12:48:55 k8smaster02.host.com etcd[24630]: {"level":"info","ts":"2020-09-09T12:48:55.159+0800","caller":"rafthttp/stream.go:425","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"a1f9be88a19d2f3c","remote-peer-id":"8ebc349f826f93fe"}
Sep 09 12:48:57 k8smaster02.host.com etcd[24630]: {"level":"info","ts":"2020-09-09T12:48:57.598+0800","caller":"membership/cluster.go:546","msg":"updated cluster version","cluster-id":"6204defa6d59332e","local-member-id":"a1f9be88a19d2f3c","from":"3.0","from":"3.4"}
Sep 09 12:48:57 k8smaster02.host.com etcd[24630]: {"level":"info","ts":"2020-09-09T12:48:57.598+0800","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}
[[email protected]:/data/etcd]# netstat -ntplu | grep etcd
tcp 0 0 192.168.13.102:2379 0.0.0.0:* LISTEN 24630/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 24630/etcd
tcp 0 0 192.168.13.102:2380 0.0.0.0:* LISTEN 24630/etcd
[[email protected]:/data/etcd]# ps -ef | grep etcd
root 24630 1 1 12:48 ? 00:00:03 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
root 24675 2602 0 12:52 pts/0 00:00:00 grep --color=auto etcd
4.2.7 启动etcd03并检查运行状态
[[email protected]:/root]# cd /data/etcd/
[[email protected]:/data/etcd]# ./etcd-server-startup.sh etcd03 192.168.13.103 etcd01=https://192.168.13.101:2380,etcd02=https://192.168.13.102:2380,etcd03=https://192.168.13.103:2380
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
[[email protected]:/data/etcd]# systemctl status etcd -l
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-09-09 12:48:55 CST; 7s ago
Docs: https://github.com/etcd-io/etcd
Main PID: 1322 (etcd)
Tasks: 20
Memory: 27.7M
CGroup: /system.slice/etcd.service
└─1322 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
Sep 09 12:48:55 k8smaster03.host.com systemd[1]: Started Etcd Server.
Sep 09 12:48:55 k8smaster03.host.com etcd[1322]: {"level":"info","ts":"2020-09-09T12:48:55.190+0800","caller":"embed/serve.go:139","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"192.168.13.103:2379"}
Sep 09 12:48:55 k8smaster03.host.com etcd[1322]: {"level":"info","ts":"2020-09-09T12:48:55.190+0800","caller":"embed/serve.go:139","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2379"}
Sep 09 12:48:55 k8smaster03.host.com etcd[1322]: {"level":"info","ts":"2020-09-09T12:48:55.192+0800","caller":"rafthttp/stream.go:250","msg":"set message encoder","from":"8ebc349f826f93fe","to":"8ebc349f826f93fe","stream-type":"stream Message"}
Sep 09 12:48:55 k8smaster03.host.com etcd[1322]: {"level":"warn","ts":"2020-09-09T12:48:55.192+0800","caller":"rafthttp/stream.go:277","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"8ebc349f826f93fe","remote-peer-id":"b255a2baf0f6222c"}
Sep 09 12:48:55 k8smaster03.host.com etcd[1322]: {"level":"info","ts":"2020-09-09T12:48:55.192+0800","caller":"rafthttp/stream.go:250","msg":"set message encoder","from":"8ebc349f826f93fe","to":"8ebc349f826f93fe","stream-type":"stream MsgApp v2"}
Sep 09 12:48:55 k8smaster03.host.com etcd[1322]: {"level":"warn","ts":"2020-09-09T12:48:55.192+0800","caller":"rafthttp/stream.go:277","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"8ebc349f826f93fe","remote-peer-id":"b255a2baf0f6222c"}
Sep 09 12:48:55 k8smaster03.host.com etcd[1322]: {"level":"info","ts":"2020-09-09T12:48:55.193+0800","caller":"etcdserver/server.go:716","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"8ebc349f826f93fe","forward-ticks":8,"forward-duration":"800ms","election-ticks":10,"election-timeout":"1s","active-remote-members":2}
Sep 09 12:48:57 k8smaster03.host.com etcd[1322]: {"level":"info","ts":"2020-09-09T12:48:57.597+0800","caller":"membership/cluster.go:546","msg":"updated cluster version","cluster-id":"6204defa6d59332e","local-member-id":"8ebc349f826f93fe","from":"3.0","from":"3.4"}
Sep 09 12:48:57 k8smaster03.host.com etcd[1322]: {"level":"info","ts":"2020-09-09T12:48:57.598+0800","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}
[[email protected]:/data/etcd]# netstat -ntplu | grep etcd
tcp 0 0 192.168.13.103:2379 0.0.0.0:* LISTEN 1322/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1322/etcd
tcp 0 0 192.168.13.103:2380 0.0.0.0:* LISTEN 1322/etcd
[[email protected]:/data/etcd]# ps -ef | grep etcd
root 1322 1 1 12:48 ? 00:00:04 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
root 1353 6455 0 12:53 pts/0 00:00:00 grep --color=auto etcd
4.2.8 检查etcd集群状态
[[email protected]:/root]# ETCDCTL_API=3 etcdctl --write-out=table \
--cacert=/opt/kubernetes/ssl/ca.pem --cert=/opt/kubernetes/ssl/server.pem --key=/opt/kubernetes/ssl/server-key.pem \
--endpoints=https://192.168.13.101:2379,https://192.168.13.102:2379,https://192.168.13.103:2379 endpoint health
+-----------------------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+-----------------------------+--------+-------------+-------+
| https://192.168.13.103:2379 | true | 20.491966ms | |
| https://192.168.13.102:2379 | true | 22.203277ms | |
| https://192.168.13.101:2379 | true | 24.576499ms | |
+-----------------------------+--------+-------------+-------+
[[email protected]:/data/etcd]# ETCDCTL_API=3 etcdctl --write-out=table \
--cacert=/opt/kubernetes/ssl/ca.pem --cert=/opt/kubernetes/ssl/server.pem \
--key=/opt/kubernetes/ssl/server-key.pem \
--endpoints=https://192.168.13.101:2379,https://192.168.13.102:2379,https://192.168.13.103:2379 member list
+------------------+---------+--------+-----------------------------+-----------------------------+------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |ISLEARNER |
+------------------+---------+--------+-----------------------------+-----------------------------+------+
| 968cc9007004cb | started | etcd03 | https://192.168.13.103:2380 | https://192.168.13.103:2379 | false |
| 437d840672a51376 | started | etcd02 | https://192.168.13.102:2380 | https://192.168.13.102:2379 | false |
| fef4dd15ed09253e | started | etcd01 | https://192.168.13.101:2380 | https://192.168.13.101:2379 | false |
+------------------+---------+--------+-----------------------------+-----------------------------+------+
4.3 部署APISERVER
4.3.1 下载kubernetes二进制安装包并解压
[[email protected]:/opt/src]# wget -c https://dl.k8s.io/v1.19.0/kubernetes-server-linux-amd64.tar.gz
[[email protected]:/data]# mkdir -p /data/package
[[email protected]:/opt/src]# tar -xf kubernetes-server-linux-amd64.tar.gz -C /data/package/
注： 解压包中的 kubernetes-src.tar.gz 为go语言编写的源码包，可删除
server目录下的*.tar 和docker_tag为docker镜像文件，可删除
[[email protected]:/opt/src]# cd /data/package/ && cd kubernetes/
[[email protected]:/data/package/kubernetes]# rm -rf kubernetes-src.tar.gz
[[email protected]:/data/package/kubernetes]# cd server/bin
[[email protected]:/data/package/kubernetes/server/bin]# rm -rf *.tar
[[email protected]:/data/package/kubernetes/server/bin]# rm -rf *.docker_tag
[[email protected]:/data/package/kubernetes/server/bin]# ls
apiextensions-apiserver kubeadm kube-aggregator kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-scheduler mounter
[[email protected]:/data/package/kubernetes/server/bin]# cp -a kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kubeadm /opt/kubernetes/bin
4.3.4 配置master组件并运行
登录 k8smaster01 k8smaster02 k8smaster03操作
# 创建 /data/master 目录，用于存放 master 配置执行脚本
mkdir -p /data/master
4.3.4.1 创建 kube-apiserver 配置文件脚本
[[email protected]:/data]# cd /data/master/
[[email protected]:/data/master]# vim apiserver.sh
#!/bin/bash
MASTER_ADDRESS=${1:-"192.168.13.101"}
ETCD_SERVERS=${2:-"http://127.0.0.1:2379"}
cat </opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \\
--v=2 \\
--log-dir=/var/log/kubernetes \\
--etcd-servers=${ETCD_SERVERS} \\
--bind-address=0.0.0.0 \\
--secure-port=6443 \\
--advertise-address=${MASTER_ADDRESS} \\
--allow-privileged=true \\
--service-cluster-ip-range=10.10.0.0/16 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--kubelet-https=true \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=5000-65000 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \\
--etcd-certfile=/opt/kubernetes/ssl/server.pem \\
--etcd-keyfile=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--proxy-client-cert-file=/opt/kubernetes/ssl/metrics-server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/metrics-server-key.pem \\
--runtime-config=api/all=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-truncate-enabled=true \\
--audit-log-path=/var/log/kubernetes/k8s-audit.log"
EOF
cat </usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver
4.3.4.2 创建生成 kube-controller-manager 配置文件脚本
1、Kubernetes控制器管理器是一个守护进程它通过apiserver监视集群的共享状态，并进行更改以尝试将当前状态移向所需状态。
2、kube-controller-manager是有状态的服务，会修改集群的状态信息。如果多个master节点上的相关服务同时生效，则会有同步与一致性问题，所以多master节点中的kube-controller-manager服务只能是主备的关系，kukubernetes采用租赁锁（lease-lock）实现leader的选举，具体到kube-controller-manager，设置启动参数"--leader-elect=true"。
[[email protected]:/data/master]# vim controller-manager.sh
#!/bin/bash
MASTER_ADDRESS=${1:-"127.0.0.1"}
cat </opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\
--v=2 \\
--master=${MASTER_ADDRESS}:8080 \\
--leader-elect=true \\
--bind-address=0.0.0.0 \\
--service-cluster-ip-range=10.10.0.0/16 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s \\
--feature-gates=RotateKubeletServerCertificate=true \\
--feature-gates=RotateKubeletClientCertificate=true \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem"
EOF
cat </usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager
4.3.4.3 创建生成 kube-scheduler 配置文件脚本
摘要：
1、Kube-scheduler作为组件运行在master节点，主要任务是把从kube-apiserver中获取的未被调度的pod通过一系列调度算法找到最适合的node，最终通过向kube-apiserver中写入Binding对象（其中指定了pod名字和调度后的node名字）来完成调度
2、kube-scheduler与kube-controller-manager一样，如果高可用，都是采用leader选举模式。启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。
[[email protected]:/data/master]# vim scheduler.sh
#!/bin/bash
MASTER_ADDRESS=${1:-"127.0.0.1"}
cat </opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true \\
--v=2 \\
--master=${MASTER_ADDRESS}:8080 \\
--address=0.0.0.0 \\
--leader-elect"
EOF
cat </usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler
4.3.5 复制执行文件
执行文件复制至k8smaster02和k8smaster03的/opt/kubernetes/bin/
[[email protected]:/data/package/kubernetes/server/bin]# scp kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kubeadm k8smaster02:/opt/kubernetes/bin
[[email protected]:/data/package/kubernetes/server/bin]# scp kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kubeadm k8smaster03:/opt/kubernetes/bin
4.3.7 添加权限并分发至master02和master03
[[email protected]:/data/master]# chmod +x *.sh
[[email protected]:/data/master]# scp apiserver.sh controller-manager.sh scheduler.sh k8smaster02:/data/master/
[[email protected]:/data/master]# scp apiserver.sh controller-manager.sh scheduler.sh k8smaster03:/data/master/
4.3.8 运行配置文件
4.3.8.1 运行k8smaster01
[[email protected]:/data/master]# ./apiserver.sh 192.168.13.101 https://192.168.13.101:2379,https://192.168.13.102:2379,https://192.168.13.103:2379
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
[[email protected]:/data/master]# ./controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
[[email protected]:/data/master]# ./scheduler.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
查看三个服务的状态
[[email protected]:/data/master]# ps -ef | grep kube
root 8918 1 0 01:11 ? 00:02:36 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=2 --config=/opt/kubernetes/cfg/kube-proxy-config.yml
root 11688 1 2 12:48 ? 00:00:48 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
root 18680 1 2 11:17 ? 00:02:35 /opt/kubernetes/bin/kubelet --logtostderr=true --v=2 --hostname-override=192.168.13.101 --anonymous-auth=false --cgroup-driver=systemd --cluster-domain=cluster.local. --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --client-ca-file=/opt/kubernetes/ssl/ca.pem --tls-cert-file=/opt/kubernetes/ssl/kubelet.pem --tls-private-key-file=/opt/kubernetes/ssl/kubelet-key.pem --image-gc-high-threshold=85 --image-gc-low-threshold=80 --kubeconfig=/opt/kubrnetes/cfg/kubelet.kubeconfig --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
root 20978 1 7 13:21 ? 00:00:27 /opt/kubernetes/bin/kube-apiserver --logtostderr=false --v=2 --apiserver-count=3 --authorization-mode=RBAC --client-ca-file=/opt/kubernetes/ssl/ca.pem --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --etcd-cafile=/opt/kubernetes/ssl/ca.pem --etcd-certfile=/opt/kubernetes/ssl/client.pem --etcd-keyfile=/optkubernetes/ssl/client-key.pem --etcd-servers=http://192.168.13.101:2379,http://192.168.13.102:2379,http://192.168.13.103:2379 --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.10.0.0/16 --service-node-port-range=30000-50000 --target-ram-mb=1024 --kubelet-client-certificate=/opt/kubernetes/ssl/client.pem --kubelet-client-key=/opt/kubernetes/ssl/client-key.pem --log-dir=/var/log/kubernetes --tls-cert-file=/opt/kubernetes/ssl/apiserver.pem --tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem
root 21830 1 2 13:24 ? 00:00:05 /opt/kubernetes/bin/kube-controller-manager --logtostderr=true --v=2 --master=http://127.0.0.1:8080 --leader-elect=true --service-cluster-ip-range=10.10.0.0/16 --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --cluster-cidr=10.244.0.0/16 --root-ca-file=/opt/kubernetes/ssl/ca.pem
root 22669 1 2 13:26 ? 00:00:01 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=2 --master=http://127.0.0.1:8080 --leader-elect
root 22937 18481 0 13:27 pts/0 00:00:00 grep --color=auto kube
[[email protected]:/data/master]# netstat -ntlp | grep kube-
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 20978/kube-apiserve
tcp6 0 0 :::10251 :::* LISTEN 22669/kube-schedule
tcp6 0 0 :::6443 :::* LISTEN 20978/kube-apiserve
tcp6 0 0 :::10252 :::* LISTEN 21830/kube-controll
tcp6 0 0 :::10257 :::* LISTEN 21830/kube-controll
tcp6 0 0 :::10259 :::* LISTEN 22669/kube-schedule
[[email protected]:/data/master]# systemctl status kube-apiserver kube-scheduler kube-controller-manager | grep active
Active: active (running) since Wed 2020-09-09 13:21:27 CST; 7min ago
Active: active (running) since Wed 2020-09-09 13:26:57 CST; 1min 57s ago
Active: active (running) since Wed 2020-09-09 13:24:08 CST; 4min 47s ago
4.3.8.2 运行k8smaster02
[[email protected]:/data/master]# ./apiserver.sh http://192.168.13.101:2379,http://192.168.13.102:2379,http://192.168.13.103:2379
[[email protected]:/data/master]# ./controller-manager.sh 127.0.0.1
[[email protected]:/data/master]# ./scheduler.sh 127.0.0.1
[[email protected]:/data/master]# ps -ef | grep kube
root 24630 1 1 12:48 ? 00:00:43 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
root 25199 1 8 13:23 ? 00:00:31 /opt/kubernetes/bin/kube-apiserver --logtostderr=false --v=2 --apiserver-count=3 --authorization-mode=RBAC --client-ca-file=/opt/kubernetes/ssl/ca.pem --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --etcd-cafile=/opt/kubernetes/ssl/ca.pem --etcd-certfile=/opt/kubernetes/ssl/client.pem --etcd-keyfile=/optkubernetes/ssl/client-key.pem --etcd-servers=http://192.168.13.101:2379,http://192.168.13.102:2379,http://192.168.13.103:2379 --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.10.0.0/16 --service-node-port-range=30000-50000 --target-ram-mb=1024 --kubelet-client-certificate=/opt/kubernetes/ssl/client.pem --kubelet-client-key=/opt/kubernetes/ssl/client-key.pem --log-dir=/var/log/kubernetes --tls-cert-file=/opt/kubernetes/ssl/apiserver.pem --tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem
root 25301 1 0 13:24 ? 00:00:02 /opt/kubernetes/bin/kube-controller-manager --logtostderr=true --v=2 --master=http://127.0.0.1:8080 --leader-elect=true --service-cluster-ip-range=10.10.0.0/16 --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --cluster-cidr=10.244.0.0/16 --root-ca-file=/opt/kubernetes/ssl/ca.pem
root 25392 1 2 13:27 ? 00:00:03 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=2 --master=http://127.0.0.1:8080 --leader-elect
root 25429 2602 0 13:29 pts/0 00:00:00 grep --color=auto kube
[[email protected]:/data/master]# netstat -ntlp | grep kube-
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 25199/kube-apiserve
tcp6 0 0 :::10251 :::* LISTEN 25392/kube-schedule
tcp6 0 0 :::6443 :::* LISTEN 25199/kube-apiserve
tcp6 0 0 :::10252 :::* LISTEN 25301/kube-controll
tcp6 0 0 :::10257 :::* LISTEN 25301/kube-controll
tcp6 0 0 :::10259 :::* LISTEN 25392/kube-schedule
[[email protected]:/data/master]# systemctl status kube-apiserver kube-scheduler kube-controller-manager | grep active
Active: active (running) since Wed 2020-09-09 13:23:19 CST; 6min ago
Active: active (running) since Wed 2020-09-09 13:27:18 CST; 2min 26s ago
Active: active (running) since Wed 2020-09-09 13:24:37 CST; 5min ago
4.3.8.3 运行k8smaster03
[[email protected]:/data/master]# ./apiserver.sh http://192.168.13.101:2379,http://192.168.13.102:2379,http://192.168.13.103:2379
[[email protected]:/data/master]# ./controller-manager.sh 127.0.0.1
[[email protected]:/data/master]# ./scheduler.sh 127.0.0.1
[[email protected]:/data/master]# ps -ef | grep kube
root 1322 1 1 12:48 ? 00:00:38 /opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml
root 1561 1 4 13:25 ? 00:00:12 /opt/kubernetes/bin/kube-apiserver --logtostderr=false --v=2 --apiserver-count=3 --authorization-mode=RBAC --client-ca-file=/opt/kubernetes/ssl/ca.pem --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --etcd-cafile=/opt/kubernetes/ssl/ca.pem --etcd-certfile=/opt/kubernetes/ssl/client.pem --etcd-keyfile=/optkubernetes/ssl/client-key.pem --etcd-servers=http://192.168.13.101:2379,http://192.168.13.102:2379,http://192.168.13.103:2379 --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.10.0.0/16 --service-node-port-range=30000-50000 --target-ram-mb=1024 --kubelet-client-certificate=/opt/kubernetes/ssl/client.pem --kubelet-client-key=/opt/kubernetes/ssl/client-key.pem --log-dir=/var/log/kubernetes --tls-cert-file=/opt/kubernetes/ssl/apiserver.pem --tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem
root 1633 1 0 13:26 ? 00:00:00 /opt/kubernetes/bin/kube-controller-manager --logtostderr=true --v=2 --master=http://127.0.0.1:8080 --leader-elect=true --service-cluster-ip-range=10.10.0.0/16 --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --cluster-cidr=10.244.0.0/16 --root-ca-file=/opt/kubernetes/ssl/ca.pem
root 1708 1 0 13:27 ? 00:00:00 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=2 --master=http://127.0.0.1:8080 --leader-elect
root 1739 6455 0 13:30 pts/0 00:00:00 grep --color=auto kube
[[email protected]:/data/master]# netstat -ntlp | grep kube-
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 1561/kube-apiserver
tcp6 0 0 :::10251 :::* LISTEN 1708/kube-scheduler
tcp6 0 0 :::6443 :::* LISTEN 1561/kube-apiserver
tcp6 0 0 :::10252 :::* LISTEN 1633/kube-controlle
tcp6 0 0 :::10257 :::* LISTEN 1633/kube-controlle
tcp6 0 0 :::10259 :::* LISTEN 1708/kube-scheduler
[[email protected]:/data/master]# systemctl status kube-apiserver kube-scheduler kube-controller-manager | grep active
Active: active (running) since Wed 2020-09-09 13:25:49 CST; 4min 30s ago
Active: active (running) since Wed 2020-09-09 13:27:33 CST; 2min 47s ago
Active: active (running) since Wed 2020-09-09 13:26:17 CST; 4min 2s ago
4.3.9 验证集群健康状态
[[email protected]:/data/master]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
4.4 部署kubelet
4.4.1 配置kubelet证书自动续期和创建Node授权用户
4.4.1.1 创建 Node节点 授权用户 kubelet-bootstrap
[[email protected]:/data/ssl]# [[email protected]:/data/master]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: null
name: kubelet-bootstrap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubelet-bootstrap
4.4.1.2 创建自动批准相关 CSR 请求的 ClusterRole
# 创建证书旋转配置存放目录
[[email protected]:/data/master]# vim tls-instructs-csr.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
# 部署
[[email protected]:/data/master]# kubectl apply -f tls-instructs-csr.yaml
clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver created
4.4.1.3 自动批准 kubelet-bootstrap 用户 TLS bootstrapping 首次申请证书的 CSR 请求
[[email protected]:/data/master]# kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/node-client-auto-approve-csr created
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: null
name: node-client-auto-approve-csr
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubelet-bootstrap
~
4.4.1.4 自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求
[[email protected]:/data/master]# kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
clusterrolebinding.rbac.authorization.k8s.io/node-client-auto-renew-crt created
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: null
name: node-client-auto-renew-crt
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
4.4.1.5 自动批准 system:nodes 组用户更新 kubelet 10250 api 端口证书的 CSR 请求
[[email protected]:/data/master]# kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes
clusterrolebinding.rbac.authorization.k8s.io/node-server-auto-renew-crt created
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: null
name: node-server-auto-renew-crt
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
4.4.2 各节点准备基础镜像
docker pull registry.cn-hangzhou.aliyuncs.com/mapplezf/google_containers-pause-amd64:3.2
或者
docker pull harbor.iot.com/kubernetes/google_containers-pause-amd64:3.2
或者
docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
或者
docker pull registry.aliyuncs.com/google_containers/pause
4.4.3 创建kubelet配置脚本
[[email protected]:/data/master]# vim kubelet.sh
#!/bin/bash
DNS_SERVER_IP=${1:-"10.10.0.2"}
HOSTNAME=${2:-"`hostname`"}
CLUETERDOMAIN=${3:-"cluster.local"}
cat </opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS=" --logtostderr=true \\
--v=2 \\
--hostname-override=${HOSTNAME} \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--network-plugin=cni \\
--cni-conf-dir=/etc/cni/net.d \\
--cni-bin-dir=/opt/cni/bin \\
--cgroup-driver=systemd \\
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/mapplezf/google_containers-pause-amd64:3.2"
EOF
# harbor.iot.com/kubernetes/google_containers-pause-amd64:3.2
cat </opt/kubernetes/cfg/kubelet-config.yml
kind: KubeletConfiguration # 使用对象
apiVersion: kubelet.config.k8s.io/v1beta1 # api版本
address: 0.0.0.0 # 监听地址
port: 10250 # 当前kubelet的端口
readOnlyPort: 10255 # kubelet暴露的端口
cgroupDriver: cgroupfs # 驱动，要于docker info显示的驱动一致
clusterDNS:
- ${DNS_SERVER_IP}
clusterDomain: ${CLUETERDOMAIN} # 集群域
failSwapOn: false # 关闭swap
# 身份验证
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/kubernetes/ssl/ca.pem
# 授权
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
# Node 资源保留
evictionHard:
imagefs.available: 15%
memory.available: 1G
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
# 镜像删除策略
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
# 旋转证书
rotateCertificates: true # 旋转kubelet client 证书
featureGates:
RotateKubeletServerCertificate: true
RotateKubeletClientCertificate: true
maxOpenFiles: 1000000
maxPods: 110
EOF
cat </usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet
4.4.4 master节点启用kubelet
操作k8smaster01
[[email protected]:/data/master]# chmod +x kubelet.sh
[[email protected]:/data/master]# ./kubelet.sh 10.10.0.2 k8smaster01.host.com cluster.local.
[[email protected]:/data/master]# systemctl status kubelet
[[email protected]:/data/master]# scp kubelet.sh k8smaster02:/data/master/
[[email protected]:/data/master]# scp kubelet.sh k8smaster03:/data/master/
操作k8smaster02
[[email protected]:/data/master]# ./kubelet.sh 10.10.0.2 k8smaster02.host.com cluster.local.
[[email protected]:/data/master]# systemctl status kubelet
操作k8smaster03
[[email protected]:/data/master]# ./kubelet.sh 10.10.0.2 k8smaster03.host.com cluster.local.
[[email protected]:/data/master]# systemctl status kubelet
4.4.5 worker节点启用kubelet
清点准备部署文件
[[email protected]:/data/master]# scp kubelet.sh k8sworker01:/data/worker/
[[email protected]:/data/master]# scp kubelet.sh k8sworker02:/data/worker/
[[email protected]:/data/master]# scp kubelet.sh k8sworker03:/data/worker/
[[email protected]:/data/master]# scp kubelet.sh k8sworker04:/data/worker/
[[email protected]:/opt/kubernetes/cfg]# scp kubelet.kubeconfig bootstrap.kubeconfig k8sworker01:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/cfg]# scp kubelet.kubeconfig bootstrap.kubeconfig k8sworker02:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/cfg]# scp kubelet.kubeconfig bootstrap.kubeconfig k8sworker03:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/cfg]# scp kubelet.kubeconfig bootstrap.kubeconfig k8sworker04:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/ssl]# scp ca.pem k8sworker01:/opt/kubernetes/ssl/
[[email protected]:/opt/kubernetes/ssl]# scp ca.pem k8sworker01:/opt/kubernetes/ssl/
[[email protected]:/opt/kubernetes/ssl]# scp ca.pem k8sworker01:/opt/kubernetes/ssl/
[[email protected]:/opt/kubernetes/ssl]# scp ca.pem k8sworker01:/opt/kubernetes/ssl/
注：其实只需要kubelet kube-proxy就可以，（分发kubectl kubeadm是为了自己后期方便，一般不推荐）
[[email protected]:/opt/kubernetes/bin]# scp kubelet kube-proxy kubectl kubeadm k8sworker01:/opt/kubernetes/bin/
[[email protected]:/opt/kubernetes/bin]# scp kubelet kube-proxy kubectl kubeadm k8sworker02:/opt/kubernetes/bin/
[[email protected]:/opt/kubernetes/bin]# scp kubelet kube-proxy kubectl kubeadm k8sworker03:/opt/kubernetes/bin/
[[email protected]:/opt/kubernetes/bin]# scp kubelet kube-proxy kubectl kubeadm k8sworker04:/opt/kubernetes/bin/
操作k8sworker01
[[email protected]:/data/worker]# ./kubelet.sh 10.10.0.2 k8sworker01.host.com cluster.local.
[[email protected]:/data/worker]# systemctl status kubelet
操作k8sworker02
[[email protected]:/data/worker]# ./kubelet.sh 10.10.0.2 k8sworker02.host.com cluster.local.
[[email protected]:/data/worker]# systemctl status kubelet
操作k8sworker03
[[email protected]:/data/worker]# ./kubelet.sh 10.10.0.2 k8sworker03.host.com cluster.local.
[[email protected]:/data/worker]# systemctl status kubelet
操作k8sworker04
[[email protected]:/data/worker]# ./kubelet.sh 10.10.0.2 k8sworker04.host.com cluster.local.
[[email protected]:/data/worker]# systemctl status kubelet
4.5 部署kube-proxy
4.5.6 创建 kube-proxy 配置脚本
摘要：
kube-proxy的作用主要是负责service的实现，具体来说，就是实现了内部从pod到service和外部的从node port向service的访问
[[email protected]:/data/master]# vim proxy.sh
#!/bin/bash
HOSTNAME=${1:-"`hostname`"}
cat </opt/kubernetes/cfg/kube-proxy.conf
KUBE_PROXY_OPTS=" --logtostderr=true \\
--v=2 \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF
cat </opt/kubernetes/cfg/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
address: 0.0.0.0 # 监听地址
metricsBindAddress: 0.0.0.0:10249 # 监控指标地址,监控获取相关信息 就从这里获取
clientConnection:
kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig # 读取配置文件
hostnameOverride: ${HOSTNAME} # 注册到k8s的节点名称唯一
clusterCIDR: 10.10.0.0/16 # service IP范围
#mode: iptables # 使用iptables模式
# 使用 ipvs 模式
mode: ipvs # ipvs 模式
ipvs:
scheduler: " nq"
iptables:
masqueradeAll: true
EOF
cat </usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy
4.5.7 master节点启动kube-proxy
操作k8smaster01
[[email protected]:/root]# chmod +x /data/master/proxy.sh
[[email protected]:/data/master]# scp /data/package/kubernetes/server/bin/kube-proxy k8smaster02:/opt/kubernetes/bin
[[email protected]:/data/master]# scp /data/package/kubernetes/server/bin/kube-proxy k8smaster03:/opt/kubernetes/bin
[[email protected]:/opt/kubernetes/cfg]# scp /opt/kubernetes/cfg/kube-proxy.kubeconfig k8smaster02:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/cfg]# scp /opt/kubernetes/cfg/kube-proxy.kubeconfig k8smaster03:/opt/kubernetes/cfg/
[[email protected]:/root]# cp /data/package/kubernetes/server/bin/kube-proxy /opt/kubernetes/bin/
[[email protected]:/root]# cd /data/master
[[email protected]:/data/master]# scp proxy.sh k8smaster02:/data/master/
[[email protected]:/data/master]# scp proxy.sh k8smaster03:/data/master/
[[email protected]:/data/master]# ./proxy.sh k8smaster01.host.com
[[email protected]:/data/master]# systemctl status kube-proxy.service
操作k8smaster02
[[email protected]:/data/master]# ./proxy.sh k8smaster02.host.com
[[email protected]:/data/master]# systemctl status kube-proxy.service
操作k8smaster03
[[email protected]:/data/master]# ./proxy.sh k8smaster03.host.com
[[email protected]:/data/master]# systemctl status kube-proxy.service
查看服务启动情况：
systemctl status kubelet kube-proxy | grep active
[[email protected]:/data/master]# netstat -ntpl | egrep "kubelet|kube-proxy"
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 2490/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 30250/kube-proxy
tcp 0 0 127.0.0.1:9943 0.0.0.0:* LISTEN 2490/kubelet
tcp6 0 0 :::10250 :::* LISTEN 2490/kubelet
tcp6 0 0 :::10255 :::* LISTEN 2490/kubelet
tcp6 0 0 :::10256 :::* LISTEN 30250/kube-proxy
[[email protected]:/data/master]# netstat -ntpl | egrep "kubelet|kube-proxy"
tcp 0 0 127.0.0.1:15495 0.0.0.0:* LISTEN 13678/kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 13678/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 12346/kube-proxy
tcp6 0 0 :::10250 :::* LISTEN 13678/kubelet
tcp6 0 0 :::10255 :::* LISTEN 13678/kubelet
tcp6 0 0 :::10256 :::* LISTEN 12346/kube-proxy
[[email protected]:/data/master]# netstat -ntpl | egrep "kubelet|kube-proxy"
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 6342/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 6220/kube-proxy
tcp 0 0 127.0.0.1:26793 0.0.0.0:* LISTEN 6342/kubelet
tcp6 0 0 :::10250 :::* LISTEN 6342/kubelet
tcp6 0 0 :::10255 :::* LISTEN 6342/kubelet
tcp6 0 0 :::10256 :::* LISTEN 6220/kube-proxy
4.5.8 worker节点启动kube-proxy
清点准备部署文件
[[email protected]:/opt/kubernetes/cfg]# scp kube-proxy.kubeconfig k8sworker01:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/cfg]# scp kube-proxy.kubeconfig k8sworker02:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/cfg]# scp kube-proxy.kubeconfig k8sworker03:/opt/kubernetes/cfg/
[[email protected]:/opt/kubernetes/cfg]# scp kube-proxy.kubeconfig k8sworker04:/opt/kubernetes/cfg/
[[email protected]:/data/master]# scp proxy.sh k8sworker01:/data/worker
[[email protected]:/data/master]# scp proxy.sh k8sworker02:/data/worker
[[email protected]:/data/master]# scp proxy.sh k8sworker03:/data/worker
[[email protected]:/data/master]# scp proxy.sh k8sworker04:/data/worker
操作k8sworker01
[[email protected]:/data/worker]# ./proxy.sh k8sworker01.host.com
[[email protected]:/data/worker]# systemctl status kube-proxy.service
操作k8sworker02
[[email protected]:/data/worker]# ./proxy.sh k8sworker02.host.com
[[email protected]:/data/worker]# systemctl status kube-proxy.service
操作k8sworker03
[[email protected]:/data/worker]# ./proxy.sh k8sworker03.host.com
[[email protected]:/data/worker]# systemctl status kube-proxy.service
操作k8sworker01
[[email protected]:/data/worker]# ./proxy.sh k8sworker04.host.com
[[email protected]:/data/worker]# systemctl status kube-proxy.service
4.6 检测状态
[[email protected]:/data/yaml]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster01.host.com Ready 165m v1.19.0
k8smaster02.host.com Ready 164m v1.19.0
k8smaster03.host.com Ready 164m v1.19.0
k8sworker01.host.com Ready 15m v1.19.0
k8sworker02.host.com Ready 14m v1.19.0
k8sworker03.host.com Ready 12m v1.19.0
k8sworker04.host.com Ready 12m v1.19.0
[[email protected]:/data/master]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-f5c7c 3m33s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
csr-hhn6n 9m14s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
csr-hqnfv 3m23s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
4.7 其他补充设置
4.7.1 master节点添加taint
[[email protected]:/root]# kubectl describe node | grep -i taint
Taints:
Taints:
Taints:
Taints:
Taints:
Taints:
Taints:
[[email protected]:/root]# kubectl describe node k8smaster01 | grep -i taint
Taints:
[[email protected]:/root]# kubectl taint nodes k8smaster01.host.com node-role.kubernetes.io/master:NoSchedule
node/k8smaster01.host.com tainted
[[email protected]:/root]# kubectl taint nodes k8smaster02.host.com node-role.kubernetes.io/master:NoSchedule
node/k8smaster02.host.com tainted
[[email protected]:/root]# kubectl taint nodes k8smaster03.host.com node-role.kubernetes.io/master:NoSchedule
node/k8smaster03.host.com tainted
[[email protected]:/root]# kubectl describe node | grep -i taint
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
Taints:
Taints:
Taints:
Taints:
4.7.2命令补全
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> /etc/rc.local
五、部署网络组件
5.1 部署calico网络 BGP模式
calico官方网址：https://docs.projectcalico.org/about/about-calico
5.1.1 修改部分配置
全部代码请查看附件或官方下载
大概位于文件的3500行
其中标记####下面需要修改 处需要重点关注
---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
namespace: kube-system
labels:
k8s-app: calico-node
spec:
selector:
matchLabels:
k8s-app: calico-node
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
k8s-app: calico-node
spec:
nodeSelector:
kubernetes.io/os: linux
hostNetwork: true
tolerations:
# Make sure calico-node gets scheduled on all nodes.
- effect: NoSchedule
operator: Exists
# Mark the pod as a critical add-on for rescheduling.
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
operator: Exists
serviceAccountName: calico-node
# Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
# deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
terminationGracePeriodSeconds: 0
priorityClassName: system-node-critical
initContainers:
# This container performs upgrade from host-local IPAM to calico-ipam.
# It can be deleted if this is a fresh installation, or if you have already
# upgraded to use calico-ipam.
- name: upgrade-ipam
image: calico/cni:v3.16.1
command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
- name: KUBERNETES_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CALICO_NETWORKING_BACKEND
valueFrom:
configMapKeyRef:
name: calico-config
key: calico_backend
volumeMounts:
- mountPath: /var/lib/cni/networks
name: host-local-net-dir
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
securityContext:
privileged: true
# This container installs the CNI binaries
# and CNI network config file on each node.
- name: install-cni
image: calico/cni:v3.16.1
command: ["/opt/cni/bin/install"]
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
# Name of the CNI config file to create.
- name: CNI_CONF_NAME
value: "10-calico.conflist"
# The CNI network config to install on each node.
- name: CNI_NETWORK_CONFIG
valueFrom:
configMapKeyRef:
name: calico-config
key: cni_network_config
# Set the hostname based on the k8s node name.
- name: KUBERNETES_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# CNI MTU Config variable
- name: CNI_MTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Prevents the container from sleeping forever.
- name: SLEEP
value: "false"
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
- mountPath: /host/etc/cni/net.d
name: cni-net-dir
securityContext:
privileged: true
# Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
# to communicate with Felix over the Policy Sync API.
- name: flexvol-driver
image: calico/pod2daemon-flexvol:v3.16.1
volumeMounts:
- name: flexvol-driver-host
mountPath: /host/driver
securityContext:
privileged: true
containers:
# Runs calico-node container on each Kubernetes node. This
# container programs network policy and routes on each
# host.
- name: calico-node
image: calico/node:v3.16.1
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
# Use Kubernetes API as the backing datastore.
- name: DATASTORE_TYPE
value: "kubernetes"
# DaemonSet中添加该环境变量 ####如果是双网络环境下面需要修改
- name: IP_AUTODETECTION_METHOD
value: skip-interface=enp0s20f0u1|em2
# Wait for the datastore.
- name: WAIT_FOR_DATASTORE
value: "true"
# Set based on the k8s node name.
- name: NODENAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# Choose the backend to use.
- name: CALICO_NETWORKING_BACKEND
valueFrom:
configMapKeyRef:
name: calico-config
key: calico_backend
# Cluster type to identify the deployment type ####下面需要修改
- name: CLUSTER_TYPE
value: "k8s,bgp"
# Auto-detect the BGP IP address. ####下面需要修改
- name: IP
value: "autodetect"
# Enable IPIP ####下面需要修改
- name: CALICO_IPV4POOL_IPIP
value: "Never"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
value: "Never"
# Set MTU for tunnel device used if ipip is enabled
- name: FELIX_IPINIPMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Set MTU for the VXLAN tunnel device.
- name: FELIX_VXLANMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Set MTU for the Wireguard tunnel device.
- name: FELIX_WIREGUARDMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "10.244.101.0/24,10.244.102.0/24,10.244.103.0/24,10.244.105.0/24,10.244.106.0/24,10.244.107.0/24,10.244.108.0/24"
# Disable file logging so `kubectl logs` works.
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"
# Set Felix endpoint to host default action to ACCEPT.
- name: FELIX_DEFAULTENDPOINTTOHOSTACTION
value: "ACCEPT"
# Disable IPv6 on Kubernetes.
- name: FELIX_IPV6SUPPORT
value: "false"
# Set Felix logging to "info"
- name: FELIX_LOGSEVERITYSCREEN
value: "info"
- name: FELIX_HEALTHENABLED
value: "true"
securityContext:
privileged: true
resources:
requests:
cpu: 250m
livenessProbe:
exec:
command:
- /bin/calico-node
- -felix-live
- -bird-live
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
readinessProbe:
exec:
command:
- /bin/calico-node
- -felix-ready
- -bird-ready
periodSeconds: 10
volumeMounts:
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /run/xtables.lock
name: xtables-lock
readOnly: false
- mountPath: /var/run/calico
name: var-run-calico
readOnly: false
- mountPath: /var/lib/calico
name: var-lib-calico
readOnly: false
- name: policysync
mountPath: /var/run/nodeagent
# For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
# parent directory.
- name: sysfs
mountPath: /sys/fs/
# Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
# If the host is known to mount that filesystem already then Bidirectional can be omitted.
mountPropagation: Bidirectional
volumes:
# Used by calico-node.
- name: lib-modules
hostPath:
path: /lib/modules
- name: var-run-calico
hostPath:
path: /var/run/calico
- name: var-lib-calico
hostPath:
path: /var/lib/calico
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
- name: sysfs
hostPath:
path: /sys/fs/
type: DirectoryOrCreate
# Used to install CNI.
- name: cni-bin-dir
hostPath:
path: /opt/cni/bin
- name: cni-net-dir
hostPath:
path: /etc/cni/net.d
# Mount in the directory for host-local IPAM allocations. This is
# used when upgrading from host-local to calico-ipam, and can be removed
# if not using the upgrade-ipam init container.
- name: host-local-net-dir
hostPath:
path: /var/lib/cni/networks
# Used to create per-pod Unix Domain Sockets
- name: policysync
hostPath:
type: DirectoryOrCreate
path: /var/run/nodeagent
# Used to install Flex Volume Driver
- name: flexvol-driver-host
hostPath:
type: DirectoryOrCreate
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
---
5.1.2 应用calico文件
[[email protected]:/opt/calicoctl]# kubectl apply -f calico.yaml
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers configured
serviceaccount/calico-kube-controllers unchanged
calico.yaml文件由于太长，代码详见附件
5.2 部署coredns
官网参考地址：https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/coredns/coredns.yaml.base
5.2.1 编辑coredns配置文件
[[email protected]:/data/yaml/coredns]# vim coredns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: Reconcile
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: EnsureExists
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local. in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
# replicas: not specified here:
# 1. In order to make Addon Manager do not reconcile this replicas parameter.
# 2. Default is 1.
# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
priorityClassName: system-cluster-critical
serviceAccountName: coredns
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: k8s-app
operator: In
values: ["kube-dns"]
topologyKey: kubernetes.io/hostname
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
kubernetes.io/os: linux
containers:
- name: coredns
image: registry.cn-hangzhou.aliyuncs.com/mapplezf/coredns:1.7.0
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 1024Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.10.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP
5.2.2 应用部署coredns
[[email protected]:/data/yaml/coredns]# kubectl apply -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
六、部署traefik
6.1 编辑traefik资源配置文件
6.1.1 编辑svc
[[email protected]:/data/yaml/traefik/ingress]# vim svc.yaml
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: kube-system
spec:
type: LoadBalancer
selector:
app: traefik
ports:
- protocol: TCP
port: 80
name: web
targetPort: 80
- protocol: TCP
port: 8080
name: admin
targetPort: 8080
6.1.2 编辑rbac
[[email protected]:/data/yaml/traefik/ingress]# vim rbac-traefik-ingress.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system
6.1.3 编辑ingress-traefik
[[email protected]:/data/yaml/traefik/ingress]# vim ingress-traefik.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: traefik-ingress
namespace: kube-system
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: traefik.lowan.com
http:
paths:
- path: /
backend:
serviceName: traefik #需要和service名字对上
servicePort: 8080
6.1.4 编辑deployment
[[email protected]:/data/yaml/traefik/ingress]# vim deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
namespace: kube-system
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik
namespace: kube-system
labels:
app: traefik
annotations:
prometheus_io_scheme: "traefik"
prometheus_io_path: "/metrics"
prometheus_io_port: "8080"
spec:
replicas: 7
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: harbor.iot.com/kubernetes/traefik:v2.3
args:
- --log.level=DEBUG
- --api
- --api.insecure
- --entrypoints.web.address=:80
- --providers.kubernetesingress
- --metrics.prometheus
- --accesslog
- --accesslog.filepath=/var/log/traefik_access.log
ports:
- name: web
containerPort: 80
hostPort: 81
- name: admin
containerPort: 8080
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
6.2 应用配置文件
kubectl apply -f svc.yaml
kubectl apply -f rbac-traefik-ingress.yaml
kubectl apply -f ingress-traefik.yaml
kubectl apply -f deployment.yaml
6.3 检查状态
[[email protected]:/data/yaml/traefik/ingress]# kubectl get pods -n kube-system -o wide | grep -i traefik
traefik-6787756cf4-7csbq 1/1 Running 0 12h 172.16.234.142 k8smaster02.host.com
traefik-6787756cf4-f6zvw 1/1 Running 0 12h 172.16.222.139 k8sworker04.host.com
traefik-6787756cf4-glkn8 1/1 Running 0 12h 172.16.15.203 k8sworker01.host.com
traefik-6787756cf4-glq9s 1/1 Running 0 12h 172.16.111.204 k8sworker02.host.com
traefik-6787756cf4-gsxqd 1/1 Running 0 12h 172.16.236.11 k8sworker03.host.com
traefik-6787756cf4-hdc79 1/1 Running 0 12h 172.16.53.78 k8smaster03.host.com
traefik-6787756cf4-scqzg 1/1 Running 0 12h 172.16.170.77 k8smaster01.host.com
6.4 添加域名解析及反向代理
[[email protected]:/root]# vim /var/named/iot.com.zone
$ORIGIN lowan.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.lowan.com. dnsadmin.lowan.com. (
2020101004 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ) ; minimum (1 day)
NS dns.lowan.com.
$TTL 60 ; 1 minutes
dns A 192.168.13.99
traefik A 192.168.13.100
:x 保存退出
root@lb03.host.com:/root]# systemctl restart named
[[email protected]:/root]# dig -t A traefik.lowan.com @192.168.13.99 +short
192.168.13.100
6.6 测试traefikdashboard
访问：http://traefik.lowan.com/
七、部署Kubernetes Dashboard
7.1 下载Dashboard的yaml文件
[[email protected]:/data/yaml/dashboard]# wget -c https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml
7.2 创建Dashboard的命令空间和存储
[[email protected]:/data/yaml/dashboard]# vim namespace-dashboard.yaml
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
[[email protected]:/data/yaml/dashboard]# vim pv-nfs-dashboard.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: k8sdashboard
labels:
volume: k8sdashboard
spec:
capacity:
storage: 100Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
storageClassName: slow
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /nfs/k8sdashboard
server: lb01.host.com
[[email protected]:/data/yaml/dashboard]# vim pvc-nfs-dashboard.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: k8sdashboard
namespace: kubernetes-dashboard
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 100Gi
7.3 应用recommended
[[email protected]:/data/yaml/dashboard]# vim recommended.yaml
修改： 添加NFS
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
nfs:
path: /nfs/k8sdashboard
server: lb01.host.com
[[email protected]:/data/yaml/dashboard]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
[[email protected]:/data/yaml/dashboard]# kubectl get pods -n kubernetes-dashboard -o wide
NAME READY STATUS RESTARTS AGE IP NODE
dashboard-metrics-scraper-66f98b8bf9-fg5z4 1/1 Running 0 32s 172.16.170.82 k8smaster01.host.com
kubernetes-dashboard-6bc5bd9b79-v7xjz 1/1 Running 0 32s 172.16.170.83 k8smaster01.host.com
7.4 创建认证
7.4.1 创建登陆认证和获取token
[[email protected]:/data/yaml/dashboard]# vim dashboard-adminuser.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
~
[[email protected]:/data/yaml/dashboard]# kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
[[email protected]:/data/yaml/dashboard]# kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
Name: admin-user-token-92zvt
Namespace: kubernetes-dashboard
Labels:
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: 91a5c456-2ac0-47ac-b664-95b3f3d000b9
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1359 bytes
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkQ4QldRbFA2emtGNzktSHVUdjRQWU11dkppRTRrZ2tsV2dwelUzeXA3OVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTkyenZ0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI5MWE1YzQ1Ni0yYWMwLTQ3YWMtYjY2NC05NWIzZjNkMDAwYjkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.HhHMq34IWWjE8mv-hu573IUfMdBCOjj2Ys9kqBzfA-phJm5HzmXAtppvnpjuTXrkh80MPwcuYaV2kMnQ74jLwiXpX1jNQLz4Dyq6Ymzkjjrxx2WIJwlCX-FBk2iyy9bjhLyfqk9ytofgKvLnqcmvT_hDo-evsSD4RHN8S_kt_KT4CT1PxRZ3BwHKviC5KS0voFNZtg3F5ulDEHW2CT5U2HoPYdzJt5xcbBh-o9Qpm44j7OjF6y3eEkNmG6DGxva-lGN2ltoJdIBRhlAlrxB0jA0FnAqWuPgBqjHwg4edKNzmuMsuGuSr-PHeYbN-8RNjc-CR9bVeSa4JLUo0LhL59A
7.4.2 创建自签证书
此处使用openssl生成证书 ，当然也可以参照文章使用cfssl工具生成证书
[[email protected]:/data/ssl]# cp ca.pem ca-key.pem /data/yaml/dashboard/
[[email protected]:/data/yaml/dashboard]# (umask 077; openssl genrsa -out kubernetes.lowan.com.key 2048)
Generating RSA private key, 2048 bit long modulus
...........................................................+++
................................................................+++
e is 65537 (0x10001)
[[email protected]:/data/yaml/dashboard]#openssl req -new -key kubernetes.lowan.com.key -out kubernetes.lowan.com.csr -subj "/CN=kubernetes.lowan.com/C=CN/ST=Beijing/L=Beijing/O=k8s/OU=system"
[[email protected]:/data/yaml/dashboard]# openssl x509 -req -in kubernetes.lowan.com.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out kubernetes.lowan.com.pem -days 3650
Signature ok
subject=/CN=kubernetes.lowans.com/C=CN/ST=Beijing/L=Beijing/O=iot/OU=system
Getting CA Private Key
7.4.3 创建secret对象来引用证书文件
[[email protected]:/data/yaml/dashboard]# kubectl create secret tls kubedashboard-tls --cert=/data/yaml/dashboard/kubernetes.lowan.com.crt --key=kubernete.lowan.com.key -n kubernetes-dashboard
7.4.4 创建dashboard的ingressrrouter
[[email protected]:/data/yaml/dashboard]# vim ingressroute-kube.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: kubernetes-dashboard-route
namespace: kubernetes-dashboard
annotations:
# traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.entrypoints: web
kubernetes.io/ingress.class: "traefik"
spec:
entryPoints:
- web
# tls:
# secretName: kubedashboard-tls
routes:
- match: Host(`kubernetes.lowan.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: kubernetes-dashboard
port: 443
~
7.4.5 配置nginx反代代理
7.4.5.1 分发pem和key文件至nginx
[[email protected]:/data/yaml/dashboard]# scp kubernetes.lowan.com.key kubernetes.lowan.com.pem lb01:/data/nginx/conf/certs/
[[email protected]:/data/yaml/dashboard]# scp kubernetes.lowan.com.key kubernetes.lowan.com.pem lb02:/data/nginx/conf/certs/
[[email protected]:/data/yaml/dashboard]# scp kubernetes.lowan.com.key kubernetes.lowan.com.pem lb03:/data/nginx/conf/certs/
或者
for i in {01..03}
do
scp kubernetes.lowan.com.key kubernetes.lowan.com.pem lbi:/data/nginx/certs/
scp rancher.lowan.com.key rancher.lowan.com.pem lbi:/data/nginx/certs/
done
7.4.5.2 修改nginx.conf
[[email protected]:/data/nginx/conf]# vim nginx.conf
#user nobody;
worker_processes 4;
worker_rlimit_nofile 40000;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 10240;
}
http {
include mime.types;
default_type application/octet-stream;
include /data/nginx/conf/conf.d/*.conf;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream default_backend_traefik {
server 192.168.13.101:81 max_fails=3 fail_timeout=10s;
server 192.168.13.102:81 max_fails=3 fail_timeout=10s;
server 192.168.13.103:81 max_fails=3 fail_timeout=10s;
server 192.168.13.105:81 max_fails=3 fail_timeout=10s;
server 192.168.13.106:81 max_fails=3 fail_timeout=10s;
server 192.168.13.107:81 max_fails=3 fail_timeout=10s;
server 192.168.13.108:81 max_fails=3 fail_timeout=10s;
}
upstream https_traefik {
server 192.168.13.101:10443 max_fails=3 fail_timeout=10s;
server 192.168.13.102:10443 max_fails=3 fail_timeout=10s;
server 192.168.13.103:10443 max_fails=3 fail_timeout=10s;
server 192.168.13.105:10443 max_fails=3 fail_timeout=10s;
server 192.168.13.106:10443 max_fails=3 fail_timeout=10s;
server 192.168.13.107:10443 max_fails=3 fail_timeout=10s;
server 192.168.13.108:10443 max_fails=3 fail_timeout=10s;
}
upstream harbor_registry {
server harbor.iot.com:1443 max_fails=3 fail_timeout=10s;
}
server {
listen 80;
server_name *.lowan.com;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
proxy_pass http://default_backend_traefik;
proxy_set_header Host $http_host;
proxy_set_header x-Forwarded-For $proxy_add_x_forwarded_for;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_name somename alias another.alias;
# location / {
# root html;
# index index.html index.htm;
# }
#}
# HTTPS server
#
server {
listen 443 ssl http2;
server_name kubernetes.lowan.com;
ssl_certificate "certs/kubernetes.lowan.com.pem";
ssl_certificate_key "certs/kubernetes.lowan.com.key";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
proxy_pass http://default_backend_traefik;
client_max_body_size 1024m;
proxy_set_header Host $http_host;
proxy_set_header x-Forwarded-For $proxy_add_x_forwarded_for;
}
}
server {
listen 443 ssl http2;
server_name harbor.iot.com;
# ssl_certificate "certs/harbor.iot.com.pem";
# ssl_certificate_key "certs/harbor.iot.com.key";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE;
ssl_prefer_server_ciphers on;
location / {
proxy_pass https://harbor_registry;
client_max_body_size 2048m;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header x-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
:x 保存退出
[[email protected]:/data/nginx/conf]# nginx -s reload
同理： 需要将nginx.conf scp至lb01 lb02节点上，三者文件保持一致
7.5 配置DNS解析
[[email protected]:/root]# vim /var/named/lowan.com.zone
添加：
kubernetes A 192.168.13.100
[[email protected]:/root]# systemctl restart named
7.6 登陆测试
登陆网址：https://kubernetes.lowan.com
7.7 配置dashboard分权
注：dashboard分权主要用于研发、测试人员使用有限权限登陆k8s dashboard进行管理
7.7.1 创建ServiceAccount 、ClusterRole和ClusterRoleBinding
注：ClusterRole和ClusterRoleBinding这两个组合的API对象的用法和Role与RoleBinding完全一样,只不过它们的定义里,没有了Namespace字段
[[email protected]:/data/yaml/role-based-access-control]# vim lister-cluster-role.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: lister-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: lister-clusterrole
rules:
- apiGroups: [""]
resources: ["pods", "daemon sets", "deployments", "jobs", "replica sets", "stateful sets", "replication controllers", "service", "ingresses", "configmaps", "namespaces", "nodes"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: lister-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: lister-clusterrole
subjects:
- kind: ServiceAccount
name: lister-user
namespace: kubernetes-dashboard
[[email protected]:/data/yaml/role-based-access-control]# kubectl apply -f lister-cluster-role.yaml
serviceaccount/lister-user created
clusterrole.rbac.authorization.k8s.io/lister-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/lister-clusterrolebinding created
[[email protected]:/data/yaml/role-based-access-control]# kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | grep lister-user | awk '{print $1}')
Name: lister-user-token-nbzzv
Namespace: kubernetes-dashboard
Labels:
Annotations: kubernetes.io/service-account.name: lister-user
kubernetes.io/service-account.uid: b8d2892e-9848-4c17-a75d-4d7e78f160ef
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1359 bytes
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkQ4QldRbFA2emtGNzktSHVUdjRQWU11dkppRTRrZ2tsV2dwelUzeXA3OVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJsaXN0ZXItdXNlci10b2tlbi1uYnp6diIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJsaXN0ZXItdXNlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImI4ZDI4OTJlLTk4NDgtNGMxNy1hNzVkLTRkN2U3OGYxNjBlZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDpsaXN0ZXItdXNlciJ9.a2g9BTnrz2CB0XgkeUcdnvJOX6ceR_9-t_ux3zNcc1ktcjqSfe1-9J1I4tS0taKpJ8BesJQ8tD8IjOGYAv_jDVHpBTcEFdJlFwXKO9ot3YWRZ1Rpoic6b8NfvAAfZnuYCDoMbcIoRPngFIPpC_oFG2DkkGqWbHc2mJHoA5XiQyudAt-uPaTavdlw1cDC3fsKkwh_lVnEHaX8eFyMslUd_voAATAXTLYgMyYk_adElfa2tL63pE2IirRWmbVow4ZoZ2tmi--ORQI6ChXqxM4p9xdSpnnz8P43rBI9HK3ewNuc50iu4Zs7ANuP1qk9W0kWiEqQ5haHpoJB_JzcMtf7Rg
注：此用户只有查看权限
**
后续参考：
02 kubernetes辅助环境设置
03 K8S集群网络ACL规则
04 Ceph集群部署
05 部署zookeeper和kafka集群
06 部署日志系统
07 部署Indluxdb-telegraf
08 部署jenkins
09 部署k3s和Helm-Rancher
10 部署maven软件
11 部署spinnaker的armory发行版