centos7.1使用kubeadm部署kubernetes 1.16.2的master高可用
机器列表，配置域名解析
cat /etc/hosts
192.168.200.210 k8s-master1
192.168.200.211 k8s-master2
192.168.200.212 k8s-node1
192.168.200.213 k8s-node2
192.168.200.214 k8s-master-vip环境版本
centos 7.1
docker 19.03.4
kubernetes 1.16.2前期准备
1、Linux查看版本当前操作系统内核信息
uname -a输出
Linux iZwz9d75c59ll4waz4y8ctZ 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
2、Linux查看当前操作系统版本信息
cat /proc/version输出
Linux version 3.10.0-693.2.2.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Sep 12 22:26:13 UTC 2017
Linux查看版本当前操作系统发行版信息
cat /etc/redhat-release输出
CentOS Linux release 7.4.1708 (Core)
3、升级内核
# 升级yum，升级系统内核，系统内核最好能是4.10以上，可以使用overlay2存储驱动
# 本人在自己虚拟机使用centos7.6，升级内核后，可以用到overlay2存储驱动
# 在服务器上由于某些原因没办法升级内核，在3.10版本也能部署成功，只是存储驱动使用overlay
yum update -y
# 安装工具
yum install -y wget curl vim 4、设置主机名，全部机器都要修改
hostnamectl set-hostname master5、关闭防火墙 、selinux和swap
systemctl disable firewalld --now
setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
# swappiness=0的时候表示最大限度使用物理内存，然后才是 swap空间
swapoff -a
echo "vm.swappiness = 0">> /etc/sysctl.conf
sed -i 's/.*swap.*/#&/' /etc/fstab
sysctl -p
6、配置内核参数，将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system一、配置软件源
1、配置yum源base repo为阿里云的yum源
cd /etc/yum.repos.d
mv CentOS-Base.repo CentOS-Base.repo.bak
mv epel.repo epel.repo.bak
curl https://mirrors.aliyun.com/repo/Centos-7.repo -o CentOS-Base.repo
sed -i 's/gpgcheck=1/gpgcheck=0/g' /etc/yum.repos.d/CentOS-Base.repo
curl https://mirrors.aliyun.com/repo/epel-7.repo -o epel.repo gpkcheck=0 表示对从这个源下载的rpm包不进行校验
2、配置docker repo
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O/etc/yum.repos.d/docker-ce.repo3、配置kubernetes源为阿里的yum源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOFbaseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 Kubernetes源设为阿里
gpgcheck=0：表示对从这个源下载的rpm包不进行校验
repo_gpgcheck=0：某些安全性配置文件会在 /etc/yum.conf 内全面启用 repo_gpgcheck，以便能检验软件库的中继数据的加密签署
如果gpgcheck设为1，会进行校验，就会报错如下，所以这里设为0
repomd.xml signature could not be verified for kubernetes4、update cache 更新缓存
yum clean all
yum makecache
yum repolist二、安装docker
查看可安装的版本
yum list docker-ce --showduplicates | sort -r安装
yum install -y docker-ce启动docker
systemctl enable docker && systemctl start docker查看docker版本
docker version设置docker的镜像源，这里我设置为国内的docker源https://www.docker-cn.com
vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"registry-mirrors": ["https://www.docker-cn.com"]
}注意：native.cgroupdriver=systemd 官方推荐此配置，地址 https://kubernetes.io/docs/setup/cri/重新加载daemon，重启docker
systemctl daemon-reload
systemctl restart docker如果docker重启失败提示start request repeated too quickly for docker.service
应该是centos的内核太低内核是3.x，不支持overlay2的文件系统，移除下面的配置
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]三、安装kubeadm、kubelet和kubectl
kubeadm不管kubelet和kubectl，所以我们安装kubeadm，还要安装kubelet和kubectl
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetesKubelet负责与其他节点集群通信，并进行本节点Pod和容器生命周期的管理。
Kubeadm是Kubernetes的自动化部署工具，降低了部署难度，提高效率。
Kubectl是Kubernetes集群管理工具。
最后启动kubelet：
systemctl enable kubelet --now注：因为kubeadm默认生成的证书有效期只有一年，所以kubeadm等安装成功后，我用自己之前编译好的kubeadm替换掉默认的kubeadm。后面初始化k8s生成的证书都是100年。编译方案看我另外一个文章：通过编译kubeadm修改证书有效期
四、master高可用（只需要在k8s-master1、k8s-master2上操作）
1、master节点全部ssh免密登录
# 先cd到/root/.ssh目录
# 如果没有.ssh目录，只需要本地ssh本机或者其他主机，就会生成.ssh文件夹
# 首先登陆master1
ssh-keygen -t rsa //一路回车
ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-master2
# 尝试登陆一下
ssh root@k8s-master2
# 然后登陆master2
ssh-keygen -t rsa //一路回车
ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-master1
# 尝试登陆一下
ssh root@k8s-master12、keepalived创建虚拟ip
在k8s的master节点的机器上安装keepalived，用于漂移虚拟ip
# 两台master节点都要安装keepalived
yum install -y keepalived在k8s-master1节点，配置keepalived角色为MASTER
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id uat-k8s-master1 #标识
}
vrrp_instance VI_1 {
state MASTER #角色是master
interface eth0 #vip 绑定端口
virtual_router_id 51 #让master 和backup在同一个虚拟路由里，id 号必须相同
priority 150 #优先级,谁的优先级高谁就是master
advert_int 1 #心跳间隔时间
authentication {
auth_type PASS #认证
auth_pass k8s #密码
}
virtual_ipaddress {
192.168.200.214 #虚拟ip
}
}
EOF
# 启动keepalived
systemctl enable keepalived.service && systemctl start keepalived.service在k8s-master2节点，配置keepalived角色为BACKUP
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id uat-k8s-master2
}
vrrp_instance VI_1 {
state BACKUP #角色是backup
interface eth0
virtual_router_id 51 #让master 和backup在同一个虚拟路由里，id 号必须相同
priority 100 #优先级,设置得比master低
advert_int 1 #心跳间隔时间
authentication {
auth_type PASS #认证
auth_pass k8s #密码
}
virtual_ipaddress {
192.168.200.214 #虚拟ip
}
}
EOF
# 启动keepalived
systemctl enable keepalived.service && systemctl start keepalived.service测试一下keepalived虚拟ip是否启动成功
分别在k8s-master1和k8s-master2执行如下命令，可以发现只有k8s-master1有结果
ip a | grep 192.168.200.214
显示如下
inet 192.168.200.214/32 scope global eth0
说明目前192.168.200.214虚拟ip是漂移到k8s-master1机器上，因为k8s-master1的keepalived设置为MASTER
测试keepalived的ip漂移功能
尝试重启k8s-master1机器，重启期间，虚拟ip会漂移到k8s-master2机器，
输入命令ip a | grep 192.168.200.21"可以看到结果，
等k8s-master1重启完毕，虚拟ip会重新漂移到k8s-master1上
证明keepalived运行正常
3、haproxy负载均衡
# 两台master节点都要安装haproxy
yum install -y haproxy在k8s-master1、k8s-master1节点，分别配置haproxy
cat > /etc/haproxy/haproxy.cfg << EOF
global
chroot /var/lib/haproxy
daemon
group haproxy
user haproxy
log 127.0.0.1:514 local0 warning
pidfile /var/lib/haproxy.pid
maxconn 20000
spread-checks 3
nbproc 8
defaults
log global
mode tcp
retries 3
option redispatch
listen https-apiserver
bind 0.0.0.0:8443 # 指定绑定的端口，ip都设置为0.0.0.0，我这里使用8443端口
mode tcp
balance roundrobin
timeout server 15s
timeout connect 15s
server apiserver1 192.168.200.210:6443 check port 6443 inter 5000 fall 5 #转发到k8s-master1的apiserver上，apiserver端口默认是6443
server apiserver2 192.168.200.211:6443 check port 6443 inter 5000 fall 5 #转发到k8s-master2的apiserver上，apiserver端口默认是6443
EOF
# 启动haproxy
systemctl start haproxy.service && systemctl enable haproxy.service上面配置只需要修改bind，还有server分发部分，其他不需要改变
分别查看haproxy的运行状态，确保haproxy正在运行
systemctl status haproxy4、初始化master
首先初始化k8s-master1，初始化之前检查haproxy是否正在运行，keepalived是否正常运作
查看所需的镜像
kubeadm config images list显示如下
k8s.gcr.io/kube-apiserver:v1.16.2
k8s.gcr.io/kube-controller-manager:v1.16.2
k8s.gcr.io/kube-scheduler:v1.16.2
k8s.gcr.io/kube-proxy:v1.16.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2能的可以通过以下命令提前把镜像pull下来
kubeadm config images pull不能，唯有在初始化时指定使用阿里的镜像库：registry.aliyuncs.com/google_containers
开始初始化
kubeadm init \
--apiserver-advertise-address=当前master机器的ip \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.16.2 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16 \
--control-plane-endpoint 192.168.200.214:8443 \
--upload-certs上面指定的参数都不能缺
参数描述
–apiserver-advertise-address：用于指定kube-apiserver监听的ip地址,就是 master本机IP地址。
–image-repository: 指定阿里云镜像仓库地址
–kubernetes-version: 用于指定k8s版本；
–pod-network-cidr：用于指定Pod的网络范围；10.244.0.0/16
–service-cidr：用于指定SVC的网络范围；
–control-plane-endpoint：指定keepalived的虚拟ip
–upload-certs：上传证书初始化成功后，会看到大概如下提示，下面信息先保留。后续添加master节点，添加node节点需要用到
Your Kubernetes control-plane has initialized successfully!
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 192.168.200.214:8443 --token fo15mb.w2c272dznmpr1qil \
--discovery-token-ca-cert-hash sha256:3455de957a699047e817f3c42b11da1cc665ee667f78661d20cfabc5abcc4478 \
--control-plane --certificate-key bcd49ad71ed9fa66f6204ba63635a899078b1be908a69a30287554f7b01a9421
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.200.214:8443 --token fo15mb.w2c272dznmpr1qil \
--discovery-token-ca-cert-hash sha256:3455de957a699047e817f3c42b11da1cc665ee667f78661d20cfabc5abcc4478 这时使用kubectl get node能看到一个master节点处于NotReady状态，那是因为没安装网络
注：如果由于初始化信息没设置好，已经执行了初始化命令，可以使用kubeadm reset重置。但本人部署时遇到问题，reset后，再次初始化，始终不成功。发现docker ps里面全部镜像都没启动。怀疑是reset时，某些docker相关的文件没有删除干净。通过yum remove docker-ce卸载docker，重启服务器，再重新安装docker，我的问题就解决了
按提示执行如下命令，kubectl就能使用了
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
5、将pod网络部署到集群
这里我是用flannel网络
下载kube-flannel.yml文件
curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml下载后的文件
注：我所在的网络可能有限制无法访问，是用自己的阿里云下载好，再拿过来使用的，并且应修改了网卡，这里贴上文件内容
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unsed in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"cniVersion": "0.2.0",
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0 # 改成你服务器在使用的网卡名
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- arm64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-arm64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-arm64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- arm
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-arm
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-arm
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-ppc64le
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- ppc64le
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-ppc64le
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-ppc64le
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-s390x
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- s390x
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-s390x
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-s390x
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg安装flannel network，还要保证你的网络能正常拉取镜像quay.io/coreos/flannel:v0.11.0-amd64
kubectl apply -f ./kube-flannel.yml这里注意kube-flannel.yml这个文件里的flannel的镜像是0.11.0，quay.io/coreos/flannel:v0.11.0-amd64执行后，显示如下，表示flannel部署成功
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
如果flannel创建失败的，则可能是服务器存在多个网卡，目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=
注意：kube-flannel.yml文件里面有多个containers，要在quay.io/coreos/flannel:v0.11.0-amd64这个containers的启动参数里面设
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0containers下的args指定参数iface=eth0，前提先ifconfig查看一下本地ip使用的网卡名称：eth0
修改之前下载的kube-flannel.yml，修改containers下的args启动参数，指定网卡名称
修改完kube-flannel.yml后，先删除之前部署的网络
kubectl delete -f ./kube-flannel.yml再重新创建
kubectl apply -f ./kube-flannel.yml执行完后，查看node状态
NAME STATUS ROLES AGE VERSION
uat-k8s-master1 Ready master 5h42m v1.16.2查看pod
kubectl get pod -n kube-systemNAME READY STATUS RESTARTS AGE
coredns-58cc8c89f4-pb8b4 1/1 Running 0 5h43m
coredns-58cc8c89f4-qwpx5 1/1 Running 0 5h43m
etcd-uat-k8s-master1 1/1 Running 1 5h42m
kube-apiserver-uat-k8s-master1 1/1 Running 1 5h43m
kube-controller-manager-uat-k8s-master1 1/1 Running 3 5h42m
kube-flannel-ds-amd64-6zjrx 1/1 Running 0 4h30m
kube-proxy-v6wcg 1/1 Running 1 5h43m
kube-scheduler-uat-k8s-master1 1/1 Running 3 5h43m
6、添加另外一个控制平面，也就是添加另外一个master
使用之前生成的提示信息，在k8s-master2执行，就能添加一个master
kubeadm join 192.168.200.214:8443 --token fo15mb.w2c272dznmpr1qil \
--discovery-token-ca-cert-hash sha256:3455de957a699047e817f3c42b11da1cc665ee667f78661d20cfabc5abcc4478 \
--control-plane --certificate-key bcd49ad71ed9fa66f6204ba63635a899078b1be908a69a30287554f7b01a9421执行完如上初始化命令，第二个master节点就能添加到集群
提示信息里面有如下内容，应该是之前指定参数上传的证书2个小时后会被删除，可以使用命令重新上传证书
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.五、添加node节点
kubeadm join 192.168.200.214:8443 --token fo15mb.w2c272dznmpr1qil \
--discovery-token-ca-cert-hash sha256:3455de957a699047e817f3c42b11da1cc665ee667f78661d20cfabc5abcc4478
完事。。。。
keepalived脑裂问题，这个我还没去处理，后面再找时间摸索一下