Kubernetes集群部署v1.17.0
文章目录
- 本地环境
- 环境准备
-
- 设置hostname及域名解析
- 关闭防火墙
- 关闭swap及selinux
- master对node节点免密登陆
- 保证时间同步
- 网桥设置
- 安装docker
- 安装kubelet和kubeadm
-
- master初始化
- node节点加入集群
- 安装网络插件flannel
- 查看集群中pod和service信息
- 测试集群
-
- nginx测试
本地环境
| 软件 | 版本 |
|---|---|
| centos | 7.4-1708 |
| docker | 18.03.0-ce |
| kubernetes | 1.17.0 |
环境准备
设置hostname及域名解析
各节点分别设置主机名
hostnamectl set-hostname master.node
cat >> /etc/hosts <<EOF
10.XX.XX.52 master.node
10.XX.XX.51 worker1.node
10.XX.XX.50 worker2.node
EOF
设置域名解析
cat >> /etc/resolv.conf <<EOF
nameserver 8.8.8.8
nameserver 8.8.4.4
EOF
关闭防火墙
systemctl stop firewalld
systemctl disable firewalld.service
关闭swap及selinux
swapoff -a
或者修改/etc/fstab，将swap那一项注释掉（需重启，永久禁用）
# /etc/fstab
# Created by anaconda on Fri Dec 20 15:28:07 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=21bbe1f9-7e7b-47db-b13a-e54edecf4342 / xfs defaults 0 0
UUID=45103268-084f-4ad5-ab3f-703edda4456a /boot xfs defaults 0 0
# UUID=af6af595-3cd9-4fdc-9489-689eb64fd8c6 swap swap defaults 0 0
关闭SELINUX
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
master对node节点免密登陆
master节点执行
ssh-keygen -t rsa
分别分发到各个节点
ssh-copy-id -i /root/.ssh/id_rsa.pub 10.XX.XX.52
ssh-copy-id -i /root/.ssh/id_rsa.pub 10.XX.XX.51
ssh-copy-id -i /root/.ssh/id_rsa.pub 10.XX.XX.50
保证时间同步
网桥设置
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
安装docker
参考
https://blog.csdn.net/zz_aiytag/article/details/103660090
安装kubelet和kubeadm
增加下载源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF
所有节点执行yum -y install kubelet kubeadm
yum -y install kubelet kubeadm
启动kubelet服务
systemctl start kubelet
systemctl enable kubelet
master初始化
kubeadm init --apiserver-advertise-address=XX.XX.XX.52 --kubernetes-version=v1.17.0 --pod-network-cidr=10.200.0.0/16 --image-repository registry.aliyuncs.com/google_containers
kubeadm init --apiserver-advertise-address=XX.XX.XX.52 \
--kubernetes-version=v1.17.0 \
--pod-network-cidr=10.200.0.0/16 \
--image-repository registry.aliyuncs.com/google_containers
[root@master ~]# kubeadm init --apiserver-advertise-address=10.XX.XX.52 --kubernetes-version=v1.17.0 --pod-network-cidr=10.200.0.0/16 --image-repository registry.aliyuncs.com/google_containers
W0103 13:49:18.212117 1928 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0103 13:49:18.212158 1928 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.03.0-ce. Latest validated version: 19.03
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master.node kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.XX.XX.52]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master.node localhost] and IPs [10.XX.XX.52 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master.node localhost] and IPs [10.XX.XX.52 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0103 13:56:47.334195 1928 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0103 13:56:47.335496 1928 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 34.001970 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master.node as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master.node as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 8mmfuc.88q0nmk4fkqlerh3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.XX.XX.52:6443 --token 8mmfuc.88q0nmk4fkqlerh3 \
--discovery-token-ca-cert-hash sha256:cb4b0096e35a9e7cbdef3cc3d029e50e01d14f0eae5176c660f5dd32ef9cf8b7
根据上面的输出提示进行操作
[root@worker centos7]# mkdir -p $HOME/.kube
[root@worker centos7]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@worker centos7]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@worker centos7]# cd /root/.kube/
[root@worker .kube]# ll
total 8
-rw------- 1 root root 5453 Dec 31 08:36 config
node节点加入集群
[root@worker1 ~]# kubeadm join 10.XX.XX.52:6443 --token 8mmfuc.88q0nmk4fkqlerh3 \
> --discovery-token-ca-cert-hash sha256:cb4b0096e35a9e7cbdef3cc3d029e50e01d14f0eae5176c660f5dd32ef9cf8b7
W0103 13:57:49.773475 1156 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.03.0-ce. Latest validated version: 19.03
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
安装网络插件flannel
在没有安装flannel之前，集群状态是这样的
[root@master .kube]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master.node NotReady master 6m4s v1.17.0
worker1.node NotReady <none> 5m9s v1.17.0
worker2.node NotReady <none> 10s v1.17.0
[root@master kubernetes]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Fri 2020-01-03 13:57:22 CST; 1h 15min ago
Docs: https://kubernetes.io/docs/
Main PID: 3319 (kubelet)
Memory: 43.2M
CGroup: /system.slice/kubelet.service
└─3319 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyunc...
Jan 03 15:12:41 master.node kubelet[3319]: W0103 15:12:41.692331 3319 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jan 03 15:12:42 master.node kubelet[3319]: E0103 15:12:42.067528 3319 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jan 03 15:12:46 master.node kubelet[3319]: W0103 15:12:46.692518 3319 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jan 03 15:12:47 master.node kubelet[3319]: E0103 15:12:47.068386 3319 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jan 03 15:12:51 master.node kubelet[3319]: W0103 15:12:51.692729 3319 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jan 03 15:12:52 master.node kubelet[3319]: E0103 15:12:52.069300 3319 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jan 03 15:12:56 master.node kubelet[3319]: W0103 15:12:56.692884 3319 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jan 03 15:12:57 master.node kubelet[3319]: E0103 15:12:57.070321 3319 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jan 03 15:13:01 master.node kubelet[3319]: W0103 15:13:01.693006 3319 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jan 03 15:13:02 master.node kubelet[3319]: E0103 15:13:02.071537 3319 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
先拉取flannel的镜像，所有的节点都操作
docker pull easzlab/flannel:v0.11.0-amd64
docker tag easzlab/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
查看docker镜像
[root@master kubernetes]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7d54289267dc 3 weeks ago 116MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.17.0 78c190f736b1 3 weeks ago 94.4MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.17.0 0cae8d5cc64c 3 weeks ago 171MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.17.0 5eb3b7486872 3 weeks ago 161MB
registry.aliyuncs.com/google_containers/coredns 1.6.5 70f311871ae1 8 weeks ago 41.6MB
registry.aliyuncs.com/google_containers/etcd 3.4.3-0 303ce5db0e90 2 months ago 288MB
easzlab/flannel v0.11.0-amd64 ff281650a721 11 months ago 52.6MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 11 months ago 52.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 2 years ago 742kB
下载flannel资源配置清单
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
修改kube-flannel.yml文件内容
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.200.0.0/16", #将默认的IP段改成和master init时指定的一致
"Backend": {
"Type": "vxlan"
}
}
安装flannel
[root@master opt]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
再查看节点状态
[root@master opt]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master.node Ready master 102m v1.17.0
worker1.node Ready <none> 101m v1.17.0
worker2.node Ready <none> 96m v1.17.0
查看集群中pod和service信息
[root@master opt]# kubectl get pods,svc -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/coredns-9d85f5447-bpkgg 1/1 Running 0 114m 10.200.0.4 master.node <none> <none>
pod/coredns-9d85f5447-shx4v 1/1 Running 0 114m 10.200.0.2 master.node <none> <none>
pod/etcd-master.node 1/1 Running 0 114m 10.180.249.52 master.node <none> <none>
pod/kube-apiserver-master.node 1/1 Running 0 114m 10.180.249.52 master.node <none> <none>
pod/kube-controller-manager-master.node 1/1 Running 0 114m 10.180.249.52 master.node <none> <none>
pod/kube-flannel-ds-amd64-cgh52 1/1 Running 0 12m 10.180.249.52 master.node <none> <none>
pod/kube-flannel-ds-amd64-cnmnj 1/1 Running 0 12m 10.180.249.50 worker2.node <none> <none>
pod/kube-flannel-ds-amd64-mv4vw 1/1 Running 0 12m 10.180.249.51 worker1.node <none> <none>
pod/kube-proxy-42hpc 1/1 Running 0 114m 10.180.249.52 master.node <none> <none>
pod/kube-proxy-qgfwt 1/1 Running 0 108m 10.180.249.50 worker2.node <none> <none>
pod/kube-proxy-whh5j 1/1 Running 0 113m 10.180.249.51 worker1.node <none> <none>
pod/kube-scheduler-master.node 1/1 Running 0 114m 10.180.249.52 master.node <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 114m k8s-app=kube-dns
测试集群
nginx测试
[root@master opt]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master opt]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
[root@master opt]# kubectl get pods,svc --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default pod/nginx-86c57db685-2klmf 1/1 Running 0 15m 10.200.2.2 worker2.node <none> <none>
kube-system pod/coredns-9d85f5447-bpkgg 1/1 Running 0 132m 10.200.0.4 master.node <none> <none>
kube-system pod/coredns-9d85f5447-shx4v 1/1 Running 0 132m 10.200.0.2 master.node <none> <none>
kube-system pod/etcd-master.node 1/1 Running 0 132m 10.180.249.52 master.node <none> <none>
kube-system pod/kube-apiserver-master.node 1/1 Running 0 132m 10.180.249.52 master.node <none> <none>
kube-system pod/kube-controller-manager-master.node 1/1 Running 0 132m 10.180.249.52 master.node <none> <none>
kube-system pod/kube-flannel-ds-amd64-cgh52 1/1 Running 0 30m 10.180.249.52 master.node <none> <none>
kube-system pod/kube-flannel-ds-amd64-cnmnj 1/1 Running 0 30m 10.180.249.50 worker2.node <none> <none>
kube-system pod/kube-flannel-ds-amd64-mv4vw 1/1 Running 0 30m 10.180.249.51 worker1.node <none> <none>
kube-system pod/kube-proxy-42hpc 1/1 Running 0 132m 10.180.249.52 master.node <none> <none>
kube-system pod/kube-proxy-qgfwt 1/1 Running 0 126m 10.180.249.50 worker2.node <none> <none>
kube-system pod/kube-proxy-whh5j 1/1 Running 0 131m 10.180.249.51 worker1.node <none> <none>
kube-system pod/kube-scheduler-master.node 1/1 Running 0 132m 10.180.249.52 master.node <none> <none>
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 132m <none>
default service/nginx NodePort 10.96.84.231 <none> 80:32751/TCP 13m app=nginx
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 132m k8s-app=kube-dns
只有一个pod，浏览器打开worker2节点的IP:32751可以看到nginx欢迎信息
修改pod副本数量
kubectl scale deployment/nginx --replicas=2
创建过程，需要等待一会
[root@master opt]# kubectl get pods,svc --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default pod/nginx-86c57db685-2klmf 1/1 Running 0 37m 10.200.2.2 worker2.node <none> <none>
default pod/nginx-86c57db685-b4t2h 1/1 Running 0 2m46s 10.200.1.2 worker1.node <none> <none>
现在有两个pod在两个节点上，浏览器测试对应节点IP:32751可以看到nginx欢迎信息