问题:生产环境部署k8s是使用二进制还是使用kubeadm?
https://studygolang.com/articles/18186?fr=sidebar
目前k8s的组件都是通过systemd来维护的,所以二进制安装,调试bug比较方便。从官方渠道来看,kubeadm是官方提供的开源安装工具,带有普适性。
普遍性来讲,使用kubeadm可以带来标准化安装的完整步骤。并且kubeadm已经是一个开源项目,团队投入精力,后面产出都融入到这个项目中了。对个人、团队都有一种荣誉感。反而,二进制安装属于优化版本,自己维护,自己使用。没有社区的协作,之后的维护成本高于回报。
如果没有特别的要求,建议直接使用kubeadm组件来搭建自己的安装k8s的工具。
kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。
官方安装文档
这个工具能通过两条指令完成一个kubernetes集群的部署:
# 创建一个 Master 节点
$ kubeadm init
# 将一个 Node 节点加入到当前集群中
$ kubeadm join <Master节点的IP和端口>
在开始之前,部署Kubernetes集群机器需要满足以下几个条件:
Kubernetes架构图:
角色 | IP地址 |
---|---|
k8s-master | 192.168.0.10 |
k8s-node1 | 192.168.0.20 |
k8s-node2 | 192.168.0.30 |
关闭防火墙:
systemctl stop firewalld
systemctl disable firewalld
关闭selinux:
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
关闭swap:
swapoff -a # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
根据规划设置主机名:
hostnamectl set-hostname <hostname>
在Master添加Hosts:
cat >>/etc/hosts <<EOF
172.16.1.119 k8s-master
172.16.1.120 k8s-node001
172.16.1.121 k8s-node002
EOF
将桥接的IPv4流量传递到iptables的链:
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
Kubernetes默认CRI(容器运行时)为Docker,因此先安装Docker。
# 1. 卸载旧版本
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
# 2. 使用存储库安装
yum install -y yum-utils
# 3. 设置镜像仓库(修改为国内源地址)
yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 5. 更新索引
yum makecache fast
# 4. 安装docker相关的依赖 默认最新版(docker-ce:社区版 ee:企业版)
yum install docker-ce docker-ce-cli containerd.io -y
#5. 安装特定docker版本(先列出列出可用版本)
yum list docker-ce --showduplicates | sort -r
yum install docker-ce-19.03.9 docker-ce-cli-19.03.9 containerd.io
# 6. 启动docker
systemctl start docker
systemctl enable docker
# 7. 查看版本
[root@k8s-master ~]# docker --version
Docker version 19.03.11, build 42e35e61f3
# 8. 配置docker镜像加速
## 后续的kubelet要与docker驱动一致为systemd,这里提前配置好
tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://registry.docker-cn.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl daemon-reload
systemctl restart docker
$ cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
由于版本更新频繁,这里可以指定版本号部署:
yum install kubelet-1.18.5 kubeadm-1.18.5 kubectl-1.18.5 -y
systemctl enable kubelet
#筛选版本号
[root@k8s-master ~]# yum list kubelet kubectl kubeadm --showduplicates | sort -r|grep 1.18.5
kubelet.x86_64 1.18.5-0 kubernetes
kubectl.x86_64 1.18.5-0 kubernetes
kubeadm.x86_64 1.18.5-0 kubernetes
# kubelet要与docker驱动一致为systemd
# 到kubeadm的启动文件中:KUBELET_KUBECONFIG_ARGS 后面追加 --cgroup-driver=systemd
[root@k8s-master ~]# vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --cgroup-driver=systemd"
注意 : 由于kubeadm 默认从官网k8s.grc.io下载所需镜像,国内无法访问,因此需要通--image-repository
指定阿里云镜像仓库地址
#在master节点执行
[root@k8s-master ~]# kubeadm init \
--apiserver-advertise-address=192.168.0.10 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.0 \
--service-cidr=10.10.0.0/16 \
--pod-network-cidr=10.244.0.0/16
## --apiserver-advertise-address : master节点的apiserver-IP通信地址
## --kubernetes-version : k8s版本
## --image-repository : 指定kubeadm镜像仓库的地址
## --service-cidr : 指定为Service分配使用的网络地址,由kubernetes管理
## --pod-network-cidr : 指定Pod分配使用的网段地址,通常应该与要部署使用的网络插件(flannel、calico等)的默认设定保持一致
....
# 记录生成的最后部分内容
kubeadm join 192.168.0.10:6443 --token g2b4cr.044kd5f4gqdqctq8 \
--discovery-token-ca-cert-hash sha256:752c37ab98b7504df0250ed9882cc42b671a11d500530880c0d1b5578498835e
# 在master节点执行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 执行下面命令,使kubectl可以自动补充
source <(kubectl completion bash)
查看节点,pod
#node节点为NotReady,因为corednspod没有启动,缺少网络pod
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 16m v1.18.5
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-qc7mf 0/1 Pending 0 6m
coredns-7ff77c879f-rs5jt 0/1 Pending 0 6m
etcd-k8s-master 1/1 Running 0 6m
kube-apiserver-k8s-master 1/1 Running 0 6m
kube-controller-manager-k8s-master 1/1 Running 0 6m
kube-proxy-nml2s 1/1 Running 0 6m
kube-scheduler-k8s-master 1/1 Running 0 6m
Kubernetes CNI网络对比参考
Kubernetes网络组件之Calico策略实践(BGP、RR、IPIP)
wget https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f calico.yaml
#查看集群pod状态和node状态
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-578894d4cd-qqj8q 1/1 Running 0 5m46s
calico-node-dzsq2 1/1 Running 0 5m46s
coredns-7ff77c879f-qc7mf 1/1 Running 0 43m
coredns-7ff77c879f-rs5jt 1/1 Running 0 43m
etcd-k8s-master 1/1 Running 0 43m
kube-apiserver-k8s-master 1/1 Running 0 43m
kube-controller-manager-k8s-master 1/1 Running 0 43m
kube-proxy-nml2s 1/1 Running 0 43m
kube-scheduler-k8s-master 1/1 Running 0 43m
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 44m v1.18.5
Calico管理工具
下载工具连接:https://github.com/projectcalico/calicoctl/releases/download/v3.14.2/calicoctl
wget https://github.com/projectcalico/calicoctl/releases/download/v3.14.2/calicoctl
chmod +x calicoctl
mv calicoctl /usr/bin/
#添加配置文件
mkdir /etc/calico
[root@k8s-master yaml]# cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "etcdv3"
etcdEndpoints: "https://192.168.0.10:2379,https://192.168.0.20:2379,https://192.168.0.30:2379"
etcdKeyFile: "/etc/kubernetes/pki/etcd/server.key"
etcdCertFile: "/etc/kubernetes/pki/etcd/server.crt"
etcdCACertFile: "/etc/kubernetes/pki/etcd/ca.crt"
[root@k8s-master yaml]# calicoctl node status
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 172.16.0.20 | node-to-node mesh | up | 02:37:05 | Established |
| 172.16.0.30 | node-to-node mesh | up | 02:37:06 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
#node-to-node mesh:就是node节点互通的意思
calicoctl get node
calicoctl get ippool
如果node节点非常多的前提下,node互通就会形成一个巨大的服务网格,连接数也成倍增加
所以需要通过Route Reflector模式(RR) 路由反射器
确定一个或多个calico节点充当路由反射器,让其他节点从这个RR节点获取路由信息
安装flannel网络插件的方法
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
# 默认镜像地址无法访问外网,可以修改为docker hub镜像仓库
# 245684979/flannel:v0.12.0-amd64
sed -ri "s#quay.io/coreos/flannel:.*-amd64#245684979/flannel:v0.11.0-amd64#g" kube-flannel.yml
# 需要注意的是如果节点有多个网卡的话,需要在 kube-flannel.yml 中使用--iface参数指定集群主机内网网卡的名称,否则可能会出现 dns 无法解析。flanneld 启动参数加上--iface=
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0
部署好calico网络插件,Node准备就绪
向集群添加新节点,使用在Master执行kubeadm init最后输出的kubeadm join命令:
#所有node节点上执行
kubeadm join 192.168.0.10:6443 --token g2b4cr.044kd5f4gqdqctq8 \
--discovery-token-ca-cert-hash sha256:752c37ab98b7504df0250ed9882cc42b671a11d500530880c0d1b5578498835e
-----------------------------------------------
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
#翻译:
该节点已加入集群:
* 证书签名请求已发送到apiserver并收到了响应。
* Kubelet被告知新的安全连接详细信息。
在控制平面上运行“ kubectl获取节点”以查看该节点是否已加入集群。
在master节点查看集群pod状态
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-578894d4cd-qqj8q 1/1 Running 0 24m
calico-node-64s8s 1/1 Running 0 10m
calico-node-dzsq2 1/1 Running 0 24m
calico-node-j4t7q 1/1 Running 0 10m
coredns-7ff77c879f-qc7mf 1/1 Running 0 61m
coredns-7ff77c879f-rs5jt 1/1 Running 0 61m
etcd-k8s-master 1/1 Running 0 61m
kube-apiserver-k8s-master 1/1 Running 0 61m
kube-controller-manager-k8s-master 1/1 Running 0 61m
kube-proxy-4n4z6 1/1 Running 0 10m
kube-proxy-jls9k 1/1 Running 0 10m
kube-proxy-nml2s 1/1 Running 0 61m
kube-scheduler-k8s-master 1/1 Running 0 61m
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 63m v1.18.5
k8s-node01 Ready <none> 12m v1.18.5
k8s-node02 Ready <none> 11m v1.18.5
在Kubernetes集群中创建一个pod,验证是否正常运行:
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
[root@k8s-master ~]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-f89759699-f2dwd 1/1 Running 0 8m59s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.10.0.1 <none> 443/TCP 73m
service/nginx NodePort 10.10.23.7 <none> 80:30480/TCP 6m56s
#本地进行访问
[root@k8s-master ~]# curl -I http://10.10.23.7:80
HTTP/1.1 200 OK
Server: nginx/1.19.1
Date: Fri, 17 Jul 2020 10:47:44 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 07 Jul 2020 15:52:25 GMT
Connection: keep-alive
ETag: "5f049a39-264"
Accept-Ranges: bytes
#测试完成后删除pod
kubectl delete deployments nginx
注意: 用kubeadm 安装的k8s集群默认情况 master节点是不参与POD负载的,即POD不会被调度到master节点上运行,只会调度到work 节点上运行,但有时候节点不够多,work node资源不够时可以通过命令让master接受POD调度,注意这种方式只能临时采用,若因参与运行POD导致 master资源不足可能会导致整个集群不稳定。
# 让master节点参与POD负载的命令为
kubectl taint nodes --all node-role.kubernetes.io/master
# 让master节点恢复不参与POD负载的命令为
kubectl taint nodes k8s-master node-role.kubernetes.io/master=:NoSchedule
# 让master节点恢复不参与POD负载,并将Node上已经存在的Pod驱逐出去的命令为
kubectl taint nodes k8s-master node-role.kubernetes.io/master=:NoExecute
官方部署dashboard的服务没使用nodeport,只能集群内部访问,修改Service为NodePort类型,暴露到外部
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc7/aio/deploy/recommended.yaml
[root@k8s-master ~]# vim recommended.yaml (32gg)
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
nodePort: 30001 #添加类型
type: NodePort
selector:
k8s-app: kubernetes-dashboard
# 进行构建
kubectl apply -f recommended.yaml
#查看dashboard的pod状态
[root@k8s-master ~]# kubectl get pods,svc -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-dc6947fbf-xzbtb 1/1 Running 0 35s
pod/kubernetes-dashboard-5d4dc8b976-28ptb 1/1 Running 0 35s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.10.18.108 <none> 8000/TCP 35s
service/kubernetes-dashboard NodePort 10.10.138.185 <none> 443:30001/TCP 36s
访问地址:https://NodeIP:30001
https://192.168.0.20:30001/
创建服务帐号并绑定默认集群管理员角色:
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
使用输出的token进行登录Dashboard,执行下面命令获取token
[root@k8s-master ~]# kubectl get secret -n kube-system|grep admin-token
dashboard-admin-token-56vcv kubernetes.io/service-account-token 3 4d14h
#会生成一串很长的base64后的字符串
[root@k8s-master ~]# kubectl get secret dashboard-admin-token-56vcv -o jsonpath={.data.token} -n kube-system |base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6InhQcU8xTFZFdVFsRXdLT3VtMDdLRDV6OEVhUUItcjNod1NiMTZYbllyWkkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW5 \
0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hb \
WUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tNTZ2Y3YiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkL \
WFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNGFlZTQwZGEtNmE1MS00Mjc1LWE1YzEtZDEwM2UyYzhjNWExI \
iwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.WUdgIC5VWhJAeg-gb95mOKc- \
YlhvMaZdhR1cXiax3bHu44a_U7513DQ5h0IYBvvd3EXsBsavjxZL-DpEHwCHA7l0M5RF1GWHDkdN6MAXVgUO62OdpXXyJMK50QUpOA0yY8PhjoDqUR4wPlWE3-I1xfK_xtxn-au5Z- \
odFxSpzShQFBqjJjWQiAZRqptmHu0o_SaGOHs4SskPmzPXcPKO9bo0iiQY85IOaKvcxwd_993VG1aqRIkvquyjrUVyB7vAxNGPHAWoDG26RwUeE7dsSZSnXUenGu \
CKLz7l68miVkal2QuIIkRX8uwGDGhm3vXmo0ovHcIgBb59I7qoqCN84A
注意 : 在 dashboard 登录页面上直接使用上面创建认集群管理员角色后得到的 token 字符串才可登录,这样就可以拥有管理员权限操作整个 kubernetes 集群的对象,当然你也可以为你的登录用户新建一个指定操作权限的用户。登陆后如果没有namespace可选,并且提示找不到资源 ,那么就是权限问题。
需要安装Metrics Server
#下载官方yaml文件
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
#修改镜像地址(默认国外地址)
containers:
- name: metrics-server
image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp
- --secure-port=4443
command:
- /metrics-server
- --kubelet-preferred-address-types=InternalIP
- --kubelet-insecure-tls
#构建
kubectl apply -f components.yaml
[root@k8s-master ~]# kubectl get pod -n kube-system metrics-server-b8ff4bb4-fvj2w
NAME READY STATUS RESTARTS AGE
metrics-server-b8ff4bb4-fvj2w 1/1 Running 0 18s
#查看node和pod的cpu和内存使用率
[root@k8s-master ~]# kubectl top pod
NAME CPU(cores) MEMORY(bytes)
java-demo-b57dd87fb-5rlsl 4m 184Mi
java-demo-b57dd87fb-cp2tq 2m 168Mi
java-demo-b57dd87fb-shr4k 2m 162Mi
web-64c686b49d-479qh 2m 182Mi
web-64c686b49d-jw95h 4m 171Mi
web-64c686b49d-vbbnx 2m 187Mi
[root@k8s-master ~]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master 370m 18% 1044Mi 36%
k8s-node01 242m 12% 1170Mi 62%
k8s-node02 230m 11% 937Mi 49%
解决Google浏览器不能打开kubernetes dashboard方法
https://blog.csdn.net/Lfwthotpt/article/details/105992874
#创建一个nginx的pod
[root@k8s-master ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
#查看pod状态
[root@k8s-master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-f89759699-mr2fk 1/1 Running 0 27s 10.244.58.198 k8s-node02 <none> <none>
#多进行访问几次
[root@k8s-master ~]# curl 10.244.58.198
在dashboard界面进行查看和访问
查看此pod的日志并登陆访问
访问登陆此pod容器