kubeadm安装k8s完整教程
一：硬件环境准备
三台机器，计划为：一台master，两台node
| 序号 | ip | 系统版本 | hostname | 配置 | 节点类型 |
|---|---|---|---|---|---|
| 1 | 192.168.137.61 | CentOS 7.4.1611 (Core) | master61 | 2核2G | Master |
| 2 | 192.168.137.62 | CentOS 7.4.1611 (Core) | node62 | 2核2G | node |
| 3 | 192.168.137.63 | CentOS 7.4.1611 (Core) | node63 | 2核2G | node |
二：系统软件环境预置
1.设置hosts
vim /etc/hosts 加入以下内容 192.168.137.61 master61 192.168.137.62 node62 192.168.137.63 node63
hostnamectl set-hostname master61
hostnamectl set-hostname node62
hostnamectl set-hostname node63
yum install -y ipvsadm
yum install -y wget vim net-tools lrzsz
如果以下命令执行不过去，可以尝试手动输入，可能复制网页代码的问题
2.关闭防火墙
[root@vm210 ~]# systemctl stop firewalld
[root@vm210 ~]# systemctl disable firewalld
[root@vm210 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@vm210 ~]#
3.安装docker
使用官方安装脚本自动安装
安装命令如下：
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
也可以使用国内 daocloud 一键安装命令：
curl -sSL https://get.daocloud.io/docker | sh
systemctl enable --now docker
4.配置yum源
vim /etc/yum.repos.d/kubernetes.repo 加入以下内容 [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0
5.设置SELinux
vim /etc/selinux/config 添加如下SELINUX=disabled 注释掉SELINUX=enforcing,SELINUXTYPE=targeted
# This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. #SELINUX=enforcing # SELINUXTYPE= can take one of three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. #SELINUXTYPE=targeted SELINUX=disabled
6.关闭swap内存
使用swap会影响性能。kubelet禁用swap
1）系统级零时关闭
swapoff -a ，重启后失效
2）系统级全部关闭
vi /etc/fstab，注释掉swap那一行
需要重启。重启后不失效
# # /etc/fstab # Created by anaconda on Mon Dec 2 21:02:22 2019 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/centos-root / xfs defaults 0 0 UUID=b232659c-bd84-46f0-928b-a46d55500934 /boot xfs defaults 0 0 #/dev/mapper/centos-swap swap
7.设置iptables
解决iptables而导致流量无法正确路由的问题
cat </etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system
三：Master节点安装kubeadm
1.安装kubelet 和kubeadm以及kubectl
yum install -y kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0 systemctl enable kubelet yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes systemctl enable --now kubelet
2.启动docker
systemctl enable docker && systemctl start docker
3.下载所需要的镜像
for i in `kubeadm config images list`; do
imageName=${i#k8s.gcr.io/}
docker pull registry.aliyuncs.com/google_containers/$imageName
docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.aliyuncs.com/google_containers/$imageName
done;
否则造成下面：
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.20.15: output: Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
4.更改kubelet的参数
vim /etc/sysconfig/kubelet 改为如下参数 KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
5.kubeadm初始化
kubeadm init 完成之后有如下结果 To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm init \ --apiserver-advertise-address=192.168.137.61 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.20.0 \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 \ --ignore-preflight-errors=all 报错（kubelete无法起动）： [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
created /etc/docker/daemon.json and added below:
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
Then
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl restart kubelet
Run kubeadm init or kubeadm join again.
kubeadm join 192.168.137.61:6443 --token uxc0ef.5t33r5ryhjf2hf32 \
--discovery-token-ca-cert-hash sha256:46e0b740b4121b8db76437d7979c7cfcba2ffd4d535b3469ed136822f3ec86fd
最后的命令需要在node节点中执行，从而加入的k8s集群
依据提示执行如下命令 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
四：node节点安装kubeadm
1.安装kubeadm kubelet
yum install -y kubelet-1.20.0 kubeadm-1.20.0 yum -y install kubeadm kubelet
2.启动docker
systemctl enable docker && systemctl start docker
3.下载所需要的镜像
for i in `kubeadm config images list`; do
imageName=${i#k8s.gcr.io/}
docker pull registry.aliyuncs.com/google_containers/$imageName
docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.aliyuncs.com/google_containers/$imageName
done;
4.更改kubelet的参数
vim /etc/sysconfig/kubelet 改为如下参数 KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
5)加入master
token来自master节点执行kubeinit的结果 kubeadm join 192.168.137.61:6443 --token uxc0ef.5t33r5ryhjf2hf32 \ --discovery-token-ca-cert-hash sha256:46e0b740b4121b8db76437d7979c7cfcba2ffd4d535b3469ed136822f3ec86fd
master中执行命令如下：
kubeadm token create --print-join-command
discovery-token-ca-cert-hash : 用于Node验证Master身份
根据CA的公钥证书数据来计算出hash值
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2>/dev/null | sha256sum | cut -d' ' -f1
这里的计算结果，跟join加入的discovery-token-ca-cert-hash后面接的结果是一样的，一致就可以说加入正确
————————————————
如果加入不进去，提示下面，应该是kubelete启动不了造成的：
[kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz’ failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
kubeadm init
kubeadm reset
rm -rf /etc/cni/net.d
rm -rf $HOME/.kube/config
rm -rf /etc/kubernetes/
或者看上面kubeadm init 故障 或者可能是redhat-release版本不同，请yum update -y
kubernetes:[kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz‘ failed_zJay-L's Blog-CSDN博客[kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz’ failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.背景：搭建k8s高可用集群，由于自己瞎操作，在初始化master（master-1）时操作失败，于是又换了一个master（mastehttps://blog.csdn.net/rookie23rook/article/details/114369501报： accepts at most 1 arg(s), received 3 可以尝试手动输入，可能复制网页代码的问题
kubernetes node节点加入容器 [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forw
输入 : sysctl -w net.ipv4.ip_forward=1 就可以解决这个问题了
五：安装网络插件
master上安装calico.yaml，解决STATUS notReady：
wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml --no-check-certificate
编辑calico.yaml 将pod cidr 192.168.0.0/16 改为 10.244.0.0/16
kubectl apply -f calico.yaml
kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico.yaml
六：查询k8s集群状态
1.查询node节点
[root@vm210 k8s]# kubectl get nodes NAME STATUS ROLES AGE VERSION vm210 Ready master 58m v1.16.3 vm211 Ready21m v1.16.3 vm212 Ready 6m29s v1.16.3
The connection to the server localhost:8080 was refused - did you specify the right host or port?解决
问题分析
环境变量
原因：kubernetes master没有与本机绑定，集群初始化的时候没有绑定，此时设置在本机的环境变量即可解决问题。
问题图片
解决方式
步骤一：设置环境变量
具体根据情况，此处记录linux设置该环境变量
方式一：编辑文件设置
vim /etc/profile
在底部增加新的环境变量 export KUBECONFIG=/etc/kubernetes/admin.conf
方式二:直接追加文件内容
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile
步骤二：使生效
source /etc/profile
The connection to the server localhost:8080 was refused - did you specify the right host or port?解决_CEVERY的博客-CSDN博客问题分析环境变量原因：kubenetes master没有与本机绑定，集群初始化的时候没有绑定，此时设置在本机的环境变量即可解决问题。问题图片解决方式步骤一：设置环境变量具体根据情况，此处记录linux设置该环境变量方式一：编辑文件设置 vim /etc/profile 在底部增加新的环境变量 export KUBECONFIG=/etc/kubernetes/admin.conf方式二:直接追加文件内容echo "export KUBECONFIG=/etc/kubhttps://blog.csdn.net/CEVERY/article/details/108753379
2.查询pods状态
root@vm210 k8s]# kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-55754f75c-7wvrb 1/1 Running 0 6m20s calico-node-9x82m 1/1 Running 0 6m20s calico-node-gn5qh 1/1 Running 0 6m20s calico-node-h8kvz 0/1 PodInitializing 0 6m20s coredns-5644d7b6d9-h9sn2 1/1 Running 0 59m coredns-5644d7b6d9-pwfl5 1/1 Running 0 59m etcd-vm210 1/1 Running 0 58m kube-apiserver-vm210 1/1 Running 0 58m kube-controller-manager-vm210 1/1 Running 0 58m kube-proxy-6hjk2 1/1 Running 0 22m kube-proxy-bcmhh 1/1 Running 0 7m31s kube-proxy-bt9rn 1/1 Running 0 59m kube-scheduler-vm210 1/1 Running 0 58m
3.查询pod具体状态
kubectl --namespace=kube-system describe podkubectl --namespace=kube-system describe pod calico-node-h8kvz
4.master节点也可以像node节点一样调度pod
kubectl taint nodes --all node-role.kubernetes.io/master- [root@vm210 k8s]# kubectl taint nodes --all node-role.kubernetes.io/master- node/vm210 untainted taint "node-role.kubernetes.io/master" not found taint "node-role.kubernetes.io/master" not found
5.查询k8s版本
[root@vm210 k8s]# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
至此， 通过kubeadm工具就实现了Kubernetes集群的快速搭建。 如果安装失败， 则可以执行kubeadm reset命令将主机恢复原状， 重新执行kubeadm init，或者kubeadm join命令， 再次进行安装。
安装的过程中，若有任何问题，欢迎添加vx：xydjun 。大家一起交流探讨
7. 测试kubernetes集群
- 验证Pod工作
- 验证Pod网络通信
- 验证DNS解析
在Kubernetes集群中创建一个pod，验证是否正常运行：
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc
访问地址：http://NodeIP:Port
8. 部署 Dashboard
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.3/aio/deploy/recommended.yaml
默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部：
vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
kubectl apply -f recommended.yaml
kubectl get pods -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-7b59f7d4df-6qglx 1/1 Running 0 21h
kubernetes-dashboard-5dbf55bd9d-hv78v 1/1 Running 6 21h
访问地址：https://NodeIP:30001
创建service account并绑定默认cluster-admin管理员集群角色：
```
创建用户
kubectl create serviceaccount dashboard-admin -n kube-system
用户授权
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
获取用户Token
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
``` 使用输出的token登录Dashboard。
9. 切换容器引擎为Containerd
https://kubernetes.io/zh/docs/setup/production-environment/container-runtimes/#containerd
1、配置先决条件
cat <设置必需的 sysctl 参数，这些参数在重新启动后仍然存在。
# Setup required sysctl params, these persist across reboots.
cat <2、安装containerd
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum update -y && sudo yum install -y containerd.io
mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
systemctl restart containerd
3、修改配置文件
vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.aliyuncs.com/googlecontainers/pause:3.2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://b9pmyelo.mirror.aliyuncs.com"]
systemctl restart containerd
4、配置kubelet使用containerd
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroup-driver=systemd
or:
echo "KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroup-driver=systemd" > /etc/sysconfig/kubelet
systemctl restart kubelet
5、验证
kubectl get node -o wide
k8s-node1 xxx containerd://1.4.4 ```
课堂笔记
``` 怎么查看容器日志？ kubectl logs <容器名称> -n kube-system
怎么查看容器事件？ kubectl describe pod <容器名称> -n kube-system
calico无法拉取镜像解决办法？
grep image calico.yaml
image: calico/cni:v3.15.1 image: calico/pod2daemon-flexvol:v3.15.1 image: calico/node:v3.15.1
docker pull xxx docker save calico/cni:v3.15.1 > cni.tar docker load < cni.tar kubectl delete -f calico.yaml kubectl apply -f calico.yaml
init失败或者情况环境可以使用： kubeadm reset
为什么部署网络组件？ Q1：每个docker主机创建的容器ip可能冲突? Q2：容器1访问容器2，容器1怎么知道容器2在哪个docker主机？ Q3：容器1访问容器2数据包怎么传输过去？
1、k8s现在可以使用docker嘛？ 可以。 2、dockershim什么时候被移除？ 预计1.23版本。 3、docker还值的学习嘛？ 值得。
kubectl get pods --show-labels # 查看资源标签 kubectl get pod -l app=web # 根据标签筛选资源