Centos7安装K8S集群环境
文章目录
- 一、系统设置
- 环境
- 1、关闭swap
- 2、 关闭SELinux
- 3、关闭防火墙
- 4、配置sysctl
- 5、配置安装源为阿里
- 5.1 配置yum安装源
- 5.2 配置k8s源
- 5.3 重建yum缓存
- 二、安装docker
- 1、安装docker
- 2、docker配置
- 3、重启docker
- 三、集群安装
- 1、安装kubeadm, kubelet和kubectl
- 2、使用kubeadm创建集群
- 3、用户设置权限(root用户也需要执行)
- 4、应用flannel网络
- 5、节点加入
- 四、安装网页界面 (Dashboard)
- 参考资料
一、系统设置
环境
[root@st01015vm192 /]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[root@st01015vm192 /]# uname -a
Linux st01015vm192 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
默认为root用户安装
1、关闭swap
临时关闭swap
swapoff -a
永久关闭
注释掉 /etc/fstab 中的下面配置
#/dev/mapper/centos-swap swap swap defaults 0 0
2、 关闭SELinux
kubelet不支持SELinux, 这里需要将SELinux设置为permissive模式
# 查看状态
# /usr/sbin/sestatus -v
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
3、关闭防火墙
systemctl disable firewalld
systemctl stop firewalld
4、配置sysctl
创建文件/etc/sysctl.d/k8s.conf， 文件内容如下
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
执行
sysctl -p /etc/sysctl.d/k8s.conf
在RHEL/CentOS 7上由于 iptables 被绕过导致网络请求被错误的路由。您得保证 在您的 sysctl 配置中 net.bridge.bridge-nf-call-iptables 被设为1。
插件将容器连接到 Linux 网桥，插件必须将 net/bridge/bridge-nf-call-iptables 系统参数设置为1，以确保 iptables 代理正常工作。
最后，在内核中启用了 IP 转发（因此内核将处理桥接容器的数据包）：
sysctl net.ipv4.ip_forward=1
所有这些的结果是所有 Pods 都可以互相访问，并且可以将流量发送到互联网。
k8s网络插件
https://kubernetes.io/zh/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
k8s 网络模型
https://kubernetes.io/zh/docs/concepts/cluster-administration/networking/
5、配置安装源为阿里
5.1 配置yum安装源
## 备份
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
## 下载阿里源
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
5.2 配置k8s源
vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
5.3 重建yum缓存
yum clean all
yum makecache fast
yum -y update
二、安装docker
1、安装docker
可以参考官网文档 https://docs.docker.com/engine/install/centos/
- 卸载旧版本
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
- 安装docker
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io
如果想要安装指定版本docker
yum list docker-ce --showduplicates | sort -r
sudo yum install docker-ce- docker-ce-cli- containerd.io
2、docker配置
创建文件/etc/docker/daemon.json，写入配置
mkdir /etc/docker/
vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
// 如果在国内安装，添加以下配置
// "registry-mirrors": [
// "https://registry.docker-cn.com"
// ],
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
国内docker源
"registry-mirrors": [
"https://1nj0zren.mirror.aliyuncs.com",
"https://docker.mirrors.ustc.edu.cn",
"http://f1361db2.m.daocloud.io",
"https://registry.docker-cn.com"
]
3、重启docker
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker
可能会报错，docker.service failed
[root@st01015vm193 ~]# journalctl -xe
May 08 15:36:09 st01015vm193 systemd[1]: Dependency failed for Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is dependency.
May 08 15:36:09 st01015vm193 systemd[1]: Job docker.service/start failed with result 'dependency'.
May 08 15:36:09 st01015vm193 systemd[1]: Unit docker.socket entered failed state.
需要给系统添加一个docker组，如果做过基线配置的话，会提示没有权限，使用 chattr -i 增加一下权限
chattr -i /etc/group
groupadd docker
systemctl enable docker && systemctl start docker
三、集群安装
1、安装kubeadm, kubelet和kubectl
yum install -y kubelet kubeadm kubectl kubernetes-cni --disableexcludes=kubernetes
systemctl enable --now kubelet && systemctl start kubelet
2、使用kubeadm创建集群
只在master节点执行
# master节点执行：
kubeadm init \
--apiserver-advertise-address 10.10.45.192 \
--pod-network-cidr=10.244.0.0/16
# --kubernetes-version=v1.15.0 \
# --image-repository=registry.aliyuncs.com/google_containers
# --apiserver-advertise-address 指定与其它节点通信的接口
# --pod-network-cidr 指定pod网络子网，使用fannel网络必须使用这个CIDR
可能会报错如下
[root@st01015vm192 ~]# kubeadm init \
> --apiserver-advertise-address 10.10.45.192 \
> --pod-network-cidr=10.244.0.0/16
W0508 15:09:35.577282 28115 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[ERROR Swap]: running with swap on is not supported. Please disable swap
[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
加入这个参数：–ignore-preflight-errors=all
报错如下
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
一般情况下是kubelet启动失败，查看状态如日志，查找原因，我这里是swap临时关闭的，导致机器重启后swap被打开了：
systemctl status kubelet
journalctl -xeu kubelet
安装成功后，有如下打印
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.10.45.192:6443 --token 82scon.3zopf5qra2b1s25i \
--discovery-token-ca-cert-hash sha256:2ea38c2a269d105b09bbf2964c089c067f0c8e7b44c0504b5854fd9acac263e0
3、用户设置权限(root用户也需要执行)
# master节点执行：
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
4、应用flannel网络
sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
## 查看flannal是否安装成功
sudo kubectl -n kube-system get po -l app=flannel -o wide
5、节点加入
按照在master节点上构建集群后的打印，执行节点加入集群操作
kubeadm join 10.10.45.192:6443 --token 82scon.3zopf5qra2b1s25i
–discovery-token-ca-cert-hash sha256:2ea38c2a269d105b09bbf2964c089c067f0c8e7b44c0504b5854fd9acac263e0
W0508 15:42:29.235510 5131 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "st01015vm194" could not be reached
[WARNING Hostname]: hostname "st01015vm194": lookup st01015vm194 on 192.168.16.24:53: no such host
修改/etc/hosts，添加hosts
10.10.45.192 st01015vm192
10.10.45.193 st01015vm193
10.10.45.194 st01015vm194
分别在节点上执行加入集群操作，执行完成后，在master节点上查看节点状态：
[root@st01015vm192 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
st01015vm192 Ready master 16d v1.18.2
st01015vm193 NotReady <none> 29s v1.18.2
st01015vm194 NotReady <none> 6m43s v1.18.2
发现节点状态一直是NotReady，检查pod状态
kubectl get pods -n kube-system -o wide
kube-flannel-ds-amd64-sdvpf 0/1 Init:ImagePullBackOff 0 8m12s 10.10.45.193 st01015vm193
kube-flannel-ds-amd64-x58td 0/1 Init:ImagePullBackOff 0 14m 10.10.45.194 st01015vm194
发现子节点上的flannel pod报错，查看详细信息
kubectl describe pod -n kube-system kube-flannel-ds-amd64-sdvpf
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled default-scheduler Successfully assigned kube-system/kube-flannel-ds-amd64-sdvpf to st01015vm193
Warning Failed 2m22s (x2 over 4m33s) kubelet, st01015vm193 Failed to pull image "quay.io/coreos/flannel:v0.12.0-amd64": rpc error: code = Unknown desc = context canceled
Warning Failed 2m22s (x2 over 4m33s) kubelet, st01015vm193 Error: ErrImagePull
Normal BackOff 2m11s (x2 over 4m33s) kubelet, st01015vm193 Back-off pulling image "quay.io/coreos/flannel:v0.12.0-amd64"
Warning Failed 2m11s (x2 over 4m33s) kubelet, st01015vm193 Error: ImagePullBackOff
Normal Pulling 118s (x3 over 7m28s) kubelet, st01015vm193 Pulling image "quay.io/coreos/flannel:v0.12.0-amd64"
Event中显示镜像拉取失败，这个可能是网络问题，pod运行失败后，会尝试重新运行，所以耐心等待一会，或者在失败的节点上手动拉取一下镜像，并且修改deployment中的imagePullPolicy: Always
By default, the kubelet will try to pull each image from the specified registry. However, if the imagePullPolicy property of the container is set to IfNotPresent or Never, then a local image is used (preferentially or exclusively, respectively).
修改为 imagePullPolicy: IfNotPresent
docker pull quay.io/coreos/flannel:v0.12.0-amd64
[root@st01015vm192 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
st01015vm192 Ready master 17d v1.18.2
st01015vm193 Ready 33m v1.18.2
st01015vm194 Ready 39m v1.18.2
四、安装网页界面 (Dashboard)
默认情况下不会部署 Dashboard。可以通过以下命令部署：
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
为了保护您的集群数据，默认情况下，Dashboard 会使用最少的 RBAC 配置进行部署。 当前，Dashboard 仅支持使用 Bearer 令牌登录。
所以，我们需要下载yaml并进行配置
我们将以下配置进行修改
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
修改后如下
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
修改镜像拉取策略为IfNotPresent或Never，并在k8s所有节点上拉取镜像，这一步是为了避免pod创建时因为网络原因拉取镜像失败
docker pull kubernetesui/metrics-scraper:v1.0.4
docker pull kubernetesui/dashboard:v2.0.0
如果某一台机器上已经拉取镜像成功了，而其他机器一直拉取不成功，可以将镜像备份后在不成功的机器上还原
备份
docker save -o dashboard.tar kubernetesui/dashboard:v2.0.0
docker save -o metrics-scraper.tar kubernetesui/metrics-scraper:v1.0.4
还原
docker load -i dashboard.tar
docker load -i metrics-scraper.tar
应用
kubectl apply -f recommended.yaml
检查状态
kubectl get pod,svc,ing,deploy -n kubernetes-dashboard
待所有pod都运行起来后，查看
https://10.10.45.192:30001/
安装完成后，在master节点上获取token
[root@st01015vm192 ~]# kubectl -n kubernetes-dashboard get secret
NAME TYPE DATA AGE
default-token-25jb9 kubernetes.io/service-account-token 3 23m
kubernetes-dashboard-certs Opaque 0 23m
kubernetes-dashboard-csrf Opaque 1 23m
kubernetes-dashboard-key-holder Opaque 2 23m
kubernetes-dashboard-token-fm795 kubernetes.io/service-account-token 3 23m
[root@st01015vm192 ~]# kubectl -n kubernetes-dashboard describe secret kubernetes-dashboard-token-fm795
Name: kubernetes-dashboard-token-fm795
Namespace: kubernetes-dashboard
Labels:
Annotations: kubernetes.io/service-account.name: kubernetes-dashboard
kubernetes.io/service-account.uid: af7a61cf-901f-42f9-bcbe-6f521d026bc2
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1025 bytes
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlhCZGZQa0MtaVFmMHJ0YTRBS083emppS0tKSENvb24xeW9scHIxY19zU0kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1mbTc5NSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImFmN2E2MWNmLTkwMWYtNDJmOS1iY2JlLTZmNTIxZDAyNmJjMiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.haW6XvBAyog0BasqbaWJxPqWjTJKiemVBwP3J8dwFE43Q93Jx41yjxK41NRNaUflL8xL3Aj4CNIJ0YUQwlpIutIzOJq7rXWkneRI6tmgr3jsCarFtjwETph7-spg-WQAHXRQxt7hwMyxcNkJprEc13q6zGO_ycx9ei_hjjliXo0O8JMuQsL0rlm2zXrWOpRer5U77Hj33dnVSGrjvlD3X_5NsI0dlzG2MmKMFZHM0_PVbYFnSvWcEmLl_04_u5CJPtPfp9Pu6RTjy1lMOZtsHgBxqDC-oXxm0UP2Tcn2qlu_UDfIPhiL3r-QrwWFy7b3WpxJCcXwcm07pfUzijQ77A
在 https://10.10.45.192:30001/ 上使用token登录即可。
参考资料
https://my.oschina.net/u/2539854/blog/3023384
https://juejin.im/post/5d089f49f265da1baa1e7611