系统:centos7.6, 3节点信息如下:
IP | hosteName |
---|---|
192.168.3.31 | master |
192.168.3.32 | node2 |
192.168.3.33 | node3 |
注意:官方建议每台机器至少双核2G内存,同时需确保MAC和product_uuid唯一
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
systemctl stop firewalld && systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sysctl -p
cat << EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
# base repo
cd /etc/yum.repos.d
mv CentOS-Base.repo CentOS-Base.repo.bak
curl -o CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
sed -i 's/gpgcheck=1/gpgcheck=0/g' /etc/yum.repos.d/CentOS-Base.repo
# docker repo
curl -o docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# k8s repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# update cache
yum clean all
yum makecache
yum repolist
yum install -y epel-release conntrack ipvsadm ipset jq sysstat curl iptables libseccomp
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --enable docker-ce-edge
yum-config-manager --enable docker-ce-test
目前k8s-1.14 使用的是 dock-18.06-ce
查看可安装的docker-ce列表
yum list docker-ce --showduplicates
安装最新版docker-ce
yum install docker-ce
要安装指定版本docker
yum install docker-ce-18.06.1.ce-3.el7
启动并设置为开机自启动
如果安装最新版的,把版本号去掉即可。
yum -y install kubelet-1.14.0 kubeadm-1.14.0 kubectl-1.14.0 kubernetes-cni-0.7.5
检查所有服务版本
[root@master ~]# rpm -qa docker-ce kubelet kubeadm kubectl kubernetes-cni
docker-ce-18.06.0.ce-3.el7.x86_64
kubelet-1.14.0-0.x86_64
kubectl-1.14.0-0.x86_64
kubeadm-1.14.0-0.x86_64
kubernetes-cni-0.7.5-0.x86_64
systemctl enable docker
systemctl enable kubelet.service
systemctl start docker
systemctl start kubelet
## 二、下载相关镜像
kubeadm config images list
[root@master ~]# kubeadm config images list
I0623 21:31:50.520669 30468 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0623 21:31:50.520780 30468 version.go:97] falling back to the local client version: v1.14.0
k8s.gcr.io/kube-apiserver:v1.14.0
k8s.gcr.io/kube-controller-manager:v1.14.0
k8s.gcr.io/kube-scheduler:v1.14.0
k8s.gcr.io/kube-proxy:v1.14.0
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1
kubeadm config print init-defaults > kubeadm.conf
kubeadm.conf
文件的镜像地址默认为google的镜像仓库地址k8s.gcr.io,国内无法访问,需要把地址修改为国内的地址,这里使用阿里云的镜像仓库地址。
编辑kubeadm.conf
,将imageRepository
修改为registry.aliyuncs.com/google_containers
。并确认Kubernetes版本是v1.14.0,和2.1中的镜像列表的版本保持一致
vim kubeadm.conf
将imageRepository: k8s.gcr.io
-->
imageRepository:registry.aliyuncs.com/google_containers
kubeadm config images pull --config kubeadm.conf
修改tag
docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.14.0 k8s.gcr.io/kube-apiserver:v1.14.0
docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.14.0 k8s.gcr.io/kube-controller-manager:v1.14.0
docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.14.0 k8s.gcr.io/kube-scheduler:v1.14.0
docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0
docker tag registry.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag registry.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag registry.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
再删除阿里云镜像
docker rmi registry.aliyuncs.com/google_containers/kube-apiserver:v1.14.0
docker rmi registry.aliyuncs.com/google_containers/kube-controller-manager:v1.14.0
docker rmi registry.aliyuncs.com/google_containers/kube-scheduler:v1.14.0
docker rmi registry.aliyuncs.com/google_containers/kube-proxy:v1.14.0
docker rmi registry.aliyuncs.com/google_containers/pause:3.1
docker rmi registry.aliyuncs.com/google_containers/etcd:3.3.10
docker rmi registry.aliyuncs.com/google_containers/coredns:1.3.1
或者使用脚本解决:
cat image.sh
#!/bin/bash
images=(kube-proxy:v1.14.0 kube-scheduler:v1.14.0 kube-controller-manager:v1.14.0 kube-apiserver:v1.14.0 etcd:3.3.10 coredns:1.3.1 pause:3.1 )
for imageName in ${images[@]} ; do
docker pull registry.aliyuncs.com/google_containers/$imageName
docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.aliyuncs.com/google_containers/$imageName
done
kubernetes集群不允许开启swap,所以我们需要忽略这个错误
编辑文件 /etc/sysconfig/kubelet,将文件里的“KUBELET_EXTRA_ARGS=”改成这样:KUBELET_EXTRA_ARGS="–fail-swap-on=false"
修改之后的文件:
cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
这里定义先POD的网段为: 10.244.0.0/16
,API Server地址为Master节点的IP地址。命令:
kubeadm init --kubernetes-version=v1.14.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.3.31
执行结果:
[root@k8smaster ~]# kubeadm init --kubernetes-version=v1.14.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.3.31
[init] Using Kubernetes version: v1.14.0
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
.......
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.3.31:6443 --token cytw42.8yq50eg2lry7ldcs \
--discovery-token-ca-cert-hash sha256:243a364fd4e7eb6084ab15b5cdf1f42e01402bd4ff6f2ae8f576e8c731a727f1
[root@k8smaster ~]#
初始化成功后,如下图,将最后两行内容记录下来,这个命令用来加入Worker节点时使用。
kubeadm join 192.168.3.31:6443 --token cytw42.8yq50eg2lry7ldcs \
--discovery-token-ca-cert-hash sha256:243a364fd4e7eb6084ab15b5cdf1f42e01402bd4ff6f2ae8f576e8c731a727f1
如果不记得上述信息了也没关系,在master上执行命令kubeadm token list
,就能看到token参数了,如下图所示:
[root@master ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
cytw42.8yq50eg2lry7ldcs 18h 2019-06-24T16:10:34+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
上一步骤初始化的要求:“To start using your cluster, you need to run the following as a regular user”,需要执行以下命令
上面输出告诉我们还需要做一些工作,执行以下命令配置kubectl,作为普通用户管理集群并在集群上工作
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get pods --all-namespaces命令查看相关状态,可以看到coredns pod处于pending状态,这是因为还没有部署pod网络:
kubectl get pods --all-namespaces
[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-fb8b8dccf-vfwg2 1/1 pending 0 5h38m
kube-system coredns-fb8b8dccf-wf2w5 1/1 pending 0 5h38m
kube-system etcd-master 1/1 Running 0 5h38m
kube-system kube-apiserver-master 1/1 Running 0 5h38m
kube-system kube-controller-manager-master 1/1 Running 0 5h37m
kube-system kube-proxy-mkmkz 1/1 Running 0 5h38m
kube-system kube-proxy-t5nmg 1/1 Running 0 137m
kube-system kube-proxy-tj2h5 1/1 Running 0 153m
kube-system kube-scheduler-master 1/1 Running 0 5h38m
kubectl get cs
命令查看健康状态
[root@master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
You must install a pod network add-on so that your pods can communicate with each other.
您必须安装一个pod网络附加组件,以便您的pod可以彼此通信。
The network must be deployed before any applications. Also, CoreDNS will not start up before a network is installed. kubeadm only supports Container Network Interface (CNI) based networks (and does not support kubenet).
pod网络附加组件是必须安装的,这样pod能够彼此通信,而且网络必须在任何应用程序之前部署。另外,CoreDNS在安装网络之前不会启动。kubeadm只支持基于容器网络接口(CNI)的网络。
支持的Pod网络有JuniperContrail/TungstenFabric、Calico、Canal、Cilium、Flannel、Kube-router、Romana、Wave Net等。
这里我们部署Calico网络,Calico是一个纯三层的方案,其好处是它整合了各种云原生平台(Docker、Mesos 与 OpenStack 等),每个 Kubernetes 节点上通过 Linux Kernel 现有的 L3 forwarding 功能来实现 vRouter 功能。
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
使用 kubectl get pods --all-namespaces
命令查看运行状态:
还没部署好的时候,状态(ContainerCreating),需要等个几分钟,几分钟之后,所有容器就变成了running状态,可以进行下一步了
[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-b8xsh 2/2 Running 0 137m
kube-system calico-node-fbn6b 2/2 Running 0 5h36m
kube-system calico-node-gfm5l 2/2 Running 0 153m
kube-system coredns-fb8b8dccf-vfwg2 1/1 Running 0 5h38m
kube-system coredns-fb8b8dccf-wf2w5 1/1 Running 0 5h38m
kube-system etcd-master 1/1 Running 0 5h38m
kube-system kube-apiserver-master 1/1 Running 0 5h38m
kube-system kube-controller-manager-master 1/1 Running 0 5h37m
kube-system kube-proxy-mkmkz 1/1 Running 0 5h38m
kube-system kube-proxy-t5nmg 1/1 Running 0 137m
kube-system kube-proxy-tj2h5 1/1 Running 0 153m
kube-system kube-scheduler-master 1/1 Running 0 5h38m
在master节点上查看当前集群的所有节点,只有master
kubectl get nodes
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 5h43m v1.14.0
在worker节点上执行
kubeadm join 192.168.3.31:6443 --token cytw42.8yq50eg2lry7ldcs \
--discovery-token-ca-cert-hash sha256:243a364fd4e7eb6084ab15b5cdf1f42e01402bd4ff6f2ae8f576e8c731a727f1
在master节点上检查加入结果
kubectl get nodes
补充:当在worker节点上刚刚执行完加入集群的令牌之后,中间有出现ErrImagePull的状态,等几分钟再看,就已经OK了
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 5h43m v1.14.0
node2 Ready <none> 142m v1.14.0
node3 Ready <none> 158m v1.14.0
等几分钟查看pod状态,就OK了:
kubectl get pods --all-namespaces
[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-b8xsh 2/2 Running 0 145m
kube-system calico-node-fbn6b 2/2 Running 0 5h43m
kube-system calico-node-gfm5l 2/2 Running 0 160m
kube-system coredns-fb8b8dccf-vfwg2 1/1 Running 0 5h46m
kube-system coredns-fb8b8dccf-wf2w5 1/1 Running 0 5h46m
kube-system etcd-master 1/1 Running 0 5h45m
kube-system kube-apiserver-master 1/1 Running 0 5h45m
kube-system kube-controller-manager-master 1/1 Running 0 5h45m
kube-system kube-proxy-mkmkz 1/1 Running 0 5h46m
kube-system kube-proxy-t5nmg 1/1 Running 0 145m
kube-system kube-proxy-tj2h5 1/1 Running 0 160m
kube-system kube-scheduler-master 1/1 Running 0 5h45m
k8s的pod默认不会调度到master节点,如果部署的是单节点的集群,就需要按照下面的方式修改pod的调度策略(https://v1-12.docs.kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network)
执行
kubectl taint nodes --all node-role.kubernetes.io/master-
输出如下,即代表完成
This will remove the
node-role.kubernetes.io/master
taint from any nodes that have it, including the master node, meaning that the scheduler will then be able to schedule pods everywhere
node "test-01" untainted
taint "node-role.kubernetes.io/master:" not found
taint "node-role.kubernetes.io/master:" not found
如果需要从集群中移除node1
这个Node执行下面的命令:
在master节点上执行:
kubectl drain node2 --delete-local-data --force --ignore-daemonsets
kubectl delete node node2
在ndoe2
上执行 …
kubeadm reset
ifconfig cni0 down
ip link delete cni0
....
rm -rf /var/lib/cni/
在master上执行
kubectl delete node node2
关于dashboard的介绍和部署方式可参考:https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#accessing-the-dashboard-ui
这里部署的是1.10.1版本
由于yaml配置文件中指定镜像从google拉取,先下载yaml文件到本地,修改配置从阿里云仓库拉取镜像。
wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
修改拉取镜像地址为阿里云的地址:
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1
kubectl create -f kubernetes-dashboard.yaml
查看Pod 的状态为running说明dashboard已经部署成功
kubectl get pods --all-namespaces
kubectl get pod --namespace=kube-system -o wide | grep dashboard
[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-b8xsh 2/2 Running 0 155m
kube-system calico-node-fbn6b 2/2 Running 0 5h53m
kube-system calico-node-gfm5l 2/2 Running 0 170m
kube-system coredns-fb8b8dccf-vfwg2 1/1 Running 0 5h56m
kube-system coredns-fb8b8dccf-wf2w5 1/1 Running 0 5h56m
kube-system etcd-master 1/1 Running 0 5h55m
kube-system kube-apiserver-master 1/1 Running 0 5h55m
kube-system kube-controller-manager-master 1/1 Running 0 5h54m
kube-system kube-proxy-mkmkz 1/1 Running 0 5h56m
kube-system kube-proxy-t5nmg 1/1 Running 0 155m
kube-system kube-proxy-tj2h5 1/1 Running 0 170m
kube-system kube-scheduler-master 1/1 Running 0 5h55m
kube-system kubernetes-dashboard-5d9599dc98-llmh8 1/1 Running 0 147m
[root@master ~]# kubectl get pod --namespace=kube-system -o wide | grep dashboard
kubernetes-dashboard-5d9599dc98-llmh8 1/1 Running 0 147m 10.244.2.2 node2 <none> <none>
同时,Dashboard 会在 kube-system namespace 中创建自己的 Deployment 和 Service:
kubectl get deployment kubernetes-dashboard --namespace=kube-system
kubectl get service kubernetes-dashboard --namespace=kube-system
如下所示
[root@master ~]# kubectl get deployment kubernetes-dashboard --namespace=kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
kubernetes-dashboard 1/1 1 1 148m
[root@master ~]# kubectl get service kubernetes-dashboard --namespace=kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.104.111.154 <none> 443:30006/TCP 149m
访问dashboard的方式有很多,这里使用的是配置nodeport的方式来访问。
修改文件kubernetes-dashboard.yaml
,将service type和nodeport添加进去,注意k8s只支持30000以上的端口
vim kubernetes-dashboard.yaml
加入type: NodePort
和nodePort: 30006
,如下所示:
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30006
selector:
k8s-app: kubernetes-dashboard
修改后,重新应用配置文件
kubectl apply -f kubernetes-dashboard.yaml
[root@master ~]# kubectl apply -f kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs unchanged
serviceaccount/kubernetes-dashboard unchanged
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged
deployment.apps/kubernetes-dashboard unchanged
service/kubernetes-dashboard unchanged
端口已经变成了30006
[root@master ~]# kubectl get service -n kube-system | grep dashboard
kubernetes-dashboard NodePort 10.104.111.154 <none> 443:30006/TCP 150m
grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"
将生成的kubecfg.p12证书导入到Windows中,直接双击打开,下一步导入即可。
注意:导入完成后需重启浏览器。
Docker教程-01.安装docker-ce-18.06
K8S学习之centos7系统下Kubeadm方式搭建k8s集群
kubeadm搭建kubernetes集群之三:加入node节点
Kubeadm部署Kubernetes1.14.3集群
使用kubeadm部署k8s测试环境(centos7)
Kubeadm部署Kubernetes1.14.1集群