通过kubeadm在Ubuntu1604下构建k8s高可用集群

2019独角兽企业重金招聘Python工程师标准>>> hot3.png

本文参考了https://github.com/cookeem/kubeadm-ha,https://kairen.github.io/2018/07/17/kubernetes/deploy/kubeadm-v1.11-ha/,以及 https://jamesdeng.github.io/2018/08/21/k8s-1.11-%E9%98%BF%E9%87%8C%E4%BA%91%E5%AE%89%E8%A3%85.html,特别是第三个方案,解决了我在阿里云部署K8s高可用集群的大问题。

本地环境基于Haproxy+Keepalived可以轻松实现k8s的HA,但是在阿里云环境中,由于ECS不支持Keepalived,且阿里云内网的SLB,不支持后端ECS实例既作为Real Server又作为客户端向所在的负载均衡实例发送请求。因为返回的数据包只在云服务器内部转发,不经过负载均衡,在后端ECS实例上去访问负载均衡的服务地址是不通的。所以这里采用的是另起两台ECS部署Haproxy,并设置内网SLB的后端为这两台ECS,这里通过docker运行haproxy容器,结合/etc/haproxy/haproxy.cfg配置。另外3台master节点的部署,这里采用一个比较巧妙的设计,设置kubeadm-init.yaml文件中的SLB地址为k8s-master-lb(对应/etc/hosts里面127.0.0.1),这里不能直接设置成SLB的内网IP地址,因为ECS的apiserver访问该SLB是不通的,即使第一台master可以添加,第二台还是会报错。通过k8s-master-lb这个指定本机ip的域名,成功生成第一个master节点的配置和certs之后,可以将/etc/kubernetes/pki+admin.conf,copy至其他两台master上,然后继续添加第二台、第三台,当然master节点上也需要部署haproxy,因为kubeadm-init.yaml文件中k8s-master-lb:8443的这个端口,是直接通过本机的Haproxy负载出来的。具体见下面第二套方案。

补充以下三点说明:

1. 虽然阿里云内网SLB不支持APIserver的四层负载均衡,但是这里采用了七层负载均衡(需要将第一台生成的apiserver的证书放置SLB中,且apiserver开启http访问),后续补充说明,目前环境部署成功。(后续会持续新增说明),且去除了第二套方案的Haproxy,因为采用了阿里云SLB,不需要再配置一个Haproxy。

2. 另外,这里考虑第一套的本地方案中,通过Keepalived可以获得一个可浮动的VIP,保证一台master掉了,可以浮动至另外两台master上,所以不需要再配置一个Haproxy,因为集群中还需要配置Nginx服务,外部访问集群直接访问VIP: Nginx-nodePort即可。

3. 目前本文方案的ETCD都是采用集群容器部署,可以考虑改成独立部署,比如直接通过系统的Systemd保证服务可用。

本文基于的环境:

Linux发行版本: Ubuntu1604

docker版本:17.03.0-ce

Kubernetes版本:V1.11.1

haproxy的docker容器:haproxy:1.7-alpine

本地环境的Keepalived容器:keepalived:1.4.5(阿里云不支持Keepalived,用它的内网SLB服务)

Flanneld网络组件:quay.io/coreos/flannel:v0.10.0-amd64

一、第一种使用场景,本地环境,一个局域网内采用Haproxy+keepalived,结合KubernetesV1.11.1的kubeadm HA实现。

首先规划3台master节点,同时3台node节点

192.168.0.1     master01   
192.168.0.2     master02
192.168.0.3     master03
192.168.0.4     node01
192.168.0.5     node02
192.168.0.6     node03

keepalived的VIP由自己设定,只要是同一网段内未使用的IP即可,本例中我设定为:192.168.0.100

这里介绍下KubernetesV1.11.1中的几个好用的新命令

# 通过yaml文件init出master节点
kubeadm init --config kubeadm-init.yaml
# 通过kubeadm 获取基础组件镜像清单
kubeadm config images list --kubernetes-version=v1.11.1
# 通过kubeadm 拉取基础镜像
kubeadm config images pull --kubernetes-version=v1.11.1

三台Master节点都需要做如下配置

  1.     mkdir -p /etc/kubernetes/manifests
  2.     mkdir -p /etc/haproxy
  3.     生成haproxy.conf配置文件
cat < /etc/haproxy/haproxy.cfg
global
  log 127.0.0.1 local0 err
  maxconn 50000
  uid 99
  gid 99
global
  log 127.0.0.1 local0 err
  maxconn 50000
  uid 99
  gid 99
  #daemon
  nbproc 1
  pidfile haproxy.pid
defaults
  mode http
  log 127.0.0.1 local0 err
  maxconn 50000
  retries 3
  timeout connect 5s
  timeout client 30s
  timeout server 30s
  timeout check 2s

listen stats
  mode http
  bind 0.0.0.0:9090
  log 127.0.0.1 local0 err
  stats refresh 30s
  stats uri     /haproxy-status
  stats realm   Haproxy\ Statistics
  stats auth    admin:admin123
  stats hide-version
  stats admin if TRUE

frontend kube-apiserver-https
   mode tcp
   bind :8443
   default_backend kube-apiserver-backend

backend kube-apiserver-backend
    mode tcp
    balance roundrobin
    server apiserver1 192.168.0.1:6443 weight 3 minconn 100 maxconn 50000 check inter 5000 rise 2 fall 5
    server apiserver2 192.168.0.2:6443 weight 3 minconn 100 maxconn 50000 check inter 5000 rise 2 fall 5
    server apiserver3 192.168.0.3:6443 weight 3 minconn 100 maxconn 50000 check inter 5000 rise 2 fall 5
EOF

    4.     新建/etc/kubernetes/manifests/haproxy.yaml

cat < /etc/kubernetes/manifests/haproxy.yaml
kind: Pod
apiVersion: v1
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  labels:
    component: haproxy
    tier: control-plane
  name: kube-haproxy
  namespace: kube-system
spec:
  hostNetwork: true
  priorityClassName: system-cluster-critical
  containers:
  - name: kube-haproxy
    image: docker.io/haproxy:1.7-alpine
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - name: haproxy-cfg
      readOnly: true
      mountPath: /usr/local/etc/haproxy/haproxy.cfg
  volumes:
  - name: haproxy-cfg
    hostPath:
      path: /etc/haproxy/haproxy.cfg
      type: FileOrCreate
EOF

   5.    新建/etc/kubernetes/manifests/keepalived.yaml

cat < /etc/kubernetes/manifests/keepalived.yaml
kind: Pod
apiVersion: v1
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  labels:
    component: keepalived
    tier: control-plane
  name: kube-keepalived
  namespace: kube-system
spec:
  hostNetwork: true
  priorityClassName: system-cluster-critical
  containers:
  - name: kube-keepalived
    image: docker.io/osixia/keepalived:1.4.5
    env:
    - name: KEEPALIVED_VIRTUAL_IPS
      value: 192.168.0.100
    - name: KEEPALIVED_INTERFACE
      value: eth1
    - name: KEEPALIVED_UNICAST_PEERS
      value: "#PYTHON2BASH:['192.168.0.1', '192.168.0.2', '192.168.0.3']"
    - name: KEEPALIVED_PASSWORD
      value: docker
    - name: KEEPALIVED_PRIORITY
      value: "100"
    - name: KEEPALIVED_ROUTER_ID
      value: "51"
    resources:
      requests:
        cpu: 100m
    securityContext:
      privileged: true
      capabilities:
        add:
        - NET_ADMIN
EOF
  • KEEPALIVED_VIRTUAL_IPS:Keepalived 提供的 VIPs。
  • KEEPALIVED_INTERFACE:VIPs 绑定的网卡。
  • KEEPALIVED_UNICAST_PEERS:其他 Keepalived 节点的单点传播 IP。
  • KEEPALIVED_PASSWORD: Keepalived auth_type 的 Password。
  • KEEPALIVED_PRIORITY:指定了备份发生時,接手的介面之順序,数字越大,优先级越高。这边master01为150,其余为100

以上的haproxy.yaml和keepalived.yaml文件直接放在/etc/kubernetes/manifests下,集群启动后,会自动加载这两个yaml文件,并部署Pod。

依次通过kubeadm部署三台Master节点,在每台master节点上执行(注意Kubernetes Control Plane新特性)

    1.   在master01节点上执行

cat < kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
apiServerCertSANs:
- "192.168.0.100"
api:
  controlPlaneEndpoint: "192.168.0.100:8443"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.0.1:2379"
      advertise-client-urls: "https://192.168.0.1:2379"
      listen-peer-urls: "https://192.168.0.1:2380"
      initial-advertise-peer-urls: "https://192.168.0.1:2380"
      initial-cluster: "u212=https://192.168.0.1:2380"
    serverCertSANs:
      - master01
      - 192.168.0.1
    peerCertSANs:
      - master01
      - 192.168.0.1
networking:
  podSubnet: "172.168.0.0/16"
EOF

通过执行kubeadm init --config kubeadm-init.yaml,成功后记录生成的token

kubeadm join 192.168.0.100:8443 --token rolz18.bm9jxwlmzagrfslg --discovery-token-ca-cert-hash sha256:*

之后,将certs文件和admin.conf文件copy至其他master节点上

export DIR=/etc/kubernetes/
for NODE in master02 master03; do
    echo "------ ${NODE} ------"
    ssh ${NODE} "mkdir -p ${DIR}/pki/etcd"
    scp ${DIR}/pki/ca.crt ${NODE}:${DIR}/pki/ca.crt
    scp ${DIR}/pki/ca.key ${NODE}:${DIR}/pki/ca.key
    scp ${DIR}/pki/sa.key ${NODE}:${DIR}/pki/sa.key
    scp ${DIR}/pki/sa.pub ${NODE}:${DIR}/pki/sa.pub
    scp ${DIR}/pki/front-proxy-ca.crt ${NODE}:${DIR}/pki/front-proxy-ca.crt
    scp ${DIR}/pki/front-proxy-ca.key ${NODE}:${DIR}/pki/front-proxy-ca.key
    scp ${DIR}/pki/etcd/ca.crt ${NODE}:${DIR}/pki/etcd/ca.crt
    scp ${DIR}/pki/etcd/ca.key ${NODE}:${DIR}/pki/etcd/ca.key
    scp ${DIR}/admin.conf ${NODE}:${DIR}/admin.conf
  done

     2.   在master02节点上执行

cat < kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
apiServerCertSANs:
- "192.168.0.100"
api:
  controlPlaneEndpoint: "192.168.0.100:8443"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.0.2:2379"
      advertise-client-urls: "https://192.168.0.2:2379"
      listen-peer-urls: "https://192.168.0.2:2380"
      initial-advertise-peer-urls: "https://192.168.0.2:2380"
      initial-cluster: "master01=https://192.168.0.2:2380,master02=https://192.168.0.2:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - master02
      - 192.168.0.2
    peerCertSANs:
      - master02
      - 192.168.0.2
networking:
  podSubnet: "172.168.0.0/16"
EOF

然后通过kubeadm phase来启动master02的kubelet,注意要在以上kubeadm-init.yaml同一目录下运行

 kubeadm alpha phase certs all --config kubeadm-init.yaml
 kubeadm alpha phase kubelet config write-to-disk --config kubeadm-init.yaml
 kubeadm alpha phase kubelet write-env-file --config kubeadm-init.yaml
 kubeadm alpha phase kubeconfig kubelet --config kubeadm-init.yaml
 systemctl start kubelet

通过etcdctl命令,将master02的etcd添加到master01的etcd容器中

 export ETCD1_NAME=master01; export ETCD1_IP=192.168.0.1
 export ETCD2_NAME=master02; export ETCD2_IP=192.168.0.2
 export KUBECONFIG=/etc/kubernetes/admin.conf
 kubectl exec -n kube-system etcd-${ETCD1_NAME} -- etcdctl \
    --ca-file /etc/kubernetes/pki/etcd/ca.crt \
    --cert-file /etc/kubernetes/pki/etcd/peer.crt \
    --key-file /etc/kubernetes/pki/etcd/peer.key \
    --endpoints=https://${ETCD1_IP}:2379 member add ${ETCD2_NAME} https://${ETCD2_IP}:2380

kubeadm alpha phase etcd local --config kubeadm-init.yaml

成功后,可以通过如下命令查看etcd集群的member list和目前的isLeader

export ETCD1_NAME=master02; export ETCD1_IP=192.168.0.2
kubectl exec -n kube-system etcd-${ETCD1_NAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt  --cert-file /etc/kubernetes/pki/etcd/peer.crt  --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${ETCD1_IP}:2379 member list

最后,执行以下命令来部署control plane

kubeadm alpha phase kubeconfig all --config kubeadm-init.yaml
kubeadm alpha phase controlplane all --config kubeadm-init.yaml
kubeadm alpha phase mark-master --config kubeadm-init.yaml

经过一段时间后,执行以下命令来使用kubeconfig

mkdir -p $HOME/.kube
cp -rp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config 

    3.   在master03节点上执行

cat < kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
apiServerCertSANs:
- "192.168.0.100"
api:
  controlPlaneEndpoint: "192.168.0.100:8443"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.0.3:2379"
      advertise-client-urls: "https://192.168.0.3:2379"
      listen-peer-urls: "https://192.168.0.3:2380"
      initial-advertise-peer-urls: "https://192.168.0.3:2380"
      initial-cluster: "master01=https://192.168.0.1:2380,master02=https://192.168.0.2:2380,master03=https://192.168.0.3:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - master03
      - 192.168.0.3
    peerCertSANs:
      - master03
      - 192.168.0.3
networking:
  podSubnet: "172.168.0.0/16"
EOF

然后通过kubeadm phase来启动master03的kubelet,注意要在以上kubeadm-init.yaml同一目录下运行

 kubeadm alpha phase certs all --config kubeadm-init.yaml
 kubeadm alpha phase kubelet config write-to-disk --config kubeadm-init.yaml
 kubeadm alpha phase kubelet write-env-file --config kubeadm-init.yaml
 kubeadm alpha phase kubeconfig kubelet --config kubeadm-init.yaml
 systemctl start kubelet

通过etcdctl命令,将master03的etcd添加到master01的etcd容器中

 export ETCD1_NAME=master01; export ETCD1_IP=192.168.0.1
 export ETCD2_NAME=master03; export ETCD2_IP=192.168.0.3
 export KUBECONFIG=/etc/kubernetes/admin.conf
 kubectl exec -n kube-system etcd-${ETCD1_NAME} -- etcdctl \
    --ca-file /etc/kubernetes/pki/etcd/ca.crt \
    --cert-file /etc/kubernetes/pki/etcd/peer.crt \
    --key-file /etc/kubernetes/pki/etcd/peer.key \
    --endpoints=https://${ETCD1_IP}:2379 member add ${ETCD2_NAME} https://${ETCD2_IP}:2380

kubeadm alpha phase etcd local --config kubeadm-init.yaml

成功后,可以通过如下命令查看etcd集群的member list和目前的isLeader

export ETCD1_NAME=master03; export ETCD1_IP=192.168.0.3
kubectl exec -n kube-system etcd-${ETCD1_NAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt  --cert-file /etc/kubernetes/pki/etcd/peer.crt  --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${ETCD1_IP}:2379 member list

最后,执行以下命令来部署control plane

kubeadm alpha phase kubeconfig all --config kubeadm-init.yaml
kubeadm alpha phase controlplane all --config kubeadm-init.yaml
kubeadm alpha phase mark-master --config kubeadm-init.yaml

经过一段时间后,执行以下命令来使用kubeconfig

mkdir -p $HOME/.kube
cp -rp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config 

至此,三台master节点的配置完成,但是都是NotReady状态,需要安装网络插件,这里选择flanneld

#直接下载官网的kube-flannel.yaml文件,不过需要制定网卡,可以制定多个网卡,因为发现新安装的ubuntu1604会有eth0,eno1,enp3s0,eth1等各个网卡。
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
#可以先curl -O https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
然后找到如下:
image: quay.io/coreos/flannel:v0.10.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0
- --iface=eth1
- --iface=eno1

Node节点的加入,根据master01 init之后的join命令

  • export KUBE_VERSION="1.11.1"
      apt-get update && apt-get install -y kubelet=${KUBE_VERSION}-00 kubeadm=${KUBE_VERSION}-00
  • mkdir -p /etc/kubernetes/manifests/
  • 下载crictl-v1.11.1-linux-amd64.tar.gz,解决一个报错,解压后将mv crictl /usr/bin/
  • kubeadm join 192.168.0.100:8443 --token *.*--discovery-token-ca-cert-hash sha256:*

至此3台master节点的高可用,etcd集群的高可用配置完成。

验证也比较简单

     1. 直接将poweroff master01,然后在查看k8s集群是否正常,etcd的isLeader属性是否正常,且自动切换。

     2. 或者通过停止服务,systemctl stop kubelet,docker stop myhaproxy,此时master01会变成NotReady状态,且etcd的isLeader已经切换。

小技巧

如果忘记初始master节点时的node节点加入集群命令时,可以通过如下命令找回,只是这个token可能24小时已经过期,可以通过kubeadm token create --ttl 0生成永不过期的token,这里也可以在kubeadm-init.yaml文件中,添加ttl参数,在kubeadm init第一个master时生成一个永不过期的token

kubeadm token create --print-join-command
kubeadm token create --ttl 0

二、第二种使用场景,阿里云ECS服务器Ubuntu1604采用Haproxy+阿里云内网SLB,结合KubernetesV1.11.1的kubeadm HA实现。

我参考的原文作者中说要加上region,我这边没有成功,而是直接用的/etc/hostname,且主机名用的小写,阿里云限制不能修改,否则会出现意想不到的问题。

我们的云上环境有三台master,三台Node,且提供一台用于安装Haproxy服务,避免内网SLB和ECS服务器直接出现请求回环,阿里云SLB的流量只能在ECS直接流转,ECS中的apiserver无法访问到内网SLB的IP。

master01  10.252.1.100  iz1111dddd01
master02  10.252.2.100  iz2222dddd02
master03  10.51.3.66    iz3333dddd03
 
node01   10.21.1.100    iz3333dddd04
node02   10.21.2.100    iz3333dddd05
node03   10.21.3.100    iz3333dddd06

node0x   10.100.3.100   iz1234567800

内网SLB IP  100.150.150.100

配置时,经常出现出错的情况,可能需要reset并清理生成的文件,以免下次init时出错或干扰

#常用命令
kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
rm -rf /etc/kubernetes/
systemctl stop kubelet;
docker rm -f -v $(docker ps -q);
find /var/lib/kubelet | xargs -n 1 findmnt -n -t tmpfs -o TARGET -T | uniq | xargs -r umount -v;
rm -rf /var/lib/kubelet /var/lib/etcd;

关闭ECS上的防火墙,配置转发规则

cat <>  /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p

加载ipvs相关内核模块

#加载ipvs模块
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4

# to check loaded modules, use
lsmod | grep -e ipvs -e nf_conntrack_ipv4
# or
cut -f1 -d " "  /proc/modules | grep -e ip_vs -e nf_conntrack_ipv4
lsmod | grep ip_vs


或者也可以通过修改configmap中的mode: "ipvs" 
kubectl edit configmap kube-proxy -n kube-system

iptables防止FORWARD链被drop

iptables -P FORWARD ACCEPT
sed -i "/ExecStart=/a\ExecStartPost=/sbin/iptables -P FORWARD ACCEPT" /lib/systemd/system/docker.service
systemctl daemon-reload
systemctl enable docker 
systemctl restart docker

依次通过kubeadm部署三台Master节点,在每台master节点上执行(注意Kubernetes Control Plane新特性)

    1.   在master01节点上执行

#生成配置文件
CP0_IP="10.252.1.100"
CP0_HOSTNAME="iz1111dddd01"
cat >kubeadm-init.yaml<

通过执行kubeadm init --config kubeadm-init.yaml,成功后记录生成的token

kubeadm join k8s-master-lb:8443 --token rolz18.bm9jxwlmzagrfslg --discovery-token-ca-cert-hash sha256:*

之后,将certs文件和admin.conf文件copy至其他master节点上

export DIR=/etc/kubernetes/
for NODE in master02 master03; do
    echo "------ ${NODE} ------"
    ssh ${NODE} "mkdir -p ${DIR}/pki/etcd"
    scp ${DIR}/pki/ca.crt ${NODE}:${DIR}/pki/ca.crt
    scp ${DIR}/pki/ca.key ${NODE}:${DIR}/pki/ca.key
    scp ${DIR}/pki/sa.key ${NODE}:${DIR}/pki/sa.key
    scp ${DIR}/pki/sa.pub ${NODE}:${DIR}/pki/sa.pub
    scp ${DIR}/pki/front-proxy-ca.crt ${NODE}:${DIR}/pki/front-proxy-ca.crt
    scp ${DIR}/pki/front-proxy-ca.key ${NODE}:${DIR}/pki/front-proxy-ca.key
    scp ${DIR}/pki/etcd/ca.crt ${NODE}:${DIR}/pki/etcd/ca.crt
    scp ${DIR}/pki/etcd/ca.key ${NODE}:${DIR}/pki/etcd/ca.key
    scp ${DIR}/admin.conf ${NODE}:${DIR}/admin.conf
  done

     2.   在master02节点上执行

CP0_IP="10.252.1.100"
CP0_HOSTNAME="iz1111dddd01"
CP1_IP="10.252.2.100"
CP1_HOSTNAME="iz2222dddd02"
cat > kubeadm-init.yaml<

然后通过kubeadm phase来启动master02的kubelet,注意要在以上kubeadm-init.yaml同一目录下运行

 kubeadm alpha phase certs all --config  kubeadm-init.yaml
 kubeadm alpha phase kubelet config write-to-disk --config  kubeadm-init.yaml
 kubeadm alpha phase kubelet write-env-file --config  kubeadm-init.yaml
 kubeadm alpha phase kubeconfig kubelet --config  kubeadm-init.yaml
 systemctl start kubelet

通过etcdctl命令,将master02的etcd添加到master01的etcd容器中

CP0_IP="10.252.1.100"
CP0_HOSTNAME="iz1111dddd01"
CP1_IP="10.252.2.100"
CP1_HOSTNAME="iz2222dddd02"
KUBECONFIG=/etc/kubernetes/admin.conf kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380

kubeadm alpha phase etcd local --config  kubeadm-init.yaml

成功后,可以通过如下命令查看etcd集群的member list和目前的isLeader

export ETCD1_NAME=iz2222dddd02; export ETCD1_IP=10.252.2.100
kubectl exec -n kube-system etcd-${ETCD1_NAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt  --cert-file /etc/kubernetes/pki/etcd/peer.crt  --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${ETCD1_IP}:2379 member list

最后,执行以下命令来部署control plane

kubeadm alpha phase kubeconfig all --config  kubeadm-init.yaml
kubeadm alpha phase controlplane all --config  kubeadm-init.yaml
kubeadm alpha phase mark-master --config  kubeadm-init.yaml

经过一段时间后,执行以下命令来使用kubeconfig

mkdir -p $HOME/.kube
cp -rp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config 

    3.   在master03节点上执行

CP0_IP="10.252.1.100"
CP0_HOSTNAME="iz1111dddd01"
CP1_IP="10.252.2.100"
CP1_HOSTNAME="iz2222dddd02"
CP2_IP="10.51.3.66"
CP2_HOSTNAME="iz3333dddd03"
cat > kubeadm-init.yaml<

然后通过kubeadm phase来启动master03的kubelet,注意要在以上kubeadm-init.yaml同一目录下运行

 kubeadm alpha phase certs all --config  kubeadm-init.yaml
 kubeadm alpha phase kubelet config write-to-disk --config  kubeadm-init.yaml
 kubeadm alpha phase kubelet write-env-file --config  kubeadm-init.yaml
 kubeadm alpha phase kubeconfig kubelet --config  kubeadm-init.yaml
 systemctl start kubelet

通过etcdctl命令,将master03的etcd添加到master01的etcd容器中

CP0_IP="10.252.1.100"
CP0_HOSTNAME="iz1111dddd01"
CP1_IP="10.51.3.66"
CP1_HOSTNAME="iz3333dddd03"
KUBECONFIG=/etc/kubernetes/admin.conf kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380

kubeadm alpha phase etcd local --config kubeadm-init.yaml

成功后,可以通过如下命令查看etcd集群的member list和目前的isLeader

export ETCD1_NAME=iz3333dddd03; export ETCD1_IP=10.51.3.66
kubectl exec -n kube-system etcd-${ETCD1_NAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt  --cert-file /etc/kubernetes/pki/etcd/peer.crt  --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${ETCD1_IP}:2379 member list

最后,执行以下命令来部署control plane

kubeadm alpha phase kubeconfig all --config  kubeadm-init.yaml
kubeadm alpha phase controlplane all --config  kubeadm-init.yaml
kubeadm alpha phase mark-master --config  kubeadm-init.yaml

在三台节点上都执行

rm -rf $HOME/.kube
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

至此,三台master节点的配置完成,但是都是NotReady状态,需要安装网络插件,这里选择flanneld

#直接下载官网的kube-flannel.yaml文件,不过需要制定网卡,可以制定多个网卡,因为发现新安装的ubuntu1604会有eth0,eno1,enp3s0,eth1等各个网卡。
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
#可以先curl -O https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
然后找到如下:
image: quay.io/coreos/flannel:v0.10.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0
- --iface=eth1
- --iface=eno1

这里还需要配置一台专门作为Haproxy的ECS,且在以上三台Master上,也要docker run一个Haproxy

mkdir /etc/haproxy
docker pull haproxy:1.7.8-alpine
cat >/etc/haproxy/haproxy.cfg<

在阿里云SLB的配置页面,将10.100.3.100的8443作为其后端ECS服务器即可。

Node节点的加入,根据master01 init之后的join命令

  • export KUBE_VERSION="1.11.1"
      apt-get update && apt-get install -y kubelet=${KUBE_VERSION}-00 kubeadm=${KUBE_VERSION}-00
  • mkdir -p /etc/kubernetes/manifests/
  • 下载crictl-v1.11.1-linux-amd64.tar.gz,解决一个报错,解压后将mv crictl /usr/bin/
  • kubeadm join k8s-master-lb:8443 --token *.*--discovery-token-ca-cert-hash sha256:* (注意这里需要在/etc/hosts配置k8s-master-lb对应的SLB的IP: 100.115.129.111)

至此3台master节点的高可用,etcd集群的高可用配置完成。

验证也比较简单

     1. 直接将poweroff master01,然后在查看k8s集群是否正常,etcd的isLeader属性是否正常,且自动切换。

     2. 或者通过停止服务,systemctl stop kubelet,docker stop myhaproxy,此时master01会变成NotReady状态,且etcd的isLeader已经切换。

小技巧

如果忘记初始master节点时的node节点加入集群命令时,可以通过如下命令找回,只是这个token可能24小时已经过期,可以通过kubeadm token create --ttl 0生成永不过期的token

kubeadm token create --print-join-command
kubeadm token create --ttl 0

 

至此,大功告成!

 

附录: 

通过git clone [email protected]:cookeem/kubeadm-ha.git之后,cd kubeadm-ha目录

主要修改create-config.sh文件,根据以上的Ip和hostname

# 这个ip是keeplived的虚拟的Ip,可以是任何的ip,这里设置成一个网段的ip
export K8SHA_VIP=192.168.0.222

# master01 ip address
export K8SHA_IP1=192.168.0.1

# master02 ip address
export K8SHA_IP2=192.168.0.2

# master03 ip address
export K8SHA_IP3=192.168.0.3

# master keepalived virtual ip hostname
export K8SHA_VHOST=k8s-master-lb

# master01 hostname
export K8SHA_HOST1=master01

# master02 hostname
export K8SHA_HOST2=maste02

# master03 hostname
export K8SHA_HOST3=maste03

# master01 network interface name
export K8SHA_NETINF1=eth0

# master02 network interface name
export K8SHA_NETINF2=eth0

# master03 network interface name
export K8SHA_NETINF3=eth0

# keepalived auth_pass config
export K8SHA_KEEPALIVED_AUTH=412f7dc3bfed32194d1600c483e10ad1d

# 这个没有用到,就设置成网关ip
export K8SHA_CALICO_REACHABLE_IP=192.168.0.1

# kubernetes CIDR pod subnet, if CIDR pod subnet is "172.168.0.0/16" please set to "172.168.0.0"
# 这个参数后续的flannel中也需要对应修改,默认的是10.244.0.0/16
export K8SHA_CIDR=172.168.0.0

执行./create-config.sh

就会生成一个config目录,里面对应是master三台节点的配置,包含一个kubeadm-config.yaml文件,当然这里不用这个方法,只是作为一种生成yaml配置的方式,可以使用。

转载于:https://my.oschina.net/beyondken/blog/1935402

你可能感兴趣的:(通过kubeadm在Ubuntu1604下构建k8s高可用集群)