kubernetes1.14使用kubeadm部署高可用master

1.下载ks8的yum源

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

主节点安装：yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
slave节点：yum install -y kubelet kubeadm --disableexcludes=kubernetes

Package kubelet-1.14.1-0.x86_64
Package kubeadm-1.14.1-0.x86_64
Package kubectl-1.14.1-0.x86_64

2.下载docker

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum  makecache 
yum list docker-ce --showduplicates|sort -r
yum install docker-ce-  （ yum install -y docker-ce-18.06.3.ce-3.el7）

3.初始化配置

3.1关闭selinux

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

3.2开启路由转发

cat < /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl -p /etc/sysctl.d/k8s.conf
如果找不到文件，输入：modprobe br_netfilter，然后在执行即可
输入：sysctl --system 查看应用了k8s.conf

3.3关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

3.4关闭swap分区

swapoff –a
sed -i 's/.swap./#&/' /etc/fstab

3.5.启动docker，设置开机自启动

设置docker的cgroupdriver：
vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}

systemctl start docker && systemctl enable docker

3.6修改kubelet配置文件,忽略某些参数

echo "KUBELET_EXTRA_ARGS="--fail-swap-on=false"" > /etc/sysconfig/kubelet

设置kubelet 开机自启动，并且启动kubelet
systemctl enable kubelet && systemctl start kubelet

3.7下载镜像

查看kubeadm需要使用的镜像：kubeadm config images list
由于无法科学上网，所以我们需要提前把镜像下载好，然后修改包名。或者使用kubeconfig指定阿里云的仓库下载镜像
k8s.gcr.io/kube-apiserver:v1.14.1
k8s.gcr.io/kube-controller-manager:v1.14.1
k8s.gcr.io/kube-scheduler:v1.14.1
k8s.gcr.io/kube-proxy:v1.14.1
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1
下面的docker镜像可以直接下载，然后修改包名，但是比较繁琐，命令如下：
docker pull gcrxio/kube-apiserver:v1.14.1
docker pull gcrxio/kube-controller-manager:v1.14.1
docker pull gcrxio/kube-scheduler:v1.14.1
docker pull gcrxio/pause:3.1
docker pull gcrxio/etcd:3.3.10
docker pull gcrxio/kube-proxy:v1.14.1
docker pull coredns/coredns:1.3.1
修改镜像名称：
docker tag gcrxio/kube-proxy:v1.14.1 k8s.gcr.io/kube-proxy:v1.14.1
docker tag gcrxio/kube-apiserver:v1.14.1 k8s.gcr.io/kube-apiserver:v1.14.1
docker tag gcrxio/kube-controller-manager:v1.14.1 k8s.gcr.io/kube-controller-manager:v1.14.1
docker tag gcrxio/kube-scheduler:v1.14.1 k8s.gcr.io/kube-scheduler:v1.14.1
docker tag gcrxio/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag gcrxio/pause:3.1 k8s.gcr.io/pause:3.1
docker tag coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

这里使用kubeconfig.yaml配置文件来初始化集群，但是必须搭建好api的负载均衡

4.搭建api负载均衡（keepalived+haproxy方案）

yum方式下载keepalived+haproxy

编辑keepalived配置：vim /etc/keepalived/keepalived.conf

在master1上

global_defs {
   router_id lb01
}
vrrp_script check_haproxy{
script "/root/check_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.254/24 dev ens33 label ens33:vip
    }

    track_script {
     check_ haproxy
    }
}

vim /root/check_haproxy.sh

#!/bin/bash
while true
do
[ `netstat -lntup|grep haproxy |wc -l` -ne 1 ]&&systemctl stop keepalived.service
done

master2和master3上修改state为BACKUP 和priority的值为100、50即可

全部master启动keepalived：
systemctl start keepalived.service && systemctl enable keepalived.service

编辑haproxy配置文件（三台机器配置相同）

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     20000
    user        haproxy
    group       haproxy
    daemon
    nbproc  2
    ulimit-n    65535  
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    balance roundrobin
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 20000

frontend k8s-api
    bind 192.168.100.254:8443
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    default_backend k8s-api

backend k8s-api
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server k8s-api-1 192.168.100.145:6443 check
    server k8s-api-2 192.168.100.146:6443 check
    server k8s-api-3 192.168.100.147:6443 check

配置haproxy的路由转发：vim /etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1

然后执行sysctl -p使其生效
启动haproxy：systemctl enable haproxy&&systemctl start haproxy
至此，负载均衡搭建完毕

5.搭建多master

首先我们需要保持服务器时间一致,执行：ntpdate 0.cn.pool.ntp.org 同步时间，或者使用自己的时间服务器，必须让服务器时间保持一致

搭建多主多etcd（此方法的etcd是以pod方式构建，若出现第三个etcd构建失败，我们可以使用外部etcd方案来创建集群，讲解完pod方式跑etcd，我会讲解外部etcd如何创建），我们需要一个配置文件：kubeadm-config.yaml,内容如下

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.1
controlPlaneEndpoint: "192.168.100.254:8443"
networking:
  podSubnet: 10.244.0.0/16
apiServer:
  certSANs:
  - 192.168.100.145
  - 192.168.100.146
  - 192.168.100.147
  - 192.168.100.254

controlPlaneEndpoint指定api的vip地址，apiServer.certSANs指定三个master和一个vip的ip地址

指定配置初始化：kubeadm init --config kubeadm-config.yaml
初始化完毕后出现了一些配置，这些配置要保留下来，我们需要接下来使用

#需要创建和拷贝的文件
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
#主节点需要加入集群时的配置
  kubeadm join 192.168.100.254:8443 --token 6dy980.sxyqhcgj1ll4ljgs \
    --discovery-token-ca-cert-hash sha256:c801532023dfcacdb4f7f9412c7f78f3cefd72f08825c50df1fb733e798cd6d5 \
    --experimental-control-plane 
#从节点加入集群时候需要的配置
    kubeadm join 192.168.100.254:8443 --token 6dy980.sxyqhcgj1ll4ljgs \
    --discovery-token-ca-cert-hash sha256:c801532023dfcacdb4f7f9412c7f78f3cefd72f08825c50df1fb733e798cd6d5 

我们第一台主节点搭建完完毕后，需要拷贝证书到剩下的两台master上。在master2和master3创建目录：mkdir -p /etc/kubernetes/pki/etcd/ ，然后拷贝文件

#################拷贝到master2###########################
  scp /etc/kubernetes/pki/ca.crt 192.168.100.146:/etc/kubernetes/pki/ca.crt
  scp /etc/kubernetes/pki/ca.key 192.168.100.146:/etc/kubernetes/pki/ca.key
  scp /etc/kubernetes/pki/sa.key 192.168.100.146:/etc/kubernetes/pki/sa.key
  scp /etc/kubernetes/pki/sa.pub 192.168.100.146:/etc/kubernetes/pki/sa.pub
  scp /etc/kubernetes/pki/front-proxy-ca.crt 192.168.100.146:/etc/kubernetes/pki/front-proxy-ca.crt
  scp /etc/kubernetes/pki/front-proxy-ca.key 192.168.100.146:/etc/kubernetes/pki/front-proxy-ca.key
  scp /etc/kubernetes/pki/etcd/ca.crt 192.168.100.146:/etc/kubernetes/pki/etcd/ca.crt
  scp /etc/kubernetes/pki/etcd/ca.key 192.168.100.146:/etc/kubernetes/pki/etcd/ca.key
  scp /etc/kubernetes/admin.conf 192.168.100.146:/etc/kubernetes/admin.conf
  scp /etc/kubernetes/admin.conf 192.168.100.146:~/.kube/config
#################拷贝到master3###########################
  scp /etc/kubernetes/pki/ca.crt 192.168.100.147:/etc/kubernetes/pki/ca.crt
  scp /etc/kubernetes/pki/ca.key 192.168.100.147:/etc/kubernetes/pki/ca.key
  scp /etc/kubernetes/pki/sa.key 192.168.100.147:/etc/kubernetes/pki/sa.key
  scp /etc/kubernetes/pki/sa.pub 192.168.100.147:/etc/kubernetes/pki/sa.pub
  scp /etc/kubernetes/pki/front-proxy-ca.crt 192.168.100.147:/etc/kubernetes/pki/front-proxy-ca.crt
  scp /etc/kubernetes/pki/front-proxy-ca.key 192.168.100.147:/etc/kubernetes/pki/front-proxy-ca.key
  scp /etc/kubernetes/pki/etcd/ca.crt 192.168.100.147:/etc/kubernetes/pki/etcd/ca.crt
  scp /etc/kubernetes/pki/etcd/ca.key 192.168.100.147:/etc/kubernetes/pki/etcd/ca.key
  scp /etc/kubernetes/admin.conf 192.168.100.147:/etc/kubernetes/admin.conf
  scp /etc/kubernetes/admin.conf 192.168.100.147:~/.kube/config

拷贝完毕后，在使用上面产生的配置使master2、master3加入主节点:
在master2和master3上创建

  kubeadm join 192.168.100.254:8443 --token 6dy980.sxyqhcgj1ll4ljgs \
    --discovery-token-ca-cert-hash sha256:c801532023dfcacdb4f7f9412c7f78f3cefd72f08825c50df1fb733e798cd6d5 \
    --experimental-control-plane

主节点加入完毕后，安装flannel网络

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

node节点加入集群：

    kubeadm join 192.168.100.254:8443 --token 6dy980.sxyqhcgj1ll4ljgs \
    --discovery-token-ca-cert-hash sha256:c801532023dfcacdb4f7f9412c7f78f3cefd72f08825c50df1fb733e798cd6d5

6.添加自动补全脚本到系统

echo "source <(kubectl completion bash)" >> ~/.bashrc

7使用外部etcd集群方式创建k8s集群

我们首先要在外部创建etcd集群

7.1环境准备

创建etcd目录：mkdir -p /opt/etcd/{cfg,bin,ssl,log}
下载etcd软件包：etcd-v3.3.10-linux-amd64.tar.gz并解压软件包

tar -zxvf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64/
cp etcd etcdctl /opt/etcd/bin/

安装cfssl证书工具：

mkdir  -p /root/k8s/etcd-cert
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
cd /root/k8s/etcd-cert

创建脚本文件：vim etcd-cert.sh

cat > ca-config.json < ca-csr.json < server-csr.json <

注意文件的最后存在空格，否则会出现乱码
bash etcd-cert.sh创建证书

[root@localhost etcd-cert]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd-cert.sh  server.csr  server-csr.json  server-key.pem  server.pem

最后把上面准备的证书移动到/opt/etcd/ssl下并拷贝到所有节点，至此基础环境准备完毕，下面开始安装etcd

[root@localhost etcd-cert]# cd  /root/k8s/etcd-cert
[root@localhost etcd-cert]# cp *.pem /opt/etcd/ssl/
cd /opt/etcd/ssl/  

创建etcd服务脚本：vim /root/script/etcd.sh

#!/bin/bash
# example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

cat <$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

cat </usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

创建生成服务的相关文件

bash /root/script/etcd.sh etcd01 192.168.100.145 etcd02=https://192.168.100.146:2380,etcd03=https://192.168.100.147:2380

将上面生成的所有文件拷贝到其它的 etcd节点

scp -r /opt/etcd/ 192.168.100.145:/opt/
scp -r /opt/etcd/ 192.168.100.146:/opt/
scp -r /usr/lib/systemd/system/etcd.service 192.168.100.145:/usr/lib/systemd/system/
scp -r /usr/lib/systemd/system/etcd.service 192.168.100.146:/usr/lib/systemd/system/

在其它节点上修改 /opt/etcd/cfg/etcd对应的etcd服务
vim /opt/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.146:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.146:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.146:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.146:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.100.145:2380,etcd02=https://192.168.100.146:2380,etcd03=https://192.168.100.147:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

etcd03也按照此方法修改配置
然后启动etcd

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

验证集群启动情况：

etcdctl --ca-file=/opt/etcd/ssl/ca.pem  --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem  --endpoints="https://192.168.100.145:2379,https://192.168.100.146:2379,https://192.168.100.147:2379"  cluster-health

至此，etcd集群搭建完毕，我们可以指定etcd集群来创建k8s集群了
kube-config.yaml配置文件如下

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.1
controlPlaneEndpoint: "192.168.100.254:8443"
networking:
  podSubnet: 10.244.0.0/16
apiServer:
  certSANs:
  - 192.168.100.145
  - 192.168.100.146
  - 192.168.100.147
  - 192.168.100.254
etcd:
  external:
    endpoints:
    - https://192.168.100.145:2379
    - https://192.168.100.146:2379
    - https://192.168.100.147:2379
    caFile: /opt/etcd/ssl/ca.pem
    certFile: /opt/etcd/ssl/server.pem
    keyFile: /opt/etcd/ssl/server-key.pem

此处添加了etcd集群地址以及证书文件位置，提供给api访问etcd，然后我们使用kubeadm init --config kube-config.yaml 来初始化集群，剩下步骤按照上面来进行，一个外部etcd方式安装k8s集群就搭建完毕了。

遇到的问题：

查看api的日志文件发现一直在报错，日志文件在/var/log/pods/kube-system_kube-apiserver-kube-masterxxxxxxxxxxxxxxxx/kube-apiserver/0.log

http: TLS handshake error from 192.168.100.145:41004: remote error: tls: bad certificate\n","stream":"std
err","time":"2019-04-28T03:15:25.700394664Z"}

一直不清楚这个问题是什么原因产生的，猜想是haproxy没有应用证书产生了此问题，希望各位大佬们看完我的文章可以在下面留言帮我解决下，或者新手想入门学习的遇到什么问题都可以在下面留言，我有时间就进行解答