继续部署。
五、部署flannel网络插件
kubernetes要求集群内各节点（包括master节点）能通过pod网段互联互通。flannel使用vxlan技术为各节点创建一个可以互通的pod网络，使用的端口为UDP 8472，需要开放该端口（如公有云aws等）。
flannel第一次启动时，从etcd获取pod网段信息，为本节点分配 一个未使用的/24段地址，然后创建flannel开头的一个接口。
flannel将分配的pod网段信息写入/run/flannel/docker文件，docker后续使用这个文件中的环境变量设置docker0网桥。
5.1 下载二进制文件
下载地址：https://github.com/coreos/flannel/releases
# 下载二进制文件 [k8s@k8s-m1 ~]$ mkdir -p /home/k8s/k8s/flannel-v0.10.0 && cd /home/k8s/k8s [k8s@k8s-m1 k8s]$ wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz [k8s@k8s-m1 k8s]$ tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel-v0.10.0 # 分发到所有节点 [k8s@k8s-m1 k8s]$ source /opt/k8s/bin/environment.sh [k8s@k8s-m1 k8s]$ for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp flannel-v0.10.0/{flanneld,mk-docker-opts.sh} k8s@${node_ip}:/opt/k8s/bin/ ssh k8s@${node_ip} "chmod +x /opt/k8s/bin/*" done
5.2 创建flannel证书和私钥
flannel从etcd集群存取网段分配信息，而etcd集群启用了双向x509证书认证，所以需要为flannel生成证书和私钥。
# 创建证书签名请求 [k8s@k8s-m1 cert]$ cd /opt/k8s/cert [k8s@k8s-m1 cert]$ cat > flanneld-csr.json <<EOF { "CN": "flanneld", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "4Paradigm" } ] } EOF # 生成证书和私钥 [k8s@k8s-m1 cert]$ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \ -ca-key=/etc/kubernetes/cert/ca-key.pem \ -config=/etc/kubernetes/cert/ca-config.json \ -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld [k8s@k8s-m1 cert]$ ls flanneld*pem # 分发证书和私钥到各节点 [k8s@k8s-m1 cert]$ for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld" scp flanneld*.pem k8s@${node_ip}:/etc/flanneld/cert done
5.3 向etcd写入集群pod网段信息
source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/cert/ca.pem \ --cert-file=/etc/flanneld/cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
- flannel当前版本（v0.10.0）不支持etcd v3，所以使用etcd v2 API写入配置key和网段数据；
- 写入的pod网段${CLUSTER_CIDR}必须是/16段地址必须与kube-controller-manager的--cluster-cidr参数值一致；
5.4 创建flanneld的system unit文件
source /opt/k8s/bin/environment.sh export IFACE=eth0 cat > flanneld.service << EOF [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/opt/k8s/bin/flanneld \\ -etcd-cafile=/etc/kubernetes/cert/ca.pem \\ -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\ -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\ -etcd-endpoints=${ETCD_ENDPOINTS} \\ -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\ -iface=${IFACE} ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service EOF
- mk-docker-opts.sh脚本将分配给flanneld的pod子网网段信息写入/run/flannel/docker文件，后续docker启动时使用这个文件中的环境变量配置docker0网桥
- flanneld使用系统默认路由所在的接口与其他节点通信，对于有多个网络接口（如内网和公网）的节点，可以用-iface参数指定通信接口，如上面的enp0s3接口
- flanneld运行时需要root权限
完整的unit文件如下
[k8s@k8s-m1 template]$ cat flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/opt/k8s/bin/flanneld \ -etcd-cafile=/etc/kubernetes/cert/ca.pem \ -etcd-certfile=/etc/flanneld/cert/flanneld.pem \ -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \ -etcd-endpoints=https://192.168.56.20:2379,https://192.168.56.21:2379 \ -etcd-prefix=/kubernetes/network \ -iface= ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service
5.5 分发flanneld systemd unit文件到所有节点
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp flanneld.service root@${node_ip}:/etc/systemd/system/ done
5.6 启动flanneld服务
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld" done
5.7 检查分配给各flanneld的pod网段信息
查看集群pod网段（/16）：
source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/cert/ca.pem \ --cert-file=/etc/flanneld/cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/config
输出：
{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}
查看已经分配的pod子网列表（/24）：
source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/cert/ca.pem \ --cert-file=/etc/flanneld/cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ ls ${FLANNEL_ETCD_PREFIX}/subnets
输出：
/kubernetes/network/subnets/172.30.17.0-24 /kubernetes/network/subnets/172.30.5.0-24 /kubernetes/network/subnets/172.30.75.0-24 /kubernetes/network/subnets/172.30.57.0-24
查看某一pod网段对应节点IP和flannel接口地址：
source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/cert/ca.pem \ --cert-file=/etc/flanneld/cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.17.0-24
输出：
{"PublicIP":"192.168.72.68","BackendType":"vxlan","BackendData":{"VtepMAC":"c2:d6:2a:e4:96:ce"}}
5.8 验证各节点能通过pod网段互通
在各节点部署flannel后，检查是否创建了flannel接口
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet" done
输出：
>>> 192.168.56.20 inet 172.30.57.0/32 scope global flannel.1 >>> 192.168.56.21 inet 172.30.17.0/32 scope global flannel.1 >>> 192.168.56.30 inet 172.30.5.0/32 scope global flannel.1 >>> 192.168.56.31 inet 172.30.75.0/32 scope global flannel.1
在各节点上ping所有flannel接口ip，确保能通：
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh ${node_ip} "ping -c 1 172.30.57.0" ssh ${node_ip} "ping -c 1 172.30.17.0" ssh ${node_ip} "ping -c 1 172.30.5.0" ssh ${node_ip} "ping -c 1 172.30.75.0" done