kubernetes安装部署-day04

五、部署flannel

在master各节点和node各节点部署flannel服务，下载地址：https://github.com/coreos/flannel/releases

5.1、证书的创建

在master创建证书

[root@k8s-master1 ssl]# mkdir flanneld
[root@k8s-master1 flanneld]# vim flanneld-csr.json
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成证书：cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

把证书COPY到node节点上

cp  flanneld*.pem /opt/kubernetes/ssl/
scp flanneld*.pem 192.168.100.108:/opt/kubernetes/ssl/
scp flanneld*.pem 192.168.100.109:/opt/kubernetes/ssl/

5.2安装flannel

5.2.1配置flannel配置文件

[root@k8s-master1 bin]# vim /opt/kubernetes/cfg/flannel
FLANNEL_ETCD="-etcd-endpoints=https://192.168.100.105:2379,https://192.168.100.106:2379,https://192.168.100.107:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"

把配置文件COPY到master2和各个node节点上
解压flannel-v0.10.0-linux-amd64.tar.gz文件，把可执行程序COPY到各个node节点

[root@k8s-master1 k8s]# tar -xvf flannel-v0.10.0-linux-amd64.tar.gz
root@k8s-master1 k8s]# scp flanneld 192.168.100.108:/opt/kubernetes/bin/
flanneld                                                                                                                           100%   35MB  81.6MB/s   00:00    
[root@k8s-master1 k8s]# scp flanneld 192.168.100.109:/opt/kubernetes/bin/
flanneld                                                                                                                           100%   35MB  83.7MB/s   00:00    
[root@k8s-master1 k8s]# cp flanneld /opt/kubernetes/bin/

5.2.2设置Flannel系统服务

root@k8s-master1:/usr/local/src/ssl/flannel# vim /lib/systemd/system/flannel.service

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
Before=docker.service

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/flannel
ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh
ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker

Type=notify

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

复制对应的脚本到/opt/kubernetes/bin/目录下

[root@k8s-master1 flanneld]# cd /usr/local/src/kubernetes/cluster/centos/node/bin/
[root@k8s-master1 bin]# ll
total 8
-rwxr-xr-x 1 root root 2598 Jul 18  2018 mk-docker-opts.sh
-rwxr-xr-x 1 root root  858 Jul 18  2018 remove-docker0.sh
[root@k8s-master1 bin]# cp *.sh /opt/kubernetes/bin/
[root@k8s-master1 bin]# 

5.2.3Flannel CNI集成

[root@k8s-master1 src]# mkdir /opt/kubernetes/bin/cni
[root@k8s-master1 src]# tar zxf cni-plugins-amd64-v0.7.1.tgz -C /opt/kubernetes/bin/cni
root@k8s-master1:/usr/local/src#  scp -r  /opt/kubernetes/bin/cni 192.168.100.108:/opt/kubernetes/bin/
root@k8s-master1:/usr/local/src#  scp -r  /opt/kubernetes/bin/cni 192.168.100.109:/opt/kubernetes/bin/

在etcd创建网络：提前将证书复制到etcd或在node节点操作：

root@k8s-master1:/usr/local/src/ssl/flannel#  
/opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file    /opt/kubernetes/ssl/flanneld-key.pem  --no-sync -C https://192.168.100.105:2379,https://192.168.100.106:2379,https://192.168.100.107:2379  mk /kubernetes/network/config  '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' 
{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}
验证网段：
[root@k8s-etcd1 ~]# /opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem     --no-sync -C  https://192.168.100.107:2379 get  /kubernetes/network/config   #以下是返回值
{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}

5.2.4、node节点启动flannel服务

node1:

root@k8s-node1:/usr/local/src# systemctl daemon-reload && systemctl enable flannel && chmod +x /opt/kubernetes/bin/* &&  systemctl start flannel &&  systemctl status  flannel
● flannel.service - Flanneld overlay address etcd agent
   Loaded: loaded (/lib/systemd/system/flannel.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-05-31 15:08:31 CST; 12ms ago
  Process: 767 ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker (code=exited, status=0/SUCCESS)
  Process: 722 ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh (code=exited, status=0/SUCCESS)
 Main PID: 734 (flanneld)
    Tasks: 9 (limit: 2323)
   CGroup: /system.slice/flannel.service
           ├─734 /opt/kubernetes/bin/flanneld -etcd-endpoints=http://10.172.160.250:2379,http://10.51.50.234:2379,http://10.170.185.97:2379 -etcd-prefix=/kubernetes/
           └─779 /opt/kubernetes/bin/flanneld -etcd-endpoints=http://10.172.160.250:2379,http://10.51.50.234:2379,http://10.170.185.97:2379 -etcd-prefix=/kubernetes/

May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.544136     734 main.go:300] Wrote subnet file to /run/flannel/subnet.env
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.544156     734 main.go:304] Running backend.
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.556014     734 vxlan_network.go:60] watching for new subnet leases
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.558342     734 main.go:396] Waiting for 22h59m59.956551757s to renew lease
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.581600     734 iptables.go:115] Some iptables rules are missing; deleting and recreating rules
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.589265     734 iptables.go:137] Deleting iptables rule: -s 10.2.0.0/16 -j ACCEPT
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.597298     734 iptables.go:137] Deleting iptables rule: -d 10.2.0.0/16 -j ACCEPT
May 31 15:08:31 k8s-node1.example.com systemd[1]: Started Flanneld overlay address etcd agent.
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.611548     734 iptables.go:125] Adding iptables rule: -s 10.2.0.0/16 -j ACCEPT
May 31 15:08:31 k8s-node1.example.com flanneld[734]: I0531 15:08:31.630543     734 iptables.go:125] Adding iptables rule: -d 10.2.0.0/16 -j ACCEPT
lines 1-21/21 (END)

查看网络

root@k8s-node1:/usr/local/src# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:00:7a:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.51.67.209/21 brd 10.51.71.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe00:7ae0/64 scope link 
       valid_lft forever preferred_lft forever
4: kube-ipvs0:  mtu 1500 qdisc noop state DOWN group default 
    link/ether 8e:61:17:89:52:98 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/32 brd 10.1.0.1 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
5: flannel.1:  mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 9e:4f:49:62:19:81 brd ff:ff:ff:ff:ff:ff
    inet 10.2.36.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::9c4f:49ff:fe62:1981/64 scope link 
       valid_lft forever preferred_lft forever
root@k8s-node1:/usr/local/src# 

5.2.3、配置docker服务使用Flannel

[root@k8s-node1 ~]# vim /lib/systemd/system/docker.service
[Unit] #在Unit下面修改After和增加Requires
After=network-online.target firewalld.service flannel.service
Wants=network-online.target
Requires=flannel.service

[Service] #增加EnvironmentFile=-/run/flannel/docker
Type=notify
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_OPTS

重启docker服务

root@k8s-node1:/usr/local/src# systemctl daemon-reload
root@k8s-node1:/usr/local/src# systemctl restart docker 
root@k8s-node1:/usr/local/src# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:00:7a:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.51.67.209/21 brd 10.51.71.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe00:7ae0/64 scope link 
       valid_lft forever preferred_lft forever
4: kube-ipvs0:  mtu 1500 qdisc noop state DOWN group default 
    link/ether 8e:61:17:89:52:98 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/32 brd 10.1.0.1 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
5: flannel.1:  mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 9e:4f:49:62:19:81 brd ff:ff:ff:ff:ff:ff
    inet 10.2.36.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::9c4f:49ff:fe62:1981/64 scope link 
       valid_lft forever preferred_lft forever
6: docker0:  mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:af:9e:f2:09 brd ff:ff:ff:ff:ff:ff
    inet 10.2.36.1/24 brd 10.2.36.255 scope global docker0
       valid_lft forever preferred_lft forever
root@k8s-node1:/usr/local/src# 

查看到docker已经使用flannel的地址
把docker启动脚本复制到其他的node节点。