一步一步二进制方式安装kubernetes集群
文章目录
- 准备工作
- 集群规划
- 一、创建ca证书
- 1、安装 cfssl 工具
- 2、创建ca证书的json配置文件。
- 3、创建ca证书签名请求文件 ca-csr.json
- 4、生成 CA 证书和私钥
- 5、在所有节点创建/etc/kubernetes/ssl目录，用来存放各种证书文件
- 6、将ca相关文件复制到所有节点的/etc/kubernetes/ssl目录下
- 二、搭建高可用etcd集群
- 1、 下载安装etcd二进制文件
- 2、创建etcd证书
- 3 、创建etcd的systemd配置文件
- 4、将systemd配置分发到其他etcd节点上
- 5、启动etcd
- 6、 验证是否搭建成功
- 三、部署kubectl命令行工具
- 1、下载二进制文件
- 2、创建 kubectl的证书和私钥
- 3、生成kubectl 的kubeconfig配置文件
- 4、将 kubectl 二进制文件和/root/.kube/config文件，分发到所有需要使用kubectl 命令的节点
- 四、部署Flannel 网络
- 1、 创建flannel证书
- 2、 向etcd 写入集群Pod 网段信息
- 3、下载安装flanneld
- 4、创建flannel的systemd配置文件
- 5、启动flanneld
- 6、检查flannel网络是否正常
- 五 、安装haproxy和keepalived
- 六、部署master节点
- 1、下载二进制文件
- 2、创建kubernetes 证书
- 3、配置和启动kube-apiserver
- 4、配置和启动kube-controller-manager
- 5、 配置和启动kube-scheduler
- 6、验证 master 节点各个组件的状态
- 七、部署node节点
- 2、安装Docker
- 3、安装和配置kubelet
- 4、安装 kube-proxy
- 八、验证测试
准备工作
1、操作系统：Centos7。
2、硬件配置：至少2核2GB内存 ,1核也可以，但是会报warning，1G可能会内存不足。
3、网络，ssh，dns，yum源
4、关闭防火墙，关闭SeLinux，关闭Swap
5、安装Docker:18:06
6、内核参数
$ cat > /etc/sysctl.d/kubernetes.conf <集群规划
- centos7.3
- kubernetes1.12.1
- flannel0.9
- etcd3.1.5
- 用keepalived和haproxy对apiserver做负载均衡，VIP是192.168.255.140，端口是8443
- 集群机器：3台master，2台node。
| 主机名 | 角色 | ip | 组件 |
|---|---|---|---|
| master1-3.hanli.com | etcd+master | 192.168.255.131-133 | etcd ，kube-apiserver，kube-scheduler，kube-controller-manager，Flanneld，kubectl |
| slave1-2.hanli.com | node | 192.168.255.121-122 | kubelet，kube-proxy ， Flanneld |
一、创建ca证书
CA 证书是集群所有节点共享的，只需要创建一个 CA 证书，后续创建的所有证书都由它签名。
在master1上操作
1、安装 cfssl 工具
[root@master1] ~$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[root@master1] ~$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[root@master1] ~$ chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
[root@master1] ~$ mv cfssl_linux-amd64 /usr/local/bin/cfssl
[root@master1] ~$ mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
2、创建ca证书的json配置文件。
用于定义使用场景 profile 和一些具体参数 (usage，过期时间、服务端认证、客户端认证、加密等)
[root@master1] ~$ vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
- profile：其中包含多类证书比如server，client，peer，可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
- signing：表示ca证书可用于签名其它证书，生成的 ca.pem 证书中 CA=TRUE；
- server auth：表示 client 可以用ca证书对 server 提供的证书进行验证；
- client auth：表示 server 可以用ca证书对 client 提供的证书进行验证；
3、创建ca证书签名请求文件 ca-csr.json
[root@master1] ~$ vim ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
-
CN：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)，浏览器使用该字段验证网站是否合法；
-
O：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；
4、生成 CA 证书和私钥
[root@master1] ~$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2019/06/03 13:37:58 [INFO] generating a new CA key and certificate from CSR
2019/06/03 13:37:58 [INFO] generate received request
2019/06/03 13:37:58 [INFO] received CSR
2019/06/03 13:37:58 [INFO] generating key: rsa-2048
2019/06/03 13:37:58 [INFO] encoded CSR
2019/06/03 13:37:58 [INFO] signed certificate with serial number 198646503974419332869251288662118700013983094096
[root@master1] ~$ ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
5、在所有节点创建/etc/kubernetes/ssl目录，用来存放各种证书文件
在所有节点创建
[root@ansible] ~$ ansible all -m shell -a " mkdir -p /etc/kubernetes/ssl"
6、将ca相关文件复制到所有节点的/etc/kubernetes/ssl目录下
如果远端机器没有提前创建/etc/kubernetes/ssl/目录，scp命令会报错。
[root@master1] ~$ mv ca* /etc/kubernetes/ssl/
[root@master1] ~$ scp -r /etc/kubernetes/ssl/* [email protected]:/etc/kubernetes/ssl
[root@master1] ~$ scp -r /etc/kubernetes/ssl/* [email protected]:/etc/kubernetes/ssl
[root@master1] ~$ scp -r /etc/kubernetes/ssl/* [email protected]:/etc/kubernetes/ssl
[root@master1] ~$ scp -r /etc/kubernetes/ssl/* [email protected]:/etc/kubernetes/ssl
查看一下，确保如下：
[root@ansible] ~$ ansible all -m shell -a "ls /etc/kubernetes/ssl"
二、搭建高可用etcd集群
在master1上操作
1、 下载安装etcd二进制文件
[root@master1] ~$ wget https://github.com/coreos/etcd/releases/download/v3.3.13/etcd-v3.1.5-linux-amd64.tar.gz -C /opt/software
[root@master1] ~$ tar zxvf etcd-v3.1.5-linux-amd64.tar.gz -C /opt/software
[root@master1] ~$ mv /opt/software/etcd-v3.1.5-linux-amd64/etcd* /usr/local/bin
# 分发到其他etcd节点上
[root@master1] ~$ scp /usr/local/bin/etcd* master2.hanli.com:/usr/local/bin/
[root@master1] ~$ scp /usr/local/bin/etcd* master3.hanli.com:/usr/local/bin/
2、创建etcd证书
为了保证通信安全，客户端(如etcdctl)与etcd 集群、etcd 集群之间的通信需要使用TLS 加密。
1）创建etcd 证书签名请求
hosts 字段指定授权使用该证书的etcd节点IP
[root@master1] ~$ vim etcd-csr.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.255.131",
"192.168.255.132",
"192.168.255.133",
"0.0.0.0"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2）生成etcd证书
[root@master1] ~$ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
2019/06/03 17:44:53 [INFO] generate received request
2019/06/03 17:44:53 [INFO] received CSR
2019/06/03 17:44:53 [INFO] generating key: rsa-2048
2019/06/03 17:44:53 [INFO] encoded CSR
2019/06/03 17:44:53 [INFO] signed certificate with serial number 246418866699320203258385930416062207987220135572
2019/06/03 17:44:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
-
如果报错
invalid character ']' looking for beginning of value Failed to parse input: unexpected end of JSON input，原因可能是你hosts字段中最后一行多了个逗号，如果是，请去掉。 -
如果报错
Failed to load config file: {"code":5200,"message":"could not read configuration file"}Failed to parse input: unexpected end of JSON input，原因可能是你执行的命令中指定的ca，ca-key，config路径不正确，请仔细检查
3） 查看生成的证书文件，并移动到/etc/etcd/ssl目录中
[root@master1] ~$ ls etcd*
etcd.csr etcd-csr.json etcd-key.pem etcd.pem
[root@master1] ~$ mkdir -p /etc/etcd/ssl/
[root@master1] ~$ mv etcd*.pem /etc/etcd/ssl/
[root@master1] ~$ ls /etc/etcd/ssl/
etcd-key.pem etcd.pem
4） 将证书分发到其他etcd节点上
先在master2和master3上创建/etc/etcd/ssl/目录，来存放etcd证书
mkdir -p /etc/etcd/ssl
然后分发
[root@master1] ~$ scp -r /etc/etcd/ssl/* [email protected]:/etc/etcd/ssl
[root@master1] ~$ scp -r /etc/etcd/ssl/* [email protected]:/etc/etcd/ssl
3 、创建etcd的systemd配置文件
在所有etcd节点创建工作目录和存放数据的目录
[root@ansible] ~$ ansible master -m shell -a "mkdir -p /var/lib/etcd"
创建etcd.service的systemd文件
[root@master1] ~$ vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd
ExecStart=/usr/local/bin/etcd \
--data-dir=/var/lib/etcd \
--name=master1 \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--peer-cert-file=/etc/etcd/ssl/etcd.pem \
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \
--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--listen-peer-urls=https://192.168.255.131:2380 \
--initial-advertise-peer-urls=https://192.168.255.131:2380 \
--listen-client-urls=https://192.168.255.131:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://192.168.255.131:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=master1=https://192.168.255.131:2380,master2=https://192.168.255.132:2380,master3=https://192.168.255.133:2380 \
--initial-cluster-state=new \
--heartbeat-interval=250 \
--election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
4、将systemd配置分发到其他etcd节点上
注意： 要根据不同的机器修改配置中的IP地址和name字段
[root@master1] ~$ scp /etc/systemd/system/etcd.service [email protected]:/etc/systemd/system/etcd.service
[root@master1] ~$ scp /etc/systemd/system/etcd.service [email protected]:/etc/systemd/system/etcd.service
5、启动etcd
在三台etcd节点上启动，（不需要按特定的顺序）
systemctl daemon-reload && systemctl enable etcd
systemctl start etcd && systemctl status etcd
如果无法启动，查看日志文件tail /var/log/messages
我在这里遇到了这个报错error verifying flags, '\' is not a valid flag. See 'etcd --help'.
原因是--name=master1 \\ 这一行中的master1和\\中间隔了两个空格，解决办法是删除一个空格。
6、 验证是否搭建成功
在任一 etcd 机器上执行如下命令：
[root@master1] ~$ etcdctl \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
cluster-health
2019-06-03 18:07:47.325908 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2019-06-03 18:07:47.326627 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 4ddac9672dc290e7 is healthy: got healthy result from https://192.168.255.132:2379
member 59f429fb1bfa3d2c is healthy: got healthy result from https://192.168.255.133:2379
member d88fe6c5f19be458 is healthy: got healthy result from https://192.168.255.131:2379
cluster is healthy
检查角色情况
[root@master1] ~$ etcdctl \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
member list
2019-06-07 22:42:36.046739 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
4ddac9672dc290e7: name=master2 peerURLs=https://192.168.255.132:2380 clientURLs=https://192.168.255.132:2379 isLeader=true
59f429fb1bfa3d2c: name=master3 peerURLs=https://192.168.255.133:2380 clientURLs=https://192.168.255.133:2379 isLeader=false
d88fe6c5f19be458: name=master1 peerURLs=https://192.168.255.131:2380 clientURLs=https://192.168.255.131:2379 isLeader=false
也可以单独检查endpoint的状态，如果你这样执行，会发现没有输出结果
[root@master1] ~$ etcdctl --endpoints=https://192.168.255.131:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem endpoint health
No help topic for 'endpoint'
你应该这样
[root@master1] ~$ ETCDCTL_API=3 etcdctl --endpoints=https://192.168.255.131:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint health
2019-06-03 18:21:07.750804 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
https://192.168.255.131:2379 is healthy: successfully committed proposal: took = 3.978082ms
[root@master1] ~$ ETCDCTL_API=3 etcdctl --endpoints=https://192.168.255.132:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint health
2019-06-03 18:21:30.268259 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
https://192.168.255.132:2379 is healthy: successfully committed proposal: took = 3.630145ms
[root@master1] ~$ ETCDCTL_API=3 etcdctl --endpoints=https://192.168.255.133:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint health
2019-06-03 18:21:40.002932 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
https://192.168.255.133:2379 is healthy: successfully committed proposal: took = 4.587359ms
三、部署kubectl命令行工具
kubectl默认从~/.kube/config配置文件中获取访问kube-apiserver 地址、证书、用户名等信息，需要正确配置该文件才能正常使用kubectl命令。
在master1上操作
1、下载二进制文件
这一步已经将所有节点二进制组件下载完成，后面不用重复下载了
wget https://dl.k8s.io/v1.6.0/kubernetes-server-linux-amd64.tar.gz
tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes
tar -xzvf kubernetes-src.tar.gz
cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/
如果你下载不下来，可以用这样的方法
[root@ansible] /opt/software$ docker pull zhangguanzhang/k8s_bin:v1.12.1-full
Trying to pull repository docker.io/zhangguanzhang/k8s_bin ...
v1.12.1-full: Pulling from docker.io/zhangguanzhang/k8s_bin
4fe2ade4980c: Pull complete
eac15947b2ef: Pull complete
8b9830294096: Pull complete
Digest: sha256:27543ca64f0d170efff424fc7d6e5c882a827e45ea078bef5bfd35ef214f85a8
[root@ansible] /opt/software$ docker run -d --name temp zhangguanzhang/k8s_bin:v1.12.1-full sleep 12
6321a4575de2b828dc70fa1d0555d26de7ce998567548c7981dd43bc8dcaa215
[root@ansible] /opt/software$ docker cp temp:/kubernetes-server-linux-amd64.tar.gz .
[root@ansible] /opt/software$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
2、创建 kubectl的证书和私钥
kubectl 与 apiserver https 安全端口通信，apiserver 对kubectl提供的证书进行认证和授权。
kubectl 作为集群的管理工具，需要被授予最高权限，这里创建具有最高权限的 admin 证书。
1） 创建证书签名请求：
[root@master1] ~$ vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
- 后续kube-apiserver使用RBAC 对客户端(如kubelet、kube-proxy、Pod)请求进行授权
- kube-apiserver 预定义了一些RBAC 使用的RoleBindings，如cluster-admin 将Group system:masters与Role cluster-admin绑定，该Role 授予了调用kube-apiserver所有API 的权限
- O 指定了该证书的Group 为system:masters，kubectl使用该证书访问kube-apiserver时，由于证书被CA 签名，所以认证通过，同时由于证书用户组为经过预授权的system:masters，所以被授予访问所有API 的劝降
- hosts 属性值为空列表
2） 生成admin证书和私钥：
[root@master1] ~$ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=/etc/kubernetes/ssl/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
2019/06/03 20:09:15 [INFO] generate received request
2019/06/03 20:09:15 [INFO] received CSR
2019/06/03 20:09:15 [INFO] generating key: rsa-2048
2019/06/03 20:09:16 [INFO] encoded CSR
2019/06/03 20:09:16 [INFO] signed certificate with serial number 212814235959110371975136196828061376235405653772
2019/06/03 20:09:16 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master1] ~$ ls admin*
admin.csr admin-csr.json admin-key.pem admin.pem
[root@master1] ~$ mv admin*.pem /etc/kubernetes/ssl/
[root@master1] ~$ ls /etc/kubernetes/ssl/admin*
/etc/kubernetes/ssl/admin-key.pem /etc/kubernetes/ssl/admin.pem
3、生成kubectl 的kubeconfig配置文件
执行以下四个命令，完成后会在~/.kube/目录下生成一个生成的config文件
1、设置集群参数，指定ca证书和apiserver地址
[root@master1] ~$ kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.255.131:6443"
Cluster "kubernetes" set.
2、设置客户端认证参数，指定kubectl的证书
[root@master1] ~$ kubectl config set-credentials admin \
--client-certificate=/etc/kubernetes/ssl/admin.pem \
--embed-certs=true \
--client-key=/etc/kubernetes/ssl/admin-key.pem
User "admin" set.
3、设置上下文参数
[root@master1] ~$ kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin
Context "kubernetes" created.
4、设置默认上下文
[root@master1] ~$ kubectl config use-context kubernetes
Switched to context "kubernetes".
查看一下
[root@master1] ~$ cat .kube/config
或
[root@master1] ~$ kubectl config view
4、将 kubectl 二进制文件和/root/.kube/config文件，分发到所有需要使用kubectl 命令的节点
这里只分发到了master2和master3，其实也可以不分发，因为我们一般只在master1上用这个命令
[root@master1] ~$ scp /usr/local/bin/kubectl master2.hanli.com:/usr/local/bin/kubectl
kubectl 100% 55MB 54.7MB/s 00:00
[root@master1] ~$ scp /usr/local/bin/kubectl master3.hanli.com:/usr/local/bin/kubectl
kubectl 100% 55MB 54.7MB/s 00:01
# 先在master2和master3上创建/root/.kube目录
[root@master1] ~$ scp /root/.kube/config master2.hanli.com:/root/.kube/config
config 100% 6263 6.1KB/s 00:00
[root@master1] ~$ scp /root/.kube/config master3.hanli.com:/root/.kube/config
config 100% 6263 6.1KB/s 00:00
四、部署Flannel 网络
flannel采用覆盖网络模型来完成对网络的打通。
注意事项：由于flannel将覆盖docker0网桥，所以如果节点上docker服务已启动，请确保先停止docker服务，否则flannel将无法正常启动。
1、 创建flannel证书
创建flanneld 证书签名请求
[root@master1] ~$ vim flanneld-csr.json
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
生成flanneld 证书和私钥：
[root@master1] ~$ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=/etc/kubernetes/ssl/ca-config.json \
-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
2019/06/03 20:33:16 [INFO] generate received request
2019/06/03 20:33:16 [INFO] received CSR
2019/06/03 20:33:16 [INFO] generating key: rsa-2048
2019/06/03 20:33:16 [INFO] encoded CSR
2019/06/03 20:33:16 [INFO] signed certificate with serial number 527590477885655817365111273046638664217916199338
2019/06/03 20:33:16 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master1] ~$ ls flanneld*
flanneld.csr flanneld-csr.json flanneld-key.pem flanneld.pem
[root@master1] ~$ mkdir -p /etc/flanneld/ssl
[root@master1] ~$ mv flanneld*.pem /etc/flanneld/ssl
将分发到其余四个节点
2、 向etcd 写入集群Pod 网段信息
只需在master1上执行一次就行了，后面 在其他节点上部署Flanneld 时无需再写入该信息
[root@master1] ~$ etcdctl \
--endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/flanneld/ssl/flanneld.pem \
--key-file=/etc/flanneld/ssl/flanneld-key.pem \
set /coreos.com/network/config '{"Network":"'172.30.0.0/16'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
2019-06-03 20:43:17.063513 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}
- 写入的 Pod 网段(==172.30.0.0/16) 必须与kube-controller-manager 的 --cluster-cidr 选项值一致；
3、下载安装flanneld
[root@master1] /opt/software$ wget https://github.com/coreos/flannel/releases/download/v0.9.0/flannel-v0.9.0-linux-amd64.tar.gz
[root@master1] /opt/software$ tar zxvf flannel-v0.9.0-linux-amd64.tar.gz
flanneld
mk-docker-opts.sh
README.md
[root@master1] ~$ mv flanneld /usr/local/bin
[root@master1] ~$ mv mk-docker-opts.sh /usr/local/bin
将这两个命令分发到其余四个节点
4、创建flannel的systemd配置文件
所有五个节点配置一样，不用修改
[root@master1] ~$ vim /etc/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/usr/local/bin/flanneld \\
-etcd-cafile=/etc/kubernetes/ssl/ca.pem \\
-etcd-certfile=/etc/flanneld/ssl/flanneld.pem \\
-etcd-keyfile=/etc/flanneld/ssl/flanneld-key.pem \\
-etcd-endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 \\
-etcd-prefix=/coreos.com/network
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
- mk-docker-opts.sh脚本将分配给flanneld 的Pod 子网网段信息写入到/run/flannel/docker 文件中，后续docker 启动时使用这个文件中的参数值为 docker0 网桥
- flanneld 使用系统缺省路由所在的接口和其他节点通信，对于有多个网络接口的机器(内网和公网)，可以用 --iface 选项值指定通信接口(上面的 systemd unit 文件没指定这个选项)
分发到其余四个节点
5、启动flanneld
启动全部节点。
[root@master1] ~$ systemctl daemon-reload && systemctl enable flanneld
[root@master1] ~$ systemctl start flanneld && systemctl status flanneld
- 如果你这里报错
Couldn't fetch network config: 100: Key not found (/coreos.com) [10]，请检查，etcdctl写入的网段key值是否正确。
我这里只有3台master能正常启动，两台node无法启动。报错如下，不知道什么原因。。。怀疑是node节点无法访问etcd节点
[root@slave1] ~$ tail /var/log/messages
Jun 7 23:18:39 slave1 flanneld: ; error #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jun 7 23:18:40 slave1 flanneld: timed out
Jun 7 23:18:40 slave1 flanneld: E0607 23:18:40.228662 14172 main.go:344] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 7 23:18:40 slave1 flanneld: ; error #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jun 7 23:18:41 slave1 flanneld: timed out
Jun 7 23:18:41 slave1 flanneld: E0607 23:18:41.229865 14172 main.go:344] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 7 23:18:41 slave1 flanneld: ; error #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jun 7 23:18:42 slave1 flanneld: timed out
Jun 7 23:18:42 slave1 flanneld: E0607 23:18:42.231224 14172 main.go:344] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 7 23:18:42 slave1 flanneld: ; error #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused
启动后，确认网络接口docker0的ip地址属于flannel的子网
[root@master1] ~$ ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens33: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:24:5f:60 brd ff:ff:ff:ff:ff:ff
inet 192.168.255.131/24 brd 192.168.255.255 scope global ens33
valid_lft forever preferred_lft forever
3: docker0: mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:55:72:8b:b6 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN
link/ether 4e:00:11:cb:4e:9a brd ff:ff:ff:ff:ff:ff
inet 172.30.13.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
重启docker服务
在这里插入代码片
到此就完成了flannel覆盖网络的设置
6、检查flannel网络是否正常
检查flanneld 服务
[root@master1] ~$ ifconfig flannel.1
flannel.1: flags=4163 mtu 1450
inet 172.30.76.0 netmask 255.255.255.255 broadcast 0.0.0.0
ether 9e:37:66:85:b7:3e txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
查看集群 Pod 网段(/16)
[root@master1] ~$ etcdctl \
--endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/flanneld/ssl/flanneld.pem \
--key-file=/etc/flanneld/ssl/flanneld-key.pem \
get /coreos.com/network/config
2019-06-04 08:14:41.608657 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}
查看已分配的 Pod 子网段列表(/24)
[root@master1] ~$ etcdctl --endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/flanneld/ssl/flanneld.pem --key-file=/etc/flanneld/ssl/flanneld-key.pem ls /coreos.com/network/subnets
2019-06-04 08:15:14.867060 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
/coreos.com/network/subnets/172.30.76.0-24
查看某一 Pod 网段对应的 flanneld 进程监听的 IP 和网络参数
[root@master1] ~$ etcdctl --endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/flanneld/ssl/flanneld.pem --key-file=/etc/flanneld/ssl/flanneld-key.pem get /coreos.com/network/subnets/172.30.76.0-24
2019-06-04 08:16:30.347131 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
{"PublicIP":"192.168.255.131","BackendType":"vxlan","BackendData":{"VtepMAC":"9e:37:66:85:b7:3e"}}
在etcd上看下flannel地址与节点ip的是否对应
[root@master1] ~$ etcdctl --endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/flanneld/ssl/flanneld.pem --key-file=/etc/flanneld/ssl/flanneld-key.pem ls /coreos.com/network/subnets
2019-06-07 23:02:15.814165 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
/coreos.com/network/subnets/172.30.13.0-24
[root@master1] ~$ etcdctl --endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/flanneld/ssl/flanneld.pem --key-file=/etc/flanneld/ssl/flanneld-key.pem get /coreos.com/network/subnets/172.30.13.0-24
2019-06-07 23:02:57.028715 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
{"PublicIP":"192.168.255.131","BackendType":"vxlan","BackendData":{"VtepMAC":"4e:00:11:cb:4e:9a"}}
2、 确保各节点间Pod 网段能互联互通
使用ping命令验证各node上docker0之间的相互访问
在master1上ping master2的docker0
查看已分配的Pod 子网段列表：
[root@master1] ~$ etcdctl --endpoints=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/flanneld/ssl/flanneld.pem --key-file=/etc/flanneld/ssl/flanneld-key.pem ls /coreos.com/network/subnets
2019-06-04 10:16:06.245571 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
/coreos.com/network/subnets/172.30.76.0-24
/coreos.com/network/subnets/172.30.83.0-24
/coreos.com/network/subnets/172.30.60.0-24
当前五个节点分配的 Pod 网段分别是：172.30.76.0-24、172.30.83.0-24、172.30.60.0-24、
五 、安装haproxy和keepalived
在三台master上安装keepalived，三台配置略有差异，根据备注自己修改。
keepalived的作用是故障转移。虚拟IP：192.168.255.140，端口：8443
[root@master1] ~$ yum install -y keepalived
[root@master1] ~$ vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
[email protected]
[email protected]
[email protected]
}
notification_email_from [email protected]
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state MASTER #备服务器上改为BACKUP
interface ens33 #改为自己的接口
virtual_router_id 51
priority 100 #备服务器上改为小于100的数字，90,80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.255.140 #虚拟vip，自己设定
}
}
三台都启动
systemctl enable keepalived && systemctl start keepalived
systemctl status keepalived
查看虚拟ip是不是在master1上
[root@master1] ~$ ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens33: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:24:5f:60 brd ff:ff:ff:ff:ff:ff
inet 192.168.255.131/24 brd 192.168.255.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.255.140/32 scope global ens33
valid_lft forever preferred_lft forever
3: docker0: mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:55:72:8b:b6 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN
link/ether 9e:37:66:85:b7:3e brd ff:ff:ff:ff:ff:ff
inet 172.30.76.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
关闭master1的keepalived服务systemctl stop keepalived，测试下vip是否漂移到master2上。
重新启动master1上的keepalived服务systemctl start keepalived，vip又会漂移到master1上
2） 安装haproxy
haproxy的作用是负载均衡
三台master上配置一样
[root@master1] ~$ yum install -y haproxy
[root@master1] ~$ vim /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
defaults
mode tcp
log global
retries 3
timeout connect 10s
timeout client 1m
timeout server 1m
frontend kubernetes
bind *:8443 #配置端口为8443
mode tcp
default_backend kubernetes-master
backend kubernetes-master #后端服务器，也就是说访问192.168.255.140:8443会将请求转发到后端的三台，这样就实现了负载均衡
balance roundrobin
server master1 192.168.255.131:6443 check maxconn 2000
server master2 192.168.255.132:6443 check maxconn 2000
server master3 192.168.255.133:6443 check maxconn 2000
三台都启动
systemctl enable haproxy && systemctl start haproxy
systemctl status haproxy
六、部署master节点
包括以下组件：
- kube-apiserver
- kube-scheduler
- kube-controller-manager
1、下载二进制文件
前面已经下载好了
[root@master1] ~$ ls /usr/local/bin/kube-*
/usr/local/bin/kube-apiserver /usr/local/bin/kube-controller-manager /usr/local/bin/kube-scheduler
2、创建kubernetes 证书
1） 创建kubernetes 证书签名请求：
[root@master1] ~$ vim kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.255.131",
"192.168.255.132",
"192.168.255.133",
"192.168.255.140", #虚拟IP
"10.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2） 生成kubernetes 证书和私钥：
[root@master1] ~$ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=/etc/kubernetes/ssl/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
2019/06/04 10:40:30 [INFO] generate received request
2019/06/04 10:40:30 [INFO] received CSR
2019/06/04 10:40:30 [INFO] generating key: rsa-2048
2019/06/04 10:40:30 [INFO] encoded CSR
2019/06/04 10:40:30 [INFO] signed certificate with serial number 531594318212721165922883938246438021866605824300
2019/06/04 10:40:30 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master1] ~$ ls kubernetes*
kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem
#移动到/etc/kubernetes/ssl/目录下
[root@master1] ~$ mv kubernetes*.pem /etc/kubernetes/ssl/
3、配置和启动kube-apiserver
1） 创建 TLS Bootstrapping Token,用来自动通过请求
[root@master1] ~$ head -c 16 /dev/urandom | od -An -t x | tr -d ' '
7c0f7452e73c322acc4044cc76eec224
[root@master1] ~$ vim /etc/kubernetes/token.csv
7c0f7452e73c322acc4044cc76eec224,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
2）创建kube-apiserver的systemd配置文件
[root@master1] ~$ vim /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
--advertise-address=192.168.255.131 \\ #修改为本机ip
--bind-address=0.0.0.0 \\
--insecure-bind-address=192.168.255.131 \\ #修改为本机ip
--authorization-mode=Node,RBAC \\
--runtime-config=rbac.authorization.k8s.io/v1alpha1 \\
--kubelet-https=true \\
#--enable-bootstrap-token-auth \\
--token-auth-file=/etc/kubernetes/token.csv \\
--service-cluster-ip-range=10.254.0.0/16 \\
--service-node-port-range=30000-32766 \\
--tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\
--tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\
--client-ca-file=/etc/kubernetes/ssl/ca.pem \\
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/etc/kubernetes/ssl/ca.pem \\
--etcd-certfile=/etc/etcd/ssl/etcd.pem \\
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\
--etcd-servers=https://192.168.255.131:2379,https://192.168.255.132:2379,https://192.168.255.133:2379 \\
--enable-swagger-ui=true \\
--allow-privileged=true \\
--apiserver-count=3 \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/var/lib/audit.log \\
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
--event-ttl=1h \\
--logtostderr=true \\
--v=6
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
审计政策文件
[root@master1] ~$ vim /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
omitStages:
- "RequestReceived"
rules:
- level: RequestResponse
resources:
- group: ""
resources: ["pods"]
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
namespaces: ["kube-system"]
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
- level: Metadata
omitStages:
- "RequestReceived"
3） 将kube-apiserver命令、公钥证书、token文件、systemd文件、审计政策文件分发到master2和amster3上
4）启动三台master上的 kube-apiserver
systemctl daemon-reload && systemctl enable kube-apiserver
systemctl start kube-apiserver && systemctl status kube-apiserver
4、配置和启动kube-controller-manager
1、创建systemd配置文件
[root@master1] ~$ vim /etc/systemd/system/kube-cintroller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
--address=127.0.0.1 \\
--master=http://192.168.255.140:8443 \\
--allocate-node-cidrs=true \\
--service-cluster-ip-range="10.254.0.0/16" \\
--cluster-cidr=172.30.0.0/16 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/etc/kubernetes/ssl/ca.pem \\
--leader-elect=true \\
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
2、分发systemd文件到master2和master3
3、启动三台master上的kube-controller-manager
systemctl daemon-reload && systemctl enable kube-controller-manager
systemctl start kube-controller-manager && systemctl status kube-controller-manager
5、 配置和启动kube-scheduler
1）systemd配置文件
[root@master1] ~$ vim /etc/systemd/system/kube-scheduler
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
--address=127.0.0.1 \\
--master=https://192.168.255.140:8443 \\
--leader-elect=true \\
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
2、分发到master2和master3
3）启动kube-scheduler
systemctl daemon-reload && systemctl enable kube-scheduler
systemctl start kube-scheduler && systemctl status kube-scheduler
6、验证 master 节点各个组件的状态
$ kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
七、部署node节点
Kubernetes node节点包含如下组件：
- Flanneld
- Docker1.12.5
- kubelet
- kube-proxy
检查一下节点上，经过前几步操作我们已经创建了如下的证书和配置文件。
$ ls /etc/kubernetes/ssl
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem kubernetes-key.pem kubernetes.pem
$ ls /etc/kubernetes/
apiserver bootstrap.kubeconfig config controller-manager kubelet kube-proxy.kubeconfig proxy scheduler ssl token.csv
2、安装Docker
1）修改docker的配置文件/usr/lib/systemd/system/docker.service，增加环境变量配置，修改后如下
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/run/flannel/docker
EnvironmentFile=-/run/docker_opts.env
EnvironmentFile=-/run/flannel/subnet.env
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
EnvironmentFile=-/run/docker_opts.env
ExecStart=/usr/bin/dockerd -H unix://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
3）启动docker
systemctl daemon-reload
systemctl enable docker
systemctl start docker && systemctl status docker
3、安装和配置kubelet
kubelet 启动时向kube-apiserver 发送TLS bootstrapping 请求，需要先将bootstrap token 文件中的kubelet-bootstrap 用户赋予system:node-bootstrapper 角色，然后kubelet 才有权限创建认证请求(certificatesigningrequests)：
kubelet就是运行在Node节点上的，所以这一步安装是在所有的Node节点上，如果你想把你的Master也当做Node节点的话，当然也可以在Master节点上安装的。
1）配置文件 vim /etc/kubernetes/kubelet
###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=192.168.255.130"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.255.130"
#
## location of the api-server
## COMMENT THIS ON KUBERNETES 1.8+
KUBELET_API_SERVER="--api-servers=http://192.168.255.130:8080"
#
## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=jimmysong/pause-amd64:3.0"
#
## Add your own!
KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"
参数说明：
- 其中前两个的IP地址更改为你的每台node节点的IP地址。
- hostname-override 在集群中显示的主机名
2）systemd管理文件 vim /usr/lib/systemd/system/kubelet.service。
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_API_SERVER \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBE_ALLOW_PRIV \
$KUBELET_POD_INFRA_CONTAINER \
$KUBELET_ARGS
#Restart=on-failure
[Install]
WantedBy=multi-user.target
4） 启动前先创建/var/lib/kubelet目录
[root@ansible] ~$ ansible kubernetes -m shell -a "mkdir -p /var/lib/kubelet"
5）启动前先执行以下两步操作，只在node1（master）上执行就行
kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求，需要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper cluster 角色(role)， 然后 kubelet 才能有权限创建认证请求(certificate signing requests)：
cd /etc/kubernetes
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
#--user=kubelet-bootstrap 是在 /etc/kubernetes/token.csv 文件中指定的用户名，同时也写入了 /etc/kubernetes/bootstrap.kubeconfig 文件；
kubelet 通过认证后向 kube-apiserver 发送 register node 请求，需要先将 kubelet-nodes 用户赋予 system:node cluster角色(role) 和 system:nodes 组(group)， 然后 kubelet 才能有权限创建节点请求：
kubectl create clusterrolebinding kubelet-nodes \
--clusterrole=system:node \
--group=system:nodes
6） 启动kublet
systemctl daemon-reload && systemctl enable kubelet
systemctl start kubelet && systemctl status kubelet
7）、 通过kublet的TLS证书请求
a . 手动通过
kubelet 首次启动时向 kube-apiserver 发送证书签名请求，必须批准通过后， kubernetes 系统才会将该 Node 加入到集群。
查看未授权的 CSR 请求
[root@master] /etc/kubernetes$ kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-4b26s 3m kubelet-bootstrap Pending
csr-cq3p1 3m kubelet-bootstrap Pending
csr-hzck7 29m kubelet-bootstrap Pending
[root@master] /etc/kubernetes$ kubectl get nodes
No resources found.
批准通过 CSR 请求
[root@master] /etc/kubernetes$ kubectl certificate approve csr-4b26s
certificatesigningrequest "csr-4b26s" approved
CSR被批准后，如下：
[root@master] ~$ kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-4b26s 27m kubelet-bootstrap Approved,Issued
查看节点
[root@master] ~$ kubectl get nodes
NAME STATUS AGE VERSION
192.168.255.121 Ready 9s v1.6.0
192.168.255.122 Ready 13m v1.6.0
192.168.255.130 Ready 1m v1.6.0
自动生成了 kubelet kubeconfig 文件和公私钥
[root@master] ~$ ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2282 Jan 11 21:07 /etc/kubernetes/kubelet.kubeconfig
$ ls -l /etc/kubernetes/ssl/kubelet*
[root@master] ~$ ls -l /etc/kubernetes/ssl/kubelet*
-rw-r--r-- 1 root root 1050 Jan 11 21:07 /etc/kubernetes/ssl/kubelet-client.crt
-rw------- 1 root root 227 Jan 11 18:52 /etc/kubernetes/ssl/kubelet-client.key
-rw-r--r-- 1 root root 1119 Jan 11 21:07 /etc/kubernetes/ssl/kubelet.crt
-rw------- 1 root root 1679 Jan 11 21:07 /etc/kubernetes/ssl/kubelet.key
假如你更新kubernetes的证书，只要没有更新token.csv，当重启kubelet后，该node就会自动加入到kuberentes集群中，而不会重新发送certificaterequest，也不需要在master节点上执行kubectl certificate approve操作。前提是不要删除node节点上的/etc/kubernetes/ssl/kubelet*和/etc/kubernetes/kubelet.kubeconfig文件。否则kubelet启动时会提示找不到证书而失败。
注意：如果启动kubelet的时候见到证书相关的报错，有个trick可以解决这个问题，可以将master节点上的~/.kube/config文件（该文件在安装kubectl命令行工具这一步中将会自动生成）拷贝到node节点的/etc/kubernetes/kubelet.kubeconfig位置，这样就不需要通过CSR，当kubelet启动后就会自动加入的集群中。
b. 自动通过csr请求
Master apiserver启用TLS认证后，Node节点kubelet组件想要加入集群，必须使用CA签发的有效证书才能与apiserver通信，当Node节点很多时，签署证书是一件很繁琐的事情，因此有了TLS Bootstrapping机制，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。
4、安装 kube-proxy
1）安装conntrack
[root@ansible] ~$ ansible kubernetes -m shell -a "yum install -y conntrack-tools"
2）、创建 kube-proxy 证书
vim proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
#生成
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server proxy-csr.json | cfssljson -bare proxy
3）、创建 kube-proxy kubeconfig 文件，生成的 kubeconfig会 被保存到 ~/.kube/config 文件
export KUBE_APISERVER="https://192.168.255.130:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kube-proxy \
--client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
--client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
4）分发 kubeconfig 文件到所有节点
5）配置文件 vim /etc/kubernetes/proxy，不同的node上需要修改ip
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=192.168.255.130 --hostname-override=192.168.255.130 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"
6）systemd管理文件 vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/kube-proxy \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
7）启动 kube-proxy
systemctl daemon-reload && systemctl enable kube-proxy
systemctl start kube-proxy && systemctl status kube-proxy
八、验证测试
我们创建一个nginx的service试一下集群是否可用。
$ kubectl run nginx --replicas=2 --labels="run=load-balancer-example" --image=nginx --port=80
$ kubectl expose deployment nginx --type=NodePort --name=example-service
$ kubectl get svc example-service
访问以下任何一个地址都可以得到nginx的页面。
192.168.255.130:30666
192.168.255.131:30666
192.168.255.132:30666