K8S1.14高可用生产集群部署方案
系统说明
系统组件版本
- 操作系统: CentOS 7.6
- Kernel: 4.4
- Kubernetes: v1.14.1
- Docker: 18.09 (支持 1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09 )
- Etcd: v3.3.12
- Flannel: v0.11
- cni-plugins: v0.7.5
- CoreDNS: 1.4.0
架构示意图
架构说明：
- 使用六台主机，三台Master节点, 三台node节点
- Master节点上部署的Kubernetes组件有kube-apiserver, kube-scheduler, kube-controller-manager,kube-proxy。部署网络组件flannel,数据存储集群Etcd.
- Master的两个高可用节点部署Haproxy和keepalived
- Node 节点部署的Kubernetes组件有Kubelet,kube-proxy。 容器组件Docker,网络组件Flannel
- 集群IP和主机名信息：
| 集群角色 | 主机名 | IP |
|---|---|---|
| Master | master-1 | 192.168.20.44 |
| Master | master-2 | 192.168.20.45 |
| Master | master-3 | 192.168.20.46 |
| Node | k8s-node-1 | 192.168.20.47 |
| Node | k8s-node-2 | 192.168.20.48 |
| Node | k8s-node-3 | 192.168.20.49 |
- Ceph 需要一个可用的Ceph集群
系统初始化
1. 主机初始化
安装好CentOS7的系统，做以下操作：
- 关闭firewalld，Selinux。
- 更新系统软件包，执行
yum update - 安装elrepo的源，更新内核为4.4或以上版本，并重启生效
- 分别设置主机名为，并在本地hosts文件中解析。
- 安装NTP服务
- 设置内核参数
设置内核参数的部分，要确认执行如下操作：
# 高可用Master节点设置内核参数
cat < /etc/sysctl.d/k8s.conf
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_port_range = 10000 65000
fs.file-max = 2000000
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF
# 其他Master节点和计算节点设置内核参数
cat < /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_port_range = 10000 65000
fs.file-max = 2000000
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF 2. 安装Docker
由于在Kubernetes1.14的版本中，支持的版本有 1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09，所以这里统一使用Docker 18.09。
下载docker-ce-18.09的rpm包和阿里云上对应的docker-ce.repo的源，在所有node节点上直接安装：
mv docker-ce.repo /etc/yum.repos.d/
yum install docker-ce-18.09.5-3.el7.x86_64.rpm -y
在所有node节点启动docker,并配置自启动：
systemctl start docker
systemctl enable docker3. 创建目录
在所有主机上执行如下命令，创建所需目录：
mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}4. 准备Kubernetes软件包
从github下载kubernetes 1.14版本的二进制软件包，下载地址: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#server-binaries
下载如下软件包：
[root@master-1 tmp]# ll
total 537520
-rw-r--r-- 1 root root 113938518 Jul 24 19:15 kubernetes-node-linux-amd64.tar.gz
-rw-r--r-- 1 root root 433740362 Jul 24 19:09 kubernetes-server-linux-amd64.tar.gz解压：
tar xf kubernetes-server-linux-amd64.tar.gz5.准备etcd和flannel组件
从github下载etcd v3.3.12和flannel v0.11.0 :
wget https://github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
制作CA证书
Kubernetes支持使用多种方式生成证书，可以使用easyrsa, openssl 或 cfssl任一一种生成。
参考链接
这里使用cfssl创建CA证书。
1.安装CFSSL
使用cfssl生成CA证书需要单独安装cfssl。
[root@master-1 ~]# cd /usr/local/src/
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /opt/kubernetes/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /opt/kubernetes/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /opt/kubernetes/bin/cfssl-certinfo
chmod +x /opt/kubernetes/bin/*
所有节点添加Kubernetes的bin目录到系统环境变量：
echo 'PATH=$PATH:/opt/kubernetes/bin' >>/etc/profile
source /etc/profile2.生成证书
- 创建需要的配置文件：
[root@master-1 ~]# cd /opt/kubernetes/ssl/
[root@master-1 ssl]# cfssl print-defaults config > config.json
[root@master-1 ssl]# cfssl print-defaults csr > csr.json
[root@master-1 ssl]# ll
total 8
-rw-r--r-- 1 root root 567 Jul 26 00:05 config.json
-rw-r--r-- 1 root root 287 Jul 26 00:05 csr.json
[root@master-1 ssl]# mv config.json ca-config.json
[root@master-1 ssl]# mv csr.json ca-csr.json
- 修改生成的文件为如下内容：
ca-config.json文件：
[root@master-1 ssl]# vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
ca-csr.json文件：
[root@master-1 ssl]# vim ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}- 生成证书（ca-key.pem）和秘钥(ca.pem):
[root@master-1 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2018/07/26 00:27:00 [INFO] generating a new CA key and certificate from CSR
2018/07/26 00:27:00 [INFO] generate received request
2018/07/26 00:27:00 [INFO] received CSR
2018/07/26 00:27:00 [INFO] generating key: rsa-2048
2018/07/26 00:27:01 [INFO] encoded CSR
2018/07/26 00:27:01 [INFO] signed certificate with serial number 479065525331838190845576195908271097044538206777
[root@master-1 ssl]# ll
total 20
-rw-r--r-- 1 root root 386 Jul 26 00:16 ca-config.json
-rw-r--r-- 1 root root 1001 Jul 26 00:27 ca.csr
-rw-r--r-- 1 root root 255 Jul 26 00:20 ca-csr.json
-rw------- 1 root root 1679 Jul 26 00:27 ca-key.pem
-rw-r--r-- 1 root root 1359 Jul 26 00:27 ca.pem
- 分发证书到各个节点：
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.45:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.46:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.47:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.48:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.49:/opt/kubernetes/sslHA 节点部署
这里选择两个Master节点部署Haproxy 和 keepalived， keepalived上需要添加监控haproxy应用的脚本。
keepalived 配置
- HA节点下载安装keepalive:
yum install keepalived -y- 配置两个虚拟IP，一个用于k8s集群的apiserver代理，另一个用于nginx ingress入口（也可以单独配置），同时要设置对haproxy的状态判断，如果节点上的haproxy进程结束需要自动切换VIP到另一节点上， 主HA配置如下：
# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 3
weight -20
}
vrrp_instance K8S {
state backup
interface eth0
virtual_router_id 44
priority 200
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.20.50
192.168.20.60
}
track_script {
check_haproxy
}
}
- 从HA配置如下：
! Configuration File for keepalived
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 3
weight -20
}
vrrp_instance K8S {
state backup
interface eth0
virtual_router_id 44
priority 190
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.20.50
192.168.20.60
}
track_script {
check_haproxy
}
}
- 在这两个节点上配置对应的监测脚本：
vim /etc/keepalived/check_haproxy.sh
#!/bin/bash
active_status=`netstat -lntp|grep haproxy|wc -l`
if [ $active_status -gt 0 ]; then
exit 0
else
exit 1
fi- 需要添加权限
chmod +x /etc/keepalived/check_haproxy.sh部署Haproxy
官方配置手册
- 需要先确认已经配置了内核参数：
echo 'net.ipv4.ip_nonlocal_bind = 1'>>/etc/sysctl.conf
echo 'net.ipv4.ip_forward = 1'>>/etc/sysctl.conf
sysctl -p- 安装haproxy
yum install haproxy -y- 配置haproxy, 我们为k8s集群设计的VIP为 192.168.20.50，使用4层代理的方式， 配置文件如下：
# cat /etc/haproxy/haproxy.cfg |egrep -v "^#"
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
defaults
mode tcp # 修改默认为四层代理
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main 192.168.20.50:6443
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
default_backend k8s-node
backend k8s-node
mode tcp # 修改为tcp
balance roundrobin
server k8s-node-1 192.168.20.44:6443 check # 三个master主机
server k8s-node-2 192.168.20.45:6443 check
server k8s-node-3 192.168.20.46:6443 check
配置完成之后检查IP是否可以自动切换。
部署ETCD集群
1.安装etcd
执行如下命令，完成etcd的安装：
[root@master-1 ~]# cd /tmp/
[root@master-1 tmp]# tar xf etcd-v3.3.12-linux-amd64.tar.gz
[root@master-1 tmp]# cd etcd-v3.3.12-linux-amd64
[root@master-1 tmp]# cp etcd* /opt/kubernetes/bin/
[root@master-1 tmp]# scp etcd* 192.168.20.45:/opt/kubernetes/bin/
[root@master-1 tmp]# scp etcd* 192.168.20.46:/opt/kubernetes/bin/
2.生成etcd的专属证书
1.创建etcd证书签名请求
[root@master-1 ~]# vim etcd-csr.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.20.44",
"192.168.20.45",
"192.168.20.46"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2.生成etcd证书
[root@master-1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd会生成如下文件：
[root@master-1 ~]# ll
total 16
-rw-r--r-- 1 root root 1062 Jul 26 01:18 etcd.csr
-rw-r--r-- 1 root root 287 Jul 26 00:50 etcd-csr.json
-rw------- 1 root root 1679 Jul 26 01:18 etcd-key.pem
-rw-r--r-- 1 root root 1436 Jul 26 01:18 etcd.pem
- 移动证书到ssl目录：
[root@master-1 ~]# cp etcd*.pem /opt/kubernetes/ssl
[root@master-1 ~]# scp etcd*.pem 192.168.20.45:/opt/kubernetes/ssl
[root@master-1 ~]# scp etcd*.pem 192.168.20.46:/opt/kubernetes/ssl3.配置etcd
- 配置ETCD配置文件
master-1上的配置为：
[root@master-1 ~]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.20.44:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.44:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.44:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=https://192.168.20.44:2380,etcd-node-2=https://192.168.20.45:2380,etcd-node-3=https://192.168.20.46:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.44:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"master-2上的配置为：
[root@master-2 tmp]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.20.45:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.45:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.45:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=https://192.168.20.44:2380,etcd-node-2=https://192.168.20.45:2380,etcd-node-3=https://192.168.20.46:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.45:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
master-3上的配置为：
[root@master-3 ~]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.20.46:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.46:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.46:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=https://192.168.20.44:2380,etcd-node-2=https://192.168.20.45:2380,etcd-node-3=https://192.168.20.46:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.46:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"在三个节点上创建etcd的systemd文件：
[root@master-1 ~]# vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service
[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd"
[Install]
WantedBy=multi-user.target
- 启动ETCD服务,在三个节点执行如下命令:
mkdir /var/lib/etcd
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd确认所以节点的etcd服务启动。
4.验证集群
[root@master-1 ~]# etcdctl --endpoints=https://192.168.20.44:2379 \
--ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/etcd.pem \
--key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health
member 32922a109cfe00b2 is healthy: got healthy result from https://192.168.20.46:2379
member 4fa519fdd3e64a84 is healthy: got healthy result from https://192.168.20.45:2379
member cab6e832332e8b2a is healthy: got healthy result from https://192.168.20.44:2379
cluster is healthyMaster节点部署
1.部署Kubernetes软件包
[root@master-1 ~]# cd /tmp/kubernetes/server/bin/
[root@master-1 bin]# cp kube-apiserver /opt/kubernetes/bin/
[root@master-1 bin]# cp kube-controller-manager /opt/kubernetes/bin/
[root@master-1 bin]# cp kube-scheduler /opt/kubernetes/bin/
2.生成API Server的认证文件
参考链接
1.创建用于生成CSR的JSON文件，这里需要指定HA代理的IP，和集群的ClusterIP：
[root@master-1 ~]# cd /opt/kubernetes/ssl
[root@master-1 ssl]# vim kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.20.50",
"10.1.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2.生成Kubernetes的证书和私钥
[root@master-1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
- 将私钥分发到其它所有的node节点中：
[root@master-1 ssl]# scp kubernetes*.pem 192.168.20.46:/opt/kubernetes/ssl/
...- 创建 API Server使用的token文件
[root@master-1 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
197f33fcbbfab2d15603dcc4408358f5
[root@master-1 ~]# vim /opt/kubernetes/ssl/bootstrap-token.csv
197f33fcbbfab2d15603dcc4408358f5,kubelet-bootstrap,10001,"system:kubelet-bootstrap"- 创建基础用户名，密码认证配置
[root@k8s-node-1 ~]# vim /opt/kubernetes/ssl/basic-auth.csv
admin,admin,1
readonly,readonly,2
- 将ssl目录的文件拷贝到其它的master节点上
scp -r -p /opt/kubernetes/ssl/* k8s-node-1:/opt/kubernetes/ssl/
scp -r -p /opt/kubernetes/ssl/* k8s-node-2:/opt/kubernetes/ssl/
scp -r -p /opt/kubernetes/ssl/* k8s-node-3:/opt/kubernetes/ssl/3.部署kube-apiserver
- 创建kube-apiserver的systemd文件
[root@k8s-node-1 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
--enable-admission-plugins=MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
--bind-address=192.168.20.44 \
--insecure-bind-address=127.0.0.1 \
--authorization-mode=Node,RBAC \
--runtime-config=rbac.authorization.k8s.io/v1 \
--kubelet-https=true \
--anonymous-auth=false \
--basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
--service-cluster-ip-range=10.1.0.0/16 \
--service-node-port-range=20000-40000 \
--tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \
--etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \
--etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \
--etcd-servers=https://192.168.20.44:2379,https://192.168.20.45:2379,https://192.168.20.46:2379 \
--enable-swagger-ui=true \
--allow-privileged=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/log/api-audit.log \
--event-ttl=1h \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
- 启动kube-apiserver服务
[root@k8s-node-1 ~]# systemctl daemon-reload
[root@k8s-node-1 ~]# systemctl start kube-apiserver
[root@k8s-node-1 ~]# systemctl enable kube-apiserver
- 查看服务状态是否正常
[root@master-1 ~]# systemctl status kube-apiserver
[root@master-1 ~]# netstat -lntp|grep kube-apiserver
tcp 0 0 192.168.20.44:6443 0.0.0.0:* LISTEN 4289/kube-apiserver
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 4289/kube-apiserver 4.部署controller-manager
- 生成controller-manager的systemd文件
[root@master-1 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/opt/kubernetes/bin/kube-controller-manager \
--bind-address=127.0.0.1 \
--master=http://127.0.0.1:8080 \
--allocate-node-cidrs=true \
--service-cluster-ip-range=10.1.0.0/16 \
--cluster-cidr=10.2.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--leader-elect=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target- 启动kube-controller-manager
[root@master-1 ~]# systemctl daemon-reload
[root@master-1 ~]# systemctl start kube-controller-manager
[root@master-1 ~]# systemctl enable kube-controller-manager
- 查看服务状态
[root@master-1 ~]# systemctl status kube-controller-manager
[root@master-1 ~]# netstat -lntp|grep kube-con
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 4390/kube-controlle
5.部署Kubernetes Scheduler
- 创建systemd文件：
[root@master-1 ~]# vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/opt/kubernetes/bin/kube-scheduler \
--address=127.0.0.1 \
--master=http://127.0.0.1:8080 \
--leader-elect=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target- 启动服务
[root@master-1 ~]# systemctl daemon-reload
[root@master-1 ~]# systemctl start kube-scheduler
[root@master-1 ~]# systemctl enable kube-scheduler- 查看服务状态
[root@master-1 ~]# systemctl status kube-scheduler
[root@master-1 ~]# netstat -lntp|grep kube-scheduler
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 4445/kube-scheduler6. Master节点部署kube-proxy(可选)
(参见node节点部署部分，需要创建对应的kube-proxy家目录)
7. 使用上述方式，配置master-1和master-2
- 将master-1上的ssl,cfg,bin 文件拷贝到其它master节点的对应位置。
- 配置各个服务的启动文件，并启动。
8.部署kubectl命令行工具
- 安装二进制包
[root@master-1 ~]# cd /tmp/kubernetes/node/bin/
[root@master-1 bin]# cp kubectl /opt/kubernetes/bin/2.创建admin证书签名
[root@master-1 ~]# vim /opt/kubernetes/ssl/admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}3.生成admin证书和私钥
[root@master-1 ~]# cd /opt/kubernetes/ssl/
[root@master-1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
- 设置集群参数
[root@master-1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.20.50:6443
Cluster "kubernetes" set.
5.设置客户端认证参数：
[root@naster-1 ~]# kubectl config set-credentials admin \
--client-certificate=/opt/kubernetes/ssl/admin.pem \
--embed-certs=true \
--client-key=/opt/kubernetes/ssl/admin-key.pem
User "admin" set.
6.设置上下文参数
[root@master-1 ~]# kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin
Context "kubernetes" created.
7.设置默认上下文：
[root@master-1 ~]# kubectl config use-context kubernetes
Switched to context "kubernetes".
8.使用Kubectl工具查看当前状态：
[root@master-1 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"} Node节点部署
1.安装所需的服务
将kubernetes-node-linux-amd64.tar.gz压缩包在node节点解压，执行如下操作
[root@k8s-node-1 ~]# cd /tmp/kubernetes/node/bin
[root@k8s-node-1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin/
[root@k8s-node-1 bin]# scp kubelet kube-proxy 192.168.20.48:/opt/kubernetes/bin/
[root@k8s-node-1 bin]# scp kubelet kube-proxy 192.168.20.49:/opt/kubernetes/bin/2.配置角色和认证参数
- 在master-1上创建角色绑定
[root@master-1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io "kubelet-bootstrap" created
- 创建kubelet bootstrapping kubeconfig文件，设置集群参数
[root@master-1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.20.50:6443 \
--kubeconfig=bootstrap.kubeconfig
Cluster "kubernetes" set.3.设置客户端认证参数
[root@master-1 ~]# kubectl config set-credentials kubelet-bootstrap \
--token=197f33fcbbfab2d15603dcc4408358f5 \
--kubeconfig=bootstrap.kubeconfig
User "kubelet-bootstrap" set.
4.设置上下文认证参数
[root@master-1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
Context "default" created.
5.选择默认上下文
[root@master-1 ~]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
Switched to context "default"6.执行上面的操作后，会在当前目录生成一个bootstrap.kubeconfig的config文件，将此文件分发到各节点：
[root@k8s-node-1 ~]# cp bootstrap.kubeconfig /opt/kubernetes/cfg/
[root@k8s-node-1 ~]# scp bootstrap.kubeconfig 192.168.20.47:/opt/kubernetes/cfg/
[root@k8s-node-1 ~]# scp bootstrap.kubeconfig 192.168.20.48:/opt/kubernetes/cfg/
[root@k8s-node-1 ~]# scp bootstrap.kubeconfig 192.168.20.49:/opt/kubernetes/cfg/
- 将master上更新的配置拷贝到其它master节点。
3.设置支持CNI
以下操作需要在所有node节点执行
- 设置Kubernetes对CNI的支持：
[root@k8s-node-2 ~]# mkdir -p /etc/cni/net.d
[root@k8s-node-2 ~]# vim /etc/cni/net.d/10-default.conf
{
"name": "flannel",
"type": "flannel",
"delegate": {
"bridge": "docker0",
"isDefaultGateway": true,
"mtu": 1400
}
}
4.配置Kubelet服务
以下操作需要在所有node节点执行
- 创建kubelet服务配置文件
[root@k8s-node-2 ~]# mkdir /var/lib/kubelet
[root@k8s-node-2 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
--address=192.168.20.48 \
--hostname-override=192.168.20.48 \
--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.1 \
--experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--cert-dir=/opt/kubernetes/ssl \
--network-plugin=cni \
--cni-conf-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/kubernetes/bin/cni \
--cluster-dns=10.1.0.2 \
--cluster-domain=cluster.local. \
--hairpin-mode hairpin-veth \
--allow-privileged=true \
--fail-swap-on=false \
--logtostderr=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
- 启动Kubelet
[root@k8s-node-2 ~]# systemctl daemon-reload
[root@k8s-node-2 ~]# systemctl start kubelet
[root@k8s-node-2 ~]# systemctl enable kubelet
[root@k8s-node-2 ~]# systemctl status kubelet
- 在master节点上查看是否收到node节点的csr请求：
[root@master-1 ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-FDH7Y3rghf1WPsEJH2EYnofvOSeyHn2f-l_-4rH-LEk 2m kubelet-bootstrap Pending
- 批准kubelet的TLS请求
[root@master-1 ~]# kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve
certificatesigningrequest.certificates.k8s.io "node-csr-FDH7Y3rghf1WPsEJH2EYnofvOSeyHn2f-l_-4rH-LEk" approved
[root@kmaster-1 ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-FDH7Y3rghf1WPsEJH2EYnofvOSeyHn2f-l_-4rH-LEk 11m kubelet-bootstrap Approved,Issued
- 之后查看node节点状态：
[root@master-1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.20.48 Ready 35s v1.14.1
node节点上查看kubelet 服务
[root@k8s-node-2 ~]# netstat -lntp|grep kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 7917/kubelet
tcp 0 0 192.168.20.32:10250 0.0.0.0:* LISTEN 7917/kubelet
tcp 0 0 192.168.20.32:10255 0.0.0.0:* LISTEN 7917/kubelet
tcp 0 0 192.168.20.32:4194 0.0.0.0:* LISTEN 7917/kubelet
5.部署kube-proxy
1.配置kube-proxy使用LVS，所有节点执行：
yum install -y ipvsadm ipset conntrack2.创建证书请求
[root@master-1 ~]# vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}3.生成证书
[root@master-1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy4.分发证书到所有node节点
[root@master-1 ~]# cp kube-proxy*.pem /opt/kubernetes/ssl/
[root@master-1 ~]# scp kube-proxy*.pem 192.168.20.47:/opt/kubernetes/ssl/
[root@master-1 ~]# scp kube-proxy*.pem 192.168.20.48:/opt/kubernetes/ssl/
[root@master-1 ~]# scp kube-proxy*.pem 192.168.20.49:/opt/kubernetes/ssl/5.创建kube-proxy配置文件
[root@k8s-node-2 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.20.50:6443 \
--kubeconfig=kube-proxy.kubeconfig
Cluster "kubernetes" set.
6.创建kube-proxy用户：
[root@k8s-node-2 ~]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \
--client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.7.设置默认上下文：
[root@k8s-node-2 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
Context "default" created.
8.切换上下文为default:
[root@k8s-node-2 ~]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
Switched to context "default".9.分发kube-proxy.kubeconfig配置文件到所有
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.44:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.45:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.46:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.47:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.48:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.459/opt/kubernetes/cfg/
10.创建kube-proxy服务配置文件
所有节点执行，注意配置文件中IP需要修改为本机对应的IP
[root@k8s-node-1 ~]# mkdir /var/lib/kube-proxy
[root@k8s-node-1 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \
--bind-address=192.168.20.47 \
--hostname-override=192.168.20.47 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \
--masquerade-all \
--feature-gates=SupportIPVSProxyMode=true \
--proxy-mode=ipvs \
--ipvs-min-sync-period=5s \
--ipvs-sync-period=5s \
--ipvs-scheduler=rr \
--logtostderr=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
11.启动服务
systemctl start kube-proxy
systemctl enable kube-proxy
systemctl status kube-proxy
12.查看服务状态,lvs状态
[root@k8s-node-1 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.0.1:443 rr
-> 192.168.20.44:6443 Masq 1 0 0
-> 192.168.20.45:6443 Masq 1 0 0
-> 192.168.20.46:6443 Masq 1 1 0
所有node节点配置成功后可以看到如下结果：
[root@master-1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.20.47 Ready 6d21h v1.14.1
192.168.20.48 Ready 4d1h v1.14.1
192.168.20.49 Ready 4d1h v1.14.1
Flannel 网络部署
所有节点都需要部署flannel。
1.创建Flannel证书
1.生成证书文件
[root@master-1 ~]# vim flanneld-csr.json
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2.生成证书
[root@master-1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
3.分发证书
[root@master-1 ~]# cp flanneld*.pem /opt/kubernetes/ssl/
[root@master-1 ~]# scp flanneld*.pem {all-k8s-node}:/opt/kubernetes/ssl/2.部署flannel
1.将之前下载的flannel压缩包解压，并分发到其它节点如下操作：
cp mk-docker-opts.sh flanneld /opt/kubernetes/bin/
scp mk-docker-opts.sh flanneld {all-k8s-node}:/opt/kubernetes/bin/
2.创建如下文件，分发到各个node节点：
[root@k8s-node-1 tmp]# vim remove-docker0.sh
#!/bin/bash
# Delete default docker bridge, so that docker can start with flannel network.
# exit on any error
set -e
rc=0
ip link show docker0 >/dev/null 2>&1 || rc="$?"
if [[ "$rc" -eq "0" ]]; then
ip link set dev docker0 down
ip link delete docker0
fi
[root@k8s-node-1 tmp]# cp remove-docker0.sh /opt/kubernetes/bin/
[root@k8s-node-1 tmp]# scp remove-docker0.sh 192.168.20.48:/opt/kubernetes/bin/
[root@k8s-node-1 tmp]# scp remove-docker0.sh 192.168.20.49:/opt/kubernetes/bin/3.配置flannel
[root@k8s-node-1 ~]# vim /opt/kubernetes/cfg/flannel
FLANNEL_ETCD="-etcd-endpoints=https://192.168.20.31:2379,https://192.168.20.32:2379,https://192.168.20.33:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"- 创建flannel服务文件
[root@k8s-node-1 ~]# vim /usr/lib/systemd/system/flannel.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
Before=docker.service
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/flannel
ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh
ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker
Type=notify
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service5.分发创建的配置文件到各个节点：
scp /opt/kubernetes/cfg/flannel {all-k8s-node}:/opt/kubernetes/cfg/
scp /usr/lib/systemd/system/flannel.service {all-k8s-node}:/usr/lib/systemd/system/
3. Flannel CNI集成
1.下载CNI插件
wget https://github.com/containernetworking/plugins/releases/download/v0.7.5/cni-plugins-amd64-v0.7.5.tgz
[root@k8s-node-1 tmp]# mkdir /opt/kubernetes/bin/cni
[root@k8s-node-1 tmp]# tar xf cni-plugins-amd64-v0.7.5.tgz -C /opt/kubernetes/bin/cni
2.分发软件到各个节点：
[root@k8s-node-1 ~]# scp -r /opt/kubernetes/bin/cni/* {all-k8s-node}:/opt/kubernetes/bin/cni/3.在etcd中创建key
[root@master-1 ~]# /opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem \
--no-sync -C https://192.168.20.44:2379,https://192.168.20.45:2379,https://192.168.20.46:2379 \
mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null 2>&1
4.各个节点启动flannel
[root@k8s-node-1 ~]# chmod +x /opt/kubernetes/bin/*
[root@k8s-node-1 ~]# systemctl daemon-reload
[root@k8s-node-1 ~]# systemctl start flannel
[root@k8s-node-1 ~]# systemctl enable flannel
配置Docker使用Flannel
1.修改docker的systemd的文件：
[Unit] #在Unit下面修改After和增加Requires
After=network-online.target firewalld.service flannel.service
Wants=network-online.target
Requires=flannel.service
[Service] #增加EnvironmentFile=-/run/flannel/docker
Type=notify
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_OPTS2.其它NODE节点也做相同的修改
[root@k8s-node-2 ~]# scp /usr/lib/systemd/system/docker.service {k8s-node}:/usr/lib/systemd/system/3.重启docker, 出现docker0网卡，且在10.2.0.0/16网段，说明配置成功
[root@k8s-node-3 ~]# systemctl daemon-reload
[root@k8s-node-3 ~]# systemctl restart docker
[root@k8s-node-3 ~]# ip a| grep -A 3 'docker0'
7: docker0: mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:e9:2b:36:86 brd ff:ff:ff:ff:ff:ff
inet 10.2.79.1/24 scope global docker0
valid_lft forever preferred_lft forever
插件部署
1.创建CoreDNS
- 创建coredns.yaml，内容如下：
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: Reconcile
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: EnsureExists
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local. in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: coredns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: coredns
template:
metadata:
labels:
k8s-app: coredns
spec:
serviceAccountName: coredns
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
- key: "CriticalAddonsOnly"
operator: "Exists"
containers:
- name: coredns
image: coredns/coredns:1.4.0
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: coredns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: coredns
clusterIP: 10.1.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- 执行此文件：
[root@master-1 tmp]# kubectl create -f coredns.yaml- 确认DNS服务运行：
[root@master-1 ~]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-76fcfc9f65-9fkfh 1/1 Running 2 3d7h 10.2.45.3 192.168.20.49
coredns-76fcfc9f65-zfplt 1/1 Running 1 3d6h 10.2.24.2 192.168.20.48
2. 部署Dashboard
1.执行目录中的yaml,部署Dashboard:
[root@master-1 ~]# ll /tmp/dashboard/
total 20
-rw-r--r-- 1 root root 356 Jul 27 03:43 admin-user-sa-rbac.yaml
-rw-r--r-- 1 root root 4253 Jul 27 03:47 kubernetes-dashboard.yaml
-rw-r--r-- 1 root root 458 Jul 27 03:49 ui-admin-rbac.yaml
-rw-r--r-- 1 root root 477 Jul 27 03:50 ui-read-rbac.yaml
[root@master-1 ~]# kubectl create -f /tmp/dashboard/2.确认服务是否正常运行：
[root@master-1 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-76fcfc9f65-9fkfh 1/1 Running 2 3d7h
coredns-76fcfc9f65-zfplt 1/1 Running 1 3d6h
kubernetes-dashboard-68ddcc97fc-w4bxf 1/1 Running 1 3d2h
[root@master-1 ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.20.50:6443
CoreDNS is running at https://192.168.20.50:6443/api/v1/namespaces/kube-system/services/coredns:dns/proxy
kubernetes-dashboard is running at https://192.168.20.50:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
3.根据提示信息，使用dashboard的url,登录，账号admin/admin, 使用如下命令生成token:
[root@master-1 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')4.复制token,选择使用令牌的方式登录：
3. Heapster 部署 (可选)
1.使用如下文件部署Heastper:
[root@master-1 ~]# ll heastper/
total 12
-rw-r--r-- 1 root root 2306 Jul 26 20:28 grafana.yaml
-rw-r--r-- 1 root root 1562 Jul 26 20:29 heapster.yaml
-rw-r--r-- 1 root root 1161 Jul 26 20:29 influxdb.yaml
[root@k8s-node-1 ~]# kubectl create -f heastper/- 登录dashboard，查看石头出现资源利用率的图表。
- 使用
kubectl cluster-info命令，查看当前服务的url地址。
补充说明
etcd无证书配置说明
在实际的生产环境中，如果都是使用的内网环境，可以将etd集群配置为无证书的模式，这样在配置和后续的故障恢复中会更简单。
etcd无证书配置需要使用http访问，安装上述文档，需要修改如下配置：
- etcd的配置文件注释掉安全证书部分，并将所有url改为http方式：
# cat /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="http://192.168.20.31:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.20.31:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.20.31:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=http://192.168.20.31:2380,etcd-node-2=http://192.168.20.32:2380,etcd-node-3=http://192.168.20.33:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.20.31:2379"
#[security]
#CLIENT_CERT_AUTH="true"
#ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
#ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
#ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
#PEER_CLIENT_CERT_AUTH="true"
#ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
#ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
#ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"- fannel网络部分注释掉etcd的证书配置参数，并将URL该位http:
# cat /opt/kubernetes/cfg/flannel
FLANNEL_ETCD="-etcd-endpoints=http://192.168.20.31:2379,http://192.168.20.32:2379,http://192.168.20.33:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
#FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
#FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
#FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"
3.kube-apiserver中去掉etcd的证书配置，这个文件中需要把参数直接删除，并将url改为http:
# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
--bind-address=192.168.20.31 \
--insecure-bind-address=127.0.0.1 \
--authorization-mode=Node,RBAC \
--runtime-config=rbac.authorization.k8s.io/v1 \
--kubelet-https=true \
--anonymous-auth=false \
--basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
--service-cluster-ip-range=10.1.0.0/16 \
--service-node-port-range=20000-40000 \
--tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-servers=http://192.168.20.31:2379,http://192.168.20.32:2379,http://192.168.20.33:2379 \
--enable-swagger-ui=true \
--allow-privileged=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/log/api-audit.log \
--event-ttl=1h \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target4.分别重启flannel、kubelet、kube-apiserver等服务。
转载于:https://blog.51cto.com/tryingstuff/2385903