Kubernetes单节点部署

Kubernetes核心概念

1、Pod

●最小部署单元

●一组容器的集合

●一个Pod中的容器共享网络命名空间

●Pod是短暂的

2、Controllers

●ReplicaSet :确保 预期的Pod副本数量

●Deployment:无状态应用 部署

●StatefulSet :有状态应用部署

●DaemonSet:确保所有Node运行同一个Pod

●Job:一次性任务

●Cronjob:定时任务

更高级层次对象，部署和管理Pod

3、Service

●防止Pod失联

●定义一组Pod的访问策略

●Label: 标签，附加到某个资源上，用于关联对象、查询和筛选

●Namespaces :命 名空间，将对象逻辑上隔离

●Annotations :注释

单节点部署集群架构图

单节点部署思路

k8s单节点部署思路：
第一部分

1：自签ETCD证书
2：ETCD部署
3：Node安装docker
4：Flannel部署（先写入子网到etcd）

第二部分（master）

1：自签APIServer证书
2：部署APIServer组件（token，csv）
3：部署controller-manager（指定apiserver证书）和scheduler组件

第三部分（node）

1：生成kubeconfig（bootstrap，kubeconfig和kube-proxy.kubeconfig）
2：部署kubelet组件
3：部署kube-proxy组件

第四部分（加入群集）

1：kubectl get csr && kubectl certificate approve 允许颁发证书，加入群集
2：添加一个node节点
3：查看kubectl get node 节点

单节点部署

Master节点
master1:192.168.100.110
master2:192.168.100.109
Node节点
node1:192.168.100.111
node2:192.168.100.112

Master操作

[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/
[root@localhost k8s]# ls    //从宿主机拖进来
etcd-cert.sh  etcd.sh
[root@localhost k8s]# mkdir etcd-cert
[root@localhost k8s]# mv etcd-cert.sh etcd-cert

下载官方包

[root@localhost k8s]# vim cfssl.sh

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

[root@localhost k8s]# bash cfssl.sh  //下载cfssl官方包
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9.8M  100  9.8M    0     0  77052      0  0:02:14  0:02:14 --:--:-- 94447
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2224k  100 2224k    0     0  66701      0  0:00:34  0:00:34 --:--:-- 71949
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6440k  100 6440k    0     0  74368      0  0:01:28  0:01:28 --:--:-- 93942

[root@localhost k8s]# ls /usr/local/bin/
cfssl  cfssl-certinfo  cfssljson
//cfssl 生成证书工具、cfssljson通过传入json文件生成证书、cfssl-certinfo查看证书信息

部署ETCD

[root@localhost k8s]# cd etcd-cert/
cat > ca-config.json <{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"     
        ]  
      } 
    }         
  }
}
EOF 
//实现证书签名
cat > ca-csr.json <{   
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生产证书，生成ca-key.pem ca.pem

//生产证书，生成ca-key.pem  ca.pem
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

指定etcd三个节点之间的通信

cat > server-csr.json <{
    "CN": "etcd",
    "hosts": [
    "192.168.100.110",
    "192.168.100.111",
    "192.168.100.112"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

//生成ETCD证书 server-key.pem server.pem

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

将软件包复制到Master里

[root@localhost etcd-cert]# cd /root/k8s/
[root@localhost k8s]# ls              //直接拉取到目录下
cfssl.sh   etcd.sh                          flannel-v0.10.0-linux-amd64.tar.gz
etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz

解压etcd

[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz

[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md

[root@localhost k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p    //配置文件，命令文件，证书

[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/

使用另外一个会话打开，会发现etcd进程已经开启

[root@localhost etcd-cert]# ps -ef | grep etcd

拷贝证书到node节点上

[root@localhost k8s]# scp -r /opt/etcd/ [email protected]:/opt/
[root@localhost k8s]# scp -r /opt/etcd/ [email protected]:/opt

分别在node节点上修改

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.111:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.111:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.111:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.111:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.100.110:2380,etcd02=https://192.168.100.111:2380,etcd03=https://192.168.100.112:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.112:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.112:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.112:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.112:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.100.110:2380,etcd02=https://192.168.100.111:2380,etcd03=https://192.168.100.112:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

在Master上开启服务

[root@localhost k8s]# bash etcd.sh etcd01 192.168.100.110 etcd02=https://192.168.100.111:2380,etcd03=https://192.168.100.112:2380

在节点服务器node1和node2上开启服务(如果这里启动报错可以试试关闭防火墙)

[root@localhost ~]# systemctl start etcd
[root@localhost ~]# systemctl status etcd

检查群集状态

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.100.110:2379,https://192.168.100.111:2379,https://192.168.100.112:2379" cluster-health
member 32293bbc65784dda is healthy: got healthy result from https://192.168.100.110:2379
member 703725a0e421bc44 is healthy: got healthy result from https://192.168.100.111:2379
member da99aab5a563863a is healthy: got healthy result from https://192.168.100.112:2379
cluster is healthy

Node节点部署

1、安装依赖包

yum install -y yum-utils device-mapper-persistent-data lvm2

2、设置阿里云镜像源

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3、安装Docker-CE

yum install -y docker-ce

4、关闭防火墙

systemctl stop firewalld.service 
setenforce 0

5、开启服务并设为开机自启

systemctl start docker.service 
systemctl enable docker.service 

镜像加速
tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://cz1numin.mirror.aliyuncs.com"]
}
EOF

重载并重启

systemctl daemon-reload
systemctl restart docker

网络优化
vim /etc/sysctl.conf
net.ipv4.ip_forward=1
sysctl -p
service network restart
systemctl restart docker

Flanner部署
1、写入分配的子网段到ETCD中，供flannel使用

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.100.110:2379,https://192.168.100.111:2379,https://192.168.100.112:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

2、查看写入的信息

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.100.110:2379,https://192.168.100.111:2379,https://192.168.100.112:2379" get /coreos.com/network/config
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

3、拷贝到所有node节点（只需要部署在node节点即可）

[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz [email protected]:/root
[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz [email protected]:/root

4、所有node节点操作解压

[root@localhost ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz 
flanneld
mk-docker-opts.sh
README.md

node1操作

[root@localhost ~]# ls /opt/
containerd  etcd  rh
[root@localhost ~]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p
[root@localhost ~]# ls /opt/
containerd  etcd  kubernetes  rh
[root@localhost ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/
[root@localhost ~]# ls /opt/kubernetes/bin/
flanneld  mk-docker-opts.sh
[root@localhost ~]# vim flannel.sh

#!/bin/bash

 

ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}

 

cat </opt/kubernetes/cfg/flanneld

 

FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \

-etcd-cafile=/opt/etcd/ssl/ca.pem \

-etcd-certfile=/opt/etcd/ssl/server.pem \

-etcd-keyfile=/opt/etcd/ssl/server-key.pem"

 

EOF

 

cat </usr/lib/systemd/system/flanneld.service

[Unit]

Description=Flanneld overlay address etcd agent

After=network-online.target network.target

Before=docker.service

 

[Service]

Type=notify

EnvironmentFile=/opt/kubernetes/cfg/flanneld

ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS

ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env

Restart=on-failure

 

[Install]

WantedBy=multi-user.target

 

EOF

 

systemctl daemon-reload

systemctl enable flanneld

systemctl restart flanneld

2、开启flannel网络功能

bash flannel.sh https://192.168.100.110:2379,https://192.168.100.111:2379,https://192.168.100.112:2379

3、配置docker连接flannel

[root@localhost ~]# vim /usr/lib/systemd/system/docker.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

4、说明：bip指定启动时的子网

[root@localhost ~]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.23.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.23.1/24 --ip-masq=false --mtu=1450"

5、重启docker服务

[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl restart docker

6、创建容器并测试网络互通性

[root@localhost ~]# docker run -it centos:7 /bin/bash
Unable to find image 'centos:7' locally
7: Pulling from library/centos
ab5ef0e58194: Pull complete 
Digest: sha256:4a701376d03f6b39b8c2a8f4a8e499441b0d567f9ab9d58e4991de4472fb813c
Status: Downloaded newer image for centos:7
[root@809ba7f7fdbe /]#  yum install net-tools -y               #安装工具后才能使用ifconfig命令
[root@localhost ~]# ping 172.17.2.2
PING 172.17.2.2(172.17.2.2) 56(84) bytes of data.
64 bytes from 172.17.56.1: icmp_seq=1 ttl=64 time=0.307 ms
64 bytes from 172.17.56.1: icmp_seq=2 ttl=64 time=0.385 ms
64 bytes from 172.17.56.1: icmp_seq=3 ttl=64 time=0.302 ms

ApiServer证书

root@localhost k8s]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p
[root@localhost k8s]# mkdir k8s-cert    //apiserver自签证书目录
[root@localhost k8s]# cd k8s-cert/
[root@localhost k8s-cert]# ls
k8s-cert.sh

cat > ca-config.json <{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
      	    "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------

cat > server-csr.json <{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.100.110",  //master1
      "192.168.100.109",  //master2
      "192.168.100."100,  //vip
      "192.168.100.109",  //lb （master）
      "192.168.100.113",  //lb （backup）
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

#-----------------------

cat > admin-csr.json <{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#-----------------------

cat > kube-proxy-csr.json <{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

2、生成k8s证书

[root@localhost k8s-cert]#  bash k8s-cert.sh
[root@localhost k8s-cert]# ls *pem
admin-key.pem  ca-key.pem  kube-proxy-key.pem  server-key.pem
admin.pem      ca.pem      kube-proxy.pem      server.pem
[root@localhost k8s-cert]# cp ca*pem server*pem /opt/kubernetes/ssl/

3、解压kubernetes压缩包

[root@localhost k8s-cert]# cd ..
[root@localhost k8s]# ls
apiserver.sh                     flannel-v0.10.0-linux-amd64.tar.gz
controller-manager.sh            k8s-cert
etcd-cert                        kubernetes-server-linux-amd64.tar.gz
etcd.sh                          master.zip
etcd-v3.3.10-linux-amd64         scheduler.sh
etcd-v3.3.10-linux-amd64.tar.gz
[root@localhost k8s]# tar zxvf kubernetes-server-linux-amd64.tar.gz 

4、复制关键命令文件

[root@localhost k8s]# cd /root/k8s/kubernetes/server/bin/
[root@localhost bin]# ls
apiextensions-apiserver              kube-controller-manager.tar
cloud-controller-manager             kubectl
cloud-controller-manager.docker_tag  kubelet
cloud-controller-manager.tar         kube-proxy
hyperkube                            kube-proxy.docker_tag
kubeadm                              kube-proxy.tar
kube-apiserver                       kube-scheduler
kube-apiserver.docker_tag            kube-scheduler.docker_tag
kube-apiserver.tar                   kube-scheduler.tar
kube-controller-manager              mounter
kube-controller-manager.docker_tag
[root@localhost bin]# cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/

使用 head -c 16 /dev/urandom | od -An -t x | tr -d ’ ’ 可以随机生成序列号

[root@localhost bin]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
010b7997c2e258ac9581ec07b66dfd2c
[root@localhost k8s]# vim /opt/kubernetes/cfg/token.csv
99fdfb838a63f2ee98d0c746bdba80b9,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

二进制文件，token，证书都准备好，开启apiserver

[root@localhost k8s]# bash apiserver.sh 192.168.100.110 https://192.168.100.1102379,https://192.168.100.111:2379,https://192.168.100.112:2379
[root@localhost k8s]# ps aux | grep kube

查看端口

[root@localhost bin]# netstat -ntap | grep 6443
tcp        0      0 192.168.100.110:6443    0.0.0.0:*               LISTEN      54075/kube-apiserve
tcp        0      0 192.168.100.110:43082   192.168.100.110:6443    ESTABLISHED 54075/kube-apiserve
tcp        0      0 192.168.100.110:6443    192.168.100.110:43082   ESTABLISHED 54075/kube-apiserve

部署controller-manager（指定apiserver证书）和scheduler组件

1、启动scheduler服务

[root@localhost k8s]# ./scheduler.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@localhost k8s]# ps aux | grep ku

2、启动controller-manager

[root@localhost k8s]#  chmod +x controller-manager.sh
[root@localhost k8s]# ./controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

3、查看master 节点状态

[root@localhost k8s]#  /opt/kubernetes/bin/kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   

生成kubeconfig（bootstrap，kubeconfig和kube-proxy.kubeconfig）
master上操作

1、把 kubelet、kube-proxy拷贝到node节点上去

[root@localhost k8s]# cd kubernetes/server/bin/
[root@localhost bin]# ls
apiextensions-apiserver              kube-controller-manager.tar
cloud-controller-manager             kubectl
cloud-controller-manager.docker_tag  kubelet
cloud-controller-manager.tar         kube-proxy
hyperkube                            kube-proxy.docker_tag
kubeadm                              kube-proxy.tar
kube-apiserver                       kube-scheduler
kube-apiserver.docker_tag            kube-scheduler.docker_tag
kube-apiserver.tar                   kube-scheduler.tar
kube-controller-manager              mounter
kube-controller-manager.docker_tag
[root@localhost bin]# scp kubelet kube-proxy [email protected]:/opt/kubernetes/bin/
[root@localhost bin]# scp kubelet kube-proxy [email protected]:/opt/kubernetes/bin/

nod01节点操作

1、复制node.zip到/root目录下再解压

[root@localhost ~]# ls
anaconda-ks.cfg                     initial-setup-ks.cfg  公共  图片  音乐
flannel.sh                          node.zip              模板  文档  桌面
flannel-v0.10.0-linux-amd64.tar.gz  README.md             视频  下载
[root@localhost ~]# unzip node.zip 
Archive:  node.zip
  inflating: proxy.sh                
  inflating: kubelet.sh    

master操作

1、创建工作目录

[root@localhost k8s]#  mkdir kubeconfig
[root@localhost k8s]# cd kubeconfig/
[root@localhost kubeconfig]# ls
kubeconfig.sh               

kubeconfig.sh脚本如下 ：

#创建 TLS Bootstrapping Token
#BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ’ ')
BOOTSTRAP_TOKEN=0fb61c46f8991b718eb38d27b605b008

cat > token.csv < ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,“system:kubelet-bootstrap”
EOF

#---------------------- #此部分删掉
2、获取token信息

[root@localhost bin]# cat /opt/kubernetes/cfg/token.csv

99fdfb838a63f2ee98d0c746bdba80b9,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

3、配置文件修改为tokenID

[root@localhost kubeconfig]# vim kubeconfig.sh
kubectl config set-credentials kubelet-bootstrap \
  --token=99fdfb838a63f2ee98d0c746bdba80b9 \
  --kubeconfig=bootstrap.kubeconfig

4、设置环境变量（可以写入到/etc/profile中）

[root@localhost kubeconfig]# vim /etc/profile

//在末行插入

export PATH=$PATH:/opt/kubernetes/bin/

[root@localhost kubeconfig]# source /etc/profile                #声明

[root@localhost kubeconfig]# kubectl get cs              #查看状态
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   

5、重命名并生成配置文件

[root@localhost kubeconfig]# mv kubeconfig.sh kubeconfig
[root@localhost kubeconfig]# ls
kubeconfig
[root@localhost kubeconfig]# vim kubeconfig 
[root@localhost kubeconfig]# bash kubeconfig 192.168.100.111 /root/k8s/k8s-cert/
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".

[root@localhost kubeconfig]# ls
bootstrap.kubeconfig  kubeconfig  kube-proxy.kubeconfig

6、拷贝配置文件到node节点

[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig [email protected]:/opt/kubernetes/cfg/
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig [email protected]:/opt/kubernetes/cfg/

7、创建bootstrap角色赋予权限用于连接apiserver请求签名（关键）

[root@localhost kubeconfig]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

部署kubelet组件
在node01节点上操作

1、执行脚本

[root@localhost ~]# bash kubelet.sh 192.168.100.111
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

2、检查kubelet服务启动并查看状态

[root@localhost ~]# ps aux | grep kube
[root@localhost ~]# systemctl status kubelet.service 

master上操作

1、检查到node1节点的请求

[root@localhost kubeconfig]# kubectl get csr

2、同意请求并颁发证书

[root@localhost kubeconfig]# kubectl certificate approve node-csr-HF551kae9hkzNGwirEbd2uAiStXnawpEI-y7nLP4sTU

3、继续查看证书状态

[root@localhost kubeconfig]# kubectl get csr

4、查看群集节点，成功加入node01节点

[root@localhost kubeconfig]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.100.111 Ready 2m18s v1.12.3
部署kube-proxy组件
在node1节点操作
1、启动proxy服务

[root@localhost ~]# bash proxy.sh 192.168.100.111
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

把现成的/opt/kubernetes目录复制到其他节点进行修改即可

[root@localhost ~]# scp -r /opt/kubernetes/ [email protected]:/opt/

2、把kubelet，kube-proxy的service文件拷贝到node2中

[root@localhost ~]# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service [email protected]:/usr/lib/systemd/system/

第四部分（加入群集）
在node02上操作，进行修改

1、首先删除复制过来的证书，等会node02会自行申请证书

[root@localhost ~]# cd /opt/kubernetes/ssl/
[root@localhost ssl]# rm -rf *

2、修改配置文件kubelet kubelet.config kube-proxy（三个配置文件）

[root@localhost ssl]# cd ../cfg/
[root@localhost cfg]# ls
bootstrap.kubeconfig  kubelet         kubelet.kubeconfig  kube-proxy.kubeconfig
flanneld              kubelet.config  kube-proxy

[root@localhost cfg]# vim kubelet

KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.100.112 \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

[root@localhost cfg]# vim kubelet.config 

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.100.112 
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true

[root@localhost cfg]# vim kube-proxy

KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.100.112 2 \
--cluster-cidr=10.0.0.0/24 \
--proxy-mode=ipvs \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"

3、启动服务

[root@localhost cfg]#  systemctl start kubelet.service
[root@localhost cfg]#  systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@localhost cfg]#  systemctl start kube-proxy.service 
[root@localhost cfg]#  systemctl enable kube-proxy.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

在master上操作

1、查看请求

[root@localhost kubeconfig]# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-HF551kae9hkzNGwirEbd2uAiStXnawpEI-y7nLP4sTU   32m     kubelet-bootstrap   Approved,Issued
node-csr-TszRzIg1T7JcfFKuwrCb82bqv7mcwvX1xZD8KMQJZCY   2m25s   kubelet-bootstrap   Pending

2、授权许可加入群集

[root@localhost kubeconfig]# kubectl certificate approve node-csr-TszRzIg1T7JcfFKuwrCb82bqv7mcwvX1xZD8KMQJZCY
certificatesigningrequest.certificates.k8s.io/node-csr-TszRzIg1T7JcfFKuwrCb82bqv7mcwvX1xZD8KMQJZCY approved

3、查看群集中的节点

[root@localhost kubeconfig]# kubectl get node
NAME             STATUS     ROLES    AGE   VERSION
192.168.35.101   Ready         21m   v1.12.3
192.168.35.102   NotReady      4s    v1.12.3