一，kunernetes部署方式和环境介绍

1.1，生产环境可部署kubenetes集群的两种方式

1.1.1 kubeadm

    kubeadm是k8s部署的一个工具，提供了kubeadm init和kubeadm join 用于快速部署k8s集群

1.1.2 二进制部署

从github上下载发行的版的kubernetes二进制包，手动部署每个组件，组成kubernetes集群
kubeadm降低了部署门槛，但是屏蔽了很多细节，但遇到问题比较难排查，如果想要更好的控制k8s集群推荐使用二进制的方式部署kubernetes集群，二进制部署相对来说是比较麻烦，部署的期间能学习很多工作原理，也比较有利于后期的维护。

1.2 部署kubernetes集群需要以下几个条件：

1.一台机器或者多台机器，操作系统centos7.x-86_x64
2.硬件配置：master和node至少是2G内存或者更高的内存cpu至少2个甚至更多，硬盘50GB或者更多，资源少了启动k8s集群可能会起不来
3.必须是可以访问外网，在部署集群时需要拉取镜像，如果不能上网需要提前下载好镜像导入节点
4.禁止swap分区，k8s不支持swap分区。

1.3 环境准备

软件环境：

软件                 版本
操作系统：           centos7.6_x64
Docker              19.3-ce
Kubernetes          1.18

服务器规划：

节点：            ip                                 组件                           
master： 10.0.0.11   kube-apiserver ,  kube-controller-manager kube-scheduler , etcd , docker
node01： 10.0.0.12   kubelet , kube-proxy , docker , etcd
node02:  10.0.0.13   kubelet , kube-proxy , docker , etcd

架构图：

image.png

单master或者多master

1.4 系统初始化配置

关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
关闭selinux
临时关闭：
setenforce 0
永久关闭： 
sed -i 's/enforcing/disabled/' /etc/selinux/config
关闭swap分区
临时关闭：
swapoff -a
永久关闭：
sed -ri 's/.*swap.*/#&/' /etc/fstab
主机名规划设置 
hostnamectl set-hostname 
桥接的ipv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  #生效
添加hosts文件
cat > /etc/hosts <

 
 二， 部署etcd集群 
 etcd是一个分布式的键值存储系统，kubernetes使用etcd进行数据存储。所以要首先准备一个etcd的数据库，为了解决单点的故障我们部署etcd时采用集群的方式部署这里使用3台机器部署可以有一台机器故障，也可以采用5台机器有两台 可以有2台机器故障 
 10.0.0.11    etcd1
10.0.0.12    etcd2
10.0.0.13    etcd3
 
 为了节省机器，我们把与k8s节点的机器进行复用，也可以单独部署k8s集群，apiserver能连接上就没有问题。 
 2.1 准备cfssl证书生成工具 
 cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl更方便使用 
 找一个服务器操作，我这里使用master节点
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
 
 2.2 生成etcd证书 
 创建工作目录
mkdir -p ~/TLS/{etcd,k8s}
cd TLS/etcd
自签CA：
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
        "expiry": "87600h",
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF
 
 生成证书： 
 cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@master etcd]# ll *.pem
-rw------- 1 root root 1679 Nov 14 11:09 ca-key.pem
-rw-r--r-- 1 root root 1265 Nov 14 11:09 ca.pem
使用自签CA 签发etcd HTTPS证书
创建证书申请文件：
cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "10.0.0.11",
    "10.0.0.12",
    "10.0.0.13"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF
注：上述的hosts字段IP为所有节点etcd的集群内部通信IP一个都不能少，可以多预留几个为方便以后扩容用。
生成证书：
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
[root@master etcd]# ll server*pem
-rw------- 1 root root 1679 Nov 14 11:17 server-key.pem
-rw-r--r-- 1 root root 1338 Nov 14 11:17 server.pem
 
 2.3 从github下载二进制文件 
 下载地址：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
 
 2.4 部署etcd集群 
 以下操作在节点1操作，简化操作会将节点1生成的文件全部copy到节点2和节点3
 创建目录并解压二进制包 
 mkdir -p /opt/etcd/{bin,cfg,ssl}
tar -xf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
 
 创建etcd配置文件 
 cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.11:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.11:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.0.0.11:2380,etcd-2=https://10.0.0.12:2380,etcd-3=https://10.0.0.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
 
 systemd管理etcd 
 cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
 
 copy刚刚生成的证书： 
 cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/
 
 将上面的节点1所有生成的文件copy到节点2和节点3： 
 scp -r /opt/etcd/ [email protected]:/opt/
scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system/
scp -r /opt/etcd/ [email protected]:/opt/
scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system/
 
 然后在节点2和节点3分别修改etcd.conf配置文件： 
 node1：
[root@node1 ~]# cat /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.12:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.12:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.0.0.11:2380,etcd-2=https://10.0.0.12:2380,etcd-3=https://10.0.0.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
node2：
[root@node2 ~]# cat /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.13:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.13:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.13:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.13:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.0.0.11:2380,etcd-2=https://10.0.0.12:2380,etcd-3=https://10.0.0.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
 
 然后启动并开机自启动： 
 systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
 
 查看集群状态： 
 [root@master ~]# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379" endpoint health
https://10.0.0.11:2379 is healthy: successfully committed proposal: took = 42.25622ms
https://10.0.0.13:2379 is healthy: successfully committed proposal: took = 47.996367ms
https://10.0.0.12:2379 is healthy: successfully committed proposal: took = 46.980311ms
输出的是上面的信息就说明部署集群成功，如有不成功先看日志、var/log/messages或者journalctl -u etcd
 
 三，安装docker 
 下载地址：https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
 
 以下在所有节点操作。这里采用二进制安装，用yum安装也一样。 
 3.1 解压二进制包 
 tar -xf docker-19.03.9.tgz
mv docker/* /usr/bin
 
 3.2 system管理docker 
 cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
 
 3.3 配置阿里云加速 
 mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
 
 3.4 启动并设置开机自启动 
 systemctl daemon-reload
systemctl start docker
systemctl enable docker
 
 四，部署Master Node 
 4.1 生成kube-apiserver证书 
 1，自签证书颁发机构（CA） 
 cd TLS/k8s
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "expiry": "87600h",
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF
生成证书：
[root@master ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@master k8s]# ll *pem
-rw------- 1 root root 1679 Nov 14 16:05 ca-key.pem
-rw-r--r-- 1 root root 1359 Nov 14 16:05 ca.pem
 
 2，使用自签CA签发kube-apiserver HTTPS证书
 创建证书申请文件： 
 cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "10.0.0.11",
      "10.0.0.12",
      "10.0.0.13",
      "10.0.0.14",
      "10.0.0.15",
      "10.0.0.16",
      "10.0.0.17",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF
注：上述文件hosts字段中IP为所有master/LB/VIP IP 一个都不能少 为了以后方便扩容可以多预留几个IP
生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
[root@master k8s]# ll server*pem
-rw------- 1 root root 1675 Nov 14 16:16 server-key.pem
-rw-r--r-- 1 root root 1659 Nov 14 16:16 server.pem
 
 4.2 从github下载二进制文件 
 下载地址： https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183
 
 注：打开连接你会发现有很多包，只需要下载一个server包就好了，里面包含了master和woker node二进制文件 
 4.3 解压二进制包 
 mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
tar xf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin/
cp kubectl /usr/bin/
 
 4.4 部署kube-apiserver 
 1，创建配置文件 
 cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379 \\
--bind-address=10.0.0.11 \\
--secure-port=6443 \\
--advertise-address=10.0.0.11 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF
注：上面\\ 第一个是转义符，第二个是换行符，使用转义符是为了使用EOF保留换行符。
--logtostderr：启用日志
---v：日志等级
--log-dir：日志目录
--etcd-servers：etcd集群地址
--bind-address：监听地址
--secure-port：https安全端口
--advertise-address：集群通告地址
--allow-privileged：启用授权
--service-cluster-ip-range：Service虚拟IP地址段
--enable-admission-plugins：准入控制模块
--authorization-mode：认证授权，启用RBAC授权和节点自管理
--enable-bootstrap-token-auth：启用TLS bootstrap机制
--token-auth-file：bootstrap token文件
--service-node-port-range：Service nodeport类型默认分配端口范围
--kubelet-client-xxx：apiserver访问kubelet客户端证书
--tls-xxx-file：apiserver https证书
--etcd-xxxfile：连接Etcd集群证书
--audit-log-xxx：审计日志
 
 2，copy刚才生成的证书 
 cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/
 
 3，启用TLS Bootstrapping 机制
 TLS Bootstrapping ：master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同事也会增加集群扩展复杂度，为了简化流程，kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前用于kubelet，kube-proxy还收有我们统一发放证书。
 TLS bootstraping工作流程： 
 
 
  
   
     
    
   
  
    image.png 
   
  
 
 
 创建上述配置文件找那个的token文件： 
 cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF
 
 格式：token，用户名，UID，用户组
 token也可自行生成替换： 
 head -c 16 /dev/urandom | od -An -t x | tr -d ' '
 
 4，systemd管理apiserver 
 cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
 
 5，启动并设置开机自启动 
 systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver
 
 6，授权kubelet-bootstrap用户允许请求证书： 
 kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
 
 4.5 部署kube-controller-manager 
 1，创建配置文件 
 cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF
--master：通过本地非安全本地端口8080连接apiserver
--leader-elect：当该组件启动多个时自动选举（HA）
--cluster-signing-cert-file/--cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致
 
 2，systemd管理controller-manager 
 cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
 
 3.启动并设置开机自启动 
 systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager
 
 4.6, 部署kube-scheduler 
 1.创建配置文件 
 cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"
EOF
--master：通过本地非安全本地端口8080连接apiserver。
--leader-elect：当该组件启动多个时，自动选举（HA）
 
  
  systemd管理scheduler 
  
 cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
 
  
  启动并设置开机启动 
  
 systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler
 
 4.查看集群状态：
 所有组件都已经启动成功，通过kubectl工具查看当前集群组件状态： 
 [root@master bin]# kubectl get cs
NAME                STATUS    MESSAGE            ERROR
scheduler            Healthy  ok                 
controller-manager  Healthy  ok                 
etcd-1              Healthy  {"health":"true"} 
etcd-2              Healthy  {"health":"true"} 
etcd-0              Healthy  {"health":"true"} 
 
 如上结果说明master节点组件运行正常。 
 五，部署Woker Node 
 下面还是在master节点上操作，也作为node节点 
 5.1 创建工作目录 
 在所有woker节点创建工作目录： 
 mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
从master节点拷贝：
[root@master ~]# cd kubernetes/server/bin
[root@master bin]# cp kubelet kube-proxy /opt/kubernetes/bin
 
 5.2部署kubelet 
 1，创建配置文件 
 cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=k8s-master \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF
--hostname-override：显示名称，集群中唯一
--network-plugin：启用CNI
--kubeconfig：空路径，会自动生成，后面用于连接apiserver
--bootstrap-kubeconfig：首次启动向apiserver申请证书
--config：配置参数文件
--cert-dir：kubelet证书生成目录
--pod-infra-container-image：管理Pod网络容器的镜像
 
 2，配置参数文件 
 cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
 
 3，生成bootstrap.kubeconfig文件 
 KUBE_APISERVER="https://10.0.0.11:6443" # apiserver IP:PORT
TOKEN="c47ffb939f5ca36231d9e3121a252940" # 与token.csv里保持一致
# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
 
 拷贝到配置文件路径： 
 [root@master bin]# cp bootstrap.kubeconfig /opt/kubernetes/cfg
 
 4，systemd管理kubelet 
 cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
 
 5，启动并设置开机自启动 
 systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
 
 5.3，批准kubelet证书申请并加入集群 
 查看证书请求: 
 kubectl get csr
NAME                                                  AGE    SIGNERNAME                                    REQUESTOR          CONDITION
node-csr--G3jTzzjPOXVvyhS4Wt_UWvVr9YxHBSWVBjqbFD-dYo  2m4s  kubernetes.io/kube-apiserver-client-kubelet  kubelet-bootstrap  Pending
 
 批准申请： 
 kubectl certificate approve node-csr--G3jTzzjPOXVvyhS4Wt_UWvVr9YxHBSWVBjqbFD-dYo
 
 查看节点： 
 kubectl get nodes
NAME        STATUS    ROLES    AGE  VERSION
k8s-master  NotReady    37s  v1.18.3
 
 注：由于还没有部署网络插件节点就会没有准备就绪 NotReady 
 5.4 部署kube-proxy 
 1.创建配置文件 
 cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF
 
 2.配置参数文件： 
 cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master
clusterCIDR: 10.0.0.0/24
EOF
 
 3,生成kube-proxy.kubeconfig文件
 生成kube-proxy证书：
 切换目录： 
 cd /root/TLS/k8s
创建请求证书文件：
cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
 
 生成证书： 
 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
[root@master k8s]# ll kube-proxy*pem
-rw------- 1 root root 1679 Nov 14 17:19 kube-proxy-key.pem
-rw-r--r-- 1 root root 1403 Nov 14 17:19 kube-proxy.pem
 
 生成kubeconfig文件： 
 KUBE_APISERVER="https://10.0.0.11:6443"
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
 
 拷贝配置文件到指定路径： 
 cp kube-proxy.kubeconfig /opt/kubernetes/cfg/
 
 4.systemd管理kube-proxy 
 cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
 
 5.启动并配置开机自启动 
 systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy
 
 5.5 部署CNI网络 
 先准备好CNI二进制文件： 
 下载地址：https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
 
 解压二进制包并移动到默认的工作目录： 
 mkdir /opt/cni/bin -p
tar xf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/
 
 部署CNI网络： 
 wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#registry.aliyuncs.com/google_containers/flannel:v0.12.0-amd64#g" kube-flannel.yml
 
 默认地址无法访问修改为docker hub镜像地址 
 kubectl apply -f kube-flannel.yml
[root@master ~]# kubectl get pods -n kube-system
NAME                    READY  STATUS    RESTARTS  AGE
kube-flannel-ds-srtq4  1/1    Running  0          11m
[root@master ~]# kubectl get nodes
NAME        STATUS  ROLES    AGE  VERSION
k8s-master  Ready      31m  v1.18.3
 
 5.6 授权apiserver访问kubelet 
 cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF
[root@master ~]# kubectl apply -f apiserver-to-kubelet-rbac.yaml
 
 5.7 新增加woker node 
 1，copy已经部署好饿node相关文件到新节点：
 在master节点将Worker Node涉及文件拷贝到新节点10.0.0.12/13 
 node1：
scp -r /opt/kubernetes [email protected]:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service [email protected]:/usr/lib/systemd/system
scp -r /opt/cni/ [email protected]:/opt/
scp -r /opt/kubernetes/ssl/ca.pem [email protected]:/opt/kubernetes/ssl
node2：
scp -r /opt/kubernetes [email protected]:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service [email protected]:/usr/lib/systemd/system
scp -r /opt/cni/ [email protected]:/opt/
scp -r /opt/kubernetes/ssl/ca.pem [email protected]:/opt/kubernetes/ssl
 
 2.删除kubelet和kubeconfig文件：在node01和node02 
 rm -f /opt/kubernetes/cfg/kubelet.kubeconfig
rm -f /opt/kubernetes/ssl/kubelet*
 
 注：这几个文件上是证书申请后自动生成的，每个node节点都不同，所以必须删除重新生成
 3，修改主机名：node1和node2都要修改 
 [root@master ~]# vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node1
[root@master ~]#  vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node1
 
  
  启动并设置开机启动 
  
 systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy
 
  
  在Master上批准新Node kubelet证书申请 
  
 [root@master ~]# kubectl get csr
NAME                                                  AGE  SIGNERNAME                                    REQUESTOR          CONDITION
node-csr--G3jTzzjPOXVvyhS4Wt_UWvVr9YxHBSWVBjqbFD-dYo  52m  kubernetes.io/kube-apiserver-client-kubelet  kubelet-bootstrap  Approved,Issued
node-csr-CyHqUAAsgTKpnPTNOxJWGissadNll5QEbQmiZ9wHjqg  60s  kubernetes.io/kube-apiserver-client-kubelet  kubelet-bootstrap  Pending
node-csr-tXc_pejNXrtq0LpMGW92RMoGwL54Bb47In4iI8kgerI  50s  kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap  Pending
[root@master ~]# kubectl certificate approve node-csr-CyHqUAAsgTKpnPTNOxJWGissadNll5QEbQmiZ9wHjqg
[root@master ~]# kubectl certificate approve node-csr-tXc_pejNXrtq0LpMGW92RMoGwL54Bb47In4iI8kgerI
 
 6，查看node状态： 
 [root@master ~]# kubectl get nodes
NAME        STATUS  ROLES    AGE    VERSION
k8s-master  Ready      4h3m  v1.18.3
k8s-node1   Ready      29m   v1.18.3
k8s-node2   Ready      29m   v1.18.3
 
 7，安装支持补全命令 
 yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
kubectl completion bash >/etc/bash_completion.d/kubectl
 
 六、部署Dashboard和CoreDNS 
 6.1部署Dashboard 
 wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
 
 默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部 
 vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

kubectl apply -f recommended.yaml

[root@master ~]# kubectl get pods,svc -n kubernetes-dashboard 
NAME                                             READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-694557449d-49gsb   1/1     Running   0          69s
pod/kubernetes-dashboard-9774cc786-zpvnq         1/1     Running   0          69s

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.183           8000/TCP        69s
service/kubernetes-dashboard        NodePort    10.0.0.56            443:30001/TCP   70s

 
 访问地址：https://NodeIP :30001 
 创建service account并绑定默认cluster-admin管理员集群角色： 
 kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name:         dashboard-admin-token-ldkwm
Namespace:    kube-system
Labels:       
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: a19a7c1a-2e9a-40bd-a98d-070b12660907

Type:  kubernetes.io/service-account-token

Data
====
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IkpGXzZuME9hMVJBbm1vRVAwdHN1TlRlMVZyRTdwQmRCWjVmbE1ZVGpRZm8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tbGRrd20iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYTE5YTdjMWEtMmU5YS00MGJkLWE5OGQtMDcwYjEyNjYwOTA3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.uKLuCPIs2pXgiT3JyWXbjYDR00A3O6kACS2SlsbFSmBuVK4hcKx0dqitS8-Awp-rwICxbu28j3Pdcq2x_nkJPg6NluIz7OMkb7VDGITTDQNUL-XJ09Gb4QUMVO24VlpPHFUk_jObdbPHieLOgRlRtMGmwZxuBq5dtBMT831GNYO93we7mYky67csadrP1U-dN3L9Wos45qGqKlbveaTIOmnoxxd6oNzytOcTbchzzLB9FbccyX5PmSrUfOPmZtvyYk9ckwqnY_ROw4-Q_GQKNrC6FvFIrukUqZGMKKJBQ8YLiYvczTJfBtE8Ly54ZZJ0-aTh12JBKtS3zlD89fAY_A
ca.crt:     1359 bytes

 
 使用输出的token登录dashboard 
  
   
     
    
   
  
    image.png 
   
  
 
 
  
   
     
    
   
  
    image.png 
   
  
 6.2部署CoreDNS 
 CoreDNS用于集群内部Service名称解析。

二进制方式部署kubernetes-v1.18集群

一，kunernetes部署方式和环境介绍

1.1，生产环境可部署kubenetes集群的两种方式

1.1.1 kubeadm

1.1.2 二进制部署

1.2 部署kubernetes集群需要以下几个条件：

1.3 环境准备

1.4 系统初始化配置

二，部署etcd集群

2.1 准备cfssl证书生成工具

2.2 生成etcd证书

2.3 从github下载二进制文件

2.4 部署etcd集群

三，安装docker

3.1 解压二进制包

3.2 system管理docker

3.3 配置阿里云加速

3.4 启动并设置开机自启动

四，部署Master Node

4.1 生成kube-apiserver证书

4.2 从github下载二进制文件

4.3 解压二进制包

4.4 部署kube-apiserver

4.5 部署kube-controller-manager

4.6, 部署kube-scheduler

五，部署Woker Node

5.1 创建工作目录

5.2部署kubelet

5.3，批准kubelet证书申请并加入集群

5.4 部署kube-proxy

5.5 部署CNI网络

5.6 授权apiserver访问kubelet

5.7 新增加woker node

六、部署Dashboard和CoreDNS

6.1部署Dashboard

6.2部署CoreDNS

你可能感兴趣的:(二进制方式部署kubernetes-v1.18集群)

二进制方式部署kubernetes-v1.18集群

一，kunernetes部署方式和环境介绍

1.1，生产环境可部署kubenetes集群的两种方式

1.1.1 kubeadm

1.1.2 二进制部署

1.2 部署kubernetes集群需要以下几个条件：

1.3 环境准备

1.4 系统初始化配置

二， 部署etcd集群

2.1 准备cfssl证书生成工具

2.2 生成etcd证书

2.3 从github下载二进制文件

2.4 部署etcd集群

三，安装docker

3.1 解压二进制包

3.2 system管理docker

3.3 配置阿里云加速

3.4 启动并设置开机自启动

四，部署Master Node

4.1 生成kube-apiserver证书

4.2 从github下载二进制文件

4.3 解压二进制包

4.4 部署kube-apiserver

4.5 部署kube-controller-manager

4.6, 部署kube-scheduler

五，部署Woker Node

5.1 创建工作目录

5.2部署kubelet

5.3，批准kubelet证书申请并加入集群

5.4 部署kube-proxy

5.5 部署CNI网络

5.6 授权apiserver访问kubelet

5.7 新增加woker node

六、部署Dashboard和CoreDNS

6.1部署Dashboard

6.2部署CoreDNS

你可能感兴趣的:(二进制方式部署kubernetes-v1.18集群)

二，部署etcd集群