二进制部署kubernetes——1.20.4
k8s二进制部署
1、环境准备
操作系统：ecntos7.9
容器引擎：docker-1.24.7
kubernetes：1.20.4
2、服务器整体规划
k8s-master1 192.168.7.100 etcd ， kube-apiserver，kube-controller-manager，kube-scheduler，kubelet，kube-proxy，docker
k8s-node1 192.168.7.101 etcd, kubelet，kube-proxy，docker
k8s-node2 192.168.7.102 etcd, kubelet，kube-proxy，docker
3、系统初始化配置
3.1、关闭防火墙(所有节点)
systemctl stop firewalld
systemctl disable firewalld
3.2、关闭selinux(所有节点)
setenforce 0 # 临时
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
3.3、关闭swap(所有节点)
swapoff -a # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
3.4、根据规划设置主机名(所有节点)
hostnamectl set-hostname
3.5、添加hosts(所有节点)
cat >> /etc/hosts << EOF
192.168.7.100 k8s-master
192.168.7.101 k8s-node1
192.168.7.102 k8s-node2
EOF
3.6、将IPv4流量传递到iptables的链(所有节点)
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 89100
fs.file-max=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans =327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 131072
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system # 生效
3.7、时间同步(所有节点)
yum install ntpdate -y
ntpdate Not Found
7. 配置免密方便部署
ssh-keygen
ssh-copy-id 192.168.7.101
ssh-copy-id 192.168.7.102
3.8 主机系统优化
ulimit -SHn 65535
cat <
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF
3.9 ipvs管理工具安装及模块加载
为集群节点安装，负载均衡节点不用安装
yum -y install ipvsadm ipset sysstat conntrack libseccomp
所有节点配置ipvs模块，再内核4.19+版本nv_conntrack_ipv4已经改为nf_conntrack,4.18以下使用nf_conntrack_ipv4即可。
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
创建 /etc/modules-load.d/ipvs.conf 并加入以下内容
Cat > /etc/modules-load.d/ipvs.conf < ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp ip_vs_sh nf_conntrack ip_tables ip_set xt_set ipt_set ipt_rpfilter ipt_REJECT ipip EOF 设置为开机启动 Systemctl enable --now systemd-modules-load.service 在这里插入图片描述 Etcd 是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，所以先准备一个Etcd数据库，为解决Etcd单点故障，应采用集群方式部署,这里使用3台组建集群，可容忍1台机器故障. 下载软件包 mkdir /home/cfssl && cd /home/cfssl/ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo 5.2.1、创建工作目录 mkdir -p /opt/{etcd,tls} && cd /opt/tls/etcd 5.2.2、自签证书颁发机构（CA） cat > ca-config.json << EOF { "signing": { "default": { "expiry": "876000h" }, "profiles": { "www": { "expiry": "876000h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json << EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Jinan", "ST": "Shandong" } ] } EOF 生成证书：会生成ca.pem和ca-key.pem文件 cfssl gencert -initca ca-csr.json | cfssljson -bare ca - 5.2.3、使用自签CA签发Etcd Https证书 创建证书请求文件 cat > server-csr.json << EOF { "CN": "etcd", "hosts": [ "192.168.7.100", "192.168.7.101", "192.168.7.102" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Jinan", "ST": "Shandong" } ] } EOF hosts字段中IP为所有etcd节点的集群内部通信IP 生成证书，会生成server.pem和server-key.pem文件 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server 5.3.1、下载etcd二进制文件 mkdir /home/etcd && cd /home/etcd wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz 5.3.2、创建工作目录并解压二进制包 mkdir /opt/etcd/{bin,cfg,ssl} -p cp /opt/tls/etcd/ca*pem /opt/tls/etcd/server*pem /opt/etcd/ssl/ tar zxvf /home/etcd/etcd-v3.4.9-linux-amd64.tar.gz cp /home/etcd/etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/ 5.3.3、创建etcd配置文件 cat > /opt/etcd/cfg/etcd.conf << EOF #[Member] ETCD_NAME="etcd-1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.7.100:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.7.100:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.100:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.100:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.7.100:2380,etcd-2=https://192.168.7.101:2380,etcd-3=https://192.168.7.102:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF 5.3.4、配置文件说明 ETCD_NAME：节点名称，集群中唯一 ETCD_DATA_DIR：数据目录 ETCD_LISTEN_PEER_URLS：集群通信监听地址 ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址 ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址 ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址 ETCD_INITIAL_CLUSTER：集群节点地址 ETCD_INITIALCLUSTER_TOKEN：集群Token ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群 5.3.5、systemd管理etcd cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF 5.3.6、拷贝生成的证书至指定位置 cp /opt/tls/etcd/ca*pem /opt/tls/etcd/server*pem /opt/etcd/ssl/ 5.3.7、启动并设置开机启动 systemctl daemon-reload systemctl start etcd systemctl enable etcd 5.3.8、将上面节点1所有生成的文件拷贝到节点2和节点3(创建2和3节点工作路劲) scp -r /opt/etcd/ etcd-2:/opt/ scp /usr/lib/systemd/system/etcd.service etcd-2:/usr/lib/systemd/system/ scp -r /opt/etcd/ etcd-3:/opt/ scp /usr/lib/systemd/system/etcd.service etcd-3:/usr/lib/systemd/system/ 5.3.9、在节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP 节点2 vim /opt/etcd/cfg/etcd.conf #[Member] ETCD_NAME="etcd-2" # 修改此处，节点2改为etcd-2，节点3改为etcd-3 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="192.168.7.101:2380" # 修改此处为当前服务器IP ETCD_LISTEN_CLIENT_URLS="https://192.168.7.101:2379" # 修改此处为当前服务器IP #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.101:2380" # 修改此处为当前服务器IP ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.101:2379" # 修改此处为当前服务器IP ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.7.100:2380,etcd-2=https://192.168.7.101:2380,etcd-3=https://192.168.7.102:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" 节点3 vim /opt/etcd/cfg/etcd.conf #[Member] ETCD_NAME="etcd-3" # 修改此处，节点2改为etcd-2，节点3改为etcd-3 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="192.168.7.102:2380" # 修改此处为当前服务器IP ETCD_LISTEN_CLIENT_URLS="https://192.168.7.102:2379" # 修改此处为当前服务器IP #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.102:2380" # 修改此处为当前服务器IP ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.102:2379" # 修改此处为当前服务器IP ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.7.100:2380,etcd-2=https://192.168.7.101:2380,etcd-3=https://192.168.7.102:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" 5.3.10、启动etcd并设置开机启动(节点2和3) systemctl daemon-reload systemctl start etcd systemctl enable etcd 5.3.11、查看集群状态 ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.7.100:2379,https://193.168.7.101:2379,https://192.168.7.102:2379" endpoint health --write-out=table +------------------------------+--------+------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +------------------------------+--------+------------+-------+ | https://192.168.7.100:2379 | true | 9.054571ms | | +------------------------------+--------+------------+-------+ | https://192.168.7.101:2379 | true | 9.054571ms | | +------------------------------+--------+------------+-------+ | https://192.168.7.102:2379 | true | 9.054571ms | | +------------------------------+--------+------------+-------+ 5.3.12、报错查看思路 journalctl -u etcd -l systemctl status etcd cat /var/log/message 1. 配置yum源 sudo yum install -y yum-utils sudo yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo 2.yum安装docker yum install docker-ce-19.03.9 3. 添加阿里云docker镜像加速 mkdir /etc/docker cat > /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] } EOF 4. 启动docker systemctl start docker systemctl enable docker systemctl status docker 下面为二进制方式安装，我们选择其中一种方式即可 cd /opt/package wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz 解压二进制软件包 tar zxvf docker-19.03.9.tgz cp docker/* /usr/bin docker version cat > /usr/lib/systemd/system/docker.service << EOF [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start docker systemctl enable docker cat > /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] } EOF systemctl restart docker cd /opt/tls/k8s cat > ca-config.json << EOF { "signing": { "default": { "expiry": "876000h" }, "profiles": { "kubernetes": { "expiry": "876000h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json << EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Jinan", "ST": "Shandong", "O": "k8s", "OU": "System" } ] } EOF 生成证书：生成ca.pem和ca-key.pem文件 cfssl gencert -initca ca-csr.json | cfssljson -bare ca - cat > server-csr.json << EOF { "CN": "kubernetes", "hosts": [ "10.0.0.1", "127.0.0.1", "192.168.7.100", "192.168.7.101", "192.168.7.102", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Jinan", "ST": "Shandong", "O": "k8s", "OU": "System" } ] } EOF 注意：上述文件hosts字段中IP为所有节点IP 生成证书，生成server.pem和server-key.pem cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server cd /home/package wget https://dl.k8s.io/v1.20.4/kubernetes-server-linux-amd64.tar.gz mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} tar zxvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin cp kubectl /usr/bin/ cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --etcd-servers=https://192.168.7.100:2379,https://192.168.7.101:2379,https://192.168.7.102:2379 \\ --bind-address=192.168.7.100 \\ --secure-port=6443 \\ --advertise-address=192.168.7.100 \\ --allow-privileged=true \\ --service-cluster-ip-range=10.0.0.0/24 \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\ --authorization-mode=RBAC,Node \\ --enable-bootstrap-token-auth=true \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --service-node-port-range=30000-32767 \\ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\ --tls-cert-file=/opt/kubernetes/ssl/server.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --service-account-issuer=api \\ --service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\ --proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\ --requestheader-allowed-names=kubernetes \\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User \\ --enable-aggregator-routing=true \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" EOF --logtostderr：启用日志 --v：日志等级 --log-dir：日志目录 --etcd-servers：etcd集群地址 --bind-address：监听地址 --secure-port：https安全端口 --advertise-address：集群通告地址 --allow-privileged：启用授权 --service-cluster-ip-range：Service虚拟IP地址段 --enable-admission-plugins：准入控制模块 --authorization-mode：认证授权，启用RBAC授权和节点自管理 --enable-bootstrap-token-auth：启用TLS bootstrap机制 --token-auth-file：bootstrap token文件 --service-node-port-range：Service nodeport类型默认分配端口范围 --kubelet-client-xxx：apiserver访问kubelet客户端证书 --tls-xxx-file：apiserver https证书 1.20版本必须加的参数：--service-account-issuer，--service-account-signing-key-file --etcd-xxxfile：连接Etcd集群证书 --audit-log-xxx：审计日志 启动聚合层相关配置：--requestheader-client-ca-file，--proxy-client-cert-file，--proxy-client-key-file，--requestheader-allowed-names，--requestheader-extra-headers-prefix，--requestheader-group-headers，--requestheader-username-headers，--enable-aggregator-routing 把刚才生成的证书拷贝到配置文件中的路径 cp /opt/tls/k8s/ca*pem /opt/tls/k8s/server*pem /opt/kubernetes/ssl/ TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。 7.9.1、自行生成token head -c 16 /dev/urandom | od -An -t x | tr -d ' ' cat > /opt/kubernetes/cfg/token.csv << EOF 5ec291f50943b0b8d4e5d879b656328a,kubelet-bootstrap,10001,"system:node-bootstrapper" EOF cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start kube-apiserver systemctl enable kube-apiserver journalctl -u kube-apiserver -l //查看报错详细信息 cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --leader-elect=true \\ --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\ --bind-address=127.0.0.1 \\ --allocate-node-cidrs=true \\ --cluster-cidr=10.244.0.0/16 \\ --service-cluster-ip-range=10.0.0.0/24 \\ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --root-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --cluster-signing-duration=876000h0m0s" EOF --kubeconfig：连接apiserver配置文件 --leader-elect：当该组件启动多个时，自动选举（HA） --cluster-signing-cert-file/--cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致 生成kube-controller-manager证书 cd /opt/k8s # 创建证书请求文件 cat > kube-controller-manager-csr.json << EOF { "CN": "system:kube-controller-manager", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Jinan", "ST": "Shandong", "O": "system:masters", "OU": "System" } ] } EOF 生成证书 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server="https://192.168.7.100:6443" \ --kubeconfig="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig" kubectl config set-credentials kube-controller-manager \ --client-certificate=./kube-controller-manager.pem \ --client-key=./kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig" kubectl config set-context default \ --cluster=kubernetes \ --user=kube-controller-manager \ --kubeconfig="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig" kubectl config use-context default --kubeconfig="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig" cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start kube-controller-manager systemctl enable kube-controller-manager cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF KUBE_SCHEDULER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --leader-elect \\ --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\ --bind-address=127.0.0.1" EOF ``` ###### 9.2、生成kubeconfig文件 ``` cd /root/liu/k8s # 创建证书请求文件 cat > kube-scheduler-csr.json << EOF { "CN": "system:kube-scheduler", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Jinan", "ST": "Shandong", "O": "system:masters", "OU": "System" } ] } EOF # 生成证书 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server="https://192.168.7.100:6443" \ --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig kubectl config set-credentials kube-scheduler \ --client-certificate=./kube-scheduler.pem \ --client-key=./kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-scheduler \ --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig kubectl config use-context default --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start kube-scheduler systemctl enable kube-scheduler cd /opt/k8s cat > admin-csr.json < { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Jinan", "ST": "Shandong", "O": "system:masters", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin mkdir /root/.kube kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server="https://192.168.7.100:6443" \ --kubeconfig="/root/.kube/config" kubectl config set-credentials cluster-admin \ --client-certificate=./admin.pem \ --client-key=./admin-key.pem \ --embed-certs=true \ --kubeconfig="/root/.kube/config" kubectl config set-context default \ --cluster=kubernetes \ --user=cluster-admin \ --kubeconfig="/root/.kube/config" kubectl config use-context default --kubeconfig="/root/.kube/config" kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-2 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"} kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap # 在所有worker node创建工作目录（master已创建，新加入节点需要创建） mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} # 从解压的k8s server压缩包中拷贝文件 cd /home/kubernetes/server/bin cp kubelet kube-proxy /opt/kubernetes/bin cat > /opt/kubernetes/cfg/kubelet.conf << EOF KUBELET_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --hostname-override=k8s-master \\ --network-plugin=cni \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\ --config=/opt/kubernetes/cfg/kubelet-config.yml \\ --cert-dir=/opt/kubernetes/ssl \\ --pod-infra-container-image=lizhenliang/pause-amd64:3.0" EOF --hostname-override：显示名称，集群中唯一 --network-plugin：启用CNI --kubeconfig：空路径，会自动生成，后面用于连接apiserver --bootstrap-kubeconfig：首次启动向apiserver申请证书 --config：配置参数文件 --cert-dir：kubelet证书生成目录 --pod-infra-container-image：管理Pod网络容器的镜像 cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0.0.2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 1000000 maxPods: 110 EOF # 生成 kubelet bootstrap kubeconfig 配置文件 kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server="https://192.168.7.100:6443" \ --kubeconfig="/opt/kubernetes/cfg/bootstrap.kubeconfig" kubectl config set-credentials "kubelet-bootstrap" \ --token="5ec291f50943b0b8d4e5d879b656328a" \ --kubeconfig="/opt/kubernetes/cfg/bootstrap.kubeconfig" kubectl config set-context default \ --cluster=kubernetes \ --user="kubelet-bootstrap" \ --kubeconfig="/opt/kubernetes/cfg/bootstrap.kubeconfig" kubectl config use-context default --kubeconfig="/opt/kubernetes/cfg/bootstrap.kubeconfig" cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start kubelet systemctl enable kubelet 查看kubelet证书请求 kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-9c-gV7fhdyBFpybXgaZXXEj8ZUr6z9UwmY5TT49jCJ8 62s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending 批准申请 kubectl certificate approve node-csr-9c-gV7fhdyBFpybXgaZXXEj8ZUr6z9UwmY5TT49jCJ8 查看节点（由于网络插件还没有部署，节点会没有准备就绪 NotReady） kubectl get node NAME STATUS ROLES AGE VERSION master NotReady cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF KUBE_PROXY_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --config=/opt/kubernetes/cfg/kube-proxy-config.yml" EOF cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: k8s-master clusterCIDR: 10.0.0.0/24 EOF 生成kube-proxy证书 cd /opt/tls/k8s 创建证书请求文件 cat > kube-proxy-csr.json << EOF { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF 生成证书 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server="https://192.168.7.100:6443" \ --kubeconfig="/opt/kubernetes/cfg/kube-proxy.kubeconfig" kubectl config set-credentials kube-proxy \ --client-certificate=./kube-proxy.pem \ --client-key=./kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig="/opt/kubernetes/cfg/kube-proxy.kubeconfig" kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig="/opt/kubernetes/cfg/kube-proxy.kubeconfig" kubectl config use-context default --kubeconfig="/opt/kubernetes/cfg/kube-proxy.kubeconfig" cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start kube-proxy systemctl enable kube-proxy 下载地址：https://docs.projectcalico.org/ [root@master yaml]# curl https://docs.projectcalico.org/v3.20/manifests/calico.yaml -O [root@master yaml]# kubectl apply -f calico.yaml [root@master yaml]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-756dd4db79-2k24z 1/1 Running 1 2d9h calico-node-6zh89 1/1 Running 0 41h calico-node-7f7zh 1/1 Running 0 41h calico-node-c6ss8 1/1 Running 1 2d9h cat > apiserver-to-kubelet-rbac.yaml << EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics - pods/log verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes EOF kubectl apply -f apiserver-to-kubelet-rbac.yaml scp -r /opt/kubernetes node1:/opt/ scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service node1:/usr/lib/systemd/system scp /opt/kubernetes/ssl/ca.pem node1:/opt/kubernetes/ssl scp -r /opt/kubernetes node2:/opt/ scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service node2:/usr/lib/systemd/system scp /opt/kubernetes/ssl/ca.pem node2:/opt/kubernetes/ssl rm -f /opt/kubernetes/cfg/kubelet.kubeconfig rm -f /opt/kubernetes/ssl/kubelet* node1节点 vim /opt/kubernetes/cfg/kubelet.conf --hostname-override=k8s-node1 vim /opt/kubernetes/cfg/kube-proxy-config.yml hostnameOverride: k8s-node1 node2节点 vim /opt/kubernetes/cfg/kubelet.conf --hostname-override=k8s-node2 vim /opt/kubernetes/cfg/kube-proxy-config.yml hostnameOverride: k8s-node2 systemctl daemon-reload systemctl start kubelet kube-proxy systemctl enable kubelet kube-proxy kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-9c-g2f5guyBFpybXgaZXXEj8ZUr6z9UwmYHT835TT49jAJ9 62s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending node-csr-XoaRNKKNZAPfkzhVF93gtog9Hj1342XqqWcngsWa0ZnW0dQ 62s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending 批准申请 kubectl certificate approve node-csr-9c-g2f5gyBFpybXgaZXXEj8ZUr6z9UwmYHT835TT49jAJ9 kubectl certificate approve node-csr-XoaRNKKNZAPfkzhVF93gtog9Hj1342XqqWcngsWa0ZnW0dQ [root@k8s-master1 ~]# kubectl get no NAME STATUS ROLES AGE VERSION k8s-master1 Ready k8s-node1 Ready k8s-node2 Ready root@k8s-master1-etcd1:~# docker pull coredns/coredns:1.8.7 root@k8s-master1-etcd1:~# docker tag docker.io/coredns/coredns:1.8.7 harbor.host.com/base/coredns:1.8.7 root@k8s-master1-etcd1:~# docker push harbor.host.com/base/coredns:1.8.7 下载这个文件 解压完后在kubernetes/cluster/addons/dns/coredns 找到coredns的yaml文件进行修改 [root@k8s-master cfg]# cat coredns.yaml # __MACHINE_GENERATED_WARNING__ apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: Reconcile name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults addonmanager.kubernetes.io/mode: EnsureExists name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system labels: addonmanager.kubernetes.io/mode: EnsureExists data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { #修改成当时部署k8s时host文件里面CLUSTER_DNS_DOMAIN对应的值，我这里是cluster.local pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: # replicas: not specified here: # 1. In order to make Addon Manager do not reconcile this replicas parameter. # 2. Default is 1. # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on. strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: securityContext: seccompProfile: type: RuntimeDefault priorityClassName: system-cluster-critical serviceAccountName: coredns affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux containers: - name: coredns image: coredns/coredns:1.8.7 #修改程自己的镜像仓库地址 imagePullPolicy: IfNotPresent resources: limits: memory: 200Mi #根据实际情况修改资源 requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile --- apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: kube-dns clusterIP: 10.0.0.2 #改成pod里面/etc/resolv.conf相同地址 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP [root@k8s-master cfg]# kubectl apply -f coredns.yaml 这里发现coredns 启动有问题 然后我们去排查下原因 Kubectl logs -f coredns-7cf597cd5f-b6f9c -n kube-system 查看到错误日志如下 E1110 02:37:36.166118 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-s ystem:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope[INFO] plugin/ready: Still waiting on: "kubernetes" 这错误是coredns没有权限访问k8s资源，修改权限： kubectl edit clusterrole system:coredns 最后追加 - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch 您在 /var/spool/mail/root 中有新邮件 [root@k8s-master cfg]# kubectl edit clusterrole system:coredns clusterrole.rbac.authorization.k8s.io/system:coredns edited [root@k8s-master cfg]# kubectl apply -f coredns.yaml serviceaccount/coredns unchanged clusterrole.rbac.authorization.k8s.io/system:coredns configured clusterrolebinding.rbac.authorization.k8s.io/system:coredns unchanged configmap/coredns unchanged deployment.apps/coredns unchanged service/kube-dns unchanged [root@k8s-master cfg]# kubectl get pod -n kube-system 恢复正常 Cp dashboard.yml /opt/kubernetes/cfg/ kubectl create serviceaccount dashboard-admin -n kube-system kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') 注意：访问需要使用火狐浏览器，谷歌浏览器需要添加某个配置才能使用，配置项自行百度4、证书说明
5、部署Etcd集群
5.1、准备cfssl证书生成工具(在master节点操作)
5.2、生成Etcd证书
5.3、开始部署ETCD集群
6、安装docker(这里选择二进制安装方式)(master和node节点)
6.1、下载并安装
6.2、systemd管理docker
6.3、启动
6.4、配置加速地址
7、部署kube-apiserver
7.1、自签证书签发机构（CA）
7.2、使用自签CA签发kube-apiserver HTTPS证书
7.3、下载并解压二进制软件包
7.4、创建配置文件
7.5、注：上面两个\ \ 第一个是转义符，第二个是换行符，使用转义符是为了使用EOF保留换行符
7.6、参数说明
7.7、启用 TLS Bootstrapping 机制
7.8、TLS bootstraping 工作流程
7.9、创建token文件7.10、systemd管理apiserver
7.11、启动并设置开机启动
7.12、注意：如果启动报错，查看是否与etcd连接不上，注意配置文件字符
8、部署kube-controller-manager
8.1、创建配置文件
8.2、配置说明
8.3、生成kubeconfig文件
8.4、生成kubeconfig文件（以下是shell命令，直接在终端执行）
8.5、systemd管理controller-manager
8.6、启动并设置开机启动
9、部署kube-scheduler
9.1、创建配置文件
9.3、成kubeconfig文件（以下是shell命令，直接在终端执行）
9.4、systemd管理scheduler
9.5、启动并设置开机启动
10、查看集群状态
10.1、生成kubectl连接集群的证书
10.2、生成kubeconfig文件
10.3、通过kubectl工具查看当前集群组件状态
10.4、授权kubelet-bootstrap用户允许请求证书
11、部署Worker Node(还是在master上操作)
11.1、创建工作目录并拷贝二进制文件
11.2、部署kubelet、创建配置文件
11.3、参数说明
11.4、配置参数文件
11.5、生成kubelet初次加入集群引导kubeconfig文件
11.6、systemd管理kubelet
11.7、启动并设置开机启动
11.8、批准kubelet证书申请并加入集群
11.9、部署kube-proxy、创建配置文件
11.10、配置参数文件
11.11、生成kube-proxy.kubeconfig文件
11.12、生成kubeconfig文件
11.13、systemd管理kube-proxy
11.14、启动并设置开机启动
12、部署网络组件Calico
13、授权apiserver访问kubelet
14、拷贝已部署好的Node相关文件到新节点(node1、node2)
14.1、删除kubelet证书和kubeconfig文件(node1、node2)
14.2、修改配置文件中的主机名(node1、node2)
14.3、启动并设置开机启动(node1、node2)
14.4、在Master上批准新Node kubelet证书申请
15、部署CoreDNS和Dashboard
1、下载所需要的镜像
下载地址：
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#downloads-for-v1204
3、修改部分内容如下：cat coredns.yaml根据实际情况修改
15.2、部署Dashboard、准备dashboard.yaml文件
15.3、访问地址：https://NodeIP:30001
15.4、创建service account并绑定默认cluster-admin管理员集群角色