CentOS 7 安装Kubernetes记录(带ca认证)
CentOS 7 安装Kubernetes记录(带ca认证)
一、环境准备
最好为服务器配置静态IP,不然关机后很多地方的ip需要修改。参考文章
1. 所有节点配置host，使各个Node间可以通过主机名称进行解析。
主机名可以参考上面的文章链接进行修改
$ vi /etc/hosts
#加入如下片段(ip地址和servername替换成自己的)
172.16.136.201 server01
172.16.136.202 server02
172.16.136.203 server032.所有节点关闭防火墙
systemctl disable firewalld
systemctl stop firewalld3.所有节点禁用SELinux
setenforce 04.所有节点关闭NetworkManager(虚拟机环境)
systemctl stop NetworkManager
systemctl disable NetworkManager在虚拟机环境有时会出现
Failed to start LSB: Bring up/down networking.错误，network无法启动。
5.关闭swap
为了性能考虑
swapoff -a二、安装Docker (所有节点)
1.安装脚本
wget -qO- https://get.docker.com/ | sh2.修改docker的网络转发规则
vi /lib/systemd/system/docker.service
#找到ExecStart=xxx，在这行上面加入一行，内容如下：(k8s的网络需要)
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT3.(可选)给docker配置加速
可以到DaoCloud上注册账号免费申请,在页面底部的加速器选项。
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://0ed63f2c.m.daocloud.io4.使配置生效并启动docker
systemctl daemon-reload
systemctl start docker
systemctl enable docker5.验证是否安装完成
$ docker version
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:09:15 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:13:03 2018
OS/Arch: linux/amd64
Experimental: false三、Kubernetes安装
1. 设置系统参数 (所有节点)
- 允许路由转发，不对bridge的数据进行处理
#写入配置文件
cat < /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF #生效配置文件
sysctl -p /etc/sysctl.d/k8s.conf2. 下载kubernetes二进制文件 (所有节点)
下载地址
下载完成后上传到服务器用户目录并解压，并将其目录加入到PATH环境变量中。
tar -zxf kubernetes-bins.tar.gz;rm -rf kubernetes-bins.tar.gz;mv kubernetes-bins bin#添加环境变量
vim /etc/profile
export PATH=$PATH:/root/bin
source /etc/profile3. 下载配置文件并生成配置文件 (所有节点)
使用
yum install git安装Git
git clone https://github.com/KingBoyWorld/kubernetes-starter3.1 生成带有权限认证的配置文件 (所有节点)
cd ~/kubernetes-starter && vi config.properties && ./gen-config.sh with-ca需要注意的是其中主节点使用https， 如下示例配置
#kubernetes二进制文件目录,eg: /home/michael/bin
BIN_PATH=/root/bin
#当前节点ip, eg: 192.168.1.102
NODE_IP=172.16.136.201
#etcd服务集群列表, eg: http://192.168.1.102:2379
#如果已有etcd集群可以填写现有的。没有的话填写：http://${MASTER_IP}:2379 （MASTER_IP自行替换成自己的主节点ip）
ETCD_ENDPOINTS=https://172.16.136.202:2379
#kubernetes主节点ip地址, eg: 192.168.1.102
MASTER_IP=172.16.136.2024. 安装cfssl并生成根证书 (所有节点)
4.1 安装cfssl(所有节点)
用来生成证书
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 && wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 && chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 && mv cfssl_linux-amd64 /usr/local/bin/cfssl && mv cfssljson_linux-amd64 /usr/local/bin/cfssljsoncfssl version4.2 生成根证书 (主节点)
mkdir -p /etc/kubernetes/ca && cp ~/kubernetes-starter/target/ca/ca-config.json /etc/kubernetes/ca && cp ~/kubernetes-starter/target/ca/ca-csr.json /etc/kubernetes/ca && cd /etc/kubernetes/ca && cfssl gencert -initca ca-csr.json | cfssljson -bare ca && ls && cd ~/kubernetes-starter5. 安装ETCD服务 (主节点)
5.1 生成ETCD证书 (主节点)
mkdir -p /etc/kubernetes/ca/etcd && cp ~/kubernetes-starter/target/ca/etcd/etcd-csr.json /etc/kubernetes/ca/etcd/ && cd /etc/kubernetes/ca/etcd/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd && ls && cd ~/kubernetes-starter5.2 安装并启动ETCD (主节点)
mkdir -p /var/lib/etcd && cp ~/kubernetes-starter/target/master-node/etcd.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable etcd && systemctl start etcd验证是否安装完成
$ ETCDCTL_API=3 etcdctl \
--endpoints=https://172.16.136.202:2379 \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd/etcd.pem \
--key=/etc/kubernetes/ca/etcd/etcd-key.pem \
endpoint health
#如下输出说明成功
https://172.16.136.202:2379 is healthy: successfully committed proposal: took = 1.04668ms6. 安装APIServer (主节点)
6.1 生成证书 (主节点)
mkdir -p /etc/kubernetes/ca/kubernetes && cp ~/kubernetes-starter/target/ca/kubernetes/kubernetes-csr.json /etc/kubernetes/ca/kubernetes/ && cd /etc/kubernetes/ca/kubernetes/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes && ls && cd ~/kubernetes-starter6.2 生成token认证文件 (主节点)
生成随机token
$ head -c 16 /dev/urandom | od -An -t x | tr -d ' '
8afdf3c4eb7c74018452423c29433609按照固定格式写入token.csv，注意替换token内容
echo "8afdf3c4eb7c74018452423c29433609,kubelet-bootstrap,10001,\"system:kubelet-bootstrap\"" > /etc/kubernetes/ca/kubernetes/token.csv6.3 启动APIServer (主节点)
cp ~/kubernetes-starter/target/master-node/kube-apiserver.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-apiserver.service && systemctl start kube-apiserver && cd ~/kubernetes-starter验证
journalctl -f -u kube-apiserver7. 安装ControllerManager (主节点)
安装
cp ~/kubernetes-starter/target/master-node/kube-controller-manager.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl start kube-controller-manager验证
journalctl -f -u kube-controller-manager8. 安装Scheduler (主节点)
安装
cp ~/kubernetes-starter/target/master-node/kube-scheduler.service /lib/systemd/system/ && systemctl enable kube-scheduler.service && systemctl start kube-scheduler验证
journalctl -f -u kube-scheduler9. 配置kubectl (主节点)
9.1 生成证书 (主节点)
mkdir -p /etc/kubernetes/ca/admin && cp ~/kubernetes-starter/target/ca/admin/admin-csr.json /etc/kubernetes/ca/admin/ && cd /etc/kubernetes/ca/admin/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin && ls && cd ~/kubernetes-starter9.2 配置kubectl (主节点)
注意修改其中的IP地址
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 && kubectl config set-credentials admin --client-certificate=/etc/kubernetes/ca/admin/admin.pem --embed-certs=true --client-key=/etc/kubernetes/ca/admin/admin-key.pem && kubectl config set-context kubernetes --cluster=kubernetes --user=admin && kubectl config use-context kubernetes查看生成的文件内容
cat ~/.kube/config验证
$ kubectl get componentstatus
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
scheduler Healthy ok10 配置calico-node (所有节点)
10.1 生成证书 (主节点)
mkdir -p /etc/kubernetes/ca/calico && cp ~/kubernetes-starter/target/ca/calico/calico-csr.json /etc/kubernetes/ca/calico/ && cd /etc/kubernetes/ca/calico/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes calico-csr.json | cfssljson -bare calico && ls && cd ~/kubernetes-starter10.2 拷贝证书到其它机器上 (主节点)
可能需要在两台机器上建立
/etc/kubernetes目录
scp -r /etc/kubernetes/ca/ root@172.16.136.201:/etc/kubernetes/ca/
scp -r /etc/kubernetes/ca/ root@172.16.136.201:/etc/kubernetes/ca/10.2 启动calio-node (所有节点)
需要下载docker镜像，会慢一些
cp ~/kubernetes-starter/target/all-node/kube-calico.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-calico && systemctl start kube-calico查看日志
journalctl -f -u kube-calico验证
calicoctl node status11 安装kubelet (工作节点)
11.1 配置查看 (主节点)
#可以通过下面命令查询clusterrole列表
$ kubectl -n kube-system get clusterrole
#可以回顾一下token文件的内容
$ cat /etc/kubernetes/ca/kubernetes/token.csv
8afdf3c4eb7c74018452423c29433609,kubelet-bootstrap,10001,"system:kubelet-bootstrap"11.2 环境准备
(工作节点)
mkdir -p /var/lib/kubelet && mkdir -p /etc/kubernetes && mkdir -p /etc/cni/net.d(主节点)
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap11.3 生成bootstrap.kubeconfig配置文件 (工作节点)
注意替换IP地址 和 token
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 --kubeconfig=bootstrap.kubeconfig && kubectl config set-credentials kubelet-bootstrap --token=8afdf3c4eb7c74018452423c29433609 --kubeconfig=bootstrap.kubeconfig && kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=bootstrap.kubeconfig && kubectl config use-context default --kubeconfig=bootstrap.kubeconfig && mv bootstrap.kubeconfig /etc/kubernetes/11.4 cni配置 (工作节点)
cp ~/kubernetes-starter/target/worker-node/10-calico.conf /etc/cni/net.d/11.5 启动kubelet服务 (工作节点)
cp ~/kubernetes-starter/target/worker-node/kubelet.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kubelet && systemctl start kubelet
#启动kubelet之后到master节点允许worker加入(批准worker的tls证书请求)
#--------*在主节点执行*---------
$ kubectl get csr|grep 'Pending' | awk '{print $1}'| xargs kubectl certificate approve
#-----------------------------检查日志
journalctl -f -u kubelet12 安装kube-proxy (工作节点)
创建工作目录 (工作节点)
mkdir -p /var/lib/kube-proxy12.1 生成证书 (工作节点)
mkdir -p /etc/kubernetes/ca/kube-proxy && cp ~/kubernetes-starter/target/ca/kube-proxy/kube-proxy-csr.json /etc/kubernetes/ca/kube-proxy/ && cd /etc/kubernetes/ca/kube-proxy/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy && ls && cd ~/kubernetes-starter12.2 生成kube-proxy.kubeconfig配置 (工作节点)
注意替换IP地址
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 --kubeconfig=kube-proxy.kubeconfig && kubectl config set-credentials kube-proxy --client-certificate=/etc/kubernetes/ca/kube-proxy/kube-proxy.pem --client-key=/etc/kubernetes/ca/kube-proxy/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig && kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig && kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig && mv kube-proxy.kubeconfig /etc/kubernetes/kube-proxy.kubeconfig12.3 启动kube-proxy (工作节点)
cp ~/kubernetes-starter/target/worker-node/kube-proxy.service /lib/systemd/system/ && systemctl daemon-reload && yum -y install conntrack && systemctl enable kube-proxy && systemctl start kube-proxy检查
journalctl -f -u kube-proxy13 部署kube-dns (主节点)
kubectl create -f ~/kubernetes-starter/target/services/kube-dns.yaml验证
kubectl -n kube-system get pods