提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档
本篇日记将记录不同部署kubernetes的production 环境,如果是学习环境的部署可以参照k8-install-learning environments。部署完毕后,进行压力以及安全测试,逐步迭代熟悉production cluster的部署考量。
本方法将不借用部署工具,在服务器端或者主机端部署相关的binary 文件和nodes。通过本轮部署,加深kubernetes架构的理解,总体流程参照kubernete the hard way,本篇日记记录的k8 cluster,同样部署2个master node,2个worker node,1个loadbalancer node。
需要工具:
安装组件如下:
由于virtual box 不同版本上对于host-only network中规定范围的不同,因此修改原github中master node 与 work node 的ip范围(Cannot create a private networck from Vagrant in VirtualBox after updating it)。
git clone https://github.com/mmumshad/kubernetes-the-hard-way.git
cd kubernetes-the-hard-way\vagrant
#修改node的ip范围,原来的范围为192.168.5.*
#IP_NW = "192.168.5."
新的ip范围
IP_NW = "192.168.56."
vagrant up
ssh进入不同worker 与master node下不同worker 和master的IP。
vagrant ssh worker-1
##进入worker-1 后,系统给出了ip
##IP address for enp0s8: 192.168.56.21
## 依次获取ips
依次获取ips,如下图vagrant 在virtubox里面部署的虚拟机的Ip以及对应的主机端口等信息。
vm | vm name | role | ip | port |
---|---|---|---|---|
master-1 | kubernetes-ha-master-1 | Master | 192.168.56.11 | 2711 |
master-2 | kubernetes-ha-master-2 | Master | 192.168.56.12 | 2712 |
work-1 | kubernetes-ha-work-1 | Worker | 192.168.56.21 | 2721 |
work-2 | kubernetes-ha-work-2 | Worker | 192.168.56.22 | 2722 |
Loadbalancer | kubernetes-ha-work-3 | Loadbalancer | 192.168.56.30 | 2730 |
通过上述命令,vagrant 部署了5个vm,并在每个虚拟机完成了以下的设置:
# to see details about the uplink DNS servers
vagrant@worker-2:/etc$ systemd-resolve --status
# 输出
Global
DNS Servers: 8.8.8.8
DNSSEC NTA: 10.in-addr.arpa
#change to sudo
vagrant@worker-2:/etc$ sudo -i
root@worker-2:~# docker --version
Docker version 20.10.17, build 100c701
sysctl net.bridge.bridge-nf-call-iptables=1
#例如:
root@worker-2:~# ping 192.168.56.11
PING 192.168.56.11 (192.168.56.11) 56(84) bytes of data.
64 bytes from 192.168.56.11: icmp_seq=1 ttl=64 time=1.12 ms
64 bytes from 192.168.56.11: icmp_seq=2 ttl=64 time=0.491 ms
vagrant ssh master-1
ssh-keygen
#得到private key 与 public key,如下
vagrant@master-1:~/.ssh$ ls
authorized_keys id_rsa id_rsa.pub
## add public key to authorized_keys
vagrant@worker-1:~/.ssh$ cat >> ~/.ssh/authorized_keys <<EOF
> ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuNtNIV6fvTWHaqYDgnSSgA/HcchVXxbEwAT26UDw7frksiMI806xgN+9/M5EtRuzUQTvUG1uFoKldNo5IjIDZnyoCBZMtNSEFntzxQ3UwPUYaHJ7+F4pGmY2wn9kHNxe6JdKnhJ4IllhjeRlZGiv8v5LDZGDYofnakcUJo7bnKQdxJXBTK8mDJ8Eu/vD5jHkxEMY+3uwSPwmJcjD+JJ2HPQVPmthdfkXGgSeFJZGWCThK91iXrLJMcGoujHV1WFz23HauGLYO+mhL7HNXEAhIOB0j+8j1caT94jd7ulrticlGP5G8T8/0GI4GGEB1qXtiL/1I0TwHEir7wOenRdb7 vagrant@master-1
> EOF
这样就能从master-1里ssh其他nodes。例如下面的界面表示已经从master-1 成功ssh入 worker-1(192.168.56.21)
vagrant@master-1:~/.ssh$ ssh 192.168.56.21
Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-191-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
wget https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/
kubectl client --version ##查看版本
本节主要通过openssl来生成kubernetes的各种组件需要访问彼此的certificate。首先选择一个node来生成文件,然后将对应文件拷贝到其他nodes里面去,参照kubernete the hard way。本篇也选择在master-1里面生成certificates。关于TLS certificate 的介绍,这篇TLS introdution就足够了。
openssl genrsa -out kube-controller-manager.key 2048
openssl req -new -key kube-controller-manager.key -subj "/CN=system:kube-controller-manager" -out kube-controller-manager.csr
openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-controller-manager.crt -days 1000
# Generate private key for admin user
openssl genrsa -out admin.key 2048
# Generate CSR for admin user. Note the OU.
openssl req -new -key admin.key -subj "/CN=admin/O=system:masters" -out admin.csr
# Sign certificate for admin user using CA servers private key
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out admin.crt -days 1000
openssl genrsa -out kube-controller-manager.key 2048
openssl req -new -key kube-controller-manager.key -subj "/CN=system:kube-controller-manager" -out kube-controller-manager.csr
openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-controller-manager.crt -days 1000
openssl genrsa -out kube-proxy.key 2048
openssl req -new -key kube-proxy.key -subj "/CN=system:kube-proxy" -out kube-proxy.csr
openssl x509 -req -in kube-proxy.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-proxy.crt -days 1000
openssl genrsa -out kube-scheduler.key 2048
openssl req -new -key kube-scheduler.key -subj "/CN=system:kube-scheduler" -out kube-scheduler.csr
openssl x509 -req -in kube-scheduler.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-scheduler.crt -days 1000
cat > openssl.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
#记得修改IP
IP.1 = 10.96.0.1
IP.2 = 192.168.56.11
IP.3 = 192.168.56.12
IP.4 = 192.168.56.30
IP.5 = 127.0.0.1
EOF
openssl genrsa -out kube-apiserver.key 2048
openssl req -new -key kube-apiserver.key -subj "/CN=kube-apiserver" -out kube-apiserver.csr -config openssl.cnf
openssl x509 -req -in kube-apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-apiserver.crt -extensions v3_req -extfile openssl.cnf -days 1000
cat > openssl-etcd.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 192.168.56.11
IP.2 = 192.168.56.12
IP.3 = 127.0.0.1
EOF
openssl genrsa -out etcd-server.key 2048
openssl req -new -key etcd-server.key -subj "/CN=etcd-server" -out etcd-server.csr -config openssl-etcd.cnf
openssl x509 -req -in etcd-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out etcd-server.crt -extensions v3_req -extfile openssl-etcd.cnf -days 1000
openssl genrsa -out service-account.key 2048
openssl req -new -key service-account.key -subj "/CN=service-accounts" -out service-account.csr
openssl x509 -req -in service-account.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out service-account.crt -days 1000
for instance in master-1 master-2; do
scp ca.crt ca.key kube-apiserver.key kube-apiserver.crt \
service-account.key service-account.crt \
etcd-server.key etcd-server.crt \
${instance}:~/
done
本篇截止,一共拉起了5个VMs,其中2个为master node,2个worker node,1个loadbalancer。并建立了master-1 和各个node的ssh连接,在master-1上部署了kubectl;生成了不同组件的crt和key文件,并用CA签名,分布到master-1 和master-2。下篇Generating Kubernetes Configuration Files for Authentication.