【Kubernetes二】Centos7.9 Kubespray 2.18 安装 Kubernetes 1.22

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks. Kubespray provides:

• a highly available cluster
• composable attributes
• support for most popular Linux distributions
• Ubuntu 16.04, 18.04, 20.04
• CentOS/RHEL/Oracle Linux 7, 8
• Debian Buster, Jessie, Stretch, Wheezy
• Fedora 31, 32
• Fedora CoreOS
• openSUSE Leap 15
• Flatcar Container Linux by Kinvolk

初始化配置（所有机器执行）

IP	hostname
10.0.70.251	n1
10.0.70.252	n2
10.0.70.253	n3

组件	版本
kubespray	2.18.0
Kubernetes	1.22.5
Etcd	3.5.0
Docker	20.10
Containerd	1.5.8
CRI-O	1.22
CNI-plugins	v1.0.1
Calico	v3.20.3
Cilium	1.9.11
Flannel	0.15.1
Kube-ovn	1.8.1
Kube-Router	1.3.2
Multus	3.8
Weave	2.8.1
CoreDNS	1.8.0
Nodelocaldns	1.21.1
Helm	3.7.1
Nginx-ingress	1.0.4
Cert-manager	1.5.4
Kubernetes Dashboard	2.4.0

1.主机配置

vi /etc/hosts
10.0.70.251 n1
10.0.70.252 n2
10.0.70.253 n3

2.关闭防火墙

# 查看防火墙状态
firewall-cmd --state
#停止firewall #禁止firewall开机启动
systemctl stop firewalld.service & systemctl disable firewalld.service

3.关闭selinux

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux

4.关闭系统交换分区

swapoff -a && sysctl -w vm.swappiness=0 && sysctl -p && free –h

5. 优化内核参数

vi /etc/security/limits.conf

* soft nofile 32768
* hard nofile 65535
* soft nproc 32768
* hadr nproc 65535

6.SSH免密登录

#生成ssh key
ssh-keygen
#其他服务器免密码：
ssh-copy-id root@n1
ssh-copy-id root@n2
ssh-copy-id root@n3
#测试
ssh root@n2
#退出
exit

7.配置代理

#全局代理设置
vi /etc/profile

export proxy="http://10.0.70.32:10801"
export http_proxy=$proxy
export https_proxy=$proxy
export ftp_proxy=$proxy
export no_proxy="localhost, 127.0.0.1, ::1"
#生效
source /etc/profile
#测试
wget google.com
#关闭代理
#要关闭代理，仅仅注释掉profile的代理内容是不行的，在文件内加入以下内容，并重新source一下
unset http_proxy
unset https_proxy
unset ftp_proxy
unset no_proxy

安装执行（仅安装节点执行）

1.下载kubespray并解压

cd /usr/local/
wget https://github.com/kubernetes-sigs/kubespray/archive/v2.18.0.tar.gz
tar -xzvf v2.18.0.tar.gz
cd kubespray-2.18.0

2.安装python3依赖环境

yum -y install epel-release python3 python3-pip yum-utils --downloadonly --downloaddir=/tmp/releases/offline-packages
yum -d 2 -y install conntrack container-selinux rsync socat unzip bash-completion ipvsadm --downloadonly --downloaddir=/tmp/releases/offline-packages
cd /tmp/releases/offline-packages
yum -y localinstall *.rpm 
python3 -V & pip3 -V

3.安装kubespray依赖环境

cd /usr/local/kubespray-2.18.0
# 1.Install dependencies from ``requirements.txt``
#获取离线包
sudo pip3 download -d /tmp/releases/python-installer -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com  -r requirements.txt 
#离线安装
sudo pip3 install --no-index --find-links=/tmp/releases/python-installer -r requirements.txt
#查看
pip3 list

4.拷贝安装目录

# 2.Copy ``inventory/sample`` as ``inventory/mycluster``
cp -rfp inventory/sample inventory/mycluster

5.配置代理

#需要socks服务端开启允许局域网访问
vi inventory/mycluster/group_vars/all/all.yml
###
http_proxy: "http://10.0.70.32:10801"
https_proxy: "http://10.0.70.32:10801"
no_proxy: "localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.svc.cluster.local"
download_validate_certs: False
download_container: true

6.其他配置修改

# 4.Review and change parameters under ``inventory/mycluster/group_vars`` 
# 自定义k8s集群信息,编辑配置文件all.yml进行修改
cat inventory/mycluster/group_vars/all/all.yml
cat inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

# 添加需要的组件
vi inventory/mycluster/group_vars/k8s_cluster/addons.yml

dashboard_enabled: true
helm_enabled: true
ingress_nginx_enabled: true
ingress_nginx_host_network: true

# 缓存镜像
vi roles/download/defaults/main.yml

download_cache_dir: /tmp/kubespray_cache
download_run_once: true

# kube_service_addresses、
# kube_pods_subnet、
# kube_proxy_mode默认ipvs（ipvs或者iptables）
vi inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
# kube_network_plugin修改网络插件默认calico（cilium, calico, weave or flannel）
kube_network_plugin: flannel
# Random shifts for retrying failed ops like pushing/downloading
retry_stagger: 1
#Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
ndots: 4

7.配置安装机器信息

# 3.Update Ansible inventory file with inventory builder
declare -a IPS=(10.0.70.251 10.0.70.252 10.0.70.253)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
#修改hosts.yaml配置主机名可以避免修改主机名

8.测试连接

ansible -i inventory/mycluster/hosts.yaml all -m ping

9.安装执行

# 5.Deploy Kubespray with Ansible Playbook - run the playbook as root
# The option `--become` is required, as for example writing SSL keys in /etc/,
# installing packages and interacting with various systemd daemons.
# Without --become the playbook will fail to run!
#配置IPS多节点部署，查看hosts.yaml即可看到，可以根据需求自己搭配各个主机的角色
ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml -vvvvv --flush-cache
#部署失败，重试
ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml --limit node1 -vvvvv
#部署失败，重置
ansible-playbook -i inventory/mycluster/hosts.yaml  reset.yml


【问题1】Stop if memory is too small for masters
修改：roles/kubernetes/preinstall/defaults/main.yml
minimal_node_memory_mb: 1024
minimal_master_memory_mb: 1500

10.查看

#查看集群状态
kubectl get nodes -o wide
kubectl get pods --all-namespaces
kubectl get pods -n kube-system
kubectl get services --all-namespaces

#查看账号 k8dash-sa 登录token：
kubectl describe secret k8dash-sa -n kube-system | awk '$1=="token:"{print $2}'
#查看证书有效期
kubeadm alpha certs check-expiration

11.查看yum安装历史

yum history
yum history info 3

根据如上命名可以看到执行过的安装命令，具体如下:

-y install yum-utils
-d 2 -y install conntrack container-selinux rsync socat unzip bash-completion ipvsadm

12.离线部署的容器镜像收集脚本

cd contrib/offline 查看README.md
# manage-offline-container-images.sh
Container image collecting script for offline deployment
This script has two features:
(1) Get container images from an environment which is deployed online.
(2) Deploy local container registry and register the container images to the registry.
Step(1) should be done online site as a preparation, then we bring the gotten images
to the target offline environment.
Then we will run step(2) for registering the images to local registry.
Step(1) can be operated with:
manage-offline-container-images.sh  create
Step(2) can be operated with:
manage-offline-container-images.sh  register

#generate_list.sh
This script generates the list of downloaded files and the list of container images by `roles/download/defaults/main.yml` file.
./generate_list.sh
cat temp/files.list
通过 wget 进行下载
wget -x -P temp/files -i temp/files.list
所有可能会用到的镜像
cat temp/images.list

13.备份二进制文件

tar -czvf kubespray-2.18.0-cache.tar.gz /tmp/kubespray_cache/
tar -czvf kubespray-2.18.0-releases.tar.gz /tmp/releases/
#解压
cd /
tar -xzvf /tmp/kubespray-2.18.0-cache.tar.gz 
tar -xzvf /tmp/kubespray-2.18.0-releases.tar.gz 

14.访问dashboard

#kubectl proxy方式访问
kubectl proxy --address='0.0.0.0'  --accept-hosts='^*$'
#kubernetes-dashboard
http://10.0.70.251:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/login

#NodePort方式访问
kubectl edit svc/kubernetes-dashboard -n kube-system
type: NodePort
#查看端口
kubectl get svc kubernetes-dashboard -n kube-system
https://10.0.70.251:32690/

#ingress-nginx方式访问
kubectl apply -f ingress-nginx-kubernetes-dashboard.yaml
kubectl get ingress -n kube-system

dashboard可以查看kubernetes系统的整体情况，为了访问dashboard页面，需要增加RBAC：

tee admin-user.yaml <<-'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
EOF

tee admin-user-role.yaml <<-'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
EOF

创建ServiceAccount和ClusterRoleBinding：

kubectl apply -f admin-user.yaml && kubectl apply -f admin-user-role.yaml

运维

扩展集群

你可以通过运行 scale playbook 向集群中添加工作节点。有关更多信息， 请参见 “添加节点”。 你可以通过运行 remove-node playbook 来从集群中删除工作节点。

升级集群

你可以通过运行 upgrade-cluster Playbook 来升级集群

清理

你可以通过 reset Playbook 重置节点并清除所有与 Kubespray 一起安装的组件。

修改node端口范围

默认Kubernetes的端口范围为30000-32767，为便于后期应用，如ingress的80、443端口，建议开放全端口。同时开放全端口范围后，需要注意避开公共端口，如8080。需要在所有Master节点操作。

vi /etc/kubernetes/manifests/kube-apiserver.yaml
--service-node-port-range=1-65535

kubectl proxy

#kubectl proxy，它在您的机器与Kubernetes API之间创建一个代理，默认情况下，只能从本地访问（启动它的机器）。
#启动代理只需执行如下命令：
$ kubectl proxy
Starting to serve on 127.0.0.1:8001
#也可以使用--address和--accept-hosts参数来允许外部访问：
kubectl proxy --address='0.0.0.0'  --accept-hosts='^*$'
#kubernetes-dashboard
http://10.0.70.251:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/login

Pod 异常排错

kubectl get pod <pod-name> -o yaml 查看 Pod 的配置是否正确
kubectl describe pod <pod-name> 查看 Pod 的事件
kubectl logs <pod-name> [-c <container-name>] 查看容器日志
#
kubectl logs kubernetes-dashboard-548847967d-6rfh9 -n kube-system
# 具体原因需要查看 Kubelet 日志 按 unit 过滤日志
journalctl -u kubelet
#实时更新日志
journalctl -f -u kubelet
# 只显示最新的 n 行
journalctl -u kubelet -n 200
# 通过日志级别进行过滤，指定的优先级如下：
# 0: emerg
# 1: alert
# 2: crit
# 3: err
# 4: warning
# 5: notice
# 6: info
# 7: debug
journalctl -u kubelet -p err