24 k8s+kubeSphere 一篇就够

24.1配置宿主机网络

查看是否支持虚拟化

grep -E '(vmx|svm)' /proc/cpuinfo

安装KVM需要的软件包

yum install qemu virt kvm -y

Question：

image.png

Solution：已安装的跳过

yum install qemu virt kvm -y --skip-broken

开机自启

systemctl start libvirtd

systemctl enable libvirtd

验证是否启动成功

virsh list

image.png

桥接工具包

yum install -y bridge-utils

#配置桥接模式

cd /etc/sysconfig/network-scripts

cp ifcfg-em2 ifcfg-br0

两个文件都要改 只保留桥接的IP

[root@localhost network-scripts]# vim ifcfg-em2

TYPE=Ethernet

BRIDGE=br0

PROXY_METHOD=none

BROWSER_ONLY=no

BOOTPROTO=none

DEFROUTE=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_FAILURE_FATAL=no

IPV6_ADDR_GEN_MODE=stable-privacy

NAME=em2

UUID=74c8085f-4c0d-4743-b0a0-70e51e3eb877

DEVICE=em2

ONBOOT=yes

IPADDR=172.16.10.5

PREFIX=24

GATEWAY=172.16.10.254

DNS1=114.114.114.114

#注意****IPADDR**** 要改为自己的

[root@localhost network-scripts]# vim ifcfg-br0

TYPE=Bridge

PROXY_METHOD=none

BROWSER_ONLY=no

BOOTPROTO=none

DEFROUTE=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_FAILURE_FATAL=no

IPV6_ADDR_GEN_MODE=stable-privacy

NAME=br0

DEVICE=br0

ONBOOT=yes

IPADDR=172.16.10.3

PREFIX=24

GATEWAY=172.16.10.254

DNS1=114.114.114.114

重启网络

systemctl restart network

#验证

brctl show

image.png

24.2 安装虚拟机

cd /home/kvm

#创建****master****虚拟机的存储盘**** 10.4

qemu-img create -f qcow2 -o cluster_size=2M k8s-master01.qcow2 200G

安装虚拟机壳子 --name k8s-master01.qcow2 --memory 8192 --vcpus 4 可改

virt-install --virt-type kvm --os-type=linux --os-variant rhel7 --name k8s-master01.qcow2 --memory 8192 --vcpus 4 --disk /home/kvm/k8s-master01.qcow2,format=qcow2 --cdrom /home/kvm/CentOS-7-x86_64-DVD-2009.iso --network bridge=br0 --graphics vnc,listen=0.0.0.0 --noautoconsole

#创建****worker****虚拟机的存储盘**** 10.5

qemu-img create -f qcow2 -o cluster_size=2M k8s-worker01.qcow2 200G

安装虚拟机壳子 --name k8s-worker01.qcow2 --memory 8192 --vcpus 4 可改

virt-install --virt-type kvm --os-type=linux --os-variant rhel7 --name k8s-worker01.qcow2 --memory 8192 --vcpus 4 --disk /home/kvm/k8s-worker01.qcow2,format=qcow2 --cdrom /home/kvm/CentOS-7-x86_64-DVD-2009.iso --network bridge=br0 --graphics vnc,listen=0.0.0.0 --noautoconsole

#创建****worker****虚拟机的存储盘**** 10.3

qemu-img create -f qcow2 -o cluster_size=2M k8s-worker02.qcow2 200G

安装虚拟机壳子--name k8s-worker02.qcow2 --memory 8192 --vcpus 4 可改

virt-install --virt-type kvm --os-type=linux --os-variant rhel7 --name k8s-worker02.qcow2 --memory 32768 --vcpus 32 --disk /home/kvm/k8s-worker02.qcow2,format=qcow2 --cdrom /home/kvm/CentOS-7-x86_64-DVD-2009.iso --network bridge=br0 --graphics vnc,listen=0.0.0.0 --noautoconsole

迅速通过vnc viewer 去安装配置操作系统

image.png

image.png

查看虚拟机启动

netstat -ntlp | grep 5900

virsh list --all

virsh shutdown k8s-master01.qcow2

virsh start k8s-master01.qcow2

虚拟机密码

ssh 172.16.10.50 root@starQuest2022

Question****：系统启动卡住

image.png

Solution：

virsh destroy k8s-master01.qcow2

virsh undefine k8s-master01.qcow2

Question****：更改桥接模式失败引发的问题

image.png

image.png

Solution：

1. 更正桥接模式 只保留桥接ip其他网卡配置均清空ip

2. 重启虚拟机 验证

   
   
   image.png

24.3 配置虚拟机****网络

配置网卡 修改网段

vi /etc/sysconfig/network-scripts/ifcfg-eth0

TYPE=Ethernet

PROXY_METHOD=none

BROWSER_ONLY=no

BOOTPROTO=static

DEFROUTE=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_FAILURE_FATAL=no

IPV6_ADDR_GEN_MODE=stable-privacy

NAME=eth0

UUID=c510f2f9-9820-45e8-9c70-65674bd35258

DEVICE=eth0

ONBOOT=yes

IPADDR=172.16.10.50

PREFIX=24

GATEWAY=172.16.10.254

DNS1=114.114.114.114

重启网络

systemctl restart network

Question：

image.png

Solution：

vi /root/.ssh/known_hosts 删除有问题IP对应行

image.png

24.4 更新系统 升级内核

#设置hostname

hostnamectl set-hostname k8s-master01

hostnamectl set-hostname k8s-worker01

hostnamectl set-hostname k8s-worker02

yum update

yum install wget

yum install vim

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available

yum --enablerepo=elrepo-kernel install kernel-lt -y

让开机内核生效

vi /etc/default/grub

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR=",,g' /etc/system-release)"

改为0

GRUB_DEFAULT=0

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL_OUTPUT="console"

GRUB_CMDLINE_LINUX="crashkernel=auto spectre_v2=retpoline rhgb quiet"

GRUB_DISABLE_RECOVERY="true"

让配置生效

grub2-mkconfig -o /boot/grub2/grub.cfg

重启并查看

reboot

uname -a

24.5 安装docker

方式一 配置yum源

cd /etc/yum.repos.d

传文件

scp -r docker-ce.repo 172.16.10.51:/etc/yum.repos.d/

安装启动

yum install docker-ce

方式二 yum 安装rpm包

yum install -y docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm

#开机自启动

systemctl start docker

systemctl enable docker

配置守护文件

vi /etc/docker/daemon.json

{

"exec-opts": [

"native.cgroupdriver=systemd"

],

"log-driver": "json-file",

"log-level": "warn",

"log-opts": {

"max-size": "1000m",

"max-file": "3"

},

"registry-mirrors": [

"https://zydiol88.mirror.aliyuncs.com"

],

"insecure-registries": ["harbor.bicisims.com"],

"selinux-enabled": false

}

让K8识别，配置镜像仓库地址

重启

systemctl restart docker

验证

docker ps

24.6 安装k8s准备工作

修改时区，同步时间

yum install ntpdate -y

ntpdate time2.aliyun.com

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

echo 'Asia/Shanghai' > /etc/timezone

crontab -e

0 12 * * * /usr/sbin/ntpdate time2.aliyun.com

关闭防火墙，selinux，swap

systemctl stop firewalld

systemctl disable firewalld

systemctl status firewalld

关闭内置selinux

sed -i 's/enforcing/disabled/' /etc/selinux/config

setenforce 0

swapoff -a

sed -ri 's/.swap./#&/' /etc/fstab

系统优化 net.ipv4.ip_forward=1 让docker互相通信

cat > /etc/sysctl.d/k8s_better.conf << EOF

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

net.ipv4.ip_forward=1

net.ipv4.tcp_tw_recycle=0

vm.swappiness=0

vm.overcommit_memory=1

vm.panic_on_oom=0

fs.inotify.max_user_instances=8192

fs.inotify.max_user_watches=1048576

fs.file-max=52706963

fs.nr_open=52706963

net.ipv6.conf.all.disable_ipv6=1

net.netfilter.nf_conntrack_max=2310720

EOF

验证

cat /etc/sysctl.d/k8s_better.conf

sysctl -p /etc/sysctl.d/k8s_better.conf

image.png

上面这两个错忽略

配置ipvs k8s需要的网络支持

cat > /etc/sysconfig/modules/ipvs.modules <

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe -- nf_conntrack

EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash

配置yum源

vi /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

enabled=1

gpgcheck=0

repo_gpgcheck=0

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg

https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

24.7 安装k8s

#必须确认是否更改hostname

hostnamectl set-hostname k8s-master01

安装kubelet

yum install -y kubelet-1.23.5 kubeadm-1.23.5 kubectl-1.23.5

启动kubelet

systemctl enable kubelet

用kubeadm 安装k8s 只在master上初始化

kubeadm init --apiserver-advertise-address=172.16.10.50 --kubernetes-version=1.23.5 --image-repository registry.aliyuncs.com/google_containers --service-cidr=10.96.0.0/16 --pod-network-cidr=10.244.0.0/16

安装成功后信息

image.png

[

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown (id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.10.50:6443 --token dak7im.w25v1sjl0kcm4y3c \

--discovery-token-ca-cert-hash sha256:afb2a0b22a3e563671103f93965f71a915f65054db74b7ffa97a84932a098f42

]

增加host

vi /etc/hosts

172.16.10.50 k8s-master01

172.16.10.51 k8s-worker01

172.16.10.52 k8s-worker02

master 验证

kubectl get nodes

重新生成token

kubeadm token create --print-join-command

worker节点 加入master kubeadm join 172.16.10.50:6443 --token 1agi0a.bsluty1ad11px2j0 --discovery-token-ca-cert-hash sha256:a630b36df4a96c76a76ced3d4ac82373ec55549464478818cb1c1c361b606835

验证

kubectl version

配置kubectl环境

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown (id -g) $HOME/.kube/config

上传配置文件

scp -r conf/ 172.16.10.50:/home/software/

image.png

运用flannel网络插件

wget https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

kubectl apply -f kube-flannel.yml

验证

kubectl get pod -A

image.png

Question:

image.png

Solution:

kubectl explain DaemonSet

Question:k8s Node 一直 pending****Solution:

修改apiserver的yaml文件

$ vim /etc/kubernetes/manifests/kube-apiserver.yaml

spec:

containers:

• command:

  • kube-apiserver

  • --feature-gates=RemoveSelfLink=false

执行apiserver文件（twice）

$ kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

$ kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

验证

kubectl get pod

image.png

安装nfs客户端

yum install -y nfs-utils

systemctl enable nfs

systemctl start nfs

24.8 重要文件

24.9安装nfs

1)安装NFS

服务端(安装在磁盘空间健康的机器 )

$ yum install -y nfs-utils rpcbind

客户端(在所有K8S的所有节点都需要安装)

$ yum install -y nfs-utils

2)启动服务

服务器端

systemctl enable rpcbind

systemctl start rpcbind

systemctl restart rpcbind

客户端

systemctl enable nfs

systemctl start nfs

systemctl restart nfs

3 )创建共享目录 服务器端

mkdir -p /home/data

vi /etc/exports

编写NFS的共享配置

/home/data *(rw,sync,no_root_squash)

*代表对所有IP都开放此目录,rw是读写

4)查看NFS共享目录 服务器端

showmount -e 172.16.10.5

5)如果要把其他服务器的磁盘加进来 就在对应的服务器安装 nfs服务端 然后建立共享文件夹

验证

kubectl get pod -A

image.png

24.10 安装storageclass

cd /root/tools/storageclass/

image.png

修改

vim nfs-provisioner.yaml

image.png

应用yml配置文件做storageclass

kubectl apply -f rbac.yaml

kubectl apply -f nfs-provisioner.yaml

kubectl apply -f nfs-StorageClass.yaml

配置默认的storageclass

kubectl patch storageclass huaweinfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

24.11安装****kubesphere

wget https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/kubesphere-installer.yaml

wget https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/cluster-configuration.yaml

kubectl apply -f kubesphere-installer.yaml

kubectl apply -f cluster-configuration.yaml

实时查看安装进度

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

Welcome to KubeSphere!

Console: http://172.16.10.50:30880

Account: admin

Password: P@88w0rd starQuest2022