k8s实验环境搭建

以下所有环境都在vmware 15完成。

===================centos7基础初始化====================================================

设置阿里云的yum源：

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

mv /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

cat < /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

yum clean all

yum makecache

yum update -y

安装基础软件：

yum install -y vim telnet net-tools docker ntp yum-utils device-mapper-persistent-data lvm2

关闭防火墙和selinux:

systemctl stop firewalld.service

systemctl disable firewalld.service

sed -i 's@SELINUX=enforcing@SELINUX=disabled@g' /etc/selinux/config

setenforce 0

getenforce

优化内核参数：

# 临时关闭swap

# 永久关闭注释/etc/fstab文件里swap相关的行

swapoff -a

# 配置转发相关参数，否则可能会出错

cat < /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

vm.swappiness=0

EOF

sysctl --system

# 加载ipvs相关内核模块

# 如果重新开机，需要重新加载

modprobe ip_vs

modprobe ip_vs_rr

modprobe ip_vs_wrr

modprobe ip_vs_sh

modprobe nf_conntrack_ipv4

lsmod | grep ip_vs

设置静态IP：

sed -i 's@BOOTPROTO=dhcp@BOOTPROTO=static@g' /etc/sysconfig/network-scripts/ifcfg-ens32

echo '''

IPADDR=192.168.1.10

NETMASK=255.255.255.0

GATEWAY=192.168.1.2

DNS1=192.168.1.2

''' >> /etc/sysconfig/network-scripts/ifcfg-ens32

cat /etc/sysconfig/network-scripts/ifcfg-ens32

systemctl restart network

同步时间，集群之间的时间同步很重要：

ntpdate cn.pool.ntp.org

===================docker搭建私有仓库、自签发证书、登录认证====================================================

设置主机名：

hostname registry.domain.com

echo registry.domain.com > /etc/hostname

启动docker:

systemctl restart docker

systemctl enable docker

生成自签发证书:

mkdir -p certs

openssl req -newkey rsa:2048 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt

ll certs/

生成鉴权密码文件:

mkdir auth

docker run --entrypoint htpasswd registry:2 -Bbn username password > auth/htpasswd

ls auth

启动Registry：

docker run -d -p 5000:5000 --restart=always --name registry -v `pwd`/auth:/auth -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v `pwd`/data:/var/lib/registry -v `pwd`/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:2

设置hosts解析域名：

echo "192.168.1.11 registry.domain.com" >> /etc/hosts

client安装我们的CA证书：

mkdir -p /etc/docker/certs.d/registry.domain.com:5000

cp certs/domain.crt /etc/docker/certs.d/registry.domain.com:5000/ca.crt

systemctl restart docker

登陆Registry:

docker login registry.domain.com:5000

pull image到registry：

docker pull busybox

docker tag busybox:latest registry.domain.com:5000/busybox:latest

docker push registry.domain.com:5000/busybox:latest

查看仓库中的镜像：

curl -u username:password -XGET https://registry.domain.com:5000/v2/_catalog -k

curl -u username:password -XGET https://registry.domain.com:5000/v2/hello-node/tags/list -k

===================Kubeadm安装Kubernetes简单集群环境====================================================

192.168.1.12 k8smaster01 软件：etcd k8smaster haproxy keepalived

192.168.1.13 k8snode01 软件：k8snode

设置主机名：

（在k8smaster01执行）

hostname k8smaster01

echo k8smaster01 > /etc/hostname

（在k8snode01执行）

hostname k8snode01

echo k8snode01 > /etc/hostname

（在所有节点执行）

设置hosts解析域名：

echo '''

192.168.1.12 k8smaster01

192.168.1.13 k8snode01

192.168.1.11 registry.domain.com

''' >> /etc/hosts

安装K8S相关组件：

（在所有节点执行）

yum install -y kubelet kubeadm kubectl ipvsadm

systemctl restart docker

systemctl enable docker

这里查到的结果为 cgroupfs.因此修改kubeadm的配置文件如下

vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

#添加如下配置

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

（在k8smaster01执行）

systemctl enable kubelet && systemctl start kubelet

启动不成功：

systemctl status kubelet 查看失败

打印日志查看:journalctl -xeu kubelet > a

cat a

Jan 16 01:48:55 k8snode01 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a

Jan 16 01:48:55 k8snode01 systemd[1]: Unit kubelet.service entered failed state.

Jan 16 01:48:55 k8snode01 systemd[1]: kubelet.service failed.

Jan 16 01:49:06 k8snode01 systemd[1]: kubelet.service holdoff time over, scheduling restart.

Jan 16 01:49:06 k8snode01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.

-- Subject: Unit kubelet.service has finished shutting down

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit kubelet.service has finished shutting down.

Jan 16 01:49:06 k8snode01 systemd[1]: Started kubelet: The Kubernetes Node Agent.

-- Subject: Unit kubelet.service has finished start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit kubelet.service has finished starting up.

-- The start-up result is done.

Jan 16 01:49:06 k8snode01 kubelet[8508]: F0116 01:49:06.193236 8508 server.go:189] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory

Jan 16 01:49:06 k8snode01 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a

Jan 16 01:49:06 k8snode01 systemd[1]: Unit kubelet.service entered failed state.

Jan 16 01:49:06 k8snode01 systemd[1]: kubelet.service failed.

网上找了一堆文档讲需要 kubeadm init成功。

比如：https://blog.csdn.net/zzq900503/article/details/81710319

运行： kubeadm init 继续报错：

[root@k8snode01 ~]# kubeadm init

I0116 02:00:49.485884 8984 version.go:94] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I0116 02:00:49.486035 8984 version.go:95] falling back to the local client version: v1.13.2

[init] Using Kubernetes version: v1.13.2

[preflight] Running pre-flight checks

error execution phase preflight: [preflight] Some fatal errors occurred:

[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

这儿有一个关键信息是：[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

意思是主机的CPU数量不能小于2，重新设置虚拟机。

到此kubelet不能启动的错误忽略！！！！！！！！！！！

继续初始化：

kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12

执行后报错:(因为无法访问国外网站，所以。。。。)

error execution phase preflight: [preflight] Some fatal errors occurred:

[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.2: output: Trying to pull repository k8s.gcr.io/kube-apiserver ...