k8s集群搭建

 

上来装centos7集群敲着命令呢，突然网断了，我以为我哪里命令搞错了，弄崩了系统，重装，之前因为已经zhua装好一个centos7并且多装了个vmtools,后面查明

cd /etc/sysconfig/network-scripts/ vim  ifcfg-ens33  （ifconfig 获取）（最基本安装连这个命令都没有，我后面都带界面安装了）

ONBOOT=no

改成

ONBOOT=yes

service network restart

先讲用kubeadm安装kubernetes集群

systemctl disable firewalld

systemctl stop firewalld

setenforce 0或者改/etc/sysconfig/selinux 将SELINUX=enforcing 改成SELINUX=disabled

以上操作我暂时没动

首先是安装docker 

yum install -y docker这个在国内貌似没问题

[lixl@localhost ~]$ docker --version
Docker version 1.13.1, build 07f3374/1.13.1

装kubeadm各路都有，主要是源的问题，我找的的方法是

https://blog.csdn.net/zhuchuangang/article/details/76572157#2

下载kubernetes镜像

https://blog.csdn.net/zhuchuangang/article/details/76572157

#kubernetes yum源
vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
然后就是

yum install -y kubelet

yum install -y kubeadm

安装好这两个，yum install -y kubectl kubernetes-cni时说都已经安装完成，估计其实是kubeadm都集成了，第一个命令都不一定需要。

systemctl start docker

systemctl enable docker

systemctl enable kubelet

systemctl start kubelet

[root@localhost ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

接下来下载kubernetes的docker镜像

由于kubeadmjian将自动下载kubernetes的相关镜像，默认gcr.io下载（没看出来）

修改docker配置/vim /etc/sysconfig/docker，增加registry mirror参数=false那个是我加的

OPTIONS='--selinux-enabled=false --log-driver=journald --signature-verification=false'
OPTIONS='--registry-mirror=http://68e02ab9.m.daocloud.io'

kubeadm  init --kubernetes-version=1.6.0
this version of kubeadm only supports deploying clusters with the control plane version >= 1.12.0. Current version: v1.6.0

[root@localhost ~]# kubeadm  init --kubernetes-version=1.12.0
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
        [ERROR Swap]: running with swap on is not supported. Please disable swap
        [ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version 

systemctl disable firewalld

systemctl stop firewalld

vim /etc/sysctl.conf 
     net.bridge.bridge-nf-call-ip6tables = 1
     net.bridge.bridge-nf-call-iptables = 1
     net.bridge.bridge-nf-call-arptables = 1重启

或者

创建/etc/sysctl.d/k8s.conf文件

sudo vi /etc/sysctl.d/k8s.conf
添加如下内容：

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

sudo sysctl -p /etc/sysctl.d/k8s.conf

https://blog.csdn.net/zzq900503/article/details/81710319

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

ansible all -m shell -a "iptables -P FORWARD ACCEPT"

永久修改：/usr/lib/sysctl.d/00-system.conf

SELINUX=enforcing

swapoff -a

https://blog.csdn.net/CSDN_duomaomao/article/details/75142769?locationNum=8&fps=1

[root@localhost sysctl.d]# kubeadm init --kubernetes-version=1.13.0
[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-apiserver ... 
Get https://k8s.gcr.io/v1/_ping: dial tcp 108.177.97.82:443: getsockopt: connection refused
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-controller-manager ... 
Get https://k8s.gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection refused
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-scheduler ... 
Get https://k8s.gcr.io/v1/_ping: dial tcp 64.233.187.82:443: getsockopt: connection refused
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-proxy ... 
Get https://k8s.gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection refused
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Trying to pull repository k8s.gcr.io/pause ... 
Get https://k8s.gcr.io/v1/_ping: dial tcp 108.177.125.82:443: getsockopt: connection refused
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Trying to pull repository k8s.gcr.io/etcd ... 
Get https://k8s.gcr.io/v1/_ping: dial tcp 74.125.204.82:443: getsockopt: connection refused
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Trying to pull repository k8s.gcr.io/coredns ... 
Get https://k8s.gcr.io/v1/_ping: dial tcp 74.125.204.82:443: getsockopt: connection refused
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

https://blog.csdn.net/jinguangliu/article/details/82792617

kubeadm config images list 根据要求写名字，不能写错，或者根据错误写docker images

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

kubeadm reset

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
        - 'docker ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

运行journalctl -xeu kubelet

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://192.168.41.137:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.41.137:6443: connect: connection refused

eviction manager: failed to get get summary stats: failed to get node 

https://blog.csdn.net/bbwangj/article/details/82024485

按照上面的方法始终不奏效，错误又是不停的循环滚动，到底也不知道哪里错了

后面http://blog.51cto.com/ghbsunny/2162205?source=dra

搜到最后这个错误，原来是内存小了，改成2G可以了

至于network那个后面kubeadm启动成功后会有系统解决方式告知

这里面的问题解决https://blog.csdn.net/u012286287/article/details/79716588

解决方案https://blog.csdn.net/zzq900503/article/details/81710319

kubelet 的cgroup dirver 与 docker的不一样。docker默认使用cgroupfs,keubelet 默认使用systemd没遇到，我是一致的

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.41.137:6443 --token 2dva0q.rr3htfw7s4ofmjh4 --discovery-token-ca-cert-hash sha256:9384042d6812ff7631f0c3c5c5b827ccbefdbbac6d9213611c077799e89bfc6e

最后选了这个apply， 版本有点多

mkdir -p ~/k8s/
cd ~/k8s
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f  kube-flannel.yml
[root@master k8s]# kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                             READY   STATUS              RESTARTS   AGE    IP               NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-86c58d9df4-6m2vv         0/1     ContainerCreating   0          31m               master            
kube-system   coredns-86c58d9df4-xttbs         0/1     ContainerCreating   0          31m               master            
kube-system   etcd-master                      1/1     Running             0          30m    192.168.41.137   master            
kube-system   kube-apiserver-master            1/1     Running             0          30m    192.168.41.137   master            
kube-system   kube-controller-manager-master   1/1     Running             0          30m    192.168.41.137   master            
kube-system   kube-flannel-ds-amd64-ftjsk      0/1     CrashLoopBackOff    3          6m4s   192.168.41.137   master            
kube-system   kube-proxy-bnzzx                 1/1     Running             0          31m    192.168.41.137   master            
kube-system   kube-scheduler-master            1/1     Running             0          30m    192.168.41.137   master            

journalctl -u kubelet -f

kubectl describe pod coredns-86c58d9df4-xttbs -n kube-system

kubectl describe pods --all-namespaces

kubectl logs coredns-86c58d9df4-6m2vv -n kube-system

kubectl get pods --all-namespaces

kubectl get pod --all-namespaces -o wide

kubectl get pods -n kube-system -o wide

kubeadm reset

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example: 
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables

kubeadm init --kubernetes-version=v1.13.0 --pod-network-cidr=10.244.0.0/16 

kubeadm join 192.168.41.137:6443 --token ycd1dl.xza4hi7b4prr0387 --discovery-token-ca-cert-hash sha256:15299a96ced577a2a865216b9240511d47e940e961ac461970a55f12e2b564be

docker pull quay.io/coreos/flannel:v0.10.0-amd64 
mkdir -p /etc/cni/net.d/
cat < /etc/cni/net.d/10-flannel.conf
{"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}}
EOF
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat < /run/flannel/subnet.env
FLANNEL_NETWORK=172.100.0.0/16
FLANNEL_SUBNET=172.100.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

集群初始化如果遇到问题，可以使用下面的命令进行清理再重新初始化：

kubeadm reset

[root@master k8s]# iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/

[runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
mkdir -p ~/k8s/
cd ~/k8s
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

问题解决参考

vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS='--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice'

https://blog.csdn.net/qq_34857250/article/details/82562514

open /run/flannel/subnet.env

http://dockone.io/question/1225

name: Invalid value: "kube_master": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

/etc/hosts中的命名有非法字符，改掉 service network restart

出处：https://www.jianshu.com/p/f9a54e553ce4

 [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[root@master ~]# sudo sysctl -p /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
net.ipv4.ip_forward = 1

或者vim /etc/rc.d/rc.local

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

less /proc/sys/net/ipv4/ip_forward

kubeadm init --kubernetes-version=v1.13.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.41.137 --token-ttl=0

kubeadm init --kubernetes-version=v1.13.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.41.137用于广播所有成员...  ken-ttl 0  集群token不过期

–apiserver-advertise-address指明用Master的哪个interface与Cluster 的其他节点通信。 如果Master有多个interface， 建议明确指定， 如果 不指定， kubeadm会自动选择有默认网关的interface。

• apiserver-advertise-address该参数一般指定为haproxy+keepalived 的vip。
• pod-network-cidr 主要是在搭建pod network（calico）时候需要在init时候指定。

The connection to the server 192.168.41.137:8080 was refused - did you specify the right host or port?

export KUBECONFIG=/etc/kubernetes/admin.conf   

解决方案https://www.jianshu.com/p/6fa06b9bbf6a

http://blog.51cto.com/ghbsunny/2162205?source=dra

https://blog.csdn.net/ximenghappy/article/details/70157361