kubelet cgroup driver: "cgroupfs" is different from dock
服务器：阿里云服务器 centos7.3 64位 2cpu 2gb内存
tips: # 后面都是我输入的命令，下面是输入命令出现的提示
干跑集群的时候
# kubeadm init --kubernetes-version="v1.14.2" --pod-network-cidr="10.244.0.0/16" --dry-run
报错，报kubelet 好像没有运行的错误，于是我就查看kubelet的状态
# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since 二 2019-06-04 11:18:42 CST; 1s ago
Docs: https://kubernetes.io/docs/
Process: 11375 ExecStart=/usr/bin/kubelet (code=exited, status=255)
Main PID: 11375 (code=exited, status=255)
6月 04 11:18:42 izuf6j4eztzg3cjw0slvhhz systemd[1]: Unit kubelet.service entered failed state.
6月 04 11:18:42 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service failed.
好像也看不出查错误，我再用日志命令看下
# journalctl -xeu kubelet
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service holdoff time over, scheduling restart.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: Starting kubelet: The Kubernetes Node Agent...
-- Subject: Unit kubelet.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has begun starting up.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.609859 11638 server.go:417] Version: v1.14.2
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.610047 11638 plugins.go:103] No cloud provider specified.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.610065 11638 server.go:556] standalone mode, no API client
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.658412 11638 server.go:474] No api server defined - no events will be sent to API server.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658435 11638 server.go:625] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658709 11638 container_manager_linux.go:261] container manager verified user specified cgroup-root exists: [
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658723 11638 container_manager_linux.go:266] Creating Container Manager object based on Node Config: {Runtim
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658840 11638 container_manager_linux.go:286] Creating device plugin manager: true
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658879 11638 state_mem.go:36] [cpumanager] initializing new in-memory state store
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.659009 11638 state_mem.go:84] [cpumanager] updated default cpuset: ""
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.659019 11638 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.661629 11638 client.go:75] Connecting to docker on unix:///var/run/docker.sock
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.661665 11638 client.go:104] Start docker client with request timeout=2m0s
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.664004 11638 docker_service.go:561] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, fal
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.664038 11638 docker_service.go:238] Hairpin mode set to "hairpin-veth"
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.664198 11638 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.667500 11638 docker_service.go:253] Docker cri networking managed by kubernetes.io/no-op
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.688598 11638 docker_service.go:258] Docker Info: &{ID:HVR5:RN62:NPRF:YA4J:YBXP:E4C4:RYGP:5WVK:MGMW:TWGE:VWQE
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: F0604 11:19:53.688710 11638 server.go:265] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgrou
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: Unit kubelet.service entered failed state.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service failed.
这个命令显示的不会换行，要看后面的，用右方向键就可以了
看到这句没有
failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
告诉我们失败的原因是你的docker 运行的cgroup driver 和 kubelet 的 cgroup driver 运行的方式不是一样的，我们需要改成一样的。这里我们修改docker的。
# vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
把native.cgroupdriver=systemd 修改成 native.cgroupdriver=cgroupfs
重启docker 并且查看docker的状态
# systemctl restart docker
# systemctl status docker
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since 二 2019-06-04 13:36:00 CST; 56s ago
Docs: https://docs.docker.com
Main PID: 7663 (dockerd)
Memory: 35.7M
CGroup: /system.slice/docker.service
└─7663 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
docker 运行正常
我们在启动下kubelet，下面的命令不能少，配置要重载入一下，不然下面创建集群会失败
# systemctl daemon-reload
# systemctl restart kubelet
# systemctl enable kubelet
# systemctl start kubelet
# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since 二 2019-06-04 13:36:06 CST; 2min 57s ago
Docs: https://kubernetes.io/docs/
Main PID: 7788 (kubelet)
Memory: 27.9M
CGroup: /system.slice/kubelet.service
└─7788 /usr/bin/kubelet
再次运行干跑命令，干跑命令会告诉你创建成功，但实际创建中会报错
# kubeadm init --kubernetes-version="v1.14.2" --pod-network-cidr="10.244.0.0/16"
[init] Using Kubernetes version: v1.14.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Hostname]: hostname "izuf6j4eztzg3cjw0slvhhz" could not be reached
[WARNING Hostname]: hostname "izuf6j4eztzg3cjw0slvhhz": lookup izuf6j4eztzg3cjw0slvhhz on 100.100.2.138:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
****
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
上面提示说，你的 kubelet 好像没有运行，kubelet 不是很健康，通过命令查看
# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: inactive (dead) since 二 2019-06-04 15:27:36 CST; 5min ago
Docs: https://kubernetes.io/docs/
Process: 11611 ExecStart=/usr/bin/kubelet (code=exited, status=0/SUCCESS)
Main PID: 11611 (code=exited, status=0/SUCCESS)
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...time.go:76
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...time.go:65
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...time.go:51
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /usr/local/go/src/runtime/panic.go:522
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...tor.go:189
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...tor.go:214
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...tor.go:125
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...ait.go:152
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Hint: Some lines were ellipsized, use -l to show in full.
什么鬼，刚才还运行正常呢，看下日志
# journalctl -xeu kubelet
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
-- Subject: Unit kubelet.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has begun shutting down.
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished shutting down.
没看到什么有用的信息，先根据 kubeadm init 时的错误日志来尝试下解决
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
****
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
上面提示到建议用 systemd 驱动,实际运行中，kubeadm init 执行后到了kubelet时
，会报超时，再去查看kubelet的状态，会发现，已经不是runngin的状态了。
利用下面的命令，把其中的systemd 也改成cgroupfs
# vim /var/lib/kubelet/kubeadm-flags.env
# vim /etc/default/kubelet
修改成一样的以后，再次执行 kubeadm init ,不会终止kubelet ，但依然会报kubelet not running 等错误
查找 资料
https://github.com/kubernetes/kubeadm/issues/1315
说缺少文件
# vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
文件内容
https://github.com/kubernetes/kubernetes/blob/master/build/rpms/10-kubeadm.conf
添加内容后，重启kubelet,并再次执行kubeadm init 命令，依然报错，但kubelet 还在执行状态。
报错日志
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
最后，把服务器系统升级到了centos7.6 64位，就可以了，升级系统，升级系统，升级系统。
操作系统:centos7.6 64位
内核
# uname -a
Linux iZuf6j4eztzg3cjw0slvhhZ 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
docker
# docker info
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 7
Server Version: 18.09.6
k8s 版本
kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
至此，k8s集群就建立成功了。其实如果版本和服务器版本能兼容的话还是挺容易的，主要是找对版本，我用的是最新的k8s版本，1.4.2的稳定版本，centos6.3 系统支持的并不是太好。