树莓派4BUbuntu server 20.04 Kubernetes-v1.17.x- Docker19.03 keadmv1.10.3部署错误一览
树莓派4BUbuntu server 20.04 keadmv1.10.3加入主节点报错
execute keadm command failed: failed to exec 'bash -c sudo ln /etc/kubeedge/edgecore.service /etc/systemd/system/edgecore.service && sudo systemctl daemon-reload && sudo systemctl enable edgecore && sudo systemctl start edgecore', err: ln: failed to create hard link '/etc/systemd/system/edgecore.service': File exists
, err: exit status 1解决办法删除'/etc/systemd/system/edgecore.service'重新加入节点，成功。
之后journalctl -u edgecore.service -xe 打印日志
报错
Version: v1.10.3
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.205200 241646 server.go:103] Use node IP address from config: 192.168.137.74
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.205291 241646 sql.go:21] Begin to register twin db model
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.205673 241646 module.go:52] Module twin registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.228239 241646 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.228365 241646 client.go:97] "Start docker client with request timeout" timeout="0s"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.262259 241646 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.263310 241646 cni.go:204] "Error validating CNI config list" configList="{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"flannel\",\n >
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.263411 241646 cni.go:239] "Unable to update cni config" err="no valid networks found in /etc/cni/net.d"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.269385 241646 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.269521 241646 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.275158 241646 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="kubernetes.io/no-op"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.312914 241646 docker_service.go:263] "Docker Info" dockerInfo=&{ID:X4KE:OB4J:5JH2:IYG4:X2FR:G4LB:OOWP:FUNK:6L46:7LM5:7CD4:ONYQ Containers:0 ContainersRunning:0 ContainersPaused:0 Containers>
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.313017 241646 docker_service.go:278] "Setting cgroupDriver" cgroupDriver="cgroupfs"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.313345 241646 edged.go:716] RemoteRuntimeEndpoint: "unix:///var/run/dockershim.sock", remoteImageEndpoint: "unix:///var/run/dockershim.sock"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.313386 241646 edged.go:719] Starting the GRPC server for the docker CRI shim.
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.361424 241646 state_mem.go:36] "Initialized new in-memory state store"
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.361691 241646 edged.go:416] failed to get the edged's cgroup: mountpoint for memory not found. Edged system container metrics may be missing.
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.361945 241646 edged.go:423] failed to get the container runtime's cgroup: failed to get container name for docker process: mountpoint for memory not found. Runtime system container metrics >
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.560195 241646 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.670596 241646 kuberuntime_manager.go:245] "Container runtime initialized" containerRuntime="docker" version="19.03.12" apiVersion="1.40.0"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.670736 241646 edged.go:600] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.693689 241646 container_manager_linux.go:280] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.693873 241646 container_manager_linux.go:285] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntim>
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.693953 241646 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694000 241646 container_manager_linux.go:320] "Creating device plugin manager" devicePluginEnabled=false
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694039 241646 state_mem.go:36] "Initialized new in-memory state store"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694294 241646 module.go:52] Module edged registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694335 241646 module.go:52] Module websocket registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694357 241646 module.go:52] Module eventbus registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694512 241646 metamanager.go:47] Begin to register metaManager db model
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694691 241646 module.go:52] Module metaManager registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.694721 241646 module.go:55] Module servicebus is disabled, do not register
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.694768 241646 module.go:55] Module edgestream is disabled, do not register
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.694791 241646 module.go:55] Module testManager is disabled, do not register
Oct 10 16:09:23 node1 edgecore[241646]: table `device` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `device_attr` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `device_twin` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `sub_topics` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `meta` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `meta_v2` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `target_urls` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698655 241646 core.go:46] starting module twin
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698757 241646 core.go:46] starting module edged
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698828 241646 process.go:114] Begin to sync sqlite
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698881 241646 core.go:46] starting module websocket
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698889 241646 edged.go:303] Starting edged...
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698968 241646 core.go:46] starting module eventbus
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699042 241646 core.go:46] starting module metaManager
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699182 241646 http.go:40] tlsConfig InsecureSkipVerify true
Oct 10 16:09:23 node1 edgecore[241646]: E1010 16:09:23.699481 241646 csi_plugin.go:222] kubernetes.io/csi: CSIDriverLister not found on KubeletVolumeHost
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699585 241646 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699596 241646 common.go:97] start connect to mqtt server with client id: hub-client-sub-1665389363
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699921 241646 common.go:99] client hub-client-sub-1665389363 isconnected: false
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.705799 241646 client.go:157] finish hub-client sub
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.705943 241646 common.go:97] start connect to mqtt server with client id: hub-client-pub-1665389363
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.705970 241646 common.go:99] client hub-client-pub-1665389363 isconnected: false
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.706533 241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/upload/#
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.707352 241646 client.go:176] finish hub-client pub
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.707411 241646 eventbus.go:69] Init Sub And Pub Client for external mqtt broker tcp://127.0.0.1:1883 successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.707714 241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/device/+/state/update
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.708535 241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/device/+/twin/+
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.709211 241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/node/+/membership/get
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.709968 241646 client.go:95] edge-hub-cli subscribe topic to SYS/dis/upload_records
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.710532 241646 client.go:95] edge-hub-cli subscribe topic to +/user/#
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.711463 241646 client.go:103] list edge-hub-cli-topics status, no record, skip sync
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.162840 241646 cpu_manager.go:209] "Starting CPU manager" policy="none"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.162909 241646 cpu_manager.go:210] "Reconciling" reconcilePeriod="0s"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.162984 241646 state_mem.go:36] "Initialized new in-memory state store"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.163639 241646 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.163709 241646 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.163743 241646 policy_none.go:49] "None policy: Start"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.167587 241646 memory_manager.go:168] "Starting memorymanager" policy="None"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.167715 241646 state_mem.go:35] "Initializing new in-memory state store"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.168271 241646 state_mem.go:75] "Updated machine memory state"
Oct 10 16:09:24 node1 edgecore[241646]: E1010 16:09:24.169212 241646 edged.go:824] Failed to start container manager, err: system validation failed - Following Cgroup subsystem not mounted: [memory]
Oct 10 16:09:24 node1 edgecore[241646]: E1010 16:09:24.169265 241646 edged.go:307] initialize module error: system validation failed - Following Cgroup subsystem not mounted: [memory]
Oct 10 16:09:24 node1 systemd[1]: edgecore.service: Main process exited, code=exited, status=1/FAILURE查看错误日志
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.212882 466671 server.go:76] Version: v1.10.3
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.225448 466671 server.go:103] Use node IP address from config: 192.168.137.74
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.225600 466671 sql.go:21] Begin to register twin db model
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.226142 466671 module.go:52] Module twin registered successfully
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.249774 466671 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.249920 466671 client.go:97] "Start docker client with request timeout" timeout="0s"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.284696 466671 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.285558 466671 cni.go:204] "Error validating CNI config list" configList="{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\": true,\n \"isDefaultGateway\": true\n }\n >
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.285634 466671 cni.go:239] "Unable to update cni config" err="no valid networks found in /etc/cni/net.d"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.291651 466671 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.291816 466671 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.298601 466671 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="kubernetes.io/no-op"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.337487 466671 docker_service.go:263] "Docker Info" dockerInfo=&{ID:X4KE:OB4J:5JH2:IYG4:X2FR:G4LB:OOWP:FUNK:6L46:7LM5:7CD4:ONYQ Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] >
Oct 10 21:40:05 node1 edgecore[466671]: E1010 21:40:05.337737 466671 edged.go:279] init new edged error, misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"vim /etc/docker/daemon.jsn
# 把native.cgroupdriver=systemd 修改成 native.cgroupdriver=cgroupfs
docker info|grep "Cgroup Driver"
#检查 是否输出 Cgroup Driver: cgroupfs#重启docker 并且查看docker的状态
systemctl restart docker
systemctl status docker
docker 运行正常
vim /var/lib/kubelet/kubeadm-flags.env
把其中的systemd 也改成cgroupfs再启动下kubelet，重新载入配置
systemctl daemon-reload
systemctl restart kubelet
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet查看日志发现
level=warning msg="Your kernel does not support cgroup rt runtime"
master dockerd[22355]: time="2022-10-10T21:51:48.629194761+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
master dockerd[22355]: time="2022-10-10T21:51:48.629205446+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"主节点docker，此条警告可以忽略，不影响。
再次加入kubeedge，发现报错
Failed to start container manager, err: system validation failed - Following Cgroup subsystem not mounted: [memory]
2388 edged.go:307] initialize module error: system validation failed - Following Cgroup subsystem not mounted: [memory]
ess exited, code=exited, status=1/FAILURE日志显示的是内存资源的cgroups机制没有开启。
查看可用的cgroup:
root@node1:~/keadm-v1.10.3-linux-arm64/keadm# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 8 1 1
cpu 2 102 1
cpuacct 2 102 1
blkio 7 102 1
memory 0 112 0
devices 6 102 1
freezer 10 2 1
net_cls 3 1 1
perf_event 11 1 1
net_prio 3 1 1
hugetlb 9 1 1
pids 4 111 1
rdma 5 1 1修改启动参数
sudo vim /boot/firmware/cmdline.txt
添加以下内容：
cgroup_memory=1 cgroup_enable=memory添加在同一行的最后面,接着内容后空格后添加, 不要换行
#重启机器配置生效
reboot
#查看edgecore启动情况
ps aux|grep edgecore
root@node1:~/keadm-v1.10.3-linux-arm64/keadm# ps aux|grep edgecore
root 697 5.4 1.9 1657340 77492 ? Ssl 23:00 0:52 /usr/local/bin/edgecore
root 2563 0.0 0.0 7696 744 pts/0 S+ 23:16 0:00 grep --color=auto edgecore
#查看启动状态
sudo systemctl status edgecore
edgecore.service
Loaded: loaded (/etc/systemd/system/edgecore.service; bad; vendor preset: enabled)
Active: active (running) since Mon 2022-10-10 23:00:25 CST; 16min ago
Main PID: 697 (edgecore)
Tasks: 13 (limit: 4435)
Memory: 88.4M
CGroup: /system.slice/edgecore.service
└─697 /usr/local/bin/edgecore
#查看docker开机启动状态 enabled:开启, disabled:关闭
systemctl is-enabled edgecore
#设置开机自启动
systemctl enable edgecoreUbuntu18.04+树莓派Ubuntu20.04部署kubeedge v1.10.3主要参考帖子
(39条消息) Ubuntu20.04+树莓派部署Kubeedge v1.6.1_YsyNoobCoder的博客-CSDN博客_kubeedge 树莓派(39条消息) KubeEdge 超详细部署记录&问题记载6.28_Jason-Ning的博客-CSDN博客
验证环节
(39条消息) K8S v1.17.17&KubeEdge v1.7部署指南+kubeedge-counter-demo示例_cike90的博客-CSDN博客
常用指令
#查看 systemd托管列表
systemctl list-unit-files
systemctl start cloudcore.service && systemctl enable cloudcore.service
#查看节点状态
kubectl get node
kubectl get po -A
kubectl get po -n kube-system
kubectl get po -o wide
#查看网络状态
yum install net-tools -y
netstat -ntlp
#查看docker状态docker
docker images | grep kubeedge-pi-counter
#传文件
scp kubeedge-pi-counter.tar 192.168.xx.xx:/root/
tmp目录下的文件
注意事项：
1、一般情况下，使用完成尽量挂起虚拟机，不然容器停掉，启动后node会处于not reday状态。
2、如果node处于not reday状态了，执行一下重启脚本，执行命令如下：
bash restart_kubeedge.sh
脚本内容：
#!/bin/bash
for i in `docker ps -a | awk '{print $1}' | grep -v CONT`;do docker restart $i;done
for i in `docker ps -a | awk '{print $1}' | grep -v CONT`;do docker restart $i;done(39条消息) 【云原生1】零基础 k8s + kubeedge v1.5.0 部署_烈日松饼的博客-CSDN博客_kubeedge和k8s
（主要）
KubeEdge 部署记录 | 生命不息 折腾不止 (konata.tech)
排除错误：
kubernetes.io/csi: CSIDriverLister not found on KubeletVolumeHost(39条消息) KubeEdge1.10从零开始详细搭建教程_孟诗远msy的博客-CSDN博客_kubeedge搭建