k8s常见报错解决

【报错】

The Service "kubernetes-dashboard" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

【原因】

如果在service中没有指定type类型，默认采用ClusterIP

【解决】

# prometheus-service.yaml

---

spec:

type: NodePort #加入

ports:

- name: web

port: 9090

nodePort: 30000

---

【报错】

Error updating node status, will retry: error getting node "k8s-master-02": Get https://192.168.80.100:16443/api/v1/nodes/k8s-master-02?timeout=10s: dial tcp 192.168.80.100:16443: connect: no route to host

【分析】

查看高可用三个keepalived的三个节点是否存在vip,发现不存在。

[root@k8s-master-01 ~]# ip address show ens33

【解决】

systemctl restart keepalived

#访问地址：https://NodeIP:30001

---

【报错】

Client sent an HTTP request to an HTTPS server.

【解决】

使用https访问

---

【报错】

[root@k8s-master-03 ~]# kubectl get nodes

The connection to the server localhost:8080 was refused - did you specify the right host or port?

【解决】

#本机只需要生成，只需要创建，不需要再加入

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

---

【部署flannel报错】

[root@localhost ~]# etcdctl --endpoints="http://192.168.31.73:2379" set /atomic.io/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}} ' 
Error:  client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://192.168.31.73:2379 exceeded header timeout

error #0: client: endpoint http://192.168.31.73:2379 exceeded header timeout

#修改etcd与之对应IP地址

[root@localhost ~]# etcdctl --endpoints="http://192.168.4.114:2379" set /atomic.io/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}} ' 
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}} 

---

【报错】error: unable to recognize "hee.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
【解决】

#查看支持apiversion版本

[root@k8s-master ~]# kubectl api-resources  |grep deployment
deployments                       deploy       apps/v1                                true         Deployment

#修改文件

apiVersion: apps/v1

【报错】

error: error validating "hee.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

【解决】

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image:  quay.io/bitnami/heapster:1.5.4
        imagePullPolicy: IfNotPresent
        command:
        - /opt/bitnami/heapster/bin/heapster
        - --source=kubernetes:https://kubernetes.default

修改为：

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app:  heapster
  template:
    metadata:
      labels:
        app:  heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image:  quay.io/bitnami/heapster:1.5.4
        imagePullPolicy: IfNotPresent
        command:
        - /opt/bitnami/heapster/bin/heapster
        - --source=kubernetes:https://kubernetes.default

重新部署解决
 

---

【报错】

Error from server (Forbidden): error when creating "mandatory.yaml": configmaps "nginx-configuration" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated

【解决】

kubectl create namespace ingress-nginx

---

【报错】

[root@master ~]# kubectl apply -f mandatory.yaml

error: error validating "mandatory.yaml": error validating data: ValidationError(DaemonSet.spec): unknown field "replicas" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false

[root@master ~]# kubectl apply -f mandatory.yaml --validate=false

【解决】

spec:

# replicas: 1 #注释掉

selector:

---

[root@master nfs-external-provisioner]# kubectl apply -f deployment-pvc-sc.yaml

【报错】

error: error parsing deployment-pvc-sc.yaml: error converting YAML to JSON: yaml: line 7: did not find expected key

【解决】

 注意：line 7: did not find expected key不是指的第7行，而是在yaml格式中无缩进的第7个字符。

【报错】

[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

【解决】

echo "sysctl -w net.ipv4.ip_forward=1" >> /etc/rc.d/rc.local \

&& echo "sysctl -p" >> /etc/rc.d/rc.local \

&& chmod +x /etc/rc.d/rc.local \

&& ll /etc/rc.d/rc.local \

&& cat /proc/sys/net/ipv4/ip_forward

#重启生效

reboot

cat /proc/sys/net/ipv4/ip_forward

---

【报错】

Kubernetes报错Failed to get system container stats for "/system.slice/kubelet.service"

【解决】

在kubelet中追加配置

--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

---

【报错】

由于不正常停止docker导致node节点挂断，node像是NotReady,查看docker启动容器存在进程

【解决】

#查看运行异常容器docker ps
[root@k8s-node2 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
92b0ba527cba        prom/prometheus     "/bin/prometheus --c…"   11 days ago         Up 11 days                              k8s_prometheus-server_prometheus-0_prometheuse-cluster_c5b6c4ee-8f3c-4713-b9d3-ee75f803aae3_0
#查看容器的进程
[root@k8s-node2 ~]# ps -aux|grep  92b0ba527cba 
root      4707  0.0  0.1 108728  6584 ?        Sl   Oct10   3:56 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/92b0ba527cbb3b2be1b269a3d60461ffb0b0016c0ff38f01a240ea4976b0a7d -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root     27324  0.0  0.0 112828  2280 pts/0    S+   01:32   0:00 grep --color=auto 92b0ba527cba
#杀死进程
[root@k8s-node2 ~]# kill -9   4707
#然后查看集群状态
[root@k8s-master1 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
k8s-master1   Ready    master   12d   v1.18.0
k8s-node1     Ready       12d   v1.18.0
k8s-node2     Ready       66m   v1.18.0
k8s-node3     Ready       12d   v1.18.0

---

【报错】

kubelet, k8s-node2 Unable to attach or mount volumes: unmounted volumes=[config-volume], unattached volumes=[default-token-8d2bc localtime mail wechat config-volume storage-volume]: timed out waiting for the condition

【解决】

#登录k8s-node2查看日志

[root@k8s-node2 ~]# tailf -n 100 /var/log/messages

#发现报错

Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring

[root@k8s-node2 ~]# ls /etc/kubernetes/manifests

ls: cannot access /etc/kubernetes/manifests: No such file or directory

#创建目录，报错解决

cd /etc/kubernetes

mkdir -p manifests

---

【报错】error execution phase preflight: couldn't validate the identity of the API Server: Get https://192.168.4.114:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: dial tcp 192.168.4.114:6443: connect: no route to host

• 在使用 Minikube 部署 kubernetes 服务时，出现 Kube DNS 服务反复重启现象（错误如上），这很可能是 iptables 规则乱了，我通过执行以下命令解决了，在此记录：

1.回到kubernees-maser  依次输入列命令
systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker

2.重新生成新token
在kubernetse-master重新生成token：
# kubeadm token create
 424mp7.nkxx07p940mkl2nd
# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
d88fb55cb1bd659023b11e61052b39bbfe99842b0636574a16c76df186fd5e0d
3.在kubernetes-slave中执行此命令 join就成功了
#  kubeadm join 192.168.4.114:6443 –token 424mp7.nkxx07p940mkl2nd \ --discovery-token-ca-cert-hash sha256:d88fb55cb1bd659023b11e61052b39bbfe99842b0636574a16c76df186fd5e0d

---

【报错】

#接着当我执行kubectl get nodes等命令时，所有的命令都会打印出错误：

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kubernetes”)

【解决】

rm -rf $HOME/.kube

---

【报错】

Error from server (Forbidden): error when creating "common.yaml": serviceaccounts "rook-ceph-admission-controller" is forbidden: unable to create new content in namespace rook-ceph because it is being terminated

[root@k8s-master-01 ~]# kubectl get ns
NAME                   STATUS        AGE
default                Active        4d23h
kube-node-lease        Active        4d23h
kube-public            Active        4d23h
kube-system            Active        4d23h
kubernetes-dashboard   Active        4d23h
prometheus-cluster     Terminating   4d22h
prometheuse-cluster    Active        3h52m
rook-ceph              Active        4d22h
[root@k8s-master-01 ~]# kubectl get namespace prometheus-cluster -o json \
>             | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" \
>             | kubectl replace --raw /api/v1/namespaces/prometheus-cluster/finalize -f -
{"kind":"Namespace","apiVersion":"v1","metadata":{"name":"prometheus-cluster","selfLink":"/api/v1/namespaces/prometheus-cluster/finalize","uid":"277c644a-0b33-4f4d-b09b-3e949e8e3412","resourceVersion":"2105215","creationTimestamp":"2022-03-23T14:14:36Z","deletionTimestamp":"2022-03-28T08:55:55Z","managedFields":[{"manager":"kubectl","operation":"Update","apiVersion":"v1","time":"2022-03-23T14:14:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:phase":{}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2022-03-28T08:56:00Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}}]},"spec":{},"status":{"phase":"Terminating","conditions":[{"type":"NamespaceDeletionDiscoveryFailure","status":"True","lastTransitionTime":"2022-03-28T08:56:00Z","reason":"DiscoveryFailed","message":"Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"},{"type":"NamespaceDeletionGroupVersionParsingFailure","status":"False","lastTransitionTime":"2022-03-28T08:56:00Z","reason":"ParsedGroupVersions","message":"All legacy kube types successfully parsed"},{"type":"NamespaceDeletionContentFailure","status":"False","lastTransitionTime":"2022-03-28T08:56:00Z","reason":"ContentDeleted","message":"All content successfully deleted, may be waiting on finalization"},{"type":"NamespaceContentRemaining","status":"False","lastTransitionTime":"2022-03-28T08:56:00Z","reason":"ContentRemoved","message":"All content successfully removed"},{"type":"NamespaceFinalizersRemaining","status":"False","lastTransitionTime":"2022-03-28T08:56:00Z","reason":"ContentHasNoFinalizers","message":"All content-preserving finalizers finished"}]}}
[root@k8s-master-01 ~]# kubectl get ns
NAME                   STATUS   AGE
default                Active   4d23h
kube-node-lease        Active   4d23h
kube-public            Active   4d23h
kube-system            Active   4d23h
kubernetes-dashboard   Active   4d23h
prometheuse-cluster    Active   3h53m
rook-ceph              Active   4d22h

【解决方式】

1、找到对应namespace,然后删除掉。

2、修改红色部分执行命令即可删除

【报错】Error from server (Forbidden): error when creating "common.yaml": serviceaccounts "rook-ceph-admission-controller" is forbidden: unable to create new content in namespace rook-ceph because it is being terminated
【解决方式一】
1、找到对应namespace,然后删除掉。
2、修改红色部分执行命令即可删除
kubectl get namespace rook-ceph -o json \
            | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" \
            | kubectl replace --raw /api/v1/namespaces/rook-ceph/finalize -f -

【报错】level=error ts=2022-03-29T03:21:34.863574497Z caller=main.go:278 msg="Loading configuration file failed" file=/etc/config/alertmanager.yml err="read /opt/prometheus-cluster/alertmanager/mail.tmpl: is a directory"

【解决】

mkdir  -p /opt/prometheus-cluster/alertmanager    #删除目录，上传文件

 rz