k8s常用操作和常见问题

写在前面

本文记录k8s的常见操作，毕竟好记性不如烂笔头。

1：常用操作

1.1：pod查看日志

dongyunqi@dongyunqi-virtual-machine:~/test$ kubectl logs busy-pod
ubuntu, on

因为pod中只有一个容器，所以不需要指定容器，当有多个容器时需要指定容器，如下:

dongyunqi@dongyunqi-virtual-machine:~/test$ kubectl logs ngx-num1-pod -c ngx1111
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
...

1.2：查询POD

可以直接查看也可以通过label过滤查看，如下：

dongyunqi@mongodaddy:~/k8s$ kubectl get pod
NAME                           READY   STATUS    RESTARTS        AGE
ngx-dep-name-dcc8b7bfd-7lbrq   1/1     Running   0               17m
ngx-dep-name-dcc8b7bfd-9m2rs   1/1     Running   0               17m
ngx-dep-name-dcc8b7bfd-wp4zt   1/1     Running   0               17m
redis-ds-5jf5x                 1/1     Running   1 (4h34m ago)   40h
redis-ds-n8p45                 1/1     Running   1 (4h34m ago)   40h
5：可以根据POD的label来过滤POD，支持使用==、!=、in、notin等表达式，如下：
    dongyunqi@mongodaddy:~/k8s$ kubectl get pod -l 'app in (ngx, nginx, ngx-dep-pod)'
    NAME                            READY   STATUS    RESTARTS   AGE
    ngx-dep-name-577dd8d59b-4xkd2   1/1     Running   0          3m34s
    ngx-dep-name-577dd8d59b-dcx8k   1/1     Running   0          3m34s
    ngx-dep-name-577dd8d59b-f2kxp   1/1     Running   0          3m34s
    dongyunqi@mongodaddy:~/k8s$ kubectl get pod -l 'app=ngx-dep-pod'
    NAME                            READY   STATUS    RESTARTS   AGE
    ngx-dep-name-577dd8d59b-4xkd2   1/1     Running   0          3m59s
    ngx-dep-name-577dd8d59b-dcx8k   1/1     Running   0          3m59s
    ngx-dep-name-577dd8d59b-f2kxp   1/1     Running   0          3m59s

1.3：查看pod中运行的容器

1.3.1：describe

dongyunqi@mongodaddy:~/k8s$ kubectl describe pod ngx-dep-name-dcc8b7bfd-7lbrq | grep Events -A 100
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  34m   default-scheduler  Successfully assigned default/ngx-dep-name-dcc8b7bfd-7lbrq to mongomummy # pod分配的Node
  Normal  Pulled     34m   kubelet            Container image "nginx:alpine" already present on machine # 创建容器使用的镜像
  Normal  Created    34m   kubelet            Created container nginx # 创建容器，显示的是容器的名称
  Normal  Started    34m   kubelet            Started container nginx
dongyunqi@mongodaddy:~/k8s$ kubectl describe pod ngx-dep-name-dcc8b7bfd-7lbrq | grep Containers: -A 100
Containers: # 所有的容器信息
  nginx: # 容器的名称
    Container ID:   docker://87afa9c8b6c1a9c8434a5dcbe39eb55863ecff4c91a68b408a3fd5f11b18a86f
    Image:          nginx:alpine # 容器使用的镜像
    Image ID:       docker-pullable://nginx@sha256:eb05700fe7baa6890b74278e39b66b2ed1326831f9ec3ed4bdc6361a4ac2f333
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 12 Jan 2023 14:59:05 +0800
    Ready:          True
    Restart Count:  0 # 容器重启的此时
    Environment:    
    Mounts:
      /etc/nginx/conf.d from ngx-conf-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlbhf (ro)

1.3.2：jsonpath过滤

dongyunqi@mongodaddy:~/k8s$ kubectl get pods ngx-dep-name-dcc8b7bfd-7lbrq -o jsonpath={.spec.containers[*].name}
nginx

1.4：通过events排查异常POD

当POD运行异常时，如一直PENDING，可通过kubectl describe查看异常信息：

dongyunqi@mongodaddy:~/k8s$ kubectl describe pod ngx-dep-name-dcc8b7bfd-7lbrq | grep Events -A 100
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  34m   default-scheduler  Successfully assigned default/ngx-dep-name-dcc8b7bfd-7lbrq to mongomummy # pod分配的Node
  Normal  Pulled     34m   kubelet            Container image "nginx:alpine" already present on machine # 创建容器使用的镜像
  Normal  Created    34m   kubelet            Created container nginx # 创建容器，显示的是容器的名称
  Normal  Started    34m   kubelet            Started container nginx

这里展示的正常，如果是发生错误，则在Message列会有对应的提示。如下是我从网上找的一个比较典型的错误信息：

0/5 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 4 Insufficient memory.

0/5是说5个容器，都不可用，1 node(s) had taint {node-role.kubernetes.io/master: }其中一个是因为有污点 , 4 Insufficient memory 另外四个是因为内存不足了，这样我们就可以对症下药了。

1.5：查看API对向所属的namespace

dongyunqi@mongodaddy:~/k8s$ kubectl get service -A | egrep 'NAMESP|ngx-svc'| awk '{print $1}'
NAMESPACE
default

1.6：获取k8s集群节点信息

dongyunqi@mongodaddy:~/k8s$ kubectl get node -o wide
NAME         STATUS   ROLES                  AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
mongodaddy   Ready    control-plane,master   6h36m   v1.23.3   192.168.64.131           Ubuntu 22.04.1 LTS   5.15.0-57-generic   docker://20.10.12
mongomummy   Ready                     6h31m   v1.23.3   192.168.64.132           Ubuntu 22.04.1 LTS   5.15.0-57-generic   docker://20.10.12

1.7：获取所有命名空间

yunqi@mongodaddy:~/k8s$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   21h
kube-flannel      Active   20h
kube-node-lease   Active   21h
kube-public       Active   21h
kube-system       Active   21h
nginx-ingress     Active   13h

1.8：POD已经RUNNING但是不READY

出现这种现象的原因一般都是配置了健康检查机制，容器已经启动成功了，但是容器内的应用健康检查没有通过，即容器内的应用没有启动成功，如下：

dongyunqi@mongodaddy:~/k8s$ kubectl get pod -n nginx-ingress
NAME                           READY   STATUS    RESTARTS   AGE
ngx-kic-dep-67c7cf6d5f-dg5cw   0/1     Running   0          43s

此时可以查看应用日志来定位具体问题，如下实例：

dongyunqi@mongodaddy:~/k8s$ kubectl logs ngx-kic-dep-67c7cf6d5f-dg5cw -n nginx-ingress
I0114 03:23:11.052364       1 main.go:213] Starting NGINX Ingress Controller Version=2.2.2 GitCommit=a88b7fe6dbde5df79593ac161749afc1e9a009c6 Date=2022-05-24T00:33:34Z Arch=linux/amd64 PlusFlag=false
....
2023/01/14 03:23:11 [notice] 12#12: OS: Linux 5.15.0-57-generic
2023/01/14 03:23:11 [notice] 12#12: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/01/14 03:23:11 [notice] 12#12: start worker processes
2023/01/14 03:23:11 [notice] 12#12: start worker process 13
I0114 03:23:11.115502       1 leaderelection.go:248] attempting to acquire leader lease nginx-ingress/nginx-ingress-leader-election...
W0114 03:23:11.117925       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1alpha1.TransportServer: the server could not find the requested resource (get transportservers.k8s.nginx.org)
E0114 03:23:11.118154       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TransportServer: failed to list *v1alpha1.TransportServer: the server could not find the requested resource (get transportservers.k8s.nginx.org)
W0114 03:23:11.137598       1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.Policy: the server could not find the requested resource (get policies.k8s.nginx.org)

以上的错误实例是在部署ingress controller时因为一些ingress controller没有初始化完成导致出现错误。

1.9：查询k8s都有哪些API对象

dongyunqi@mongodaddy:~/k8s$ kubectl api-resources|egrep -w 'Ingress|KIND'
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
ingresses                         ing          networking.k8s.io/v1                   true         Ingress
dongyunqi@mongodaddy:~/k8s$ kubectl api-resources
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
bindings                                       v1                                     true         Binding
componentstatuses                 cs           v1                                     false        ComponentStatus
...

2：常用问题

2.1：POD一直ContainerCreating

通过kubectl describe pod {pod名称}查看错误原因,如下就是因为网络插件出现问题导致：

dongyunqi@mongodaddy:~/k8s$ kubectl describe pod busy-pod
Name:         busy-pod
...
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               4m5s                   default-scheduler  Successfully assigned default/busy-pod to mongomummy
  Warning  FailedCreatePodSandBox  4m5s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "81294866bfbdb13363f63bab5b2a87434994b7890ec03e601869cb586d1fad87" network for pod "busy-pod": networkPlugin cni failed to set up pod "busy-pod_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  4m3s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "533c9ceee9a99087be62bb1efdc624cf72f81397443cd908252abc4cea7b7c1b" network for pod "busy-pod": networkPlugin cni failed to set up pod "busy-pod_default" network: open /run/flannel/subnet.env: no such file or directory
 ...

只需要在k8s集群中创建网络插件就行了，具体可以参考这篇文章 。