Helm：问题对应：缺少PV或者PV不足导致pod一直pending的对应方法

使用Helm 安装MySQL，缺省方式下发现pod一直处于pending状态，确认之后发现未创建足够PV所导致的问题。

问题现象

master $ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
amber-ibex-mysql-5cd6bd76bb-hz8mh   0/1     Pending   0          74s
master $

环境说明

master $ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
master $
master $ helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
master $

步骤1: 安装mysql

master $ kubectl get pv
No resources found.
master $ kubectl get pvc
No resources found.
master $ helm install stable/mysql
NAME:   amber-ibex
LAST DEPLOYED: Wed Dec  4 08:07:55 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME              TYPE    DATA  AGE
amber-ibex-mysql  Opaque  2     0s

==> v1/ConfigMap
NAME                   DATA  AGE
amber-ibex-mysql-test  1     0s

==> v1/PersistentVolumeClaim
NAME              STATUS   VOLUME  CAPACITY  ACCESS MODES  STORAGECLASS  AGE
amber-ibex-mysql  Pending  0s

==> v1/Service
NAME              TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
amber-ibex-mysql  ClusterIP  10.109.73.123         3306/TCP  0s

==> v1/Deployment
NAME              DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
amber-ibex-mysql  1        1        1           0          0s

==> v1/Pod(related)
NAME                               READY  STATUS   RESTARTS  AGE
amber-ibex-mysql-5cd6bd76bb-hz8mh  0/1    Pending  0         0s


NOTES:
MySQL can be accessed via port 3306 on the following DNS name from within your cluster:
amber-ibex-mysql.default.svc.cluster.local

To get your root password run:

    MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default amber-ibex-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)

To connect to your database:

1. Run an Ubuntu pod that you can use as a client:

    kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il

2. Install the mysql client:

    $ apt-get update && apt-get install mysql-client -y

3. Connect using the mysql cli, then provide your password:
    $ mysql -h amber-ibex-mysql -p

To connect to your database directly from outside the K8s cluster:
    MYSQL_HOST=127.0.0.1
    MYSQL_PORT=3306

    # Execute the following command to route the connection:
    kubectl port-forward svc/amber-ibex-mysql 3306

    mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}


master $

步骤2: 确认结果

master $ helm ls
NAME            REVISION        UPDATED                         STATUS          CHART       NAMESPACE
amber-ibex      1               Wed Dec  4 08:07:55 2019        DEPLOYED        mysql-1.4.0 default
master $
master $ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
amber-ibex-mysql-5cd6bd76bb-hz8mh   0/1     Pending   0          74s
master $

继续确认pod的详细信息

master $ kubectl describe pods amber-ibex-mysql-5cd6bd76bb-hz8mh
Name:               amber-ibex-mysql-5cd6bd76bb-hz8mh
Namespace:          default
Priority:           0
PriorityClassName:  
Node:               
Labels:             app=amber-ibex-mysql
                    pod-template-hash=5cd6bd76bb
                    release=amber-ibex
Annotations:        
Status:             Pending
IP:
Controlled By:      ReplicaSet/amber-ibex-mysql-5cd6bd76bb
Init Containers:
  remove-lost-found:
    Image:      busybox:1.29.3
    Port:       
    Host Port:  
    Command:
      rm
      -fr
      /var/lib/mysql/lost+found
    Requests:
      cpu:        10m
      memory:     10Mi
    Environment:  
    Mounts:
      /var/lib/mysql from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dkmsx (ro)
Containers:
  amber-ibex-mysql:
    Image:      mysql:5.7.14
    Port:       3306/TCP
    Host Port:  0/TCP
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   exec [sh -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=30s timeout=5s period=10s #success=1 #failure=3
    Readiness:  exec [sh -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MYSQL_ROOT_PASSWORD:    Optional: false
      MYSQL_PASSWORD:           Optional: true
      MYSQL_USER:
      MYSQL_DATABASE:
    Mounts:
      /var/lib/mysql from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dkmsx (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  amber-ibex-mysql
    ReadOnly:   false
  default-token-dkmsx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dkmsx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  38s (x3 over 107s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims
master $

确认pvc，发现原因

master $ kubectl get pvc
NAME               STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
amber-ibex-mysql   Pending                                                     3m23s
master $

进一步确认describe信息，发现没有pv

master $ kubectl describe pvc amber-ibex-mysql
Name:          amber-ibex-mysql
Namespace:     default
StorageClass:
Status:        Pending
Volume:
Labels:        app=amber-ibex-mysql
               chart=mysql-1.4.0
               heritage=Tiller
               release=amber-ibex
Annotations:   
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type       Reason         Age                  From                         Message
  ----       ------         ----                 ----                         -------
  Normal     FailedBinding  6s (x18 over 3m59s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
Mounted By:  amber-ibex-mysql-5cd6bd76bb-hz8mh
master $

使用helm inspect命令确认pvc相关的需求

master $ helm inspect stable/mysql
apiVersion: v1
appVersion: 5.7.27
description: Fast, reliable, scalable, and easy to use open-source relational database
  system.
engine: gotpl
home: https://www.mysql.com/
icon: https://www.mysql.com/common/logos/logo-mysql-170x115.png
keywords:
- mysql
- database
- sql
...省略
## Persist data to a persistent volume
persistence:
  enabled: true
  ## database data Persistent Volume Storage Class
  ## If defined, storageClassName: 
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # storageClass: "-"
  accessMode: ReadWriteOnce
  size: 8Gi
  annotations: {}
...省略
master $

可以看到需要一个大小为8G的pv，使用如下yaml文件创建所需的pv

master $ cat mysql-pv-volume.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-volume-mysql
  labels:
    type: local
spec:
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql/data"
master $ kubectl get pvc
NAME               STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
amber-ibex-mysql   Pending                                                     22m
master $ kubectl get pv
No resources found.
master $ kubectl create -f mysql-pv-volume.yaml
persistentvolume/pv-volume-mysql created
master $

确认pv和pvc可以看到pv状态从Available到Bound的变化过程

master $ kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv-volume-mysql   8Gi        RWO            Retain           Available            4s
master $ kubectl get pvc
NAME               STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
amber-ibex-mysql   Bound    pv-volume-mysql   8Gi        RWO                           23m
master $ kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM    STORAGECLASS   REASON   AGE
pv-volume-mysql   8Gi        RWO            Retain           Bound    default/amber-ibex-mysql                           16s
master $

再次确认pod也已经变成Running的状态了

master $ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
amber-ibex-mysql-5cd6bd76bb-hz8mh   0/1     Running   0          23m
master $

STATUS仍然不正确，是因为实际仍然在启动中，稍等一会再次确认就发现已经正常了

master $ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
amber-ibex-mysql-5cd6bd76bb-hz8mh   1/1     Running   0          26m
master $