Kubernetes -- NFS作为动态存储创建pvc/pv

NFS由于自身的问题，不常用于生产环境，这里仅作为demo展示动态存储的使用。生产环境可以使用ceph，rook-ceph来管理ceph存储。

假设已部署好NFS Server，这里演示如何在集群中部署动态存储与创建storageclass/pvc/pv。

1.创建独立的namespace

# kubectl create ns storage

2.创建rbac给serviceAccount赋权

创建一个serviceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: storage

为serviceAccount赋权：

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: storage
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

3.部署Provisioner

privisioner可以理解为底层存储的驱动，由privisioner管理底层存储。
privisioner以deploy方式部署了1个pod，pod内container指定了nfs的环境信息(包括name/ip/path等)，serviceAccountName=上一步创建的serviceAccount名称；

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nfs-provisioner
  name: nfs-provisioner
  namespace: storage
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-provisioner
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-volume
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 178.104.163.63
            - name: NFS_PATH
              value: /var/nfs
      volumes:
        - name: nfs-volume
          nfs:
            server: 178.104.163.63
            path: /var/nfs

查看部署的pod:

# kubectl get pod -n storage
NAME                               READY   STATUS    RESTARTS   AGE
nfs-provisioner-778c655cbd-4twcz   1/1     Running   0          19s

provisioner部署完毕后，观察pod的log；若有报错，需及时排查：

# kubectl logs nfs-provisioner-778c655cbd-4twcz -n storage
I0330 08:08:27.981652       1 leaderelection.go:185] attempting to acquire leader lease  storage/fuseim.pri-ifs...
I0330 08:08:27.992969       1 leaderelection.go:194] successfully acquired lease storage/fuseim.pri-ifs
I0330 08:08:27.993028       1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776!
I0330 08:08:27.993504       1 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"storage", Name:"fuseim.pri-ifs", UID:"35414cb4-5e1c-45e4-9d0e-9b031e0c3df2", APIVersion:"v1", ResourceVersion:"2323405", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776 became leader
I0330 08:08:28.093343       1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776!

4.创建storageclass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-sc
  namespace: storage
provisioner: fuseim.pri/ifs    # 这里的provisioner==上面env.PROVISIONER_NAME

查看storageclass:

# kubectl get sc -n storage
NAME     PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-sc   fuseim.pri/ifs   Delete          Immediate           false                  19s

5.storageclass创建pvc给pod使用

创建pvc：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
  namespace: storage
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-sc    //使用storageClass

查看自动创建的pv:

# kubectl get sc,pv,pvc -n storage
NAME                                 PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/nfs-sc   fuseim.pri/ifs   Delete          Immediate           false                  13m

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
persistentvolume/pvc-2fa56db1-fe02-4722-a9e9-d0dfad565934   1Gi        RWX            Delete           Bound    storage/myclaim   nfs-sc                  3m12s

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/myclaim   Bound    pvc-2fa56db1-fe02-4722-a9e9-d0dfad565934   1Gi        RWX            nfs-sc         3m12s

将pvc提供给pod使用：

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
  namespace: storage
spec:
  containers:
  - name: test-pod
    image: nginx:1.15.2
    volumeMounts:
      - name: nfs-pvc
        mountPath: /mnt/nginx
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: myclaim        ## claimName==上面创建的pvc的名称

pod创建完毕，到容器中查看挂载的目录：

# kubectl exec -it test-pod -n storage -- bash
# df -h
Filesystem                                                                        Size  Used Avail Use% Mounted on
overlay                                                                           100G  6.2G   94G   7% /
tmpfs                                                                              64M     0   64M   0% /dev
tmpfs                                                                             3.0G     0  3.0G   0% /sys/fs/cgroup
/dev/vda1                                                                         100G  6.2G   94G   7% /etc/hosts
178.104.163.63:/var/nfs/storage-myclaim-pvc-2fa56db1-fe02-4722-a9e9-d0dfad565934  100G  1.4G   99G   2% /mnt/nginx

6.storageclass为statefulset创建pvc/pv

statefulset.spec指定storageClass，部署时会自动为其创建pvc、pv：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: storage
spec:
  serviceName: "nginx"        # 等于headless service的名称
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.15.2
        imagePullPolicy: IfNotPresent
        name: nginx
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /mnt/nginx
          name: nfs-sc-volume
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
  volumeClaimTemplates:                ## 指定连接到sc去申请pvc
  - metadata:
      name: nfs-sc-volume
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        limits:
          storage: 2Gi
        requests:
          storage: 1Gi
      storageClassName: nfs-sc
      volumeMode: Filesystem

这里的statefulset创建了2个pod，为每个pod都绑定了一个pvc，2个pod使用独立的存储卷:

# kubectl get sc,pvc,pv -n storage
NAME                                 PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/nfs-sc   fuseim.pri/ifs   Delete          Immediate           false                  4m58s

NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/nfs-sc-volume-nginx-0   Bound    pvc-63389a42-a00d-4b34-bd51-4542cebb42aa   1Gi        RWO            nfs-sc         75s
persistentvolumeclaim/nfs-sc-volume-nginx-1   Bound    pvc-e01d4f5c-004b-4618-94d6-9556612cd198   1Gi        RWO            nfs-sc         70s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
persistentvolume/pvc-63389a42-a00d-4b34-bd51-4542cebb42aa   1Gi        RWO            Delete           Bound    storage/nfs-sc-volume-nginx-0   nfs-sc                  75s
persistentvolume/pvc-e01d4f5c-004b-4618-94d6-9556612cd198   1Gi        RWO            Delete           Bound    storage/nfs-sc-volume-nginx-1   nfs-sc                  70s