【kubernetes/k8s概念】rook

     https://github.com/rook/rook

     云原生计算基金会(CNCF)的孵化级项目,是Kubernetes的开源云本地存储协调器,为各种存储解决方案提供平台,框架和支持,以便与云原生环境本地集成

    Ceph is a highly scalable distributed storage solution for block storageobject storage, and shared file systems with years of production deployments.

    实现了一个自动管理的、自动扩容的、自动修复的分布式存储服务。Rook 支持 Ceph 存储,基于 Kubernetes 使用 Rook 可以大大简化 Ceph 存储集群的搭建以及使用

 

Desigin

【kubernetes/k8s概念】rook_第1张图片

                        引用:https://github.com/rook/rook/blob/master/Documentation/ceph-storage.md

operator:作用启动监控存储集群,包括mon,osd,mgr等服务,可以支持 block 块存储,file system 文件存储,object 对象存储 (S3/Swift)。 启动包括ceph agent 与 discover服务,agent 包含 Flexvolume plugin,包括attaching network storage devices, mounting volumes, and formating the filesystem 三种操作。

 

Rook 架构

 

【kubernetes/k8s概念】rook_第2张图片

 

Block 块存储

       Ceph 块设备也可以简称为 RBD 或 RADOS 块设备

       Rook 创建 Ceph 块设备,先创建 CephBlockPool 和 StorageClass,CephBlockPool 是 Rook 的 CRD (创建一个 Block 池),在创建一个 StorageClass,这个是动态卷配置 (Dynamic provisioning) 

File System 文件系统

      A shared file system can be mounted with read/write permission from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.

      ROOK_ALLOW_MULTIPLE_FILESYSTEM定义允许多个文件系统共享

Object 对象存储     

     Ceph RGW 是基于 librados,为应用提供 RESTful 类型的对象存储接口,其接口方式支持 S3(兼容 Amazon S3 RESTful API) 和 Swift(兼容 OpenStack Swift API) 两种类型

 

组件Mon

       Mon作用是监控、管理和协调整个分布式系统环境中其它各个OSD/PG、Client、MDS角色的工作,保证整个分布环境中的数据一致性

       MON采用主备模式(leader/follower),即使系统中有多个MON角色,实际工作的也只有一个MON,其它MON都处于standby状态

       当Ceph失去了Leader MON后,其它MON会基于PaxOS算法,投票选出新的Leader

组件osd

    存储落地操作的节点叫做OSD(Object Storage Device),基于它工作的RADOS、PG等模块

 

1. rook快速部署:

    路径rook/cluster/example/kubernetes/ceph

1. kubectl apply -f common.yaml

2. kubectl apply -f operator.yaml

3. kubectl aplply -f cluster.yaml

  • dataDirHostPath: 在宿主机上生成的,保存的是ceph的相关的配置文件,再重新生成集群目录为空,否则mon会无法启动
  • useAllDevices: 使用所有的设备,建议为false,否则会把宿主机所有可用的磁盘都干掉
  • useAllNodes:使用所有的node节点,建议为false,肯定不会用k8s集群内的所有node来搭建ceph的
  • databaseSizeMB和journalSizeMB:当磁盘大于100G的时候,就注释这俩项就行了

 

2. rook csi部署

  2.1 需求

  1. a Kubernetes v1.13+ is needed in order to support CSI Spec 1.0.
  2. --allow-privileged flag set to true in kubelet and your API server

    需要的镜像:

       quay.io/k8scsi/csi-provisioner:v1.0.1

       quay.io/cephcsi/rbdplugin:v1.0.0

       quay.io/k8scsi/csi-attacher:v1.0.1

       quay.io/k8scsi/csi-snapshotter:v1.0.1     

       quay.io/k8scsi/csi-node-driver-registrar:v1.0.2

       quay.io/cephcsi/cephfsplugin:v1.0.0

  2.2 为CSI驱动create RBRC

     绑定的ClusterRole的rule未设置,是不是被骗了,搞这么一大堆

      kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/rbd/csi-nodeplugin-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-csi-rbd-plugin-sa
  namespace: rook-ceph

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-nodeplugin
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.ceph.rook.io/aggregate-to-rbd-csi-nodeplugin: "true"
rules: []
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-nodeplugin-rules
  labels:
    rbac.ceph.rook.io/aggregate-to-rbd-csi-nodeplugin: "true"
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-nodeplugin
subjects:
  - kind: ServiceAccount
    name: rook-csi-rbd-plugin-sa
    namespace: rook-ceph
roleRef:
  kind: ClusterRole
  name: rbd-csi-nodeplugin
  apiGroup: rbac.authorization.k8s.io

      kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/rbd/csi-provisioner-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-csi-rbd-provisioner-sa
  namespace: rook-ceph

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-external-provisioner-runner
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.ceph.rook.io/aggregate-to-rbd-external-provisioner-runner: "true"
rules: []
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-external-provisioner-runner-rules
  labels:
    rbac.ceph.rook.io/aggregate-to-rbd-external-provisioner-runner: "true"
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete", "update"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "create", "update"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "create", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-provisioner-role
subjects:
  - kind: ServiceAccount
    name: rook-csi-rbd-provisioner-sa
    namespace: rook-ceph
roleRef:
  kind: ClusterRole
  name: rbd-external-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

    同理创建cephfs的rbac,就不展示了

    kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/cephfs

    2.3 创建operator-with-csi.yaml

     kubectl apply -f operaor-with-csi.yaml

#################################################################################################################
# The deployment for the rook operator that enables the ceph-csi driver for beta testing.
# For example, to create the rook-ceph cluster:
#   kubectl create -f common.yaml
#   kubectl create -f operator-with-csi.yaml
#   kubectl create -f cluster.yaml
#################################################################################################################

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-operator
  namespace: rook-ceph
  labels:
    operator: rook
    storage-backend: ceph
spec:
  selector:
    matchLabels:
      app: rook-ceph-operator
  replicas: 1
  template:
    metadata:
      labels:
        app: rook-ceph-operator
    spec:
      serviceAccountName: rook-ceph-system
      containers:
      - name: rook-ceph-operator
        image: rook/ceph:master
        args: ["ceph", "operator"]
        volumeMounts:
        - mountPath: /var/lib/rook
          name: rook-config
        - mountPath: /etc/ceph
          name: default-config-dir
        env:
        - name: ROOK_CURRENT_NAMESPACE_ONLY
          value: "true"
        # CSI enablement
        - name: ROOK_CSI_ENABLE_CEPHFS
          value: "true"
        - name: ROOK_CSI_CEPHFS_IMAGE
          value: "quay.io/cephcsi/cephfsplugin:v1.0.0"
        - name: ROOK_CSI_ENABLE_RBD
          value: "true"
        - name: ROOK_CSI_RBD_IMAGE
          value: "quay.io/cephcsi/rbdplugin:v1.0.0"
        - name: ROOK_CSI_REGISTRAR_IMAGE
          value: "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
        - name: ROOK_CSI_PROVISIONER_IMAGE
          value: "quay.io/k8scsi/csi-provisioner:v1.0.1"
        - name: ROOK_CSI_SNAPSHOTTER_IMAGE
          value: "quay.io/k8scsi/csi-snapshotter:v1.0.1"
        - name: ROOK_CSI_ATTACHER_IMAGE
          value: "quay.io/k8scsi/csi-attacher:v1.0.1"
        # The name of the node to pass with the downward API
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        # The pod name to pass with the downward API
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        # The pod namespace to pass with the downward API
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      volumes:
      - name: rook-config
        emptyDir: {}
      - name: default-config-dir
        emptyDir: {}

 

【kubernetes/k8s概念】rook_第3张图片

 

【kubernetes/k8s概念】rook_第4张图片

         核心是RADOS,也即高可用、自动化的分布式对象存储,该模块负责对OSD的自动化运行,保证整个存储系统的可用性。同时,该模块通过一个名为librados的公共库对外提供存储服务,块存储和对象存储都依赖该动态库。

 

参考资料

  • https://github.com/rook/rook/blob/master/Documentation/ceph-block.md
  • https://github.com/rook/rook/blob/master/Documentation/ceph-filesystem.md
  • Rook Ceph Object Storage
  • https://github.com/rook/rook/blob/master/Documentation/ceph-storage.md

https://github.com/rook/rook

https://rook.io/

https://edgefs.io/

你可能感兴趣的:(kubernetes,CSI,存储)