ROOK-02 优化集群参数部署ROOK

一、环境信息

主机名	IP	k8s角色	rook角色	磁盘20G
rook01	192.168.86.36	master,slave	mon，mgr，osd，csi	/dev/sdb-sdd
rook02	192.168.86.37	master,slave	mon，mgr，osd，csi	/dev/sdb-sdd
rook03	192.168.86.38	master,slave	mon，mgr，osd，csi	/dev/sdb-sdd
rook04	192.168.86.39	slave	osd，csi	/dev/sdb-sdd

二、部署和卸载

1、获取安装源码（使用写博客时最新的版本）

git clone --single-branch --branch v1.7.2 https://github.com/rook/rook.git

2、调整mon调度策略参数，⽣产环境有⼀些专⻔的节点⽤于mon、mgr，存储节点节点使⽤单独的节点承担,利⽤调度机制实现
rook/cluster/examples/kubernetes/ceph/cluster.yaml

#定义mon节点分布和mgr节点分布
  placement:
    mon:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: ceph-mon
              operator: In
              values:
              - enable
    mgr:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: ceph-mgr
              operator: In
              values:
              - enable

设置磁盘的参数，调整为false，⽅便后⾯定制

  storage: # cluster level storage configuration and selection
    useAllNodes: false
    useAllDevices: false
# nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    nodes:
    - name: "rook01"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
    - name: "rook02"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
    - name: "rook03"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
    - name: "rook04"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdb"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdc"
        config:
          storeType: bluestore
          journalSizeMB: "4096"
      - name: "sdd"
        config:
          storeType: bluestore
          journalSizeMB: "4096"

设置资源限制，默认组件没有设置资源分配，当出现资源争抢的时候可能会出现驱逐，为了保证Ceph的核⼼组件能分配到特定的资源，需要设置合理的资源分配；
mon,内存推荐128G
mds
osd,每T磁盘建议需要有4G以上内存

  resources:
    mon:
      limits:
        cpu: "500m"
        memory: "1Gi"
      requests:
        cpu: "500m"
        memory: "1Gi"
    mgr:
      limits:
        cpu: "500m"
        memory: "1Gi"
      requests:
        cpu: "500m"
        memory: "1Gi"
    osd:
      limits:
        cpu: "1000m"
        memory: "2Gi"
      requests:
        cpu: "1000m"
        memory: "2Gi"

3、给节点打label并部署rook

kubectl label node rook01 ceph-mon=enable
kubectl label node rook01 ceph-mgr=enable
kubectl label node rook02 ceph-mon=enable
kubectl label node rook02 ceph-mgr=enable
kubectl label node rook03 ceph-mon=enable
kubectl label node rook03 ceph-mgr=enable

cd rook/cluster/examples/kubernetes/ceph/
kubectl apply -f common.yaml
kubectl apply -f crds.yaml
kubectl apply -f operator.yaml
kubectl apply -f cluster.yaml
#部署客户端，提供ceph集群命令操作入口
kubectl apply -f toolbox.yaml

扩容osd或者主机直接在cluster.yaml中添加新节点信息即可，加入节点后osd无法被创建需要修改operator.yaml中的rook-discover扫描时间，默认1小时

            # The duration between discovering devices in the rook-discover daemonset.
            - name: ROOK_DISCOVER_DEVICES_INTERVAL
              value: "60s"