ceph-crush map区分ssd和hdd资源池

环境

  • 3块SSD

  • 6块HDD

    # ceph -s
      cluster:
        id:     b4c125fd-60ab-41ce-b51c-88833089a3ad
        health: HEALTH_OK
     
      services:
        mon: 3 daemons, quorum node1,node2,node3 (age 47m)
        mgr: node1(active, since 56m), standbys: node2, node3
        osd: 9 osds: 9 up (since 44m), 9 in (since 44m)
     
      data:
        pools:   1 pools, 1 pgs
        objects: 0 objects, 0 B
        usage:   9.1 GiB used, 30 TiB / 30 TiB avail
        pgs:     1 active+clean
       
    # ceph osd tree
    ID  CLASS  WEIGHT    TYPE NAME       STATUS  REWEIGHT  PRI-AFF
    -1         29.83720  root default                             
    -3          8.73286      host node1                           
     0    hdd   3.63869          osd.0       up   1.00000  1.00000
     1    hdd   3.63869          osd.1       up   1.00000  1.00000
     6    ssd   1.45549          osd.6       up   1.00000  1.00000
    -5         12.37148      host node2                           
     2    hdd   5.45799          osd.2       up   1.00000  1.00000
     3    hdd   5.45799          osd.3       up   1.00000  1.00000
     7    ssd   1.45549          osd.7       up   1.00000  1.00000
    -7          8.73286      host node3                           
     4    hdd   3.63869          osd.4       up   1.00000  1.00000
     5    hdd   3.63869          osd.5       up   1.00000  1.00000
     8    ssd   1.45549          osd.8       up   1.00000  1.00000
    

需求

根据crush map将ssd和hdd分为两个资源池,iops高的放在ssd的pool,不高的放在hdd的pool

部署

  1. 查看crush map分布:
# ceph osd crush tree
ID  CLASS  WEIGHT    TYPE NAME     
-1         29.83720  root default  
-3          8.73286      host node1
 0    hdd   3.63869          osd.0 
 1    hdd   3.63869          osd.1 
 6    ssd   1.45549          osd.6 
-5         12.37148      host node2
 2    hdd   5.45799          osd.2 
 3    hdd   5.45799          osd.3 
 7    ssd   1.45549          osd.7 
-7          8.73286      host node3
 4    hdd   3.63869          osd.4 
 5    hdd   3.63869          osd.5 
 8    ssd   1.45549          osd.8 
  1. 查看详细crush规则

    # ceph osd crush dump
    {
        "devices": [
            {
                "id": 0,
                "name": "osd.0",
                "class": "hdd"
            },
            {
                "id": 1,
                "name": "osd.1",
                "class": "hdd"
            },
            {
                "id": 2,
                "name": "osd.2",
                "class": "hdd"
            },
            {
                "id": 3,
                "name": "osd.3",
                "class": "hdd"
            },
            {
                "id": 4,
                "name": "osd.4",
                "class": "hdd"
            },
            {
                "id": 5,
                "name": "osd.5",
                "class": "hdd"
            },
            {
                "id": 6,
                "name": "osd.6",
                "class": "ssd"
            },
            {
                "id": 7,
                "name": "osd.7",
                "class": "ssd"
            },
            {
                "id": 8,
                "name": "osd.8",
                "class": "ssd"
            }
        ],
        "types": [
            {
                "type_id": 0,
                "name": "osd"
            },
            {
                "type_id": 1,
                "name": "host"
            },
            {
                "type_id": 2,
                "name": "chassis"
            },
            {
                "type_id": 3,
                "name": "rack"
            },
            {
                "type_id": 4,
                "name": "row"
            },
            {
                "type_id": 5,
                "name": "pdu"
            },
            {
                "type_id": 6,
                "name": "pod"
            },
            {
                "type_id": 7,
                "name": "room"
            },
            {
                "type_id": 8,
                "name": "datacenter"
            },
            {
                "type_id": 9,
                "name": "zone"
            },
            {
                "type_id": 10,
                "name": "region"
            },
            {
                "type_id": 11,
                "name": "root"
            }
        ],
        "buckets": [
            {
                "id": -1,
                "name": "default",
                "type_id": 11,
                "type_name": "root",
                "weight": 1955411,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": -3,
                        "weight": 572317,
                        "pos": 0
                    },
                    {
                        "id": -5,
                        "weight": 810777,
                        "pos": 1
                    },
                    {
                        "id": -7,
                        "weight": 572317,
                        "pos": 2
                    }
                ]
            },
            {
                "id": -2,
                "name": "default~hdd",
                "type_id": 11,
                "type_name": "root",
                "weight": 1669250,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": -4,
                        "weight": 476930,
                        "pos": 0
                    },
                    {
                        "id": -6,
                        "weight": 715390,
                        "pos": 1
                    },
                    {
                        "id": -8,
                        "weight": 476930,
                        "pos": 2
                    }
                ]
            },
            {
                "id": -3,
                "name": "node1",
                "type_id": 1,
                "type_name": "host",
                "weight": 572317,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 0,
                        "weight": 238465,
                        "pos": 0
                    },
                    {
                        "id": 1,
                        "weight": 238465,
                        "pos": 1
                    },
                    {
                        "id": 6,
                        "weight": 95387,
                        "pos": 2
                    }
                ]
            },
            {
                "id": -4,
                "name": "node1~hdd",
                "type_id": 1,
                "type_name": "host",
                "weight": 476930,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 0,
                        "weight": 238465,
                        "pos": 0
                    },
                    {
                        "id": 1,
                        "weight": 238465,
                        "pos": 1
                    }
                ]
            },
            {
                "id": -5,
                "name": "node2",
                "type_id": 1,
                "type_name": "host",
                "weight": 810777,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 2,
                        "weight": 357695,
                        "pos": 0
                    },
                    {
                        "id": 3,
                        "weight": 357695,
                        "pos": 1
                    },
                    {
                        "id": 7,
                        "weight": 95387,
                        "pos": 2
                    }
                ]
            },
            {
                "id": -6,
                "name": "node2~hdd",
                "type_id": 1,
                "type_name": "host",
                "weight": 715390,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 2,
                        "weight": 357695,
                        "pos": 0
                    },
                    {
                        "id": 3,
                        "weight": 357695,
                        "pos": 1
                    }
                ]
            },
            {
                "id": -7,
                "name": "node3",
                "type_id": 1,
                "type_name": "host",
                "weight": 572317,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 4,
                        "weight": 238465,
                        "pos": 0
                    },
                    {
                        "id": 5,
                        "weight": 238465,
                        "pos": 1
                    },
                    {
                        "id": 8,
                        "weight": 95387,
                        "pos": 2
                    }
                ]
            },
            {
                "id": -8,
                "name": "node3~hdd",
                "type_id": 1,
                "type_name": "host",
                "weight": 476930,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 4,
                        "weight": 238465,
                        "pos": 0
                    },
                    {
                        "id": 5,
                        "weight": 238465,
                        "pos": 1
                    }
                ]
            },
            {
                "id": -9,
                "name": "node1~ssd",
                "type_id": 1,
                "type_name": "host",
                "weight": 95387,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 6,
                        "weight": 95387,
                        "pos": 0
                    }
                ]
            },
            {
                "id": -10,
                "name": "node2~ssd",
                "type_id": 1,
                "type_name": "host",
                "weight": 95387,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 7,
                        "weight": 95387,
                        "pos": 0
                    }
                ]
            },
            {
                "id": -11,
                "name": "node3~ssd",
                "type_id": 1,
                "type_name": "host",
                "weight": 95387,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": 8,
                        "weight": 95387,
                        "pos": 0
                    }
                ]
            },
            {
                "id": -12,
                "name": "default~ssd",
                "type_id": 11,
                "type_name": "root",
                "weight": 286161,
                "alg": "straw2",
                "hash": "rjenkins1",
                "items": [
                    {
                        "id": -9,
                        "weight": 95387,
                        "pos": 0
                    },
                    {
                        "id": -10,
                        "weight": 95387,
                        "pos": 1
                    },
                    {
                        "id": -11,
                        "weight": 95387,
                        "pos": 2
                    }
                ]
            }
        ],
        "rules": [
            {
                "rule_id": 0,
                "rule_name": "replicated_rule",
                "ruleset": 0,
                "type": 1,
                "min_size": 1,
                "max_size": 10,
                "steps": [
                    {
                        "op": "take",
                        "item": -1,
                        "item_name": "default"
                    },
                    {
                        "op": "chooseleaf_firstn",
                        "num": 0,
                        "type": "host"
                    },
                    {
                        "op": "emit"
                    }
                ]
            }
        ],
        "tunables": {
            "choose_local_tries": 0,
            "choose_local_fallback_tries": 0,
            "choose_total_tries": 50,
            "chooseleaf_descend_once": 1,
            "chooseleaf_vary_r": 1,
            "chooseleaf_stable": 1,
            "straw_calc_version": 1,
            "allowed_bucket_algs": 54,
            "profile": "jewel",
            "optimal_tunables": 1,
            "legacy_tunables": 0,
            "minimum_required_version": "jewel",
            "require_feature_tunables": 1,
            "require_feature_tunables2": 1,
            "has_v2_rules": 0,
            "require_feature_tunables3": 1,
            "has_v3_rules": 0,
            "has_v4_buckets": 1,
            "require_feature_tunables5": 1,
            "has_v5_rules": 0
        },
        "choose_args": {}
    }
    
  2. 查看当前pool写入osd的规则

    # ceph osd crush rule ls
    replicated_rule
    

手动编辑crush规则

  1. 导出crushmap规则

    # ceph osd getcrushmap -o crushmap20200922.bin
    19
    
  2. 编译导出的crushmap文件

    # crushtool -d crushmap20200922.bin -o crushmap20200922.txt
    [root@node1 ceph]# ll
    total 8
    -rw-r--r-- 1 root root 1326 Sep 22 23:06 crushmap20200922.bin
    -rw-r--r-- 1 root root 1935 Sep 22 23:07 crushmap20200922.txt
    
  3. 修改规则

    1. 添加三个buckets来定义ssd host 并且将ssd host从原来的hdd host中删除 并且将以前host顺便改名

      # 原来的host
      ...
      
      # buckets  定义osd怎么存放的
      host node1 {
      	id -3		# do not change unnecessarily
      	id -4 class hdd		# do not change unnecessarily
      	id -9 class ssd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.0 weight 3.639
      	item osd.1 weight 3.639
      	item osd.6 weight 1.455
      }
      host node2 {
      	id -5		# do not change unnecessarily
      	id -6 class hdd		# do not change unnecessarily
      	id -10 class ssd		# do not change unnecessarily
      	# weight 12.371
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.2 weight 5.458
      	item osd.3 weight 5.458
      	item osd.7 weight 1.455
      }
      host node3 {
      	id -7		# do not change unnecessarily
      	id -8 class hdd		# do not change unnecessarily
      	id -11 class ssd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.4 weight 3.639
      	item osd.5 weight 3.639
      	item osd.8 weight 1.455
      }
      ...
      
      # 修改后的host
      ...
      # buckets  定义osd怎么存放的
      host node1-hdd {
      	id -3		# do not change unnecessarily
      	id -4 class hdd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.0 weight 3.639
      	item osd.1 weight 3.639
      }
      host node2-hdd {
      	id -5		# do not change unnecessarily
      	id -6 class hdd		# do not change unnecessarily
      	# weight 12.371
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.2 weight 5.458
      	item osd.3 weight 5.458
      }
      host node3-hdd {
      	id -7		# do not change unnecessarily
      	id -8 class hdd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.4 weight 3.639
      	item osd.5 weight 3.639
      }
      host node1-ssd {
      	id -9 class ssd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.6 weight 1.455
      }
      host node2-ssd {
      	id -10 class ssd		# do not change unnecessarily
      	# weight 12.371
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.7 weight 1.455
      }
      host node3-ssd {
      	id -11 class ssd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.8 weight 1.455
      }
      ...
      
    2. 定义一个上层规则去调用新加的三个host,并且将以前的root default改为root hdd 并且调节权重,权重就是所有osd容量大小相加,比如单盘1.8TB,系统容量为1.635,权重为1.63539

      #之前的default调用
      ...
      root default {
      	id -1		# do not change unnecessarily
      	id -2 class hdd		# do not change unnecessarily
      	id -12 class ssd		# do not change unnecessarily
      	# weight 29.837
      	alg straw2
      	hash 0	# rjenkins1
      	item node1 weight 8.733
      	item node2 weight 12.371
      	item node3 weight 8.733
      }
      ...
      
      #修改后的调用
      ...
      root hdd {
      	id -1		# do not change unnecessarily
      	id -2 class hdd		# do not change unnecessarily
      	# weight 29.837
      	alg straw2
      	hash 0	# rjenkins1
      	item node1-hdd weight 7.278
      	item node2-hdd weight 10.916
      	item node3-hdd weight 7.278
      }
      root ssd {
      	# weight 29.837
      	alg straw2
      	hash 0	# rjenkins1
      	item node1-ssd weight 1.456
      	item node2-ssd weight 1.456
      	item node3-ssd weight 1.456
      }
      
      ...
      
    3. 最后定义一个rule规则关联到上层规则,并且将以前的replicated_rule改为hdd_rule

      #之前的rule
      ...
      rule replicated_rule { 
      	id 0
      	type replicated
      	min_size 1
      	max_size 10
      	step take default
      	step chooseleaf firstn 0 type host
      	step emit
      }
      ...
      #修改后rule
      ...
      rule hdd_rule { 
      	id 0
      	type replicated
      	min_size 1
      	max_size 10
      	step take hdd
      	step chooseleaf firstn 0 type host
      	step emit
      }
      rule ssd_rule { 
      	id 1
      	type replicated
      	min_size 1
      	max_size 10
      	step take ssd
      	step chooseleaf firstn 0 type host
      	step emit
      }
      ...
      
    4. 修改完成后的规则

      # begin crush map
      tunable choose_local_tries 0
      tunable choose_local_fallback_tries 0
      tunable choose_total_tries 50
      tunable chooseleaf_descend_once 1
      tunable chooseleaf_vary_r 1
      tunable chooseleaf_stable 1
      tunable straw_calc_version 1
      tunable allowed_bucket_algs 54
      
      # devices 定义osd信息
      device 0 osd.0 class hdd
      device 1 osd.1 class hdd
      device 2 osd.2 class hdd
      device 3 osd.3 class hdd
      device 4 osd.4 class hdd
      device 5 osd.5 class hdd
      device 6 osd.6 class ssd
      device 7 osd.7 class ssd
      device 8 osd.8 class ssd
      
      # types  定义数组组织类型  osd 主机 机柜 机房等等
      type 0 osd
      type 1 host
      type 2 chassis
      type 3 rack
      type 4 row
      type 5 pdu
      type 6 pod
      type 7 room
      type 8 datacenter
      type 9 zone
      type 10 region
      type 11 root
      
      # buckets  定义osd怎么存放的
      host node1-hdd {
      	id -3		# do not change unnecessarily
      	id -4 class hdd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.0 weight 3.639
      	item osd.1 weight 3.639
      }
      host node2-hdd {
      	id -5		# do not change unnecessarily
      	id -6 class hdd		# do not change unnecessarily
      	# weight 12.371
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.2 weight 5.458
      	item osd.3 weight 5.458
      }
      host node3-hdd {
      	id -7		# do not change unnecessarily
      	id -8 class hdd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.4 weight 3.639
      	item osd.5 weight 3.639
      }
      host node1-ssd {
      	id -9 class ssd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.6 weight 1.455
      }
      host node2-ssd {
      	id -10 class ssd		# do not change unnecessarily
      	# weight 12.371
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.7 weight 1.455
      }
      host node3-ssd {
      	id -11 class ssd		# do not change unnecessarily
      	# weight 8.733
      	alg straw2
      	hash 0	# rjenkins1
      	item osd.8 weight 1.455
      }
      root hdd {
      	id -1		# do not change unnecessarily
      	id -2 class hdd		# do not change unnecessarily
      	# weight 29.837
      	alg straw2
      	hash 0	# rjenkins1
      	item node1-hdd weight 7.278
      	item node2-hdd weight 10.916
      	item node3-hdd weight 7.278
      }
      root ssd {
      	# weight 29.837
      	alg straw2
      	hash 0	# rjenkins1
      	item node1-ssd weight 1.456
      	item node2-ssd weight 1.456
      	item node3-ssd weight 1.456
      }
      
      # rules 最终的root规则 pool应用到哪个
      rule hdd_rule { 
      	id 0
      	type replicated
      	min_size 1
      	max_size 10
      	step take hdd
      	step chooseleaf firstn 0 type host
      	step emit
      }
      rule ssd_rule { 
      	id 1
      	type replicated
      	min_size 1
      	max_size 10
      	step take ssd
      	step chooseleaf firstn 0 type host
      	step emit
      }
      
      # end crush map
      
      
    5. 将修改完的规则编译为bin文件

      # crushtool -c crushmap20200922.txt  -o crushmap20200922-new.bin
      # ll
      total 12
      -rw-r--r-- 1 root root 1326 Sep 22 23:06 crushmap20200922.bin
      -rw-r--r-- 1 root root 2113 Sep 22 23:33 crushmap20200922-new.bin
      -rw-r--r-- 1 root root 2516 Sep 22 23:33 crushmap20200922.txt
      
    6. 应用新的规则

      # ceph osd setcrushmap -i crushmap20200922-new.bin 
      20
      
    7. 应用后新老规则对比,已经可以区分出ssd和hdd规则

      # 应用前
      # ceph osd tree
      ID  CLASS  WEIGHT    TYPE NAME       STATUS  REWEIGHT  PRI-AFF
      -1         29.83720  root default                             
      -3          8.73286      host node1                           
       0    hdd   3.63869          osd.0       up   1.00000  1.00000
       1    hdd   3.63869          osd.1       up   1.00000  1.00000
       6    ssd   1.45549          osd.6       up   1.00000  1.00000
      -5         12.37148      host node2                           
       2    hdd   5.45799          osd.2       up   1.00000  1.00000
       3    hdd   5.45799          osd.3       up   1.00000  1.00000
       7    ssd   1.45549          osd.7       up   1.00000  1.00000
      -7          8.73286      host node3                           
       4    hdd   3.63869          osd.4       up   1.00000  1.00000
       5    hdd   3.63869          osd.5       up   1.00000  1.00000
       8    ssd   1.45549          osd.8       up   1.00000  1.00000
       
       # 应用后 可以看出有两个root规则,分为hdd和ssd
       ceph osd tree
      ID   CLASS  WEIGHT    TYPE NAME           STATUS  REWEIGHT  PRI-AFF
      -15          4.36798  root ssd                                     
      -12          1.45599      host node1-ssd                           
        6    ssd   1.45499          osd.6           up   1.00000  1.00000
      -13          1.45599      host node2-ssd                           
        7    ssd   1.45499          osd.7           up   1.00000  1.00000
      -14          1.45599      host node3-ssd                           
        8    ssd   1.45499          osd.8           up   1.00000  1.00000
       -1         25.47200  root hdd                                     
       -3          7.27800      host node1-hdd                           
        0    hdd   3.63899          osd.0           up   1.00000  1.00000
        1    hdd   3.63899          osd.1           up   1.00000  1.00000
       -5         10.91600      host node2-hdd                           
        2    hdd   5.45799          osd.2           up   1.00000  1.00000
        3    hdd   5.45799          osd.3           up   1.00000  1.00000
       -7          7.27800      host node3-hdd                           
        4    hdd   3.63899          osd.4           up   1.00000  1.00000
        5    hdd   3.63899          osd.5           up   1.00000  1.00000
      
    8. 验证,将之前的ceph-demo资源池的规则改为ssd的

      # ceph osd pool get ceph-demo crush_rule
      crush_rule: hdd_rule
      # ceph osd pool set ceph-demo crush_rule ssd_rule
      set pool 2 crush_rule to ssd_rule
      # ceph osd pool get ceph-demo crush_rule
      crush_rule: ssd_rule
      
      #创建一个100G的rbd
      # rbd create ceph-demo/rbd-demo.img --size 100G
      # rbd info ceph-demo/rbd-demo.img
      rbd image 'rbd-demo.img':
              size 100 GiB in 25600 objects
              order 22 (4 MiB objects)
              snapshot_count: 0
              id: 39a814402607
              block_name_prefix: rbd_data.39a814402607
              format: 2
              features: layering
              op_features: 
              flags: 
              create_timestamp: Tue Sep 22 23:42:55 2020
              access_timestamp: Tue Sep 22 23:42:55 2020
              modify_timestamp: Tue Sep 22 23:42:55 2020
      #将ceph-demo设置为rbd类型:
      # ceph osd pool application enable ceph-demo rbd
      
      # 挂载到本地
      # rbd map ceph-demo/rbd-demo.img
      /dev/rbd0
      ]# mkfs.xfs /dev/rbd0
      meta-data=/dev/rbd0              isize=512    agcount=16, agsize=1638400 blks
               =                       sectsz=512   attr=2, projid32bit=1
               =                       crc=1        finobt=0, sparse=0
      data     =                       bsize=4096   blocks=26214400, imaxpct=25
               =                       sunit=1024   swidth=1024 blks
      naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
      log      =internal log           bsize=4096   blocks=12800, version=2
               =                       sectsz=512   sunit=8 blks, lazy-count=1
      realtime =none                   extsz=4096   blocks=0, rtextents=0
      [root@node1 ceph]# mkdir /mnt/rdb-demo
      [root@node1 ceph]# mount /dev/rbd0 /mnt/rdb-demo
      
      #最后测速测一把 写入1.2G 符合ssd
      # time dd if=/dev/zero of=test.dbf bs=8k count=300000
      300000+0 records in
      300000+0 records out
      2457600000 bytes (2.5 GB) copied, 2.06784 s, 1.2 GB/s
      
      real    0m2.072s
      user    0m0.318s
      sys     0m1.742s
      
      #同时也可以看到文件落在三个ssd上
      # ceph osd map ceph-demo rbd-demo.img
      osdmap e81 pool 'ceph-demo' (2) object 'rbd-demo.img' -> pg 2.7d92bf55 (2.15) -> up ([6,8,7], p6) acting ([6,8,7], p6)
      
    9. 禁用掉禁用在增加或者修改osd的时候自动分配curshmap,否在在重启服务或者添加osd的时候 会有大量的pg迁移。出现事故,在配置文件中添加osd crush update on start = false 添加了这个参数后,后面加了osd后 需要手动将osd添加到curshmap规则里 重新应用

      # vim ceph.conf 
      ...
      [osd]
      osd crush update on start = false
      ...
      
      #推送到其他节点
      # ceph-deploy --overwrite-conf config push node1 node2 node3
      [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
      [ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node1 node2 node3
      [ceph_deploy.cli][INFO  ] ceph-deploy options:
      [ceph_deploy.cli][INFO  ]  username                      : None
      [ceph_deploy.cli][INFO  ]  verbose                       : False
      [ceph_deploy.cli][INFO  ]  overwrite_conf                : True
      [ceph_deploy.cli][INFO  ]  subcommand                    : push
      [ceph_deploy.cli][INFO  ]  quiet                         : False
      [ceph_deploy.cli][INFO  ]  cd_conf                       : 
      [ceph_deploy.cli][INFO  ]  cluster                       : ceph
      [ceph_deploy.cli][INFO  ]  client                        : ['node1', 'node2', 'node3']
      [ceph_deploy.cli][INFO  ]  func                          : 
      [ceph_deploy.cli][INFO  ]  ceph_conf                     : None
      [ceph_deploy.cli][INFO  ]  default_release               : False
      [ceph_deploy.config][DEBUG ] Pushing config to node1
      [node1][DEBUG ] connected to host: node1 
      [node1][DEBUG ] detect platform information from remote host
      [node1][DEBUG ] detect machine type
      [node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
      [ceph_deploy.config][DEBUG ] Pushing config to node2
      [node2][DEBUG ] connected to host: node2 
      [node2][DEBUG ] detect platform information from remote host
      [node2][DEBUG ] detect machine type
      [node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
      [ceph_deploy.config][DEBUG ] Pushing config to node3
      [node3][DEBUG ] connected to host: node3 
      [node3][DEBUG ] detect platform information from remote host
      [node3][DEBUG ] detect machine type
      [node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
      
      #重启osd 三台节点全部重启
      # systemctl restart ceph-osd.target
      

      注意事项

      • 在扩容和删除osd还有编辑curshmap的时候最好备份一个

      • 初始化就规划好curshmap的规则,否则应用过程中改变的话,就会有大量的pg迁移

      • 重启服务的时候就会自动修改curshmap,所以需要备份,也可以禁用在增加或者修改osd的时候自动分配curshmap

你可能感兴趣的:(ceph,ceph)