ceph 维护系列(一)删除一个ceph 节点

零 修订记录

序号 修订内容 修订时间
1 新增 2021/2/19

一 摘要

本文是基于(ceph 纵向扩容 nautilus版本)[https://www.cnblogs.com/weiwei2021/p/14381416.html],对cephtest003.ceph.kxdigit.com 节点进行卸载。

二 环境信息

主机名 状态 IP 磁盘 角色
cephtest001.ceph.kxdigit.com 已完成 10.3.176.10 系统盘:/dev/sda 数据盘:/dev/sdb /dev/sdc /dev/sdd ceph-deploy,monitor,mgr,mds,osd
cephtest002.ceph.kxdigit.com 已完成 10.3.176.16 系统盘:/dev/sda 数据盘:/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf monitor,mgr,mds,osd
cephtest003.ceph.kxdigit.com 已完成(待删除节点) 10.3.176.44 系统盘:/dev/sda 数据盘:/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg monitor,mgr,mds,osd
cephtest004.ceph.kxdigit.com 已完成 10.3.176.36 系统盘:/dev/sda 数据盘:/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf(待纵向扩容) monitor,mgr,mds,osd

三 实施

(一)ceph 集群当前状态

3.1.1 集群健康状态

可见 cephtest003 上有 mon,mgr,osd 服务。

[cephadmin@cephtest001 ~]$ ceph -s
  cluster:
    id:     6cd05235-66dd-4929-b697-1562d308d5c3
    health: HEALTH_WARN
            1 pools have many more objects per pg than average

  services:
    mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age                                                                13d)
    mgr: cephtest001(active, since 7w), standbys: cephtest002, cephtest003, ceph                                                               test004
    osd: 19 osds: 19 up (since 11d), 19 in (since 11d)
    rgw: 1 daemon active (cephtest004)

  task status:

  data:
    pools:   8 pools, 400 pgs
    objects: 24.12k objects, 167 GiB
    usage:   522 GiB used, 70 TiB / 71 TiB avail
    pgs:     400 active+clean

  io:
    client:   77 KiB/s rd, 341 B/s wr, 85 op/s rd, 0 op/s wr

3.1.2 osd 状态

可见 cephtest003 上有osd.8\osd.9\osd.10\osd.11\osd.12\osd.13 共6块osd。

[cephadmin@cephtest001 cephcluster]$ ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       70.57448 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-7       43.66132     host cephtest003
 8   hdd  7.27689         osd.8            up  1.00000 1.00000
 9   hdd  7.27689         osd.9            up  1.00000 1.00000
10   hdd  7.27689         osd.10           up  1.00000 1.00000
11   hdd  7.27689         osd.11           up  1.00000 1.00000
12   hdd  7.27689         osd.12           up  1.00000 1.00000
13   hdd  7.27689         osd.13           up  1.00000 1.00000
-9       18.19099     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
18   hdd  3.63820         osd.18           up  1.00000 1.00000
[cephadmin@cephtest001 cephcluster]$

(二)方法一 移除cephtest003节点上所有osd(cephtest003 机器上操作)

逐个删除该节点上所有osd,先把数据迁移走,然后再删除。
###3.2.1 调整osd的crush weight(osd.8 为例)
cephadmin 用户登录cephtest003 ,

[cephadmin@cephtest003 ~]$ ceph osd crush reweight osd.8 0.1
reweighted item id 8 name 'osd.8' to 0.1 in crush map

执行了之后,把osd.8 里 数据迁移出来较慢,并且还会迁移到该节点cephtest003 其他机器上。
现在改为另一种方法 移除cephtest003

(三)方法二 移除cephtest003节点上所有osd(cephtest003 机器上操作)

该方法 直接关闭节点osd 服务,直接删除各个osd。

3.3.1 停掉cephtest003 osd 服务

先看下当前节点osd 服务。

[cephadmin@cephtest003 ~]$ systemctl status ceph-osd.target
● ceph-osd.target - ceph target allowing to start/stop all [email protected] instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd.target; enabled; vendor preset: enabled)
   Active: active since Thu 2020-12-31 09:09:07 CST; 1 months 19 days ago
[cephadmin@cephtest003 ~]$ ps -ef | grep osd
ceph       20806       1  0  2020 ?        03:43:45 /usr/bin/ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
ceph       20809       1  0  2020 ?        04:00:41 /usr/bin/ceph-osd -f --cluster ceph --id 12 --setuser ceph --setgroup ceph
ceph       20816       1  0  2020 ?        03:45:35 /usr/bin/ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph
ceph       20819       1  0  2020 ?        04:04:24 /usr/bin/ceph-osd -f --cluster ceph --id 13 --setuser ceph --setgroup ceph
ceph       20821       1  0  2020 ?        03:28:47 /usr/bin/ceph-osd -f --cluster ceph --id 10 --setuser ceph --setgroup ceph
ceph       20824       1  0  2020 ?        05:05:17 /usr/bin/ceph-osd -f --cluster ceph --id 11 --setuser ceph --setgroup ceph
cephadm+  770980  770940  0 13:54 pts/1    00:00:00 grep --color=auto osd
[cephadmin@cephtest003 ~]$

停服务,需要root 权限

[cephadmin@cephtest003 ~]$ systemctl stop ceph-osd.target
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:
==== AUTHENTICATION COMPLETE ===

可见osd 已经down


[cephadmin@cephtest003 ~]$ ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       63.39758 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-7       36.48442     host cephtest003
 8   hdd  0.09999         osd.8          down  1.00000 1.00000
 9   hdd  7.27689         osd.9          down  1.00000 1.00000
10   hdd  7.27689         osd.10         down  1.00000 1.00000
11   hdd  7.27689         osd.11         down  1.00000 1.00000
12   hdd  7.27689         osd.12         down  1.00000 1.00000
13   hdd  7.27689         osd.13         down  1.00000 1.00000
-9       18.19099     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
18   hdd  3.63820         osd.18           up  1.00000 1.00000
[cephadmin@cephtest003 ~]$

3.3.2 删除该节点上所有osd

[cephadmin@cephtest003 ~]$ ceph osd rm 8
removed osd.8
[cephadmin@cephtest003 ~]$ ceph osd rm 9
removed osd.9
[cephadmin@cephtest003 ~]$ ceph osd rm 10
removed osd.10
[cephadmin@cephtest003 ~]$ ceph osd rm 11
removed osd.11
[cephadmin@cephtest003 ~]$ ceph osd rm 12
removed osd.12
[cephadmin@cephtest003 ~]$ ceph osd rm 13
removed osd.13
[cephadmin@cephtest003 ~]$

[cephadmin@cephtest003 ~]$ ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       63.39758 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-7       36.48442     host cephtest003
 8   hdd  0.09999         osd.8           DNE        0
 9   hdd  7.27689         osd.9           DNE        0
10   hdd  7.27689         osd.10          DNE        0
11   hdd  7.27689         osd.11          DNE        0
12   hdd  7.27689         osd.12          DNE        0
13   hdd  7.27689         osd.13          DNE        0
-9       18.19099     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
18   hdd  3.63820         osd.18           up  1.00000 1.00000
[cephadmin@cephtest003 ~]$

3.3.3 删除该节点上所有osd 的crush map

[cephadmin@cephtest003 ~]$ ceph osd crush rm osd.8
removed item id 8 name 'osd.8' from crush map
[cephadmin@cephtest003 ~]$ ceph osd crush rm osd.9
removed item id 9 name 'osd.9' from crush map
[cephadmin@cephtest003 ~]$ ceph osd crush rm osd.10
removed item id 10 name 'osd.10' from crush map
[cephadmin@cephtest003 ~]$ ceph osd crush rm osd.11
removed item id 11 name 'osd.11' from crush map
[cephadmin@cephtest003 ~]$ ceph osd crush rm osd.12
removed item id 12 name 'osd.12' from crush map
[cephadmin@cephtest003 ~]$ ceph osd crush rm osd.13
removed item id 13 name 'osd.13' from crush map
[cephadmin@cephtest003 ~]$ ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       26.91316 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-7              0     host cephtest003
-9       18.19099     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
18   hdd  3.63820         osd.18           up  1.00000 1.00000
[cephadmin@cephtest003 ~]$

3.3.3 删除该节点上所有osd 的认证

[cephadmin@cephtest003 ~]$ ceph auth list | grep osd.8
installed auth entries:

osd.8
[cephadmin@cephtest003 ~]$ ceph auth del osd.8
updated

[cephadmin@cephtest003 ~]$ ceph auth del osd.9
updated
[cephadmin@cephtest003 ~]$ ceph auth del osd.10
updated
[cephadmin@cephtest003 ~]$ ceph auth del osd.11
updated
[cephadmin@cephtest003 ~]$ ceph auth del osd.12
updated
[cephadmin@cephtest003 ~]$ ceph auth del osd.13
updated
[cephadmin@cephtest003 ~]$

3.3.4 ceph osd tree中删除此节点的crush map

[cephadmin@cephtest003 ~]$ ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       26.91316 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-7              0     host cephtest003
-9       18.19099     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
18   hdd  3.63820         osd.18           up  1.00000 1.00000
[cephadmin@cephtest003 ~]$ ceph osd crush rm  cephtest003
removed item id -7 name 'cephtest003' from crush map
[cephadmin@cephtest003 ~]$ ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       26.91316 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-9       18.19099     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
18   hdd  3.63820         osd.18           up  1.00000 1.00000
[cephadmin@cephtest003 ~]$

3.3.5 卸载该节点所有挂载在osd的硬盘(root 用户)

[root@cephtest003 osd]# ll -ah
total 0
drwxr-x---.  8 ceph ceph  94 Dec 25 09:29 .
drwxr-x---. 15 ceph ceph 222 Dec 24 16:49 ..
drwxrwxrwt   2 ceph ceph 180 Dec 31 09:09 ceph-10
drwxrwxrwt   2 ceph ceph 180 Dec 31 09:09 ceph-11
drwxrwxrwt   2 ceph ceph 180 Dec 31 09:09 ceph-12
drwxrwxrwt   2 ceph ceph 180 Dec 31 09:09 ceph-13
drwxrwxrwt   2 ceph ceph 180 Dec 31 09:09 ceph-8
drwxrwxrwt   2 ceph ceph 180 Dec 31 09:09 ceph-9
[root@cephtest003 osd]# cd ceph-8/
[root@cephtest003 ceph-8]# ll
total 24
lrwxrwxrwx 1 ceph ceph 93 Dec 31 09:09 block -> /dev/ceph-7db02a0b-fe54-46c9-ac16-22272491f5ab/osd-block-60527c79-6c65-4807-bb4b-93b1eafc586b
-rw------- 1 ceph ceph 37 Dec 31 09:09 ceph_fsid
-rw------- 1 ceph ceph 37 Dec 31 09:09 fsid
-rw------- 1 ceph ceph 55 Dec 31 09:09 keyring
-rw------- 1 ceph ceph  6 Dec 31 09:09 ready
-rw------- 1 ceph ceph 10 Dec 31 09:09 type
-rw------- 1 ceph ceph  2 Dec 31 09:09 whoami
[root@cephtest003 ceph-8]# cd ../
[root@cephtest003 osd]# cd ~
[root@cephtest003 ~]# umount /var/lib/ceph/osd/ceph-8
[root@cephtest003 ~]# ll /var/lib/ceph/osd/
total 0
drwxrwxrwt  2 ceph ceph 180 Dec 31 09:09 ceph-10
drwxrwxrwt  2 ceph ceph 180 Dec 31 09:09 ceph-11
drwxrwxrwt  2 ceph ceph 180 Dec 31 09:09 ceph-12
drwxrwxrwt  2 ceph ceph 180 Dec 31 09:09 ceph-13
drwxr-xr-x. 2 ceph ceph   6 Dec 25 09:28 ceph-8
drwxrwxrwt  2 ceph ceph 180 Dec 31 09:09 ceph-9
[root@cephtest003 ~]# umount /var/lib/ceph/osd/ceph-9
[root@cephtest003 ~]# umount /var/lib/ceph/osd/ceph-10
[root@cephtest003 ~]# umount /var/lib/ceph/osd/ceph-11
[root@cephtest003 ~]# umount /var/lib/ceph/osd/ceph-12
[root@cephtest003 ~]# umount /var/lib/ceph/osd/ceph-13
[root@cephtest003 ~]# cd /var/lib/ceph/osd/
[root@cephtest003 osd]# ll
total 0
drwxr-xr-x. 2 ceph ceph 6 Dec 25 09:29 ceph-10
drwxr-xr-x. 2 ceph ceph 6 Dec 25 09:29 ceph-11
drwxr-xr-x. 2 ceph ceph 6 Dec 25 09:29 ceph-12
drwxr-xr-x. 2 ceph ceph 6 Dec 25 09:29 ceph-13
drwxr-xr-x. 2 ceph ceph 6 Dec 25 09:28 ceph-8
drwxr-xr-x. 2 ceph ceph 6 Dec 25 09:28 ceph-9
[root@cephtest003 osd]# cd ceph-8/
[root@cephtest003 ceph-8]# ll
total 0
[root@cephtest003 ceph-8]#

(四) 移除该节点mon

3.4.1 当前集群状态

[root@cephtest003 ceph-8]# su - cephadmin
Last login: Fri Feb 19 13:54:40 CST 2021 on pts/1
[cephadmin@cephtest003 ~]$ ceph -s
  cluster:
    id:     6cd05235-66dd-4929-b697-1562d308d5c3
    health: HEALTH_WARN
            1 pools have many more objects per pg than average
            Degraded data redundancy: 17889/72348 objects degraded (24.726%), 97 pgs degraded, 97 pgs undersized

  services:
    mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age 13d)
    mgr: cephtest001(active, since 7w), standbys: cephtest002, cephtest003, cephtest004
    osd: 13 osds: 13 up (since 28m), 13 in (since 11d); 97 remapped pgs
    rgw: 1 daemon active (cephtest004)

  task status:

  data:
    pools:   8 pools, 400 pgs
    objects: 24.12k objects, 167 GiB
    usage:   406 GiB used, 27 TiB / 27 TiB avail
    pgs:     17889/72348 objects degraded (24.726%)
             16284/72348 objects misplaced (22.508%)
             302 active+clean
             95  active+undersized+degraded+remapped+backfill_wait
             2   active+undersized+degraded+remapped+backfilling
             1   active+remapped+backfill_wait

  io:
    client:   219 KiB/s rd, 243 op/s rd, 0 op/s wr
    recovery: 25 MiB/s, 4 objects/s

[cephadmin@cephtest003 ~]$

3.4.2 移除cephtest003 mon

[cephadmin@cephtest003 ~]$ ceph mon stat
e2: 4 mons at {cephtest001=[v2:10.3.176.10:3300/0,v1:10.3.176.10:6789/0],cephtest002=[v2:10.3.176.16:3300/0,v1:10.3.176.16:6789/0],cephtest003=[v2:10.3.176.44:3300/0,v1:10.3.176.44:6789/0],cephtest004=[v2:10.3.176.36:3300/0,v1:10.3.176.36:6789/0]}, election epoch 144, leader 0 cephtest001, quorum 0,1,2,3 cephtest001,cephtest002,cephtest003,cephtest004
[cephadmin@cephtest003 ~]$ ceph mon remove cephtest003
removing mon.cephtest003 at [v2:10.3.176.44:3300/0,v1:10.3.176.44:6789/0], there will be 3 monitors
[cephadmin@cephtest003 ~]$ ceph mon stat
e3: 3 mons at {cephtest001=[v2:10.3.176.10:3300/0,v1:10.3.176.10:6789/0],cephtest002=[v2:10.3.176.16:3300/0,v1:10.3.176.16:6789/0],cephtest004=[v2:10.3.176.36:3300/0,v1:10.3.176.36:6789/0]}, election epoch 150, leader 0 cephtest001, quorum 0,1,2 cephtest001,cephtest002,cephtest004
[cephadmin@cephtest003 ~]$

3.4.3 修改ceph.conf

修改部署节点 /home/cephadmin/cephcluster/ceph.conf 待批量推送的配置文件,删除cephtest003 相关配置。

[cephadmin@cephtest001 cephcluster]$ pwd
/home/cephadmin/cephcluster
[cephadmin@cephtest001 cephcluster]$ cat ceph.conf
[global]
fsid = 6cd05235-66dd-4929-b697-1562d308d5c3
mon_initial_members = cephtest001, cephtest002, cephtest004
mon_host = 10.3.176.10,10.3.176.16,10.3.176.36
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

public network = 10.3.176.0/22
cluster network = 10.3.176.0/22

[cephadmin@cephtest001 cephcluster]$

批量推送配置文件到节点。

3.4.4 推送 修改/home/cephadmin/cephcluster/ceph.conf 到四台节点(部署节点执行)

[cephadmin@cephtest001 cephcluster]$ ceph-deploy --overwrite-conf admin cephtest001 cephtest002  cephtest004

3.4.5 修改 /etc/ceph 目录权限(所有节点执行)

[cephadmin@cephtest004 ~]$ sudo chown -R cephadmin:cephadmin /etc/ceph

##(五)关闭mgr
到待删除节点上关闭mgr 服务即可。

[cephadmin@cephtest003 ~]$ ceph -s
  cluster:
    id:     6cd05235-66dd-4929-b697-1562d308d5c3
    health: HEALTH_WARN
            1 pools have many more objects per pg than average

  services:
    mon: 3 daemons, quorum cephtest001,cephtest002,cephtest004 (age 23m)
    mgr: cephtest001(active, since 7w), standbys: cephtest002, cephtest003, cephtest004
    osd: 13 osds: 13 up (since 71m), 13 in (since 11d); 1 remapped pgs
    rgw: 1 daemon active (cephtest004)

  task status:

  data:
    pools:   8 pools, 400 pgs
    objects: 24.12k objects, 167 GiB
    usage:   517 GiB used, 26 TiB / 27 TiB avail
    pgs:     28/72348 objects misplaced (0.039%)
             399 active+clean
             1   active+clean+remapped

  io:
    client:   77 KiB/s rd, 1.3 KiB/s wr, 86 op/s rd, 0 op/s wr

[cephadmin@cephtest003 ~]$ systemctl status ceph-mgr.target
● ceph-mgr.target - ceph target allowing to start/stop all [email protected] instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph-mgr.target; enabled; vendor preset: enabled)
   Active: active since Thu 2020-12-31 09:09:07 CST; 1 months 19 days ago
[cephadmin@cephtest003 ~]$ systemctl stop ceph-mgr.target
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:
==== AUTHENTICATION COMPLETE ===

等一会再执行ceph-s

[cephadmin@cephtest003 ~]$ ceph -s
  cluster:
    id:     6cd05235-66dd-4929-b697-1562d308d5c3
    health: HEALTH_WARN
            1 pools have many more objects per pg than average

  services:
    mon: 3 daemons, quorum cephtest001,cephtest002,cephtest004 (age 26m)
    mgr: cephtest001(active, since 7w), standbys: cephtest002, cephtest004
    osd: 13 osds: 13 up (since 75m), 13 in (since 11d); 1 remapped pgs
    rgw: 1 daemon active (cephtest004)

  task status:

  data:
    pools:   8 pools, 400 pgs
    objects: 24.12k objects, 167 GiB
    usage:   517 GiB used, 26 TiB / 27 TiB avail
    pgs:     28/72348 objects misplaced (0.039%)
             399 active+clean
             1   active+clean+remapped

  io:
    client:   7.2 KiB/s rd, 27 KiB/s wr, 10 op/s rd, 7 op/s wr

[cephadmin@cephtest003 ~]$

你可能感兴趣的:(#,ceph)