【Ceph】Ceph错误记录 Ceph 运维手册

Ceph 运维手册

第一部分：常用操作 - 12. 日志和调试 - 《Ceph 运维手册》 - 书栈网 · BookStack

分布式存储ceph运维操作

(摘抄自：https://www.cnblogs.com/mm2021/p/15121746.html）

—、统一节点上ceph.conf 文件

如果是在admin节点修改的ceph.conf，想推送到所有其他节点，则需要执行下述命令

ceph-deploy --overwrite-conf config push mon01 mon02 mon03 osd01 osd02 osd03

修改完毕配置文件后需要重启服务生效

二、ceph集群服务管理

@@!!!下述操作均需要在具体运行服务的那个节点上运行，而不是admin节点!!! @@

1.方式一

在各具体节点执行下行命令，重启节点上的所有ceph守护进程

systemctl restart ceph.target

2.方式二

在各具体节点执行下行命令，按类型重启相应的守护进程

# 1.重启mgr守护进程
systemctl restart ceph-mgr.target

# 2.重启mds守护进程
systemctl restart ceph-mds.target

# 3.重启rgw守护进程
systemctl restart ceph-radosgw.target

# 4.重启mon守护进程
systemctl restart ceph-mon.target

# 5.重启osd守护进程
登录到osd01节点上，该节点上运行有三个osd daemon进程osd.0、osd.l、osd.2

# 5.1重启所有的osd daemoon
systemctl restart ceph-osd.target

# 5.2挨个重启
syste

建议大家遇到集群状态发现报错或警告后，使用 ceph health detail命令 来查看到系统给出的处理建议。

Ceph错误记录

mds is damaged

mds.0 is damaged_discsthnew：https://blog.csdn.net/mailjoin/article/details/79694965

强制恢复（注意会丢数据，仅供开发测试时候用）

cephfs-journal-tool journal reset
ceph mds repaired 0 

unable to open OSD superblock on /var/lib/ceph/osd/ceph-45

key: use_rdma, val: 1
default pool attr 0, nr_hugepages 0, no neet set hugepages
2021-03-28 21:46:07.548765 7f904d555ec0 2818102 20 ERROR bluestore(/var/lib/ceph/osd/ceph-45/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-45/block: (2) No such file or directory
2021-03-28 21:46:07.548788 7f904d555ec0 2818102 20 ERROR  ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-45: (2) No such file or directory

 OSD 号不对，ceph osd tree 查看

[root@rdma61 osd]# ceph osd tree
ID  CLASS WEIGHT    TYPE NAME                   STATUS REWEIGHT PRI-AFF 
-10               0 root maintain                                       
 -9       123.71407 root hddpool                                        
-11       123.71407     rack rack.hddpool                               
-15        36.38649         host rdma61.hddpool                         
  3   hdd   7.27730             osd.3             down  1.00000 1.00000 
  6   hdd   7.27730             osd.6             down  1.00000 1.00000 
 10   hdd   7.27730             osd.10            down  1.00000 1.00000 
 13   hdd   7.27730             osd.13            down  1.00000 1.00000 
 16   hdd   7.27730             osd.16            down  1.00000 1.00000 
 -6        43.66379         host rdma63.hddpool                         
  1   hdd   7.27730             osd.1             down  1.00000 1.00000 
  4   hdd   7.27730             osd.4             down  1.00000 1.00000 
  7   hdd   7.27730             osd.7             down  1.00000 1.00000 
  9   hdd   7.27730             osd.9             down  1.00000 1.00000 
 12   hdd   7.27730             osd.12            down  1.00000 1.00000 
 15   hdd   7.27730             osd.15            down  1.00000 1.00000 
 -3        43.66379         host rdma64.hddpool                         
  2   hdd   7.27730             osd.2             down  1.00000 1.00000 
  5   hdd   7.27730             osd.5             down  1.00000 1.00000 
  8   hdd   7.27730             osd.8             down  1.00000 1.00000 
 11   hdd   7.27730             osd.11            down  1.00000 1.00000 
 14   hdd   7.27730             osd.14            down  1.00000 1.00000 
 17   hdd   7.27730             osd.17            down  1.00000 1.00000 
 -5        10.47839 root ssdpool                                        
-25        10.47839     rack rack.ssdpool                               
-28         3.49280         host rdma61.ssdpool                         
 18   ssd   0.87320             osd.18            down  1.00000 1.00000 
 21   ssd   0.87320             osd.21            down  1.00000 1.00000 
 24   ssd   0.87320             osd.24            down  1.00000 1.00000 
 28   ssd   0.87320             osd.28            down  1.00000 1.00000 
-31         3.49280         host rdma63.ssdpool                         
 20   ssd   0.87320             osd.20            down  1.00000 1.00000 
 22   ssd   0.87320             osd.22            down  1.00000 1.00000 
 25   ssd   0.87320             osd.25            down  1.00000 1.00000 
 27   ssd   0.87320             osd.27            down  1.00000 1.00000 
-34         3.49280         host rdma64.ssdpool                         
 19   ssd   0.87320             osd.19            down  1.00000 1.00000 
 23   ssd   0.87320             osd.23            down  1.00000 1.00000 
 26   ssd   0.87320             osd.26            down  1.00000 1.00000 
 29   ssd   0.87320             osd.29            down        0 1.00000 
 -1               0 root default  

未挂盘

[root@ceph]#lsblk

解决Ceph ERROR: unable to open OSD superblock - 简书

换回部署的时候的ceph包，然后激活：ceph-disk activate-all

fio 报错

[root@localhost ~]# ./fio --ioengine=rbd --iodepth=4 --numjobs=8 --pool=.rbdpool.rbd --rbdname=lun0 --name=write5 --rw=randwrite --bs=1M --size=10G --group_reporting --direct=1
write5: (g=0): rw=randwrite, bs=1M-1M/1M-1M/1M-1M, ioengine=rbd, iodepth=4
...
fio-2.1.10
Starting 8 processes
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.
rbd engine: RBD version: 1.12.0
rados_ioctx_create failed.
fio_rbd_connect failed.

OSD 池不对

fio 测到一半段错误

[‎2021/‎9/‎18 14:54] liangchaoxi (Cloud): 
无标题
Core was generated by `fio fio_10.conf'.
Program terminated with signal 11, Segmentation fault.
#0  Mutex::lock (this=0xab1aaa000, no_lockdep=false) at /compiledir/zhangtao/ceph-L/src/common/Mutex.cc:97
97      /compiledir/zhangtao/ceph-L/src/common/Mutex.cc: 没有那个文件或目录.
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.176-5.el7.x86_64 glibc-2.17-324.el7_9.x86_64 nspr-4.25.0-2.el7_9.x86_64 nss-3.53.1-7.el7_9.x86_64 nss-softokn-3.53.1-6.el7_9.x86_64 nss-softokn-freebl-3.53.1-6.el7_9.x86_64 nss-util-3.53.1-1.el7_9.x86_64 xz-libs-5.2.2-1.el7.x86_64
(gdb) bt
#0  Mutex::lock (this=0xab1aaa000, no_lockdep=false) at /compiledir/zhangtao/ceph-L/src/common/Mutex.cc:97
#1  0x00007f90effe1710 in librbd::io::AioCompletion::get_return_value() () from /opt/h3c/lib/librbd.so.1
#2  0x000000000045d6f4 in _fio_rbd_finish_aiocb (comp=, data=) at engines/rbd.c:190
#3  0x00007f90effe0c35 in librbd::io::AioCompletion::complete() () from /opt/h3c/lib/librbd.so.1
#4  0x00007f90effe1ba3 in librbd::io::AioCompletion::complete_request(long, bool) () from /opt/h3c/lib/librbd.so.1
#5  0x00007f90f0000ee6 in librbd::io::ReadResult::C_SparseReadRequestBase::finish(std::map, std::allocator > >&, std::vector, std::allocator > > const&, unsigned long, unsigned long, ceph::buffer::list&, int) () from /opt/h3c/lib/librbd.so.1
#6  0x00007f90effe43fd in librbd::io::ReadResult::C_SparseReadRequest::finish(int) () from /opt/h3c/lib/librbd.so.1
#7  0x00007f90eff06dc9 in Context::complete(int) () from /opt/h3c/lib/librbd.so.1
#8  0x00007f90efffbc8f in librbd::io::ObjectRequest::complete(int) () from /opt/h3c/lib/librbd.so.1
#9  0x00007f90efb292b2 in librados::C_AioComplete::finish(int) () from /opt/h3c/lib/librados.so.2
#10 0x00007f90efaff5b9 in Context::complete(int) () from /opt/h3c/lib/librados.so.2
#11 0x00007f90e5c92268 in Finisher::finisher_thread_entry (this=0x7f90ac12cd90) at /compiledir/zhangtao/ceph-L/src/common/Finisher.cc:83
#12 0x00007f90eef35ea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f90eea5a9fd in clone () from /lib64/libc.so.6 

原来是fio 版本不对，将fio 2.20 替换成fio 3.20就好了

ceph -s 失败……timed out

例子：

[root@rdma55 ~]# ceph -s
2021-08-24 19:02:27.925124 7f6a05f63700 548321 2 WARNING monclient(hunting): authenticate timed out after 75
2021-08-24 19:02:27.925168 7f6a05f63700 548321 2 WARNING librados: client.admin authentication error (110) Connection timed out

原因：

查看/var/log/ceph/ceph-mon.log 看到

monitor data filesystem reached concerning levels of available storage space
you may adjust mon data avail crit to a lower value to make this go away (default: 5%)

解决办法见本文：Mon节点启动失败

 Mon节点启动失败

错误描述：monitor data filesystem reached concerning levels of available storage space
you may adjust mon data avail crit to a lower value to make this go away (default: 5%)

这里涉及到一个参数：mon-data-avail-crit，这个参数用来监控mon节点的数据存储，ceph mon节点数据存储default情况是在目录: /var/lib/ceph目录下，如果该目录的使用率超过95%，就会导致Mon进程被kill掉。

解决方法：

说明磁盘空间不够了，查看一下磁盘是什么数据占用大量的空间。（存储的数据、core文件等），清除无用数据腾出空间。
清理 /var/lib/这个目录的数据，或者设置参数：mon-data-avail-crit

《Ceph集群常见问题处理方法》第六篇：Ceph集群常见问题处理方法_cheney的专栏-CSDN博客

《ceph搭建过程中遇到的问题汇总》ceph搭建过程中遇到的问题汇总_Yannick Jiang 的专栏-CSDN博客

经验教训：

fio 版本问题可能会导致fio测试的过程中崩溃。

日志默认是开启的，日志没关将导致性能很低。

部署ceph后，配置文件中缺少rdma和libgo的配置、/etc/ceph/目录下没有libgo的配置文件libgo_josn

编译环境上的libgo版本需要与运行环境上从libgo版本相同。

性能太低

日志没关

broken_new

 engine.15.20 engine_create_state: broken_new

可能是元数据坏了，删池善卷重建
 

dse起不来:terminate called after throwing an instance of 'ceph::buffer::malformed_input'
  what():  buffer::malformed_input: void CSDSuperblock::decode(ceph::buffer::list::iterator&) no longer understand old encoding version 1 < struct_compat

terminate called after throwing an instance of 'ceph::buffer::malformed_input'
  what():  buffer::malformed_input: void CSDSuperblock::decode(ceph::buffer::list::iterator&) no longer understand old encoding version 1 < struct_compat
*** Caught signal (Aborted) **
 in thread 7f88997fd700 thread_name:Processer_1
 ceph version 0cc873 (f0cc873864ed88a23d19b32381a348dfbb151b2e) luminous (stable)
 1: (()+0x6d8fd0) [0x55616bcd8fd0]
 2: (()+0xf630) [0x7f8933c01630]
 3: (gsignal()+0x37) [0x7f8929c21387]
 4: (abort()+0x148) [0x7f8929c22a78]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f892a531ac5]
 6: (()+0x5ea36) [0x7f892a52fa36]
 7: (()+0x5ea63) [0x7f892a52fa63]
 8: (()+0x5ec83) [0x7f892a52fc83]
 9: (CSDSuperblock::decode(ceph::buffer::list::iterator&)+0x20f) [0x7f8936e6960f]
 10: (CSD::read_csd_superblock(CSDSuperblock&)+0x105) [0x55616c0a81d5]
 11: (CSD::init()+0xb6) [0x55616c0fdf86]
 12: (CSD::try_create(int, co::Scheduler*, std::vector >&, unsigned char, CSD*&, co::CoSem&)+0x2f4f) [0x55616c12122f]
 13: (()+0xdc830) [0x7f893620b830]
 14: (co::Task::Run()+0xc9) [0x7f893620b90f]
 15: (co::Task::StaticRun(long)+0x2f) [0x7f893620bd85]
 16: (make_fcontext()+0x21) [0x7f89362c7941]
2021-10-25 15:25:11.844783 7f88997fd700 921594 13 ERROR *** Caught signal (Aborted) **
 in thread 7f88997fd700 thread_name:Processer_1

原因，集群dse的元数据（启动之后读取自己身的元数据，因为数据损坏，所以decode失败）

解决：重新安装产品包

 zengjun ：
看下rpm -qf /usr/lib64/libibverbs*.so ， 估计你们编译时链接的verbs库和实际运行时链接的库不一样。 
你们ceph产品包目录下还有一个目录指定了libibvers*的库

zhangtao解决了libgo造成的编译不通过的问题：先前安装的旧版libgo的task.h 头文件放在了目录A下，新版的libgo把task.h 放在了目录B下。
编译的时候系统默认优先include A目录下。旧版libgo的stack.h 缺少新版的一些成员变量导致。

too few PGs per OSD 错误

RDMA 

failed to transition to RTR state: (22) Invalid argument

两端的配置不一致，一边posix，一边rdma。或一边预留内存，一边不预留
 

ceph -s
  cluster:
    id:     da54ea6a-111a-434c-b78e-adad1ac66abb
    health: HEALTH_WARN
            too few PGs per OSD (10 < min 30)
 
  services:
    mon: 3 daemons, quorum master1,master2,master3
    mgr: master1(active), standbys: master2
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   1 pools, 10 pgs
    objects: 17  objects, 24 MiB
    usage:   3.0 GiB used, 24 GiB / 27 GiB avail
    pgs:     10 active+clean

从上面可以看到，提示说每个osd上的pg数量小于最小的数目30个。pgs为10，因为是2副本的配置，所以当有3个osd的时候，每个osd上均分了10/3 *2=6个pgs,也就是出现了如上的错误 小于最小配置30个。
集群这种状态如果进行数据的存储和操作，会发现集群卡死，无法响应io，同时会导致大面积的osd down。

• 修改默认pool rbd的pgs
  ceph osd pool set rbd pg_num 50
  此时，ceph -s 查看会提示pg_num 大于 pgp_num，所以还需要修改pgp_num
  ceph osd pool set rbd pgp_num 50
  再次查看：

ceph -s
  cluster:
    id:     da54ea6a-111a-434c-b78e-adad1ac66abb
    health: HEALTH_WARN
            application not enabled on 1 pool(s)

• 提示 application not enabled

[root@master1 ~]# ceph health detail
HEALTH_WARN application not enabled on 1 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)
    application not enabled on pool 'rbd'
    use 'ceph osd pool application enable  ', where  is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.
[root@master1 ~]# ceph osd pool application enable rbd rbd
enabled application 'rbd' on pool 'rbd'

根据上面的提示，选择存储方式：cephfs ，rbd ，rgw 。此处我是rbd块存储。
查看：

root@master1 ~]# ceph -s
  cluster:
    id:     da54ea6a-111a-434c-b78e-adad1ac66abb
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum master1,master2,master3
    mgr: master1(active), standbys: master2
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   1 pools, 50 pgs
    objects: 17  objects, 24 MiB
    usage:   3.0 GiB used, 24 GiB / 27 GiB avail
    pgs:     50 active+clean

ceph 常见的Error和HEALTH_WARN解决办法 - fuhaizi - 博客园

HEALTH_WARN application not enabled on pool '.rgw.root'

• 在创建rgw对象存储时，报错：

root@master1 ceph-cluster]# ceph health detail
HEALTH_WARN application not enabled on 1 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)
    application not enabled on pool '.rgw.root'
    use 'ceph osd pool application enable  ', where  is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.
[root@master1 ceph-cluster]# ceph osd pool application enable .rgw.root rgw
enabled application 'rgw' on pool '.rgw.root'
ceph -s  查看已恢复正常

• too many PGs per OSD告警的处理
  ceph自L版本后，mon_max_pg_per_osd,默认值也从300变更为200
  修改ceph.conf文件

[root@master1 ceph-cluster]# cat ceph.conf 
[global]
......
mon_max_pg_per_osd = 1000
在global下面追加一行。
推送到各节点：
ceph-deploy --overwrite-conf config push master1 master2 master3
重启mgr和各节点mon
systemctl restart ceph-mgr@master1
systemctl restart ceph-mon@master1
systemctl restart ceph-mon@master2
systemctl restart ceph-mon@master3
ceph -s  查看已恢复正常

一般ceph.conf在不修改配置文件的情况下，好的参数都使用的默认配置，此时创建rgw用户时也会报错：

rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)
[root@master1 ceph-cluster]# radosgw-admin user create --uid=radosgw --display-name='radosgw'
{
    "user_id": "radosgw",
    "display_name": "radosgw",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,      #最大值1000已生效
    "auid": 0,
    "subusers": [],
......

解决办法同上，修改默认配置文件后重启即可。

ceph 常见的Error和HEALTH_WARN解决办法 - fuhaizi - 博客园

health_warn：clock skew detected on mon

造成集群状态health_warn：clock skew detected on mon节点的原因有两个，一个是mon节点上ntp服务器未启动，另一个是ceph设置的mon的时间偏差阈值比较小。

排查时也应遵循先第一个原因，后第二个原因的方式。

1、确认ntp服务是否正常工作

$ systemctl status ntpd
如果没有安装ntpd，可以参照以下文章进行安装，传送门：Centos7 搭建NTP服务器及客户端同步时间

centos7 查看启动ntp服务命令 - sisimi_2017 - 博客园

2、如果1步还不能解决问题，那么有可能设置的偏差阈值过小。

使用 ceph health detail命令 来查看到系统给出的处理建议

修改ceph配置中的时间偏差阈值

在deploy节点修改配置文件调整时间偏差阈值，命令如下：

$ vim /ceph-install/ceph.conf
在global字段下添加：

mon clock drift allowed = 2
mon clock drift warn backoff = 30
向需要同步的mon节点推送配置文件，命令如下：

[root@cephnode01 my-cluster]# ceph-deploy --overwrite-conf config push cephnode{01,02,03}
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push cephnode01 cephnode02 cephnode03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       :
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['cephnode01', 'cephnode02', 'cephnode03']
[ceph_deploy.cli][INFO  ]  func                          :
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to cephnode01
[cephnode01][DEBUG ] connected to host: cephnode01 
[cephnode01][DEBUG ] detect platform information from remote host
[cephnode01][DEBUG ] detect machine type
[cephnode01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to cephnode02
[cephnode02][DEBUG ] connected to host: cephnode02 
[cephnode02][DEBUG ] detect platform information from remote host
[cephnode02][DEBUG ] detect machine type
[cephnode02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to cephnode03
[cephnode03][DEBUG ] connected to host: cephnode03 
[cephnode03][DEBUG ] detect platform information from remote host
[cephnode03][DEBUG ] detect machine type
[cephnode03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
这里是向node1、node2、node3节点推送，重启mon服务，命令如下：

$ systemctl restart ceph-mon.target
3、验证

[root@cephnode01 my-cluster]# ceph -s
  cluster:
    id:     406e0c23-755f-4378-bbc9-13548c4d3d64
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum cephnode01,cephnode02,cephnode03 (age 8m)
    mgr: cephnode01(active, since 71m), standbys: cephnode03, cephnode02
    mds:  3 up:standby
    osd: 3 osds: 3 up (since 11m), 3 in (since 47m)
    rgw: 1 daemon active (cephnode01)
 
  task status:
 
  data:
    pools:   4 pools, 128 pgs
    objects: 187 objects, 1.2 KiB
    usage:   3.0 GiB used, 12 GiB / 15 GiB avail
    pgs:     128 active+clean
显示health_ok说明问题解决。
————————————————
版权声明：本文为CSDN博主「富士康质检员张全蛋」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/qq_34556414/article/details/116095875

通信 
 

通信不通、超时，特别是前端：可能是ceph的前端网络配置改为RDMA并重启了，但是ganesha登前端client 没有重启加载配置