【ceph相关】ceph常见问题处理

一、pg相关

1、xx objects unfound

- 问题描述:

dmesg查看磁盘发现读写异常,部分对象损坏(处于objects nofound状态),集群处于ERR状态

root@node1101:~# ceph health detail
HEALTH_ERR noscrub,nodeep-scrub flag(s) set; 13/409798 objects unfound(0.003%);17 stuck requests are blocked > 4096 sec. Implicated osds 38
OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set
OBJECT_UNFOUND 13/409798 objects unfound (0.003%)
  pg 5.309 has 1 unfound objects
  pg 5.2da has 1 unfound objects
  pg 5.2c9 has 1 unfound objects
  pg 5.1e2 has 1 unfound objects
  pg 5.6a has 1 unfound objects
  pg 5.120 has 1 unfound objects
  pg 5.148 has 1 unfound objects
  pg 5.14b has 1 unfound objects
  pg 5.160 has 1 unfound objects
  pg 5.35b has 1 unfound objects
  pg 5.39c has 1 unfound objects
  pg 5.3ad has 1 unfound objects
REQUEST_STUCK 17 stuck requests are blocked > 4096 sec. Implicated osds 38
  17 ops are blocked > 67108.9 sec
  osd.38 has stuck requests > 67108.9 sec

- 处理措施:

将unfound pg强制删除,参考命令:ceph pg {pgid} mark_unfound_lost delete
注:如需批量删除unfound pg,则参考命令如下

for i in `ceph health detail | grep pg | awk '{print $2}'`;do ceph pg $i mark_unfound_lost delete;done

2、Reduced data availability: xx pgs inactive

- 问题描述:

磁盘出现读写异常,osd无法启动,强制替换故障盘为新盘加入到集群,出现pgs inactive(unkown)

root@node1106:~# ceph -s
  cluster:
    id:     7f1aa879-afbb-4b19-9bc3-8f55c8ecbbb4
    health: HEALTH_WARN
            4 clients failing to respond to capability release
            3 MDSs report slow metadata IOs
            1 MDSs report slow requests
            3 MDSs behind on trimming
            noscrub,nodeep-scrub flag(s) set
            Reduced data availability: 25 pgs inactive
            6187 slow requests are blocked > 32 sec. Implicated osds 41


  services:
    mon: 3 daemons, quorum node1101,node1102,node1103
    mgr: node1103(active), standbys: node1102, node1101
    mds: ceph-3/3/3 up  {0=node1103=up:active,1=node1102=up:active,2=node1104=up:active}, 2 up:standby
    osd: 48 osds: 48 up, 48 in
         flags noscrub,nodeep-scrub


  data:
    pools:   6 pools, 2888 pgs
    objects: 474.95k objects, 94.5GiB
    usage:   267GiB used, 202TiB / 202TiB avail
    pgs:     0.866% pgs unknown
             2863 active+clean
             25   unknown


root@node1101:~# ceph pg dump_stuck inactive
ok
PG_STAT STATE   UP UP_PRIMARY ACTING ACTING_PRIMARY 
1.166   unknown []         -1     []             -1 
1.163   unknown []         -1     []             -1 
1.26f   unknown []         -1     []             -1 
1.228   unknown []         -1     []             -1 
1.213   unknown []         -1     []             -1 
1.12f   unknown []         -1     []             -1 
1.276   unknown []         -1     []             -1 
1.264   unknown []         -1     []             -1 
1.32a   unknown []         -1     []             -1 
1.151   unknown []         -1     []             -1 
1.20d   unknown []         -1     []             -1 
1.298   unknown []         -1     []             -1 
1.306   unknown []         -1     []             -1 
1.2f7   unknown []         -1     []             -1 
1.2c8   unknown []         -1     []             -1 
1.223   unknown []         -1     []             -1 
1.204   unknown []         -1     []             -1 
1.374   unknown []         -1     []             -1 
1.b5    unknown []         -1     []             -1 
1.b6    unknown []         -1     []             -1 
1.2b    unknown []         -1     []             -1 
1.9f    unknown []         -1     []             -1 
1.2ac   unknown []         -1     []             -1 
1.78    unknown []         -1     []             -1 
1.1c3   unknown []         -1     []             -1 
1.1a    unknown []         -1     []             -1 
1.d9    unknown []         -1     []             -1 

- 处理措施:

强制创建unkown pg,参考命令:ceph osd force-create-pg {pgid}
注:如需批量创建unkown pg,则参考命令如下:

for i in `ceph pg dump_stuck inactive | awk '{if (NR>2){print $1}}'`;do ceph osd force-create-pg $i;done

二、OSD相关

1、osd端口与其他服务固定绑定端口冲突

- 问题描述:

osd先行启动,占用其他服务固定绑定端口,导致其他服务绑定端口失败,无法启动

- 处理措施:

考虑到其他服务涉及组件太多,担心修改不完全导致其他问题发生,尝试修改osd启动端口范围为其他服务之外

  • 修改osd作为服务端的启动端口范围
    ceph可通过ms_bind_port_minms_bind_port_max参数限制osd和mds守护进程使用端口范围,默认范围为6800:7300
    设置端口使用范围为9600:10000,追加参数设置至/etc/ceph/ceph.conf文件中[global]字段内
[root@node111 ~]# cat /etc/ceph/ceph.conf | grep ms_bind_port
ms_bind_port_min = 9600
ms_bind_port_max = 10000
[root@node111 ~]#
[root@node111 ~]# ceph --show-config | grep ms_bind_port
ms_bind_port_max = 10000
ms_bind_port_min = 9600
  • 修改osd作为客户端的启动端口范围
    osd作为客户端的启动端口为随机分配的,可通过内核去限制随机端口分配范围
    默认端口范围为1024:65000,修改端口范围为7500:65000
--默认端口范围为1024:65000
[root@node111 ~]# cat /proc/sys/net/ipv4/ip_local_port_range 
1024    65000
--修改范围为7500:65000
[root@node111 ~]# sed -i 's/net.ipv4.ip_local_port_range=1024 65000/net.ipv4.ip_local_port_range=7500 65000/g' /etc/sysctl.conf 
[root@node111 ~]# sysctl -p

2、磁盘热插拔,osd无法上线

- 问题描述:

使用bluestore部署ceph集群,对osd所在磁盘进行热插拔操作,当重新插回之后,osd对应lvm不能自动恢复,导致osd无法上线成功
【ceph相关】ceph常见问题处理_第1张图片

- 处理措施:

  • 查找故障osd对应uuid:
ceph osd dump | grep {osd-id} | awk '{print $NF}'
参考示例:查找osd.0对应uuid
[root@node127 ~]# ceph osd dump | grep osd.0 | awk '{print $NF}'
57377809-fba4-4389-8703-f9603f16e60d
  • 查找故障osd对应lvm路径:
ls /dev/mapper/ | grep `ceph osd dump | grep {osd-id} | awk '{print $NF}' | sed 's/-/--/g'`
参考示例:通过uuid查找对应lvm路径
注:由于lvm路径对uuid做了处理,需要sed 's/-/--/g'`将-替换为--
[root@node127 ~]# ls /dev/mapper/ | grep `ceph osd dump | grep osd.0 | awk '{print $NF}' | sed 's/-/--/g'`
ceph--3182c42e--f8d8--4c13--ad92--987463d626c8-osd--block--57377809--fba4--4389--8703--f9603f16e60d
  • 删除故障osd对应lvm路径
dmsetup remove /dev/mapper/{lvm-path}
参考示例:删除故障osd对应lvm路径
[root@node127 ~]# dmsetup remove /dev/mapper/ceph--3182c42e--f8d8--4c13--ad92--987463d626c8-osd--block--57377809--fba4--4389--8703--f9603f16e60d 
  • 重新激活所有lvm卷组
    注:此时可以查看到对应故障osd的lvm信息
vgchange -ay
  • 重新启动osd使得osd上线
systemctl start ceph-volume@lvm-{osd-id}-{osd-uuid}

三、集群相关

1、clock skew detected on mon.node2

- 问题描述:

集群mon节点时间偏差过大,出现clock skew detected on mon.node2 告警信息

- 处理措施:

1、检查集群mon节点时间偏差,使用chronyd时间进行集群时间同步
2、调大集群参数阈值,调整mon_clock_drift_allowed 参数值为2,调整mon_clock_drift_warn_backoff 参数值为30

sed -i "2 amon_clock_drift_allowed = 2" /etc/ceph/ceph.conf
sed -i "3 amon_clock_drift_warn_backoff = 30" /etc/ceph/ceph.conf
ceph tell mon.* injectargs '--mon_clock_drift_allowed 2'
ceph tell mon.* injectargs '--mon_clock_drift_warn_backoff 30'

注:相关参数说明如下:

[root@node147 ~]# ceph --show-config | grep mon_clock_drift
mon_clock_drift_allowed = 0.050000
--当mon节点之间时间偏移超过0.05秒,则不正常
mon_clock_drift_warn_backoff = 5.000000
--当出现5次偏移,则上报告警

你可能感兴趣的:(存储相关,云计算,linux)