ceph-objectstore-tool工具使用详解

文章目录

      • 简介
      • 使用
        • OSD相关操作
        • PG相关操作
        • 对象相关操作
      • 总结

简介

ceph-objectstore-tool工具,能够操作到ceph最底层的数据,包括pg,对象层级。它能够对底层pg以及对象相关数据进行获取、修改。并能够对一些问题pg和对象进行简单修复。所以使用该工具进行操作的时候需要谨慎(涉及到修改的操作最好备份一份数据),防止误操作造成数据丢失。
该工具的使用主要针对对象PG,所以这里我们的使用主要为以下两种

  • PG的相关操作
    info #查看pg的信息
    log  #查看pg的元数据信息
    remove #从当前osd移除pg
    mkfs #重新初始化osd
    fsck #检查bluefs数据正确性
    export #导出pg元数据
    import #导入pg元数据
    list #列出pg内部对象或者当前osd所有对象
    fix-lost #修复pg丢失的对象
    list-pgs #列出当前osd所有pg
    rm-past-intervals 
    dump-journal #针对filestore,打印日志
    dump-super #打印osd的超级块信息
    meta-list #打印元数据信息列表
    get-osdmap #获取osdmap
    set-osdmap #设置osd map
    get-inc-osdmap #从当前osd 获取inc 信息
    set-inc-osdmap #将inc设置进入当前osd信息
    mark-complete #标记为complete,让bluestore认为可以pg可以选举出权威日志
    
  • 对象的相关操作,这里主要列出常用的对象操作
     list-attrs  #列出对象的一些属性
     list-omap #列出对象的omp信息
     remove|removeall #移除对象或者移除所有对象
     dump #打印对象元数据信息
    

使用

基本命令使用如下:
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore xxx一般前半部分如左,指定osd路径,指定存储引擎(bluestore或者filestore

PS:使用之前需要停止当前操作的osd,否则会报错

OSD相关操作

  • 检查bluestore的bluefs是否文件系统被损坏,并尝试修复
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op fsck

  • 列出当前osd所有对象
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op list

  • 获取当前osd的osdmap信息,并指定对应的输出文件
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op get-osdmap --file 1.txt

    [root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op get-osdmap --file 1.txt
    osdmap#1457 exported.
    
  • 导入一个osdmap信息到当前osd
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op set-osdmap --file 1.txt

    [root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op set-osdmap --file 1.txt
    Wrote osdmap.145
    
  • 获取当前osd的超级块信息
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op dump-super 输出如下:

    {
    "cluster_fsid": "fa27f041-0ce9-4df1-a4bd-5e37678834bd",
    "osd_fsid": "c03c2fdb-cfd4-42bd-8e32-61e359281078",
    "whoami": 8,
    "current_epoch": 1457,
    "oldest_map": 784,
    "newest_map": 1457,
    "weight": 0.000000,
    "compat": {
        "compat": {},
        "ro_compat": {},
        "incompat": {
            "feature_1": "initial feature set(~v.18)",
            "feature_2": "pginfo object",
            "feature_3": "object locator",
            "feature_4": "last_epoch_clean",
            "feature_5": "categories",
            "feature_6": "hobjectpool",
            "feature_7": "biginfo",
            "feature_8": "leveldbinfo",
            "feature_9": "leveldblog",
            "feature_10": "snapmapper",
            "feature_11": "sharded objects",
            "feature_12": "transaction hints",
            "feature_13": "pg meta object",
            "feature_14": "explicit missing set",
            "feature_15": "fastinfo pg attr",
            "feature_16": "deletes in missing set"
        }
    },
    "clean_thru": 1457,
    "last_epoch_mounted": 1456
    }
    
  • 获取当前osd的所有pg
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op list-pgs 其中--op表示对当前osd或者当前pg的操作

    17.es1
    17.as0
    17.8s1
    17.4s2
    17.3s1
    17.3fs0
    17.3ds0
    17.3cs2
    17.23s2
    17.2es0
    

PG相关操作

  • 针对单个pg,列出当前pg的基本信息
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op info输出如下
    "pgid": "17.es1",
    "last_update": "1380'5116",
    "last_complete": "1380'5116",
    "log_tail": "1376'4854",
    "last_user_version": 5118,
    "last_backfill": "MAX",
    "last_backfill_bitwise": 0,
    "purged_snaps": [],
    "history": {
        "epoch_created": 1290,
        "epoch_pool_created": 1290,
        "last_epoch_started": 1457,
        "last_interval_started": 1456,
        "last_epoch_clean": 1350,
        "last_interval_clean": 1349,
        "last_epoch_split": 0,
        "last_epoch_marked_full": 0,
        "same_up_since": 1456,
        "same_interval_since": 1456,
        "same_primary_since": 1290,
        "last_scrub": "0'0",
        "last_scrub_stamp": "2019-08-12 11:14:04.515869",
        "last_deep_scrub": "0'0",
        "last_deep_scrub_stamp": "2019-08-12 11:14:04.515869",
        "last_clean_scrub_stamp": "2019-08-12 11:14:04.515869"
        ...
    
  • 针对单个pg,列出当前pg元数据信息
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op log
    该命令的输出主要包括pg的历史版本信息,记录着每个对象的相关操作,以及当时pg epoch和对应的pg小版本
    "pg_log_t": {
       "head": "1380'5116",
       "tail": "1376'4854",
       "log": [
           {
               "op": "modify",
               "object": "17:713bbf48:::rbd_data.18.cf83c74b0dc51.0000000000023ba0:head",
               "version": "1376'4855",
               "prior_version": "0'0",
               "reqid": "client.852676.0:164001",
               "extra_reqids": [],
               "mtime": "2019-08-12 14:40:08.571459",
               "return_code": 0,
               "mod_desc": {
                   "object_mod_desc": {
                       "can_local_rollback": true,
                       "rollback_info_completed": true,
                       "ops": [
                           {
                               "code": "CREATE"
                           }
                       ]
                   }
               }
           },
           ...
    
  • 打印pg中对象的元数据
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op meta-list
    ["meta",{"oid":"osdmap.1372","key":"","snapid":0,"hash":168878088,"max":0,"pool":-1,"namespace":"","max":0}]
    ["meta",{"oid":"osdmap.1053","key":"","snapid":0,"hash":168945672,"max":0,"pool":-1,"namespace":"","max":0}]
    ["meta",{"oid":"osdmap.1266","key":"","snapid":0,"hash":168892424,"max":0,"pool":-1,"namespace":"","max":0}]
    ["meta",{"oid":"osdmap.1101","key":"","snapid":0,"hash":168920072,"max":0,"pool":-1,"namespace":"","max":0}]
    ...
    
  • 将pg的元数据导出到指定文件,一般用作 备份
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op export --file /mnt/test.obj
  • 将指定的pg元数据导入到当前pg
    这里导入前需要先移除当前pg(remove之前请先export备份一下pg数据)
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op remove
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op import --file /mnt/test.obj
  • 将集群中incomplete的pg标记为complete,这里需要根据集群的副本数将相关的所有osd上的pg进行标记
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op mark-complete

对象相关操作

  • 列出当前pg的所有对象
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op list
    输出如下:
    ["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.000000000001bda5","key":"","snapid":-2,"hash":1725693966,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0}]
    ["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.0000000000033137","key":"","snapid":-2,"hash":1782448142,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0}]
    ["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.000000000003f128","key":"","snapid":-2,"hash":1246789646,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0
    ...
    ...
    
  • 从以上获取的对象中取一个,打印对象的元数据信息
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a dump
    {
    "id": {
        "oid": "rbd_data.18.cf83c74b0dc51.000000000003925a",
        "key": "",
        "snapid": -2,
        "hash": 1962672078,
        "max": 0,
        "pool": 17,
        "namespace": "",
        "shard_id": 1,
        "max": 0
    },
    "info": {
        "oid": {
            "oid": "rbd_data.18.cf83c74b0dc51.000000000003925a",
            "key": "",
            "snapid": -2,
            "hash": 1962672078,
            "max": 0,
            "pool": 17,
            "namespace": ""
        },
        ...
    
    • 获取对象的属性信息
      ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore
       [root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a list-attrs
      _
      hinfo_key
      snapset
      
  • 删除对象
    ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove请慎重测试,如有pg备份,则可以进行测试
    [root@node1 ceph]# ceph-objectstore-tool-bak --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove
    remove 1#17:73ffdf2e:::rbd_data.18.cf83c74b0dc51.000000000003925a:head#
    
    查看pg中是否还有该对象
    ceph-objectstore-tool-bak --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op list|grep rbd_data.18.cf83c74b0dc51.000000000003925a并无输出

总结

由于该工具是对Ceph底层数据以及元数据进行操作,所以使用需谨慎,处处要备份。如有兴趣,可以对该工具的具体操作源码进行阅读
src/tools/ceph_objectstore_tool.cc路径,修改对应代码之后需要同时将libceph-common.so.0以及libceph-common.so.0拷贝到测试设备才能够正常使用

你可能感兴趣的:(分布式存储:ceph技能树,#,ceph运维:常用工具)