ceph(luminous)-Bluestore,ceph-objectstoe-tool工具

前言

本文主要介绍了一些ceph-objectstrore-tool的常用操作,主要从pg操作层面,osdmap操作层面和对象操作层面介绍的.
下面是help说明.

Allowed options:
  --help                      produce help message
  --type arg                  Arg is one of [bluestore, filestore (default),
                              memstore]
  --data-path arg             path to object store, mandatory
  --journal-path arg          path to journal, use if tool can't find it
  --pgid arg                  PG id, mandatory for info, log, remove, export,
                              export-remove, rm-past-intervals, mark-complete,
                              trim-pg-log, and mandatory for
                              apply-layout-settings if --pool is not specified
  --pool arg                  Pool name, mandatory for apply-layout-settings if
                              --pgid is not specified
  --op arg                    Arg is one of [info, log, remove, mkfs, fsck,
                              repair, fuse, dup, export, export-remove, import,
                              list, fix-lost, list-pgs, rm-past-intervals,
                              dump-journal, dump-super, meta-list, get-osdmap,
                              set-osdmap, get-inc-osdmap, set-inc-osdmap,
                              mark-complete, apply-layout-settings,
                              update-mon-db, dump-import, trim-pg-log]
  --epoch arg                 epoch# for get-osdmap and get-inc-osdmap, the
                              current epoch in use if not specified
  --file arg                  path of file to export, export-remove, import,
                              get-osdmap, set-osdmap, get-inc-osdmap or
                              set-inc-osdmap
  --mon-store-path arg        path of monstore to update-mon-db
  --fsid arg                  fsid for new store created by mkfs
  --target-data-path arg      path of target object store (for --op dup)
  --mountpoint arg            fuse mountpoint
  --format arg (=json-pretty) Output format which may be json, json-pretty,
                              xml, xml-pretty
  --debug                     Enable diagnostic output to stderr
    --force                     Ignore some types of errors and proceed with
                              operation - USE WITH CAUTION: CORRUPTION POSSIBLE
                              NOW OR IN THE FUTURE
  --skip-journal-replay       Disable journal replay
  --skip-mount-omap           Disable mounting of omap
  --head                      Find head/snapdir when searching for objects by
                              name
  --dry-run                   Don't modify the objectstore
  --namespace arg             Specify namespace when searching for objects


Positional syntax:

ceph-objectstore-tool ...  (get|set)-bytes [file]
ceph-objectstore-tool ...  set-(attr|omap)  [file]
ceph-objectstore-tool ...  (get|rm)-(attr|omap) 
ceph-objectstore-tool ...  get-omaphdr
ceph-objectstore-tool ...  set-omaphdr [file]
ceph-objectstore-tool ...  list-attrs
ceph-objectstore-tool ...  list-omap
ceph-objectstore-tool ...  remove|removeall
ceph-objectstore-tool ...  dump
ceph-objectstore-tool ...  set-size
ceph-objectstore-tool ...  remove-clone-metadata 

 can be a JSON object description as displayed
by --op list.
 can be an object name which will be looked up in all
the OSD's PGs.
 can be the empty string ('') which with a provided pgid
specifies the pgmeta object



  1. pg相关
    (1)list-pgs
    列出osd的所有pg
[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --op list-pgs --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore
6.a
6.22
6.18
6.16
4.2a
4.1e
4.36
6.10
6.9
6.37
4.27
4.b
4.25
4.1a
4.0
6.3d
6.34
6.2
4.5
4.3c
4.2f
4.16
4.1
4.30
4.15
4.31
4.17
4.8
4.37
4.3a
6.0

(2)info
显示pg的信息

[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --pgid 6.22 --op info --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore
{
    "pgid": "6.22",
    "last_update": "379'1162",
    "last_complete": "379'1162",
    "log_tail": "0'0",
    "last_user_version": 840,
    "last_backfill": "MAX",
    "last_backfill_bitwise": 0,
    "purged_snaps": [],
    "history": {
        "epoch_created": 193,
        "epoch_pool_created": 193,
        "last_epoch_started": 414,
        "last_interval_started": 413,
        "last_epoch_clean": 414,
        "last_interval_clean": 413,
        "last_epoch_split": 0,
        "last_epoch_marked_full": 0,
        "same_up_since": 413,
        "same_interval_since": 413,
        "same_primary_since": 193,
        "last_scrub": "379'1162",
        "last_scrub_stamp": "2018-10-12 00:22:33.984871",
        "last_deep_scrub": "379'1162",
        "last_deep_scrub_stamp": "2018-10-12 00:22:33.984871",
        "last_clean_scrub_stamp": "2018-10-12 00:22:33.984871"
    },
    "stats": {
        "version": "379'1162",
        "reported_seq": "6242",
        "reported_epoch": "381",
        "state": "peering",
        "last_fresh": "2018-10-11 10:47:37.628784",
        "last_change": "2018-10-11 10:47:36.545884",
        "last_active": "2018-10-11 10:47:36.545475",
        "last_peered": "2018-10-10 19:56:19.684867",
        "last_clean": "2018-10-10 19:56:19.684867",
        "last_became_active": "2018-10-09 17:07:33.567688",
        "last_became_peered": "2018-10-09 17:07:33.567688",
        "last_unstale": "2018-10-11 10:47:37.628784",
        "last_undegraded": "2018-10-11 10:47:37.628784",
        "last_fullsized": "2018-10-11 10:47:37.628784",
        "mapping_epoch": 413,
        "log_start": "0'0",
        "ondisk_log_start": "0'0",
        "created": 193,
        "last_epoch_clean": 379,
        "parent": "0.0",
        "parent_split_bits": 0,
        "last_scrub": "379'1162",
        "last_scrub_stamp": "2018-10-10 19:56:19.684833",
        "last_deep_scrub": "379'1162",
        "last_deep_scrub_stamp": "2018-10-10 19:56:19.684833",
        "last_clean_scrub_stamp": "2018-10-10 19:56:19.684833",
        "log_size": 1162,
        "ondisk_log_size": 1162,
        "stats_invalid": false,
        "dirty_stats_invalid": false,
        "omap_stats_invalid": false,
        "hitset_stats_invalid": false,
        "hitset_bytes_stats_invalid": false,
        "pin_stats_invalid": false,
        "snaptrimq_len": 0,
        "stat_sum": {
            "num_bytes": 1350565888,
            "num_objects": 322,
            "num_object_clones": 0,
            "num_object_copies": 644,
            "num_objects_missing_on_primary": 0,
            "num_objects_missing": 0,
            "num_objects_degraded": 0,
            "num_objects_misplaced": 0,
            "num_objects_unfound": 0,
            "num_objects_dirty": 322,
            "num_whiteouts": 0,
            "num_read": 0,
            "num_read_kb": 0,
            "num_write": 1421,
            "num_write_kb": 2379776,
            "num_scrub_errors": 0,
            "num_shallow_scrub_errors": 0,
            "num_deep_scrub_errors": 0,
            "num_objects_recovered": 1733,
            "num_bytes_recovered": 1585446912,
            "num_keys_recovered": 0,
            "num_objects_omap": 0,
            "num_objects_hit_set_archive": 0,
            "num_bytes_hit_set_archive": 0,
            "num_flush": 0,
            "num_flush_kb": 0,
            "num_evict": 0,
            "num_evict_kb": 0,
            "num_promote": 0,
            "num_flush_mode_high": 0,
            "num_flush_mode_low": 0,
            "num_evict_mode_some": 0,
            "num_evict_mode_full": 0,
            "num_objects_pinned": 0,
            "num_legacy_snapsets": 0,
            "num_large_omap_objects": 0
        },
        "up": [
            8,
            0
        ],
        "acting": [
            8,
            0
        ],
        "blocked_by": [],
        "up_primary": 8,
        "acting_primary": 8
    },
    "empty": 0,
    "dne": 0,
    "incomplete": 0,
    "last_epoch_started": 414,
    "hit_set_history": {
        "current_last_update": "0'0",
        "history": []
    }
}

(3)log
打印出当前pg的pglog,可以重定向到文件中查看.

[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --pgid 6.22 --op log --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore > pglog.txt
[root@test-1 ceph-objectstore-tool]# vim pglog.txt 
    1 {
    2     "pg_log_t": {
    3         "head": "379'1162",
    4         "tail": "0'0",
    5         "log": [
    6             {
    7                 "op": "modify",
    8                 "object": "6:4690ec49:::benchmark_data_test-2_1267539_object129:head",
    9                 "version": "290'1",
   10                 "prior_version": "0'0",
   11                 "reqid": "client.34918.0:130",
   12                 "extra_reqids": [],
   13                 "mtime": "2018-10-08 16:01:30.541217",
   14                 "return_code": 0,
   15                 "mod_desc": {
   16                     "object_mod_desc": {
   17                         "can_local_rollback": false,
   18                         "rollback_info_completed": false,
   19                         "ops": []
   20                     }
   21                 }
   22             },
.
.中间省略
.
.
19743             {
19744                 "op": "modify",
19745                 "object": "6:47fd92df:::benchmark_data_test-2_1392793_object10373:head",
19746                 "version": "379'1162",
19747                 "prior_version": "343'689",
19748                 "reqid": "osd.8.0:6135",
19749                 "extra_reqids": [],
19750                 "mtime": "0.000000",
19751                 "return_code": 0,
19752                 "mod_desc": {
19753                     "object_mod_desc": {
19754                         "can_local_rollback": false,
19755                         "rollback_info_completed": false,
19756                         "ops": []
19757                     }
19758                 }
19759             }
19760         ],
19761         "dups": []
19762     },
19763     "pg_missing_t": {
19764         "missing": [],
19765         "may_include_deletes": true
19766     }
19767 }


(4)export
导出pg为一个二进制文件.

[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --pgid 6.22 --op export --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore --file /home/tool/ceph-objectstore-tool/pg6.22
Exporting 6.22
Read #6:44032eb3:::benchmark_data_test-2_1392793_object19725:head#
Read #6:4406def7:::benchmark_data_test-2_1392793_object18137:head#
.
.
.
.
Read #6:47f9c22b:::benchmark_data_test-2_1392793_object20268:head#
Read #6:47fa1ea8:::benchmark_data_test-2_1392793_object5260:head#
Read #6:47fc6475:::benchmark_data_test-2_1392793_object11319:head#
Read #6:47fd92df:::benchmark_data_test-2_1392793_object10373:head#
Export successful
[root@test-1 ceph-objectstore-tool]# ll
total 1320204
-rw-r--r-- 1 root root       3466 Oct 12 16:00 objtool.txt
-rw-r--r-- 1 root root 1351124663 Oct 12 16:34 pg6.22
-rw-r--r-- 1 root root     757340 Oct 12 16:29 pglog.txt

(5)remove
删除一个osd内的pg,需要加–force才能删除.

[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --pgid 6.22 --op remove --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore  
Please use export-remove or you must use --force option
[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --pgid 6.22 --op remove --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore --force 
 marking collection for removal
setting '_remove' omap key
finish_remove_pgs 6.22_head removing 6.22
Remove successful

(6)import
从一个文件中把pg导入osd.

[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --pgid 6.22 --op import --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore --file /home/tool/ceph-objectstore-tool/pg6.22
Importing pgid 6.22
Write #6:44032eb3:::benchmark_data_test-2_1392793_object19725:head#
snapset 0=[]:{}
Write #6:4406def7:::benchmark_data_test-2_1392793_object18137:head#
snapset 0=[]:{}
Write #6:4409649e:::benchmark_data_test-2_1392793_object11369:head#
snapset 0=[]:{}
Write #6:4409b917:::benchmark_data_test-2_1392793_object19047:head#
.
.
.
.
Write #6:47f6ec56:::benchmark_data_test-2_1392793_object2565:head#
snapset 0=[]:{}
Write #6:47f9c22b:::benchmark_data_test-2_1392793_object20268:head#
snapset 0=[]:{}
Write #6:47fa1ea8:::benchmark_data_test-2_1392793_object5260:head#
snapset 0=[]:{}
Write #6:47fc6475:::benchmark_data_test-2_1392793_object11319:head#
snapset 0=[]:{}
Write #6:47fd92df:::benchmark_data_test-2_1392793_object10373:head#
snapset 0=[]:{}
Import successful

(4)(5)(6)可以联合使用用来修复无法通过scrub修复的pg.
(7)mark-complete
用于修改pg的状态,把incomplete的pg强制改为complete.

[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --pgid 6.22 --op mark-complete --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore 
Marking complete 
Marking complete succeeded

(8) trim-pg-log
如果pglog的长度大于osd_max_pg_log_entries了,就对pglog进行trim.

  1. osdmap相关
    主要就是get-osdmap,set-osdmap,get-inc-osdmap,set-inc-osdmap4个函数,用于拿到/替换osdmap/incosdmap,就像类似于filestore中current/meta目录下的内容一样,如果osdlog报错由于osdmap无法正常启动,可以尝试从正常的osd中拿osdmap来替换无法启动的osdmap.
[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --op get-osdmap  --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore --file ./osdmaplast
osdmap#418 exported 
##如果不+‘--epoch’就是取得最新的osdmap,incosdmap同理.
[root@test-1 ceph-objectstore-tool]# ceph-objectstore-tool --op get-osdmap --epoch 200 --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore --file ./osdmap200
osdmap#200 exported.
##添加epoch 就会拿到指定epoch的osdmap.
  1. 对象操作相关
    (1)list
    列举pg内的对象
[root@test-1 lqbz]# ceph-objectstore-tool --op list --pgid 10.5e --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore 
["10.5e",{"oid":"benchmark_data_test-1.novalocal_2041528_object10107","key":"","snapid":-2,"hash":3017920606,"max":0,"pool":10,"namespace":"","max":0}]
["10.5e",{"oid":"benchmark_data_test-1.novalocal_2041528_object3606","key":"","snapid":-2,"hash":3774810206,"max":0,"pool":10,"namespace":"","max":0}]
["10.5e",{"oid":"benchmark_data_test-1.novalocal_2041528_object10321","key":"","snapid":-2,"hash":1194600542,"max":0,"pool":10,"namespace":"","max":0}]
["10.5e",{"oid":"benchmark_data_test-1.novalocal_2041528_object1121","key":"","snapid":-2,"hash":1660721246,"max":0,"pool":10,"namespace":"","max":0}]
.
.
.
["10.5e",{"oid":"benchmark_data_test-1.novalocal_2041528_object1703","key":"","snapid":-2,"hash":3044276190,"max":0,"pool":10,"namespace":"","max":0}]
["10.5e",{"oid":"benchmark_data_test-1.novalocal_2041528_object10448","key":"","snapid":-2,"hash":239562718,"max":0,"pool":10,"namespace":"","max":0}]
["10.5e",{"oid":"benchmark_data_test-1.novalocal_2041528_object7254","key":"","snapid":-2,"hash":1379041246,"max":0,"pool":10,"namespace":"","max":0}]

(2) (get|set)-bytes
下载(上传)对象,并保存为文件.

[root@test-1 lqbz]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore benchmark_data_test-1.novalocal_2041528_object14236 get-bytes obj.dat 
//把对象名为下载到工作目录的obj.dat文件中.

(3) list-attrs
列举对象的属性,原先filestore把attr都保存再xfs的attr中了,bluestore脱离了xfs,需要通过这个工具拿到attr.

[root@test-1 lqbz]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore benchmark_data_test-1.novalocal_2041528_object14236 list-attrs
_
snapset

(4) (get|rm)-(attr|omap)
用于获取(删除)对象的attr或者omap,可以导出为文件用ceph-dencoder解码得到可读性的属性.

[root@test-1 lqbz]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore benchmark_data_test-1.novalocal_2041528_object14236 get-attr _
Base64:EQgsAQAABANUAAAAAAAAADMAAABiZW5jaG1hcmtfZGF0YV90ZXN0LTEubm92YWxvY2FsXzIwNDE1Mjhfb2JqZWN0MTQyMzb+/15JBu4AAAAAAAoAAAAAAAAABgMcAAAACgAAAAAAAAD/AAAAAAAAAAD//wAAAAByAAAAAAAAABwDAAAAAAAAAAAAAAAAAAACAhUAAAAIK1oCAAAAAACdNwAAAAAAAAAAAAAAAEAAAAAAADuMzlv0MD8AAgIVAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcgAAAAAAAAAAAAAAAAAAAAAUAAAAQIzOW/IHLyVfMvDR/wAAQAAAAAAAAABAAAAAAAA1AAAA
//刚才listattr中有“_”和"snapset"两个属性,本例中获取第一个属性,其值为base64编码,需要用ceph-dencoder解码.

(5)dump
dump操作可以展示对象的一些信息.

[root@test-1 lqbz]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore benchmark_data_test-1.novalocal_2041528_object14236 dump
{
    "id": {
        "oid": "benchmark_data_test-1.novalocal_2041528_object14236",
        "key": "",
        "snapid": -2,
        "hash": 3993389406,
        "max": 0,
        "pool": 10,
        "namespace": "",
        "max": 0
    },
    "info": {
        "oid": {
            "oid": "benchmark_data_test-1.novalocal_2041528_object14236",
            "key": "",
            "snapid": -2,
            "hash": 3993389406,
            "max": 0,
            "pool": 10,
            "namespace": ""
        },
        "version": "796'114",
        "prior_version": "0'0",
        "last_reqid": "client.154155.0:14237",
        "user_version": 114,
        "size": 4194304,
        "mtime": "2018-10-23 10:49:31.004141",
        "local_mtime": "2018-10-23 10:49:36.623839",
        "lost": 0,
        "flags": [
            "dirty",
            "data_digest"
        ],
        "legacy_snaps": [],
        "truncate_seq": 0,
        "truncate_size": 0,
        "data_digest": "0xd1f0325f",
        "omap_digest": "0xffffffff",
        "expected_object_size": 4194304,
        "expected_write_size": 4194304,
        "alloc_hint_flags": 53,
        "manifest": {
            "type": 0,
            "redirect_target": {
                "oid": "",
                "key": "",
                "snapid": 0,
                "hash": 0,
                "max": 0,
                "pool": -9223372036854775808,
                "namespace": ""
            }
        },
        "watchers": {}
    },
    "stat": {
        "size": 4194304,
        "blksize": 4096,
        "blocks": 1024,
        "nlink": 1
    },
    "SnapSet": {
        "snap_context": {
            "seq": 0,
            "snaps": []
        },
        "head_exists": 1,
        "clones": []
    }
}

(6)remove
remove操作可以删除此对象

[root@test-1 lqbz]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore benchmark_data_test-1.novalocal_2041528_object14236 remove
remove #10:7a926077:::benchmark_data_test-1.novalocal_2041528_object14236:head#
[root@test-1 lqbz]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --type bluestore benchmark_data_test-1.novalocal_2041528_object14236 dump
No object id 'benchmark_data_test-1.novalocal_2041528_object14236' found or invalid JSON specified

结束语

主要整理了一些常用的,还有一些操作没有说明,平时用到的不是很多,后续用到了会补上.

你可能感兴趣的:(ceph,ceph,bluestore)