ceph（luminous）-Bluestore，ceph-kvstore-tool工具说明
前言
此工具主要是作用是针对ceph中各种kv存储进行操作和展示的工具.
先看一下help列表
[root@test-1 ceph]# ceph-kvstore-tool --help
Usage: ceph-kvstore-tool command [args...]
Commands:
list [prefix] //列出所有key，可以指定prefix
list-crc [prefix] //列出crc
exists [key] //判断某个key是否存在
get [out ] //获取具体key的value，保存到文件中
crc //获取crc
get-size [ ] //获取db的大小或者某个kv的大小
set [ver |in ] //设置kv
rm //删除kv
rm-prefix //删除指定前缀的kv组
store-copy [num-keys-per-tx] //拷贝kv
store-crc //store的crc
compact //合并kv
compact-prefix
compact-range
基本上所有的操作都需要把进程停止了以后再进行，因为rocksdb或者leveldb有一个文件锁，每次打开db之前都会给文件上锁，如果文件已经被锁上了（osd或者mon进程还在），就无法再次进行上锁（ceph-kvstore-tool进程还需要对db上锁）.
数据库主要分为3类，leveldb和rocksdb都是可配的，bluestore-kv是bluestore下专用的kv存储，其内部也是rocksdb。
- list
可以列出数据库的所有key值
下面列出了bluestore-key
[root@test-1 crushmap]# ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-0/ list > key.list
2018-10-23 10:58:25.645999 7fcd99345ec0 1 bluestore(/var/lib/ceph/osd/ceph-0/) _mount path /var/lib/ceph/osd/ceph-0/
.
.
.
//列出的key如下
B blocks //B是prefix，blocks是key
B blocks_per_key
B bytes_per_block
B size
C 10.0_head// C是prefix，表示collection的意思.
C 10.10_head
C 10.11_head
C 10.12_head
C 10.1e_head
C 10.20_head
C 10.21_head
C 10.25_head
C 10.29_head
C 10.2a_head
C 10.2d_head
C 10.2e_head
C 10.2f_head
C 10.32_head
C 10.33_head
C 10.36_head
C 10.38_head
C 10.3c_head
C 10.3d_head
C 10.42_head
bluestore-kv中prefix的含义:
S* superblock block size、configure options，整个存储的核心数据
B* block allocation： freeblock、block that used
C* collection name -> cnode_t
O对象名存储到ONODE的映射关系
L wal 写前日志
M元数据信息
const string PREFIX_SUPER = “S”; // field -> value
const string PREFIX_STAT = “T”; // field -> value(int64 array)
const string PREFIX_COLL = “C”; // collection name -> cnode_t
const string PREFIX_OBJ = “O”; // object name -> onode_t
const string PREFIX_OMAP = “M”; // u64 + keyname -> value
const string PREFIX_DEFERRED = “L”; // id -> deferred_transaction_t
const string PREFIX_ALLOC = “B”; // u64 offset -> u64 length (freelist)
const string PREFIX_SHARED_BLOB = “X”; // u64 offset -> shared_blob_t
omap中保存了pglog，pglog中的***epoch_为小端存储，65 1c 00 00 = 7269
下面列出了mon-rocksdb的key
[root@test-1 crushmap]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db/ list > mon.list
2018-10-23 11:03:48.824528 7f9fef12cec0 4 rocksdb: RocksDB version: 5.4.0
2018-10-23 11:03:48.824571 7f9fef12cec0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
.
.
.
- list-crc
显示kv的crc - exists
判断key是否存在
[root@test-1 ceph]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db/ exists monmap
2018-10-23 11:08:09.189204 7f5611b27ec0 4 rocksdb: RocksDB version: 5.4.0
2018-10-23 11:08:09.189247 7f5611b27ec0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-10-23 11:08:09.189250 7f5611b27ec0 4 rocksdb: Compile date Sep 20 2018
.
.
.
2018-10-23 11:08:09.201440 7f5611b27ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540264089201431, "job": 1, "event": "recovery_finished"}
2018-10-23 11:08:09.201987 7f5611b27ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_files.cc:314] [JOB 2] Tried to delete a non-existing file /var/lib/ceph/mon/ceph-test-1/store.db///MANIFEST-024885 type=3 #24885 -- IO error: /var/lib/ceph/mon/ceph-test-1/store.db///MANIFEST-024885: No such file or directory
2018-10-23 11:08:09.202022 7f5611b27ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_files.cc:314] [JOB 2] Tried to delete a non-existing file /var/lib/ceph/mon/ceph-test-1/store.db//024886.log type=0 #24886 -- IO error: /var/lib/ceph/mon/ceph-test-1/store.db//024886.log: No such file or directory
2018-10-23 11:08:09.210464 7f5611b27ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7f5614128000
(monmap, ) exists //存在prefix为monmap
2018-10-23 11:08:09.211004 7f5611b27ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2018-10-23 11:08:09.211307 7f5611b27ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:343] Shutdown complete
- get
从kv数据库中获取某个指定key的value，并输出为文件.可以用于导出osdmap，monmap时使用.
[root@test-1 crushmap]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db/ get monmap 9 out monmap9 //把prefix为monmap key为9的monmap（epoch为9的monmap输出为文件 monmap9）
2018-10-23 11:27:56.087672 7fa96ec0eec0 4 rocksdb: RocksDB version: 5.4.0
.
.
.
2018-10-23 11:27:56.103530 7fa96ec0eec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_files.cc:314] [JOB 2] Tried to delete a non-existing file /var/lib/ceph/mon/ceph-test-1/store.db//024898.log type=0 #24898 -- IO error: /var/lib/ceph/mon/ceph-test-1/store.db//024898.log: No such file or directory
2018-10-23 11:27:56.111757 7fa96ec0eec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7fa970420000
(monmap, 9)
2018-10-23 11:27:56.112429 7fa96ec0eec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2018-10-23 11:27:56.112698 7fa96ec0eec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:343] Shutdown complete
[root@test-1 crushmap]# monmaptool --print monmap9
monmaptool: monmap file monmap9
epoch 9
fsid acc6dc6a-79cd-45dc-bf1f-83a576eb8039
last_changed 2018-10-22 16:32:01.468845
created 2018-09-29 17:21:07.728600
0: 192.168.125.109:6789/0 mon.test-1
1: 192.168.125.110:6789/0 mon.test-2
2: 192.168.125.111:6789/0 mon.test-3
-
crc
获取指定prefix下key的crc -
get-size
获取指定prefix下key的大小,或者获取整个db的大小. -
set
设置指定prefix下key的value，可以通过文件方式或者变量赋值.
[root@test-1 crushmap]# ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-test-1/store.db set monmap 9 in monmap9 //通过文件方式设置变量
2018-10-23 11:39:23.720162 7fd3b7176ec0 4 rocksdb: RocksDB version: 5.4.0
2018-10-23 11:39:23.720204 7fd3b7176ec0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-10-23 11:39:23.720207 7fd3b7176ec0 4 rocksdb: Compile date Sep 20 2018
.
.
.
2018-10-23 11:39:23.723255 7fd3b7176ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540265963723231, "job": 1, "event": "recovery_started", "log_files": [24919]}
2018-10-23 11:39:23.723279 7fd3b7176ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:482] Recovering log #24919 mode 2
2018-10-23 11:39:23.728876 7fd3b7176ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540265963728847, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 24920, "file_size": 1147, "table_properties": {"data_size": 199, "index_size": 28, "filter_size": 18, "raw_key_size": 16, "raw_average_key_size": 16, "raw_value_size": 402, "raw_average_value_size": 402, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2018-10-23 11:39:23.728944 7fd3b7176ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/version_set.cc:2395] Creating manifest 24921
2018-10-23 11:39:23.744083 7fd3b7176ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1540265963744074, "job": 1, "event": "recovery_finished"}
2018-10-23 11:39:23.753370 7fd3b7176ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7fd3b9210000
2018-10-23 11:39:23.757713 7fd3b7176ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2018-10-23 11:39:23.758026 7fd3b7176ec0 4 rocksdb: [/root/rpmbuild/BUILD/ceph-luminous/src/rocksdb/db/db_impl.cc:343] Shutdown complete
-
rm
删除指定prefix下的kv值. -
rm-prefix
删除指定prefix -
store-copy
拷贝kvstore，只能拷贝为同类型的kv,如果是bluestore-kv想导出为rocksdb可以使用ceph-bluestore-tool进行导出. -
store-crc
显示store的crc -
compact
合并kv，由于rocksdb和leveldb的特性都是一直追加写log文件，等到log文件到了一定大小再找出相同key值进行合并，此操作可以强制kvdb进行合并操作，一般合并后kvdb大小会减小. -
compact-prefix
合并指定prefix的kv -
compact-range
合并指定范围的kv