RocksDB系列四:MANIFEST

  RocksDB对文件系统和存储介质是不可知的。文件系统的操作不是原子的,所以很有可能在系统故障时导致不一致。即使打开了日志记录功能,文件系统也不能在unclean restart时保证一致性。POSIX文件系统也不支持批量操作的原子性。所以,在RocksDB重启时,不能依靠存储在RocksDB data store file中的元信息来重建启动前的一致性状态。
  RocksD有一个内建的机制来克服POSIX文件系统的各种限制,这种机制就是通过一个MANIFEST文件记录RocksDB状态改变的所有事物日志。所以,MANIFEST文件可以在DB重启时恢复到最近一次的一致性状态。

  • MANIFEST
    记录RocksDB状态改变的所有transction log的一种机制
  • Manifest log
    一个单独的文件,记录了RocksDB状态的snapshot和edit log
  • CURRENT
    最新最近的manifest log

how does it work?

  MANIFEST是RocksDB状态变更的transction log 记录。MANIFEST包含 manifest log文件和最新的manifest 文件指针。Manifest logs是滚动的日子文件,命名为MANIFEST-(序列号,递增)。CURRENT是一个特殊的文件用于指定latest manifest log 文件。
  在系统启动或者重启时,latest manifest log文件包含了RocksDB的一致性状态。RocksDB后续的所有变更记录都写入这个manifest log 文件。当一个manifest log 文件超过了固定大小时,一个新的manifest文件会被创建,这个新的文件是基于当前DB的snapshot。CURRENT中的最新manifest 文件指针会更新,同时sync到文件系统中。一旦成功写入CURRENT file,冗余的manifest log文件会被清除。

MANIFEST = { CURRENT, MANIFEST-* } 
CURRENT = File pointer to the latest manifest log
MANIFEST- = Contains snapshot of RocksDB state and subsequent modifications

Version Edit

  RocksDB在特定时间的特定状态会关联一个version。针对version的任何改动都被视为一个version edit。version(or RocksDB state snapshot)是通过join 一连串的version-edit构建起来。总之,一个manifest log文件是一连串的version edits。

version-edit      = Any RocksDB state change
version           = { version-edit* }
manifest-log-file = { version, version-edit* }
                  = { version-edit* }

Version Edit Layout

   Manifest log是一串version edit记录。version edit 记录的类型是通过 edit number指定。

1、Data Types

简单数据类型

VarX   - Variable character encoding of intX
FixedX - Fixed character encoding of intX

复杂数据类型

String - Length prefixed string data
+-----------+--------------------+
| size (n)  | content of string  |
+-----------+--------------------+
|<- Var32 ->|<-- n            -->|

2、Version Edit Record Format

version edit record 使用下面的格式,解码器通过Record ID来识别记录类型。

+-------------+------ ......... ----------+
| Record ID   | Variable size record data |
+-------------+------ .......... ---------+
<-- Var32 --->|<-- varies by type       -->

3、Version Edit Record Types and Layout

根据对RocksDB的状态变更的不同,有不同类型的edit record。

  • Comparator edit record
Captures the comparator name

+-------------+----------------+
| kComparator | data           |
+-------------+----------------+
<-- Var32 --->|<-- String   -->|
  • Log number edit record
Latest WAL log file number

+-------------+----------------+
| kLogNumber  | log number     |
+-------------+----------------+
<-- Var32 --->|<-- Var64    -->|
  • Previous File Number edit record
Previous manifest file number

+------------------+----------------+
| kPrevFileNumber  | log number     |
+------------------+----------------+
<-- Var32      --->|<-- Var64    -->|
  • Next File Number edit record
Next manifest file number

+------------------+----------------+
| kNextFileNumber  | log number     |
+------------------+----------------+
<-- Var32      --->|<-- Var64    -->|
  • Last Sequence Number edit record
Last sequence number of RocksDB

+------------------+----------------+
| kLastSequence    | log number     |
+------------------+----------------+
<-- Var32      --->|<-- Var64    -->|
  • Max Column Family edit record
Adjust the maximum number of family columns allowed.

+---------------------+----------------+
| kMaxColumnFamily    | log number     |
+---------------------+----------------+
<-- Var32         --->|<-- Var32    -->|
  • Deleted File edit record
Mark a file as deleted from database.

+-----------------+-------------+--------------+
| kDeletedFile    | level       | file number  |
+-----------------+-------------+--------------+
<-- Var32     --->|<-- Var32 -->|<-- Var64  -->|
  • 新文件的edit record
    表明这个文件是新添加到数据库中的,同时提供RocksDB元信息。
    a、File edit record with compaction information
+--------------+-------------+--------------+------------+----------------+--------------+----------------+----------------+
| kNewFile4    | level       | file number  | file size  | smallest_key   | largest_key  | smallest_seqno | largest_seq_no |
+--------------+-------------+--------------+------------+----------------+--------------+----------------+----------------+
|<-- var32  -->|<-- var32 -->|<-- var64  -->|<-  var64 ->|<-- String   -->|<-- String -->|<-- var64    -->|<-- var64    -->|

+-----------+---------------+-------+------------------+-------+--------------+
|kPathID ---| Path size(n)  | path  | kNeedCompaction  | 1     | value (0/1)  |
+-----------+---------------+-------+------------------+-------+--------------+
<- var32  ->|<-- var32   -->|<- n ->|<-- var32      -->|<- 1 ->|<-- 1      -->|

  b、File edit record backward compatible

+--------------+-------------+--------------+------------+----------------+--------------+----------------+----------------+
| kNewFile2    | level       | file number  | file size  | smallest_key   | largest_key  | smallest_seqno | largest_seq_no |
+--------------+-------------+--------------+------------+----------------+--------------+----------------+----------------+
<-- var32   -->|<-- var32 -->|<-- var64  -->|<-  var64 ->|<-- String   -->|<-- String -->|<-- var64    -->|<-- var64    -->|

  c、File edit record with path information

+--------------+-------------+--------------+-------------+-------------+----------------+--------------+
| kNewFile3    | level       | file number  | Path ID     | file size   | smallest_key   | largest_key  |
+--------------+-------------+--------------+-------------+-------------+----------------+--------------+
|<-- var32  -->|<-- var32 -->|<-- var64  -->|<-- var32 -->|<-- var64 -->|<-- String   -->|<-- String -->|
+----------------+----------------+
| smallest_seqno | largest_seq_no |
+----------------+----------------+
<-- var64     -->|<-- var64    -->|
  • Column family status edit record
Note the status of column family feature (enabled/disabled)

+------------------+----------------+
| kColumnFamily    | 0/1            |
+------------------+----------------+
<-- Var32      --->|<-- Var32    -->|
  • Column family add edit record
Add a column family

+---------------------+----------------+
| kColumnFamilyAdd    | cf name        |
+---------------------+----------------+
<-- Var32         --->|<-- String   -->|
  • Column family drop edit record
Drop all column family

+---------------------+
| kColumnFamilyDrop   |
+---------------------+
<-- Var32         --->|

你可能感兴趣的:(RocksDB系列四:MANIFEST)