HDFS fsimage文件解析与反解析

参考：
http://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html
https://blog.csdn.net/androidlushangderen/article/details/50937069

目录

• 概述
• 使用
  • 命令使用说明
  • XMl Processor
  • FileDistribution Processor
  • Delimited Processor

概述

HDFS中解析fsImage的工具是 Offline Image Viewer ，对应的命令是hdfs oiv。Offline Image Viewer提供了几下几种处理器：

• XML：将fsimage文件中的所有内容解析为XML格式的文件，该处理器的输出可以通过XML工具进行自动化处理和分析。由于XML语法的冗长，该处理器生成的输出文件非常大。
• FileDistribution：分析namespace image中的文件大小分布的一个工具
• Web：启动一个HTTP服务，对外暴露一个只读的WebHDFS API。不支持secure 模式。
• Delimited（experimental）：生成一个文本文件，其中包含inodes-under-construction和inode所共有的所有元素，用分隔符分隔，默认分隔符是\t，但可以通过-delimiter参数更改。
• ReverseXML（experimental）：对应XML processor，将XML格式的文件反解析为FSImage。这个处理器可以很容易地创建用于测试的fsimages，并在fsimage出现损坏时手动编辑fsimages。

使用

命令使用说明

$ hdfs oiv -h
Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE
Offline Image Viewer
View a Hadoop fsimage INPUTFILE using the specified PROCESSOR,
saving the results in OUTPUTFILE.

The oiv utility will attempt to parse correctly formed image files
and will abort fail with mal-formed image files.

The tool works offline and does not require a running cluster in
order to process an image file.

The following image processors are available:
  * XML: This processor creates an XML document with all elements of
    the fsimage enumerated, suitable for further analysis by XML
    tools.
  * ReverseXML: This processor takes an XML file and creates a
    binary fsimage containing the same elements.
  * FileDistribution: This processor analyzes the file size
    distribution in the image.
    -maxSize specifies the range [0, maxSize] of file sizes to be
     analyzed (128GB by default).
    -step defines the granularity of the distribution. (2MB by default)
    -format formats the output result in a human-readable fashion
     rather than a number of bytes. (false by default)
  * Web: Run a viewer to expose read-only WebHDFS API.
    -addr specifies the address to listen. (localhost:5978 by default)
    It does not support secure mode nor HTTPS.
  * Delimited (experimental): Generate a text file with all of the elements common
    to both inodes and inodes-under-construction, separated by a
    delimiter. The default delimiter is \t, though this may be
    changed via the -delimiter argument.

Required command line arguments:
-i,--inputFile    FSImage or XML file to process.

Optional command line arguments:
-o,--outputFile   Name of output file. If the specified
                       file exists, it will be overwritten.
                       (output to stdout by default)
                       If the input file was an XML file, we
                       will also create an .md5 file.
-p,--processor    Select which type of processor to apply
                       against image file. (XML|FileDistribution|
                       ReverseXML|Web|Delimited)
                       The default is Web.
-delimiter        Delimiting string to use with Delimited processor.  
-t,--temp         Use temporary dir to cache intermediate result to generate
                       Delimited outputs. If not set, Delimited processor constructs
                       the namespace in memory before outputting text.
-h,--help              Display usage information and exit

  选项说明：

• -i,–inputFile 必选项，指定FSImage或者XML文件
• -o,–outputFile 可选项，指定输出文件，默认是stdout
• -p,–processor 可选项，指定processor，默认是Web processor
• -delimiter 可选项，用于使用Delimited processor时，指定输出字符串的分隔符
• -t,–temp 可选项，用于使用Delimited processor时，指定临时目录缓存输出。

XMl Processor

  使用示例

bin/hdfs oiv -p XML -i fsimage_0000000000007905322 -o fsimage.xml

  文件内容截取如下：

-641Unknown
5855296211000304682010740433787895152


5RS-10-4-1024k1048576DISABLED
rs104

...
...


219664253934
16385DIRECTORY1556176976929hadoop:hadoop:07559223372036854775807-1
...

...
...

  从XML文件中可以看到fsimage中包含了几类section

• 命名空间类Section, 包括namespaceId, rollingUpgradeStartTime等类型的变量.
• INode相关Section,包含了文件,目录相关inode的信息
• fileUnderConstructionSection正在构建中的文件信息.
• SnapShot快照相关信息.
• SecretManager安全管理相关信息
• CacheManager缓存管理相关信息

FileDistribution Processor

  使用示例

hdfs oiv -i fsimage_0000000000007905322 -p FileDistribution -format
Processed 0 inodes.
Size Range      NumFiles
[0 B, 0 B]      863
(0 B, 2 MB]     36238
(2 MB, 4 MB]    3072
(4 MB, 6 MB]    1780
(6 MB, 8 MB]    573
(8 MB, 10 MB]   70
(10 MB, 12 MB]  922
(12 MB, 14 MB]  33
(14 MB, 16 MB]  10
...
...
totalFiles = 49974
totalDirectories = 4000
totalBlocks = 54112
totalSpace = 6468980524520
maxFileSize = 19466558832

Delimited Processor

  使用示例

hdfs oiv -i current/fsimage_0000000000007961778 -p Delimited 
Path    Replication     ModificationTime        AccessTime      PreferredBlockSize      BlocksCount     FileSize        NSQUOTA DSQUOTA Permission      UserName   GroupName
/       0       2019-04-25 15:22        1970-01-01 08:00        0       0       0       9223372036857     -1      drwxr-xr-x      hadoop  hadoop
/tmp    0       2019-07-15 16:25        1970-01-01 08:00        0       0       0       -1      -1      drwxrwxrwx      hadoop  hadoop
...