Hadoop fsck命令

今天在安装CDH的时候遇到了一些错误，典型一个关于HDFS的错误如下，

Bad : 255 missing blocks in the cluster. 266 total blocks in the cluster. Percentage missing blocks: 95.86%. Critical threshold: any.

网上查看这个问题的解决办法一般都是用Hadoop命令fsck，我也确实用这个命令解决了问题，解决方法的命令如下，

hdfs fsck / -delete 

不过虽然知道可以用这种方法解决，对内在的原理倒不是很清楚，所以专门了解了一下fsck命令的用法。

语法

hdfs fsck  [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]
              检查这个目录中的文件是否完整
-move               破损的文件移至/lost+found目录
-delete             删除破损的文件
-openforwrite       打印正在打开写操作的文件
-files              打印正在check的文件名
-blocks             打印block报告 （需要和-files参数一起使用）
-locations          打印每个block的位置信息（需要和-files参数一起使用）
-racks              打印位置信息的网络拓扑图 （需要和-files参数一起使用）

例子

bash-4.1$ hdfs fsck -h
Usage: DFSck  [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]
          start checking from this path
        -move   move corrupted files to /lost+found
        -delete delete corrupted files
        -files  print out files being checked
        -openforwrite   print out files opened for write
        -includeSnapshots       include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
        -list-corruptfileblocks print out list of missing blocks and files they belong to
        -blocks print out block report
        -locations      print out locations for every block
        -racks  print out network topology for data-node locations

        -blockId        print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)

Please Note:
        1. By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually  tagged CORRUPT or HEALTHY depending on their block allocation status
        2. Option -includeSnapshots should not be used for comparing stats, should be used only for HEALTH check, as this may contain duplicates if the same file present in both original fs tree and inside snapshots.

Generic options supported are
-conf file>     specify an application configuration file
-D value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]


bash-4.1$ hdfs fsck /
Connecting to namenode via http://n12.trafodion.local:50070
FSCK started by hdfs (auth:SIMPLE) from /10.10.11.11 for path / at Thu Mar 16 13:23:49 CST 2017
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
.........................Status: HEALTHY
 Total size:    28617859656 B (Total open files size: 332 B)
 Total dirs:    11866
 Total files:   5925
 Total symlinks:                0 (Files currently being written: 5)
 Total blocks (validated):      3830 (avg. block size 7472026 B) (Total open file blocks (not validated): 4)
 Minimally replicated blocks:   3830 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.9874673
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Thu Mar 16 13:23:49 CST 2017 in 215 milliseconds


The filesystem under path '/' is HEALTHY