HDFS工具DsfAdmin和FsShell

使用HDFS时两个常用工具是多任务工具DfsAdmin和文件系统Shell命令。

FsShell
Hadoop文件系统Shell命令可以执行其他 文件系统中常见的操作，如读取文件、移动文件、创建目录、删除数据等等。在终端上可以通过下面命令，获得Shell命令的详细帮助信息，

[hdfs@cent-2 ~]$ hadoop fs -help
Usage: hadoop fs [generic options]
        [-appendToFile  ... ]
        [-cat [-ignoreCrc]  ...]
        [-checksum  ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R] ... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-copyFromLocal [-f] [-p] [-l]  ... ]
        [-copyToLocal [-p] [-ignoreCrc] [-crc]  ... ]
        [-count [-q] [-h]  ...]
        [-cp [-f] [-p | -p[topax]]  ... ]
        [-createSnapshot  []]
        [-deleteSnapshot  ]
        [-df [-h] [ ...]]
        [-du [-s] [-h]  ...]
        [-expunge]
        [-get [-p] [-ignoreCrc] [-crc]  ... ]
        [-getfacl [-R] ]
        [-getfattr [-R] {-n name | -d} [-e en] ]
        [-getmerge [-nl]  ]
        [-help [cmd ...]]
        [-ls [-d] [-h] [-R] [ ...]]
        [-mkdir [-p]  ...]
        [-moveFromLocal  ... ]
        [-moveToLocal  ]
        [-mv  ... ]
        [-put [-f] [-p] [-l]  ... ]
        [-renameSnapshot   ]
        [-rm [-f] [-r|-R] [-skipTrash]  ...]
        [-rmdir [--ignore-fail-on-non-empty]  ...]
        [-setfacl [-R] [{-b|-k} {-m|-x } ]|[--set  ]]
        [-setfattr {-n name [-v value] | -x name} ]
        [-setrep [-R] [-w]   ...]
        [-stat [format]  ...]
        [-tail [-f] ]
        [-test -[defsz] ]
        [-text [-ignoreCrc]  ...]
        [-touchz  ...]
        [-usage [cmd ...]]
...

“hadoop fs”命令还可以在本地文件系统和HDFS之间进行文件复制，copyFromLocal表示将本地文件复制到HDFS上。
FsShell是一个Java程序，实现了应用入口的main()方法，是一个典型的基于ToolRunner实现的应用。

DfsAdmin
DfsAdmin继承自FsShell，它的实现和FsShell类似，也是通过ToolRunner.run()执行DFSAdmin.run()方法，该方法根据不同的命令调用相应的处理函数。可以通过以下命令获取帮助信息，

[hdfs@cent-2 ~]$ hadoop dfsadmin
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Usage: hdfs dfsadmin
Note: Administrative commands can only be run as the HDFS superuser.
        [-report [-live] [-dead] [-decommissioning]]
        [-safemode ]
        [-saveNamespace]
        [-rollEdits]
        [-restoreFailedStorage true|false|check]
        [-refreshNodes]
        [-setQuota  ...]
        [-clrQuota ...]
        [-setSpaceQuota  ...]
        [-clrSpaceQuota ...]
        [-finalizeUpgrade]
        [-rollingUpgrade []]
        [-refreshServiceAcl]
        [-refreshUserToGroupsMappings]
        [-refreshSuperUserGroupsConfiguration]
        [-refreshCallQueue]
        [-refresh   [arg1..argn]
        [-reconfig ...>  ]
        [-printTopology]
        [-refreshNamenodes datanode_host:ipc_port]
        [-deleteBlockPool datanode_host:ipc_port blockpoolId [force]]
        [-setBalancerBandwidth in bytes per second>]
        [-fetchImage ]
        [-allowSnapshot ]
        [-disallowSnapshot ]
        [-shutdownDatanode  [upgrade]]
        [-getDatanodeInfo ]
        [-metasave filename]
        [-triggerBlockReport [-incremental] ]
        [-help [cmd]]

Generic options supported are
-conf      specify an application configuration file
-D             use value for given property
-fs       specify a namenode
-jt     specify a ResourceManager
-files     specify comma separated files to be copied to the map reduce cluster
-libjars     specify comma separated jar files to include in the classpath.
-archives     specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

一个典型的应用是使用如下命令在新增节点或注销节点时使用，

[hdfs@cent-2 ~]$ hadoop dfsadmin -refreshNodes
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Refresh nodes successful

另外使用如下命令查看Hadoop集群的各节点的状态，

[hdfs@cent-2 ~]$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Configured Capacity: 54380585780 (50.65 GB)
Present Capacity: 27026104320 (25.17 GB)
DFS Remaining: 26990800896 (25.14 GB)
DFS Used: 35303424 (33.67 MB)
DFS Used%: 0.13%
Under replicated blocks: 270
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (2):

Name: 192.168.0.47:50010 (cent-2.novalocal)
Hostname: cent-2.novalocal
Rack: /default
Decommission Status : Normal
Configured Capacity: 27190292890 (25.32 GB)
DFS Used: 17649664 (16.83 MB)
Non DFS Used: 8935477658 (8.32 GB)
DFS Remaining: 18237165568 (16.98 GB)
DFS Used%: 0.06%
DFS Remaining%: 67.07%
Configured Cache Capacity: 568328192 (542 MB)
Cache Used: 0 (0 B)
Cache Remaining: 568328192 (542 MB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 8
Last contact: Tue Dec 13 14:31:22 CST 2016


Name: 192.168.0.16:50010 (cent-1.novalocal)
Hostname: cent-1.novalocal
Rack: /default
Decommission Status : Normal
Configured Capacity: 27190292890 (25.32 GB)
DFS Used: 17653760 (16.84 MB)
Non DFS Used: 18419003802 (17.15 GB)
DFS Remaining: 8753635328 (8.15 GB)
DFS Used%: 0.06%
DFS Remaining%: 32.19%
Configured Cache Capacity: 758120448 (723 MB)
Cache Used: 0 (0 B)
Cache Remaining: 758120448 (723 MB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 8
Last contact: Tue Dec 13 14:31:21 CST 2016