HDFS常用命令整理

简介：

• HDFS 文件系统提供了相当多的shell 操作命令，大大方便了程序员和系统管理人员查看、修改HDFS 上的文件。进一步，HDFS 的操作命令和Unix/Linux 的命令名称和格式相当一致，因而学习HDFS 命令的成本也大为缩小。

1.命令帮助

[hadoop@hadoop000 ~]$ hdfs dfs
Usage: hadoop fs [generic options]
        [-appendToFile  ... ]
        [-cat [-ignoreCrc]  ...]
        [-checksum  ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R]  PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-copyFromLocal [-f] [-p] [-l] [-d]  ... ]
        [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc]  ... ]
        [-count [-q] [-h] [-v] [-t []] [-u] [-x]  ...]
        [-cp [-f] [-p | -p[topax]] [-d]  ... ]
        [-createSnapshot  []]
        [-deleteSnapshot  ]
        [-df [-h] [ ...]]
        [-du [-s] [-h] [-x]  ...]
        [-expunge]
        [-find  ...  ...]
        [-get [-f] [-p] [-ignoreCrc] [-crc]  ... ]
        [-getfacl [-R] ]
        [-getfattr [-R] {-n name | -d} [-e en] ]
        [-getmerge [-nl] [-skip-empty-file]  ]
        [-help [cmd ...]]
        [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [ ...]]
        [-mkdir [-p]  ...]
        [-moveFromLocal  ... ]
        [-moveToLocal  ]
        [-mv  ... ]
        [-put [-f] [-p] [-l] [-d]  ... ]
        [-renameSnapshot   ]
        [-rm [-f] [-r|-R] [-skipTrash] [-safely]  ...]
        [-rmdir [--ignore-fail-on-non-empty]  ...]
        [-setfacl [-R] [{-b|-k} {-m|-x } ]|[--set  ]]
        [-setfattr {-n name [-v value] | -x name} ]
        [-setrep [-R] [-w]   ...]
        [-stat [format]  ...]
        [-tail [-f] ]
        [-test -[defsz] ]
        [-text [-ignoreCrc]  ...]
        [-touchz  ...]
        [-truncate [-w]   ...]
        [-usage [cmd ...]]
# 注：hdfs dfs等同于hadoop fs

2.文件操作命令案例

hdfs dfs -copyFromLocal /local/data /hdfs/data：将本地文件上传到 hdfs 上（原路径只能是一个文件）
hdfs dfs -put /tmp/ /hdfs/ ：和 copyFromLocal 区别是，put 原路径可以是文件夹等
hadoop fs -ls / ：查看根目录文件
hadoop fs -ls /tmp/data：查看/tmp/data目录
hadoop fs -cat /tmp/a.txt ：查看 a.txt，与 -text 一样
hadoop fs -mkdir dir：创建目录dir
hadoop fs -get /aaa/jdk.tar.gz :等同于copyToLocal，就是从hdfs下载文件到本地
hadoop fs -df -h / :统计文件系统的可用空间信息
hadoop fs -cp /aaa/jdk.tar.gz /bbb/jdk.tar.gz.2 :从hdfs的一个路径拷贝hdfs的另一个路径
hadoop fs -mv /aaa/jdk.tar.gz / :在hdfs目录中移动文件
hadoop fs -du -s -h /aaa/* :统计文件夹的大小信息
hadoop fs -rm -r /aaa/bbb/ :删除文件或文件夹
hadoop fs -rmdir /aaa/bbb/ccc :删除目录
hadoop fs -tail /weblog/access_log.1 :显示一个文件的末尾

3.管理命令

# 命令帮助
[hadoop@hadoop000 ~]$ hdfs dfsadmin
Usage: hdfs dfsadmin
Note: Administrative commands can only be run as the HDFS superuser.
        [-report [-live] [-dead] [-decommissioning]]
        [-safemode ]
        [-saveNamespace]
        [-rollEdits]
        [-restoreFailedStorage true|false|check]
        [-refreshNodes]
        [-setQuota  ...]
        [-clrQuota ...]
        [-setSpaceQuota  [-storageType ] ...]
        [-clrSpaceQuota [-storageType ] ...]
        [-finalizeUpgrade]
        [-rollingUpgrade []]
        [-refreshServiceAcl]
        [-refreshUserToGroupsMappings]
        [-refreshSuperUserGroupsConfiguration]
        [-refreshCallQueue]
        [-refresh   [arg1..argn]
        [-reconfig   ]
        [-printTopology]
        [-refreshNamenodes datanode_host:ipc_port]
        [-deleteBlockPool datanode_host:ipc_port blockpoolId [force]]
        [-setBalancerBandwidth ]
        [-getBalancerBandwidth ]
        [-fetchImage ]
        [-allowSnapshot ]
        [-disallowSnapshot ]
        [-shutdownDatanode  [upgrade]]
        [-evictWriters ]
        [-getDatanodeInfo ]
        [-metasave filename]
        [-triggerBlockReport [-incremental] ]
        [-help [cmd]]

hdfs dfsadmin -report :显示所有dataNode
hdfs dfsadmin -safemode leave :离开安全模式