大数据入门与实战-HDFS的常用操作

1 命令

1.1 dfs命令

bin/hdfs dfs
Usage: hadoop fs [generic options]
    [-appendToFile  ... ]
    [-cat [-ignoreCrc]  ...]
    [-checksum  ...]
    [-chgrp [-R] GROUP PATH...]
    [-chmod [-R]  PATH...]
    [-chown [-R] [OWNER][:[GROUP]] PATH...]
    [-copyFromLocal [-f] [-p] [-l] [-d]  ... ]
    [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc]  ... ]
    [-count [-q] [-h] [-v] [-t []] [-u] [-x]  ...]
    [-cp [-f] [-p | -p[topax]] [-d]  ... ]
    [-createSnapshot  []]
    [-deleteSnapshot  ]
    [-df [-h] [ ...]]
    [-du [-s] [-h] [-x]  ...]
    [-expunge]
    [-find  ...  ...]
    [-get [-f] [-p] [-ignoreCrc] [-crc]  ... ]
    [-getfacl [-R] ]
    [-getfattr [-R] {-n name | -d} [-e en] ]
    [-getmerge [-nl] [-skip-empty-file]  ]
    [-help [cmd ...]]
    [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [ ...]]
    [-mkdir [-p]  ...]
    [-moveFromLocal  ... ]
    [-moveToLocal  ]
    [-mv  ... ]
    [-put [-f] [-p] [-l] [-d]  ... ]
    [-renameSnapshot   ]
    [-rm [-f] [-r|-R] [-skipTrash] [-safely]  ...]
    [-rmdir [--ignore-fail-on-non-empty]  ...]
    [-setfacl [-R] [{-b|-k} {-m|-x } ]|[--set  ]]
    [-setfattr {-n name [-v value] | -x name} ]
    [-setrep [-R] [-w]   ...]
    [-stat [format]  ...]
    [-tail [-f] ]
    [-test -[defsz] ]
    [-text [-ignoreCrc]  ...]
    [-touchz  ...]
    [-truncate [-w]   ...]
    [-usage [cmd ...]]

上面的命令的作用基本上和linux文件操作等命令的效果差不多

文件夹创建与删除

上传文件

1.2 常用启动关闭命令

• sbin/hadoop-daemon.sh start namenode:

启动namenode

• sbin/hadoop-daemon.sh start datanode

启动datanode

• sbin/yarn-daemon.sh start resourcemanager

启动resourcemanager

• sbin/yarn-daemon.sh start nodemanager

启动nodemanager

• sbin/mr-jobhistory-daemon.sh start historyserver

启动historyserver

• jps

查看进程

与start对应的stop为停止相关的进程，比如sbin/hadoop-daemon.sh stop namenode

1.3常用的Web UI地址

• 文件操作系统
  http://192.168.31.135:50070/dfshealth.html#tab-datanode
  
• 监控集群任务运行状况
  http://192.168.31.135:8088/cluster/apps
  
• 历史任务管理器
  http://192.168.31.135:19888/jobhistory/app
  

1.4 其他命令

• bin/yarn jar

执行jar文件
例如下面的词频统计实例：
bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /user/root/mapreduce/input/news.txt /user/root/mapreduce/output/

2 配置文件

2.1 yarn-site.xml

     
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    
    
    
        yarn.resourcemanager.hostname
        quincyqiang.com
    
    
    
        yarn.log-aggregation-enable
        True
    
    
    
        yarn.log-aggregation.retain-seconds
        604800
    

• yarn.log-aggregation-enable：打开日志聚合
• yarn.log-aggregation.retain-seconds：日志保留时间