Hadoop总结(更新)

面试题：hadoop组成
hadoop1.X MapReduce(计算+资源调度) HDFS(数据存储)
hadoop2.X MapReduce(计算) Yarn(资源调度) HDFS(数据存储)

HDFS 有一个ResourceManager+一个NameNode+若干个(几台单点服务器)NodeManager

网络配置：vim /etc/sysconfig/network-scripts/ifcfg-ens33

如何理解：
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
## Allows people in group wheel to run all commands
%wheel ALL=(ALL) ALL
atguigu ALL=(ALL) NOPASSWD:ALL

集群配置

NameNode、SecondNameNode、ResourceManager三个都很耗费资源，装在三台不同机器上。
NameNode内部通讯hadoop102:8020
NameNode web 端访问地址 hadoop102:9870
RescoureManager web hadoop103:8088
SecondaryNameNode web 端hadoop104:9868

默认配置文件 core-default.xml、hdfs-default.xml、yarn-default.xml、mapred-default.xml
自定义配置文件：core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml

core-site.xml
     
    <property> 
        <name>fs.defaultFSname> 
        <value>hdfs://hadoop102:8020value> 
    property>
     
    <property> 
        <name>hadoop.tmp.dirname> 
        <value>/opt/module/hadoop-3.1.3/datavalue> 
property> 
     
    <property> 
        <name>hadoop.http.staticuser.username> 
        <value>tmvalue> 
property> 

hdfs-site.xml
   
  <property> 
        <name>dfs.namenode.http-addressname> 
        <value>hadoop102:9870value> 
    property> 
   
    <property> 
        <name>dfs.namenode.secondary.http-addressname> 
        <value>hadoop104:9868value> 
    property> 

yarn-site.xml
     
    <property> 
        <name>yarn.nodemanager.aux-servicesname> 
        <value>mapreduce_shufflevalue> 
    property> 
 
     
    <property> 
        <name>yarn.resourcemanager.hostnamename> 
        <value>hadoop103value> 
    property> 
 
     
    <property> 
        <name>yarn.nodemanager.env-whitelistname> 
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOMEvalue> 
    property> 

mapred-site.xml
   
    <property> 
        <name>mapreduce.framework.namename> 
        <value>yarnvalue> 
    property>