Hadoop(四)

目录

Hadoop部署

一、配置core-site.xml。代理主机

二、HDFS配置文件

三、YARN配置

四、MapReduce配置文件

五、配置workers

配置历史服务器

一、配置mapred-site.xml

配置日志的聚集

---

Hadoop部署

一、配置core-site.xml。代理主机

[doudou@hadoop101 module]$ cd hadoop-3.1.3/

[doudou@hadoop101 hadoop-3.1.3]$ cd etc/hadoop/

[doudou@hadoop101 hadoop]$ cd core-site.xml

[doudou@hadoop101 hadoop]$ vim /core-site.xml

  
  
     fs.defaultFS
     hdfs://hadoop102:8020
  
  
     hadoop.tmp.dir
     /opt/module/hadoop-3.1.3/data
  
  
     hadoop.http.staticuser.user
     doudou
  
  
       hadoop.proxyuser.doudou.hosts
       *
  
  
     hadoop.proxyuser.doudou.grups
     *
  
  
       hadoop.proxyuser.doudou.users
         *
  

二、HDFS配置文件

配置hdfs-site.xml

[doudou@hadoop101 hadoop]$ vim hdfs-site.xml 

  
   dfs.namenode.http-address
   hadoop101:9870
  
  
    dfs.namenode.secondary.http-address
    hadoop103:9868
  
  
     dfs.replication
     1
  


~                         

分发

[doudou@hadoop101 hadoop]$ xsync hdfs-site.xml 

三、YARN配置

配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml

   
    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    

    
    
        yarn.resourcemanager.hostname
        hadoop103
    

    
    
        yarn.nodemanager.env-whitelist
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    
   
       yarn.scheduler.minimum-allocation-mb
       512
   
   
       yarn.scheduler.maximum-allocation-mb
       4096
   
   
        yarn.nodemanager.resource.memory-mb
        4096
   
   
      yarn.nodemanager.vmem-check-enabled
      false
   

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml

四、MapReduce配置文件

配置mapred-site.xml

[doudou@hadoop101 hadoop]$ vim mapred-site.xml 

	
    
        mapreduce.framework.name
        yarn
    

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml 

五、配置workers

[doudou@hadoop101 ~]$ vim /opt/module/hadoop-3.1.3/etc/hadoop/workers

hadoop101
hadoop102
hadoop103

[doudou@hadoop101 hadoop]$ xsync workers 

配置历史服务器

一、配置mapred-site.xml

[doudou@hadoop101 hadoop]$ vim mapred-site.xml 

    mapreduce.jobhistory.webapp.address
    hadoop101:19888

>mapreduce.framework.name
        yarn
    

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml 

配置日志的聚集

一、配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml 

    yarn.log-aggregation-enable
    true


  
    yarn.log.server.url  
    http://hadoop101:19888/jobhistory/logs



    yarn.log-aggregation.retain-seconds
    604800

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml