hadoop-2.6.0-cdh5.7.1搭建

一、配置ssh、hosts、静态ip、hostname，java环境

二、解压 tar  -zxvf  hadoop，mv改名，配置环境变量bin和sbin

     1.hadoop  version验证一下

三、创建数据存储目录

${HAFDOOP_HOME}根据自己的路径写

1. ${HAFDOOP_HOME}/data/dfs/name
2. ${HAFDOOP_HOME}/data/dfs/data
3. ${HAFDOOP_HOME}/tmp
4. ${HAFDOOP_HOME}/tmp/logs
5. ${HAFDOOP_HOME}/logs

四、修改配置文件

1.vim ${HAFDOOP_HOME}/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/software/jdk1.8.0_191
export HADOOP_HOME=/usr/local/hadoop-2.6.0

2. vim ${HAFDOOP_HOME}/etc/hadoop/core-site.xml

    fs.defaultFS
    hdfs://master:9000



    hadoop.tmp.dir
    /usr/local/hadoop-2.6.0/tmp

 

3.vim ${HAFDOOP_HOME}/etc/hadoop/hdfs-site.xml

 

        
        
                dfs.namenode.http-address
                master:50070
        
        
        
                dfs.namenode.secondary.http-address
                master:50090
        
        
        
                dfs.namenode.name.dir
                /usr/local/hadoop-2.6.0/data/dfs/name
        
        
                dfs.datanode.data.dir
                /usr/local/hadoop-2.6.0/data/dfs/data
        
        
        
                dfs.replication
                3
        

 

4.vim ${HAFDOOP_HOME}/etc/hadoop/mapred-site.xml

        
        
                mapreduce.framework.name
                yarn
        
        
                mapreduce.application.classpath
                
                        $HADOOP_HOME/etc/hadoop,
                        $HADOOP_HOME/share/hadoop/common/*,
                        $HADOOP_HOME/share/hadoop/common/lib/*,
                        $HADOOP_HOME/share/hadoop/hdfs/*,
                        $HADOOP_HOME/share/hadoop/hdfs/lib/*,
                        $HADOOP_HOME/share/hadoop/mapreduce/*,
                        $HADOOP_HOME/share/hadoop/mapreduce/lib/*,
                        $HADOOP_HOME/share/hadoop/yarn/*,
                        $HADOOP_HOME/share/hadoop/yarn/lib/*
                
        
         
        
                mapreduce.jobhistory.address
                master:10020
        
         
        
                mapreduce.jobhistory.webapp.address
                master:19888
        

5.vim ${HAFDOOP_HOME}/etc/hadoop/yarn-site.xml

        
		yarn.resourcemanager.hostname
		master
	
	
		yarn.nodemanager.aux-services
		mapreduce_shuffle
	
	
		yarn.nodemanager.aux-services.mapreduce.shuffle.class
		org.apache.hadoop.mapred.ShuffleHandler
	
	
		RM中应用程序管理器接口的地址。
		yarn.resourcemanager.address
		master:8032
	
	
		调度程序接口的地址。
		yarn.resourcemanager.scheduler.address
		master:8030
	
	
		RM web应用程序的http地址.
		yarn.resourcemanager.webapp.address
		master:8088
	
	
		RM web应用程序的https adddress.
		yarn.resourcemanager.webapp.https.address
		master:8090
	
	
		yarn.resourcemanager.resource-tracker.address
		master:8031
	
	
		RM admin接口的地址
		yarn.resourcemanager.admin.address
		master:8033
	
	
		yarn.nodemanager.local-dirs
		/usr/local/hadoop-2.6.0/yarn/local
	
	
		yarn.log-aggregation-enable
		true
	
	
		yarn.nodemanager.remote-app-log-dir
		/usr/local/hadoop-2.6.0/tmp/logs
	
	
		yarn.log.server.url
		http://master:19888/jobhistory/logs
	
	
		yarn.nodemanager.vmem-check-enabled
		false
	

6.配置slaves

node1
node2

五、分发，并配置环境变量生效

六、格式化namenode，只能格式化一次，得到集群号。

hdfs namenode -format

七、启动集群

start-all.sh
stop-all.sh

 

 

遇到的错误：

1、曾经配置过hadoop3.1.1集群，确定删除干净，启动时报错

Error: Cannot find configuration directory: /usr/local/hadoop-3.1.1/etc/hadoop

解决方法：

配置环境变量
export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.0/etc/hadoop
或者
source hadoop-env.sh (不错)