【大数据】分布式集群部署

1、集群规划部署

节点名称	NN1	NN2	DN	RM	NM
hadoop01	NameNode		DataNode		NodeManager
hadoop02		SecondaryNameNode	DataNode	ResourceManager	NodeManager
hadoop03			DataNode		NodeManager

 2、参考单机部署，拷贝安装目录至相同目录，使用ln -s 建立软连接

 

 

3、修改配置文件参数及sh启动文件--根据集群规划部署配置

 

slaves：记录了机器名

*.sh：修改JAVA_HOME

yarn-site.xml 

          
        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        
          
        
                yarn.resourcemanager.hostname
                hadoop02
        

hdfs-site.xml 

         
        
                dfs.replication
                3
        
    
        dfs.permissions
        false
        
            If "true", enable permission checking in HDFS.
            If "false", permission checking is turned off,
            but all other behavior is unchanged.
            Switching from one parameter value to the other does not change the mode,
            owner or group of files or directories.
        
    
    
    
        dfs.namenode.secondary.http-address
        hadoop02:50090
    

mapred-site.xml 

          
        
                mapreduce.framework.name
                yarn
        

 

core-site.xml

        
        
                fs.defaultFS
                hdfs://hadoop01:9000
        
          
        
                hadoop.tmp.dir
                /hadoop/tmp
        

 

4、由于是在单机基础上升级扩展，需要删除hadoop.tmp.dir目录文件，并用root授权 chmod 777 -R /hadoop

5、重新格式化：hdfs namenode -foamate

6、配置拷贝：scp -r /home/hadoop/Soft/hadoop-2.7.6/etc/hadoop hadoop@hadoop03:/home/hadoop/Soft/hadoop-2.7.6/etc/

7、Hadoop01：start-dfs.sh

8、Hadoop02：start-yarn.sh

10、使用jps查看进程

 

 

参考：

https://blog.csdn.net/frank409167848/article/details/80968531

https://www.cnblogs.com/frankdeng/p/9047698.html

转载于:https://www.cnblogs.com/defineconst/p/10982576.html