hadoop集群及单节点安装教程

1、hadoop下载

1、安装jdk参考：jdk下载
2、下载hadoop，原理同1

2、环境修改

#vi /etc/hostname
分别修改三台主机名为：hadoop1、hadoop2、hadoop3

添加环境变量：
HADOOP_HOME=/usr/local/hadoop/hadoop-3.2.0
PATH=$HADOOP_HOME/bin

#vi /etc/hosts
登录服务将三台服务信息添加进host中
192.168.101.88 hadoop01
192.168.101.89 hadoop02
192.168.101.90 hadoop03

设置ssh免密登录：ssh教程
注：此处为集群，但是即使是单节点hadoop安装也需要配置ssh，否则node无法启动

3、修改配置文件

创建文件夹：

mkdir  /usr/local/hadoop/tmp
mkdir  /usr/local/hadoop/var
mkdir  /usr/local/hadoop/dfs
mkdir  /usr/local/hadoop/dfs/name
mkdir  /usr/local/hadoop/dfs/data

#/usr/local/hadoop/hadoop-3.2.0/etc/hadoop
core-site.xml

 
        hadoop.tmp.dir
        /usr/local/hadoop/tmp
   
   
        fs.default.name
        hdfs://hadoop01:9000
   

hadoop-env.sh

将文件中的export   JAVA_HOME= 修改为自己的JAVA_HOME

hdfs-site.xml

    
        dfs.name.dir
        /usr/local/hadoop/dfs/name
    
    
        dfs.data.dir
        /usr/local/hadoop/dfs/data
    
    
        dfs.replication
        2
    
    
        dfs.permissions
        false
    
    
        dfs.namenode.http.address
        hadoop01:50070
    

mapred-site.xml

    
        mapred.job.tracker
        hadoop01:49001
    
    
        mapred.local.dir
        /usr/local/hadoop/var
    
    
        mapreduce.framework.name
        yarn
    

workers:
此文件只需要在hadoop1服务器添加

hadoop01
hadoop02
hadoop03

yarn-site.xml

    
        yarn.resourcemanager.hostname
        hadoop01
    
    
        yarn.resourcemanager.address
        ${yarn.resourcemanager.hostname}:8032
    
    
        yarn.resourcemanager.scheduler.address
        ${yarn.resourcemanager.hostname}:8030
    
    
        yarn.resourcemanager.webapp.address
        ${yarn.resourcemanager.hostname}:8088
    
    
        yarn.resourcemanager.webapp.https.address
        ${yarn.resourcemanager.hostname}:8090
    
    
        yarn.resourcemanager.resource-tracker.address
        ${yarn.resourcemanager.hostname}:8031
    
    
        yarn.resourcemanager.admin.address
        ${yarn.resourcemanager.hostname}:8033
    
    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    
    
        yarn.scheduler.maximum-allocation-mb
        1024
    
    
        yarn.nodemanager.vmem-pmem-ratio
        2.1
    
    
        yarn.nodemanager.resource.memory-mb
        2048
    
    
        yarn.nodemanager.vmem-check-enabled
        false
    

将/usr/local/hadoop文件夹复制到各个节点/usr/local/目录下

4、启动

hadoop1机器的namenode初始化(hadoop2和hadoop3不需要此步骤)
cd /usr/local/hadoop/hadoop-3.2.0/bin
./hadoop namenode -format
格式化成功后，/usr/local/hadoop/dfs/name/目录多了一个current目录
启动执行命令
./usr/local/hadoop/hadoop-3.2.0/sibn/start-all.sh
验证
访问浏览器，输入网址：http://192.168.101.88:8088/
成功显示hadoop的Nodes of the cluster页面。正常情况会显示3个nodes。

5、单节点

单节点部署只需要把配置文件中的workers只保留本机，同时配置ssh即可