Mac上搭建Hadoop环境(2) — Hadoop下载及安装

@[TOC]

前言

在Mac上搭建Hadoop环境(1) — 虚拟机的安装及SSH免密设置中，我们已经配置好了节点之间基本的网络环境，接下来，只要在master(MBP)上下载安装hadoop即可。

下载hadoop

可以前往Apache Hadoop官网，选择你想要的版本进行下载。这里，我选择下载的是hadoop-2.7.7的binary版本。

安装hadoop

在下载完hadoop-2.7.7.tar.gz后，将其解压到你想安装的目录即可。我将其解压到 ==/opt== 文件夹下。

sudo tar -C /opt -xvf ~/Downloads/hadoop-2.7.7.tar.gz;

sudo mv /opt/hadoop-2.7.7 /opt/hadoop;
mkdir /opt/hadoop/dfs;
mkdir /opt/hadoop/dfs/name;
mkdir /opt/hadoop/dfs/data;
mkdir /opt/hadoop/tmp;

配置hadoop

hadoop-env.sh

编辑 /opt/hadoop/etc/hadoop/hadoop-env.sh，设置JAVA_HOME值，如下

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home

yarn-env.sh

编辑 /opt/hadoop/etc/hadoop/yarn-env.sh，设置JAVA_HOME值，如下

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home

slaves

编辑 /opt/hadoop/etc/hadoop/slaves，加入之前设置的slave节点的hostname：

slave1
slave2

更新*.xml配置文件

core-site.xml


       
                fs.defaultFS
                hdfs://master:8020
       
       
                io.file.buffer.size
                131072
        
       
               hadoop.tmp.dir
               file:/opt/hadoop/tmp
               Abase for other temporary directories.
       
        
               hadoop.proxyuser.lestat.hosts
               *
       
       
               hadoop.proxyuser.lestat.groups
               *

hdfs-site.xml


       
                dfs.namenode.secondary.http-address
               master:9001
       
     
             dfs.namenode.name.dir
             file:/opt/hadoop/dfs/name
       
      
              dfs.datanode.data.dir
              file:/opt/hadoop/dfs/data
       
       
               dfs.replication
               3
        
        
                 dfs.webhdfs.enabled
                  true

mapred-site.xml


                                                                        
          mapreduce.framework.name
                yarn
           
          
                  mapreduce.jobhistory.address
                  master:10020
          
          
                mapreduce.jobhistory.webapp.address
                master:19888

yarn-site.xml


        
               yarn.nodemanager.aux-services
               mapreduce_shuffle
        
                                                                        
yarn.nodemanager.aux-services.mapreduce.shuffle.class
               org.apache.hadoop.mapred.ShuffleHandler
        
        
               yarn.resourcemanager.address
               master:8032
       
       
               yarn.resourcemanager.scheduler.address
               master:8030
       
       
            yarn.resourcemanager.resource-tracker.address
             master:8031
      
      
              yarn.resourcemanager.admin.address
               master:8033
       
       
               yarn.resourcemanager.webapp.address
               master:8088

将hadoop复制到slave1和slave1上

scp -r /opt/hadoop parallels@slave1:~/
scp -r /opt/hadoop parallels@slave2:~/

然后分别在slave节点执行

sudo mv ~/hadoop /opt/hadoop;
sudo chown -R parallels:parallels /opt/hadoop

修改slave上的hadoop-env.sh和yarn-env.sh中的JAVA_HOME值

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64

添加hadoop环境变量

export PATH=$PATH:/opt/hadoop/bin:/opt/hadoop/sbin

启动hadoop

在启动之前，先执行hadoop namenode -format 进行格式化，
然后执行 start-all.sh来启动hadoop集群，以下是我的启动日志

Lestats-MBP:~ lestat$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
19/03/24 09:57:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: namenode running as process 45208. Stop it first.
slave1: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave1.out
slave2: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave2.out
Starting secondary namenodes [master]
master: secondarynamenode running as process 45331. Stop it first.
19/03/24 09:57:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 45435. Stop it first.
slave1: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave1.out
slave2: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave2.out

然后可以执行master上jps来查看启动的JVM。

Lestats-MBP:~ lestat$ jps
45331 SecondaryNameNode
45846 Jps
45208 NameNode
45435 ResourceManager

执行slave上jps来查看启动的JVM。
如果jps命令找不到，可以运行sudo yum install java-1.8.0-openjdk-devel进行安装

[parallels@slave1 ~]$ jps
28832 Jps
26454 DataNode
26590 NodeManager

[parallels@slave2 ~]$ jps
26034 DataNode
26180 NodeManager
27931 Jps

这样hadoop伪分布式集群的安装就告一段落了。

测试

例如我们执行一些hadoop fs的命令：

lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -mkdir /input
19/03/24 10:09:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - lestat supergroup          0 2019-03-24 10:09 /input