Mac上搭建Hadoop环境(2) — Hadoop下载及安装

前言

在Mac上搭建Hadoop环境(1) — 虚拟机的安装及SSH免密设置中，我们已经配置好了节点之间基本的网络环境，接下来，只要在master(MBP)上下载安装hadoop即可。

下载hadoop

可以前往Apache Hadoop官网，选择你想要的版本进行下载。 这里，我选择下载的是hadoop-2.7.7的binary版本。

安装hadoop

在下载完hadoop-2.7.7.tar.gz后，将其解压到你想安装的目录即可。 我将其解压到 /opt 文件夹下。

sudo tar -C /opt -xvf ~/Downloads/hadoop-2.7.7.tar.gz;

然后对目录重命名并创建相应的子目录

sudo mv /opt/hadoop-2.7.7 /opt/hadoop;
mkdir /opt/hadoop/dfs;
mkdir /opt/hadoop/dfs/name;
mkdir /opt/hadoop/dfs/data;
mkdir /opt/hadoop/tmp;

配置hadoop

hadoop-env.sh

编辑 /opt/hadoop/etc/hadoop/hadoop-env.sh，设置JAVA_HOME值，如下

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home

yarn-env.sh

编辑 /opt/hadoop/etc/hadoop/yarn-env.sh，设置JAVA_HOME值，如下

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home

slaves

编辑 /opt/hadoop/etc/hadoop/slaves，加入之前设置的slave节点的hostname：

slave1
slave2

更新*.xml配置文件

core-site.xml

<configuration>
       <property>
                <name>fs.defaultFSname>
                <value>hdfs://master:8020value>
       property>
       <property>
                <name>io.file.buffer.sizename>
                <value>131072value>
        property>
       <property>
               <name>hadoop.tmp.dirname>
               <value>file:/opt/hadoop/tmpvalue>
               <description>Abase for other temporary directories.description>
       property>
        <property>
               <name>hadoop.proxyuser.lestat.hostsname>
               <value>*value>
       property>
       <property>
               <name>hadoop.proxyuser.lestat.groupsname>
               <value>*value>
       property>
configuration>

hdfs-site.xml

<configuration>
       <property>
                <name>dfs.namenode.secondary.http-addressname>
               <value>master:9001value>
       property>
     <property>
             <name>dfs.namenode.name.dirname>
             <value>file:/opt/hadoop/dfs/namevalue>
       property>
      <property>
              <name>dfs.datanode.data.dirname>
              <value>file:/opt/hadoop/dfs/datavalue>
       property>
       <property>
               <name>dfs.replicationname>
               <value>3value>
        property>
        <property>
                 <name>dfs.webhdfs.enabledname>
                  <value>truevalue>
         property>
configuration>

mapred-site.xml

                                                                        
          mapreduce.framework.name
                yarn
           
          
                  mapreduce.jobhistory.address
                  master:10020
          
          
                mapreduce.jobhistory.webapp.address
                master:19888
       

yarn-site.xml

<configuration>
        <property>
               <name>yarn.nodemanager.aux-servicesname>
               <value>mapreduce_shufflevalue>
        property>
        <property>                                                                
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.classname>
               <value>org.apache.hadoop.mapred.ShuffleHandlervalue>
        property>
        <property>
               <name>yarn.resourcemanager.addressname>
               <value>master:8032value>
       property>
       <property>
               <name>yarn.resourcemanager.scheduler.addressname>
               <value>master:8030value>
       property>
       <property>
            <name>yarn.resourcemanager.resource-tracker.addressname>
             <value>master:8031value>
      property>
      <property>
              <name>yarn.resourcemanager.admin.addressname>
               <value>master:8033value>
       property>
       <property>
               <name>yarn.resourcemanager.webapp.addressname>
               <value>master:8088value>
       property>
configuration>

将hadoop复制到slave1和slave1上

scp -r /opt/hadoop parallels@slave1:~/
scp -r /opt/hadoop parallels@slave2:~/

然后分别在slave节点执行

sudo mv ~/hadoop /opt/hadoop;
sudo chown -R parallels:parallels /opt/hadoop

修改slave上的hadoop-env.sh和yarn-env.sh中的JAVA_HOME值

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64

添加hadoop环境变量

export PATH=$PATH:/opt/hadoop/bin:/opt/hadoop/sbin

启动hadoop

在启动之前，先执行hadoop namenode -format 进行格式化，
然后执行 start-all.sh来启动hadoop集群，以下是我的启动日志

Lestats-MBP:~ lestat$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
19/03/24 09:57:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: namenode running as process 45208. Stop it first.
slave1: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave1.out
slave2: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave2.out
Starting secondary namenodes [master]
master: secondarynamenode running as process 45331. Stop it first.
19/03/24 09:57:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 45435. Stop it first.
slave1: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave1.out
slave2: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave2.out

然后可以执行master上jps来查看启动的JVM。

Lestats-MBP:~ lestat$ jps
45331 SecondaryNameNode
45846 Jps
45208 NameNode
45435 ResourceManager

执行slave上jps来查看启动的JVM。
如果jps命令找不到， 可以运行sudo yum install java-1.8.0-openjdk-devel进行安装

[parallels@slave1 ~]$ jps
28832 Jps
26454 DataNode
26590 NodeManager

[parallels@slave2 ~]$ jps
26034 DataNode
26180 NodeManager
27931 Jps

这样hadoop伪分布式集群的安装就告一段落了。

测试

例如我们执行一些hadoop fs的命令：

lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -mkdir /input
19/03/24 10:09:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - lestat supergroup          0 2019-03-24 10:09 /input