Mac下单机安装部署Hadoop3.2.1

环境准备

修改主机名

sudo scutil --set HostName localhost

ssh免密登录

ssh-keygen -t rsa      （一路回车直到完成）
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod og-wx ~/.ssh/authorized_keys

设置完之后 ssh localhost 不需要输入密码便登录，就表示设置成功。

JDK环境

在安装hadoop之前，jdk环境已经配置好了，但如果版本不合适，需要重新下载，在环境变量（～/.bash_profile）中修改路径就可以了，不需要卸载以前的jdk。

查看jdk版本：

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

 安装Hadoop

配置环境变量

将hadoop-3.2.1.tar.gz 解压至安装目录 /usr/local/hadoop-3.2.1。

接下来，在~/.bash_profile 中增加 Hadoop 相关用户环境变量，文件完整内容如下： 

export HADOOP_HOME=/usr/local/hadoop-3.2.1
export HADOOP_ROOT_LOGGER=INFO,console
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native:$HADOOP_COMMON_LIB_NATIVE_DIR"

export PATH=$PATH:$HADOOP_HOME/bin:/usr/local/bin

环境变量配置完之后，记得执行 source ~/.bash_profile 命令使环境变量生效。

接下来，还需要修改 %HADOOP_HOME%/etc/hadoop/ 目录下面的几个配置文件。

hadoop-env.sh

# The java implementation to use.
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home

# Location of Hadoop.
export HADOOP_HOME=/usr/local/hadoop-3.2.1

mapred-site.xml

        
                mapreduce.framework.name
                yarn
                true
                The runtime framework for executing MapReduce jobs
        
        
                yarn.app.mapreduce.am.env
                HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.1
        
        
                mapreduce.map.env
                HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.1
        
        
                mapreduce.reduce.env
                HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.1
        

core-site.xml

    
    
        fs.default.name
        hdfs://localhost:9000
    
    
    
        hadoop.tmp.dir
        /usr/local/hadoop-3.2.1/temp
    

hdfs-site.xml 

        
                dfs.permissions.enabled
                false
        
        
                dfs.replication
                1
                
        
                dfs.namenode.name.dir
                /usr/local/hadoop-3.2.1/data/namenode
        
        
                dfs.datanode.data.dir
                /usr/local/hadoop-3.2.1/data/datanode
        
        
                dfs.namenode.secondary.http-address
                localhost:9001
        
        
                dfs.webhdfs.enabled
                true
        
        
                dfs.http.address
                0.0.0.0:50070
        

yarn-site.xml

        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        

启动Hadoop

格式化HDFS

hdfs namenode -format

启动所以节点

一次启动hadoop所有进程：

start-all.sh 

通过 jps 查看启动的进程。

dengdeyudeMacBook-Pro:hadoop dengdeyu$ jps
25889 NodeManager
27009 Jps
25346 NameNode
25589 SecondaryNameNode
25451 DataNode
25789 ResourceManager

 启动成功之后，可以通过 http://localhost:50070 进入hdfs管理页面 ，访问 http://localhost:8088 进入hadoop进程管理页面。

MapReduce测试

dengdeyudeMacBook-Pro:hadoop dengdeyu$ hdfs dfs -mkdir /input
dengdeyudeMacBook-Pro:hadoop dengdeyu$ hdfs dfs -put /usr/local/hadoop-3.2.1/etc/hadoop/hdfs-site.xml /input
dengdeyudeMacBook-Pro:hadoop dengdeyu$ hadoop jar /usr/local/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep /input /output 'dfs[a-z.]+'
dengdeyudeMacBook-Pro:hadoop dengdeyu$ hdfs dfs -cat /output/1/part-r-00000
2020-03-22 19:24:40,387 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1	dfs.webhdfs.enabled
1	dfs.replication
1	dfs.permissions.enabled
1	dfs.namenode.secondary.http
1	dfs.namenode.name.dir
1	dfs.http.address
1	dfs.datanode.data.dir