Linux 搭建单机版Hadoop

本文记录从初始系统安装Hadoop单机版的具体过程。

环境

系统环境	版本
操作系统	Ubuntu 18.04
Java	openjdk version "1.8.0_212"
Hadoop	3.2.0

1. 更新APT镜像源

添加apt阿里镜像源，加速资源的下载速度，参考官方文档

编辑/etc/apt/sources.list.d/aliyun.list

deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse

执行更新

sudo apt update

2. 安装相关工具

Java JDK推荐使用jdk8，jdk9以上默认禁用了许多javax接口，会导致nodemanager和resourceManager启动失败。

sudo apt install -y openjdk-8-jdk

sudo apt install -y git
sudo apt install -y vim
sudo apt install -y curl
sudo apt install -y zsh
sudo apt install -y openssh-server

3. 安装Zsh（个人爱好）

sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

chsh -s /bin/zsh

4. 新建Hadoop用户

sudo adduser hduser

5. 配置SSH权限（Hadoop用户）

su hduser

ssh-keygen -t rsa
cat ~/.sshid_rsa.pub >> ~/.sshauthorized_keys

测试

ssh localhost

6. 下载Hadoop3.2.0

其余版本链接参考清华大学开源软件镜像站

wget [https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-3.2.0.tar.gz](https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-3.2.0.tar.gz)

7. 修改Hadoop配置文件

为了方便，首先编辑~/.bashrc，配置用户环境变量

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export HADOOP_HOME=/home/hduser/hadoop-family/hadoop-3.2.0
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
  
export PATH=${PATH}:${HADOOP_HOME}/bin/:${HADOOP_HOME}/sbin/

配置hadoop-3.2.0/etc/hadoop/hadoop-env.sh

# The java implementation to use. By default, this environment
export JAVA_HOME=${JAVA_HOME}

配置hadoop-3.2.0/etc/hadoop/core-site.xml


        
                fs.defaultFS
                hdfs://localhost:9000
        
        
                hadoop.tmp.dir
                file:/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp
                Abase for other temporary directories.

配置hdfs-site.xml


        
                dfs.namenode.secondary.http-address
                localhost:50090
        
        
                dfs.namenode.name.dir
                file:/home/hduser/hadoop-family/hadoop-3.2.0/file-tmp/dfs/name
        
        
                dfs.datanode.data.dir
                file:/home/hduser/hadoop-family/hadoop-3.2.0/file-tmp/dfs/data
        
        
                dfs.replication
                1

配置hadoop-3.2.0/etc/hadoop/mapred-site.xml


        
                mapreduce.framework.name
                yarn
        
        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        
        
            mapreduce.jobhistory.address
            localhost:10020
        
        
            mapreduce.jobhistory.webapp.address
            localhost:19888
        

        
            yarn.app.mapreduce.am.env
            HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp
        
        
            mapreduce.map.env
            HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp
        
        
            mapreduce.reduce.env
            HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp

配置hadoop-3.2.0/etc/hadoop/yarn-site.xml



        
                yarn.resourcemanager.hostname
                localhost
        
        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        
        
                 yarn.resourcemanager.webapp.address
                 localhost:8088
         
         
                 yarn.resourcemanager.scheduler.address
                 localhost:8030
         
         
                 yarn.resourcemanager.address
                 localhost:8032
         
         
                 yarn.resourcemanager.resource-tracker.address
                 localhost:8031
         
         
                 yarn.resourcemanager.admin.address
                 localhost:8033

8. 格式化Hadoop文件系统

hdfs namenode -format

9. 启动Hadoop

Tips:
若出现DataNode未启动的现象，原因可能是由于hdfs-tmp/dfs/name/current/VERSION和hdfs-tmp/dfs/name/current/VERSION中的clusterID不匹配。
可以清空hdfs-tmp，并重新执行hadoop-3.2.0/bin/hdfs namenode -format。
或将hdfs-tmp/dfs/name/current/VERSION的clusterID复制进hdfs-tmp/dfs/name/current/VERSION。

start-all.sh

访问Hadoop

命令行执行jps可以看到：

命令行执行hdfs dfsadmin -report可以看到：

浏览器访问http://localhost:9870 可以看到:

浏览器访问http://localhost:8088 可以看到: