本文记录从初始系统安装Hadoop单机版的具体过程。
环境
| 系统环境 | 版本 |
|---|---|
| 操作系统 | Ubuntu 18.04 |
| Java | openjdk version "1.8.0_212" |
| Hadoop | 3.2.0 |
1. 更新APT镜像源
添加apt阿里镜像源，加速资源的下载速度，参考官方文档
- 编辑
/etc/apt/sources.list.d/aliyun.list
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
- 执行更新
sudo apt update
2. 安装相关工具
Java JDK推荐使用jdk8，jdk9以上默认禁用了许多javax接口，会导致nodemanager和resourceManager启动失败。
sudo apt install -y openjdk-8-jdk
sudo apt install -y git
sudo apt install -y vim
sudo apt install -y curl
sudo apt install -y zsh
sudo apt install -y openssh-server
3. 安装Zsh（个人爱好）
sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
chsh -s /bin/zsh
4. 新建Hadoop用户
sudo adduser hduser
5. 配置SSH权限（Hadoop用户）
su hduser
ssh-keygen -t rsa
cat ~/.sshid_rsa.pub >> ~/.sshauthorized_keys
测试
ssh localhost
6. 下载Hadoop3.2.0
其余版本链接参考清华大学开源软件镜像站
wget [https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-3.2.0.tar.gz](https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-3.2.0.tar.gz)
7. 修改Hadoop配置文件
为了方便，首先编辑~/.bashrc，配置用户环境变量
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/home/hduser/hadoop-family/hadoop-3.2.0
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
export PATH=${PATH}:${HADOOP_HOME}/bin/:${HADOOP_HOME}/sbin/
- 配置
hadoop-3.2.0/etc/hadoop/hadoop-env.sh
# The java implementation to use. By default, this environment
export JAVA_HOME=${JAVA_HOME}
- 配置
hadoop-3.2.0/etc/hadoop/core-site.xml
fs.defaultFS
hdfs://localhost:9000
hadoop.tmp.dir
file:/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp
Abase for other temporary directories.
- 配置
hdfs-site.xml
dfs.namenode.secondary.http-address
localhost:50090
dfs.namenode.name.dir
file:/home/hduser/hadoop-family/hadoop-3.2.0/file-tmp/dfs/name
dfs.datanode.data.dir
file:/home/hduser/hadoop-family/hadoop-3.2.0/file-tmp/dfs/data
dfs.replication
1
- 配置
hadoop-3.2.0/etc/hadoop/mapred-site.xml
mapreduce.framework.name
yarn
yarn.nodemanager.aux-services
mapreduce_shuffle
mapreduce.jobhistory.address
localhost:10020
mapreduce.jobhistory.webapp.address
localhost:19888
yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp
mapreduce.map.env
HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp
mapreduce.reduce.env
HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp
- 配置
hadoop-3.2.0/etc/hadoop/yarn-site.xml
yarn.resourcemanager.hostname
localhost
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.resourcemanager.webapp.address
localhost:8088
yarn.resourcemanager.scheduler.address
localhost:8030
yarn.resourcemanager.address
localhost:8032
yarn.resourcemanager.resource-tracker.address
localhost:8031
yarn.resourcemanager.admin.address
localhost:8033
8. 格式化Hadoop文件系统
hdfs namenode -format
9. 启动Hadoop
Tips:
若出现DataNode未启动的现象，原因可能是由于hdfs-tmp/dfs/name/current/VERSION和hdfs-tmp/dfs/name/current/VERSION中的clusterID不匹配。
可以清空hdfs-tmp，并重新执行hadoop-3.2.0/bin/hdfs namenode -format。
或将hdfs-tmp/dfs/name/current/VERSION的clusterID复制进hdfs-tmp/dfs/name/current/VERSION。
start-all.sh
访问Hadoop
命令行执行jps可以看到：
命令行执行hdfs dfsadmin -report可以看到：
浏览器访问http://localhost:9870 可以看到:
浏览器访问http://localhost:8088 可以看到: