有空，上一个完整过程。
| software | version |
|---|---|
| operating system | Ubuntu 18.04.3 LTS |
| jdk version | 1.8.0_232 |
| hadoop version | 3.1.3 |
一、install java
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jre openjdk-8-jdk -y
$ dpkg -L openjdk-8-jdk | grep '/bin' (or whereis)
$ vim ~/.profile
```
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
```
$ source ~/.profile
将以上写入shell脚本（记得echo时$符号前加反斜杠 ' \ '）可能不识别source命令，则:
$ ls -l \`which sh\` # 发现sh不对：bin/sh -> dash
$ sudo dpkg-reconfigure dash # 选择 no
$ ls -l \`which sh\` # 此时为 bin/sh -> bash ，即可以在shell脚本中执行source
二、install hadoop
$ sudo apt-get install ssh
$ sudo apt-get install pdsh -y
$ wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
$ tar -zxvf hadoop-3.1.3.tar.gz
三、single node配置
将java路径添加至etc/hadoop/hadoop-env.sh
将Hadoop路径添加至系统环境配置 ~/.profile
$ echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> etc/hadoop/hadoop-env.sh
$ echo "export HADOOP_HOME=/home/vickee/bigData/hadoop-3.1.3" >> ~/.profile
$ echo "export PATH=\$PATH:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin" >> ~/.profile
$ hadoop
四、fully distributed（完全分布式配置）
三台主机：
| role | hostname |
|---|---|
| master | blockchain004 |
| slave1 | blockchain002 |
| slave2 | blockchain003 |
1. Install java && Hadoop for each node（同上文一、二、，不再赘述）
2. 配置主机名（on each node）
$ vim /etc/hostname
$ vim /etc/hosts # IP and hostname of all nodes inthe expected clusetr
3. 关闭防火墙
- on ubuntu
$ sudo ufw status
$ sudo ufw disable
- on centos
$ sudo systemctl status firewalld
$ sudo systemctl stop firewalld
4. 时间同步
$ sudo apt-get install ntp
$ ntpdate -u ntp1.aliyun.com
$ date # test if ok
5. 配置ssh免密
- on each node
$ ssh-keygen -t rsa
- on each slave
$ scp .ssh/id_rsa.pub [email protected]:.ssh/id_rsa_002.pub
- on master
$ cd .ssh
$ cp id_rsa.pub authorized_keys
$ cat id_rsa_002.pub >> authorized_keys # for each slave pub
$ scp authorized_keys [email protected]:.ssh # for each slave ip
6. 配置Hadoop
在master节点更改配置文件后后发送至slave节点。
- hadoop-env.sh
$ echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> etc/hadoop/hadoop-env.sh
- * yarn-env.sh
$ echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> etc/hadoop/yarn-env.sh
- core-site.xml
$ vim etc/hadoop/core-site.sh
fs.defaultFS
hdfs://blockchain-004:9000
io.file.buffer.size
131072
hadoop.tmp.dir
file:/home/vickee/bigData/temp
- hdfs-site.xml
$ vim etc/hadoop/hdfs-site.sh
dfs.namenode.secondary.http-address
blockchain-004:9001
dfs.namenode.name.dir
file:/home/vickee/bigData/dfs/name
dfs.datanode.data.dir
file:/home/vickee/bigData/dfs/data
dfs.replication
2
dfs.webhdfs.enabled
true
dfs.permissions
false
dfs.web.ugi
supergroup
- mapred-site.xml
$ vim etc/hadoop/mapred-site.sh
mapreduce.framework.name
yarn
mapreduce.jobhistory.address
blockchain-004:10020
mapreduce.jobhistory.webapp.address
blockchain-004:19888
yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=/home/vickee/bigData/hadoop-3.1.3
mapreduce.map.env
HADOOP_MAPRED_HOME=/home/vickee/bigData/hadoop-3.1.3
mapreduce.reduce.env
HADOOP_MAPRED_HOME=/home/vickee/bigData/hadoop-3.1.3
note: 我初始配置时没有添加最后三个属性，导致运行基准测试工具TestDFSIO时出现错误：
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
......
- yarn-site.xml
$ vim etc/hadoop/yarn-site.sh
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
yarn.resourcemanager.address
blockchain-004:8032
yarn.resourcemanager.scheduler.address
blockchain-004:8030
yarn.resourcemanager.resource-tracker.address
blockchain-004:8031
yarn.resourcemanager.admin.address
blockchain-004:8033
yarn.resourcemanager.webapp.address
blockchain-004:8088
- worker
$ vim etc/hadoop/workers
blockchain-002
blockchain-003
- 发送配置文件夹到salves上
$ scp -r etc/hadoop/ [email protected]:/home/vickee/bigData/hadoop-3.1.3/etc/
- 格式化主节点hadoop
$ hadoop namenode -format
提示：successfully formatted
- 启动
$ ./sbin/start-all.sh
可通过jps命令在各节点中查看Hadoop的相关进程