Hadoop2.7.2集群的部署

原文地址：http://happyshome.cn/blog/deploy/centos/hadoop2.7.2/cluster.html

本文使用了4台虚拟机成功的部署了hadoop2.7.2的集群开发环境。

1. 前言

假设你的环境已经具备了运行Hadoop的必要条件，如果没有你可以参考Centos上源码安装Hadoop2.7.2。

2. 准备工作

这里我准备了4台Centos7虚拟机，下面我们马上来进行配置。

# 编辑host
vi /etc/hosts

# 每个节点添加IP、主机名称
192.168.10.162 hmaster
192.168.10.163 hslave1
192.168.10.164 hslave2
192.168.10.166 hslave3

# 创建hadoop用户
useradd hadoop
# 设置密码
passwd hadoop

# 设置权限
chown hadoop /app/hadoop-2.7.2/ -R
chgrp hadoop /app/hadoop-2.7.2/ -R

# ssh自动登陆
su hadoop
# 这里一路回车即可
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hmaster
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave1
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave2
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave3
chmod 0600 ~/.ssh/authorized_keys

# 关闭防火墙
systemctl stop firewalld

# 也可以编辑sysctl.conf
vi /etc/sysctl.conf

# 添加以下内容
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

3. 部署

3.1. 设置hadoop环境变量

# 切换到hadoop用户，添加hadoop环境变量
vi ~/.bashrc

# 添加以下内容
export HADOOP_PREFIX=/app/hadoop-2.7.2
export HADOOP_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export HADOOP_YARN_HOME=$HADOOP_PREFIX
export PATH=$PATH:$HADOOP_PREFIX/sbin:$HADOOP_PREFIX/bin

# 退出，使之生效
source ~/.bashrc

3.2. 设置master

# 1. 编辑core-site.xml
vi /app/hadoop-2.7.2/etc/hadoop/core-site.xml

# 添加以下内容
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hmaster:9000/</value>
</property>
</configuration>

# 2. 创建NameNode目录
mkdir /home/hadoop/namenode
chown hadoop /home/hadoop/namenode/
chgrp hadoop /home/hadoop/namenode/

# 3. 编辑hdfs-site.xml
vi /app/hadoop-2.7.2/etc/hadoop/hdfs-site.xml

# 添加以下内容
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/datanode</value>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>/home/hadoop/namenode</value>
</property>
</configuration>

# 4. 编辑mapred-site.xml
vi /app/hadoop-2.7.2/etc/hadoop/mapred-site.xml

# 添加以下内容
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

# 5. 编辑yarn-site.xml设置ResourceManager和NodeManagers
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hmaster</value>
</property>
<property>
<name>yarn.nodemanager.hostname</name>
<value>hmaster</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

# 6. 编辑slaves添加DataNode
vi /app/hadoop-2.7.2/etc/hadoop/slaves

# 添加以下内容
hmaster
hslave1
hslave2
hslave3

3.3. 设置DataNode

# 1. 编辑core-site.xml
vi /app/hadoop-2.7.2/etc/hadoop/core-site.xml

# 添加以下内容
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hmaster:9000/</value>
</property>
</configuration>

# 2. 创建DataNode目录
mkdir /home/hadoop/datanode
chown hadoop /home/hadoop/datanode/
chgrp hadoop /home/hadoop/datanode/

# 3. 编辑hdfs-site.xml
vi /app/hadoop-2.7.2/etc/hadoop/hdfs-site.xml

# 添加以下内容
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/datanode</value>
</property>
</configuration>

3.4. 启动

# 在master上切换到hadoop用户
su hadoop
hdfs namenode -format

# 启动命令后, 使用jps查看任务是否启动, Slave节点:DataNode, Master节点: NameNode, DataNode, SecondaryNameNode
cd /app/hadoop-2.7.2/etc/hadoop
sbin/start-dfs.sh

# 启动YARN, 查看Master节点上的ResourceManager和NodeManager是否启动
sbin/start-yarn.sh

# 查看集群状态
hdfs dfsadmin -report

Configured Capacity: 74985766912 (69.84 GB)
Present Capacity: 21625159680 (20.14 GB)
DFS Remaining: 21625114624 (20.14 GB)
DFS Used: 45056 (44 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (4):

Name: 192.168.10.163:50010 (hslave1)
Hostname: hslave1
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 13337149440 (12.42 GB)
DFS Remaining: 5409280000 (5.04 GB)
DFS Used%: 0.00%
DFS Remaining%: 28.85%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sat Apr 09 02:18:49 CST 2016


Name: 192.168.10.164:50010 (hslave2)
Hostname: hslave2
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 13339152384 (12.42 GB)
DFS Remaining: 5407277056 (5.04 GB)
DFS Used%: 0.00%
DFS Remaining%: 28.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sat Apr 09 02:18:50 CST 2016


Name: 192.168.10.162:50010 (hmaster)
Hostname: hmaster
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 13345161216 (12.43 GB)
DFS Remaining: 5401268224 (5.03 GB)
DFS Used%: 0.00%
DFS Remaining%: 28.81%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sat Apr 09 02:18:50 CST 2016


Name: 192.168.10.166:50010 (hslave3)
Hostname: hslave3
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 13339144192 (12.42 GB)
DFS Remaining: 5407289344 (5.04 GB)
DFS Used%: 0.00%
DFS Remaining%: 28.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sat Apr 09 02:18:51 CST 2016

你也可以通过查看 http://hmaster:50070进行查看