Hadoop 2.6.0分布式部署参考手册
Hadoop 2.6.0分布式部署参考手册
关于本参考手册的word文档，可以到如下地址下载：http://download.csdn.net/detail/u012875880/8291493
1.环境说明
1.1安装环境说明
本列中，操作系统为Centos 7.0，JDK版本为Oracle HotSpot 1.7，Hadoop版本为Apache Hadoop 2.6.0，操作用户为hadoop。
2.2 Hadoop集群环境说明：
集群各节点信息参考如下：
| 主机名 |
IP地址 |
角色 |
| ResourceManager |
172.15.0.2 |
ResourceManager & MR JobHistory Server |
| NameNode |
172.15.0.3 |
NameNode |
| SecondaryNameNode |
172.15.0.4 |
SecondaryNameNode |
| DataNode01 |
172.15.0.5 |
DataNode & NodeManager |
| DataNode02 |
172.15.0.6 |
DataNode & NodeManager |
| DataNode03 |
172.15.0.7 |
DataNode & NodeManager |
| DataNode04 |
172.15.0.8 |
DataNode & NodeManager |
| DataNode05 |
172.15.0.9 |
DataNode & NodeManager |
注：上述表中用”&”连接多个角色，如主机”ResourceManager”有两个角色，分别为ResourceManager和MR JobHistory Server。
2.基础环境安装及配置
2.1 添加hadoop用户
useradd hadoop
用户“hadoop”即为Hadoop集群的安装和使用用户。
2.2 JDK 1.7安装
Centos 7自带的JDK版本为 OpenJDK 1.7，本例中需要将其更换为Oracle HotSpot 1.7版，本例中采用解压二进制包方式安装，安装目录为/opt/。
① 查看当前JDK rpm包
rpm -qa | grep jdk
java-1.7.0-openjdk-1.7.0.51-2.4.5.5.el7.x86_64
java-1.7.0-openjdk-headless-1.7.0.51-2.4.5.5.el7.x86_64② 删除自带JDK
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.51-2.4.5.5.el7.x86_64
rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.51-2.4.5.5.el7.x86_64
③ 安装指定JDK
进入安装包所在目录并解压
④ 配置环境变量
编辑~/.bashrc或者/etc/profile，添加如下内容：
#JAVA
export JAVA_HOME=/opt/jdk1.7
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$JAVA_HOME/lib
export CLASSPATH=$CLASSPATH:$JAVA_HOME/jre/lib2.3 SSH无密码登陆配置
① 需要设置如上表格所示8台主机间的SSH无密码登陆。
② 进入hadoop用户的根目录下并通过命令ssh-keygen -t rsa 生成秘钥对
③ 创建公钥认证文件authorized_keys并将生成的~/.ssh目录下的id_rsa.pub文件 的内容输出至该文件：
more id_rsa.pub > auhorized_keys
④ 分别改变~/.ssh目录和authorized_keys文件的权限：
chmod 700 ~/.ssh;chmod 600 ~/.ssh/authorized_keys
⑤ 每个节点主机都重复以上步骤，并将各自的~/.ssh/id_rsa.pub文件的公钥拷贝至其 他主机。
对于以上操作，也可以通过一句命令搞定：
rm -rf ~/.ssh;ssh-keygen -t rsa;chmod 700 ~/.ssh;more ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys;chmod 600 ~/.ssh/authorized_keys;
注：在centos 6中可以用dsa方式：ssh-keygen -t dsa命令来设置无密码登陆，在centos 7中只能用rsa方式，否则只能ssh无密码登陆本机，无能登陆它机。
2.4 修改hosts映射文件
分别编辑各节点上的/etc/hosts文件，添加如下内容：
172.15.0.2 ResourceManager
172.15.0.3 NameNode
172.15.0.4 SecondaryNameNode
172.15.0.5 DataNode01
172.15.0.6 DataNode02
172.15.0.7 DataNode03
172.15.0.8 DataNode04
172.15.0.9 DataNode05
172.15.0.5 NodeManager01
172.15.0.6 NodeManager02
172.15.0.7 NodeManager03
172.15.0.8 NodeManager04
172.15.0.9 NodeManager053.Hadoop安装及配置
3.1 通用部分安装及配置
以下操作内容为通用操作部分，及在每个节点上的内容一样。
分别在每个节点上重复如下操作：
① 将hadoop安装包（hadoop-2.6.0.tar）拷贝至/opt目录下，并解压：
tar -xvf hadoop-2.6.0.tar
解压后的hadoop-2.6.0目录(/opt/hadoop-2.6.0)即为hadoop的安装根目录。
② 更改hadoop安装目录hadoop-2.6.0的所有者为hadoop用户：
chown -R hadoop.hadoop /opt/hadoop-2.6.0
③ 添加环境变量：
#hadoop
export HADOOP_HOME=/opt/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin3.2 各节点配置
分别将如下配置文件解压并分发至每个节点的Hadoop“$HADOOP_HOME/etc/hadoop”目录中，如提示是否覆盖文件，确认即可。
Hadoop配置文件：http://download.csdn.net/detail/u012875880/8291517
注：关于各节点的配置参数设置，请参考后面的“附录1”或“附录2”
4.格式化/启动集群
4.1 格式化集群HDFS文件系统
安装完毕后，需登陆NameNode节点或任一DataNode节点执行hdfs namenode -format格式化集群HDFS文件系统；
注：如果非第一次格式化HDFS文件系统，则需要在进行格式化操作前分别将NameNode的dfs.namenode.name.dir和各个DataNode节点的dfs.datanode.data.dir目录(在本例中为/home/hadoop/hadoopdata)下的所有内容清空。
4.2启动Hadoop集群
分别登陆如下主机并执行相应命令：
① 登陆ResourceManger执行start-yarn.sh命令启动集群资源管理系统yarn
② 登陆NameNode执行start-dfs.sh命令启动集群HDFS文件系统
③ 分别登陆SecondaryNameNode、DataNode01、DataNode02、DataNode03、DataNode04 节点执行jps命令，查看每个节点是否有如下Java进程运行：
ResourceManger节点运行的进程：ResouceNamager
NameNode节点运行的进程：NameNode
SecondaryNameNode节点运行的进程：SecondaryNameNode
各个DataNode节点运行的进程：DataNode & NodeManager
如果以上操作正常则说明Hadoop集群已经正常启动。
附录1 关键配置内容参考
1 core-site.xml
fs.defaultFS
hdfs://NameNode:9000
NameNode URI
- 属性”fs.defaultFS“表示NameNode节点地址，由”hdfs://主机名(或ip)：端口号”组成。
2 hdfs-site.xml
dfs.namenode.name.dir
file:///home/hadoop/hadoopdata/hdfs/namenode
dfs.datanode.data.dir
file:///home/jack/hadoopdata/hdfs/datanode
dfs.namenode.secondary.http-address
SecondaryNameNode:50090
- 属性“dfs.namenode.name.dir”表示NameNode存储命名空间和操作日志相关的元数据信息的本地文件系统目录，该项默认本地路径为”/tmp/hadoop-{username}/dfs/name”；
- 属性”dfs.datanode.data.dir“表示DataNode节点存储HDFS文件的本地文件系统目录，由”file://本地目录”组成,该项默认本地路径为”/tmp/hadoop-{username}/dfs/data”。
- 属性“dfs.namenode.secondary.http-address”表示SecondNameNode主机及端口号（如果无需额外指定SecondNameNode角色，可以不进行此项配置）；
3 mapred-site.xml
mapreduce.framework.name
yarn
Execution framework set to Hadoop YARN.
- 属性”mapreduce.framework.name“表示执行mapreduce任务所使用的运行框架，默认为local，需要将其改为”yarn”
4 yarn-site.xml
yarn.resourcemanager.hostname
ResourceManager
ResourceManager host
yarn.nodemanager.aux-services
mapreduce_shuffle
Shuffle service that needs to be set for Map Reduce applications.
- 属性”yarn.resourcemanager.hostname”用来指定ResourceManager主机地址；
- 属性”yarn.nodemanager.aux-service“表示MR applicatons所使用的shuffle工具类
5 hadoop-env.sh
JAVA_HOME表示当前的Java安装目录
export JAVA_HOME=/opt/jdk-1.76 slaves
集群中的master节点(NameNode、ResourceManager)需要配置其所拥有的slaver节点，其中：
NameNode节点的slaves内容为：
DataNode01
DataNode02
DataNode03
DataNode04
DataNode05ResourceManager节点的slaves内容为：
NodeManager01
NodeManager02
NodeManager03
NodeManager04
NodeManager05附录2 详细配置内容参考
注：以下的红色字体部分的配置参数为必须配置的部分，其他配置皆为默认配置。
1 core-site.xml
l 属性”fs.defaultFS“表示NameNode节点地址，由”hdfs://主机名(或ip)：端口号”组成。
2 hdfs-site.xml
- l 属性“dfs.namenode.name.dir”表示NameNode存储命名空间和操作日志相关的元数据信息的本地文件系统目录，该项默认本地路径为”/tmp/hadoop-{username}/dfs/name”；
- l 属性”dfs.datanode.data.dir“表示DataNode节点存储HDFS文件的本地文件系统目录，由”file://本地目录”组成,该项默认本地路径为”/tmp/hadoop-{username}/dfs/data”。
- l 属性“dfs.namenode.secondary.http-address”表示SecondNameNode主机及端口号（如果无需额外指定SecondNameNode角色，可以不进行此项配置）；
3 mapred-site.xml
- l 属性”mapreduce.framework.name“表示执行mapreduce任务所使用的运行框架，默认为local，需要将其改为”yarn”
4 yarn-site.xml
The default value is 8192.
NOTES:Defines total available resources on the NodeManager to be made available to running containers
The default value is 2.1
NOTES:The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio.
The default value is "${hadoop.tmp.dir}/nm-local-dir"
NOTES:Multiple paths help spread disk i/o.
The default value is "${yarn.log.dir}/userlogs"
NOTES:Multiple paths help spread disk i/o.
The default value is "10800"
The default value is "/logs" or "/tmp/logs"
- 属性”yarn.resourcemanager.hostname”用来指定ResourceManager主机地址；
- l 属性”yarn.nodemanager.aux-service“表示MR applicatons所使用的shuffle工具类
5 hadoop-env.sh
JAVA_HOME表示当前的Java安装目录
export JAVA_HOME=/opt/jdk-1.76 slaves
集群中的master节点(NameNode、ResourceManager)需要配置其所拥有的slaver节点，其中：
NameNode节点的slaves内容为：
DataNode01
DataNode02
DataNode03
DataNode04
DataNode05ResourceManager节点的slaves内容为：
NodeManager01
NodeManager02
NodeManager03
NodeManager04
NodeManager05
附录3 详细参数配置参考
Configuring the Hadoop Daemons in Non-Secure Mode
This section deals with important parameters to be specified in the given configuration files:
· conf/core-site.xml
| Parameter |
Value |
Notes |
| fs.defaultFS |
NameNode URI |
hdfs://host:port/ |
| io.file.buffer.size |
131072 |
Size of read/write buffer used in SequenceFiles. |
· conf/hdfs-site.xml
o Configurations for NameNode:
| Parameter |
Value |
Notes |
| dfs.namenode.name.dir |
Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. |
If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. |
| dfs.namenode.hosts /dfs.namenode.hosts.exclude |
List of permitted/excluded DataNodes. |
If necessary, use these files to control the list of allowable datanodes. |
| dfs.blocksize |
268435456 |
HDFS blocksize of 256MB for large file-systems. |
| dfs.namenode.handler.count |
100 |
More NameNode server threads to handle RPCs from large number of DataNodes. |
o Configurations for DataNode:
| Parameter |
Value |
Notes |
| dfs.datanode.data.dir |
Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks. |
If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. |
· conf/yarn-site.xml
o Configurations for ResourceManager and NodeManager:
| Parameter |
Value |
Notes |
| yarn.acl.enable |
true / false |
Enable ACLs? Defaults to false. |
| yarn.admin.acl |
Admin ACL |
ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which means anyone. Special value of just space means no one has access. |
| yarn.log-aggregation-enable |
false |
Configuration to enable or disable log aggregation |
o Configurations for ResourceManager:
| Parameter |
Value |
Notes |
| yarn.resourcemanager.address |
ResourceManager host:port for clients to submit jobs. |
host:port |
| yarn.resourcemanager.scheduler.address |
ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources. |
host:port |
| yarn.resourcemanager.resource-tracker.address |
ResourceManager host:port for NodeManagers. |
host:port |
| yarn.resourcemanager.admin.address |
ResourceManager host:port for administrative commands. |
host:port |
| yarn.resourcemanager.webapp.address |
ResourceManager web-ui host:port. |
host:port |
| yarn.resourcemanager.hostname |
ResourceManager host. |
host |
| yarn.resourcemanager.scheduler.class |
ResourceManager Scheduler class. |
CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler |
| yarn.scheduler.minimum-allocation-mb |
Minimum limit of memory to allocate to each container request at theResource Manager. |
In MBs |
| yarn.scheduler.maximum-allocation-mb |
Maximum limit of memory to allocate to each container request at theResource Manager. |
In MBs |
| yarn.resourcemanager.nodes.include-path /yarn.resourcemanager.nodes.exclude-path |
List of permitted/excluded NodeManagers. |
If necessary, use these files to control the list of allowable NodeManagers. |
o Configurations for NodeManager:
| Parameter |
Value |
Notes |
| yarn.nodemanager.resource.memory-mb |
Resource i.e. available physical memory, in MB, for givenNodeManager |
Defines total available resources on the NodeManager to be made available to running containers |
| yarn.nodemanager.vmem-pmem-ratio |
Maximum ratio by which virtual memory usage of tasks may exceed physical memory |
The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio. |
| yarn.nodemanager.local-dirs |
Comma-separated list of paths on the local filesystem where intermediate data is written. |
Multiple paths help spread disk i/o. |
| yarn.nodemanager.log-dirs |
Comma-separated list of paths on the local filesystem where logs are written. |
Multiple paths help spread disk i/o. |
| yarn.nodemanager.log.retain-seconds |
10800 |
Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled. |
| yarn.nodemanager.remote-app-log-dir |
/logs |
HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled. |
| yarn.nodemanager.remote-app-log-dir-suffix |
logs |
Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled. |
| yarn.nodemanager.aux-services |
mapreduce_shuffle |
Shuffle service that needs to be set for Map Reduce applications. |
o Configurations for History Server (Needs to be moved elsewhere):
| Parameter |
Value |
Notes |
| yarn.log-aggregation.retain-seconds |
-1 |
How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node. |
| yarn.log-aggregation.retain-check-interval-seconds |
-1 |
Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node. |
· conf/mapred-site.xml
o Configurations for MapReduce Applications:
| Parameter |
Value |
Notes |
| mapreduce.framework.name |
yarn |
Execution framework set to Hadoop YARN. |
| mapreduce.map.memory.mb |
1536 |
Larger resource limit for maps. |
| mapreduce.map.java.opts |
-Xmx1024M |
Larger heap-size for child jvms of maps. |
| mapreduce.reduce.memory.mb |
3072 |
Larger resource limit for reduces. |
| mapreduce.reduce.java.opts |
-Xmx2560M |
Larger heap-size for child jvms of reduces. |
| mapreduce.task.io.sort.mb |
512 |
Higher memory-limit while sorting data for efficiency. |
| mapreduce.task.io.sort.factor |
100 |
More streams merged at once while sorting files. |
| mapreduce.reduce.shuffle.parallelcopies |
50 |
Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. |
o Configurations for MapReduce JobHistory Server:
| Parameter |
Value |
Notes |
| mapreduce.jobhistory.address |
MapReduce JobHistory Server host:port |
Default port is 10020. |
| mapreduce.jobhistory.webapp.address |
MapReduce JobHistory Server Web UI host:port |
Default port is 19888. |
| mapreduce.jobhistory.intermediate-done-dir |
/mr-history/tmp |
Directory where history files are written by MapReduce jobs. |