之前在centos7安装hadoop2.2.0集群-环境准备及centos7安装hadoop2.2.0集群-软件安装中讲过从最开始的环境准备及软件安装,从设置hostname到jdk的安装都没有问题,只是在这里重新回顾一下完整版的hadoop安装,验证过的,比较靠谱一点。
因为之前jdk安装之前的都没有问题,所以这里只讲hadoop的安装。
1、首先下载hadoop压缩包,hadoop-2.2.0.tar.gz,解压到安装目录下(自己定),然后设置环境变量,如下:
export HADOOP_HOME=/export/servers/hadoop-2.2.0
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export PATH=$JAVA_HOME/bin:JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
2、修改配置文件,hadoop2.2的配置文件目录是$HADOOP_HOME/etc/hadoop,主要修改的有以下几个配置文件:
core-site.xml:(后两项是设置oozie的代理)
fs.defaultFS
hdfs://hadoop-master
hadoop.tmp.dir
/export/data/hadoop_tmp
hadoop.proxyuser.root.hosts
*
hadoop.proxyuser.root.groups
*
hadoop-en.sh:
在文件末尾添加java安装目录,例如:export JAVA_HOME=/usr/local/java/jdk1.7.0_71
hdfs-site.xml:
dfs.nameservices
ns1
Logical name for this newnameservice
dfs.ha.namenodes.ns1
nn1
Unique identifiers for each NameNode in the nameservice
dfs.namenode.rpc-address.ns1.nn1
hadoop-master:8020
dfs.namenode.http-address.ns1.nn1
hadoop-master:50070
dfs.replication
1--注释:子节点的个数,超出会报错
dfs.namenode.name.dir
/export/data/nn
dfs.datanode.data.dir
/export/data/dfs
mapred-env.sh:
在文件末尾添加Java安装目录,同上。
mapred-site.xml:
mapreduce.framework.name
yarn
mapreduce.jobhistory.address
hadoop-master:10020
mapreduce.jobhistory.webapp.address
hadoop-master:19888
mapreduce.map.output.compress
false
mapreduce.map.output.compress.codec
org.apache.hadoop.io.compress.GzipCodec
yarn.app.mapreduce.am.staging-dir
/user
slaves文件:
hadoop-slave
yarn-env.sh:
在文件末尾添加Java安装目录,同上。
yarn-site.xml:
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
yarn.resourcemanager.address
hadoop-master:8032
yarn.resourcemanager.scheduler.address
hadoop-master:8030
yarn.resourcemanager.resource-tracker.address
hadoop-master:8031
yarn.resourcemanager.admin.address
hadoop-master:8033
yarn.resourcemanager.webapp.address
hadoop-master:8088
yarn.log-aggregation-enable
true
yarn.nodemanager.log-dirs
/export/data/yarn/logs
yarn.nodemanager.log.retain-seconds
86400
yarn.nodemanager.remote-app-log-dir
/tmp/app-logs
yarn.nodemanager.remote-app-log-dir-suffix
logs
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
yarn.scheduler.fair.allocation.file
/export/servers/hadoop-2.2.0/etc/hadoop/fair-scheduler.xml
yarn.scheduler.assignmultiple
true
yarn.scheduler.fair.allow-undeclared-pools
false
yarn.scheduler.fair.locality.threshold.node
0.1
yarn.scheduler.fair.locality.threshold.rack
0.1
以上,配置已经完成,现在启动hadoop,第一次启动需要初始化:
hdfs namenode -format
进入$HADOOP_HOME/sbin目录,sh start-all.sh
启动后,在浏览器输入以下地址测试,如图表示启动成功:
http://hadoop-master:8088
http://hadoop-master:50070