Hadoop2.9.1环境搭建
Typically one machine in the cluster is designated asthe NameNode and another machine as the ResourceManager, exclusively. These arethe masters. Other services (such as Web App Proxy Server and MapReduce JobHistory server) are usually run either on dedicated hardware or on sharedinfrastructure, depending upon the load.
The rest of the machines in the cluster act as bothDataNode and NodeManager. These are the slaves.
FROMHadoop2.9.1官网安装指南
1.Hadoop安装
1.修改profile，创建Hadoop文件夹
先安装好jdk8，使用java -version检验是否安装Java环境
下载Hadoop2.9.1源码，解压缩，配置profile文件如下
export HADOOP_HOME=/opt/workspace/hadoop-2.9.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:${MAVEN_HOME}/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH使用source /etc/profile使配置生效
hadoop version验证：
创建Hadoop需要的各文件夹如下：
mkdir /root/hadoop
mkdir /root/hadoop/tmp
mkdir /root/hadoop/var
mkdir /root/hadoop/dfs
mkdir /root/hadoop/dfs/name
mkdir /root/hadoop/dfs/data注：以下各配置文件中勿使用中文，造成编码格式不匹配会导致Hadoop初始化失败
2.修改core-site.xml
修改/opt/workspace/hadoop-2.9.1/etc/hadoop/core-site.xml文件
在节点内加入配置:
<configuration>
<property>
<name>hadoop.tmp.dirname>
<value>/root/hadoop/tmpvalue>
<description>Abase for other temporary directories.description>
property>
<property>
<name>fs.default.namename>
<value>hdfs://master1:9000value>
property>
<property>
<name>io.file.buffer.sizename>
<value>131072value>
property>
configuration>说明：
fs.defaultFS:这个属性用来指定namenode的hdfs协议的文件系统通信地址，可以指定一个主机+端口，也可以指定一个namenode服务（这个服务内部可以有多台namenode实现ha的namenode服务）
hadoop.tmp.dir:hadoop集群在工作的时候存储的一些临时文件的目录
当前使single-namenode，先如此配置，启动成功后再改成高可用性集群(HA)
查询资料：Hadoop官方文档翻译—— YARN ResourceManager High Availability 2.7.3
eclipse配置暂未用到，仅作记录
hadoop.proxyuser.root.hosts
hadoop.proxyuser.root.groups
*
3.修改 hdfs-site.xml
<property>
<name>dfs.namenode.name.dirname>
/root/hadoop/dfs/name
Path on the local filesystem where theNameNode stores the namespace and transactions logs persistently.
property>
<property>
<name>dfs.datanode.data.dirname>
/root/hadoop/dfs/data
Comma separated list of paths on the localfilesystem of a DataNode where it should store its blocks.
property>
<property>
<name>dfs.replicationname>
2
property>说明：dfs.replication:hdfs的副本数设置。也就是上传一个文件，其分割的block块后，每个block的冗余副本个数，默认配置是3。
4.修改yarn-site.xml
<property>
<name>yarn.resourcemanager.hostnamename>
<value>master1value>
property>
<property>
<description>The address of the applications manager interface in the RM.description>
<name>yarn.resourcemanager.addressname>
<value>${yarn.resourcemanager.hostname}:8032value>
property>
<property>
<description>The address of the scheduler interface.description>
<name>yarn.resourcemanager.scheduler.addressname>
<value>${yarn.resourcemanager.hostname}:8030value>
property>
<property>
<description>The http address of the RM web application.description>
<name>yarn.resourcemanager.webapp.addressname>
<value>${yarn.resourcemanager.hostname}:8088value>
property>
<property>
<description>The https adddress of the RM web application.description>
<name>yarn.resourcemanager.webapp.https.addressname>
<value>${yarn.resourcemanager.hostname}:8090value>
property>
<property>
<name>yarn.resourcemanager.resource-tracker.addressname>
<value>${yarn.resourcemanager.hostname}:8031value>
property>
<property>
<description>The address of the RM admin interface.description>
<name>yarn.resourcemanager.admin.addressname>
<value>${yarn.resourcemanager.hostname}:8033value>
property>
<property>
<name>yarn.nodemanager.aux-servicesname>
<value>mapreduce_shufflevalue>
property>
<property>
<name>yarn.scheduler.maximum-allocation-mbname>
<value>5120value>
property>
<property>
<name>yarn.nodemanager.vmem-pmem-rationame>
<value>2.1value>
property>
<property>
<name>yarn.nodemanager.resource.memory-mbname>
<value>5120value>
property>5. 修改mapred-site.xml
<property>
<name>mapred.job.trackername>
<value>master1:49001value>
property>
<property>
<name>mapred.local.dirname>
<value>/root/hadoop/varvalue>
property>
<property>
<name>mapreduce.framework.namename>
<value>yarnvalue>
property>6.修改 hadoop-env.sh
export JAVA_HOME=${JAVA_HOME}修改为：
export JAVA_HOME=/opt/workspace/jdk1.87.修改slaves
修改/opt/workspace/hadoop-2.9.1/etc/hadoop/slaves文件
改为
slave1
slave2
slave38.启动
- 将环境传输到其他主机
- 只在master主机上进行初始化：
在/opt/workspace/hadoop-2.9.1/bin目录下输入
./hdfs namenode -format[root@master1 bin]# ./hdfs namenode -format
18/08/15 16:09:37 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = localhost/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.9.1
STARTUP_MSG: classpath = /opt/workspace/hadoop-2.9.1/etc/hadoop:/opt/workspace/hadoop-2.9.1/share/h adoop/common/lib/snappy-java-1.0.5.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/curator-fr amework-2.7.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt /workspace/hadoop-2.9.1/share/hadoop/common/lib/json-smart-1.3.1.jar:/opt/workspace/hadoop-2.9.1/shar e/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib /slf4j-log4j12-1.7.25.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-lang3-3.4.jar:/ opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/workspace/hadoop-2.9. 1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/j axb-api-2.2.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/op t/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/workspace/hadoop-2.9 .1/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/curato r-recipes-2.7.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-beanutils-core-1.8.0. jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/workspace/hadoo p-2.9.1/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/comm on/lib/zookeeper-3.4.6.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-collections-3. 2.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/workspace/hado op-2.9.1/share/hadoop/common/lib/asm-3.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/avro -1.7.7.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/workspace/h adoop-2.9.1/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/commo n/lib/jets3t-0.9.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-configuration-1.6. jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/hadoop-annotations-2.9.1.jar:/opt/workspace/h adoop-2.9.1/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/comm on/lib/paranamer-2.3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-logging-1.1.3.ja r:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/work space/hadoop-2.9.1/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/opt/workspace/hadoop-2. 9.1/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib /jersey-core-1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar: /opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/opt/workspace/hadoop- 2.9.1/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/co mmon/lib/log4j-1.2.17.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/junit-4.11.jar:/opt/wor kspace/hadoop-2.9.1/share/hadoop/common/lib/activation-1.1.jar:/opt/workspace/hadoop-2.9.1/share/hado op/common/lib/stax2-api-3.1.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jetty-6.1.26.ja r:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/workspace/hadoop-2.9.1 /share/hadoop/common/lib/jettison-1.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/guava-1 1.0.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/workspace/ hadoop-2.9.1/share/hadoop/common/lib/commons-io-2.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/comm on/lib/hadoop-auth-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jsr305-3.0.0.jar:/op t/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/workspace/hadoop-2.9.1/sha re/hadoop/common/lib/jsp-api-2.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/hamcrest-cor e-1.3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/workspace/ hadoop-2.9.1/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/opt/workspace/hadoop-2.9.1/share/had oop/common/lib/slf4j-api-1.7.25.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/httpcore-4.4. 4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/workspace/hadoop -2.9.1/share/hadoop/common/lib/commons-net-3.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/li b/jaxb-impl-2.2.3-1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/xz-1.0.jar:/opt/workspace /hadoop-2.9.1/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/opt/workspace/hadoop-2.9.1/share/ha doop/common/lib/jsch-0.1.54.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/curator-client-2. 7.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/workspace/h adoop-2.9.1/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/co mmon/lib/httpclient-4.5.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/api-util-1.0.0-M20. jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/workspace/hadoop-2.9 .1/share/hadoop/common/lib/gson-2.2.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/hadoop-comm on-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/common/hadoop-common-2.9.1-tests.jar:/opt/works pace/hadoop-2.9.1/share/hadoop/common/hadoop-nfs-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/h dfs:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/workspace/ha doop-2.9.1/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoo p/hdfs/lib/jackson-databind-2.7.8.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/protobuf-java -2.5.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/workspace/ha doop-2.9.1/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/workspace/hadoop-2.9.1/share/hadoop /hdfs/lib/xmlenc-0.52.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/jackson-annotations-2.7.8 .jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/workspace/hadoop-2. 9.1/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/jersey-server -1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/workspace/hadoop- 2.9.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/l ib/commons-logging-1.1.3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/htrace-core4-4.1.0-inc ubating.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/workspace/ha doop-2.9.1/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/li b/log4j-1.2.17.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/ workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/opt/workspace/hadoop-2.9.1/share /hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/servlet-api-2.5.j ar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/workspace/hadoop-2.9.1/ share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/okio-1.6.0.j ar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/workspace/hadoop-2.9.1/s hare/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/commons-lang- 2.6.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/workspace/hadoop- 2.9.1/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/lib/o khttp-2.7.5.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/hadoop-hdfs-client-2.9.1.jar:/opt/works pace/hadoop-2.9.1/share/hadoop/hdfs/hadoop-hdfs-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hd fs/hadoop-hdfs-native-client-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/hadoop-hdfs-rbf- 2.9.1-tests.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.1.jar:/opt/workspac e/hadoop-2.9.1/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.1-tests.jar:/opt/workspace/hadoop-2.9. 1/share/hadoop/hdfs/hadoop-hdfs-2.9.1-tests.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/hadoop- hdfs-client-2.9.1-tests.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.1.jar:/ opt/workspace/hadoop-2.9.1/share/hadoop/yarn:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/snappy -java-1.0.5.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/curator-framework-2.7.1.jar:/opt/wo rkspace/hadoop-2.9.1/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar:/opt/workspace/hadoop-2.9.1/sha re/hadoop/yarn/lib/json-smart-1.3.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/jackson-map per-asl-1.9.13.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/json-io-2.5.1.jar:/opt/workspace /hadoop-2.9.1/share/hadoop/yarn/lib/guice-3.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/c ommons-lang3-3.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/w orkspace/hadoop-2.9.1/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/workspace/hadoop-2.9.1/share/h adoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/leveldbjni-all-1. 8.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/java-util-1.9.0.jar:/opt/workspace/hadoop-2.9 .1/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/li b/commons-compress-1.4.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/metrics-core-3.0.1.jar :/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/xmlenc-0.52.jar:/opt/workspace/hadoop-2.9.1/share/ hadoop/yarn/lib/curator-recipes-2.7.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/javax.inj ect-1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/commons-beanutils-core-1.8.0.jar:/opt/wor kspace/hadoop-2.9.1/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/workspace/hadoop-2.9.1/share/ hadoop/yarn/lib/woodstox-core-5.0.3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/zookeeper-3 .4.6.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/workspa ce/hadoop-2.9.1/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/ yarn/lib/asm-3.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/avro-1.7.7.jar:/opt/workspace/ hadoop-2.9.1/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yar n/lib/stax-api-1.0-2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/opt/work space/hadoop-2.9.1/share/hadoop/yarn/lib/commons-configuration-1.6.jar:/opt/workspace/hadoop-2.9.1/sh are/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/paranamer-2 .3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/opt/workspace/had oop-2.9.1/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/l ib/commons-logging-1.1.3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/apacheds-kerberos-code c-2.0.0-M15.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar:/ opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/workspace/hadoop-2.9.1/sh are/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/apacheds-i1 8n-2.0.0-M15.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/jcip-annotations-1.0-1.jar:/opt/wo rkspace/hadoop-2.9.1/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar:/opt/workspace/hadoop-2.9.1/sh are/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/activation-1.1 .jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/stax2-api-3.1.4.jar:/opt/workspace/hadoop-2.9. 1/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/ehcache-3. 3.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/opt/workspace/ha doop-2.9.1/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/li b/jettison-1.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/workspace/ hadoop-2.9.1/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/y arn/lib/commons-io-2.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/wo rkspace/hadoop-2.9.1/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/workspace/hadoop-2.9.1/share/had oop/yarn/lib/jsp-api-2.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/commons-math3-3.1.1.ja r:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/fst-2.50.jar:/opt/workspace/hadoop-2.9.1/share/ha doop/yarn/lib/nimbus-jose-jwt-4.41.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/httpcore-4 .4.4.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/workspace/hadoo p-2.9.1/share/hadoop/yarn/lib/commons-net-3.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/j axb-impl-2.2.3-1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/works pace/hadoop-2.9.1/share/hadoop/yarn/lib/xz-1.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/ jetty-sslengine-6.1.26.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/jsch-0.1.54.jar:/opt/wor kspace/hadoop-2.9.1/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/opt/workspace/hadoop-2.9.1/share/ hadoop/yarn/lib/commons-digester-1.8.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/httpclient -4.5.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar :/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar:/opt/workspace/hadoop-2.9.1 /share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/jersey-g uice-1.9.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/lib/gson-2.2.4.jar:/opt/workspace/hadoop-2 .9.1/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop -yarn-server-sharedcachemanager-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-s erver-applicationhistoryservice-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-r egistry-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginsto rage-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-server-router-2.9.1.jar:/opt /workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.9.1.jar:/o pt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-server-common-2.9.1.jar:/opt/workspace/hadoop -2.9.1/share/hadoop/yarn/hadoop-yarn-common-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/h adoop-yarn-server-web-proxy-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-clien t-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.9.1.ja r:/opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.9.1.jar:/ opt/workspace/hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.9.1.jar:/opt/workspace/ hadoop-2.9.1/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/h adoop/yarn/hadoop-yarn-api-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/snappy-ja va-1.0.5.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/op t/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/workspace/hadoop-2.9.1/share/h adoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/le veldbjni-all-1.8.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.j ar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/workspace/h adoop-2.9.1/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/ma preduce/lib/asm-3.2.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/avro-1.7.7.jar:/opt/wo rkspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/workspace/hadoop-2.9.1/sha re/hadoop/mapreduce/lib/hadoop-annotations-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapredu ce/lib/paranamer-2.3.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar :/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/workspace/hadoop-2.9 .1/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib /junit-4.11.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/worksp ace/hadoop-2.9.1/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/workspace/hadoop-2.9.1/share/h adoop/mapreduce/lib/aopalliance-1.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/xz-1.0 .jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/workspace/hadoo p-2.9.1/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/map reduce/hadoop-mapreduce-client-hs-plugins-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduc e/hadoop-mapreduce-client-jobclient-2.9.1-tests.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduc e/hadoop-mapreduce-client-app-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/hadoop-map reduce-examples-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/hadoop-mapreduce-client- core-2.9.1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9. 1.jar:/opt/workspace/hadoop-2.9.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.1.jar:/opt/wo rkspace/hadoop-2.9.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.1.jar:/opt/workspac e/hadoop-2.9.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.1.jar:/opt/workspace/hadoop -2.9.1/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e30710aea4e6e55e69372929106cf119af06fd 0e; compiled by 'root' on 2018-04-16T09:33Z
STARTUP_MSG: java = 1.8.0_181
************************************************************/
18/08/15 16:09:37 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
18/08/15 16:09:37 INFO namenode.NameNode: createNameNode [-format]
18/08/15 16:09:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform. .. using builtin-java classes where applicable
18/08/15 16:09:37 WARN common.Util: Path /root/hadoop/dfs/name should be specified as a URI in config uration files. Please update hdfs configuration.
18/08/15 16:09:37 WARN common.Util: Path /root/hadoop/dfs/name should be specified as a URI in config uration files. Please update hdfs configuration.
Formatting using clusterid: CID-f6d9410e-dc34-4bd9-99cd-226fd1c70b22
18/08/15 16:09:37 INFO namenode.FSEditLog: Edit logging is async:true
18/08/15 16:09:37 INFO namenode.FSNamesystem: KeyProvider: null
18/08/15 16:09:37 INFO namenode.FSNamesystem: fsLock is fair: true
18/08/15 16:09:37 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
18/08/15 16:09:38 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
18/08/15 16:09:38 INFO namenode.FSNamesystem: supergroup = supergroup
18/08/15 16:09:38 INFO namenode.FSNamesystem: isPermissionEnabled = true
18/08/15 16:09:38 INFO namenode.FSNamesystem: HA Enabled: false
18/08/15 16:09:38 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disab ling file IO profiling
18/08/15 16:09:38 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
18/08/15 16:09:38 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostnam e-check=true
18/08/15 16:09:38 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
18/08/15 16:09:38 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Aug 15 16:09:38
18/08/15 16:09:38 INFO util.GSet: Computing capacity for map BlocksMap
18/08/15 16:09:38 INFO util.GSet: VM type = 64-bit
18/08/15 16:09:38 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
18/08/15 16:09:38 INFO util.GSet: capacity = 2^21 = 2097152 entries
18/08/15 16:09:38 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
18/08/15 16:09:38 WARN conf.Configuration: No unit for dfs.heartbeat.interval(3) assuming SECONDS
18/08/15 16:09:38 WARN conf.Configuration: No unit for dfs.namenode.safemode.extension(30000) assumin g MILLISECONDS
18/08/15 16:09:38 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0. 9990000128746033
18/08/15 16:09:38 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
18/08/15 16:09:38 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
18/08/15 16:09:38 INFO blockmanagement.BlockManager: defaultReplication = 2
18/08/15 16:09:38 INFO blockmanagement.BlockManager: maxReplication = 512
18/08/15 16:09:38 INFO blockmanagement.BlockManager: minReplication = 1
18/08/15 16:09:38 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
18/08/15 16:09:38 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
18/08/15 16:09:38 INFO blockmanagement.BlockManager: encryptDataTransfer = false
18/08/15 16:09:38 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
18/08/15 16:09:38 INFO namenode.FSNamesystem: Append Enabled: true
18/08/15 16:09:38 INFO util.GSet: Computing capacity for map INodeMap
18/08/15 16:09:38 INFO util.GSet: VM type = 64-bit
18/08/15 16:09:38 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
18/08/15 16:09:38 INFO util.GSet: capacity = 2^20 = 1048576 entries
18/08/15 16:09:38 INFO namenode.FSDirectory: ACLs enabled? false
18/08/15 16:09:38 INFO namenode.FSDirectory: XAttrs enabled? true
18/08/15 16:09:38 INFO namenode.NameNode: Caching file names occurring more than 10 times
18/08/15 16:09:38 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: falseskipCaptureAcce ssTimeOnlyChange: false
18/08/15 16:09:38 INFO util.GSet: Computing capacity for map cachedBlocks
18/08/15 16:09:38 INFO util.GSet: VM type = 64-bit
18/08/15 16:09:38 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
18/08/15 16:09:38 INFO util.GSet: capacity = 2^18 = 262144 entries
18/08/15 16:09:38 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
18/08/15 16:09:38 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
18/08/15 16:09:38 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
18/08/15 16:09:38 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
18/08/15 16:09:38 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
18/08/15 16:09:38 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/08/15 16:09:38 INFO util.GSet: VM type = 64-bit
18/08/15 16:09:38 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
18/08/15 16:09:38 INFO util.GSet: capacity = 2^15 = 32768 entries
18/08/15 16:09:38 INFO namenode.FSImage: Allocated new BlockPoolId: BP-914791333-127.0.0.1-1534320578 871
18/08/15 16:09:38 INFO common.Storage: Storage directory /root/hadoop/dfs/name has been successfully formatted.
18/08/15 16:09:38 INFO namenode.FSImageFormatProtobuf: Saving image file /root/hadoop/dfs/name/curren t/fsimage.ckpt_0000000000000000000 using no compression
18/08/15 16:09:39 INFO namenode.FSImageFormatProtobuf: Image file /root/hadoop/dfs/name/current/fsima ge.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds .
18/08/15 16:09:39 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/08/15 16:09:39 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/
注：出现以上信息即为初始化成功，初始化只需第一次启动前执行，再次启动无需初始化。
- 启动HDFS
初始化成功后切换到/opt/workspace/hadoop-2.9.1/sbin
启动Hadoop和yarn
第一次登录会询问是否连接，输入yes
[root@master1 sbin]# ./start-dfs.sh
18/08/15 16:27:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master1]
The authenticity of host 'master1 (127.0.0.1)' can't be established.
ECDSA key fingerprint is 63:c8:af:d0:1d:43:b5:80:b2:54:02:6a:a2:13:97:f8.
Are you sure you want to continue connecting (yes/no)? yes
master1: Warning: Permanently added 'master1' (ECDSA) to the list of known hosts.
master1: starting namenode, logging to /opt/workspace/hadoop-2.9.1/logs/hadoop-root-namenode-master1.out
slave3: starting datanode, logging to /opt/workspace/hadoop-2.9.1/logs/hadoop-root-datanode-slave3.out
slave2: starting datanode, logging to /opt/workspace/hadoop-2.9.1/logs/hadoop-root-datanode-slave2.out
slave1: starting datanode, logging to /opt/workspace/hadoop-2.9.1/logs/hadoop-root-datanode-slave1.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 63:c8:af:d0:1d:43:b5:80:b2:54:02:6a:a2:13:97:f8.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /opt/workspace/hadoop-2.9.1/logs/hadoop-root-secondarynamenode-master1.out
18/08/15 16:28:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@master1 sbin]# jps
3065 NameNode
3452 Jps
3261 SecondaryNameNode
启动yarn
[root@master1 sbin]# ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/workspace/hadoop-2.9.1/logs/yarn-root-resourcemanager-master1.out
slave3: starting nodemanager, logging to /opt/workspace/hadoop-2.9.1/logs/yarn-root-nodemanager-slave3.out
slave2: starting nodemanager, logging to /opt/workspace/hadoop-2.9.1/logs/yarn-root-nodemanager-slave2.out
slave1: starting nodemanager, logging to /opt/workspace/hadoop-2.9.1/logs/yarn-root-nodemanager-slave1.out
[root@master1 sbin]# jps
3526 ResourceManager
3065 NameNode
3261 SecondaryNameNode
3806 Jps
[root@slave1 ~]# jps
2981 DataNode
3238 Jps
3099 NodeManager
master上新增一个ResourceManager，slave上新增一个NodeManager，yarn启动成功
一开始slave2,slave3的datanode启动失败，因为第一次启动时这两个节点的yarn-site.xml文件尚有汉字未去掉。stop-all.sh后统一yarn-site.xml，再次启动dfs,yarn，成功
启动namenode或datanode失败，参考hadoop 集群开启之后datanode没有启动
简而言之：
若namenode未启动，先stop-all.sh，删除core-site.xml下所指向的目录，删除后切记要重新建立配置的空目录，然后运行hadoop namenode -format重新初始化hadoop.tmp.dir
若datanode未启动，先stop-all.sh，将所有Salve节点上的tmp(即 hdfs-site.xml 中指定的文件夹，DataNode存放数据块的位置)文件夹删除 ， 然后重新建立tmp文件夹；再将master节点的core-site.xml文件统一到各slave节点。dfs.datanode.data.dir
参考链接：
Linux搭建Hive On Spark环境(spark-1.6.3-without-hive+hadoop2.8.0+hive2.1.1)
hadoop2.9.1伪分布式环境搭建以及文件系统的简单操作
大数据学习系列之七 —– Hadoop+Spark+Zookeeper+HBase+Hive集群搭建 图文详解
Hadoop之——重新格式化HDFS的方案