hadoop2.6.0集群的搭建方法
1.集群环境的安装
1.1工具软件版本说明（软件尽量去官网下载）：
VMware Workstation :10.0.0 build-1295980
Ubuntn:Ubuntn15.10
JDK：jdk-8u60-linux-x64.tar.gz
Hadoop:hadoop-2.6.0.tar.gz
SecureCRT：Version 6.2.3 (build 313)
WinSCP:5.1.6(build 3394)
VM先按照ubuntu虚拟机步骤网上搜下，自行安装。
Ubuntu15.10设置用root账号登录: http://kevin12.iteye.com/blog/2271687
VMware中安装了ubuntu全屏设置: http://kevin12.iteye.com/blog/2271690
1.2 虚拟机的一般配置
1.2.1修改主机名称：运行命令vim /etc/hostname 将原先的主机名称改成Master,保存并退出，然后重启虚拟机即可；可以用命令hostname验证一下；
1.2.2 设置静态ip：可参考： http://kevin12.iteye.com/blog/2273491
1.2.3 绑定主机名和IP
运行vim /etc/hosts 绑定后保存并退出，如下图：
1.2.4 关闭防火墙
ubuntu防火墙用的是ufw
查看防火墙状态：sudo ufw status
关闭防火墙：sudo ufw disable
1.2.5 设置免密码登陆
Ubuntu默认安装openssh-client，但是木有安装server，如果运行ssh localhost会报错误：ssh: connect to host localhost port 22: Connection refused
通过命令sudo apt-get install ssh进行安装，过程如下：
2.安装jdk
2.1 将jdk-8u60-linux-x64.tar.gz拷贝到虚拟机中，用命令tar -zxvf jdk-8u60-linux-x64.tar.gz 解压，并将解压包移动到/usr/local/jdk下2.安装JDK
2.2 编辑.bashrc文件配置java的环境变量，添加下面的代码,保存并退出，执行命令：source /.bashrc使配置生效。配置如下：
2.3 验证命令:java -version,出现下面的代码说明jdk安装成功；
3.将master1克隆出来四台节点，分别命名为worker1,worker2,worker3，worker4
3.1 修改worker1的主机名 vim /etc/hostname
3.2 设置worker1的静态ip
3.3 设置worker1的静态ip映射 192.168.112.131 worker1
3.4 关闭防火墙 然后重启
3.5 设置ssh：删除.ssh目录下的内容，重新生成ssh
3.6 将worker1中~/.ssh/id_rsa.pub的内容拷贝到master1中的authorized_keys中
按照上面同样的方法操作worker2,worker3,最后修改将master1中的authorized_keys复制到三台worker中的~/.ssh目录下；
修改master1的/etc/hosts文件，worker的机器ip映射添加进来，如下：
用scp命令将master1中的authorized_keys拷贝到其他4个节点上，命令如下：
用ssh worker1 第一次要输入密码，之后就不用输入密码了，其他的节点也是如此。
4.1官网下载hadoop-2.6.0.tar.gz，复制到ubuntu虚拟机/usr/local/中，运行命令tar -zxvf hadoop-2.6.0.tar.gz ./hadoop 命令解压。
配置相关文件：
配置hadoop的环境变量 ~/.bashrc
说明：
HADOOP_CONF_DIR配置是为了更好的支持yarn的运行，包括spark运行yarn;
下面两项可选
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"
core-site.xml配置：
hdfs-site.xml配置：
yarn-site.xml配置：
yarn-env.sh配置：
hadoop-env.sh配置：
slaves配置：
4.2在master1上配置好后，用scp命令将配置好的环境变量.bashrc和hadoop拷贝到其他四台worker节点上。
拷贝.bashrc命令：
然后每台worker节点上运行命令source ~/.bashrc，使之生效。
拷贝配置好的hadoop命令：
查看worker上的/usr/local/目录下面是否拷贝过来。
4.3 在master1上格式化namenode
启动hdfs: ./sbin/start-dfs.sh 打印出如下信息：
用命令jps验证master1上进程，启动了NameNode和SecondaryNameNode后台进程；
在worker1上用jps验证，启动了DataNode后台进程；
在master1上启动yarn: ./sbin/start-yarn.sh 打印出如下信息：
在master1上用jps验证，启动了ResourceManager后台进程；
在worker1上用jps验证，启动了NodeManager后台进程；
关闭hdfs用命令./sbin/stop-dfs.sh
关闭yarn用命令./sbin/stop-yarn.sh
查看集群状态用命令./bin/hdfs dfsadmin -report
查看文件块组成： ./bin/hdf sfsck / -files -blocks
root@master1:/usr/local/hadoop/hadoop-2.6.0/bin# hdfs fsck / -files -blocks
16/01/23 22:13:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connecting to namenode via http://master1:50070
FSCK started by root (auth:SIMPLE) from /192.168.112.130 for path / at Sat Jan 23 22:13:47 CST 2016
/ <dir>
Status: HEALTHY
Total size: 0 B
Total dirs: 1
Total files: 0
Total symlinks: 0
Total blocks (validated): 0
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 2
Average block replication: 0.0
Corrupt blocks: 0
Missing replicas: 0
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Sat Jan 23 22:13:47 CST 2016 in 2 milliseconds
The filesystem under path '/' is HEALTHY
4.4 用浏览器访问集群
在master1上输入http://192.168.112.130:50070 或者master1:50070访问
用http://192.168.112.130:8088/查看RM的监控页面
到此为止，hadoop集群已经安装成功！！！
王家林：中国Spark第一人，Spark亚太研究院院长和首席专家
DT大数据梦工厂
新浪微博：http://weibo.com.ilovepains/
手机：18610086859
QQ：1740415547
联系邮箱[email protected]
1.1工具软件版本说明（软件尽量去官网下载）：
VMware Workstation :10.0.0 build-1295980
Ubuntn:Ubuntn15.10
JDK：jdk-8u60-linux-x64.tar.gz
Hadoop:hadoop-2.6.0.tar.gz
SecureCRT：Version 6.2.3 (build 313)
WinSCP:5.1.6(build 3394)
VM先按照ubuntu虚拟机步骤网上搜下，自行安装。
Ubuntu15.10设置用root账号登录: http://kevin12.iteye.com/blog/2271687
VMware中安装了ubuntu全屏设置: http://kevin12.iteye.com/blog/2271690
1.2 虚拟机的一般配置
1.2.1修改主机名称：运行命令vim /etc/hostname 将原先的主机名称改成Master,保存并退出，然后重启虚拟机即可；可以用命令hostname验证一下；
1.2.2 设置静态ip：可参考： http://kevin12.iteye.com/blog/2273491
1.2.3 绑定主机名和IP
运行vim /etc/hosts 绑定后保存并退出，如下图：
1.2.4 关闭防火墙
ubuntu防火墙用的是ufw
查看防火墙状态：sudo ufw status
关闭防火墙：sudo ufw disable
1.2.5 设置免密码登陆
Ubuntu默认安装openssh-client，但是木有安装server，如果运行ssh localhost会报错误：ssh: connect to host localhost port 22: Connection refused
通过命令sudo apt-get install ssh进行安装，过程如下：
root@master1:~# sudo apt-get install ssh 正在读取软件包列表... 完成 正在分析软件包的依赖关系树 正在读取状态信息... 完成 将会安装下列额外的软件包： libck-connector0 ncurses-term openssh-client openssh-server openssh-sftp-server ssh-import-id 建议安装的软件包： libpam-ssh keychain monkeysphere rssh molly-guard 下列【新】软件包将被安装： libck-connector0 ncurses-term openssh-server openssh-sftp-server ssh ssh-import-id 下列软件包将被升级： openssh-client 升级了 1 个软件包，新安装了 6 个软件包，要卸载 0 个软件包，有 106 个软件包未被升级。 需要下载 8,784 B/1,262 kB 的软件包。 解压缩后会消耗掉 3,523 kB 的额外空间。 您希望继续执行吗？ [Y/n] Y 【警告】：下列软件包不能通过验证！ libck-connector0 openssh-client openssh-sftp-server openssh-server ssh ncurses-term ssh-import-id 不经验证就安装这些软件包吗？ [y/N] y 获取：1 http://cn.archive.ubuntu.com/ubuntu/ wily/main libck-connector0 amd64 0.4.6-5 [8,784 B] 下载 8,784 B，耗时 0秒 (127 kB/s) 正在预设定软件包 ... 正在选中未选择的软件包 libck-connector0:amd64。 (正在读取数据库 ... 系统当前共安装有 180732 个文件和目录。) 正准备解包 .../libck-connector0_0.4.6-5_amd64.deb ... 正在解包 libck-connector0:amd64 (0.4.6-5) ... 正准备解包 .../openssh-client_1%3a6.9p1-2ubuntu0.1_amd64.deb ... 正在将 openssh-client (1:6.9p1-2ubuntu0.1) 解包到 (1:6.9p1-2) 上 ... 正在选中未选择的软件包 openssh-sftp-server。 正准备解包 .../openssh-sftp-server_1%3a6.9p1-2ubuntu0.1_amd64.deb ... 正在解包 openssh-sftp-server (1:6.9p1-2ubuntu0.1) ... 正在选中未选择的软件包 openssh-server。 正准备解包 .../openssh-server_1%3a6.9p1-2ubuntu0.1_amd64.deb ... 正在解包 openssh-server (1:6.9p1-2ubuntu0.1) ... 正在选中未选择的软件包 ssh。 正准备解包 .../ssh_1%3a6.9p1-2ubuntu0.1_all.deb ... 正在解包 ssh (1:6.9p1-2ubuntu0.1) ... 正在选中未选择的软件包 ncurses-term。 正准备解包 .../ncurses-term_5.9+20150516-2ubuntu1_all.deb ... 正在解包 ncurses-term (5.9+20150516-2ubuntu1) ... 正在选中未选择的软件包 ssh-import-id。 正准备解包 .../ssh-import-id_4.5-0ubuntu1_all.deb ... 正在解包 ssh-import-id (4.5-0ubuntu1) ... 正在处理用于 man-db (2.7.4-1) 的触发器 ... 正在处理用于 ureadahead (0.100.0-19) 的触发器 ... 正在处理用于 systemd (225-1ubuntu9) 的触发器 ... 正在处理用于 ufw (0.34-2) 的触发器 ... 正在设置 libck-connector0:amd64 (0.4.6-5) ... 正在设置 openssh-client (1:6.9p1-2ubuntu0.1) ... 正在设置 openssh-sftp-server (1:6.9p1-2ubuntu0.1) ... 正在设置 openssh-server (1:6.9p1-2ubuntu0.1) ... Creating SSH2 RSA key; this may take some time ... 2048 SHA256:Uo5idLAsqmt7e8rAUxXt4hkB0L33zzVgIKv6Gx/TnX4 root@master1 (RSA) Creating SSH2 DSA key; this may take some time ... 1024 SHA256:fFT1QElhDnmT//1tFoymVPKUVN/b/EnL+ODdzooF7no root@master1 (DSA) Creating SSH2 ECDSA key; this may take some time ... 256 SHA256:ctbDQwKOmzr0Ixl43vm1akZAP360sbX4rHWk5FZ2gpo root@master1 (ECDSA) Creating SSH2 ED25519 key; this may take some time ... 256 SHA256:NP0yHA9iwPNNXN3jXz6h+V+J/lYgGXJd5f2MUV4XG+8 root@master1 (ED25519) 正在设置 ncurses-term (5.9+20150516-2ubuntu1) ... 正在设置 ssh-import-id (4.5-0ubuntu1) ... 正在处理用于 ufw (0.34-2) 的触发器 ... 正在设置 ssh (1:6.9p1-2ubuntu0.1) ... 正在处理用于 libc-bin (2.21-0ubuntu4) 的触发器 ... 正在处理用于 ureadahead (0.100.0-19) 的触发器 ... 正在处理用于 systemd (225-1ubuntu9) 的触发器 ... root@master1:~# /etc/init.d/ssh start [ ok ] Starting ssh (via systemctl): ssh.service. root@master1:~# ps -e | grep ssh 3146 ? 00:00:00 sshd //在home目录下面，执行命令ssh -keygen -t rsa 并一路回车即可；再运行命令cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys root@master1:~# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:CCBcvdZn1zlL033Gv6rdQqC1zY0fM73emg+ra+ELDt0 root@master1 The key's randomart image is: +---[RSA 2048]----+ |o o.. | | o . . | | . o . o..| | + o o + * .=| | . . S + * *.+| | o o.* =o| | . o.E.o *| | o .+..B.| | .o=*B=o| +----[SHA256]-----+ root@master1:~# ssh localhost The authenticity of host 'localhost (127.0.0.1)' can't be established. ECDSA key fingerprint is SHA256:ctbDQwKOmzr0Ixl43vm1akZAP360sbX4rHWk5FZ2gpo. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. Welcome to Ubuntu 15.10 (GNU/Linux 4.2.0-16-generic x86_64) * Documentation: https://help.ubuntu.com/ 107 packages can be updated. 0 updates are security updates. The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. root@master1:~#ssh localhost --再次运行时就不用输密码了 Welcome to Ubuntu 15.10 (GNU/Linux 4.2.0-16-generic x86_64) * Documentation: https://help.ubuntu.com/ 107 packages can be updated. 0 updates are security updates. Last login: Tue Jan 19 22:17:00 2016 from 127.0.0.1 root@master1:~#
2.安装jdk
2.1 将jdk-8u60-linux-x64.tar.gz拷贝到虚拟机中，用命令tar -zxvf jdk-8u60-linux-x64.tar.gz 解压，并将解压包移动到/usr/local/jdk下2.安装JDK
2.2 编辑.bashrc文件配置java的环境变量，添加下面的代码,保存并退出，执行命令：source /.bashrc使配置生效。配置如下：
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
export JRE_HOME=${JAVA_HOME}/jre
export PATH=.:${JAVA_HOME}/bin:$PATH
export CLASS_PATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=.:${JAVA_HOME}/bin:$PATH
2.3 验证命令:java -version,出现下面的代码说明jdk安装成功；
root@jylu:~# java -version java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode
3.将master1克隆出来四台节点，分别命名为worker1,worker2,worker3，worker4
3.1 修改worker1的主机名 vim /etc/hostname
3.2 设置worker1的静态ip
3.3 设置worker1的静态ip映射 192.168.112.131 worker1
3.4 关闭防火墙 然后重启
3.5 设置ssh：删除.ssh目录下的内容，重新生成ssh
3.6 将worker1中~/.ssh/id_rsa.pub的内容拷贝到master1中的authorized_keys中
按照上面同样的方法操作worker2,worker3,最后修改将master1中的authorized_keys复制到三台worker中的~/.ssh目录下；
修改master1的/etc/hosts文件，worker的机器ip映射添加进来，如下：
192.168.112.130 master1 192.168.112.131 worker1 192.168.112.132 worker2 192.168.112.133 worker3 192.168.112.134 worker4
用scp命令将master1中的authorized_keys拷贝到其他4个节点上，命令如下：
scp ~/.ssh/authorized_keys root@worker1:~/.ssh/ scp ~/.ssh/authorized_keys root@worker2:~/.ssh/ scp ~/.ssh/authorized_keys root@worker3:~/.ssh/ scp ~/.ssh/authorized_keys root@worker4:~/.ssh/
用ssh worker1 第一次要输入密码，之后就不用输入密码了，其他的节点也是如此。
4.在master1上安装hadoop
4.1官网下载hadoop-2.6.0.tar.gz，复制到ubuntu虚拟机/usr/local/中，运行命令tar -zxvf hadoop-2.6.0.tar.gz ./hadoop 命令解压。
配置相关文件：
配置hadoop的环境变量 ~/.bashrc
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
export JRE_HOME=${JAVA_HOME}/jre
export CLASS_PATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export SCALA_HOME=/usr/local/scala/scala-2.10.4
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
说明：
HADOOP_CONF_DIR配置是为了更好的支持yarn的运行，包括spark运行yarn;
下面两项可选
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"
core-site.xml配置：
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop/hadoop-2.6.0/tmp</value> </property> <property> <name>hadoop.native.lib</name> <value>true</value> <description>Should native hadoop libraries,if present,be used.</description> </property> </configuration>
hdfs-site.xml配置：
<configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>master1:50090</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/usr/local/hadoop/hadoop-2.6.0/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/usr/local/hadoop/hadoop-2.6.0/dfs/data</value> </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>file:///usr/local/hadoop/hadoop-2.6.0/dfs/namesecondary</value> </property> </configuration>
yarn-site.xml配置：
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.resourcemanager.hostname</name> <value>master1</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
yarn-env.sh配置：
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
hadoop-env.sh配置：
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPS="-Djava.library.path=${HADOOP_HOME}/lib"
slaves配置：
worker1 worker2 worker3 worker4
4.2在master1上配置好后，用scp命令将配置好的环境变量.bashrc和hadoop拷贝到其他四台worker节点上。
拷贝.bashrc命令：
scp ~/.bashrc root@worker1:~/ scp ~/.bashrc root@worker2:~/ scp ~/.bashrc root@worker3:~/ scp ~/.bashrc root@worker4:~/
然后每台worker节点上运行命令source ~/.bashrc，使之生效。
拷贝配置好的hadoop命令：
scp -r /usr/local/hadoop/ root@worker1:/usr/local/ scp -r /usr/local/hadoop/ root@worker2:/usr/local/ scp -r /usr/local/hadoop/ root@worker3:/usr/local/ scp -r /usr/local/hadoop/ root@worker4:/usr/local/
查看worker上的/usr/local/目录下面是否拷贝过来。
4.3 在master1上格式化namenode
root@master1:/usr/local/hadoop/hadoop-2.6.0/bin# hdfs namenode -format 16/01/23 21:57:57 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = master1/192.168.112.130 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.6.0 STARTUP_MSG: classpath = /usr/local/hadoop/hadoop-2.6.0/etc/hadoop:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/curator-client-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/curator-framework-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/hadoop-nfs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on 2014-11-13T21:10Z STARTUP_MSG: java = 1.8.0_60 ************************************************************/ 16/01/23 21:57:57 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 16/01/23 21:57:57 INFO namenode.NameNode: createNameNode [-format] 16/01/23 21:57:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/01/23 21:57:58 WARN common.Util: Path /usr/local/hadoop/hadoop-2.6.0/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 16/01/23 21:57:58 WARN common.Util: Path /usr/local/hadoop/hadoop-2.6.0/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration. Formatting using clusterid: CID-9bd3d990-8e12-4581-b131-5021e35cfe78 16/01/23 21:57:58 INFO namenode.FSNamesystem: No KeyProvider found. 16/01/23 21:57:58 INFO namenode.FSNamesystem: fsLock is fair:true 16/01/23 21:57:58 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 16/01/23 21:57:58 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 16/01/23 21:57:58 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 16/01/23 21:57:58 INFO blockmanagement.BlockManager: The block deletion will start around 2016 一月 23 21:57:58 16/01/23 21:57:58 INFO util.GSet: Computing capacity for map BlocksMap 16/01/23 21:57:58 INFO util.GSet: VM type = 64-bit 16/01/23 21:57:58 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 16/01/23 21:57:58 INFO util.GSet: capacity = 2^21 = 2097152 entries 16/01/23 21:57:58 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 16/01/23 21:57:58 INFO blockmanagement.BlockManager: defaultReplication = 2 16/01/23 21:57:58 INFO blockmanagement.BlockManager: maxReplication = 512 16/01/23 21:57:58 INFO blockmanagement.BlockManager: minReplication = 1 16/01/23 21:57:58 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 16/01/23 21:57:58 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 16/01/23 21:57:58 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 16/01/23 21:57:58 INFO blockmanagement.BlockManager: encryptDataTransfer = false 16/01/23 21:57:58 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 16/01/23 21:57:58 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE) 16/01/23 21:57:58 INFO namenode.FSNamesystem: supergroup = supergroup 16/01/23 21:57:58 INFO namenode.FSNamesystem: isPermissionEnabled = true 16/01/23 21:57:58 INFO namenode.FSNamesystem: HA Enabled: false 16/01/23 21:57:58 INFO namenode.FSNamesystem: Append Enabled: true 16/01/23 21:57:58 INFO util.GSet: Computing capacity for map INodeMap 16/01/23 21:57:58 INFO util.GSet: VM type = 64-bit 16/01/23 21:57:58 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 16/01/23 21:57:58 INFO util.GSet: capacity = 2^20 = 1048576 entries 16/01/23 21:57:58 INFO namenode.NameNode: Caching file names occuring more than 10 times 16/01/23 21:57:59 INFO util.GSet: Computing capacity for map cachedBlocks 16/01/23 21:57:59 INFO util.GSet: VM type = 64-bit 16/01/23 21:57:59 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 16/01/23 21:57:59 INFO util.GSet: capacity = 2^18 = 262144 entries 16/01/23 21:57:59 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 16/01/23 21:57:59 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 16/01/23 21:57:59 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 16/01/23 21:57:59 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 16/01/23 21:57:59 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 16/01/23 21:57:59 INFO util.GSet: Computing capacity for map NameNodeRetryCache 16/01/23 21:57:59 INFO util.GSet: VM type = 64-bit 16/01/23 21:57:59 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 16/01/23 21:57:59 INFO util.GSet: capacity = 2^15 = 32768 entries 16/01/23 21:57:59 INFO namenode.NNConf: ACLs enabled? false 16/01/23 21:57:59 INFO namenode.NNConf: XAttrs enabled? true 16/01/23 21:57:59 INFO namenode.NNConf: Maximum size of an xattr: 16384 16/01/23 21:57:59 INFO namenode.FSImage: Allocated new BlockPoolId: BP-773392318-192.168.112.130-1453557479065 16/01/23 21:57:59 INFO common.Storage: Storage directory /usr/local/hadoop/hadoop-2.6.0/dfs/name has been successfully formatted. 16/01/23 21:57:59 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 16/01/23 21:57:59 INFO util.ExitUtil: Exiting with status 0 16/01/23 21:57:59 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master1/192.168.112.130 ************************************************************/ root@master1:/usr/local/hadoop/hadoop-2.6.0/bin#
启动hdfs: ./sbin/start-dfs.sh 打印出如下信息：
root@master1:/usr/local/hadoop/hadoop-2.6.0/sbin# start-dfs.sh 16/01/23 22:01:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [master1] The authenticity of host 'master1 (192.168.112.130)' can't be established. ECDSA key fingerprint is SHA256:ctbDQwKOmzr0Ixl43vm1akZAP360sbX4rHWk5FZ2gpo. Are you sure you want to continue connecting (yes/no)? yes master1: Warning: Permanently added 'master1,192.168.112.130' (ECDSA) to the list of known hosts. master1: starting namenode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-root-namenode-master1.out worker4: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-worker4.out worker1: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-worker1.out worker3: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-worker3.out worker2: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-worker2.out Starting secondary namenodes [master1] master1: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-root-secondarynamenode-master1.out 16/01/23 22:01:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
用命令jps验证master1上进程，启动了NameNode和SecondaryNameNode后台进程；
root@master1:/usr/local/hadoop/hadoop-2.6.0/sbin# jps 4498 Jps 4379 SecondaryNameNode 4175 NameNode
在worker1上用jps验证，启动了DataNode后台进程；
root@worker1:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# jps 2563 DataNode 2646 Jps
在master1上启动yarn: ./sbin/start-yarn.sh 打印出如下信息：
root@master1:/usr/local/hadoop/hadoop-2.6.0/sbin# start-yarn.sh starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-root-resourcemanager-master1.out worker1: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-worker1.out worker3: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-worker3.out worker2: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-worker2.out worker4: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-worker4.out
在master1上用jps验证，启动了ResourceManager后台进程；
root@master1:/usr/local/hadoop/hadoop-2.6.0/sbin# jps 4821 Jps 4551 ResourceManager 4379 SecondaryNameNode 4175 NameNode
在worker1上用jps验证，启动了NodeManager后台进程；
root@worker1:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# jps 2563 DataNode 2713 NodeManager 2829 Jps
关闭hdfs用命令./sbin/stop-dfs.sh
关闭yarn用命令./sbin/stop-yarn.sh
查看集群状态用命令./bin/hdfs dfsadmin -report
查看文件块组成： ./bin/hdf sfsck / -files -blocks
root@master1:/usr/local/hadoop/hadoop-2.6.0/bin# hdfs fsck / -files -blocks
16/01/23 22:13:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connecting to namenode via http://master1:50070
FSCK started by root (auth:SIMPLE) from /192.168.112.130 for path / at Sat Jan 23 22:13:47 CST 2016
/ <dir>
Status: HEALTHY
Total size: 0 B
Total dirs: 1
Total files: 0
Total symlinks: 0
Total blocks (validated): 0
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 2
Average block replication: 0.0
Corrupt blocks: 0
Missing replicas: 0
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Sat Jan 23 22:13:47 CST 2016 in 2 milliseconds
The filesystem under path '/' is HEALTHY
4.4 用浏览器访问集群
在master1上输入http://192.168.112.130:50070 或者master1:50070访问
用http://192.168.112.130:8088/查看RM的监控页面
到此为止，hadoop集群已经安装成功！！！
王家林：中国Spark第一人，Spark亚太研究院院长和首席专家
DT大数据梦工厂
新浪微博：http://weibo.com.ilovepains/
手机：18610086859
QQ：1740415547
联系邮箱[email protected]