前言:
使用云服务器有一个特别需要注意的地方,就是 /etc/hosts 文件中
ip地址要是私网ip,不能用公网ip,否则报 java.net.BindException
2017-06-28 15:25:07,371 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.net.BindException: Problem binding to [ecs-node-0001:50070] java.net.BindException: Cannot assign requested address; For more details see: http://wiki.apache.org/hadoop/BindException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
at org.apache.hadoop.ipc.Server.bind(Server.java:552)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:919)
at org.apache.hadoop.ipc.Server.(Server.java:2587)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:958)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:366)
at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:341)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:800)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:415)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:755)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:697)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:898)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:877)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1603)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1671)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:535)
... 13 more
这个错误折磨了我三天,刚开始完全不知道怎么回事,
总是以为自己的配置文件写错了,到处找配置文件的实例,后来在 stackoverflow.com 找到了问题的原因
Problem was with azure server, it did not allow to bind on PUBLIC VIRTUAL IP (VIP) ADDRESS,
it just allow binding on INTERNAL IP ADDRESS, so I create [virtual network]
(https://en.wikipedia.org/wiki/Virtual_network),
add all 3 nodes to the virtual network.
https://stackoverflow.com/questions/35313781/failed-to-start-namenode-cannot-assign-requested-address
根据这个提示,测试了华为云服务器和阿里云服务器,真的好使.成功启动了!
解决了这个问题之后,启动hadoop机群就非常简单了!
1.主机信息
6台华为云服务器
注意一定是
私有IP地址
192.168.1.197 ecs-node-0001 namenode yarn 节点
192.168.1.201 ecs-node-0002
192.168.1.200 ecs-node-0003
192.168.1.199 ecs-node-0004
192.168.1.198 ecs-node-0005
192.168.1.202 ecs-node-0006
2.免密登陆设置
可以参看我的上一篇文章,有详细说明
http://www.jianshu.com/p/0cc626fcf4f5
以下操作,6台都要设置
[root@ecs-4cbe-0001 ~]# vim /etc/hosts
添加
192.168.1.197 ecs-node-0001
192.168.1.201 ecs-node-0002
192.168.1.200 ecs-node-0003
192.168.1.199 ecs-node-0004
192.168.1.198 ecs-node-0005
192.168.1.202 ecs-node-0006
[root@ecs-4cbe-0001 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
59:a8:ba:10:e5:88:e2:1c:c0:7c:ec:3f:91:1a:fd:79 root@ecs-4cbe-0001
The key's randomart image is:
+--[ RSA 2048]----+
| |
|o . . |
|.o o. . . |
|..o+. .. o |
|o.oo.+. S |
|+ ..+.o . |
| o...o o E |
| . .. . |
| . |
+-----------------+
[root@ecs-4cbe-0001 ~]#
以下是 ecs-node-0001上的操作:
[root@ecs-4cbe-0001 ~]# sh getId_rsa.pub.sh
[root@ecs-4cbe-0001 ~]# sh sendAuthorized_keys.sh
[root@ecs-4cbe-0001 ~]# cat getId_rsa.pub.sh
#!/bin/bash
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
for i in $( seq 2 6 )
do
scp root@ecs-node-000${i}:~/.ssh/id_rsa.pub ~/.ssh/p${i}
cat ~/.ssh/p${i} >> ~/.ssh/authorized_keys
done
[root@ecs-4cbe-0001 ~]# cat sendAuthorized_keys.sh
#!/bin/bash
for i in $( seq 2 6 )
do
scp ~/.ssh/authorized_keys root@ecs-node-000${i}:~/.ssh
done
[root@ecs-4cbe-0001 ~]#
免密登陆配置成功
3.更改hadoop2.8.0配置文件
etc/hadoop/core-site.xml
fs.defaultFS
hdfs://ecs-node-0001:9000/
etc/hadoop/yarn-site.xml
yarn.resourcemanager.hostname
ecs-node-0001
yarn.nodemanager.aux-services
mapreduce_shuffle
etc/hadoop/hdfs-site.xml
dfs.namenode.secondary.http-address
ecs-node-0001:9001
dfs.replication
3
etc/hadoop/mapred-site.xml
此文件没有,自己复制一个
[root@ecs-4cbe-0001 hadoop]# cat mapred-site.xml.template > mapred-site.xml
mapreduce.framework.name
yarn
etc/hadoop/slaves文件
ecs-node-0002
ecs-node-0003
ecs-node-0004
ecs-node-0005
ecs-node-0006
最后,把ecs-node-0001节点上的etc/hadoop 文件分发到其它5个节点
[root@ecs-4cbe-0001 hadoop]# cd ~
[root@ecs-4cbe-0001 ~]# sh sendHadoop.sh
[root@ecs-4cbe-0001 ~]# cat sendHadoop.sh
#!/bin/bash
for i in $( seq 2 6 )
do
scp -r /usr/local/src/hadoop/hadoop/etc/hadoop root@ecs-node-000${i}:/usr/local/src/hadoop/hadoop/etc
done
[root@ecs-4cbe-0001 ~]#
4.启动hadoop机群
[root@ecs-4cbe-0001 ~]# source goToHadoopRootDir.sh
[root@ecs-4cbe-0001 hadoop]# bin/hdfs namenode -format
[root@ecs-4cbe-0001 hadoop]# sbin/start-dfs.sh
Starting namenodes on [ecs-node-0001]
ecs-node-0001: Warning: Permanently added 'ecs-node-0001,192.168.1.197' (ECDSA) to the list of known hosts.
ecs-node-0001: starting namenode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-namenode-ecs-4cbe-0001.out
ecs-node-0006: Warning: Permanently added 'ecs-node-0006,192.168.1.202' (ECDSA) to the list of known hosts.
ecs-node-0005: Warning: Permanently added 'ecs-node-0005,192.168.1.198' (ECDSA) to the list of known hosts.
ecs-node-0002: Warning: Permanently added 'ecs-node-0002,192.168.1.201' (ECDSA) to the list of known hosts.
ecs-node-0004: Warning: Permanently added 'ecs-node-0004,192.168.1.199' (ECDSA) to the list of known hosts.
ecs-node-0003: Warning: Permanently added 'ecs-node-0003,192.168.1.200' (ECDSA) to the list of known hosts.
ecs-node-0006: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0006.out
ecs-node-0005: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0005.out
ecs-node-0003: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0003.out
ecs-node-0002: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0002.out
ecs-node-0004: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0004.out
Starting secondary namenodes [ecs-node-0001]
ecs-node-0001: Warning: Permanently added 'ecs-node-0001,192.168.1.197' (ECDSA) to the list of known hosts.
ecs-node-0001: starting secondarynamenode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-secondarynamenode-ecs-4cbe-0001.out
[root@ecs-4cbe-0001 hadoop]# jps
5936 Jps
5810 SecondaryNameNode
5609 NameNode
[root@ecs-4cbe-0001 hadoop]#
[root@ecs-4cbe-0001 hadoop]# sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-resourcemanager-ecs-4cbe-0001.out
ecs-node-0006: Warning: Permanently added 'ecs-node-0006,192.168.1.202' (ECDSA) to the list of known hosts.
ecs-node-0004: Warning: Permanently added 'ecs-node-0004,192.168.1.199' (ECDSA) to the list of known hosts.
ecs-node-0002: Warning: Permanently added 'ecs-node-0002,192.168.1.201' (ECDSA) to the list of known hosts.
ecs-node-0003: Warning: Permanently added 'ecs-node-0003,192.168.1.200' (ECDSA) to the list of known hosts.
ecs-node-0005: Warning: Permanently added 'ecs-node-0005,192.168.1.198' (ECDSA) to the list of known hosts.
ecs-node-0002: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0002.out
ecs-node-0004: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0004.out
ecs-node-0006: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0006.out
ecs-node-0003: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0003.out
ecs-node-0005: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0005.out
[root@ecs-4cbe-0001 hadoop]# jps
5810 SecondaryNameNode
6259 Jps
5609 NameNode
5997 ResourceManager
[root@ecs-4cbe-0001 hadoop]#
[root@ecs-4cbe-0004 ~]# jps
5714 Jps
5587 NodeManager
5447 DataNode
[root@ecs-4cbe-0004 ~]#
hadoop机群成功启动
5.关闭hadoop机群
[root@ecs-4cbe-0001 hadoop]# sbin/stop-yarn.sh
[root@ecs-4cbe-0001 hadoop]# sbin/stop-dfs.sh
大功告成!
忙了三天终于成功了,自己探索获的果实是最美好的,也是最牢固的!