安装环境: MacBook 10.9.1
Hadoop版本:2.2.0
单机伪同步环境。
启动start-dfs.sh后,发现本地没有datanode进程,查看datanode的日志如下:
2014-03-24 23:48:11,357 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-127.0.0.1-50010-1395699655564) service to localhost/192.168.1.101:9000 beginning handshake with NN
2014-03-24 23:48:11,381 FATAL org.apache.hadoop.hdfs.server.datanode
.DataNode: Initialization failed for block pool Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-
127.0.0.1-50010-1395699655564) service to localhost/
192.168.1.101:9000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException):
Datanode denied communication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-2053126411-
127.0.0.1-50010-1395699655564, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-ba95b66c-d94b-4390-8d44-adb486ba5682;nsid=2073699254;c=0)
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:739)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3929)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:948)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24079)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
at java.lang.Thread.run(Thread.java:695)
2014-03-24 23:48:11,382 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-127.0.0.1-50010-1395699655564) service to localhost/192.168.1.101:9000
2014-03-24 23:48:11,484 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-1316134947-127.0.0.1-1395699640023 (storage id DS-2053126411-127.0.0.1-50010-1395699655564)
2014-03-24 23:48:11,484 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Removed bpid=BP-1316134947-127.0.0.1-1395699640023 from blockPoolScannerMap
2014-03-24 23:48:11,484 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing block pool BP-1316134947-127.0.0.1-1395699640023
2014-03-24 23:48:13,485 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2014-03-24 23:48:13,486 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2014-03-24 23:48:13,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at localhost/127.0.0.1
************************************************************/
导致这个问题的原因很多,在stackoverflow上有很多解决方案,但都不适用我。经过摸索,通过修改/etc/hosts解决。
/etc/hosts原来为:
127.0.0.1 localhost
由于我本机IP地址为 192.168.1.101,所以将hosts文件修改为这个ip地址:
192.168.1.101 localhost
理论上127.0.0.1也是可行的,但不知道我的为什么不行。
修改后,重新删除dfs.namenode.name.dir和dfs.datanode.data.dir以及hadoop.tmp.dir目录,分别执行:
hdfs namenode -format
start-dfs.sh
start-yarn.sh
后,用jps查看服务,全部正常:
37619 NodeManager
37798 Jps
37247 NameNode
37330 DataNode
37536 ResourceManager
37432 SecondaryNameNode