Hadoop学习过程中遇到的问题锦集

  

2013-04-13 12:25:47,472 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = xiaohua-PC/192.168.1.100
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2013-04-13 12:25:47,550 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException: Problem binding to localhost/127.0.0.1:9000 : Address already in use: bind
at org.apache.hadoop.ipc.Server.bind(Server.java:190)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:253)
at org.apache.hadoop.ipc.Server.(Server.java:1026)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:488)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:191)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
Caused by: java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:344)
at sun.nio.ch.Net.bind(Net.java:336)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:188)               

解决：netstat -an（查看全部进程）

查看那个进程占用了9000端口

杀掉

也有可能你的HADOOP重复拉起造成的 

netstat -an|grep 9000     

---

   

在进行scp传输“文件夹”的时候，出现not a regular file，研究了很久，发现是在进行scp时候没有加上参数r导致此问题，所以如果在出现此问题，可以用scp -r来进行传输，马虎啊！

    2013-08-16 01:57:39,447 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to master.hadoop/192.168.70.135:9000 failed on local exception: java.net.NoRouteToHostException: No route to host

        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)

        at org.apache.hadoop.ipc.Client.call(Client.java:1075)

        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)

        at $Proxy5.getProtocolVersion(Unknown Source)

        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)

        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:370)

        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:429)

        at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:331)

        at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:296)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:356)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)

Caused by: java.net.NoRouteToHostException: No route to host

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)

        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)

        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)

        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)

        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)

        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)

        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)

        at org.apache.hadoop.ipc.Client.call(Client.java:1050)

        ... 14 more

解决：

    关闭 防火墙  

    如果还不行的话，重启相应的机器

---

启动Hadoop集群测试HBase时候，发现三台DataNode只启动成功了两台，未启动成功的那一台日志中出现了下列异常：

2012-09-07 23:58:51,240 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting down: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 192.168.100.11:50010 is attempting to report storage ID DS-1282452139-218.196.207.181-50010-1344220553439. Node 192.168.100.12:50010 is expected to serve this storage.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:4608)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:3460)
at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:1001)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy5.blockReport(Unknown Source)
at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:958)
at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1458)
at java.lang.Thread.run(Thread.java:722)

     此异常是因为，两台DataNode的storageID出现了冲突，应该是因为我直接备份安装的原因吧。

     解决方法就是直接将出现异常的那台机器的data目录删除！

---

2013-08-19 19:24:47,433 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: 

/************************************************************

STARTUP_MSG: Starting DataNode

STARTUP_MSG:   host = localhost.localdomain/127.0.0.1

STARTUP_MSG:   args = []

STARTUP_MSG:   version = 1.0.4

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012

************************************************************/

2013-08-19 19:24:47,686 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

2013-08-19 19:24:47,706 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.

2013-08-19 19:24:47,707 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).

2013-08-19 19:24:47,707 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started

2013-08-19 19:24:48,036 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.

2013-08-19 19:24:48,730 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /tmp/hadoop-lei/dfs/data: namenode namespaceID = 103896252; datanode namespaceID = 726256761

        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)

        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)

        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)

2013-08-19 19:24:48,747 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 

/************************************************************

SHUTDOWN_MSG: Shutting down DataNode at localhost.localdomain/127.0.0.1

解决方法是：  /tmp/hadoop-lei/dfs/data/current/VERSION 删除掉

参考的信息为：http://blog.csdn.net/wanghai__/article/details/5752199

---

提交文件的时候遇到 不能提交的情况

描述：在window下使用Eclipse进行hadoop的程序编写，然后Run on hadoop 后，出现如下错误：

11/10/28 16:05:53 INFO mapred.JobClient: Running job: job_201110281103_0003
11/10/28 16:05:54 INFO mapred.JobClient: map 0% reduce 0%
11/10/28 16:06:05 INFO mapred.JobClient: Task Id : attempt_201110281103_0003_m_000002_0, Status : FAILED
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=DrWho, access=WRITE, inode="hadoop":hadoop:supergroup:rwxr-xr-x
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

解决：

http://www.cnblogs.com/acmy/archive/2011/10/28/2227901.html

---

3. Invalid Hadoop Runtime specified; please click 'Configure Hadoop install directory' or fill in library location input
 field

   解决：

  主要是因为 Hadoop的Jar包没有导入

  eclipse window->preferences - > Map/Reduce  选择hadoop根目录