hadoop secondary namenode 部署出错所产生的错误及解决方法

只叙述secondary namenode部署出错所产生的错误及解决方法

环境：suse 10.1

namenode 单独部署在cloud1

secondary namenode 单独部署在 cloud3

集群部署完成后使用Jps查看进程,发现该有的进程都有,hdfs也能上传下载文件

 查看secondary name 上的log,发现在doCheckpoint都失败

2011-11-11 00:02:58,154 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
2011-11-11 00:02:58,155 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:395)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:234)
        at sun.net.www.http.HttpClient.New(HttpClient.java:307)
        at sun.net.www.http.HttpClient.New(HttpClient.java:324)

发现secondary namenode不能产生image文件夹更别说image内的fimage,edits等文件

查看name node上的log 

2011-11-11 23:13:03,628 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.0.1.162
2011-11-11 23:18:03,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.0.1.162

没有错误,但是没有没有提示获取成功信息

关闭集群，在hdfs-site.xml中指定

<property>
	<name>dfs.http.address</name>
	<value>{your_namenode_ip}:50070</value>
</property>

分发到下去,重启,发现image文件夹内文件都有了，再次产看secondar namenode上的日志仍然doCheckPoint失败

查看namenode上的log

2011-11-17 13:31:57,434 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:395)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:234)
        at sun.net.www.http.HttpClient.New(HttpClient.java:307)
        at sun.net.www.http.HttpClient.New(HttpClient.java:324)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172)

可以得知secondary namenode上传image失败,所以namenode也就getimage失败了

查了下发现时由于没有指定secondary 的地址,因此很多人在测试的时候由于namenode和secondary namenode是在同一台机子上

而默认的dfs.secondary.http.adress值是0.0.0.0:50090，所以两者能进行通信。但是当分开部署的时候没有指定这个值而默认去在本机上取数据上传数据

当然就出错了。这个我想很多人在部署的时候都没有太注意

       <property>
                <name>dfs.secondary.http.address</name>
                <value>cloud3:50090</value>
        </property>

将上述参数指定好，重启下hadoop,再次查看secondary namenode上的log，log变成了

2011-11-17 14:20:31,435 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
2011-11-17 14:20:31,443 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Downloaded file fsimage size 6699 bytes.
2011-11-17 14:20:31,445 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Downloaded file edits size 4 bytes.
2011-11-17 14:20:31,445 INFO org.apache.hadoop.hdfs.util.GSet: VM type       = 64-bit
2011-11-17 14:20:31,445 INFO org.apache.hadoop.hdfs.util.GSet: 2% max memory = 17.77875 MB
2011-11-17 14:20:31,445 INFO org.apache.hadoop.hdfs.util.GSet: capacity      = 2^21 = 2097152 entries
2011-11-17 14:20:31,445 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152
2011-11-17 14:20:31,447 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=ppstat
2011-11-17 14:20:31,447 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2011-11-17 14:20:31,447 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
2011-11-17 14:20:31,447 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100
2011-11-17 14:20:31,448 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
2011-11-17 14:20:31,448 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2011-11-17 14:20:31,448 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 53
2011-11-17 14:20:31,460 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 7
2011-11-17 14:20:31,463 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /data/cloud/hadoop/tmp/dfs/namesecondary/current/edits of size 4 edits # 0 loaded in 0 seconds.
2011-11-17 14:20:31,464 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
2011-11-17 14:20:31,470 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 6699 saved in 0 seconds.
2011-11-17 14:20:31,477 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 6699 saved in 0 seconds.
2011-11-17 14:20:31,480 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL cloud1:50070putimage=1&port=50090&machine=cloud3&token=-31:114395553:0:1321510831000:1321510231897
2011-11-17 14:20:31,491 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done. New Image Size: 6699

说明我们获取image,并上传成功到namenode成功了

在查看namenode

2011-11-17 14:20:31,972 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll FSImage from 10.0.1.162
2011-11-17 14:20:31,972 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 1 
2011-11-17 14:30:31,997 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.0.1.162
2011-11-17 14:30:31,997 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 

说明就是由于没有指定secondary namenode地址所造成的。

ppstat@cloud3:/data/cloud/hadoop/tmp/dfs/namesecondary/current> ll
总计 20
-rw-r--r-- 1 ppstat users    4 2011-11-17 14:40 edits
-rw-r--r-- 1 ppstat users 6699 2011-11-17 14:40 fsimage
-rw-r--r-- 1 ppstat users    8 2011-11-17 14:40 fstime
-rw-r--r-- 1 ppstat users  100 2011-11-17 14:40 VERSION

PS：抱怨下,csdn的blog越弄越垃圾了，编辑起来一点都不方便，坑爹的玩意