Hadoop secondarynamenode应用
本文参考：http://www.cnblogs.com/ggjucheng/archive/2012/04/18/2454693.html，本文使用的是伪分布式的hadoop集群，版本为1.0.4，主要的配置文件如下：
core-site.xml:
<!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/fs/tmp</value> </property> <property> <name>fs.default.name</name> <value>hdfs://hadoop:9000</value> </property> <property> <name>fs.checkpoint.period</name> <value>300</value> </property> <property> <name>fs.checkpoint.size</name> <value>67108864</value> </property> <property> <name>fs.checkpoint.dir</name> <value>/home/hadoop/checkpoint/namesecodary</value> </property> </configuration>
hdfs-site.xml:
<!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replicatioin</name> <value>1</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/fs/data</value> </property> <property> <name>dfs.name.dir</name> <value>/home/hadoop/fs/name</value> </property> <property> <name>dfs.back.http.address</name> <value>hadoop:50070</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> </configuration>
mapred-site.xml:
<!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>hadoop:9001</value> </property> </configuration>
1.首先输入jps，查看hadoop进程，如下：
hadoop@hadoop:~/hadoop-1.0.4/bin$ jps 16018 TaskTracker 15646 DataNode 17250 NameNode 15870 JobTracker 17296 Jps 15790 SecondaryNameNode
2.kill掉namenode的进程，继续查看hadoop进程，如下：
hadoop@hadoop:~/hadoop-1.0.4/bin$ kill 17250 hadoop@hadoop:~/hadoop-1.0.4/bin$ jps 16018 TaskTracker 15646 DataNode 17328 Jps 15870 JobTracker 15790 SecondaryNameNode
3. 删除dfs.name.dir的目录的全部内容；
4. 这里没有按照参考的博客那样把secondarynamenode的fs.checkpoint.dir的内容拷贝到namenode的dfs.name.dir下面，而是直接使用./hadoop namenode -importCheckPoint命令，这个命令应该已经含有了拷贝的命令了，出现下面的logs信息：
hadoop@hadoop:~/hadoop-1.0.4/bin$ ./hadoop namenode -importCheckPoint 13/05/28 15:37:14 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoop/192.168.128.138 STARTUP_MSG: args = [-importCheckPoint] STARTUP_MSG: version = 1.0.4 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012 ************************************************************/ 13/05/28 15:37:14 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 13/05/28 15:37:14 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 13/05/28 15:37:14 INFO impl.MetricsSystemImpl: NameNode metrics system started 13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source ugi registered. 13/05/28 15:37:14 WARN impl.MetricsSystemImpl: Source name ugi already exists! 13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source jvm registered. 13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source NameNode registered. 13/05/28 15:37:14 INFO util.GSet: VM type = 32-bit 13/05/28 15:37:14 INFO util.GSet: 2% max memory = 19.33375 MB 13/05/28 15:37:14 INFO util.GSet: capacity = 2^22 = 4194304 entries 13/05/28 15:37:14 INFO util.GSet: recommended=4194304, actual=4194304 13/05/28 15:37:14 INFO namenode.FSNamesystem: fsOwner=hadoop 13/05/28 15:37:14 INFO namenode.FSNamesystem: supergroup=supergroup 13/05/28 15:37:14 INFO namenode.FSNamesystem: isPermissionEnabled=false 13/05/28 15:37:14 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 13/05/28 15:37:14 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 13/05/28 15:37:15 INFO namenode.FSNamesystem: Registered FSNamesystemStateMBean and NameNodeMXBean 13/05/28 15:37:15 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/05/28 15:37:15 INFO common.Storage: Storage directory /home/hadoop/fs/name is not formatted. 13/05/28 15:37:15 INFO common.Storage: Formatting ... 13/05/28 15:37:15 INFO common.Storage: Number of files = 8 13/05/28 15:37:15 INFO common.Storage: Number of files under construction = 0 13/05/28 15:37:15 INFO common.Storage: Image file of size 755 loaded in 0 seconds. 13/05/28 15:37:15 INFO common.Storage: Edits file /home/hadoop/checkpoint/namesecodary/current/edits of size 4 edits # 0 loaded in 0 seconds. 13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds. 13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds. 13/05/28 15:37:15 INFO namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds. 13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds. 13/05/28 15:37:15 INFO namenode.NameCache: initialized with 0 entries 0 lookups 13/05/28 15:37:15 INFO namenode.FSNamesystem: Finished loading FSImage in 345 msecs 13/05/28 15:37:15 INFO hdfs.StateChange: STATE* Safe mode ON. The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically. 13/05/28 15:37:15 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list 13/05/28 15:37:15 INFO impl.MetricsSourceAdapter: MBean for source FSNamesystemMetrics registered. 13/05/28 15:37:15 INFO impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9000 registered. 13/05/28 15:37:15 INFO impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9000 registered. 13/05/28 15:37:15 INFO ipc.Server: Starting SocketReader 13/05/28 15:37:15 INFO namenode.NameNode: Namenode up at: hadoop/192.168.128.138:9000 13/05/28 15:37:35 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 13/05/28 15:37:35 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 13/05/28 15:37:35 INFO http.HttpServer: dfs.webhdfs.enabled = false 13/05/28 15:37:35 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070 13/05/28 15:37:35 INFO http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070 13/05/28 15:37:35 INFO http.HttpServer: Jetty bound to port 50070 13/05/28 15:37:35 INFO mortbay.log: jetty-6.1.26 13/05/28 15:37:35 INFO mortbay.log: Started [email protected]:50070 13/05/28 15:37:35 INFO namenode.NameNode: Web-server up at: 0.0.0.0:50070 13/05/28 15:37:35 INFO ipc.Server: IPC Server Responder: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server listener on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 0 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 1 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 2 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 3 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 4 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 5 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 6 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 7 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 8 on 9000: starting 13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 9 on 9000: starting 13/05/28 15:37:35 INFO hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.128.138:50010 storage DS-2106362203-192.168.128.138-50010-1368229536640 13/05/28 15:37:35 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.128.138:50010 13/05/28 15:37:35 INFO hdfs.StateChange: STATE* Safe mode extension entered. The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 29 seconds. 13/05/28 15:37:35 INFO hdfs.StateChange: *BLOCK* NameSystem.processReport: from 192.168.128.138:50010, blocks: 1, processing time: 10 msecs 13/05/28 15:37:56 INFO hdfs.StateChange: STATE* Safe mode ON. The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 9 seconds. 13/05/28 15:38:06 INFO namenode.FSNamesystem: Total number of blocks = 1 13/05/28 15:38:06 INFO namenode.FSNamesystem: Number of invalid blocks = 0 13/05/28 15:38:06 INFO namenode.FSNamesystem: Number of under-replicated blocks = 1 13/05/28 15:38:06 INFO namenode.FSNamesystem: Number of over-replicated blocks = 0 13/05/28 15:38:06 INFO hdfs.StateChange: STATE* Safe mode termination scan for invalid, over- and under-replicated blocks completed in 56 msec 13/05/28 15:38:06 INFO hdfs.StateChange: STATE* Leaving safe mode after 51 secs. 13/05/28 15:38:06 INFO hdfs.StateChange: STATE* Safe mode is OFF. 13/05/28 15:38:06 INFO hdfs.StateChange: STATE* Network topology has 1 racks and 1 datanodes 13/05/28 15:38:06 INFO hdfs.StateChange: STATE* UnderReplicatedBlocks has 1 blocks 13/05/28 15:38:06 INFO namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec 13/05/28 15:38:06 INFO namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec processing time, 1 msec clock time, 1 cycles 13/05/28 15:38:06 INFO namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec 13/05/28 15:38:06 INFO namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles 13/05/28 15:38:06 INFO namenode.FSNamesystem: Roll Edit Log from 192.168.128.138 13/05/28 15:38:06 INFO namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 13/05/28 15:38:06 INFO namenode.FSNamesystem: Roll FSImage from 192.168.128.138 13/05/28 15:38:06 INFO namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 5
5.在kill掉namenode后，50070界面就不能访问了，但是使用了4的命令后，这个界面又可以访问了。但是使用CTRL+c后50070页面又不可以访问了，这时使用命令./hadoop-daemon.sh start namenode 后稍等片刻就又可以访问50070界面了。