hadoop2 HA启动时Namenode均进入安全模式,没有出现active的Namenode

版权声明:本文为博主原创文章,未经博主允许不得转载。https://www.jianshu.com/p/b79c5d333684

HDFS服务重启后的报错:

1、HDFS已使用/总容量为0

2、hdfs有block丢失,丢失99.99%

3、两个NameNode状态都为安全模式,无法退出

4、所有datanode启动后状态均为离线,无法连接至namenode

明确一点,namenode即便进入安全模式,datanode也是能连接上的。

原因:

查看datanode最新日志,HDFS在重启后,每个datanode会去加载本地数据目录下的block到hdfs上,因此在HDFS刚重启时可看到总容量是0,因为datanode还没有加载好,或者说加载很慢。只有当namenode检测到block的数量满足一定阈值,才会自主退出安全模式。这时若试图通过不断重启hdfs或datanode只是重新开始加载block而已。当一个个datanode逐渐加载完成,可以看到hdfs的总容量也会同步增加。

而节点加载block慢的原因分析来看是和磁盘挂载形式有关。

附:datanode加载block日志

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-117f916b-942b-4fce-99df-961cb6f33e12   

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /mnt/data1/dfs/dn/current, StorageType: DISK       

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-ab2f9189-e659-4fd7-bcd7-e5cbd0d4e90a 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /mnt/data2/dfs/dn/current, StorageType: DISK         

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-ac7f14dd-165b-4134-bcfd-6b5dac89af5e 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /mnt/data3/dfs/dn/current, StorageType: DISK         

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean                                                     

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Volume reference is released.  

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1452924002-IP1-1527602017139                 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1452924002-IP1-1527602017139 on volume /mnt/data2/dfs/dn/current...                 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1452924002-IP1-1527602017139 on volume /mnt/data1/dfs/dn/current...                 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1452924002-IP1-1527602017139 on volume /mnt/data3/dfs/dn/current...                 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Cached dfsUsed found for /mnt/data1/dfs/dn/current/BP-1452924002-IP1-1527602017139/current: 10567324426240 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Cached dfsUsed found for /mnt/data2/dfs/dn/current/BP-1452924002-IP1-1527602017139/current: 10610305351680 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Cached dfsUsed found for /mnt/data3/dfs/dn/current/BP-1452924002-IP1-1527602017139/current: 10560120340480 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1452924002-IP1-1527602017139 on /mnt/data1/dfs/dn/current: 24ms           

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1452924002-IP1-1527602017139 on /mnt/data3/dfs/dn/current: 23ms           

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1452924002-IP1-1527602017139 on /mnt/data2/dfs/dn/current: 24ms           

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1452924002-IP1-1527602017139: 26ms                           

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-1452924002-IP1-1527602017139 on volume /mnt/data1/dfs/dn/current...

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-1452924002-IP1-1527602017139 on volume /mnt/data3/dfs/dn/current...

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-1452924002-IP1-1527602017139 on volume /mnt/data2/dfs/dn/current...

当日志打到这时,该datnode基本就完成block的加载了,状态也就恢复正常了。

你可能感兴趣的:(hadoop2 HA启动时Namenode均进入安全模式,没有出现active的Namenode)