HDFS block丢失过多进入安全模式（safe mode）的解决方法

Safe mode is ON. The reported blocks 3 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 5. The number of live datanodes 2 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

说明我们的损坏的文件比例超过了阈值, 这个阈值配置在hdfs中,  也就是说不允许任何一个块损坏掉. 如果我们配置成99%应该就不会触发safemode了.

HDFS block丢失过多进入安全模式（Safe mode）的解决方法

背景及现象描述(Background and Symptom)

 因磁盘空间不足，内存不足，系统掉电等其他原因导致dataNode datablock丢失，出现如下类似日志：

The number of live datanodes 3 has reached the minimum number 0.

Safe mode will be turned off automatically once the thresholds have been reached.

Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled.

 Name node is in safe mode.

The reported blocks 632758 needs additional 5114 blocks to reach the threshold 0.9990

of total blocks 638510.

The number of live datanodes 3 has reached the minimum number 0.

Safe mode will be turned off automatically once the thresholds have been reached.

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode

(FSNamesystem.java:1209)

    ... 12 more

 

 

原因分析(Cause Analysis)*

由于系统断电，内存不足等原因导致dataNode丢失超过设置的丢失百分比，系统自动进入安全模式

解决办法(Solution)*

安装HDFS客户端，并执行如下命令：

                                步骤 1     执行命令退出安全模式：hadoop dfsadmin -safemode leave

                                步骤 2     执行健康检查，删除损坏掉的block。  hdfs fsck  /  -delete

 

注意: 这种方式会出现数据丢失，损坏的block会被删掉