[解决方案]org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block

[解决方案]org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block...

  • 报错情况
  • 查看受损block
  • 有可能当前路径下的受损block不止一个
  • 删除损坏的block

报错情况

Caused by: java.io.IOException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: xxxxxxxxxxxxxxxxxxxx file=/path/xxxx/xxxxx/xxxxxx/xxxxxxx
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:888)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:568)

查看受损block

hadoop fsck /你的path  将报错中file后面的路径粘贴过来即可

运行后会显示受损块文件的相关信息
/你的path: MISSING 1 blocks of total size 69 B.Status: CORRUPT
 Total size:	69 B
 Total dirs:	0
 Total files:	1
 Total symlinks:		0
 Total blocks (validated):	1 (avg. block size 69 B)
  ********************************
  CORRUPT FILES:	1
  MISSING BLOCKS:	1
  MISSING SIZE:		69 B
  CORRUPT BLOCKS: 	1
  ********************************
 Minimally replicated blocks:	0 (0.0 %)
 Over-replicated blocks:	0 (0.0 %)
 Under-replicated blocks:	0 (0.0 %)
 Mis-replicated blocks:		1 (100.0 %)
 Default replication factor:	3
 Average block replication:	0.0
 Corrupt blocks:		1
 Missing replicas:		0
 Number of data-nodes:		31
 Number of racks:		2
FSCK ended at Thu Apr 14 13:37:15 CST 2022 in 25 milliseconds


The filesystem under path '/你的path' is CORRUPT    
CORRUPT说明文件受损

有可能当前路径下的受损block不止一个

hadoop fsck /path -list-corruptfileblocks  
运行后可以输出当前路径下所有的受损文件块

删除损坏的block

hadoop fsck /path -delete

你可能感兴趣的:(笔记,hadoop)