hadoop 3.x 案例3:datanode 异常

一. 问题描述

hdfs web页面显示有的文件是损坏的，需要使用fsck来检查

image.png

二. 解决方案

通过fsck检查根目录，看是否有损坏的文件

hdfs fsck /

image.png

将检查到的损坏的文件进行删除

hdfs fsck /spark-jars -move
hdfs fsck /spark-jars -delete
hdfs fsck /spark-jars -files

测试记录:

[root@hadoop1 shell]# hdfs fsck /spark-jars -move
Connecting to namenode via http://hadoop1:9870/fsck?ugi=root&move=1&path=%2Fspark-jars
FSCK started by root (auth:SIMPLE) from /192.168.0.150 for path /spark-jars at Sun Nov 20 21:15:14 CST 2022


/spark-jars/JLargeArrays-1.5.jar: MISSING 1 blocks of total size 232470 B.
/spark-jars/JTransforms-3.1.jar: MISSING 1 blocks of total size 1175798 B.
/spark-jars/RoaringBitmap-0.9.0.jar: MISSING 1 blocks of total size 386529 B.
/spark-jars/aircompressor-0.21.jar: MISSING 1 blocks of total size 183588 B.
/spark-jars/algebra_2.12-2.0.1.jar: MISSING 1 blocks of total size 1168417 B.
/spark-jars/annotations-17.0.0.jar: MISSING 1 blocks of total size 19014 B.
/spark-jars/antlr4-runtime-4.8.jar: MISSING 1 blocks of total size 337864 B.
/spark-jars/aopalliance-repackaged-2.6.1.jar: MISSING 1 blocks of total size 27006 B.
/spark-jars/arpack-2.2.1.jar: MISSING 1 blocks of total size 77537 B.
/spark-jars/arpack_combined_all-0.1.jar: MISSING 1 blocks of total size 1194003 B.
/spark-jars/arrow-format-2.0.0.jar: MISSING 1 blocks of total size 72668 B.
......
......
......
/spark-jars/tink-1.6.0.jar: MISSING 1 blocks of total size 1323184 B.
/spark-jars/univocity-parsers-2.9.1.jar: MISSING 1 blocks of total size 447005 B.
/spark-jars/xbean-asm9-shaded-4.20.jar: MISSING 1 blocks of total size 294763 B.
/spark-jars/xz-1.8.jar: MISSING 1 blocks of total size 108555 B.
/spark-jars/zjsonpatch-0.3.0.jar: MISSING 1 blocks of total size 35518 B.
/spark-jars/zstd-jni-1.5.0-4.jar: MISSING 1 blocks of total size 6768681 B.
Status: CORRUPT
 Number of data-nodes:  1
 Number of racks:               1
 Total dirs:                    1
 Total symlinks:                0

Replicated Blocks:
 Total size:    199191055 B
 Total files:   170
 Total blocks (validated):      170 (avg. block size 1171712 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:      170 (100.0 %)
  MINIMAL BLOCK REPLICATION:    1
  CORRUPT FILES:        170
  MISSING BLOCKS:       170
  MISSING SIZE:         199191055 B
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     0.0
 Missing blocks:                170
 Corrupt blocks:                0
 Missing replicas:              0
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Sun Nov 20 21:15:15 CST 2022 in 1408 milliseconds


The filesystem under path '/spark-jars' is CORRUPT
[root@hadoop1 shell]# hdfs fsck /spark-jars -delete
Connecting to namenode via http://hadoop1:9870/fsck?ugi=root&delete=1&path=%2Fspark-jars
FSCK started by root (auth:SIMPLE) from /192.168.0.150 for path /spark-jars at Sun Nov 20 21:15:59 CST 2022


/spark-jars/JLargeArrays-1.5.jar: MISSING 1 blocks of total size 232470 B.
......
/spark-jars/zstd-jni-1.5.0-4.jar: MISSING 1 blocks of total size 6768681 B.
Status: CORRUPT
 Number of data-nodes:  1
 Number of racks:               1
 Total dirs:                    1
 Total symlinks:                0

Replicated Blocks:
 Total size:    199191055 B
 Total files:   170
 Total blocks (validated):      170 (avg. block size 1171712 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:      170 (100.0 %)
  MINIMAL BLOCK REPLICATION:    1
  CORRUPT FILES:        170
  MISSING BLOCKS:       170
  MISSING SIZE:         199191055 B
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     0.0
 Missing blocks:                170
 Corrupt blocks:                0
 Missing replicas:              0
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Sun Nov 20 21:16:00 CST 2022 in 755 milliseconds


The filesystem under path '/spark-jars' is CORRUPT
[root@hadoop1 shell]# 
[root@hadoop1 shell]# 
[root@hadoop1 shell]# 
[root@hadoop1 shell]# hdfs fsck /spark-jars -files
Connecting to namenode via http://hadoop1:9870/fsck?ugi=root&files=1&path=%2Fspark-jars
FSCK started by root (auth:SIMPLE) from /192.168.0.150 for path /spark-jars at Sun Nov 20 21:16:35 CST 2022

/spark-jars 

Status: HEALTHY
 Number of data-nodes:  1
 Number of racks:               1
 Total dirs:                    1
 Total symlinks:                0

Replicated Blocks:
 Total size:    0 B
 Total files:   0
 Total blocks (validated):      0
 Minimally replicated blocks:   0
 Over-replicated blocks:        0
 Under-replicated blocks:       0
 Mis-replicated blocks:         0
 Default replication factor:    3
 Average block replication:     0.0
 Missing blocks:                0
 Corrupt blocks:                0
 Missing replicas:              0
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Sun Nov 20 21:16:35 CST 2022 in 0 milliseconds


The filesystem under path '/spark-jars' is HEALTHY
[root@hadoop1 shell]# 

参考:

1. https://blog.csdn.net/aohuang8877/article/details/101116099