一. 问题描述
hdfs web页面显示有的文件是损坏的，需要使用fsck来检查
image.png
二. 解决方案
通过fsck检查根目录，看是否有损坏的文件
hdfs fsck /
image.png
将检查到的损坏的文件进行删除
hdfs fsck /spark-jars -move
hdfs fsck /spark-jars -delete
hdfs fsck /spark-jars -files
测试记录:
[root@hadoop1 shell]# hdfs fsck /spark-jars -move
Connecting to namenode via http://hadoop1:9870/fsck?ugi=root&move=1&path=%2Fspark-jars
FSCK started by root (auth:SIMPLE) from /192.168.0.150 for path /spark-jars at Sun Nov 20 21:15:14 CST 2022
/spark-jars/JLargeArrays-1.5.jar: MISSING 1 blocks of total size 232470 B.
/spark-jars/JTransforms-3.1.jar: MISSING 1 blocks of total size 1175798 B.
/spark-jars/RoaringBitmap-0.9.0.jar: MISSING 1 blocks of total size 386529 B.
/spark-jars/aircompressor-0.21.jar: MISSING 1 blocks of total size 183588 B.
/spark-jars/algebra_2.12-2.0.1.jar: MISSING 1 blocks of total size 1168417 B.
/spark-jars/annotations-17.0.0.jar: MISSING 1 blocks of total size 19014 B.
/spark-jars/antlr4-runtime-4.8.jar: MISSING 1 blocks of total size 337864 B.
/spark-jars/aopalliance-repackaged-2.6.1.jar: MISSING 1 blocks of total size 27006 B.
/spark-jars/arpack-2.2.1.jar: MISSING 1 blocks of total size 77537 B.
/spark-jars/arpack_combined_all-0.1.jar: MISSING 1 blocks of total size 1194003 B.
/spark-jars/arrow-format-2.0.0.jar: MISSING 1 blocks of total size 72668 B.
......
......
......
/spark-jars/tink-1.6.0.jar: MISSING 1 blocks of total size 1323184 B.
/spark-jars/univocity-parsers-2.9.1.jar: MISSING 1 blocks of total size 447005 B.
/spark-jars/xbean-asm9-shaded-4.20.jar: MISSING 1 blocks of total size 294763 B.
/spark-jars/xz-1.8.jar: MISSING 1 blocks of total size 108555 B.
/spark-jars/zjsonpatch-0.3.0.jar: MISSING 1 blocks of total size 35518 B.
/spark-jars/zstd-jni-1.5.0-4.jar: MISSING 1 blocks of total size 6768681 B.
Status: CORRUPT
Number of data-nodes: 1
Number of racks: 1
Total dirs: 1
Total symlinks: 0
Replicated Blocks:
Total size: 199191055 B
Total files: 170
Total blocks (validated): 170 (avg. block size 1171712 B)
********************************
UNDER MIN REPL'D BLOCKS: 170 (100.0 %)
MINIMAL BLOCK REPLICATION: 1
CORRUPT FILES: 170
MISSING BLOCKS: 170
MISSING SIZE: 199191055 B
********************************
Minimally replicated blocks: 0 (0.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 0.0
Missing blocks: 170
Corrupt blocks: 0
Missing replicas: 0
Blocks queued for replication: 0
Erasure Coded Block Groups:
Total size: 0 B
Total files: 0
Total block groups (validated): 0
Minimally erasure-coded block groups: 0
Over-erasure-coded block groups: 0
Under-erasure-coded block groups: 0
Unsatisfactory placement block groups: 0
Average block group size: 0.0
Missing block groups: 0
Corrupt block groups: 0
Missing internal blocks: 0
Blocks queued for replication: 0
FSCK ended at Sun Nov 20 21:15:15 CST 2022 in 1408 milliseconds
The filesystem under path '/spark-jars' is CORRUPT
[root@hadoop1 shell]# hdfs fsck /spark-jars -delete
Connecting to namenode via http://hadoop1:9870/fsck?ugi=root&delete=1&path=%2Fspark-jars
FSCK started by root (auth:SIMPLE) from /192.168.0.150 for path /spark-jars at Sun Nov 20 21:15:59 CST 2022
/spark-jars/JLargeArrays-1.5.jar: MISSING 1 blocks of total size 232470 B.
......
/spark-jars/zstd-jni-1.5.0-4.jar: MISSING 1 blocks of total size 6768681 B.
Status: CORRUPT
Number of data-nodes: 1
Number of racks: 1
Total dirs: 1
Total symlinks: 0
Replicated Blocks:
Total size: 199191055 B
Total files: 170
Total blocks (validated): 170 (avg. block size 1171712 B)
********************************
UNDER MIN REPL'D BLOCKS: 170 (100.0 %)
MINIMAL BLOCK REPLICATION: 1
CORRUPT FILES: 170
MISSING BLOCKS: 170
MISSING SIZE: 199191055 B
********************************
Minimally replicated blocks: 0 (0.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 0.0
Missing blocks: 170
Corrupt blocks: 0
Missing replicas: 0
Blocks queued for replication: 0
Erasure Coded Block Groups:
Total size: 0 B
Total files: 0
Total block groups (validated): 0
Minimally erasure-coded block groups: 0
Over-erasure-coded block groups: 0
Under-erasure-coded block groups: 0
Unsatisfactory placement block groups: 0
Average block group size: 0.0
Missing block groups: 0
Corrupt block groups: 0
Missing internal blocks: 0
Blocks queued for replication: 0
FSCK ended at Sun Nov 20 21:16:00 CST 2022 in 755 milliseconds
The filesystem under path '/spark-jars' is CORRUPT
[root@hadoop1 shell]#
[root@hadoop1 shell]#
[root@hadoop1 shell]#
[root@hadoop1 shell]# hdfs fsck /spark-jars -files
Connecting to namenode via http://hadoop1:9870/fsck?ugi=root&files=1&path=%2Fspark-jars
FSCK started by root (auth:SIMPLE) from /192.168.0.150 for path /spark-jars at Sun Nov 20 21:16:35 CST 2022
/spark-jars
Status: HEALTHY
Number of data-nodes: 1
Number of racks: 1
Total dirs: 1
Total symlinks: 0
Replicated Blocks:
Total size: 0 B
Total files: 0
Total blocks (validated): 0
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 3
Average block replication: 0.0
Missing blocks: 0
Corrupt blocks: 0
Missing replicas: 0
Blocks queued for replication: 0
Erasure Coded Block Groups:
Total size: 0 B
Total files: 0
Total block groups (validated): 0
Minimally erasure-coded block groups: 0
Over-erasure-coded block groups: 0
Under-erasure-coded block groups: 0
Unsatisfactory placement block groups: 0
Average block group size: 0.0
Missing block groups: 0
Corrupt block groups: 0
Missing internal blocks: 0
Blocks queued for replication: 0
FSCK ended at Sun Nov 20 21:16:35 CST 2022 in 0 milliseconds
The filesystem under path '/spark-jars' is HEALTHY
[root@hadoop1 shell]#
参考:
- https://blog.csdn.net/aohuang8877/article/details/101116099