Hadoop distcp命令遇到的异常及解决方案

1 异常信息

Caused by: java.io.IOException: Mismatch in length of source:hdfs://xxx and target:hdfs://xxx

2 原因

需要远程复制的文件没有关闭，还处于写的状态。

3 解决方案：

1） 检查文件状态

hdfs fsck hdfs://10.10.10.10:80/flume/xxx/xxxxxxxx/day=2018-03-12/xxx.2018-03-12.1520841735508

Connecting to namenode via http://xxx:50070/fsck?ugi=hadoop&path=%2Fflume%2Fxxx%xxx%2Fday%3D2018-03-12%xxx.2018-03-12.1520841735508

FSCK started by hadoop (auth:SIMPLE) from /139.5.108.244 for path /flume/xxx/xxx/day=2018-03-12/xxx.2018-03-12.1520841735508 at Tue Mar 13 16:12:39 CST 2018

Status: HEALTHY

Total size: 0 B (Total open files size: 420 B)

Total dirs: 0

Total files: 0

Total symlinks: 0 (Files currently being written: 1)

Total blocks (validated): 0 (Total open file blocks (not validated): 1)

Minimally replicated blocks: 0

Over-replicated blocks: 0

Under-replicated blocks: 0

Mis-replicated blocks: 0

Default replication factor: 3

Average block replication: 0.0

Corrupt blocks: 0

Missing replicas: 0

Number of data-nodes: 8

Number of racks: 1

FSCK ended at Tue Mar 13 16:12:39 CST 2018 in 2 milliseconds

The filesystem under path '/flume/xxx/xxx/day=2018-03-12/xxx.2018-03-12.1520841735508' is HEALTHY

2） 关闭文件

hdfs debug recoverLease -path hdfs://10.10.10.10:8020/flume/xxx/xxx/day=2018-03-12/xxx.2018-03-12.1520841735508

可能会失败：

recoverLease returned false.

Giving up on recoverLease for hdfs://10.10.10.10:8020/flume/xxx/xxx/day=2018-03-12/xxx.2018-03-12.1520841735508 after 1 try.

尝试再次关闭：

recoverLease SUCCEEDED on hdfs:// 10.10.10.10:8020/flume/xxx/xxx/day=2018-03-12/xxx.2018-03-12.1520841735508

3）再次检查状态

Connecting to namenode via http://xxx:50070/fsck?ugi=hadoop&path=%2Fflume%2Fxxx%xxx%2Fday%3D2018-03-12%xxx.2018-03-12.1520841735508

FSCK started by hadoop (auth:SIMPLE) from /xx.xx.xx.xx for path /flume/xxx/xxx/day=2018-03-12/xxx.2018-03-12.1520841735508 at Tue Mar 13 16:19:57 CST 2018

.Status: HEALTHY

Total size: 838 B

Total dirs: 0

Total files: 1

Total symlinks: 0

Total blocks (validated): 1 (avg. block size 838 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 0 (0.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 3

Average block replication: 3.0

Corrupt blocks: 0

Missing replicas: 0 (0.0 %)

Number of data-nodes: 8

Number of racks: 1

FSCK ended at Tue Mar 13 16:19:57 CST 2018 in 2 milliseconds

The filesystem under path '/flume/xxx/xxx/day=2018-03-12/xxx.2018-03-12.1520841735508' is HEALTHY

4） 再次复制

hadoop distcp -bandwidth 15 -m 50 -pb hdfs://10.10.10.10:8020//flume/xxx/xxx/day=2018-03-12 /flume/xxx/xxx/day=2018-03-12

* 注意：在使用该命令时最好指定带宽限制（-bandwidth），同时拷贝的最大数目（-m）。我在首次迁移数据时没有设置，一次性迁移了好几个月的数据，导致流量超标。