提高Hadoop Balancer 迁移block速度的方法

如何提高Hadoop Balancer迁移block的速度？

1）增加DataNode用于balancer的bandwidth。

dfs.datanode.balance.bandwidthPerSec
52428800

这个值是DataXceiverServer上BlockBalanceThrottler控制的带宽大小。该单位是Byte，如果机器的网卡和交换机的带宽有限，可以适当降低该速度。Hadoop系统默认是1048576 （1MB）。

2）增加DataNode上转移block的Xceiver的个数上限。
DataNode上同时用于balancer的Xceiver的个数受到了BlockBananceThrottler限制。可以适当调大如下的配置。

dfs.datanode.balance.max.concurrent.moves
50

这个值默认是5。如果仅仅在Balancer的hdfs-site.xml修改配置而没有修改DataNode下的配置，Balancer会抛出如下的WARN LOG：

2015-06-18 15:54:24,253 WARN org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move blk_1366768180_1100055981849 with size=134217728 from 172.22.6.25:1004:DISK to 172.22.5.23:1004:DISK through 172.22.5.99:1004: block move is failed: Not able to receive block 1366768180 from /172.22.6.5:33544 because threads quota is exceeded.

查看DataXceiverServer，如果同时执行Balancer的Xceiver的个数upperlimit是5个，将DataNode上这个参数调大才可以增加迁移Block的速度。
附带SourceCode：

/** Check if the block move can start.

*

* Return true if the thread quota is not exceeded and

* the counter is incremented; False otherwise.

*/

synchronizedboolean acquire() {

if (numThreads >= maxThreads) {

returnfalse;

}

numThreads++;

returntrue;

}

(这里maxThreads 就是’dfs.datanode.balance.max.concurrent.moves’控制的)

From Binospace, post 提高Hadoop Balancer 迁移block速度的方法

文章的脚注信息由WordPress的 wp-posturl插件自动生成

Copyright © 2008
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)