HDFS源码解析---DataTransferThrottler（流量控制）

简介

        DataTransferThrottler类别Datanode读取和写入数据时控制传输数据速率。这个类是线程安全的，它可以由多个线程共享。

        用途是构建DataTransferThrottler对象，并设置期限period和带宽bandwidthPerSec，际读写前调用DataTransferThrottler.throttle()方法。假设I/O的速率相对给定的带宽太快，则该方法会将当前线程wait。

构造函数

• 双參构造函数，能够设置周期period和带宽bandwidthPerSec。

  /**
     * Constructor
     * @param period in milliseconds. Bandwidth is enforced over this
     *        period.
     * @param bandwidthPerSec bandwidth allowed in bytes per second. 
     */
    public DataTransferThrottler(long period, long bandwidthPerSec) {
      this.curPeriodStart = monotonicNow();
      this.period = period;
      this.curReserve = this.bytesPerPeriod = bandwidthPerSec*period/1000;
      this.periodExtension = period*3;
    }

• 单參构造函数，能够设置带宽bandwidthPerSec, 周期period默认被设置为500ms。

    /**
     * @return current throttle bandwidth in bytes per second.
     */
    public synchronized long getBandwidth() {
      return bytesPerPeriod*1000/period;
    }

   

重要属性

period	周期，单位毫秒
periodExtension	周期扩展时间。单位毫秒
bytesPerPeriod	一个周期内能够发送/接收的byte总数
curPeriodStart	当前周期开始时间，单位毫秒
curReserve	当前周期内还能够发送/接收的byte数
bytesAlreadyUsed	当前周期内已经使用的byte数

DataTransferThrottler.throttle()

        DataTransferThrottler.throttle()方法会循环判断请求发送数据量是否比剩余数据量小， 如
果throttle()方法能满足请求量则立即返回，调用线程就可以立即执行发送／ 接收数据的操作； 否则， DataTransferThrottler.throttle()方法会循环等待检查周期结束， 并在每个周期结束时增加剩余请求量（每个周期添加bytesPerPeriod） ， 直到当前请求量得到满足时throttle()方法才会退出。

/** Given the numOfBytes sent/received since last time throttle was called,
   * make the current thread sleep if I/O rate is too fast
   * compared to the given bandwidth.  Allows for optional external cancelation.
   *
   * @param numOfBytes
   *     number of bytes sent/received since last time throttle was called
   * @param canceler
   *     optional canceler to check for abort of throttle
   */
  public synchronized long throttle(long numOfBytes, Canceler canceler) {
    if ( numOfBytes <= 0 ) {
      return 0;
    }

    long currentWaitTime = 0;
    // 当前周期余量减去须要发送/接收的byte数numOfBytes
    curReserve -= numOfBytes;
    bytesAlreadyUsed += numOfBytes;
    
    // 假设curReserve小于等于0，则说明当前周期已经没有余量，要开始限流了！
    while (curReserve <= 0) {
      // 假设传入了有效取消器canceler，而且取消器的取消状态isCancelled是true，则直接退出while循环
      if (canceler != null && canceler.isCancelled()) {
        return currentWaitTime;
      }
      long now = monotonicNow();
      // 计算当前周期结束时间。并存放在curPeriodEnd变量中
      long curPeriodEnd = curPeriodStart + period;

      if ( now < curPeriodEnd ) {
        // Wait for next period so that curReserve can be increased.
        // 等待下一个周期，这样curReserve就能够添加
        totalWaitCount.incrementAndGet();
        long start = Time.monotonicNow();
        try {
          wait( curPeriodEnd - now );
        } catch (InterruptedException e) {
          // Abort throttle and reset interrupted status to make sure other
          // interrupt handling higher in the call stack executes.
          // 终止throttle, 而且重置interrupted状态来确保在调用栈中其他interrupt处理器能够正确运行
          Thread.currentThread().interrupt();
          break;
        } finally {
          long wait = Time.monotonicNow() - start;
          currentWaitTime += wait;
          totalWaitTime.addAndGet(wait);
        }
      // 假设当前时间now比当前结束时间curPeriodEnd晚，而且小于curPeriodStart+periodExtension(周期3倍时间)，则进入下一个周期
      // 并添加bytesPerPeriod到curReserve   
      } else if ( now <  (curPeriodStart + periodExtension)) {
        curPeriodStart = curPeriodEnd;
        // 增加当前周期余量
        curReserve += bytesPerPeriod;
      // 假设当前时间now大于curPeriodStart+periodExtension，则可能Throttler非常长时间没有使用。抛弃上一个周期
      } else {
        // discard the prev period. Throttler might not have
        // been used for a long time.
        curPeriodStart = now;
        curReserve = bytesPerPeriod - bytesAlreadyUsed;
      }
    }

    bytesAlreadyUsed -= numOfBytes;
    return currentWaitTime;
  }