提高RPC Server throughput的请求延时回复处理

前言

---

在一套完整的分布式系统中，client端向server端发起一个请求，然后client等待此请求被server端处理完毕，然后接受到serve的返回结果。自此一个请求就算作是被处理完了。这种block等待处理结果的请求处理行为在我们日常的系统中十分的常见。但是这种处理方式的一个明显弊端是，未处理完成的请求势必会占住server端的处理资源。因此一般常见的改进做法是提高server端的Handler数量，来提高服务端的请求并发处理能力，这种做法是比较简单直接的。但其实这里还有另外一个方向点的优化，是否能够提高单个请求的处理时间来做优化呢？比如一些已经被处理完毕的请求，但是正处于返回response结果的，这也是会占着Handler资源的。因为返回response操作也是在请求被处理环节的一部分。假设说我们能将回复请求的阶段从处理请求方法中拆分出去，通过延时返回的方式，毫无疑问，这也会在一定程度上提高server端的throughput。本文笔者来聊聊关于RPC Server请求的回复延时处理以及Hadoop RPC Server是如何做这部分优化的。

Server端延时请求回复的优劣势

---

按照前面我们所说的，如果将server端的<请求即刻处理->请求即时回复>变为<请求即刻处理->请求延时恢复>，它会给整个系统带来怎样的变化呢？

以下是其所带来的好的一面和不好的一面：

优势：
增大系统整体的throughput，因为请求结果回复变为了延时异步的方式，这相当于提早释放了Handler的资源，让Handler能够马上接下来处理其它客户端的请求。

弊端：

• 请求回复的延时处理意味着server端于client的open connection会增多，因为client的请求回复还没有被执行，这些连接并没有被关闭。
• Server端因为此改动接受了过多的请求，导致请求回复阶段成为bottleneck。

Hadoop RPC Server的请求延时回复处理

---

下面我们通过Hadoop RPC Server内部目前已经优化了的请求回复处理例子，来具体了解下这部分的处理逻辑。

样例一：增加延时请求计数的处理

这个改动源自Hadoop社区JIRA：HADOOP-10300：Allowed deferred sending of call responses。

它的一个主要思路是这样的：

1） 在每个PRC call里面多添加了一个请求等待的计数值，初始值为1
2）正常情况下，Server端在处理完请求后，会执行sendResponse方法，然后会将上述计数值做减1操作，然后执行请求回复操作。
3）但是，如果我们想要做请求的延时回复处理，我们可以额外执行一个postponeResponse的方法来增大请求回复等待的计数值。这样的话，在正常逻辑中的sendResponse则不会实际执行请求回复操作，它只做计数值的减操作。只有再第二次Server被触发执行了sendResponse后，才会执行请求回复操作。

相关代码如下：

/** 请求回复等待计数值 */
private AtomicInteger responseWaitCount = new AtomicInteger(1);

...

 
/**
 * Allow a IPC response to be postponed instead of sent immediately
 * after the handler returns from the proxy method.  The intended use
 * case is freeing up the handler thread when the response is known,
 * but an expensive pre-condition must be satisfied before it's sent
 * to the client.
 */
@InterfaceStability.Unstable
@InterfaceAudience.LimitedPrivate({"HDFS"})
public void postponeResponse() {
  // 执行延时回复处理，将计数值加1
  int count = responseWaitCount.incrementAndGet();
  assert count > 0 : "response has already been sent";
}

@InterfaceStability.Unstable
@InterfaceAudience.LimitedPrivate({"HDFS"})
public void sendResponse() throws IOException {
  // 执行请求回复操作时，减小计数值
  int count = responseWaitCount.decrementAndGet();
  assert count >= 0 : "response has already been sent";
  // 如果计数值为0了，则进行实际回复返回操作，否则不进行response信息的返回
  if (count == 0) {
    assert rpcResponse != null : "response has not been set";
    connection.sendResponse(this);
  }
}

下面是对应的testcase：

// Test that IPC calls can be marked for a deferred response.
// call 0: immediate
// call 1: immediate
// call 2: delayed with wait for 1 sendResponse, check if blocked
// call 3: immediate, proves handler is freed
// call 4: delayed with wait for 2 sendResponses, check if blocked
// call 2: sendResponse, should return
// call 4: sendResponse, should remain blocked
// call 5: immediate, prove handler is still free
// call 4: sendResponse, expect it to return
@Test(timeout=10000)
public void testDeferResponse() throws IOException, InterruptedException {
  final AtomicReference<Call> deferredCall = new AtomicReference<Call>();
  final AtomicInteger count = new AtomicInteger();
  final Writable wait0 = new IntWritable(0);
  final Writable wait1 = new IntWritable(1);
  final Writable wait2 = new IntWritable(2);
  
  // use only 1 handler to prove it's freed after every call
  Server server = new Server(ADDRESS, 0, IntWritable.class, 1, conf){
    @Override
    public Writable call(RPC.RpcKind rpcKind, String protocol,
        Writable waitCount, long receiveTime) throws IOException {
      Call call = Server.getCurCall().get();
      int wait = ((IntWritable)waitCount).get();
      // 根据传入的wait次数值，做postponeResponse的处理，意为这个call需要做额外对应次数的sendResponse方法才会有结果返回
      while (wait-- > 0) {
        call.postponeResponse();
        deferredCall.set(call);
      }
      return new IntWritable(count.getAndIncrement());
    }
  };
  server.start();
  
  final InetSocketAddress address = NetUtils.getConnectAddress(server);
  final Client client = new Client(IntWritable.class, conf);
  Call[] waitingCalls = new Call[2];
      
  // calls should return immediately, check the sequence number is
  // increasing
  assertEquals(0,
      ((IntWritable)client.call(wait0, address)).get());
  assertEquals(1,
      ((IntWritable)client.call(wait0, address)).get());
  
  // do a call in the background that will have a deferred response
  final ExecutorService exec = Executors.newCachedThreadPool();
  Future<Integer> future1 = exec.submit(new Callable<Integer>() {
    @Override
    public Integer call() throws IOException {
      return ((IntWritable)client.call(wait1, address)).get();
    }
  });
  // make sure it blocked 
  try {
    future1.get(1, TimeUnit.SECONDS);
    Assert.fail("ipc shouldn't have responded");
  } catch (TimeoutException te) {
    // ignore, expected
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  assertFalse(future1.isDone());
  waitingCalls[0] = deferredCall.get();
  assertNotNull(waitingCalls[0]);  // proves the handler isn't tied up, and that the prior sequence number
  // was consumed
  assertEquals(3,
      ((IntWritable)client.call(wait0, address)).get());  // another call with wait count of 2
  Future<Integer> future2 = exec.submit(new Callable<Integer>() {
    @Override
    public Integer call() throws IOException {
      return ((IntWritable)client.call(wait2, address)).get();
    }
  });
  // make sure it blocked 
  try {
    future2.get(1, TimeUnit.SECONDS);
    Assert.fail("ipc shouldn't have responded");
  } catch (TimeoutException te) {
    // ignore, expected
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  assertFalse(future2.isDone());
  waitingCalls[1] = deferredCall.get();
  assertNotNull(waitingCalls[1]);  // the background calls should still be blocked
  assertFalse(future1.isDone());
  assertFalse(future2.isDone());  // trigger responses
  waitingCalls[0].sendResponse();
  waitingCalls[1].sendResponse();
  try {
    int val = future1.get(1, TimeUnit.SECONDS);
    assertEquals(2, val);
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }  // make sure it's still blocked 
  try {
    future2.get(1, TimeUnit.SECONDS);
    Assert.fail("ipc shouldn't have responded");
  } catch (TimeoutException te) {
    // ignore, expected
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  assertFalse(future2.isDone());  // call should return immediately
  assertEquals(5,
      ((IntWritable)client.call(wait0, address)).get());  // trigger last waiting call
  waitingCalls[1].sendResponse();
  try {
    int val = future2.get(1, TimeUnit.SECONDS);
    assertEquals(4, val);
  } catch (Exception ex) {
    Assert.fail("unexpected exception:"+ex);
  }
  
  server.stop();
}

样例二：在RPC Call内添加延时回复标记

---

第二种改动方法相对就比较直接了，直接在RPC call内标明是是否需要将此RPC call的请求回复行为变为延时回复模式。

此优化源自Hadoop社区JIRA HADOOP-11552：Allow handoff on the server side for RPC requests。

相关核心改动如下：

/** A generic call queued for handling. */
public static class Call implements Schedulable,
PrivilegedExceptionAction<Void> {
...
// 是否需要做请求延时回复的标记
rivate boolean deferredResponse = false;
...

@InterfaceStability.Unstable
public void deferResponse() {
  this.deferredResponse = true;
}

@InterfaceStability.Unstable
public boolean isResponseDeferred() {
  return this.deferredResponse;
}

// 以下两个方法在实际使用中需要被覆写
// 延时返回正常结果方法
public void setDeferredResponse(Writable response) {
}

// 延时返回错误response结果
public void setDeferredError(Throwable t) {
}

请求call的处理方法

    @Override
    public Void run() throws Exception {
      ...
      try {
        // 1）执行请求处理操作，并得到结果值
        value = call(
            rpcKind, connection.protocolName, rpcRequest, timestampNanos);
      } catch (Throwable e) {
        populateResponseParamsOnError(e, responseParams);
      }
      // 2）如果不需要做延时回复处理的话
      if (!isResponseDeferred()) {
        long deltaNanos = Time.monotonicNowNanos() - startNanos;
        ProcessingDetails details = getProcessingDetails();

        details.set(Timing.PROCESSING, deltaNanos, TimeUnit.NANOSECONDS);
        deltaNanos -= details.get(Timing.LOCKWAIT, TimeUnit.NANOSECONDS);
        deltaNanos -= details.get(Timing.LOCKSHARED, TimeUnit.NANOSECONDS);
        deltaNanos -= details.get(Timing.LOCKEXCLUSIVE, TimeUnit.NANOSECONDS);
        details.set(Timing.LOCKFREE, deltaNanos, TimeUnit.NANOSECONDS);
        startNanos = Time.monotonicNowNanos();

        // 3）设置请求response信息并返回response
        setResponseFields(value, responseParams);
        sendResponse();

        deltaNanos = Time.monotonicNowNanos() - startNanos;
        details.set(Timing.RESPONSE, deltaNanos, TimeUnit.NANOSECONDS);
      } else {
        if (LOG.isDebugEnabled()) {
          LOG.debug("Deferring response for callId: " + this.callId);
        }
      }
      // 否则是延时返回处理，直接返回此方法
      return null;
    }

以下是一个简单的延时回复处理Server的testcase：

public class TestRpcServerHandoff {

  public static final Log LOG =
      LogFactory.getLog(TestRpcServerHandoff.class);

  private static final String BIND_ADDRESS = "0.0.0.0";
  private static final Configuration conf = new Configuration();


  public static class ServerForHandoffTest extends Server {

    private final AtomicBoolean invoked = new AtomicBoolean(false);
    private final ReentrantLock lock = new ReentrantLock();
    private final Condition invokedCondition = lock.newCondition();

    private volatile Writable request;
    private volatile Call deferredCall;

    protected ServerForHandoffTest(int handlerCount) throws IOException {
      super(BIND_ADDRESS, 0, BytesWritable.class, handlerCount, conf);
    }

    @Override
    public Writable call(RPC.RpcKind rpcKind, String protocol, Writable param,
                         long receiveTime) throws Exception {
      request = param;
      deferredCall = Server.getCurCall().get();
      Server.getCurCall().get().deferResponse();
      lock.lock();
      try {
        invoked.set(true);
        invokedCondition.signal();
      } finally {
        lock.unlock();
      }
      return null;
    }

    void awaitInvocation() throws InterruptedException {
      lock.lock();
      try {
        while (!invoked.get()) {
          invokedCondition.await();
        }
      } finally {
        lock.unlock();
      }
    }

    void sendResponse() {
      deferredCall.setDeferredResponse(request);
    }

    void sendError() {
      deferredCall.setDeferredError(new IOException("DeferredError"));
    }
  }

  @Test(timeout = 10000)
  public void testDeferredResponse() throws IOException, InterruptedException,
      ExecutionException {


    ServerForHandoffTest server = new ServerForHandoffTest(2);
    server.start();
    try {
      InetSocketAddress serverAddress = NetUtils.getConnectAddress(server);
      byte[] requestBytes = generateRandomBytes(1024);
      ClientCallable clientCallable =
          new ClientCallable(serverAddress, conf, requestBytes);

      FutureTask<Writable> future = new FutureTask<Writable>(clientCallable);
      Thread clientThread = new Thread(future);
      clientThread.start();

      server.awaitInvocation();
      awaitResponseTimeout(future);

      server.sendResponse();
      BytesWritable response = (BytesWritable) future.get();

      Assert.assertEquals(new BytesWritable(requestBytes), response);
    } finally {
      if (server != null) {
        server.stop();
      }
    }
  }

  @Test(timeout = 10000)
  public void testDeferredException() throws IOException, InterruptedException,
      ExecutionException {
    ServerForHandoffTest server = new ServerForHandoffTest(2);
    server.start();
    try {
      InetSocketAddress serverAddress = NetUtils.getConnectAddress(server);
      byte[] requestBytes = generateRandomBytes(1024);
      ClientCallable clientCallable =
          new ClientCallable(serverAddress, conf, requestBytes);

      FutureTask<Writable> future = new FutureTask<Writable>(clientCallable);
      Thread clientThread = new Thread(future);
      clientThread.start();

      server.awaitInvocation();
      awaitResponseTimeout(future);

      server.sendError();
      try {
        future.get();
        Assert.fail("Call succeeded. Was expecting an exception");
      } catch (ExecutionException e) {
        Throwable cause = e.getCause();
        Assert.assertTrue(cause instanceof RemoteException);
        RemoteException re = (RemoteException) cause;
        Assert.assertTrue(re.toString().contains("DeferredError"));
      }
    } finally {
      if (server != null) {
        server.stop();
      }
    }
  }

  private void awaitResponseTimeout(FutureTask<Writable> future) throws
      ExecutionException,
      InterruptedException {
    long sleepTime = 3000L;
    while (sleepTime > 0) {
      try {
        future.get(200L, TimeUnit.MILLISECONDS);
        Assert.fail("Expected to timeout since" +
            " the deferred response hasn't been registered");
      } catch (TimeoutException e) {
        // Ignoring. Expected to time out.
      }
      sleepTime -= 200L;
    }
    LOG.info("Done sleeping");
  }

  private static class ClientCallable implements Callable<Writable> {

    private final InetSocketAddress address;
    private final Configuration conf;
    final byte[] requestBytes;


    private ClientCallable(InetSocketAddress address, Configuration conf,
                           byte[] requestBytes) {
      this.address = address;
      this.conf = conf;
      this.requestBytes = requestBytes;
    }

    @Override
    public Writable call() throws Exception {
      Client client = new Client(BytesWritable.class, conf);
      Writable param = new BytesWritable(requestBytes);
      final Client.ConnectionId remoteId =
          Client.ConnectionId.getConnectionId(address, null,
              null, 0, null, conf);
      Writable result = client.call(RPC.RpcKind.RPC_BUILTIN, param, remoteId,
          new AtomicBoolean(false));
      return result;
    }
  }

  private byte[] generateRandomBytes(int length) {
    Random random = new Random();
    byte[] bytes = new byte[length];
    for (int i = 0; i < length; i++) {
      bytes[i] = (byte) ('a' + random.nextInt(26));
    }
    return bytes;
  }
}

Server端的请求延时回复需要根据实际的场景进行运用，并不是说异步延时回复的方式就比RPC同步等待response结果的方式好。上述相关代码的改动感兴趣的同学可阅读下文对应JIRA的链接。

引用

---

[1].https://issues.apache.org/jira/browse/HADOOP-10300
[2].https://issues.apache.org/jira/browse/HADOOP-11552