HBase rpc调用

HBase rpc 0.94中


例如在client put数据时，会调用htable的flushCommits，再调HConnectionImplementationr的processBatch，再调processBatchCallback
中，在这里异步调用线程，并使用future取得结果，最终执行的是call方法。

        // step 2: make the requests

        Map<HRegionLocation, Future<MultiResponse>> futures =
            new HashMap<HRegionLocation, Future<MultiResponse>>(
                actionsByServer.size());

        for (Entry<HRegionLocation, MultiAction<R>> e: actionsByServer.entrySet()) {
          futures.put(e.getKey(), pool.submit(createCallable(e.getKey(), e.getValue(), tableName)));
        }

        // step 3: collect the failures and successes and prepare for retry

        for (Entry<HRegionLocation, Future<MultiResponse>> responsePerServer
             : futures.entrySet()) {
          HRegionLocation loc = responsePerServer.getKey();

    private <R> Callable<MultiResponse> createCallable(final HRegionLocation loc,
        final MultiAction<R> multi, final byte [] tableName) {
      // TODO: This does not belong in here!!! St.Ack  HConnections should
      // not be dealing in Callables; Callables have HConnections, not other
      // way around.
      final HConnection connection = this;
      return new Callable<MultiResponse>() {
       public MultiResponse call() throws IOException {
         ServerCallable<MultiResponse> callable =
           new ServerCallable<MultiResponse>(connection, tableName, null) {
             public MultiResponse call() throws IOException {
               return server.multi(multi);
             }
             @Override
             public void connect(boolean reload) throws IOException {
               server = connection.getHRegionConnection(loc.getHostname(), loc.getPort());
             }
           };
         return callable.withoutRetries();
       }
     };
   }

其中的异步调用的线程池

    int maxThreads = conf.getInt("hbase.htable.threads.max", Integer.MAX_VALUE);
    if (maxThreads == 0) {
      maxThreads = 1; // is there a better default?
    }
    long keepAliveTime = conf.getLong("hbase.htable.threads.keepalivetime", 60);

    // Using the "direct handoff" approach, new threads will only be created
    // if it is necessary and will grow unbounded. This could be bad but in HCM
    // we only create as many Runnables as there are region servers. It means
    // it also scales when new region servers are added.
    this.pool = new ThreadPoolExecutor(1, maxThreads,
        keepAliveTime, TimeUnit.SECONDS,
        new SynchronousQueue<Runnable>(),
        new DaemonThreadFactory());
    ((ThreadPoolExecutor)this.pool).allowCoreThreadTimeOut(true);

    this.finishSetup();

HMaster or HRegionServer初始化创建HBaseServer调用HBaseRPC.getServer->

    this.rpcServer = HBaseRPC.getServer(this,
      new Class<?>[]{HMasterInterface.class, HMasterRegionInterface.class},
        initialIsa.getHostName(), // BindAddress is IP we got for this server.
        initialIsa.getPort(),
        numHandlers,
        0, // we dont use high priority handlers in master
        conf.getBoolean("hbase.rpc.verbose", false), conf,
        0); // this is a DNC w/o high priority handlers

HBase 0.94 HRegionServer默认启动10个handler线程用于处理rpc请求（hbase.regionserver.handler.count），

    this.rpcServer = HBaseRPC.getServer(this,
      new Class<?>[]{HRegionInterface.class, HBaseRPCErrorHandler.class,
        OnlineRegions.class},
        initialIsa.getHostName(), // BindAddress is IP we got for this server.
        initialIsa.getPort(),
        conf.getInt("hbase.regionserver.handler.count", 10),
        conf.getInt("hbase.regionserver.metahandler.count", 10),
        conf.getBoolean("hbase.rpc.verbose", false),
        conf, QOS_THRESHOLD);
    // Set our address.
    this.isa = this.rpcServer.getListenerAddress();

    this.rpcServer.setErrorHandler(this);
    this.rpcServer.setQosFunction(new QosFunction());

HBaseServer的Listerner的reader线程接到RPC请求后，会丢到queue中，
其中的reader线程数是由一个线程池决定

     this.readThreads = conf.getInt(
        "ipc.server.read.threadpool.size",
        10);

    public Listener() throws IOException {
      address = new InetSocketAddress(bindAddress, port);
      // Create a new server socket and set to non blocking mode
      acceptChannel = ServerSocketChannel.open();
      acceptChannel.configureBlocking(false);

      // Bind the server socket to the local host and port
      bind(acceptChannel.socket(), address, backlogLength);
      port = acceptChannel.socket().getLocalPort(); //Could be an ephemeral port
      // create a selector;
      selector= Selector.open();

      readers = new Reader[readThreads];
      readPool = Executors.newFixedThreadPool(readThreads,
        new ThreadFactoryBuilder().setNameFormat(
          "IPC Reader %d on port " + port).setDaemon(true).build());
      for (int i = 0; i < readThreads; ++i) {
        Reader reader = new Reader();
        readers[i] = reader;
        readPool.execute(reader);
      }

      // Register accepts on the server socket with the selector.
      acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
      this.setName("IPC Server listener on " + port);
      this.setDaemon(true);
    }

1. HBaseServer创建后有几个重要的角色.
1.1 Listener deamon线程,负责接收HMaster,HRegionServer,HBase Client的http请求.
1.2 Responder demon线程,负责将处理完的请求,发送回调用者.
1.3 Connection listener接收到的每个Socket请求都会创建一个Connection 实例.
1.4 Call 每一个客户端的发送的请求由Connection读取到有效数据后都会生成一个Call实例
1.5 LinkedBlockingQueue callQueue 每个由新生成的call都会放入到callQueue这个队列里.
1.6 Handler 从callQueue中取出call,并对call进行反射的调用,生成的结果value,交由responder处理.
1.7 LinkedList Connection.responseQueue ,用来存放已经由handler处理过的属于同一个Connection的call.
HBaseServer->listener = new Listener();->Reader.run->doRunLoop->doRead->Connection.readAndProcess->processData中将rpc请求生成call并加入到queue中。

      Call call = new Call(id, param, this, responder, callSize);
      callQueueSize.add(callSize);

HBaseServer的下面方法决定了接收RPC请求的queue大小maxQueueLength ，默认是100

    String oldMaxQueueSize = this.conf.get("ipc.server.max.queue.size");
    if (oldMaxQueueSize == null) {
      this.maxQueueLength =
        this.conf.getInt("ipc.server.max.callqueue.length",
          handlerCount * DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER);
    } else {
      LOG.warn("ipc.server.max.queue.size was renamed " +
               "ipc.server.max.callqueue.length, " +
               "please update your configuration");
      this.maxQueueLength = Integer.getInteger(oldMaxQueueSize);
    }

    this.maxQueueSize =
      this.conf.getInt("ipc.server.max.callqueue.size",
        DEFAULT_MAX_CALLQUEUE_SIZE);
     this.readThreads = conf.getInt(
        "ipc.server.read.threadpool.size",
        10);
    this.callQueue  = new LinkedBlockingQueue<Call>(maxQueueLength);

当regionserver启动时调用startServiceThreads内会调用HBaseServer的下面方法启动10个hander线程。

  public synchronized void startThreads() {
    responder.start();
    listener.start();
    handlers = new Handler[handlerCount];

    for (int i = 0; i < handlerCount; i++) {
      handlers[i] = new Handler(callQueue, i);
      handlers[i].start();
    }

    if (priorityHandlerCount > 0) {
      priorityHandlers = new Handler[priorityHandlerCount];
      for (int i = 0 ; i < priorityHandlerCount; i++) {
        priorityHandlers[i] = new Handler(priorityCallQueue, i);
        priorityHandlers[i].start();
      }
    }
  }

handler线程最终会调用到run的call方法，再此方法中会射出相应的方法并最终调用，也就是会调用相应的regionserver的方法。

        Method method =
          protocol.getMethod(call.getMethodName(),
                                   call.getParameterClasses());
        method.setAccessible(true);

        //Verify protocol version.
        //Bypass the version check for VersionedProtocol
        if (!method.getDeclaringClass().equals(VersionedProtocol.class)) {
          long clientVersion = call.getProtocolVersion();
          ProtocolSignature serverInfo = ((VersionedProtocol) instance)
              .getProtocolSignature(protocol.getCanonicalName(), call
                  .getProtocolVersion(), call.getClientMethodsHash());
          long serverVersion = serverInfo.getVersion();
          if (serverVersion != clientVersion) {
            LOG.warn("Version mismatch: client version=" + clientVersion
                + ", server version=" + serverVersion);
            throw new RPC.VersionMismatch(protocol.getName(), clientVersion,
                serverVersion);
          }
        }
        Object impl = null;
        if (protocol.isAssignableFrom(this.implementation)) {
          impl = this.instance;
        }
        else {
          throw new HBaseRPC.UnknownProtocolException(protocol);
        }

        long startTime = System.currentTimeMillis();
        Object[] params = call.getParameters();
        Object value = method.invoke(impl, params);