RequestChannel
Processor线程与Handler线程之间传递数据是通过RequestChannel完成的。在RequestChannel中包含了一个requestQueue队列和多个responseQueue队列,每个Processor线程对应一个responseQueue队列。Processor线程将读取到的请求存入requestQueue队列中，Handler线程从requestQueue队列中取出请求进行处理；Handler线程处理请求产生的响应会放到Processor对应的responseQueue中，Processor线程从其对应的responseQueue中取出响应发送给客户端。RequestChannel的结构如下：
RequestChannel核心字段：
- requestQueue:ArrayBlockingQueue[RequestChannel.Request]类型。Processor线程向Handler线程传递请求的队列。因为多个Processor线程和多个Handler线程并发操作，这个队列是线程安全的队列。
- responseQueues:LinkedBlockingQueue[RequestChannel.Response]队列的数组，Handler线程向Processor线程传递响应的队列，每个Processor对应数组的一个队列。
- numProcessors: Processor线程的个数，也是responseQueues数组的长度。
- queueSize:缓存请求的最大个数，即requestQueue的长度。
- responseListeners:List[(Int)=>Unit]类型，该字段是监听器列表，监听器的主要功能是Handler线程向responseQueue存放响应时唤醒对应的Processor线程。
RequestChannel提供了增删requestQueue队列，responseQueues集合以及responseListeners列表中元素的方法。在SocketServer的初始化过程中，有向RequestChannel.responseListeners集合中添加一个唤醒对应Processor线程的监听，如下：
requestChannel.addResponseListener(id => processors(id).wakeup())
每次向responseQueues添加请求时都要触发responseListeners列表中的监听器：
//向对应responseQueue队列中添加SendAction类型的Response
/** Send a response back to the socket server to be sent over the network */
def sendResponse(response: RequestChannel.Response) {
responseQueues(response.processor).put(response)
for(onResponse <- responseListeners)//调用responseListeners集合中所有的监听器
onResponse(response.processor)
}
//向对应responseQueue队列中添加NoOpAction类型的Response
/** No operation to take for the request, need to read more over the network */
def noOperation(processor: Int, request: RequestChannel.Request) {
responseQueues(processor).put(new RequestChannel.Response(processor, request, null, RequestChannel.NoOpAction))
for(onResponse <- responseListeners)//调用responseListeners集合中所有的监听器
onResponse(processor)
}
//向对应responseQueue队列中添加CloseConnectionAction类型的Response
/** Close the connection for the request */
def closeConnection(processor: Int, request: RequestChannel.Request) {
responseQueues(processor).put(new RequestChannel.Response(processor, request, null, RequestChannel.CloseConnectionAction))
for(onResponse <- responseListeners)//调用responseListeners集合中所有的监听器
onResponse(processor)
}
在RequestChannel中保存的是RequestChannel.Request和RequestChannel.Response两个类的对象。RequestChannel.Request会对请求进行解析，组成requestId（请求类型ID），header(请求头)，body(请求体)等字段，供Handler线程使用，并提供了一些记录操作时间的字段供监控用。RequestChannel.Request的解析过程：
case class Request(processor: Int, connectionId: String, session: Session, private var buffer: ByteBuffer, startTimeMs: Long, securityProtocol: SecurityProtocol) {
// These need to be volatile because the readers are in the network thread and the writers are in the request
// handler threads or the purgatory threads
//下面是一些记录操作时间的字段，由于这些字段会被多个线程修改和读取，使用@volatile保证可见性
@volatile var requestDequeueTimeMs = -1L
@volatile var apiLocalCompleteTimeMs = -1L
@volatile var responseCompleteTimeMs = -1L
@volatile var responseDequeueTimeMs = -1L
@volatile var apiRemoteCompleteTimeMs = -1L
val requestId = buffer.getShort()//请求类型id
// TODO: this will be removed once we migrated to client-side format
// for server-side request / response format
// NOTE: this map only includes the server-side request/response handlers. Newer
// request types should only use the client-side versions which are parsed with
// o.a.k.common.requests.AbstractRequest.getRequest()
private val keyToNameAndDeserializerMap: Map[Short, (ByteBuffer) => RequestOrResponse]=
Map(ApiKeys.FETCH.id -> FetchRequest.readFrom,
ApiKeys.CONTROLLED_SHUTDOWN_KEY.id -> ControlledShutdownRequest.readFrom
)
// TODO: this will be removed once we migrated to client-side format
val requestObj =
keyToNameAndDeserializerMap.get(requestId).map(readFrom => readFrom(buffer)).orNull
// if we failed to find a server-side mapping, then try using the
// client-side request / response format
//请求头
val header: RequestHeader =
if (requestObj == null) {
buffer.rewind
try RequestHeader.parse(buffer)
catch {
case ex: Throwable =>
throw new InvalidRequestException(s"Error parsing request header. Our best guess of the apiKey is: $requestId", ex)
}
} else
null
//请求体
val body: AbstractRequest =
if (requestObj == null)
try {
// For unsupported version of ApiVersionsRequest, create a dummy request to enable an error response to be returned later
if (header.apiKey == ApiKeys.API_VERSIONS.id && !Protocol.apiVersionSupported(header.apiKey, header.apiVersion))
new ApiVersionsRequest
else
//解析请求
AbstractRequest.getRequest(header.apiKey, header.apiVersion, buffer)
} catch {
case ex: Throwable =>
throw new InvalidRequestException(s"Error getting request for apiKey: ${header.apiKey} and apiVersion: ${header.apiVersion}", ex)
}
else
null
buffer = null//解析完成
RequestChannel.Response需要注意，字段ResponseAction有三种类型：SendAction,NoOpAction,CloseConnectionAction。
当请求进入RequestChannel.requestQueue之后，会有多个Handler线程并发处理从其中取出请求处理，如何保证客户端请求的顺序性呢？
可以看下Processor.run()方法，有多处注册/取消OP_READ事件以及注册/取消OP_WRITE事件的操作，这样可以保证每个连接上只有一个请求和对应的响应，从而实现请求的顺序性。
回顾一个请求数据从生产者发送到服务端的流程过程：
KafkaProducer线程创建ProducerRecord后，会将ProducerRecord缓存进RecordAccumulator。Sender线程从RecordAccumulator中获取缓存的消息，放入KafkaChannel.send字段中等待发送，同时放入InflightRequests队列中等待响应。之后，客户端会通过KSelector将请求发送出去。在服务端，Processor线程使用KSelector读取请求并暂存到stagedReceives队列中，KSelector.poll()方法结束后，请求被转移到completedReceives队列中。之后，Processor将请求做一系列的解析后放入RequestChannel.requestQueue队列。Handler线程会从RequestChannel.requestQueue队列中取出请求进行处理，将处理之后的响应放入RequestChannel.responseQueue队列。Processor线程从其对应的RequestChannel.responseQueue队列中取出响应并放入inflightResponse队列中缓存，当响应发送出去后会将其从inflightResponse中删除。生产者读取响应的过程与服务端读取请求的过程类似，去吧是inFlightRequest中的请求进行确认。消费者与服务端直接的请求和响应的流转过程与上述过程类似。
在高性能的分布式框架中经常采用这种Reactor模式的设计，如：HDFS RPC框架的服务端，ZooKeeper等。也有实现了Reactor模式的框架，如：Netty。