rocketMq底层使用netty作为网络通信框架,在上一篇 rocketmq rpc 通信(一) 编解码 介绍了rocketMq如何编解码,而本篇将重点介绍rocketMq netty底层的线程模型和通信方式。
1、为什么使用netty
- netty封装了java底层的nio,而不是使用传统的bio,底层使用reactor线程模型,使用几个线程可以轻松处理上万socket连接,效率更加高效
- netty底层解决了jdk epoll 空轮训的bug,同时提供了许多处理网络粘包拆包的工具类,开发更加高效
- netty的api相对简单,开发人员不用被nio的各种事件困扰,底层优秀的pipeline机制更加利于各种handler的扩展,使开发人员更加注重业务逻辑的实现
- netty是异步编程,可以处理更高的并发,很多优秀的中间件都是基于netty做网络通信
2、rocketMq 使用netty的线程模型介绍
rocketMq中一个有4类线程,如图表所示
线程数 | name | 说明 |
---|---|---|
1 | BossThread | reactor主线程,处理tcp连接建立 |
3 | SelectorWorkerThread | reactor worker线程,负责select网络事件,之后转发到编解码线程处理 |
8 | CodecThread | netty 的编解码线程,源码中数量为8 |
N | ProcessorThread | 业务处理线程池,负责各种业务处理,如读写消息 |
说明:在rocketMq 对应的netty 的worker线程,实际工作就是select socket 事件,之后转发给code线程,所以源码中的worker线程只设置了3个线程。上图中的线程处理时间长短是逐渐增大的。各个线程之间的交互如图所示
还有一张图更加清晰的解释了 netty 的pipeline 和 server 端的如何做出处理的,此图来自rocketMq社区
3、server端流程
3.1、server端启动流程
服务端启动时,tcp参数说明
- ChannelOption.SO_BACKLOG对应的是tcp/ip协议listen函数中的backlog参数,函数listen(int socketfd,int backlog)用来初始化服务端可连接队列, 服务端处理客户端连接请求是顺序处理的,所以同一时间只能处理一个客户端连接,多个客户端来的时候,服务端将不能处理的客户端连接请求放在队列中等待处理,backlog参数指定了队列的大小
- ChanneOption.SO_REUSEADDR对应于套接字选项中的SO_REUSEADDR,这个参数表示允许重复使用本地地址和端口,比如,某个服务器进程占用了TCP的80端口进行监听,此时再次监听该端口就会返回错误,使用该参数就可以解决问题,该参数允许共用该端口,这个在服务器程序中比较常使用
- Channeloption.SO_KEEPALIVE参数对应于套接字选项中的SO_KEEPALIVE,该参数用于设置TCP连接,当设置该选项以后,连接会测试链接的状态,这个选项用于可能长时间没有数据交流的连接。当设置该选项以后,如果在两小时内没有数据的通信时,TCP会自动发送一个活动探测数据报文。由于rocketMq有心跳机制,所以此参数设置为false。
- ChannelOption.TCP_NODELAY 该参数的作用就是禁止使用Nagle算法,使用于小数据即时传输
server端构造方法,这些都是netty的一下api,需要对netty相对熟悉
public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
final ChannelEventListener channelEventListener) {
super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
this.serverBootstrap = new ServerBootstrap();
this.nettyServerConfig = nettyServerConfig;
this.channelEventListener = channelEventListener;
int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads();
if (publicThreadNums <= 0) {
publicThreadNums = 4;
}
this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
}
});
// 是否使用epoll
if (useEpoll()) {
this.eventLoopGroupBoss = new EpollEventLoopGroup(1, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyEPOLLBoss_%d", this.threadIndex.incrementAndGet()));
}
});
this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
private int threadTotal = nettyServerConfig.getServerSelectorThreads();
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyServerEPOLLSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
}
});
} else {
this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyNIOBoss_%d", this.threadIndex.incrementAndGet()));
}
});
this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
private int threadTotal = nettyServerConfig.getServerSelectorThreads();
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyServerNIOSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
}
});
}
loadSslContext();
}
服务端启动方法
@Override
public void start() {
// 处理编解码的线程池
this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
nettyServerConfig.getServerWorkerThreads(),
new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
}
});
ServerBootstrap childHandler =
this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
.channel(useEpoll() ? EpollServerSocketChannel.class :NioServerSocketChannel.class)// 是否使用epoll
.option(ChannelOption.SO_BACKLOG, 1024)
.option(ChannelOption.SO_REUSEADDR, true)
.option(ChannelOption.SO_KEEPALIVE, false)
.childOption(ChannelOption.TCP_NODELAY, true)
.childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
.childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
.localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
.childHandler(new ChannelInitializer() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline()
.addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME,
new HandshakeHandler(TlsSystemConfig.tlsMode))
// 这些handler共用defaultEventExecutorGroup这个线程池
.addLast(defaultEventExecutorGroup,
new NettyEncoder(),
new NettyDecoder(),
new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
new NettyConnectManageHandler(),
new NettyServerHandler()// 具体处理
);
}
});
if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
}
try {
ChannelFuture sync = this.serverBootstrap.bind().sync();
InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
this.port = addr.getPort();
} catch (InterruptedException e1) {
throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
}
if (this.channelEventListener != null) {
this.nettyEventExecutor.start();
}
}
说明:可以看到,在构造函数中初始化boss和worker线程池时,判断是否可以使用epoll,下面大体上来说明下epoll比select好在哪里~
select的最大缺陷是能打开fd的个数是有限的,linux上默认为1024,对socket的扫描是线性扫描,就是采用轮训的方法效率低下,当socket比较多时并且不管哪个socket是活跃的,都会遍历一遍。select需要存储一个维护fd的数据结构,在用户态和内核态切换时需要拷贝,效率也比较低下。
epoll支持水平触发和边缘触发,最大的特点在于边缘触发,它只告诉进程哪些fd刚刚变为就绪态,并且只会通知一次。还有一个特点是,epoll使用“事件”的就绪通知方式,通过epoll_ctl注册fd,一旦该fd就绪,内核就会采用类似callback的回调机制来激活该fd,epoll_wait便可以收到通知。epoll没有最大连接数限制存储使用红黑树,没有用户态和内核态之间的内存拷贝,使用mmap做内存映射,效率更加高效。
3.2、server端处理请求
由最上面类继承图可知,NettyRemotingServer继承了NettyRemotingAbstract,server只会处理request请求,处理完成后将response写回到channel返给客户端。
class NettyServerHandler extends SimpleChannelInboundHandler {
@Override
protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
// 这个方法是NettyRemotingAbstract的方法
processMessageReceived(ctx, msg);
}
}
// 看下server端如何处理请求,代码在NettyRemotingAbstract
public void processRequestCommand(final ChannelHandlerContext ctx, final RemotingCommand cmd) {
// 服务端启动时会注册相应的processor和处理processor对应的线程池
final Pair matched = this.processorTable.get(cmd.getCode());
final Pair pair = null == matched ? this.defaultRequestProcessor : matched;
// client生成的一个请求码,server端原路返回
final int opaque = cmd.getOpaque();
if (pair != null) {
Runnable run = new Runnable() {
@Override
public void run() {
try {
// 执行before钩子方法
doBeforeRpcHooks(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd);
// 具体的processor处理
final RemotingCommand response = pair.getObject1().processRequest(ctx, cmd);
// 执行after钩子方法
doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd, response);
if (!cmd.isOnewayRPC()) {
if (response != null) {
// 将客户端请求码原路返回,之后client根据opaque从map中获取相应的ResponseFuture,让client端的线程不在等待
response.setOpaque(opaque);
response.markResponseType();
try {
ctx.writeAndFlush(response);
} catch (Throwable e) {
log.error("process request over, but response failed", e);
log.error(cmd.toString());
log.error(response.toString());
}
} else {
}
}
} catch (Throwable e) {
log.error("process request exception", e);
log.error(cmd.toString());
if (!cmd.isOnewayRPC()) {
final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_ERROR,
RemotingHelper.exceptionSimpleDesc(e));
response.setOpaque(opaque);
ctx.writeAndFlush(response);
}
}
}
};
// 执行对应的processor方法,看是否拒绝,这里是在code线程中执行的,所以这个方法不能太耗时
if (pair.getObject1().rejectRequest()) {
final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_BUSY,
"[REJECTREQUEST]system busy, start flow control for a while");
response.setOpaque(opaque);
ctx.writeAndFlush(response);
return;
}
try {
final RequestTask requestTask = new RequestTask(run, ctx.channel(), cmd);
// 提交到对应的processor线程池中执行
pair.getObject2().submit(requestTask);
} catch (RejectedExecutionException e) {
if ((System.currentTimeMillis() % 10000) == 0) {
log.warn(RemotingHelper.parseChannelRemoteAddr(ctx.channel())
+ ", too many requests and system thread pool busy, RejectedExecutionException "
+ pair.getObject2().toString()
+ " request code: " + cmd.getCode());
}
// 由于Oneway 类型的rpc 客户端不关心结果,所以不用返回客户端信息
if (!cmd.isOnewayRPC()) {
final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_BUSY,
"[OVERLOAD]system busy, start flow control for a while");
response.setOpaque(opaque);
ctx.writeAndFlush(response);
}
}
} else {
// 非法的code码,返回客户端错误
String error = " request type " + cmd.getCode() + " not supported";
final RemotingCommand response =
RemotingCommand.createResponseCommand(RemotingSysResponseCode.REQUEST_CODE_NOT_SUPPORTED, error);
response.setOpaque(opaque);
ctx.writeAndFlush(response);
log.error(RemotingHelper.parseChannelRemoteAddr(ctx.channel()) + error);
}
}
说明:服务端在接收到客户端的请求后,由上面注册的defaultEventExecutorGroup线程也就是 编解码线程执行,通过RemotingCommand的code码获取相应的pair key : processor,value : processor对应的执行线程池,编解码线程提交到processor对应的执行线程池中执行,线程池执行完成之后,将结果写入pipeline中,做后续处理。
4、client端流程
client端的类继承图如图所示
4.1、client端rpc通信方式
- sync,客户端线程同步等待
- async,客户端线程不同步等待,通过netty异步编程的callback进行通知
- oneway,客户端只像server端请求,不关心请求是否成功
4.1.1、client端handler处理
由上面的类继承图,NettyRemotingClient也继承了NettyRemotingAbstract,在NettyRemotingAbstract#processMessageReceived方法中声明了RESPONSE_COMMAND,就是客户端收到server端的响应具体处理的逻辑
public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
final RemotingCommand cmd = msg;
if (cmd != null) {
switch (cmd.getType()) {
case REQUEST_COMMAND:
processRequestCommand(ctx, cmd);
break;
// client端调用这个方法
case RESPONSE_COMMAND:
processResponseCommand(ctx, cmd);
break;
default:
break;
}
}
}
public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
final int opaque = cmd.getOpaque();
final ResponseFuture responseFuture = responseTable.get(opaque);
if (responseFuture != null) {
responseFuture.setResponseCommand(cmd);
responseTable.remove(opaque);
if (responseFuture.getInvokeCallback() != null) {
// async模式下,执行回调方法
executeInvokeCallback(responseFuture);
} else {
// sync模式下,唤醒client端发送线程,发送线程返回RemotingCommand 对应的response
responseFuture.putResponse(cmd);
responseFuture.release();
}
} else {
log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
log.warn(cmd.toString());
}
}
4.1.2、sync调用
调用流程图如图所示
代码分析如下
public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
final long timeoutMillis)
throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
final int opaque = request.getOpaque();
try {
// 初始化时,ResponseFuture 内部成员变量 CountDownLatch countDownLatch = new CountDownLatch(1);
final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
// 将客户端请求码和ResponseFuture put 到map中,在server端返回结果回调Listener时可以根据客户端请求码获取ResponseFuture
this.responseTable.put(opaque, responseFuture);
final SocketAddress addr = channel.remoteAddress();
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
// 成功,将future设置SendRequestOK为true,直接返回
responseFuture.setSendRequestOK(true);
return;
} else {
responseFuture.setSendRequestOK(false);
}
// 以下操作是失败的情况,删除客户端请求码对应的future,唤醒client端发送线程
responseTable.remove(opaque);
responseFuture.setCause(f.cause());
// 这个方法执行了countDownLatch.countDown(),因此会唤醒client端发送线程。
responseFuture.putResponse(null);
log.warn("send a request command to channel <" + addr + "> failed.");
}
});
// 客户端发送线程同步等待,就是调用了ResponseFuture 中的 countDownLatch.await
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
if (null == responseCommand) {
if (responseFuture.isSendRequestOK()) {
throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
responseFuture.getCause());
} else {
throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
}
}
return responseCommand;
} finally {
this.responseTable.remove(opaque);
}
}
说明:在同步调用过程中,有两个地方执行了responseFuture中countDownLatch的countdown操作,一个是在发送数据注册的Listener中,一个是在NettyClientHandler中,这两个地方都会唤醒client端发送线程,Listener中对应的是rpc通信失败,NettyClientHandler对应的是请求成功
4.1.3、async调用
调用流程图如图所示
代码如下
public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
final InvokeCallback invokeCallback)
throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
long beginStartTime = System.currentTimeMillis();
final int opaque = request.getOpaque();
// 获取一个信号量,防止发送太快,release的逻辑在执行完callback方法后
boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
if (acquired) {
// 在 once 的release 方法里,会将信号量release 减一
final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
long costTime = System.currentTimeMillis() - beginStartTime;
if (timeoutMillis < costTime) {
once.release();
throw new RemotingTimeoutException("invokeAsyncImpl call timeout");
}
final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
this.responseTable.put(opaque, responseFuture);
try {
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
// 这里是执行成功了,设置为sendRequestOK为true,直接返回,回调方法在NettyClientHandler执行了
responseFuture.setSendRequestOK(true);
return;
}
// 这里执行失败,将失败结果传递给回调方法执行
requestFail(opaque);
log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
}
});
} catch (Exception e) {
responseFuture.release();
log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
}
} else {
if (timeoutMillis <= 0) {
throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
} else {
String info =
String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
timeoutMillis,
this.semaphoreAsync.getQueueLength(),
this.semaphoreAsync.availablePermits()
);
log.warn(info);
throw new RemotingTimeoutException(info);
}
}
}
说明:异步调用其实就是传入了一个callback,这样做的好处是不用阻塞client端的调用线程,在server端处理成功后,利用netty的异步编程,调用回调函数,完全实现了异步化,吞吐量有所上升,注意:异步调用有可能会丢失数据(server端突然宕机,数据会丢失)。rocketMq异步调用为了防止发送过快服务端压力过大,在client端设置了信号量 Semaphore,调用时先获取一个,在回调成功执行完callback函数后会将信号量 release。
4.1.3、oneway调用
说明:oneway调用客户端不关心server端是否处理成功,这种模式适合哪种不重要的调用,同时server端发生异常时发现是oneway调用,也不会向client端发送结果数据
4.1.4、async调用定时清理
异步调用时,有可能服务端宕机,这时候客户端会永远接收不到服务端的响应数据,因此async调用时put到map中的ResponseFuture就会一直在客户端的内存中,还有重要的一点是async调用时获取了一个信号量也得不到释放,最终async有可能调用不了,这时候需要把这些数据清除掉,代码如下
public void scanResponseTable() {
final List rfList = new LinkedList();
Iterator> it = this.responseTable.entrySet().iterator();
while (it.hasNext()) {
Entry next = it.next();
ResponseFuture rep = next.getValue();
if ((rep.getBeginTimestamp() + rep.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) {
// 释放信号量
rep.release();
// 从map中remove
it.remove();
rfList.add(rep);
log.warn("remove timeout request, " + rep);
}
}
for (ResponseFuture rf : rfList) {
try {
// 异步执行callback方法
executeInvokeCallback(rf);
} catch (Throwable e) {
log.warn("scanResponseTable, operationComplete Exception", e);
}
}
}
5、客户端连接管理
在server端启动时,设置了一个TCP选项 ChannelOption.SO_KEEPALIVE, false
,表示禁用tcp的keepalive,由于rocketMq有心跳机制,所以取消了这个选项。
在server端初始化pipeline时,添加了一个 IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds())
,nettyServerConfig.getServerChannelMaxIdleTimeSeconds()
默认是120秒,表示120秒没有发生读写请求,netty 会实例化一个 IdleStateEvent 沿着pipeline传输,NettyConnectManageHandler中监听了这个事件,会将这个channel remove掉,具体看下源码。
IdleStateHandler是netty里面的一个类,内部逻辑就是在channel有读或写事件时,更新一下lastReadTime和lastWriteTime,然后起一个定时任务检测和当前时间比较是否超过了allIdleTimeNanos设置的120秒。
// 初始化IdleStateHandler时,readerIdleTime = 0,writerIdleTime = 0,allIdleTime = 120
// 在超过120秒没有读写请求后,这是
public void run() {
if (!ctx.channel().isOpen()) {
return;
}
long nextDelay = allIdleTimeNanos;
if (!reading) {
nextDelay -= System.nanoTime() - Math.max(lastReadTime, lastWriteTime);
}
if (nextDelay <= 0) {
// Both reader and writer are idle - set a new timeout and
// notify the callback.
allIdleTimeout = ctx.executor().schedule(
this, allIdleTimeNanos, TimeUnit.NANOSECONDS);
try {
IdleStateEvent event = newIdleStateEvent(IdleState.ALL_IDLE, firstAllIdleEvent);
if (firstAllIdleEvent) {
firstAllIdleEvent = false;
}
// 发送IdleStateEvent事件到pipeline中,NettyConnectManageHandler会处理这个事件
channelIdle(ctx, event);
} catch (Throwable t) {
ctx.fireExceptionCaught(t);
}
} else {
// Either read or write occurred before the timeout - set a new
// timeout with shorter delay.
allIdleTimeout = ctx.executor().schedule(this, nextDelay, TimeUnit.NANOSECONDS);
}
}
// NettyConnectManageHandler处理
class NettyConnectManageHandler extends ChannelDuplexHandler {
@Override
public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
final String remoteAddress = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
log.info("NETTY SERVER PIPELINE: channelRegistered {}", remoteAddress);
super.channelRegistered(ctx);
}
@Override
public void channelUnregistered(ChannelHandlerContext ctx) throws Exception {
final String remoteAddress = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
log.info("NETTY SERVER PIPELINE: channelUnregistered, the channel[{}]", remoteAddress);
super.channelUnregistered(ctx);
}
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
final String remoteAddress = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
log.info("NETTY SERVER PIPELINE: channelActive, the channel[{}]", remoteAddress);
super.channelActive(ctx);
if (NettyRemotingServer.this.channelEventListener != null) {
// channel 连接事件
NettyRemotingServer.this.putNettyEvent(new NettyEvent(NettyEventType.CONNECT, remoteAddress, ctx.channel()));
}
}
@Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
final String remoteAddress = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
log.info("NETTY SERVER PIPELINE: channelInactive, the channel[{}]", remoteAddress);
super.channelInactive(ctx);
if (NettyRemotingServer.this.channelEventListener != null) {
// channel关闭事件
NettyRemotingServer.this.putNettyEvent(new NettyEvent(NettyEventType.CLOSE, remoteAddress, ctx.channel()));
}
}
@Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
// 这里处理IdleStateHandler 中发出来的事件
if (evt instanceof IdleStateEvent) {
IdleStateEvent event = (IdleStateEvent) evt;
if (event.state().equals(IdleState.ALL_IDLE)) {
final String remoteAddress = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
log.warn("NETTY SERVER PIPELINE: IDLE exception [{}]", remoteAddress);
RemotingUtil.closeChannel(ctx.channel());
if (NettyRemotingServer.this.channelEventListener != null) {
// 封装NettyEvent,放入队列,别的线程从队列中取相应的事件做处理
NettyRemotingServer.this
.putNettyEvent(new NettyEvent(NettyEventType.IDLE, remoteAddress, ctx.channel()));
}
}
}
ctx.fireUserEventTriggered(evt);
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
final String remoteAddress = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
log.warn("NETTY SERVER PIPELINE: exceptionCaught {}", remoteAddress);
log.warn("NETTY SERVER PIPELINE: exceptionCaught exception.", cause);
if (NettyRemotingServer.this.channelEventListener != null) {
// 发生异常事件
NettyRemotingServer.this.putNettyEvent(new NettyEvent(NettyEventType.EXCEPTION, remoteAddress, ctx.channel()));
}
RemotingUtil.closeChannel(ctx.channel());
}
}
// 从队列中取事件的线程
class NettyEventExecutor extends ServiceThread {
private final LinkedBlockingQueue eventQueue = new LinkedBlockingQueue();
private final int maxSize = 10000;
public void putNettyEvent(final NettyEvent event) {
if (this.eventQueue.size() <= maxSize) {
this.eventQueue.add(event);
} else {
log.warn("event queue size[{}] enough, so drop this event {}", this.eventQueue.size(), event.toString());
}
}
@Override
public void run() {
log.info(this.getServiceName() + " service started");
final ChannelEventListener listener = NettyRemotingAbstract.this.getChannelEventListener();
while (!this.isStopped()) {
try {
NettyEvent event = this.eventQueue.poll(3000, TimeUnit.MILLISECONDS);
if (event != null && listener != null) {
switch (event.getType()) {
case IDLE:
listener.onChannelIdle(event.getRemoteAddr(), event.getChannel());
break;
case CLOSE:
listener.onChannelClose(event.getRemoteAddr(), event.getChannel());
break;
case CONNECT:
listener.onChannelConnect(event.getRemoteAddr(), event.getChannel());
break;
case EXCEPTION:
listener.onChannelException(event.getRemoteAddr(), event.getChannel());
break;
default:
break;
}
}
} catch (Exception e) {
log.warn(this.getServiceName() + " service has exception. ", e);
}
}
log.info(this.getServiceName() + " service end");
}
@Override
public String getServiceName() {
return NettyEventExecutor.class.getSimpleName();
}
}
说明:当有客户端连接或关闭或超过一定时间没有读写请求时,NettyConnectManageHandler会捕捉到相应的事件,之后放到阻塞队列中,有一个单独的NettyEventExecutor从队列中获取事件,调用相应的listener做处理。
6、总结
本文介绍了rocketMq 使用netty的线程模型,client端发送数据,server端接收数据并根据相应的请求code码找到相应的processor提交到线程池中处理,处理完成后将结果写会客户端。客户端调用又分为3中模式,还介绍了channel的管理机制,netty底层其实还是挺复杂的,还需要深入学习。以上就是通过看rocketMq rpc模块相关代码总结出来的,如有错误,欢迎指出讨论~