Rocketmq 的通信层是基于通信框架 netty,下面来看看rocketmq底层继承图。
NettyRemotingAbstract是NettyRemotingClient和NettyRemotingServer的抽象父类,对发送和接收的公共部分进行了处理
一 . 首先在数据结构方面使用了responseFuture模式
1.保存了RPC处理器 ,Broker 接收请求将 opaque 直接把这个值设置回响应对象,客户端接收到这个响应,通过 opaque 从缓存查找对应的 ResponseFuture 对象
protected final HashMap> processorTable =
new HashMap>(64);
当服务端接受客户端响应的时候,会调用NettyServerHandler处理客户端的请求。
ServerBootstrap childHandler = //
this.serverBootstrap.group(this.eventLoopGroupBoss,
this.eventLoopGroupSelector).channel(NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 1024)
.option(ChannelOption.SO_REUSEADDR, true)
.option(ChannelOption.SO_KEEPALIVE, false)
.childOption(ChannelOption.TCP_NODELAY, true)
.option(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
//
.option(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
//
.localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
.childHandler(new ChannelInitializer() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(
defaultEventExecutorGroup, //
new NettyEncoder(), //
new NettyDecoder(), //
new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()), //
new NettyConnetManageHandler(), //
new NettyServerHandler());
}
});
NettyServerHandler处理channelRead的时候调用processMessageRecevived
class NettyServerHandler extends SimpleChannelInboundHandler {
@Override
protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
processMessageReceived(ctx, msg);
}
}
processMesssageReceived方法就会判断当前是作为Server端,接收的消息是请求,那么调用processTable对应的事件进行处理,如果作为Client端,接收的消息是回复,即接收到Server端的回复,那么从responseTable中,首先获取opaque对应的ResponseFuture,如果这个response是异步回调,则有InvokeCallback,那么调用invokeBack函数,然后将Response塞入ResponseFuture后返回;
public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
final RemotingCommand cmd = msg;
if (cmd != null) {
switch (cmd.getType()) {
case REQUEST_COMMAND:
processRequestCommand(ctx, cmd);
break;
case RESPONSE_COMMAND:
processResponseCommand(ctx, cmd);
break;
default:
break;
}
}
}
2.其次还保存了对外请求
protected final ConcurrentHashMap responseTable =
new ConcurrentHashMap(256);
在服务端启动 start() 方法的时候调用线程启动扫描将超时的responseFuture直接删除掉
public void start() {
.............................
this.timer.scheduleAtFixedRate(new TimerTask() {
[@Override](https://my.oschina.net/u/1162528)
public void run() {
try {
NettyRemotingServer.this.scanResponseTable();
} catch (Exception e) {
log.error("scanResponseTable exception", e);
}
}
}, 1000 * 3, 1000);
}
扫描方法如下,判断哪些future超时:
public void scanResponseTable() {
final List rfList = new LinkedList();
Iterator> it = this.responseTable.entrySet().iterator();
while (it.hasNext()) {
Entry next = it.next();
ResponseFuture rep = next.getValue();
if ((rep.getBeginTimestamp() + rep.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) {
rep.release();
it.remove();
rfList.add(rep);
plog.warn("remove timeout request, " + rep);
}
}
for (ResponseFuture rf : rfList) {
try {
rf.executeInvokeCallback();
} catch (Throwable e) {
plog.warn("scanResponseTable, operationComplete Exception", e);
}
}
}
二. 上面看了基本的数据结构和数据处理方式,下面看下发送方信息的发送逻辑。
- invokeSyncImpl: 同步发送,发送时,生成ResponseFuture,放入responseTable中;然后发送后等待设置的timeout(3s)时间,如果对应的ResponseFuture为空,则报错;否则返回RemoteCommand进行业务逻辑处理;发送失败设置 ResponseFuture 发送失败,并且从缓存中移除 ResponseFuture(没有响 应过来,就用不到缓存中的 ResponseFuturel)
public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
final int opaque = request.getOpaque();
try {
final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, null, null);
this.responseTable.put(opaque, responseFuture);
final SocketAddress addr = channel.remoteAddress();
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
responseFuture.setSendRequestOK(true);
return;
} else {
responseFuture.setSendRequestOK(false);
}
responseTable.remove(opaque);
responseFuture.setCause(f.cause());
responseFuture.putResponse(null);
plog.warn("send a request command to channel <" + addr + "> failed.");
}
});
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
if (null == responseCommand) {
if (responseFuture.isSendRequestOK()) {
throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
responseFuture.getCause());
} else {
throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
}
}
return responseCommand;
} finally {
this.responseTable.remove(opaque);
}
}
2 invokeAsyncImpl:异步发送,发送时,生成ResponseFuture,设置 opaque, callback, once,超时时间等值,并放入缓存集合,放入responseTable中;如果超过scanResponseTable的timeout (30s),则报错;否则调用注册的invokeCallback进行回调处理;异步一般链路耗时比较长, 为了防止本地缓存的 netty 请求过多, 使用信号量控制上,限默认 2048 个,发送成功 responseFuture.setSendRequestOK(true); 发送失败 responseFuture.setSendRequestOK(false), 信号量通过 once 释放, 删除缓存 Netty 接收 server 端响应,根据 opaque 从缓存获取 responseFuture,调用回调方法即接 口 InvokeCallback 实现
public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
final int opaque = request.getOpaque();
try {
final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, null, null);
this.responseTable.put(opaque, responseFuture);
final SocketAddress addr = channel.remoteAddress();
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
responseFuture.setSendRequestOK(true);
return;
} else {
responseFuture.setSendRequestOK(false);
}
responseTable.remove(opaque);
responseFuture.setCause(f.cause());
responseFuture.putResponse(null);
plog.warn("send a request command to channel <" + addr + "> failed.");
}
});
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
if (null == responseCommand) {
if (responseFuture.isSendRequestOK()) {
throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
responseFuture.getCause());
} else {
throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
}
}
return responseCommand;
} finally {
this.responseTable.remove(opaque);
}
}
rocketmq大概的通信流程基本如此,具体还是要去github上下载源码研读,学下源代码对自身的提高确实蛮大的。