我们带着问题去看源码,下面是我的问题:
- serverSocketChannel是什么时候与selector绑定的,流程是什么
- 启动的线程数有多少,什么时候new的,什么时候start的
- 有那些地方需要用锁
前提概要:netty代码非常深,尽量写详细,贴出代码中可以看todo的注解。2.1章节为重点,提到了线程的创建与start
可以先看最后的总结再看。
服务端代码
主要是分析
- NioEventLoopGroup bossGroup = new NioEventLoopGroup();
- ChannelFuture cf = bootstrap.bind(6666).sync();
public static void main(String[] args) {
//1.创建Boss NioEventLoopGroup , Worker NioEventLoopGroup
NioEventLoopGroup bossGroup = new NioEventLoopGroup();
NioEventLoopGroup workerGroup = new NioEventLoopGroup();
try {
//2.创建服务器端的启动对象,配置参数
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
//线程队列得到连接个数
.option(ChannelOption.SO_BACKLOG, 128)
//设置保持活动链接状态
.childOption(ChannelOption.SO_KEEPALIVE, true)
//给workGroup的EventLoop对应的管道设置handler
.childHandler(new ChannelInitializer() {
@Override
protected void initChannel(SocketChannel socketChannel) throws Exception {
socketChannel.pipeline().addLast(new NettyServerHandler());
}
});
//3.绑定端口并且同步
ChannelFuture cf = bootstrap.bind(6666).sync();
//4.对关闭通道进行监听
cf.channel().closeFuture().sync();
} catch (InterruptedException e) {
e.printStackTrace();
}finally {
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
}
猜想
NioEventLoopGroup bossGroup = new NioEventLoopGroup();
- 创建NioEventLoop
- 创建Selector
ChannelFuture cf = bootstrap.bind(6666).sync();
- 创建ServerSocketChannel
- 选出一个Selector于ServerSocketChannel绑定
实际
1.NioEventLoopGroup bossGroup = new NioEventLoopGroup();
protected MultithreadEventExecutorGroup(int nThreads, Executor executor,
EventExecutorChooserFactory chooserFactory, Object... args) {
// TODO: 不指定的时候,线程数默认cpu核数 *2
if (nThreads <= 0) {
throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
}
//TODO:executor.execute(Runnable command) 使用ThreadFactory创建一个新线程并且start();
if (executor == null) {
executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
}
children = new EventExecutor[nThreads];
for (int i = 0; i < nThreads; i ++) {
boolean success = false;
try {
// TODO: 创建NioEventLoop
children[i] = newChild(executor, args);
success = true;
} catch (Exception e) {
throw new IllegalStateException("failed to create a child event loop", e);
} finally {
if (!success) {
for (int j = 0; j < i; j ++) {
children[j].shutdownGracefully();
}
for (int j = 0; j < i; j ++) {
EventExecutor e = children[j];
try {
while (!e.isTerminated()) {
e.awaitTermination(Integer.MAX_VALUE, TimeUnit.SECONDS);
}
} catch (InterruptedException interrupted) {
// Let the caller handle the interruption.
Thread.currentThread().interrupt();
break;
}
}
}
}
}
// TODO: 根据nThreads线程数来选出选择选择器
chooser = chooserFactory.newChooser(children);
final FutureListener
我觉得需要着重注意一下executor这个属性类型ThreadPerTaskExecutor,因为代码中充斥了大量的使用这个,下面我们去看NioEventLoop的创建,时刻记得我们的问题什么时候创建的线程,什么时候启动的线程?
1.2 children[i] = newChild(executor, args);
这里调用的逻辑为NioEventLoopGroup类中的
创建好的NioEventLoop
- 所以这个时候并没有创建NioEventLoop里的线程
- 创建了Selector
- 有一个队列底层使用:Mpsc.newMpscQueue();,底层使用MpscUnboundedArrayQueue.class,传值1024
- 有一个threadLock
- exector较NioEventLoopGroup的exector又包装了一层 ,exector类型为ThreadExecutorMap.class 。很显然eventLoop.exector()的时候在开始的时候会来调用这个NioEventLoopGroup的exector中的ThreadFactory去newThread并且start()。
- ThreadExecutorMap里面存储:每个线程一个NioEventLoop
1.3 chooser = chooserFactory.newChooser(children);
public EventExecutorChooser newChooser(EventExecutor[] executors) {
if (isPowerOfTwo(executors.length)) {
return new PowerOfTwoEventExecutorChooser(executors);
} else {
return new GenericEventExecutorChooser(executors);
}
}
根据线程数初始化选择器,如果是偶数,则选PowerOfTwoEventExecutorChooser(count &(len -1)),这样的效率比取余数效率更高。
1 小总结
- NioEventLoopGroup除了有NioEventLoop的数组,还要一个只读的集合。ServerBootstrap.class里面也有一个多有数据只读的属性ServerBootstrapConfig.class
- 这一步只是创建了Selector和choose
2.ChannelFuture cf = bootstrap.bind(6666).sync();
代码太多了,这里打算不粘贴全部源码了,只写出我觉得是重点的部分,可以打开源码看文档,搭配食用
调用io.netty.bootstrap.AbstractBootstrap#doBind
//TODO: 根据反射创建ServerSocketChannel,完成注册
final ChannelFuture regFuture = initAndRegister();
//TODO: ServerSocketChannel地址绑定
doBind0(regFuture, channel, localAddress, promise);
2.1 final ChannelFuture regFuture = initAndRegister();
final ChannelFuture initAndRegister() {
Channel channel = null;
try {
//TODO: 反射初始化Channel,
channel = channelFactory.newChannel();
//TODO: 小重点,初始化channel
init(channel);
} catch (Throwable t) {
if (channel != null) {
// channel can be null if newChannel crashed (eg SocketException("too many open files"))
channel.unsafe().closeForcibly();
// as the Channel is not registered yet we need to force the usage of the GlobalEventExecutor
return new DefaultChannelPromise(channel, GlobalEventExecutor.INSTANCE).setFailure(t);
}
// as the Channel is not registered yet we need to force the usage of the GlobalEventExecutor
return new DefaultChannelPromise(new FailedChannel(), GlobalEventExecutor.INSTANCE).setFailure(t);
}
//TODO: 没有异常一般都会执行到这部分逻辑
ChannelFuture regFuture = config().group().register(channel);
if (regFuture.cause() != null) {
if (channel.isRegistered()) {
channel.close();
} else {
channel.unsafe().closeForcibly();
}
}
return regFuture;
}
channel = channelFactory.newChannel();
初始化后的channel
在初始化的时候创建java的ServerSocketChannel,并且设置为非阻塞
io.netty.bootstrap.ServerBootstrap#init
@Override
void init(Channel channel) {
setChannelOptions(channel, options0().entrySet().toArray(EMPTY_OPTION_ARRAY), logger);
setAttributes(channel, attrs0().entrySet().toArray(EMPTY_ATTRIBUTE_ARRAY));
ChannelPipeline p = channel.pipeline();
final EventLoopGroup currentChildGroup = childGroup;
final ChannelHandler currentChildHandler = childHandler;
final Entry, Object>[] currentChildOptions =
childOptions.entrySet().toArray(EMPTY_OPTION_ARRAY);
final Entry, Object>[] currentChildAttrs = childAttrs.entrySet().toArray(EMPTY_ATTRIBUTE_ARRAY);
p.addLast(new ChannelInitializer() {
//TODO: 需要关注这个方法执行的时间
@Override
public void initChannel(final Channel ch) {
final ChannelPipeline pipeline = ch.pipeline();
//TODO: 在pipline上添加配置的handler
ChannelHandler handler = config.handler();
if (handler != null) {
pipeline.addLast(handler);
}
//TODO: 给pipLines中添加一个ServerBootstrapAcceptor的handler
ch.eventLoop().execute(new Runnable() {
@Override
public void run() {
pipeline.addLast(new ServerBootstrapAcceptor(
ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
}
});
}
});
}
别想错了,这个时候还没有执行那个ChannelInitializer#initChannel方法。所以目前都还没有执行eventLoop.executor(..)
ChannelFuture regFuture = config().group().register(channel);
io.netty.channel.MultithreadEventLoopGroup#register(io.netty.channel.Channel)
@Override
public ChannelFuture register(Channel channel) {
//TODO: 调用choose选出一个NioEventLoop注册上去
return next().register(channel);
}
register方法最后会调用的逻辑在,io.netty.channel.AbstractChannel.AbstractUnsafe#register
public final void register(EventLoop eventLoop, final ChannelPromise promise) {
ObjectUtil.checkNotNull(eventLoop, "eventLoop");
if (isRegistered()) {
promise.setFailure(new IllegalStateException("registered to an event loop already"));
return;
}
if (!isCompatible(eventLoop)) {
promise.setFailure(
new IllegalStateException("incompatible event loop type: " + eventLoop.getClass().getName()));
return;
}
AbstractChannel.this.eventLoop = eventLoop;
//TODO:当前线程是main线程,返回false
if (eventLoop.inEventLoop()) {
register0(promise);
} else {
try {
//TODO:第一次调用调用 eventLoop.execute,这次会在NioEventLoop创建线程并且start
eventLoop.execute(new Runnable() {
@Override
public void run() {
register0(promise);
}
});
} catch (Throwable t) {
logger.warn(
"Force-closing a channel whose registration task was not accepted by an event loop: {}",
AbstractChannel.this, t);
closeForcibly();
closeFuture.setClosed();
safeSetFailure(promise, t);
}
}
}
io.netty.channel.AbstractChannel.AbstractUnsafe#register0
private void register0(ChannelPromise promise) {
try {
// check if the channel is still open as it could be closed in the mean time when the register
// call was outside of the eventLoop
if (!promise.setUncancellable() || !ensureOpen(promise)) {
return;
}
boolean firstRegistration = neverRegistered;
//TODO:重点1
doRegister();
neverRegistered = false;
registered = true;
// Ensure we call handlerAdded(...) before we actually notify the promise. This is needed as the
// user may already fire events through the pipeline in the ChannelFutureListener.
//TODO:重点2
pipeline.invokeHandlerAddedIfNeeded();
//TODO:重点3,在doBind0()绑定地址哪里有讲
safeSetSuccess(promise);
pipeline.fireChannelRegistered();
// Only fire a channelActive if the channel has never been registered. This prevents firing
// multiple channel actives if the channel is deregistered and re-registered.
if (isActive()) {
if (firstRegistration) {
pipeline.fireChannelActive();
} else if (config().isAutoRead()) {
// This channel was registered before and autoRead() is set. This means we need to begin read
// again so that we process inbound data.
//
// See https://github.com/netty/netty/issues/4805
beginRead();
}
}
} catch (Throwable t) {
// Close the channel directly to avoid FD leak.
closeForcibly();
closeFuture.setClosed();
safeSetFailure(promise, t);
}
}
重点1:io.netty.channel.nio.AbstractNioChannel#doRegister
@Override
protected void doRegister() throws Exception {
boolean selected = false;
for (;;) {
try {
//TODO:给这个channel注册了0事件,注意不是OP_ACCEPT
selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this);
return;
} catch (CancelledKeyException e) {
if (!selected) {
// Force the Selector to select now as the "canceled" SelectionKey may still be
// cached and not removed because no Select.select(..) operation was called yet.
eventLoop().selectNow();
selected = true;
} else {
// We forced a select operation on the selector before but the SelectionKey is still cached
// for whatever reason. JDK bug ?
throw e;
}
}
}
}
重点2:最后逻辑调用io.netty.channel.DefaultChannelPipeline#callHandlerAddedForAllHandlers
private void callHandlerAddedForAllHandlers() {
final PendingHandlerCallback pendingHandlerCallbackHead;
synchronized (this) {
assert !registered;
// This Channel itself was registered.
registered = true;
pendingHandlerCallbackHead = this.pendingHandlerCallbackHead;
// Null out so it can be GC'ed.
this.pendingHandlerCallbackHead = null;
}
// This must happen outside of the synchronized(...) block as otherwise handlerAdded(...) may be called while
// holding the lock and so produce a deadlock if handlerAdded(...) will try to add another handler from outside
// the EventLoop.
PendingHandlerCallback task = pendingHandlerCallbackHead;
while (task != null) {
//todo:这个方法最后其实就是调用了pipLine.addLast(ChannelInitializer#initChannel)
//完成最后NioServerSocketChannel的pipline中handler的初始化
task.execute();
task = task.next;
}
}
强势总结:
- 代码执行到现在NioServerSocketChannel初始化了。regregister0()里面的逻辑是channel与selector绑定,并且完成最后的pipline的初始化,但是异步执行,所以需要看那个地方执行了regregister0()
2.1 重点中的重点第一次调用eventLoop.executor(),所以分析逻辑
会调用io.netty.util.concurrent.SingleThreadEventExecutor#execute(java.lang.Runnable)
@Override
public void execute(Runnable task) {
ObjectUtil.checkNotNull(task, "task");
execute(task, !(task instanceof LazyRunnable) && wakesUpForTask(task));
}
execute:
private void execute(Runnable task, boolean immediate) {
boolean inEventLoop = inEventLoop();
// TODO: 加入NioEventLoop的taskQueue中,什么时候执行等我揭晓。
addTask(task);
//TODO:当前线程仍然是main,则当我们在NioEventLoop线程中exeutor(Runable)会直接加入队列中
if (!inEventLoop) {
//TODO:重点
startThread();
if (isShutdown()) {
boolean reject = false;
try {
if (removeTask(task)) {
reject = true;
}
} catch (UnsupportedOperationException e) {
// The task queue does not support removal so the best thing we can do is to just move on and
// hope we will be able to pick-up the task before its completely terminated.
// In worst case we will log on termination.
}
if (reject) {
reject();
}
}
}
if (!addTaskWakesUp && immediate) {
wakeup(inEventLoop);
}
}
startThread:
private void startThread() {
if (state == ST_NOT_STARTED) {
if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
boolean success = false;
try {
doStartThread();
success = true;
} finally {
if (!success) {
STATE_UPDATER.compareAndSet(this, ST_STARTED, ST_NOT_STARTED);
}
}
}
}
}
有点绕,仔细看箭头:
结论:doStartThread()里面的这个Runable的方法在NioEventLoop中的线程.start()的时候调用的。则在threadFactory.newThread(这里的runable是doStartThread中new出来的).start()
看到现在强势总结:
- 线程也创建了但是仍然没有看到要执行register0()所以现在可以看doStartThread()中SingleThreadEventExecutor.this.run(),执行这行代码的是在NioEventLoop线程中的,所以这个this没有问题吧。SingleThreadEventExecutor里面的代码大致是轮训遍历执行Selector里准备就绪的事件和taskQueue里面的事件,在下一章会有介绍。
2.2 doBind0(regFuture, channel, localAddress, promise);
什么时候被调用的?
- 因为它是在 regFuture.addListener(new ChannelFutureListener() {})中的回调方法中的执行的,所以他在 io.netty.channel.AbstractChannel.AbstractUnsafe#register0 的 safeSetSuccess(promise);行代码被调用的。
- 代码非常深,逻辑处理:io.netty.channel.socket.nio.NioServerSocketChannel#doBind
@SuppressJava6Requirement(reason = "Usage guarded by java version check")
@Override
protected void doBind(SocketAddress localAddress) throws Exception {
if (PlatformDependent.javaVersion() >= 7) {
javaChannel().bind(localAddress, config.getBacklog());
} else {
javaChannel().socket().bind(localAddress, config.getBacklog());
}
}
最后还会执行这一段代码:
invokeLater(new Runnable() {
@Override
public void run() {
pipeline.fireChannelActive();
}
});
pipeline.fireChannelActive();代码非常深:会执行这段逻辑
io.netty.channel.nio.AbstractNioChannel#doBeginRead
@Override
protected void doBeginRead() throws Exception {
// Channel.read() or ChannelHandlerContext.read() was called
final SelectionKey selectionKey = this.selectionKey;
if (!selectionKey.isValid()) {
return;
}
readPending = true;
final int interestOps = selectionKey.interestOps();
if ((interestOps & readInterestOp) == 0) {
//TODO:注册16,即OP_ACCEPT事件
selectionKey.interestOps(interestOps | readInterestOp);
}
}
总结
那么Netty的服务端的启动流程:
- NioEventLoopGroup bossGroup = new NioEventLoopGroup();
- 创建NioEventLoop,即创建Selector,Selector是NioEventLoop的一个属性变量
- 创建choose : 轮训让Channel绑定NioEventLoop的Selector上的算法
- ChannelFuture cf = bootstrap.bind(6666).sync();
- 通过反射创建NioServerSocketChannel
- NioServerSocketChannel初始化,添加一个回调函数为pipLine上加一个ServerBootstrapAcceptor的handler
- 添加一个回调函数绑定地址
- (在NioEventLoop线程中)NioServerSocketChannel注册上NioEventLoop的Selector上,绑定的是0
- (在NioEventLoop线程中)调用之前的回调函数NioServerSocketChannel绑定地址
- (在NioEventLoop线程中) 为NioServerSocektChannel注册OP_ACCEPT事件。