tars源码解析二

tars源码解析二

准备知识

这节准备解析下tars的网络层和协议转换的源码，为了更好的理解这节，需要具备java网络层知识，主要是nio和Reactor模式。

nio的知识可以参考这个系列的文章
http://www.iteye.com/magazines/132-Java-NIO

这里简单说下Nio的概念，在bio模式下，因为socket读取数据是阻塞的，所以导致整个线程都卡在这里。在nio模式下，会用事件机制（表相是这样，底层是依赖操作系统的机制实现，linux的epoll，windows的iocp）用一个线程去托管n个socket，该线程不停的遍历所有socket的事件，如果某个socket读取好了数据发送通知通知该线程，该线程拿出该socket去处理（通俗的解释，不严格正确）

Reactor模式
https://www.cnblogs.com/doit8791/p/7461479.html

Reactor有几张图很重要，代表了几种模式

• 单个线程充当Reactor，业务也在该线程里处理
  

• 单个线程充当Reactor，业务放在线程池处理
  

• Reactor拆出来2个，1个负责连接，1个负责读，业务放在线程池处理
  

这3张图理解了，基本上Ractor模式就掌握了

服务端的网络层

上一节我们忽略了整个网络层的细节，对于上层来说，就是这段的代码
ServantAdapter.bind

public void bind(AppService appService) throws IOException {
        this.skeleton = (ServantHomeSkeleton) appService;
        ServerConfig serverCfg = ConfigurationManager.getInstance().getServerConfig();

        boolean keepAlive = true;
        Codec codec = createCodec(serverCfg);
        Processor processor = createProcessor(serverCfg);
        Executor threadPool = ServantThreadPoolManager.get(servantAdapterConfig);

        Endpoint endpoint = this.servantAdapterConfig.getEndpoint();
        if (endpoint.type().equals("tcp")) {
            this.selectorManager = new SelectorManager(Utils.getSelectorPoolSize(), new ServantProtocolFactory(codec), threadPool, processor, keepAlive, "server-tcp-reactor", false);
            this.selectorManager.setTcpNoDelay(serverCfg.isTcpNoDelay());
            this.selectorManager.start();

            System.out.println("[SERVER] server starting at " + endpoint + "...");
            ServerSocketChannel serverChannel = ServerSocketChannel.open();
            serverChannel.socket().bind(new InetSocketAddress(endpoint.host(), endpoint.port()), 1024);
            serverChannel.configureBlocking(false);

            selectorManager.getReactor(0).registerChannel(serverChannel, SelectionKey.OP_ACCEPT);

            System.out.println("[SERVER] server started at " + endpoint + "...");

        } else if (endpoint.type().equals("udp")) {

            this.selectorManager = new SelectorManager(1, new ServantProtocolFactory(codec), threadPool, processor, false, "server-udp-reactor", true);
            this.selectorManager.start();

            System.out.println("[SERVER] server starting at " + endpoint + "...");
            DatagramChannel serverChannel = DatagramChannel.open();
            DatagramSocket socket = serverChannel.socket();
            socket.bind(new InetSocketAddress(endpoint.host(), endpoint.port()));
            serverChannel.configureBlocking(false);

            this.selectorManager.getReactor(0).registerChannel(serverChannel, SelectionKey.OP_READ);
            System.out.println("[SERVER] servant started at " + endpoint + "...");
        }
    }

先看 SelectorManager初始化
获取线程池的数目，我们大部分确定线程数一般是cpu+1，但是这里对于cpu大于8的，做了特殊处理，原因猜测是因为如果cpu大于8的时候，不想把cpu都占满？

public class Utils {
    public static int getSelectorPoolSize() {
        int processors = Runtime.getRuntime().availableProcessors();
        return processors > 8 ? 4 + (processors * 5 / 8) : processors + 1;
    }

我们看下SelectorManager成员变量

public final class SelectorManager {

    private final AtomicLong sets = new AtomicLong(0);

    //这里是前面说的reactor模式里的reactor
    private final Reactor[] reactorSet;

    //这里是codec的工厂，理解简单点就是codec，默认的情况是TarsCodec这个类
    private ProtocolFactory protocolFactory = null;

    //这个treadPool是处理业务的线程池
    private Executor threadPool = null;

    //request<-> response的处理逻辑 默认是TarsServantProcessor
    private Processor processor = null;

    //select的个数，基本就是reactor的个数
    private final int selectorPoolSize;

    private volatile boolean started;

    private boolean keepAlive;

    private boolean isTcpNoDelay = false;

selectManager的构造函数,主要是赋值，构建Reactors

public SelectorManager(int selectorPoolSize, ProtocolFactory protocolFactory, Executor threadPool,
                           Processor processor, boolean keepAlive, String reactorNamePrefix, boolean udpMode) throws IOException {
    if (udpMode) selectorPoolSize = 1;

    this.selectorPoolSize = selectorPoolSize;
    this.protocolFactory = protocolFactory;
    this.threadPool = threadPool;
    this.processor = processor;
    this.keepAlive = keepAlive;

    reactorSet = new Reactor[selectorPoolSize];

    for (int i = 0; i < reactorSet.length; i++) {
        reactorSet[i] = new Reactor(this, reactorNamePrefix + "-" + protocolFactory.getClass().getSimpleName().toLowerCase() + "-" + String.valueOf(i), udpMode);
    }
}

整个网络层，tars提供了一个单独的net包来封装

我们看下Reactor,可以看到Reactor是个Thread，里面包含nio的一个重要组件select

public final class Reactor extends Thread {

    protected volatile Selector selector = null;

    private volatile boolean crashed = false;

    private final Queue register = new LinkedBlockingQueue();

    private final Queue unregister = new LinkedBlockingQueue();

    private Acceptor acceptor = null;

    public Reactor(SelectorManager selectorManager, String name) throws IOException {
        this(selectorManager, name, false);
    }

    public Reactor(SelectorManager selectorManager, String name, boolean udpMode) throws IOException {
        super(name);

        if (udpMode) {
            this.acceptor = new UDPAcceptor(selectorManager);
        } else {
            this.acceptor = new TCPAcceptor(selectorManager);
        }

        this.selector = Selector.open();
    }

accpetor是处理网络事件的逻辑

public abstract class Acceptor {

    protected SelectorManager selectorManager = null;

    public Acceptor(SelectorManager selectorManager) {
        this.selectorManager = selectorManager;
    }

    //处理网络OP_CONNECT的事件
    public abstract void handleConnectEvent(SelectionKey key) throws IOException;

    //处理网络OP_ACCEPT的事件
    public abstract void handleAcceptEvent(SelectionKey key) throws IOException;

    //处理网络OP_READ的事件
    public abstract void handleReadEvent(SelectionKey key) throws IOException;

    //处理网络OP_WRITE的事件
    public abstract void handleWriteEvent(SelectionKey key) throws IOException;
}

我们稍后再看下TCPAcceptor具体的处理逻辑，回到上面
this.selectorManager.start()；
这里启动了Reactor，其实就是调用Thread的run方法，这里基本属于nio的处理，可以简单理解，不停的循环遍历socket看看有没有事件抛出来，抛出来就处理。

public synchronized void start() {
    if (this.started) {
        return;
    }

    this.started = true;
    for (Reactor reactor : this.reactorSet) {
        reactor.start();
    }
}


public void run() {
    try {
        while (!Thread.interrupted()) {
            selector.select();
            processRegister();
            Iterator iter = selector.selectedKeys().iterator();
            while (iter.hasNext()) {
                SelectionKey key = iter.next();
                iter.remove();

                if (!key.isValid()) continue;

                try {
                    //1. Update the last operation time
                    if (key.attachment() != null && key.attachment() instanceof Session) {
                        ((Session) key.attachment()).updateLastOperationTime();
                    }

                    //2. Dispatch I/O event
                    dispatchEvent(key);

                } catch (Throwable ex) {
                    disConnectWithException(key, ex);
                }
            }
            processUnRegister();
        }
    } catch (Throwable e) {
        crashed = true;
        e.printStackTrace();
    } finally {
        try {
            selector.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

抛出来的事件怎么处理，在dispatchEvent处理，其实是调用前面的acceptor去处理

private void dispatchEvent(final SelectionKey key) throws IOException {
    if (key.isConnectable()) {
        acceptor.handleConnectEvent(key);
    } else if (key.isAcceptable()) {
        acceptor.handleAcceptEvent(key);
    } else if (key.isReadable()) {
        acceptor.handleReadEvent(key);
    } else if (key.isValid() && key.isWritable()) {
        acceptor.handleWriteEvent(key);
    }
}

先不急着看acceptor的逻辑，我们停下来想想，这里的reactor线程会发生什么？会走哪个逻辑？如果你熟悉网络编程的话，你会发现这个服务端还没绑定端口。回到上层

启动ractor之后，绑定端口，随后注册OP_ACCEPT消息

ServerSocketChannel serverChannel = ServerSocketChannel.open();
serverChannel.socket().bind(new InetSocketAddress(endpoint.host(), endpoint.port()), 1024);
serverChannel.configureBlocking(false);

selectorManager.getReactor(0).registerChannel(serverChannel, SelectionKey.OP_ACCEPT);

System.out.println("[SERVER] server started at " + endpoint + "...");

我们现在来看TcpAcceptor处理逻辑, 先看怎么处理OP_ACCEPT，如果一个客户端连接过来，建立一个TCPSession，然后注册到SessionManager里管理，同时使用一个新的reactor来处理这个connection的OP_READ的消息。

public void handleAcceptEvent(SelectionKey key) throws IOException {
    //1. Accept TCP request
    ServerSocketChannel server = (ServerSocketChannel) key.channel();
    SocketChannel channel = server.accept();
    channel.socket().setTcpNoDelay(selectorManager.isTcpNoDelay());
    channel.configureBlocking(false);
    Utils.setQosFlag(channel.socket());

    //2. Create NioSession for each TCP connection
    TCPSession session = new TCPSession(selectorManager);
    session.setChannel(channel);
    session.setStatus(SessionStatus.SERVER_CONNECTED);
    session.setKeepAlive(selectorManager.isKeepAlive());
    session.setTcpNoDelay(selectorManager.isTcpNoDelay());

    //3. Register session
    SessionManager.getSessionManager().registerSession(session);

    //4. Register channel with the specified session
    selectorManager.nextReactor().registerChannel(channel, SelectionKey.OP_READ, session);
}

这里符合我们前面讲的reactor第三张图，一个reactor去处理连接请求，连接上来让一个reactor组来处理读消息分散压力，本应该如此，但是我们看tars的nextReactor这个方法.假设我们有2个reactor，我们固定用第0个reactor去处理连接，但是这么写会导致我第0个reactor也会处理read消息，其实不是很符合reactor的标准设计思路。不知道tars这么设计的理由？

public final Reactor nextReactor() {
    return this.reactorSet[(int) (this.sets.incrementAndGet() % this.selectorPoolSize)];
}

看下OP_READ消息处理,调用TCPSession的read方法

public void handleReadEvent(SelectionKey key) throws IOException {
    TCPSession session = (TCPSession) key.attachment();
    if (session == null) throw new RuntimeException("The session is null when reading data...");
    session.read();
}

TCPSession也很好理解，就是一个connection。这里的处理都是nio的部分，如果对nio很熟悉，很好理解，简单来说
1. 把读取的二进制数据调用我们前面说的codec去处理获取request
2. 把request形成一个workTread丢到业务线程池里取处理

protected void read() throws IOException {
    int ret = readChannel();

    if (this.status == SessionStatus.CLIENT_CONNECTED) {
        readResponse();
    } else if (this.status == SessionStatus.SERVER_CONNECTED) {
        readRequest();
    } else {
        throw new IllegalStateException("The current session status is invalid. [status:" + this.status + "]");
    }

    if (ret < 0) {
        close();
        return;
    }
}

public void readRequest() throws IOException {
    Request request = null;
    IoBuffer tempBuffer = null;

    try {
        tempBuffer = readBuffer.duplicate().flip();

        while (true) {
            tempBuffer.mark();

            if (tempBuffer.remaining() > 0) {
                request = selectorManager.getProtocolFactory().getDecoder().decodeRequest(tempBuffer, this);
            } else {
                request = null;
            }

            if (request != null) {
                try {
                    request.resetBornTime();
                    selectorManager.getThreadPool().execute(new WorkThread(request, selectorManager));
                } catch (Exception ex) {
                    ex.printStackTrace();
                }
            } else {
                tempBuffer.reset();
                readBuffer = resetIoBuffer(tempBuffer);
                break;
            }
        }
    } catch (ProtocolException ex) {
        close();
        ex.printStackTrace();
    }
}

我们先看codec的decodeRequest的逻辑，这里是在TarsCodec里,这里如果想看懂，需要对tars通信协议具备一定理解，我们可以不关心这个。
简单理解就是一个4字节的int去表明我这个消息长度是多少

public Request decodeRequest(IoBuffer buffer, Session session) throws ProtocolException {
    //如果一个int都没有返回
    if (buffer.remaining() < 4) {
        return null;
    }
    //算下真个包体多长，如果包体太长，认为出错，buffer.getInt()就是这个包体的长度
    int length = buffer.getInt() - TarsHelper.HEAD_SIZE;
    if (length > TarsHelper.PACKAGE_MAX_LENGTH || length <= 0) {
        throw new ProtocolException("the length header of the package must be between 0~10M bytes. data length:" + Integer.toHexString(length));
    }
    //如果接受的消息长度不足，说明还需要读取，返回继续读
    if (buffer.remaining() < length) {
        return null;
    }

    //这里就是把二进制数据按照协议的格式去填充进TarsServantRequest
    byte[] reads = new byte[length];
    buffer.get(reads);
    TarsInputStream jis = new TarsInputStream(reads);
    TarsServantRequest request = new TarsServantRequest(session);
    try {
        short version = jis.read(TarsHelper.STAMP_SHORT.shortValue(), 1, true);
        byte packetType = jis.read(TarsHelper.STAMP_BYTE.byteValue(), 2, true);
        int messageType = jis.read(TarsHelper.STAMP_INT.intValue(), 3, true);
        int requestId = jis.read(TarsHelper.STAMP_INT.intValue(), 4, true);
        String servantName = jis.readString(5, true);
        String methodName = jis.readString(6, true);
        request.setVersion(version);
        request.setPacketType(packetType);
        request.setMessageType(messageType);
        request.setRequestId(requestId);
        request.setServantName(servantName);
        request.setFunctionName(methodName);
        request.setInputStream(jis);
        request.setCharsetName(charsetName);
    } catch (Exception e) {
        System.err.println(e);
        request.setRet(TarsHelper.SERVERDECODEERR);
    }
    return request;
}

在我们开始看第二步之前，我们思考一个问题，我们不停的读取出来Request的，tars能不能保证同一个session执行的先后顺序。比如同一个客户端连续发送了消息abc，希望接受的顺序也是abc，tars能不能做到呢?

答案是no，tars做不到，为什么？因为消息abc是交由线程池处理，也许消息c先处理完就先返回了，所以同一个session是无法保证顺序的。

如果我们想保证一个session的消息严格按顺序返回，怎么做（hits：可以参看mina源码，mina是可以做到的）

做不到顺序，会对rpc有影响么？答案是no，具体答案在客户端实现里
从这里可以看到我们不能把Tars当做消息队列来使用，因为消息队列要严格顺序

我们再看第二步如何处理的session，主要是看WorkThread的执行方法
“`java
public void run() {
try {
if (udpSession != null) {
parseDatagramPacket();
udpBuffer = null;
}

    if (req != null) {
        req.setProcessTime(System.currentTimeMillis());
        req.init();
        Response res = selectorManager.getProcessor().process(req, req.getIoSession());
        if (!res.isAsyncMode()) req.getIoSession().write(res);
    } else if (resp != null) {
        resp.init();
        Ticket ticket = TicketManager.getTicket(resp.getTicketNumber());
        if (ticket == null) {
            String s = "failed to fetch request for this response. [from:" + resp.getSession().getRemoteIp() + ":" + resp.getSession().getRemotePort() + "]";
            System.out.println(s);
            return;
        }
        fillDitributedContext(ticket.request().getDistributedContext());
        ticket.notifyResponse(resp);
        ticket.countDown();
        TicketManager.removeTicket(ticket.getTicketNumber());
    }
} catch (Exception ex) {
    ex.printStackTrace();
} finally {
    clearDistributedContext();
}

}
“`
这里先调用req.init方法,这里会继续解析request消息。

这里为啥把request的解析分成2个部分，这里体现了设计的精巧，因为第一部分解析是在reactor线程里解析，所以需要尽量快，防止阻塞，而这里的body解析是在业务线程里解析。（深刻理解了Reactor模式）

public void init() {
    ((TarsCodec) this.session.getProtocolFactory().getDecoder()).decodeRequestBody(this);
//        TarsCodecHelper.decodeRequestBody(this);
}

 public ServantRequest decodeRequestBody(ServantRequest req) {
    TarsServantRequest request = (TarsServantRequest) req;
    if (request.getRet() != TarsHelper.SERVERSUCCESS) {
        return request;
    }
    if (TarsHelper.isPing(request.getFunctionName())) {
        return request;
    }

    TarsInputStream jis = request.getInputStream();
    ClassLoader oldClassLoader = null;
    try {
        oldClassLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(resolveProtocolClassLoader());
        String methodName = request.getFunctionName();
        byte[] data = jis.read(TarsHelper.STAMP_BYTE_ARRAY, 7, true);//数据
        int timeout = jis.read(TarsHelper.STAMP_INT.intValue(), 8, true);//超时时间
        Map context = (Map) jis.read(TarsHelper.STAMP_MAP, 9, true);//Map context
        Map status = (Map) jis.read(TarsHelper.STAMP_MAP, 10, true);

        request.setTimeout(timeout);
        request.setContext(context);
        request.setStatus(status);

        String servantName = request.getServantName();
        //在启动的时候会把暴露的接口的方法和参数缓存起来，这里就是取出来
        Map methodInfoMap = AnalystManager.getInstance().getMethodMapByName(servantName);

        if (methodInfoMap == null || methodInfoMap.isEmpty()) {
            request.setRet(TarsHelper.SERVERNOSERVANTERR);
            throw new ProtocolException("no found methodInfo, the context[ROOT], serviceName[" + servantName + "], methodName[" + methodName + "]");
        }
        TarsMethodInfo methodInfo = methodInfoMap.get(methodName);
        if (methodInfo == null) {
            request.setRet(TarsHelper.SERVERNOFUNCERR);
            throw new ProtocolException("no found methodInfo, the context[ROOT], serviceName[" + servantName + "], methodName[" + methodName + "]");
        }

        request.setMethodInfo(methodInfo);
        List parametersList = methodInfo.getParametersList();
        if (!CommonUtils.isEmptyCollection(parametersList)) {
            Object[] parameters = new Object[parametersList.size()];
            int i = 0;
            //不同版本解析不同
            if (TarsHelper.VERSION == request.getVersion()) {//request
                parameters = decodeRequestBody(data, request.getCharsetName(), methodInfo);
            } else if (TarsHelper.VERSION2 == request.getVersion() || TarsHelper.VERSION3 == request.getVersion()) {
                //wup request
                UniAttribute unaIn = new UniAttribute();
                unaIn.setEncodeName(request.getCharsetName());

                if (request.getVersion() == TarsHelper.VERSION2) {
                    unaIn.decodeVersion2(data);
                } else if (request.getVersion() == TarsHelper.VERSION3) {
                    unaIn.decodeVersion3(data);
                }

                Object value = null;
                for (TarsMethodParameterInfo parameterInfo : parametersList) {
                    if (TarsHelper.isHolder(parameterInfo.getAnnotations())) {
                        String holderName = TarsHelper.getHolderName(parameterInfo.getAnnotations());
                        if (!StringUtils.isEmpty(holderName)) {
                            value = new Holder