dubbo源码第13篇 本地调用和结果处理

服务提供者filter链条处理

上一篇我们看到了服务提供者的一段关键的代码：

// 执行Invoker调用链
Result result = invoker.invoke(inv);

这里接下来就是经过前面的ProtocolFilterWrapper定义的提供者filter链条：

// 使用责任链模式调用所有的filter
// 如果是服务提供者，则依次执行8个filter：
// 0 = {EchoFilter@1770}
// 1 = {ClassLoaderFilter@1771}
// 2 = {GenericFilter@1772}
// 3 = {ContextFilter@1773}
// 4 = {TraceFilter@1774}
// 5 = {TimeoutFilter@1775}
// 6 = {MonitorFilter@1776}
// 7 = {ExceptionFilter@1777}

这里分别简单看下这些链条的作用。

@Activate(group = CommonConstants.PROVIDER, order = -110000)
public class EchoFilter implements Filter {
     

    /**
     * 作为回声测试的filter，直接返回参数值
     * @param invoker
     * @param inv
     * @return
     * @throws RpcException
     */
    @Override
    public Result invoke(Invoker<?> invoker, Invocation inv) throws RpcException {
     
        if (inv.getMethodName().equals($ECHO) && inv.getArguments() != null && inv.getArguments().length == 1) {
     
            return AsyncRpcResult.newDefaultAsyncResult(inv.getArguments()[0], inv);
        }
        return invoker.invoke(inv);
    }

}
/**
 * Set the current execution thread class loader to service interface's class loader.
 * 将当前执行线程类加载器设置为服务接口的类加载器
 */
@Activate(group = CommonConstants.PROVIDER, order = -30000)
public class ClassLoaderFilter implements Filter {
     

    @Override
    public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
     
        ClassLoader ocl = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(invoker.getInterface().getClassLoader());
        try {
     
            return invoker.invoke(invocation);
        } finally {
     
            Thread.currentThread().setContextClassLoader(ocl);
        }
    }

}

GenericFilter就是将服务消费者调用时传入的经过序列化的参数，反序列化为对象。
ContextFilter将消费者定义的invocation参数写入到当前的上下文，并且设置必要的提供者的参数。
TraceFilter对每个invoker的调用次数和调用时间进行统计，如果超时则发送超时异常到消费者。
TimeoutFilter响应超时则打印日志。
MonitorFilter收集各种统计指标，包括调用次数，并发送到Monitor监控中心。
ExceptionFilter对接口调用产生的异常进行包装。

代理调用本地类方法

经过filter链条之后，就是InvokerDelegate委托类，然后InvokerDelegate会调用服务提供方启动时 AbstractProxyInvoker代理类的invoke()方法。

public abstract class AbstractProxyInvoker<T> implements Invoker<T> {
     
    Logger logger = LoggerFactory.getLogger(AbstractProxyInvoker.class);

    @Override
    public Result invoke(Invocation invocation) throws RpcException {
     
        try {
     
            // 具体执行本地服务调用
            Object value = doInvoke(proxy, invocation.getMethodName(), invocation.getParameterTypes(), invocation.getArguments());
			CompletableFuture<Object> future = wrapWithFuture(value);
            CompletableFuture<AppResponse> appResponseFuture = future.handle((obj, t) -> {
     
                AppResponse result = new AppResponse();
                // 处理异常情况
                if (t != null) {
     
                    if (t instanceof CompletionException) {
     
                        result.setException(t.getCause());
                    } else {
     
                        result.setException(t);
                    }
                } else {
     
                    result.setValue(obj);
                }
                return result;
            });
            // 返回结果
            return new AsyncRpcResult(appResponseFuture, invocation);
        } catch (InvocationTargetException e) {
     
            if (RpcContext.getContext().isAsyncStarted() && !RpcContext.getContext().stopAsync()) {
     
                logger.error("Provider async started, but got an exception from the original method, cannot write the exception back to consumer because an async result may have returned the new thread.", e);
            }
            return AsyncRpcResult.newDefaultAsyncResult(null, e.getTargetException(), invocation);
        } catch (Throwable e) {
     
            throw new RpcException("Failed to invoke remote proxy method " + invocation.getMethodName() + " to " + getUrl() + ", cause: " + e.getMessage(), e);
        }
    }

	private CompletableFuture<Object> wrapWithFuture(Object value) {
     
        // 开启异步执行
        if (RpcContext.getContext().isAsyncStarted()) {
     
            return ((AsyncContextImpl)(RpcContext.getContext().getAsyncContext())).getInternalFuture();
        }
        // 返回结果是 CompletableFuture类型
        else if (value instanceof CompletableFuture) {
     
            return (CompletableFuture<Object>) value;
        }
        return CompletableFuture.completedFuture(value);
    }

    protected abstract Object doInvoke(T proxy, String methodName, Class<?>[] parameterTypes, Object[] arguments) throws Throwable;

}

这里的关键就是执行doInvoke()方法。

public class JavassistProxyFactory extends AbstractProxyFactory {
     

    @Override
    @SuppressWarnings("unchecked")
    public <T> T getProxy(Invoker<T> invoker, Class<?>[] interfaces) {
     
        return (T) Proxy.getProxy(interfaces).newInstance(new InvokerInvocationHandler(invoker));
    }

    @Override
    public <T> Invoker<T> getInvoker(T proxy, Class<T> type, URL url) {
     
        // TODO Wrapper cannot handle this scenario correctly: the classname contains '$'
        // 创建Wrapper类，以便减少反射调用
        final Wrapper wrapper = Wrapper.getWrapper(proxy.getClass().getName().indexOf('$') < 0 ? proxy.getClass() : type);
        return new AbstractProxyInvoker<T>(proxy, type, url) {
     
            @Override
            protected Object doInvoke(T proxy, String methodName,
                                      Class<?>[] parameterTypes,
                                      Object[] arguments) throws Throwable {
     
                // 直接反射调用本地类方法
                return wrapper.invokeMethod(proxy, methodName, parameterTypes, arguments);
            }
        };
    }

}

最终其实就是通过wrapper.invokeMethod()，调用我们在例子中设置的setRef()的具体实现类的方法。

private static void startWithBootstrap() {
     
    ServiceConfig<DemoServiceImpl> service = new ServiceConfig<>();
    service.setInterface(DemoService.class);
    // 设置接口实现类
    service.setRef(new DemoServiceImpl());

    DubboBootstrap bootstrap = DubboBootstrap.getInstance();
    bootstrap.application(new ApplicationConfig("dubbo-demo-api-provider"))
            .registry(new RegistryConfig("zookeeper://127.0.0.1:2181"))
            .service(service)
            .start()
            .await();
}

看到这里，服务提供者的主要逻辑就梳理完毕了，简单总结下：
首先是提供者的netty server接收到了消费者发来的请求，将请求数据读取完整之后，将请求构造成一个ChannelEventRunnable，丢到netty server的worker线程池执行，执行的过程是先经过层层filter链条过滤处理，处理正常的话就通过前面JavassistProxyFactory生成的实现类的代理wrapper，代理wrapper直接调用对应的实现类的方法。

返回结果给消费者

下面我们开始从实现类的方法执行完毕后，如何层层返回到消费者的过程，再继续梳理。

上面的AbstractProxyInvoker的invoke()方法最后就是返回了AsyncRpcResult异步结果。
这里我们直接返回HeaderExchangeHandler看看如何处理返回结果的。

public class HeaderExchangeHandler implements ChannelHandlerDelegate {
     
	void handleRequest(final ExchangeChannel channel, Request req) throws RemotingException {
     
        Response res = new Response(req.getId(), req.getVersion());
        if (req.isBroken()) {
     
            Object data = req.getData();

            String msg;
            if (data == null) {
     
                msg = null;
            } else if (data instanceof Throwable) {
     
                msg = StringUtils.toString((Throwable) data);
            } else {
     
                msg = data.toString();
            }
            res.setErrorMessage("Fail to decode request due to: " + msg);
            res.setStatus(Response.BAD_REQUEST);

            channel.send(res);
            return;
        }
        // find handler by message class.
        Object msg = req.getData();
        try {
     
            // 调用DubboProtocol的reply方法
            CompletionStage<Object> future = handler.reply(channel, msg);
            // 否则等返回结果后一步调用回调
            future.whenComplete((appResult, t) -> {
     
                try {
     
                    // 如果请求已经完成，则设置结果并写回
                    if (t == null) {
     
                        res.setStatus(Response.OK);
                        res.setResult(appResult);
                    } else {
     
                        res.setStatus(Response.SERVICE_ERROR);
                        res.setErrorMessage(StringUtils.toString(t));
                    }
                    channel.send(res);
                } catch (RemotingException e) {
     
                    logger.warn("Send result to consumer failed, channel is " + channel + ", msg is " + e);
                }
            });
        } catch (Throwable e) {
     
            res.setStatus(Response.SERVICE_ERROR);
            res.setErrorMessage(StringUtils.toString(e));
            channel.send(res);
        }
    }
}

可以看到future.whenComplete()里面的处理，其实就是设置一下status和result，如果失败就是设置ErrorMessage，然后就调用channel.send()，将结果通过channel写回到消费者。

到了这里，提供者的处理就结束了，所以我们来看看消费者是如何处理的。在初始化过程中，提供者和消费者的handler处理器是同一个：HeaderExchangeHandler，由下面的代码可知：

public class HeaderExchanger implements Exchanger {
     

    public static final String NAME = "header";

    @Override
    public ExchangeClient connect(URL url, ExchangeHandler handler) throws RemotingException {
     
        // 初始化client，其中的handler是HeaderExchangeHandler
        return new HeaderExchangeClient(Transporters.connect(url, new DecodeHandler(new HeaderExchangeHandler(handler))), true);
    }

    @Override
    public ExchangeServer bind(URL url, ExchangeHandler handler) throws RemotingException {
     
        // 初始化server，其中的handler是HeaderExchangeHandler
        return new HeaderExchangeServer(Transporters.bind(url, new DecodeHandler(new HeaderExchangeHandler(handler))));
    }

}

所以这里，我们就继续看看HeaderExchangeHandler的响应消息的处理逻辑。

public class HeaderExchangeHandler implements ChannelHandlerDelegate {
     
	static void handleResponse(Channel channel, Response response) throws RemotingException {
     
        // 响应不为空并且是非心跳的响应
        if (response != null && !response.isHeartbeat()) {
     
            // 处理异步的结果
            DefaultFuture.received(channel, response);
        }
    }
}

关键的处理在DefaultFuture。

public class DefaultFuture extends CompletableFuture<Object> {
     
	// 通道缓存
    private static final Map<Long, Channel> CHANNELS = new ConcurrentHashMap<>();

    // future缓存
    private static final Map<Long, DefaultFuture> FUTURES = new ConcurrentHashMap<>();

    // 超时检查定时器
    public static final Timer TIME_OUT_TIMER = new HashedWheelTimer(
            new NamedThreadFactory("dubbo-future-timeout", true),
            30,
            TimeUnit.MILLISECONDS);

	public static void received(Channel channel, Response response) {
     
        received(channel, response, false);
    }

    public static void received(Channel channel, Response response, boolean timeout) {
     
        try {
     
            // 从FUTURES中移除id对应的future对象
            DefaultFuture future = FUTURES.remove(response.getId());
            // 如果future存在，则调用doReceived
            if (future != null) {
     
                Timeout t = future.timeoutCheckTask;
                // 非超时
                if (!timeout) {
     
                    // decrease Time
                    t.cancel();
                }
                future.doReceived(response);
            } else {
     
                logger.warn("The timeout response finally returned at "
                        + (new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS").format(new Date()))
                        + ", response status is " + response.getStatus()
                        + (channel == null ? "" : ", channel: " + channel.getLocalAddress()
                        + " -> " + channel.getRemoteAddress()) + ", please check provider side for detailed result.");
            }
        } finally {
     
            // 从缓存CHANNELS中移除请求id对应的通道
            CHANNELS.remove(response.getId());
        }
    }

	private void doReceived(Response res) {
     
        if (res == null) {
     
            throw new IllegalStateException("response cannot be null");
        }
        // 如果返回状态是成功
        if (res.getStatus() == Response.OK) {
     
            // CompletableFuture设置为完成
            this.complete(res.getResult());
        }
        // 客户端超时或服务端超时
        else if (res.getStatus() == Response.CLIENT_TIMEOUT || res.getStatus() == Response.SERVER_TIMEOUT) {
     
            // CompletableFuture设置为异常
            this.completeExceptionally(new TimeoutException(res.getStatus() == Response.SERVER_TIMEOUT, channel, res.getErrorMessage()));
        }
        // 远程调用异常
        else {
     
            this.completeExceptionally(new RemotingException(channel, res.getErrorMessage()));
        }

        // the result is returning, but the caller thread may still waiting
        // to avoid endless waiting for whatever reason, notify caller thread to return.
        // 结果已经拿到，但调用线程可能仍在等待，为了避免等待，通知调用线程返回。
        if (executor != null && executor instanceof ThreadlessExecutor) {
     
            ThreadlessExecutor threadlessExecutor = (ThreadlessExecutor) executor;
            if (threadlessExecutor.isWaiting()) {
     
                threadlessExecutor.notifyReturn(new IllegalStateException("The result has returned, but the biz thread is still waiting" +
                        " which is not an expected state, interrupt the thread manually by returning an exception."));
            }
        }
    }
}

/**
 * 通知阻塞的线程返回结果
 * tells the thread blocking on {@link #waitAndDrain()} to return, despite of the current status, to avoid endless waiting.
 */
public void notifyReturn(Throwable t) {
     
    // an empty runnable task.
    execute(() -> {
     
        waitingFuture.completeExceptionally(t);
    });
}

上面的代码其实就是将结果设置为OK，然后唤醒阻塞的线程返回结果。
如果是超时的话，就会执行TimeoutCheckTask线程的内容：

private static class TimeoutCheckTask implements TimerTask {
     

    private final Long requestID;

    TimeoutCheckTask(Long requestID) {
     
        this.requestID = requestID;
    }

    @Override
    public void run(Timeout timeout) {
     
        // 如果future为null或者已经完成，则返回
        DefaultFuture future = DefaultFuture.getFuture(requestID);
        if (future == null || future.isDone()) {
     
            return;
        }

        if (future.getExecutor() != null) {
     
            future.getExecutor().execute(() -> notifyTimeout(future));
        } else {
     
            notifyTimeout(future);
        }
    }

    /**
     * 设置为超时异常，并作为响应处理
     * @param future
     */
    private void notifyTimeout(DefaultFuture future) {
     
        // create exception response.
        Response timeoutResponse = new Response(future.getId());
        // set timeout status.
        timeoutResponse.setStatus(future.isSent() ? Response.SERVER_TIMEOUT : Response.CLIENT_TIMEOUT);
        timeoutResponse.setErrorMessage(future.getTimeoutMessage(true));
        // handle response.
        // 把超时响应信息设置到future内的通道
        DefaultFuture.received(future.getChannel(), timeoutResponse, true);
    }
}

最终将结果设置为超时异常。

最后，让我们回到消费者调用的起点位置，即是对接口方法进行代理的InvokerInvocationHandler类的invoke方法。

public class InvokerInvocationHandler implements InvocationHandler {
     
    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
     
        if (method.getDeclaringClass() == Object.class) {
     
            return method.invoke(invoker, args);
        }
        String methodName = method.getName();
        Class<?>[] parameterTypes = method.getParameterTypes();
        if (parameterTypes.length == 0) {
     
            if ("toString".equals(methodName)) {
     
                return invoker.toString();
            } else if ("$destroy".equals(methodName)) {
     
                invoker.destroy();
                return null;
            } else if ("hashCode".equals(methodName)) {
     
                return invoker.hashCode();
            }
        } else if (parameterTypes.length == 1 && "equals".equals(methodName)) {
     
            return invoker.equals(args[0]);
        }
        // RPC调用参数
        RpcInvocation rpcInvocation = new RpcInvocation(method, invoker.getInterface().getName(), args);
        String serviceKey = invoker.getUrl().getServiceKey();
        rpcInvocation.setTargetServiceUniqueName(serviceKey);
      
        if (consumerModel != null) {
     
            rpcInvocation.put(Constants.CONSUMER_MODEL, consumerModel);
            rpcInvocation.put(Constants.METHOD_MODEL, consumerModel.getMethodModel(method));
        }

        // 开始执行MockClusterInvoker的invoke方法
        // recreate()其实就是调用AppResponse的recreate()，判断是否异常并进行抛出
        return invoker.invoke(rpcInvocation).recreate();
    }
}

invoker.invoke()返回的是前面返回的DecodeableRpcResult对象，然后调用recreate()进行返回。在DecodeableRpcResult中，会先调用decode()将原本的CompatibleResult对象转换为对应的返回值类型对象。

public class DecodeableRpcResult extends AppResponse implements Codec, Decodeable {
     
	/**
     * 将RpcResult的结果按照接口方法的返回类型，构造输出对象
     * @param channel channel.
     * @param input   input stream.
     * @return
     * @throws IOException
     */
    @Override
    public Object decode(Channel channel, InputStream input) throws IOException {
     
        if (log.isDebugEnabled()) {
     
            Thread thread = Thread.currentThread();
            log.debug("Decoding in thread -- [" + thread.getName() + "#" + thread.getId() + "]");
        }

        ObjectInput in = CodecSupport.getSerialization(channel.getUrl(), serializationType)
                .deserialize(channel.getUrl(), input);

        byte flag = in.readByte();
        switch (flag) {
     
            case DubboCodec.RESPONSE_NULL_VALUE:
                break;
            case DubboCodec.RESPONSE_VALUE:
                handleValue(in);
                break;
            case DubboCodec.RESPONSE_WITH_EXCEPTION:
                handleException(in);
                break;
            case DubboCodec.RESPONSE_NULL_VALUE_WITH_ATTACHMENTS:
                handleAttachment(in);
                break;
                // 默认是这种，处理返回值和隐式参数
            case DubboCodec.RESPONSE_VALUE_WITH_ATTACHMENTS:
                handleValue(in);
                handleAttachment(in);
                break;
            case DubboCodec.RESPONSE_WITH_EXCEPTION_WITH_ATTACHMENTS:
                handleException(in);
                handleAttachment(in);
                break;
            default:
                throw new IOException("Unknown result flag, expect '0' '1' '2' '3' '4' '5', but received: " + flag);
        }
        if (in instanceof Cleanable) {
     
            ((Cleanable) in).cleanup();
        }
        return this;
    }

    @Override
    public void decode() throws Exception {
     
        if (!hasDecoded && channel != null && inputStream != null) {
     
            try {
     
                decode(channel, inputStream);
            } catch (Throwable e) {
     
                if (log.isWarnEnabled()) {
     
                    log.warn("Decode rpc result failed: " + e.getMessage(), e);
                }
                response.setStatus(Response.CLIENT_ERROR);
                response.setErrorMessage(StringUtils.toString(e));
            } finally {
     
                hasDecoded = true;
            }
        }
    }

    /**
     * 按照方法的返回类型转换对象
     * @param in
     * @throws IOException
     */
    private void handleValue(ObjectInput in) throws IOException {
     
        try {
     
            Type[] returnTypes;
            if (invocation instanceof RpcInvocation) {
     
                returnTypes = ((RpcInvocation) invocation).getReturnTypes();
            } else {
     
                returnTypes = RpcUtils.getReturnTypes(invocation);
            }
            Object value = null;
            if (ArrayUtils.isEmpty(returnTypes)) {
     
                // This almost never happens?
                value = in.readObject();
            } else if (returnTypes.length == 1) {
     
                value = in.readObject((Class<?>) returnTypes[0]);
            } else {
     
                value = in.readObject((Class<?>) returnTypes[0], returnTypes[1]);
            }
            setValue(value);
        } catch (ClassNotFoundException e) {
     
            rethrow(e);
        }
    }
}

最后在调用一下recreate()判断是否有异常并返回结果。

public class AppResponse implements Result {
     
	/**
     * 判断一下是否有异常，如果有则抛出，如果没有则返回结果
     * @return
     * @throws Throwable
     */
    @Override
    public Object recreate() throws Throwable {
     
        if (exception != null) {
     
            // fix issue#619
            try {
     
                // get Throwable class
                Class clazz = exception.getClass();
                while (!clazz.getName().equals(Throwable.class.getName())) {
     
                    clazz = clazz.getSuperclass();
                }
                // get stackTrace value
                Field stackTraceField = clazz.getDeclaredField("stackTrace");
                stackTraceField.setAccessible(true);
                Object stackTrace = stackTraceField.get(exception);
                if (stackTrace == null) {
     
                    exception.setStackTrace(new StackTraceElement[0]);
                }
            } catch (Exception e) {
     
                // ignore
            }
            throw exception;
        }
        return result;
    }
}

主流程到这里就结束了，到了这里我们就清楚了整个dubbo的提供者、消费者的初始化，包括netty的服务端和客户端的初始化和注册信息的初始化，然后消费者发起一次远程调用，提供者是如何处理的，最后消费者是如何拿到结果并进行转换的。

下一篇，准备写一些扩展的部分。