万万没想到一个xxl-job源码分析，竟然能引发这么多血案！(上)

前言

XxlJob 2.0.1版本源码，其实在去年这个时候我已经看完了，并且做了很详细的注释。但是由于自己太懒了，没有写成博客分享。俗话说好记性不如烂笔头，于是乎我挑出几个源码实现中我认为不错的知识点并且结合自己的见解，来分享一波。

image.png

因为篇幅过于长，不让发，要分成上下节！

---

源码思考

• 1.executor端是开启一个jettyServer,其中配置了JettyServerHandler。将请求交给XxlRpcResponse xxlRpcResponse = xxlRpcProviderFactory.invokeService(xxlRpcRequest);

详细可以看JettyServerHandler中的handle函数

    @Override
    public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {

        if ("/services".equals(target)) {   // services mapping

            StringBuffer stringBuffer = new StringBuffer("");
            for (String serviceKey: xxlRpcProviderFactory.getServiceData().keySet()) {
                stringBuffer.append("
").append(serviceKey).append(": ").append(xxlRpcProviderFactory.getServiceData().get(serviceKey)).append("
");
            }
            stringBuffer.append("");

            writeResponse(baseRequest, response, stringBuffer.toString().getBytes());
            return;
        } else {    // default remoting mapping

            // request parse
            XxlRpcRequest xxlRpcRequest = null;
            try {

                xxlRpcRequest = parseRequest(request);
            } catch (Exception e) {
                writeResponse(baseRequest, response, ThrowableUtil.toString(e).getBytes());
                return;
            }

            // invoke
            XxlRpcResponse xxlRpcResponse = xxlRpcProviderFactory.invokeService(xxlRpcRequest);

            // response-serialize + response-write
            byte[] responseBytes = xxlRpcProviderFactory.getSerializer().serialize(xxlRpcResponse);
            writeResponse(baseRequest, response, responseBytes);
        }

    }

executor端通过上文中xxlRpcProviderFactory.invokeService(xxlRpcRequest);函数执行本地暴露的服务方法

    /**
     * invoke service
     *
     * @param xxlRpcRequest
     * @return
     */
    public XxlRpcResponse invokeService(XxlRpcRequest xxlRpcRequest) {

        //  make response
        XxlRpcResponse xxlRpcResponse = new XxlRpcResponse();
        xxlRpcResponse.setRequestId(xxlRpcRequest.getRequestId());

        // match service bean
        String serviceKey = makeServiceKey(xxlRpcRequest.getClassName(), xxlRpcRequest.getVersion());
        Object serviceBean = serviceData.get(serviceKey);

        // valid
        if (serviceBean == null) {
            xxlRpcResponse.setErrorMsg("The serviceKey["+ serviceKey +"] not found.");
            return xxlRpcResponse;
        }

        if (System.currentTimeMillis() - xxlRpcRequest.getCreateMillisTime() > 3*60*1000) {
            xxlRpcResponse.setErrorMsg("The timestamp difference between admin and executor exceeds the limit.");
            return xxlRpcResponse;
        }
        if (accessToken!=null && accessToken.trim().length()>0 && !accessToken.trim().equals(xxlRpcRequest.getAccessToken())) {
            xxlRpcResponse.setErrorMsg("The access token[" + xxlRpcRequest.getAccessToken() + "] is wrong.");
            return xxlRpcResponse;
        }

        // invoke
        try {
            Class serviceClass = serviceBean.getClass();
            String methodName = xxlRpcRequest.getMethodName();
            Class[] parameterTypes = xxlRpcRequest.getParameterTypes();
            Object[] parameters = xxlRpcRequest.getParameters();

            Method method = serviceClass.getMethod(methodName, parameterTypes);
            method.setAccessible(true);
            Object result = method.invoke(serviceBean, parameters);

            /*FastClass serviceFastClass = FastClass.create(serviceClass);
            FastMethod serviceFastMethod = serviceFastClass.getMethod(methodName, parameterTypes);
            Object result = serviceFastMethod.invoke(serviceBean, parameters);*/

            xxlRpcResponse.setResult(result);
        } catch (Throwable t) {
            logger.error("xxl-rpc provider invokeService error.", t);
            xxlRpcResponse.setErrorMsg(ThrowableUtil.toString(t));
        }

        return xxlRpcResponse;
    }

executor端服务提供者维护在serviceData中, serviceData是map结构。key是服务提供者className拼接版本号, value则是服务提供者本身实例。

    // ---------------------- server invoke ----------------------

    /**
     * init local rpc service map
     */
    private Map serviceData = new HashMap();
    public Map getServiceData() {
        return serviceData;
    }

admin端其实是没有开启jettyServer服务器，可以通过XxlJobDynamicScheduler的
initRpcProvider()一看究竟。

    // ---------------------- init + destroy ----------------------
    public void start() throws Exception {
        // valid
        Assert.notNull(scheduler, "quartz scheduler is null");

        // init i18n
        initI18n();

        // admin registry monitor run
        /*
         启动自动注册线程，获取类型为自动注册的执行器信息，完成机器的自动注册与发现
         */
        JobRegistryMonitorHelper.getInstance().start();

        // admin monitor run
        /**
         * 启动失败日志监控线程
         */
        JobFailMonitorHelper.getInstance().start();

        // admin-server
        /**
         * 暴露AdminBiz服务,并设置jettyServerHandler
         */
        initRpcProvider();

        logger.info(">>>>>>>>> init xxl-job admin success.");
    }

    // ---------------------- admin rpc provider (no server version) ----------------------
    private static JettyServerHandler jettyServerHandler;
    private void initRpcProvider(){
        // init
        XxlRpcProviderFactory xxlRpcProviderFactory = new XxlRpcProviderFactory();
        xxlRpcProviderFactory.initConfig(NetEnum.JETTY, Serializer.SerializeEnum.HESSIAN.getSerializer(), null, 0, XxlJobAdminConfig.getAdminConfig().getAccessToken(), null, null);

        // add services
        xxlRpcProviderFactory.addService(AdminBiz.class.getName(), null, XxlJobAdminConfig.getAdminConfig().getAdminBiz());

        // jetty handler
        jettyServerHandler = new JettyServerHandler(xxlRpcProviderFactory);
    }
    private void stopRpcProvider() throws Exception {
        new XxlRpcInvokerFactory().stop();
    }
    public static void invokeAdminService(HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {
        jettyServerHandler.handle(null, new Request(null, null), request, response);
    }

admin端通过API服务暴露出自己的服务。比如执行器注册服务，任务结果回调服务。admin端服务暴露都是通过JobApiController实现的，来达到和executor端类似的效果,请求交给JettyServerHandler处理, 然后通过xxlRpcProviderFacotry调用本地方法。

@Controller
public class JobApiController implements InitializingBean {


    @Override
    public void afterPropertiesSet() throws Exception {

    }

    @RequestMapping(AdminBiz.MAPPING)
    @PermessionLimit(limit=false)
    public void api(HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {
        XxlJobDynamicScheduler.invokeAdminService(request, response);
    }
}

XxlRpcReferenceBean这个factoryBean中的getObject方法用于创建代理对象，
代理逻辑中过滤掉了非业务方法,也就是Object类中的方法。只是将目标类的方法名，参数,类名等信息包装成XxlRpcRequest，通过JettyClient发送给调度中心。调度中心的接口地址为"admin端的ip/api"。调度中心的API接口拿到请求之后通过参数里面的类名,方法,参数,版本号等信息反射出来一个服务实例,调用invoke执行方法，也就是上文中提到的JobApiController。

每个XxlRpcReferenceBean对象中都会初始化一个JettyClient对象。感觉这样做有点耗性能，需要优化啊。毕竟创建一个JettyClient对象开销并不小，也许你使用操作不恰当会造成OOM。之前有一位朋友就是对每一个请求都创建了一个HttpClient，这样由于创建每一个HttpClient实例的时候都会调用evictExpireConnections，造成有多少请求就会创建多少个定时线程，最后造成系统OOM。所以建议这里最好采用单例或者将JettyClient缓存起来。

    // ---------------------- initClient ----------------------

    Client client = null;

    private void initClient() {
        try {
            client = netType.clientClass.newInstance();
            client.init(this);
        } catch (InstantiationException | IllegalAccessException e) {
            throw new XxlRpcException(e);
        }
    }

创建发起RPC请求的代理对象

public Object getObject() {
        return Proxy.newProxyInstance(Thread.currentThread()
                .getContextClassLoader(), new Class[] { iface },
                new InvocationHandler() {
                    @Override
                    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                        String className = method.getDeclaringClass().getName();

                        // filter method like "Object.toString()"
                        if (Object.class.getName().equals(className)) {
                            logger.info(">>>>>>>>>>> xxl-rpc proxy class-method not support [{}.{}]", className, method.getName());
                            throw new XxlRpcException("xxl-rpc proxy class-method not support");
                        }

                        // address
                        String address = routeAddress();
                        if (address==null || address.trim().length()==0) {
                            throw new XxlRpcException("xxl-rpc reference bean["+ className +"] address empty");
                        }

                        // request
                        XxlRpcRequest xxlRpcRequest = new XxlRpcRequest();
                        xxlRpcRequest.setRequestId(UUID.randomUUID().toString());
                        xxlRpcRequest.setCreateMillisTime(System.currentTimeMillis());
                        xxlRpcRequest.setAccessToken(accessToken);
                        xxlRpcRequest.setClassName(className);
                        xxlRpcRequest.setMethodName(method.getName());
                        xxlRpcRequest.setParameterTypes(method.getParameterTypes());
                        xxlRpcRequest.setParameters(args);
                        
                        // send
                        if (CallType.SYNC == callType) {
                            try {
                                // future set
                                XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(xxlRpcRequest, null);

                                // do invoke
                                client.asyncSend(address, xxlRpcRequest);

                                // future get
                                XxlRpcResponse xxlRpcResponse = futureResponse.get(timeout, TimeUnit.MILLISECONDS);
                                if (xxlRpcResponse.getErrorMsg() != null) {
                                    throw new XxlRpcException(xxlRpcResponse.getErrorMsg());
                                }
                                return xxlRpcResponse.getResult();
                            } catch (Exception e) {
                                logger.info(">>>>>>>>>>> xxl-job, invoke error, address:{}, XxlRpcRequest{}", address, xxlRpcRequest);

                                throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
                            } finally{
                                // remove-InvokerFuture
                                XxlRpcFutureResponseFactory.removeInvokerFuture(xxlRpcRequest.getRequestId());
                            }
                        } else if (CallType.FUTURE == callType) {

                            // thread future set
                            XxlRpcInvokeFuture invokeFuture = null;
                            try {
                                // future set
                                invokeFuture = new XxlRpcInvokeFuture(new XxlRpcFutureResponse(xxlRpcRequest, null));
                                XxlRpcInvokeFuture.setFuture(invokeFuture);

                                // do invoke
                                client.asyncSend(address, xxlRpcRequest);

                                return null;
                            } catch (Exception e) {
                                logger.info(">>>>>>>>>>> xxl-job, invoke error, address:{}, XxlRpcRequest{}", address, xxlRpcRequest);

                                // remove-InvokerFuture
                                invokeFuture.stop();

                                throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
                            }

                        } else if (CallType.CALLBACK == callType) {

                            // get callback
                            XxlRpcInvokeCallback finalInvokeCallback = invokeCallback;
                            XxlRpcInvokeCallback threadInvokeCallback = XxlRpcInvokeCallback.getCallback();
                            if (threadInvokeCallback != null) {
                                finalInvokeCallback = threadInvokeCallback;
                            }
                            if (finalInvokeCallback == null) {
                                throw new XxlRpcException("xxl-rpc XxlRpcInvokeCallback（CallType="+ CallType.CALLBACK.name() +"） cannot be null.");
                            }

                            try {
                                // future set
                                XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(xxlRpcRequest, finalInvokeCallback);

                                client.asyncSend(address, xxlRpcRequest);
                            } catch (Exception e) {
                                logger.info(">>>>>>>>>>> xxl-job, invoke error, address:{}, XxlRpcRequest{}", address, xxlRpcRequest);

                                // future remove
                                XxlRpcFutureResponseFactory.removeInvokerFuture(xxlRpcRequest.getRequestId());

                                throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
                            }

                            return null;
                        } else if (CallType.ONEWAY == callType) {
                            client.asyncSend(address, xxlRpcRequest);
                            return null;
                        } else {
                            throw new XxlRpcException("xxl-rpc callType["+ callType +"] invalid");
                        }

                    }
                });
    }

这里只讲下SYNC模式的请求，XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(xxlRpcRequest, null);创建XxlRpcFutureResponse对象(Future任务执行的结果), 这里的invokeCallback回调是null。

    public XxlRpcFutureResponse(XxlRpcRequest request, XxlRpcInvokeCallback invokeCallback) {
        this.request = request;
        this.invokeCallback = invokeCallback;

        // set-InvokerFuture
        XxlRpcFutureResponseFactory.setInvokerFuture(request.getRequestId(), this);
    }

当我们获取Future执行结果时, XxlRpcFutureResponse中的done变量如果是false, 一直阻塞线程(还有超时机制), 除非有调用setResponse(XxlRpcResponse response)方法

使done变量为true, 并获取锁,调用lock.notifyAll(), 唤醒线程,并返回执行结果。

在JettyClient中发送请求，调用asyncSend方法

@Override
public void asyncSend(String address, XxlRpcRequest xxlRpcRequest) throws Exception {
    // do invoke
    postRequestAsync(address, xxlRpcRequest);
}

在postRequestAsync方法拿到RPC请求执行结果，我们应该通知future，唤醒等待。

// deserialize response
XxlRpcResponse xxlRpcResponse = (XxlRpcResponse) xxlRpcReferenceBean.getSerializer().deserialize(responseBytes, XxlRpcResponse.class);
// notify response
XxlRpcFutureResponseFactory.notifyInvokerFuture(xxlRpcResponse.getRequestId(), xxlRpcResponse);

在初始化XxlRpcFutureResponse中, 我们有调用setInvokerFuture方法将消息和XxlRpcFutureResponse结果维护起来。

当在JettyClient执行完请求获取结果时, 调用notifyInvokerFuture方法设置XxlRpcFutureResponse中的xxlRpcResponse属性
也就是真实的执行结果。

JettyClient方法请求虽然是异步的, 但是这里还是同步阻塞获取执行结果。

public class XxlRpcFutureResponseFactory {

    private static ConcurrentMap futureResponsePool = new ConcurrentHashMap();

    public static void setInvokerFuture(String requestId, XxlRpcFutureResponse futureResponse){
        // TODO，running future method-isolation and limit
        futureResponsePool.put(requestId, futureResponse);
    }

    public static void removeInvokerFuture(String requestId){
        futureResponsePool.remove(requestId);
    }

    public static void notifyInvokerFuture(String requestId, XxlRpcResponse xxlRpcResponse){
        XxlRpcFutureResponse futureResponse = futureResponsePool.get(requestId);
        if (futureResponse != null) {
            futureResponse.setResponse(xxlRpcResponse);
            futureResponsePool.remove(requestId);
        }
    }

}

• 2.XxlJob的核心包括XxlRpc,上文大致提到过其简单的流程。如果我们想要实现一个RPC，应该要做些什么呢？比如序列化，压缩算法，协议，动态代理，服务注册，加密，网络编码，连接管理，健康检测，负载均衡，优雅启停机，异常重试，业务分组以及熔断限流等等。由于之前写过一个基于Netty简单的RPC框架，因此可以通过和XxlRpc对比来查漏补缺。(这里不会讲Netty，只会粗略讲一下Rpc代理)

netty-rpc-client端扫描需要代理服务的接口并且修改BeanDefinition初始化的方式。在Spring容器中实例化一个对象的方式有:Supplier，FactoryBean，指定FactoryMethodName和FactoryBeanName,Constructor等等。

@Slf4j
public class ClassPathRpcScanner extends ClassPathBeanDefinitionScanner {

    private RpcFactoryBean rpcFactoryBean = new RpcFactoryBean