通过分布式远程调用框架xxl-rpc的源码领会rpc实现思想
目前版本：1.4.2-SNAPSHOT
官网概述：XXL-RPC 是一个分布式服务框架，提供稳定高性能的RPC远程服务调用功能。拥有"高性能、分布式、注册中心、负载均衡、服务治理"等特性。现已开放源代码，开箱即用。
一、rpc框架一般会提供的功能
1、服务提供端(Provider)
2、服务消费端(Invoker)：通过生成代理对象，封装底层通信细节 完成类似“本地方法”的调用。
3、注册中心：xxl-rpc提供 local本地注册、xxl-registry注册、zk注册等方式
4、负载均衡策略：xxl-rpc提供 随机、轮询、一致性hash、最不经常使用(LFU)、最近最久未使用(LRU)供invoker选择
5、序列化方式：xxl-rpc提供 Hessian、Jackson、Kryo、Protostuff等序列化方式
6、通信方式：xxl-rpc提供 NETTY、NETTY_HTTP、MINA、JETTY等几种通信方式
二、xxl-rpc实现思路：
Provider向注册中心注册服务信息(提供了三种注册方式)，Invoker的代理对象会从注册中心发现可用的服务信息、根据选择的负载均衡策略调用服务 并根据选择的底层通信方式和序列化方式进行网络间通信。当然这些由xxl-rpc封装实现，对用户透明，使用时只需要通过配置+注解即可工作。
三、系统架构图(虽然v1.3版本, 但和1.4版本思想不变)：
四、实现
1、系统核心结构图：
2、服务提供者(Provider):
a、核心类：XxlRpcProviderFactory
部分参数结构：
public class XxlRpcProviderFactory {
private static final Logger logger = LoggerFactory.getLogger(XxlRpcProviderFactory.class);
// ---------------------- config ----------------------
private NetEnum netType; // Netty/Jetty等网络通信类型
private Serializer serializer; // Protostuff/Jackson/Hessian等序列化方式
private String ip; // for registry
private int port; // default port
private String accessToken;
private Class serviceRegistryClass; // local/xxl-rpc/zk等注册中心
private Map serviceRegistryParam; // 服务注册参数 start方法，注册实例的初始化，服务注册/下线后的一系列抽象方法进行善后工作：
public void start() throws Exception {
// start server
serviceAddress = IpUtil.getIpPort(this.ip, port);
// 实例化net通信服务类
server = netType.serverClass.newInstance();
// 通过钩子方法在通信服务启动关闭时，对服务注册中心进行一系列操作
server.setStartedCallback(new BaseCallback() { // serviceRegistry started
@Override
public void run() throws Exception {
// start registry
if (serviceRegistryClass != null) {
// 服务启动时注册中心实例化
serviceRegistry = serviceRegistryClass.newInstance();
serviceRegistry.start(serviceRegistryParam);
if (serviceData.size() > 0) {
serviceRegistry.registry(serviceData.keySet(), serviceAddress);
}
}
}
});
server.setStopedCallback(new BaseCallback() { // serviceRegistry stoped
@Override
public void run() {
// stop registry
if (serviceRegistry != null) {
if (serviceData.size() > 0) {
// 服务下线时, 服务从注册中心摘除
serviceRegistry.remove(serviceData.keySet(), serviceAddress);
}
serviceRegistry.stop();
serviceRegistry = null;
}
}
});
server.start(this);
}真正的服务提供实现方法invokeService：
/**
* 由于与服务消费者的交互都是通过net封装的通信层实现，因此该方法在net通信实现中调用
* invoke service
*
* @param xxlRpcRequest
* @return
*/
public XxlRpcResponse invokeService(XxlRpcRequest xxlRpcRequest) {
// make response
XxlRpcResponse xxlRpcResponse = new XxlRpcResponse();
xxlRpcResponse.setRequestId(xxlRpcRequest.getRequestId());
// match service bean
String serviceKey = makeServiceKey(xxlRpcRequest.getClassName(), xxlRpcRequest.getVersion());
Object serviceBean = serviceData.get(serviceKey);
// valid
if (serviceBean == null) {
xxlRpcResponse.setErrorMsg("The serviceKey["+ serviceKey +"] not found.");
return xxlRpcResponse;
}
if (System.currentTimeMillis() - xxlRpcRequest.getCreateMillisTime() > 3*60*1000) {
xxlRpcResponse.setErrorMsg("The timestamp difference between admin and executor exceeds the limit.");
return xxlRpcResponse;
}
if (accessToken!=null && accessToken.trim().length()>0 && !accessToken.trim().equals(xxlRpcRequest.getAccessToken())) {
xxlRpcResponse.setErrorMsg("The access token[" + xxlRpcRequest.getAccessToken() + "] is wrong.");
return xxlRpcResponse;
}
// 通过反射拿到对应的实现类，并调用，将处理结果返回给服务消费者
try {
// invoke
Class serviceClass = serviceBean.getClass();
String methodName = xxlRpcRequest.getMethodName();
Class[] parameterTypes = xxlRpcRequest.getParameterTypes();
Object[] parameters = xxlRpcRequest.getParameters();
Method method = serviceClass.getMethod(methodName, parameterTypes);
method.setAccessible(true);
Object result = method.invoke(serviceBean, parameters);
/*FastClass serviceFastClass = FastClass.create(serviceClass);
FastMethod serviceFastMethod = serviceFastClass.getMethod(methodName, parameterTypes);
Object result = serviceFastMethod.invoke(serviceBean, parameters);*/
xxlRpcResponse.setResult(result);
} catch (Throwable t) {
// catch error
logger.error("xxl-rpc provider invokeService error.", t);
xxlRpcResponse.setErrorMsg(ThrowableUtil.toString(t));
}
return xxlRpcResponse;
}以上实现已可以正常提供服务，实现服务注册、摘除并可被消费端调用；但强大的Spring提供了一系列扩张工具类，下面看看：
b、XxlRpcProviderFactory的实现类XxlRpcSpringProviderFactory实现Spring的一些拓展类，方便处理初始化和“善后”工作：
/**
* xxl-rpc provider (for spring)
*
* @author xuxueli 2018-10-18 18:09:20
*/
public class XxlRpcSpringProviderFactory extends XxlRpcProviderFactory
implements ApplicationContextAware, InitializingBean,DisposableBean {
………
………
// ---------------------- util ----------------------
// 从应用上下文中拿到XxlRpcService注解的bean，实例化后并向注册中心添加服务
@Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
Map serviceBeanMap = applicationContext.getBeansWithAnnotation(XxlRpcService.class);
if (serviceBeanMap!=null && serviceBeanMap.size()>0) {
for (Object serviceBean : serviceBeanMap.values()) {
// valid
if (serviceBean.getClass().getInterfaces().length ==0) {
throw new XxlRpcException("xxl-rpc, service(XxlRpcService) must inherit interface.");
}
// add service
XxlRpcService xxlRpcService = serviceBean.getClass().getAnnotation(XxlRpcService.class);
String iface = serviceBean.getClass().getInterfaces()[0].getName();
String version = xxlRpcService.version();
super.addService(iface, version, serviceBean);
}
}
// TODO，addServices by api + prop
}
// 初始化和“善后”操作
@Override
public void afterPropertiesSet() throws Exception {
this.prepareConfig();
super.start();
}
@Override
public void destroy() throws Exception {
super.stop();
} 这样结合Spring的强大注解，即可轻松实现配置(这里是针对使用spring的服务可以轻松简洁的接入)-相比之下就不用在手动的注册服务了，如下：
@Configuration
public class XxlRpcProviderConfig {
private Logger logger = LoggerFactory.getLogger(XxlRpcProviderConfig.class);
@Value("${xxl-rpc.remoting.port}")
private int port;
@Value("${xxl-rpc.registry.xxlregistry.address}")
private String address;
@Value("${xxl-rpc.registry.xxlregistry.env}")
private String env;
@Bean
public XxlRpcSpringProviderFactory xxlRpcSpringProviderFactory() {
XxlRpcSpringProviderFactory providerFactory = new XxlRpcSpringProviderFactory();
providerFactory.setPort(port);
providerFactory.setServiceRegistryClass(XxlRegistryServiceRegistry.class);
providerFactory.setServiceRegistryParam(new HashMap(){{
put(XxlRegistryServiceRegistry.XXL_REGISTRY_ADDRESS, address);
put(XxlRegistryServiceRegistry.ENV, env);
}});
logger.info(">>>>>>>>>>> xxl-rpc provider config init finish.");
return providerFactory;
}
} ------------------------OK以上是Provider基本思路----------------------
2、服务消费者(Invoker):
a、核心实现类XxlRpcReferenceBean，该类会提供代理对象，封装底层通信细节等
public class XxlRpcReferenceBean {
private static final Logger logger = LoggerFactory.getLogger(XxlRpcReferenceBean.class);
// ---------------------- config ----------------------
private NetEnum netType; // net 通信方式
private Serializer serializer; // 序列化方式
private CallType callType; // 回调方式(sync/future等)
private LoadBalance loadBalance; // 负载均衡方式
private Class iface;
private String version;
private long timeout = 1000;
private String address;
private String accessToken;
private XxlRpcInvokeCallback invokeCallback; // 回调方式
private XxlRpcInvokerFactory invokerFactory; // 调用工厂，注册中心及服务选择，下一步详细介绍
主要实现方法->创建代理对象，如下
// ---------------------- util ----------------------
public Object getObject() {
return Proxy.newProxyInstance(Thread.currentThread()
.getContextClassLoader(), new Class[] { iface },
new InvocationHandler() {
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
// 1、target方法名以及传递参数等获取
// method param
String className = method.getDeclaringClass().getName(); // iface.getName()
String varsion_ = version;
String methodName = method.getName();
Class[] parameterTypes = method.getParameterTypes();
Object[] parameters = args;
// filter for generic
if (className.equals(XxlRpcGenericService.class.getName()) && methodName.equals("invoke")) {
Class[] paramTypes = null;
if (args[3]!=null) {
String[] paramTypes_str = (String[]) args[3];
if (paramTypes_str.length > 0) {
paramTypes = new Class[paramTypes_str.length];
for (int i = 0; i < paramTypes_str.length; i++) {
paramTypes[i] = ClassUtil.resolveClass(paramTypes_str[i]);
}
}
}
className = (String) args[0];
varsion_ = (String) args[1];
methodName = (String) args[2];
parameterTypes = paramTypes;
parameters = (Object[]) args[4];
}
// filter method like "Object.toString()"
if (className.equals(Object.class.getName())) {
logger.info(">>>>>>>>>>> xxl-rpc proxy class-method not support [{}#{}]", className, methodName);
throw new XxlRpcException("xxl-rpc proxy class-method not support");
}
// 2、注册中心地址集选择，并选择合适的负载均衡策略 得到服务的地址
// address
String finalAddress = address;
if (finalAddress==null || finalAddress.trim().length()==0) {
if (invokerFactory!=null && invokerFactory.getServiceRegistry()!=null) {
// discovery
String serviceKey = XxlRpcProviderFactory.makeServiceKey(className, varsion_);
TreeSet addressSet = invokerFactory.getServiceRegistry().discovery(serviceKey);
// load balance
if (addressSet==null || addressSet.size()==0) {
// pass
} else if (addressSet.size()==1) {
finalAddress = addressSet.first();
} else {
finalAddress = loadBalance.xxlRpcInvokerRouter.route(serviceKey, addressSet);
}
}
}
if (finalAddress==null || finalAddress.trim().length()==0) {
throw new XxlRpcException("xxl-rpc reference bean["+ className +"] address empty");
}
// 3、以下是请求方式的选择sync/future/callback等，选择之后调用服务即可
// request
XxlRpcRequest xxlRpcRequest = new XxlRpcRequest();
xxlRpcRequest.setRequestId(UUID.randomUUID().toString());
xxlRpcRequest.setCreateMillisTime(System.currentTimeMillis());
xxlRpcRequest.setAccessToken(accessToken);
xxlRpcRequest.setClassName(className);
xxlRpcRequest.setMethodName(methodName);
xxlRpcRequest.setParameterTypes(parameterTypes);
xxlRpcRequest.setParameters(parameters);
// send
if (CallType.SYNC == callType) {
// future-response set
XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(invokerFactory, xxlRpcRequest, null);
try {
// do invoke
client.asyncSend(finalAddress, xxlRpcRequest);
// future get
XxlRpcResponse xxlRpcResponse = futureResponse.get(timeout, TimeUnit.MILLISECONDS);
if (xxlRpcResponse.getErrorMsg() != null) {
throw new XxlRpcException(xxlRpcResponse.getErrorMsg());
}
return xxlRpcResponse.getResult();
} catch (Exception e) {
logger.info(">>>>>>>>>>> xxl-rpc, invoke error, address:{}, XxlRpcRequest{}", finalAddress, xxlRpcRequest);
throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
} finally{
// future-response remove
futureResponse.removeInvokerFuture();
}
} else if (CallType.FUTURE == callType) {
// future-response set
XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(invokerFactory, xxlRpcRequest, null);
try {
// invoke future set
XxlRpcInvokeFuture invokeFuture = new XxlRpcInvokeFuture(futureResponse);
XxlRpcInvokeFuture.setFuture(invokeFuture);
// do invoke
client.asyncSend(finalAddress, xxlRpcRequest);
return null;
} catch (Exception e) {
logger.info(">>>>>>>>>>> xxl-rpc, invoke error, address:{}, XxlRpcRequest{}", finalAddress, xxlRpcRequest);
// future-response remove
futureResponse.removeInvokerFuture();
throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
}
} else if (CallType.CALLBACK == callType) {
// get callback
XxlRpcInvokeCallback finalInvokeCallback = invokeCallback;
XxlRpcInvokeCallback threadInvokeCallback = XxlRpcInvokeCallback.getCallback();
if (threadInvokeCallback != null) {
finalInvokeCallback = threadInvokeCallback;
}
if (finalInvokeCallback == null) {
throw new XxlRpcException("xxl-rpc XxlRpcInvokeCallback（CallType="+ CallType.CALLBACK.name() +"） cannot be null.");
}
// future-response set
XxlRpcFutureResponse futureResponse = new XxlRpcFutureResponse(invokerFactory, xxlRpcRequest, finalInvokeCallback);
try {
client.asyncSend(finalAddress, xxlRpcRequest);
} catch (Exception e) {
logger.info(">>>>>>>>>>> xxl-rpc, invoke error, address:{}, XxlRpcRequest{}", finalAddress, xxlRpcRequest);
// future-response remove
futureResponse.removeInvokerFuture();
throw (e instanceof XxlRpcException)?e:new XxlRpcException(e);
}
return null;
} else if (CallType.ONEWAY == callType) {
client.asyncSend(finalAddress, xxlRpcRequest);
return null;
} else {
throw new XxlRpcException("xxl-rpc callType["+ callType +"] invalid");
}
}
});
}
2、Invoker调用工厂XxlRpcInvokerFactory：
/**
* xxl-rpc invoker factory, init service-registry
*
* @author xuxueli 2018-10-19
*/
public class XxlRpcInvokerFactory {
private static Logger logger = LoggerFactory.getLogger(XxlRpcInvokerFactory.class);
// ---------------------- default instance ----------------------
private static volatile XxlRpcInvokerFactory instance = new XxlRpcInvokerFactory(LocalServiceRegistry.class, null);
public static XxlRpcInvokerFactory getInstance() {
return instance;
}
// ---------------------- config ----------------------
private Class serviceRegistryClass; // class.forname
private Map serviceRegistryParam; 和服务端类似，负载与注册中心交互的核心类，不在赘述
--------- 以上服务消费端已经基本可以完成通信，但强大的Spring仍提供的一些很好的扩张，如下 --------
3、基于Spring的实现类XxlRpcSpringInvokerFactory：
/**
* xxl-rpc invoker factory, init service-registry and spring-bean by annotation (for spring)
*
* @author xuxueli 2018-10-19
*/
public class XxlRpcSpringInvokerFactory extends InstantiationAwareBeanPostProcessorAdapter
implements InitializingBean,DisposableBean, BeanFactoryAware {
private Logger logger = LoggerFactory.getLogger(XxlRpcSpringInvokerFactory.class);
// ---------------------- config ----------------------
private Class serviceRegistryClass; // class.forname
private Map serviceRegistryParam;
public void setServiceRegistryClass(Class serviceRegistryClass) {
this.serviceRegistryClass = serviceRegistryClass;
}
public void setServiceRegistryParam(Map serviceRegistryParam) {
this.serviceRegistryParam = serviceRegistryParam;
}
// ---------------------- util ----------------------
private XxlRpcInvokerFactory xxlRpcInvokerFactory; 实现Spring的一些拓展类 以方便更好的支持注解、服务init、destory等，如下：
// 1、该类参数初始化完成后进行处理
@Override
public void afterPropertiesSet() throws Exception {
// start invoker factory
xxlRpcInvokerFactory = new XxlRpcInvokerFactory(serviceRegistryClass, serviceRegistryParam);
xxlRpcInvokerFactory.start();
}
// 2、实例化之后进行处理
@Override
public boolean postProcessAfterInstantiation(final Object bean, final String beanName) throws BeansException {
// 3、这里其实通过遍历所有的bean找到通过有XxlRpcReference注解过的bean的字段，并将该字段实例化，通过XxlRpcReferenceBean(开始介绍的核心类)拿到代理对象，然后就和之前一样了
// collection
final Set serviceKeyList = new HashSet<>();
// parse XxlRpcReferenceBean
ReflectionUtils.doWithFields(bean.getClass(), new ReflectionUtils.FieldCallback() {
@Override
public void doWith(Field field) throws IllegalArgumentException, IllegalAccessException {
if (field.isAnnotationPresent(XxlRpcReference.class)) {
// valid
Class iface = field.getType();
if (!iface.isInterface()) {
throw new XxlRpcException("xxl-rpc, reference(XxlRpcReference) must be interface.");
}
XxlRpcReference rpcReference = field.getAnnotation(XxlRpcReference.class);
// init reference bean
XxlRpcReferenceBean referenceBean = new XxlRpcReferenceBean(
rpcReference.netType(),
rpcReference.serializer().getSerializer(),
rpcReference.callType(),
rpcReference.loadBalance(),
iface,
rpcReference.version(),
rpcReference.timeout(),
rpcReference.address(),
rpcReference.accessToken(),
null,
xxlRpcInvokerFactory
);
Object serviceProxy = referenceBean.getObject();
// set bean
field.setAccessible(true);
field.set(bean, serviceProxy);
logger.info(">>>>>>>>>>> xxl-rpc, invoker factory init reference bean success. serviceKey = {}, bean.field = {}.{}",
XxlRpcProviderFactory.makeServiceKey(iface.getName(), rpcReference.version()), beanName, field.getName());
// collection
String serviceKey = XxlRpcProviderFactory.makeServiceKey(iface.getName(), rpcReference.version());
serviceKeyList.add(serviceKey);
}
}
});
// mult discovery
if (xxlRpcInvokerFactory.getServiceRegistry() != null) {
try {
xxlRpcInvokerFactory.getServiceRegistry().discovery(serviceKeyList);
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
return super.postProcessAfterInstantiation(bean, beanName);
}
// 4、服务销毁, “善后”与注册中心的联系
@Override
public void destroy() throws Exception {
// stop invoker factory
xxlRpcInvokerFactory.stop();
}
private BeanFactory beanFactory;
@Override
public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
this.beanFactory = beanFactory;
} 这样之后Spring接入方可以通过简洁的配置即可 如下：
@Configuration
public class XxlRpcInvokerConfig {
private Logger logger = LoggerFactory.getLogger(XxlRpcInvokerConfig.class);
@Value("${xxl-rpc.registry.xxlregistry.address}")
private String address;
@Value("${xxl-rpc.registry.xxlregistry.env}")
private String env;
@Bean
public XxlRpcSpringInvokerFactory xxlJobExecutor() {
XxlRpcSpringInvokerFactory invokerFactory = new XxlRpcSpringInvokerFactory();
invokerFactory.setServiceRegistryClass(XxlRegistryServiceRegistry.class);
invokerFactory.setServiceRegistryParam(new HashMap(){{
put(XxlRegistryServiceRegistry.XXL_REGISTRY_ADDRESS, address);
put(XxlRegistryServiceRegistry.ENV, env);
}});
logger.info(">>>>>>>>>>> xxl-rpc invoker config init finish.");
return invokerFactory;
}
} --------------------以上基本的服务provider/invoker介绍完毕-----------------------
还有一些通信细节没有介绍，如序列化方式、负载均衡策略、Net通信方式、注册方式以及回调方式，这些不是本文重点，可以看作工具 以供xxl-rpc的用户选择使用, 接入方可以通过配置即可轻松选择；其中：
1、Net通信方式如Netty、Mina等都是不小的话题，可以拓展学习
2、注册方式：这里通过local、xxl-registry、zk等方式，在生产环境也很常用，可以拓展学习
以上是个人理解，如有问题请指出，谢谢！