以下面最小化代码初始pigeon服务提供者为例,分析pigeon完成服务注册的流程:
public static void main(String[] args) throws Exception {
// 初始化ProviderConfig,服务接口是EchoService,具体服务提供者是EchoServiceDefaultImpl
ProviderConfig<EchoService> providerConfig = new ProviderConfig<>(EchoService.class,
new EchoServiceDefaultImpl());
// 注册服务
ServiceFactory.addService(providerConfig);
System.in.read();
}
每个接口服务都以一个ProviderConfig类作为元数据,里面定义的相关属性如下:
// T为提供服务的抽象接口类型
public class ProviderConfig<T> {
// 提供服务的抽象接口类型
private Class<?> serviceInterface;
// 访问服务的url
private String url;
// 服务版本,调用时会附到url上区分不同版本调用
private String version;
// 具体服务示例类
private T service;
// 服务器配置
private ServerConfig serverConfig = new ServerConfig();
// 是否已发布
private boolean published = false;
// 调用超时
private boolean cancelTimeout = Constants.DEFAULT_TIMEOUT_CANCEL;
// 配置中心
private ConfigManager configManager = ConfigManagerLoader.getConfigManager();
// 如果useSharedPool为false,pigeon就会为每个方法设置独立的线程池执行请求。
private boolean useSharedPool = configManager.getBooleanValue(Constants.KEY_SERVICE_SHARED,
Constants.DEFAULT_SERVICE_SHARED);
// 保存当前服务接口的相关调用方法
private Map<String, ProviderMethodConfig> methods;
// 单独设置某个方法的最大并发数,如果并发超过设置的最大并发数,服务端会抛出com.dianping.pigeon.remoting.common.exception.RejectedException异常,客户端也会收到这个异常。
private int actives = 0;
// 是否支持新协议,如thrift
private boolean supported;
// 独立线程池配置,通过poolConfig --> threadPool找到对应的线程池实例
private PoolConfig poolConfig;
}
如果说一个ProviderConfig映射一个服务调用接口,一个ServerConfig则对应一个服务器进程,具体服务器可能为基于netty的RPC服务器或jetty的http服务器。
ServerConfig相关配置如下:
public class ServerConfig {
// 配置中心,默认为本地文件,可以拓展为zk,mysql或其他的存储中心
private static ConfigManager configManager = ConfigManagerLoader.getConfigManager();
// 默认非http服务器的监听端口
public static final int DEFAULT_PORT = getDefaultPort();
// 默认http服务器的监听端口
public static final int DEFAULT_HTTP_PORT = 4080;
// 实际非http服务器的监听端口
private int port = configManager.getIntValue("pigeon.server.defaultport", DEFAULT_PORT);
// 实际http服务器的监听端口
private int httpPort = configManager.getIntValue("pigeon.httpserver.defaultport", DEFAULT_HTTP_PORT);
// 是否允许在端口冲突时,自动选择新端口
private boolean autoSelectPort = true;
private boolean enableTest = configManager
.getBooleanValue(Constants.KEY_TEST_ENABLE, Constants.DEFAULT_TEST_ENABLE);
// 处理服务请求的线程池核心数
private int corePoolSize = Constants.PROVIDER_POOL_CORE_SIZE;
// 处理服务请求的线程池最大数
private int maxPoolSize = Constants.PROVIDER_POOL_MAX_SIZE;
// 处理服务请求的线程池最大队列容量
private int workQueueSize = Constants.PROVIDER_POOL_QUEUE_SIZE;
//
private String suffix = configManager.getGroup();
// 服务器访问协议,default=tcp或http
private String protocol = Constants.PROTOCOL_DEFAULT;
// 当前环境,区分不同环境调用,如dev,prod等。
private String env;
// 当前服务ip
private String ip;
// 最终监听端口,在port的基础上,可能因为实际监听冲突改变
private int actualPort = port;
}
在示例中,通过一行代码:ServiceFactory.addService(providerConfig)
来完成服务注册,在具体调用方法前,会触发ServiceFactory的静态初始化,具体初始化工作包括三部分:
ProviderBootStrap是服务提供者的核心启动类,内部包括以下初始化流程:
在初始化前,通过双重检查判断是否已经初始化,具体实现源码如下所示:
public static void init() {
if (!isInitialized) {
synchronized (ProviderBootStrap.class) {
if (!isInitialized) {
// 初始化所有相关拦截器
ProviderProcessHandlerFactory.init();
// 初始化所有序列化方式
SerializerFactory.init();
// 加载所有pigeon相关类
ClassUtils.loadClasses("com.dianping.pigeon");
// 注册关闭钩子,进程清理工作
Thread shutdownHook = new Thread(new ShutdownHookListener());
shutdownHook.setDaemon(true);
shutdownHook.setPriority(Thread.MAX_PRIORITY);
Runtime.getRuntime().addShutdownHook(shutdownHook);
ServerConfig config = new ServerConfig();
// 设置配置协议为http
config.setProtocol(Constants.PROTOCOL_HTTP);
//确保注册管理器已被初始化
RegistryManager.getInstance();
// 加载服务器,包括NettyServer和JettyHttpServer
List<Server> servers = ExtensionLoader.getExtensionList(Server.class);
for (Server server : servers) {
if (!server.isStarted()) {
// config的协议类型是http,因而只初始化HttpServer
if (server.support(config)) {
server.start(config);
registerConsoleServer(config);
initRegistryConfig(config);
httpServer = server;
serversMap.put(server.getProtocol() + server.getPort(), server);
logger.warn("pigeon " + server + "[version:" + VersionUtils.VERSION + "] has been started");
}
}
}
isInitialized = true;
}
}
}
}
在pigeon中,发送和接受处理请求实际上都通过一系列的拦截器链实现,结合拦截器链,我们可以很方便地通过定义一个拦截器,添加到拦截器链中来拓展我们自己的逻辑,具体注册的拦截器如下所示,具体每个拦截器的实现原理在服务被调用部分文档解析:
public static void init() {
/*
业务逻辑拦截器开始
*/
// 调用链追踪拦截器,记录分布式链路追踪上下文
registerBizProcessFilter(new TraceFilter());
if (Constants.MONITOR_ENABLE) {
// 调用监控拦截器,添加调用上下文信息打点
registerBizProcessFilter(new MonitorProcessFilter());
}
// 记录当前处理流程,处理自定义拦截器逻辑,在拦截器回调时调用ProviderProcessInterceptor的postInvoke方法
registerBizProcessFilter(new WriteResponseProcessFilter());
// 调用链上线文信息传递处理拦截器
registerBizProcessFilter(new ContextTransferProcessFilter());
// 异常处理拦截器
registerBizProcessFilter(new ExceptionProcessFilter());
// 鉴权处理拦截器
registerBizProcessFilter(new SecurityFilter());
// 网关处理拦截器,限制调用来源等
registerBizProcessFilter(new GatewayProcessFilter());
// 业务逻辑拦截器,先调用ProviderProcessInterceptor的preInvoke方法,再调用实际的服务提供方方法
registerBizProcessFilter(new BusinessProcessFilter());
bizInvocationHandler = createInvocationHandler(bizProcessFilters);
/*
业务逻辑拦截器结束
*/
// 注册心跳处理拦截器
registerHeartBeatProcessFilter(new WriteResponseProcessFilter());
registerHeartBeatProcessFilter(new HeartbeatProcessFilter());
heartBeatInvocationHandler = createInvocationHandler(heartBeatProcessFilters);
// 注册心跳检查拦截器
registerHealthCheckProcessFilter(new WriteResponseProcessFilter());
registerHealthCheckProcessFilter(new HealthCheckProcessFilter());
healthCheckInvocationHandler = createInvocationHandler(healthCheckProcessFilters);
// 注册心跳扫描拦截器
registerScannerHeartBeatProcessFilter(new WriteResponseProcessFilter());
registerScannerHeartBeatProcessFilter(new ScannerHeartBeatProcessFilter());
scannerHeartBeatInvocationHandler = createInvocationHandler(scannerHeartBeatProcessFilters);
}
在SerializerFactory#init,注册相关序列化类型和其对应的序列化实例,具体实现如下:
public static void init() {
if (!isInitialized) {
synchronized (SerializerFactory.class) {
if (!isInitialized) {
// Java原声序列化
registerSerializer(SerializerType.JAVA, new JavaSerializer());
// HESSIAN序列化
registerSerializer(SerializerType.HESSIAN, new HessianSerializer());
registerSerializer(SerializerType.HESSIAN1, new Hessian1Serializer());
// Protostuff序列化
registerSerializer(SerializerType.PROTO, new ProtostuffSerializer());
// Thrift序列化
registerSerializer(SerializerType.THRIFT, new ThriftSerializer());
// Protobuf3序列化
registerSerializer(SerializerType.PROTOBUF3, new Protobuf3Serializer());
try {
// FST序列化
registerSerializer(SerializerType.FST, new FstSerializer());
} catch (Throwable t) {
logger.warn("failed to initialize fst serializer:" + t.getMessage());
}
// 判断是否支持json序列化
boolean supportJackson = true;
try {
ClassUtils.getClass("com.fasterxml.jackson.databind.ObjectMapper");
} catch (ClassNotFoundException e) {
supportJackson = false;
}
if (supportJackson) {
try {
// 注册jackson序列化
registerSerializer(SerializerType.JSON, new JacksonSerializer());
} catch (Throwable t) {
logger.warn("failed to initialize jackson serializer:" + t.getMessage());
}
}
// 加载所有的序列化注册中心,并进行相关注册操作
List<SerializerRegister> serializerRegisters = ExtensionLoader.getExtensionList(SerializerRegister.class);
for (SerializerRegister serializerRegister : serializerRegisters) {
if (!serializerRegister.isRegistered()) {
serializerRegister.registerSerializer();
}
}
isInitialized = true;
}
}
}
}
大致创建流程:
在ProviderBootStrap#init中,有一行代码RegistryManager.getInstance()
会触发注册管理器的初始化。
注册管理器主要用于管理所有注册的服务,pigeon默认使用的注册中心是zookeeper,在pigeon中的实现为CuratorRegistry,主要保存服务地址、端口、权重、心跳等信息的注册管理。
具体实现代码如下:
public static RegistryManager getInstance() {
if (!isInit) {
synchronized (RegistryManager.class) {
if (!isInit) {
// 具体的init方法在下面
instance.init();
initializeException = null;
// 注册服务器信息变更监听,更新当前ConfigManager对象,基于特定服务地址的hostInfo
RegistryEventListener.addListener(new InnerServerInfoListener());
isInit = true;
}
}
}
return instance;
}
private void init() {
// 加载所有的注册中心
List<Registry> _registryList = ExtensionLoader.getExtensionList(Registry.class);
try {
if (_registryList.size() > 0) {
//根据配置使用相应的注册中心,默认为curator
String customizedRegistryName = configManager.getStringValue(KEY_PIGEON_REGISTRY_CUSTOMIZED,
Constants.REGISTRY_CURATOR_NAME);
for (Registry registry : _registryList) {
if (registry.getName().equals(customizedRegistryName)) {
// 初始化注册中心,如建立zk连接等。
registry.init();
RegistryManager.registry = registry;
logger.info(registry.getName() + " registry started.");
}
}
} else {
throw new RegistryException("failed to find registry extension type, please check dependencies!");
}
// 监听KEY_PIGEON_REGISTRY_CUSTOMIZED配置,如果改变,初始化对应新的注册中心
configManager.registerConfigChangeListener(new InnerConfigChangeListener());
} catch (Throwable t) {
initializeException = t;
throw new RuntimeException(t);
}
}
调用ServiceFactory#addService,会进一步调用DefaultPublishPolicy#doAddService,最后调用的是抽象父类AbstractPublishPolicy#doAddService方法。
AbstractPublishPolicy#doAddService方法具体源码如下:
public void doAddService(ProviderConfig providerConfig) {
try {
// 检查服务名
checkServiceName(providerConfig);
// 发布指定版本服务,同时解析服务方法
ServicePublisher.addService(providerConfig);
// 启动netty RPC服务器,作为服务提供方供调用方调用服务
ServerConfig serverConfig = ProviderBootStrap.startup(providerConfig);
// 更新serverConfig
providerConfig.setServerConfig(serverConfig);
// 实际发布服务,会将服务注册到注册中心,供调用方发现调用
ServicePublisher.publishService(providerConfig, false);
} catch (Throwable t) {
throw new RpcException("error while adding service:" + providerConfig, t);
}
}
这一步如果自定义了访问url,和默认服务名(接口全限定名)不一致,会检查服务是否为存量服务,如果不是,会判断配置是否允许自定义服务名,如果允许,则使用自定义服务名,否则会抛出异常或强制转换为类路径服务名,具体源码实现如下:
private void checkServiceName(ProviderConfig providerConfig) {
// 默认服务名(接口全限定名)
String serviceUrl = ServiceFactory.getServiceUrl(providerConfig);
// 自定义url
String customUrl = providerConfig.getUrl();
if (StringUtils.isBlank(customUrl)) {
// 没有自定义则使用默认
providerConfig.setUrl(serviceUrl);
} else if (!serviceUrl.equals(customUrl) && !isStockService(customUrl)) {
// 自定义url且和服务名不一直,并且不是存量服务
if (IS_ALLOW_CUSTOMIZED_SERVICENAME) {
// 允许自定义
return;
}
// 非存量服务,不允许注册,抛出异常或强制转换为类路径服务名
if (IS_CHECK_SERVICE_EXCEPTION_DEFAULT) {
logger.error("customized [serviceName]: " + customUrl
+ " cannot provide service to OCTO invoker "
+ "unless set the [serviceName] to full class name "
+ "or just keep [serviceName] config to blank.\n"
+ "[serviceName] should be replaced by full class name: "
+ serviceUrl + ", more help refer to: "
+ configManager.getStringValue("pigeon.help.provider.octo.url"
, "http://wiki.sankuai.com/pages/viewpage.action?pageId=606809899"));
System.exit(1);
} else {
logger.warn("customized [serviceName]: " + customUrl
+ " cannot provide service to OCTO invoker "
+ "unless set the [serviceName] to full class name "
+ "or just keep [serviceName] config to blank.\n"
+ "[serviceName] will be replaced by full class name: "
+ serviceUrl + ", more help refer to: "
+ configManager.getStringValue("pigeon.help.provider.octo.url"
, "http://wiki.sankuai.com/pages/viewpage.action?pageId=606809899"));
providerConfig.setUrl(serviceUrl);
}
}
}
在添加解析服务中,主要进行了以下流程:
具体实现源码如下所示:
public static <T> void addService(ProviderConfig<T> providerConfig) throws Exception {
if (logger.isInfoEnabled()) {
logger.info("add service:" + providerConfig);
}
String version = providerConfig.getVersion();
String url = providerConfig.getUrl();
// 默认版本,直接以url为key
if (StringUtils.isBlank(version)) {
serviceCache.put(url, providerConfig);
} else {
// urlWithVersion = url + "_" + version
String urlWithVersion = getServiceUrlWithVersion(url, version);
if (serviceCache.containsKey(url)) {
// 如果已经存在,覆盖服务
serviceCache.put(urlWithVersion, providerConfig);
ProviderConfig<?> providerConfigDefault = serviceCache.get(url);
String defaultVersion = providerConfigDefault.getVersion();
// 如果默认服务存在默认版本,并且小于当前版本,用当前版本服务更新默认服务版本
if (!StringUtils.isBlank(defaultVersion)) {
if (VersionUtils.compareVersion(defaultVersion, providerConfig.getVersion()) < 0) {
serviceCache.put(url, providerConfig);
}
}
} else {
// 将当前版本设为指定版本服务和默认版本服务
serviceCache.put(urlWithVersion, providerConfig);
// use this service as the default provider
serviceCache.put(url, providerConfig);
}
}
// 如果服务实现了InitializingService接口,调用实现的initialize方法
T service = providerConfig.getService();
if (service instanceof InitializingService) {
((InitializingService) service).initialize();
}
// 解析接口自定义方法,根据方法名,参数等相关信息记录方法
ServiceMethodFactory.init(url);
}
// ServiceMethodFactory.init(url);方法实现如下:
public static void init(String url) {
getServiceMethodCache(url);
}
// 具体调用了
public static ServiceMethodCache getServiceMethodCache(String url) {
// 是否存在指定url的ServiceMethodCache
ServiceMethodCache serviceMethodCache = methods.get(url);
if (serviceMethodCache == null) {
// 获取指定url的providerConfig
Map<String, ProviderConfig<?>> services = ServicePublisher.getAllServiceProviders();
ProviderConfig<?> providerConfig = services.get(url);
if (providerConfig != null) {
Object service = providerConfig.getService();
Method[] methodArray = service.getClass().getMethods();
serviceMethodCache = new ServiceMethodCache(url, service);
// 遍历指定url的所有服务方法
for (Method method : methodArray) {
// 忽略掉Object和Class的所有方法
if (!ingoreMethods.contains(method.getName())) {
method.setAccessible(true);
serviceMethodCache.addMethod(method.getName(), new ServiceMethod(service, method));
if (isCompact) {
// 压缩url,方法名等调用所需信息
int id = LangUtils.hash(url + "#" + method.getName(), 0, Integer.MAX_VALUE);
ServiceId serviceId = new ServiceId(url, method.getName());
ServiceId lastId = CompactRequest.PROVIDER_ID_MAP.putIfAbsent(id, serviceId);
// 检查如果存在相同id服务方法,抛异常
if (lastId != null && !serviceId.equals(lastId)) {
throw new IllegalArgumentException("same id for service:" + url + ", method:"
+ method.getName());
}
}
}
}
// 更新缓存
methods.put(url, serviceMethodCache);
}
}
return serviceMethodCache;
}
具体代码实现流程大致:
具体源码实现如下:
public static ServerConfig startup(ProviderConfig<?> providerConfig) {
ServerConfig serverConfig = providerConfig.getServerConfig();
if (serverConfig == null) {
throw new IllegalArgumentException("server config is required");
}
// 根据protocol+port判断当前服务器列表中是否存在相关的服务器实例
Server server = serversMap.get(serverConfig.getProtocol() + serverConfig.getPort());
if (server != null) {
// 存在直接返回
server.addService(providerConfig);
return server.getServerConfig();
} else {
// 不存在先创建一个
synchronized (ProviderBootStrap.class) {
List<Server> servers = ExtensionLoader.newExtensionList(Server.class);
for (Server s : servers) {
if (!s.isStarted()) {
if (s.support(serverConfig)) {
s.start(serverConfig);
// 添加服务
s.addService(providerConfig);
serversMap.put(s.getProtocol() + serverConfig.getPort(), s);
logger.warn("pigeon " + s + "[version:" + VersionUtils.VERSION + "] has been started");
break;
}
}
}
server = serversMap.get(serverConfig.getProtocol() + serverConfig.getPort());
// 预启动内部的请求处理器核心线程
if (server != null) {
server.getRequestProcessor().getRequestProcessThreadPool().prestartAllCoreThreads();
return server.getServerConfig();
}
return null;
}
}
}
在调用s.addService(providerConfig)
方法一步,会调用RequestTheadPoolProcessor#addService方法。
RequestTheadPoolProcessor是pigeon请求线程池封装,并提供了线程池的刷新功能。在接收到调用方方法请求时,会讲请求交由RequestTheadPoolProcessor处理,RequestTheadPoolProcessor会根据配置看是否需要创建独立的线程池调用实际的服务方法。
具体源码:
public <T> void addService(ProviderConfig<T> providerConfig) {
requestProcessor.addService(providerConfig);
doAddService(providerConfig);
List<ServiceChangeListener> listeners = ServiceChangeListenerContainer.getListeners();
for (ServiceChangeListener listener : listeners) {
listener.notifyServiceAdded(providerConfig);
}
}
RequestTheadPoolProcessor#addService方法实现如下:
ublic synchronized <T> void addService(ProviderConfig<T> providerConfig) {
String url = providerConfig.getUrl();
Map<String, ProviderMethodConfig> methodConfigs = providerConfig.getMethods();
ServiceMethodCache methodCache = ServiceMethodFactory.getServiceMethodCache(url);
Set<String> methodNames = methodCache.getMethodMap().keySet();
// 根据useSharedPool参数判断是否用不同的线程池隔离不同服务调用
if (needStandalonePool(providerConfig)) {
// 服务的poolConfig方式,支持方法的fallback
if (providerConfig.getPoolConfig() != null) {
// 根据配置刷新线程池
DynamicThreadPoolFactory.refreshThreadPool(providerConfig.getPoolConfig());
} else if (providerConfig.getActives() > 0 && CollectionUtils.isEmpty(methodConfigs)) {
// 服务的actives方式,不支持方法的fallback,不支持动态修改
DynamicThreadPool pool = serviceThreadPools.get(url);
if (pool == null) {
int actives = providerConfig.getActives();
int coreSize = (int) (actives / DEFAULT_POOL_RATIO_CORE) > 0 ? (int) (actives / DEFAULT_POOL_RATIO_CORE)
: actives;
int maxSize = actives;
int queueSize = actives;
pool = new DynamicThreadPool("Pigeon-Server-Request-Processor-service", coreSize, maxSize,
queueSize);
serviceThreadPools.putIfAbsent(url, pool);
}
}
// 方法级设置方式,根据url#method,为特定方法
if (!CollectionUtils.isEmpty(methodConfigs)) {
for (String name : methodNames) {
String key = url + "#" + name;
ProviderMethodConfig methodConfig = methodConfigs.get(name);
DynamicThreadPool pool = methodThreadPools.get(key);
if (methodConfig != null) {
if (methodConfig.getPoolConfig() != null) {
// 方法poolConfig方式
DynamicThreadPoolFactory.refreshThreadPool(methodConfig.getPoolConfig());
} else if (pool == null) {
// 方法actives方式
int actives = DEFAULT_POOL_ACTIVES;
if (methodConfig.getActives() > 0) {
actives = methodConfig.getActives();
}
int coreSize = (int) (actives / DEFAULT_POOL_RATIO_CORE) > 0 ? (int) (actives / DEFAULT_POOL_RATIO_CORE)
: actives;
int maxSize = actives;
int queueSize = actives;
pool = new DynamicThreadPool("Pigeon-Server-Request-Processor-method", coreSize, maxSize,
queueSize);
methodThreadPools.putIfAbsent(key, pool);
}
}
}
}
}
}
具体流程如下:
源码实现如下所示:
public static <T> void publishService(ProviderConfig<T> providerConfig, boolean forcePublish)
throws RegistryException {
// 判断是否是存在于serviceCache的服务
String url = providerConfig.getUrl();
boolean existingService = false;
for (String key : serviceCache.keySet()) {
ProviderConfig<?> pc = serviceCache.get(key);
if (pc.getUrl().equals(url)) {
existingService = true;
break;
}
}
if (logger.isInfoEnabled()) {
logger.info("try to publish service to registry:" + providerConfig + ", existing service:"
+ existingService);
}
if (existingService) {
boolean autoPublishEnable = ConfigManagerLoader.getConfigManager().getBooleanValue(
Constants.KEY_AUTOPUBLISH_ENABLE, true);
// 是否自动发布和强制发布
if (autoPublishEnable || forcePublish) {
// 获取所有服务器列表
List<Server> servers = ProviderBootStrap.getServers(providerConfig);
int registerCount = 0;
for (Server server : servers) {
// 发布服务到注册中心
publishServiceToRegistry(url, server.getRegistryUrl(url), server.getPort(),
RegistryManager.getInstance().getGroup(url), providerConfig.isSupported());
registerCount++;
}
if (registerCount > 0) {
// 注册心跳监听器,定时发布时间戳到注册中心
boolean isHeartbeatEnable = configManager.getBooleanValue(Constants.KEY_HEARTBEAT_ENABLE,
DEFAULT_HEARTBEAT_ENABLE);
if (isHeartbeatEnable) {
HeartBeatListener.registerHeartBeat(providerConfig);
}
// 通知服务发生变化
boolean isNotify = configManager
.getBooleanValue(Constants.KEY_NOTIFY_ENABLE, false);
if (isNotify && serviceChangeListener != null) {
serviceChangeListener.notifyServicePublished(providerConfig);
}
// 更新服务权重以上线
boolean autoRegisterEnable = ConfigManagerLoader.getConfigManager().getBooleanValue(
Constants.KEY_AUTOREGISTER_ENABLE, true);
if (autoRegisterEnable) {
ServiceOnlineTask.start();
} else {
logger.info("auto register is disabled");
}
providerConfig.setPublished(true);
}
} else {
logger.info("auto publish is disabled");
}
}
}
基于xml配置流程更复杂一些,往往通过配置一个ServiceBean实现,具体流程如下:
Spring初始化所有非懒加载单例Bean,触发创建ServiceBean:
以zookeeper为注册中心,在注册后,会在zk注册以下节点:
Pigeon服务端每次启动后会将自身ip:port注册到ZooKeeper集群中,具体会注册到组值,每一组属性值含义: