RabbitMQ 客户端 利用反射技术实现自定义springboot stater，实现多系统之间数据同步
1. 导读
本文主要讲述RabbitMQ客户端的封装和使用，思路不一定准确，但目前生产中是这样使用的，有不对的地方，欢迎批评指正。
本文需要对springboot、spring的spel表达式、rabbitmq工作流程、死信队列机制有一定的了解，熟悉springboot starter的大概思路。
客户端思路
定义客户端
定义所有实体类的父类InterfaceFactoryBean
实现ImportBeanDefinitionRegistrar,扫描自定义注解标注的路径
公用service接口，第三方持久层需实现
2. 客户端
最近公司微服务架构中有需要同步数据库表数据的需求，考虑到封装性和扩展性，减少第三方项目的开发工作量，思路如下(本文不涉及缓存)：
2.1. 服务端创建Fanout模式的交换机，提供数据发送接口
2.2. 封装客户端，客户端作为jar包存在，供第三方项目(其他项目，以下都以第三方代称)依赖使用
2.3. 第三方项目依赖的客户端启动创建队列绑定上面的交换机，并创建死信交换机、队列
2.4. 第三方项目依赖的客户端接到数据后，通过反射执行第三方实现的客户端中定义的持久层接口，实现数据入库。
2.5定义死信队列接收消费失败的消息，死信队列通过RestTemplate入库，客户端实现
2.6. 失败的消息会重试3次，重试后再次失败，消息进入死信队列。每次新接收的消息会检查库中是否存在同类型的死信，如果有，直接入库，然后尝试消费当前第一条，成功就继续，失败则终止。（同步的是人员、机构等数据，对顺序性要求高）
3. 开始撸代码 – 客户端
3.1 创建一个spring boot 项目引入依赖如下：
org.springframework.boot
spring-boot-starter-web
org.springframework.boot
spring-boot-starter-logging
org.springframework.boot
spring-boot-starter-amqp
org.projectlombok
lombok
com.chinacoal.microservice
common-utils
1.0-SNAPSHOT
org.springframework.boot
spring-boot-configuration-processor
true
com.alibaba
fastjson
1.1.23
org.springframework.boot
spring-boot-starter-jdbc
3.2 定义客户端：
@Autowired
private QueueConfig queueConfig;
@RabbitListener(exclusive = false,bindings=@QueueBinding(
value=@Queue(autoDelete = "false",durable = "true",value = "#{queueConfig.getName()}",arguments = {
@Argument(name = "x-dead-letter-exchange", value = "#{queueConfig.getDLX_EXCHANGE_NAME()}"),
@Argument(name = "x-dead-letter-routing-key", value ="#{queueConfig.getROUTE_KEY()}"),
@Argument(name = "x_message_ttl", value = "30000")
}),exchange=@Exchange(value = "orgFanoutExchange", durable = "true",type = ExchangeTypes.FANOUT,autoDelete = "false")))
public void receiveMessage(String receiveMessage, Message message, Channel channel) throws Exception {
// 手动签收
log.info("接收到消息:[{}]", receiveMessage);
//执行业务逻辑
try {
if(doWork(receiveMessage,message)) {
//throw new FileUploadException();
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}else {
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false,false);
}
} catch (Exception e) {
log.error("消息签收失败:", e);
Integer count = Integer.valueOf(String.valueOf(message.getMessageProperties().getHeaders().get("count")));
if(count >= 3) {
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false,false);
}else {
message.getMessageProperties().getHeaders().put("count", count + 1);
throw e;
}
}
}
主要属性解释：
@RabbitListener //声明客户端
@Queue //声明一个队列
exclusive //表示该消息队列是否只在当前connection生效
durable //是否开启持久化
value = "#{queueConfig.getName()}" //队列的名称，该表达式为spel表达式，queueConfig 需要提供getName方法
@Argument(name = "x-dead-letter-exchange", value = "#{queueConfig.getDLX_EXCHANGE_NAME()}"),@Argument(name = "x-dead-letter-routing-key", value ="#{queueConfig.getROUTE_KEY()}"),
//绑定死信队列和死信队列的routeKey,@Argument中的name为固定值
死信队列声明代码
@Autowired
private QueueConfig queueConfig;
@RabbitListener(exclusive = false,bindings=@QueueBinding(
value=@Queue(autoDelete = "false",durable = "true",value = "#{queueConfig.getDLX_QUEUE_NAME()}"),
exchange=@Exchange(value = "#{queueConfig.getDLX_EXCHANGE_NAME()}", durable = "true",type = ExchangeTypes.DIRECT,autoDelete = "false")))
public void receiveMessage(String receiveMessage, Message message, Channel channel) throws Exception {
log.info("接收到死信消息:[{}]", receiveMessage);
//执行业务逻辑
try {
if(doWork(receiveMessage)) {
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}else {
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false,false);
}
} catch (Exception e) {
log.error("死信消息签收失败:", e);
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false,true);
throw e;
}
}
3.3 自定义注解，实现实体类和持久层扫描，和所有实体类的父类型
扫描实体类的注解
@Documented
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Import(BeanDefinitionRegistrar.class)
public @interface Model {
/**
* @return
*/
String[] value() default {};
/**
* 扫描包
*
* @return
*/
String[] basePackages() default {};
/**
* 扫描的基类
*
* @return
*/
Class[] basePackageClasses() default {};
/**
* 包含过滤器
*
* @return
*/
Filter[] includeFilters() default {};
/**
* 排斥过滤器
*
* @return
*/
Filter[] excludeFilters() default {};
扫描持久层的注解
@Documented
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Import(BeanDefinitionRegistrar.class)
public @interface ServiceScan {
/**
* @return
*/
String[] value() default {};
/**
* 扫描包
*
* @return
*/
String[] basePackages() default {};
/**
* 扫描的基类
*
* @return
*/
Class[] basePackageClasses() default {};
/**
* 包含过滤器
*
* @return
*/
Filter[] includeFilters() default {};
/**
* 排斥过滤器
*
* @return
*/
Filter[] excludeFilters() default {};
}
}
定义InterfaceFactoryBean，以此为所有实体的父类，为后续json转实体类型提供便利，解决代码耦合
@Data
public class InterfaceFactoryBean implements FactoryBean{
private Class interfaceClass;
@Override
public T getObject() throws Exception {
// 检查 h 不为空，否则抛异常
Objects.requireNonNull(interfaceClass);
return (T) Enhancer.create(interfaceClass,new DymicInvocationHandler());
}
@Override
public Class getObjectType() {
return interfaceClass;
}
@Override
public boolean isSingleton() {
return true;
}
}
3.4 实现ImportBeanDefinitionRegistrar 实现扫描注解标注的包路径下的类，加入静态集合
@Slf4j
public class BeanDefinitionRegistrar implements ImportBeanDefinitionRegistrar {
private static final String RESOURCE_PATTERN = "**/*.class";
public static final Map> MODEL_MAPPING = new ConcurrentHashMap>();
public static final Map> SERVICE_MAPPING = new ConcurrentHashMap>();
/**
* @param importingClassMetadata
* @param registry
*/
@Override
public void registerBeanDefinitions(AnnotationMetadata importingClassMetadata, BeanDefinitionRegistry registry) {
register(importingClassMetadata,registry,MODEL_MAPPING,ModelScan.class.getName(),0);
register(importingClassMetadata,registry,SERVICE_MAPPING,ServiceScan.class.getName(),1);
}
private void register(AnnotationMetadata importingClassMetadata,BeanDefinitionRegistry registry,Map> mapping,String annotationName,Integer flag) {
AnnotationAttributes annAttr = AnnotationAttributes.fromMap(importingClassMetadata.getAnnotationAttributes(annotationName));
String[] basePackages = annAttr.getStringArray("value");
if (ObjectUtils.isEmpty(basePackages)) {
basePackages = annAttr.getStringArray("basePackages");
}
if (ObjectUtils.isEmpty(basePackages)) {
basePackages = getPackagesFromClasses(annAttr.getClassArray("basePackageClasses"));
}
if (ObjectUtils.isEmpty(basePackages)) {
basePackages = new String[] {ClassUtils.getPackageName(importingClassMetadata.getClassName())};
}
List includeFilters = extractTypeFilters(annAttr.getAnnotationArray("includeFilters"));
//增加一个包含的过滤器,扫描到的类只要不是抽象的,接口,枚举,注解,及匿名类那么就算是符合的
includeFilters.add(new CustomTypeFilter());
List excludeFilters = extractTypeFilters(annAttr.getAnnotationArray("excludeFilters"));
excludeFilters.add(new CustomTypeFilter());
List> candidates = scanPackages(basePackages, includeFilters, excludeFilters);
if (candidates.isEmpty()) {
log.info("扫描指定基础包[{}]时未发现复合条件的基础类", basePackages.toString());
return;
}
registerBeanDefinitions(candidates, registry,mapping,flag);
}
/**
* @param basePackages
* @param includeFilters
* @param excludeFilters
* @return
*/
private List> scanPackages(String[] basePackages, List includeFilters, List excludeFilters) {
List> candidates = new ArrayList>();
for (String pkg : basePackages) {
try {
candidates.addAll(findCandidateClasses(pkg, includeFilters, excludeFilters));
} catch (IOException e) {
log.error("扫描指定基础包[{}]时出现异常", pkg);
continue;
}
}
return candidates;
}
/**
* @param basePackage
* @return
* @throws IOException
*/
private List> findCandidateClasses(String basePackage, List includeFilters, List excludeFilters) throws IOException {
if (log.isDebugEnabled()) {
log.debug("开始扫描指定包{}下的所有类" + basePackage);
}
List> candidates = new ArrayList>();
String packageSearchPath = ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX + replaceDotByDelimiter(basePackage) + '/' + RESOURCE_PATTERN;
ResourceLoader resourceLoader = new DefaultResourceLoader();
MetadataReaderFactory readerFactory = new SimpleMetadataReaderFactory(resourceLoader);
Resource[] resources = ResourcePatternUtils.getResourcePatternResolver(resourceLoader).getResources(packageSearchPath);
for (Resource resource : resources) {
MetadataReader reader = readerFactory.getMetadataReader(resource);
if (isCandidateResource(reader, readerFactory, includeFilters, excludeFilters)) {
Class candidateClass = transform(reader.getClassMetadata().getClassName());
if (candidateClass != null) {
candidates.add(candidateClass);
log.debug("扫描到符合要求基础类:{}" + candidateClass.getName());
}
}
}
return candidates;
}
/**
* 注册 Bean,
* Bean的名称格式:
* @param internalClasses
* @param registry
*/
private void registerBeanDefinitions(List> internalClasses, BeanDefinitionRegistry registry,Map> mapping,Integer flag) {
for (Class clazz : internalClasses) {
if (mapping.values().contains(clazz)) {
log.debug("重复扫描{}类,忽略重复注册", clazz.getName());
continue;
}
if(flag == 0 ) {
BeanDefinitionBuilder builder = BeanDefinitionBuilder.genericBeanDefinition(clazz);
GenericBeanDefinition definition = (GenericBeanDefinition) builder.getRawBeanDefinition();
definition.getPropertyValues().add("interfaceClass", clazz);
definition.setBeanClass(InterfaceFactoryBean.class);
definition.setAutowireMode(GenericBeanDefinition.AUTOWIRE_BY_TYPE);
registry.registerBeanDefinition(ClassUtils.getShortNameAsProperty(clazz), definition);
}
mapping.put(ClassUtils.getShortNameAsProperty(clazz), clazz);
}
}
代码解释：
基于spring的bean注册器，扫描自定义注解声明的包路径下的所有类，实体类加入MODEL_MAPPING集合，serviceImp加入SERVICE_MAPPING集合，然后通过BeanDefinitionBuilder将所有上面定义的InterfaceFactoryBean的子类注入到spring容器
3.5 执行业务逻辑
@Autowired
private Map maps;//获取所有InterfaceFactoryBean的实现
private boolean doWork(String receiveMessage, Message m) throws Exception {
Map map = JSON.parseObject(receiveMessage);
String className = (String) map.get("className");
Integer operator = Integer.valueOf(String.valueOf(map.get("operator")));
//校验数据库是否存在死信数据，如果有继续入库，如果没有往下执行业务逻辑
String checkSql = "select count(1) from data_exchange_record where className = ? group by className";
MessageWrapper parseObject = JSON.parseObject(receiveMessage, MessageWrapper.class);
if(jDBCUtils.statistics(checkSql,parseObject.getClassName()) > 0) {
//解决重试重复入库
if("false".equals(m.getMessageProperties().getHeaders().get("status"))) {
jDBCUtils.execInsert(JDBCUtils.INSERT_SQL, parseObject);
m.getMessageProperties().getHeaders().put("status","true");
log.info("数据库存在该类型，直接入库:[{}]", receiveMessage);
}
String sql = "select * from data_exchange_record where className= ? order by recordTime asc";
List list = jDBCUtils.getList(sql,parseObject.getClassName());
if(!CollectionUtils.isEmpty(list)) {
//尝试更新
//成功 更新所有的数据库中的数据，直到失败（不重试）
for (int i = 0; i < list.size(); i++) {
MessageWrapper message = list.get(i);
Integer operatorTmp = message.getOperator();
Boolean handleMessage = handleMessage(operatorTmp,JSON.toJSONString(message),message.getClassName());
boolean flag = false;
if(handleMessage) {
flag = jDBCUtils.execDelete(JDBCUtils.DELETE_SQL,message.getId());
if(!flag) {
break;
}
}else {
break;
}
}
}
return true;
}
return handleMessage(operator,receiveMessage,className);
}
private Boolean handleMessage(Integer operator,String receiveMessage,String className) throws Exception {
if(operator == 4) { // sql
String sql = (String) JSON.parseObject(receiveMessage).get("json");
return jDBCUtils.execSql(sql);
}
Class forName = Class.forName(className);
if(!BeanDefinitionRegistrar.MODEL_MAPPING.containsValue(forName)){
return false;
}
String shorName = ClassUtils.getShortNameAsProperty(forName);
if(BeanDefinitionRegistrar.SERVICE_MAPPING.keySet().size() == 0) {
return false;
}
String tempKeyName = "";
for (String key : BeanDefinitionRegistrar.SERVICE_MAPPING.keySet()) {
if(key.contains(shorName)) {
tempKeyName = key;
break;
}
}
if(StringUtils.isBlank(tempKeyName)) {
return false;
}
Class clazz = BeanDefinitionRegistrar.SERVICE_MAPPING.get(tempKeyName);
String jsonString = (String) JSON.parseObject(receiveMessage).get("json");
InterfaceFactoryBean factoryBean = maps.get(CommonUtil.toLowCaseFirstOne(shorName));
Method method = null;
//由于操作的是同一个对象，除删除意外的所有方法的实体类属性都应该有默认值
if(operator == 1) { //insert
method = getMethod(clazz,"insert");
method = invokeMethod( clazz, method, jsonString,shorName,factoryBean);
return (Boolean) method.invoke(clazz.newInstance(),factoryBean);
}else if(operator == 2) { // update
method = getMethod(clazz,"update");
method = invokeMethod( clazz, method, jsonString,shorName,factoryBean);
return (Boolean) method.invoke(clazz.newInstance(),factoryBean);
}else if(operator == 3) { // delete
method = getMethod(clazz,"delete");
method = invokeMethod( clazz, method, jsonString,shorName,factoryBean);
return (Boolean) method.invoke(clazz.newInstance(),factoryBean);
}
return false;
}
private Method invokeMethod(Class clazz, Method method,String jsonString, String shorName, InterfaceFactoryBean factoryBean2) throws Exception {
JSONObject parse = JSON.parseObject(jsonString);
InterfaceFactoryBean factoryBean = maps.get(CommonUtil.toLowCaseFirstOne(shorName));
if(!Objects.isNull(parse)) {
Method[] methods = factoryBean.getClass().getMethods();
for (int j = 0; j < methods.length; j++) {
Method method2 = methods[j];
String name = method2.getName();
String lowCaseFirstOne = CommonUtil.toLowCaseFirstOne(name.substring(3));
Object param =parse.get(lowCaseFirstOne);
if(name.startsWith("set") && !Objects.isNull(param)) {
String object = String.valueOf(param);
Type[] parameters = method2.getGenericParameterTypes();
Class forName2 = Class.forName(parameters[0].getTypeName());
Method declaredMethod = factoryBean.getClass().getDeclaredMethod(method2.getName(),forName2);
if(forName2 == BigDecimal.class) {
declaredMethod.invoke(factoryBean,new BigDecimal(object));
}else if(forName2 == Date.class){
Date date = new Date();
date.setTime(Long.valueOf(object));
declaredMethod.invoke(factoryBean,date);
}else {
declaredMethod.invoke(factoryBean,object);
}
}
}
}
return method;
}
代码解释：
通过上面定义的SERVICE_MAPPING集合对比当前消息的className，然后从容器中取出，通过消息中的operator判断是什么操作，然后反射相应的方法。
再从maps中获取具体的实体类型class对象，反射调用set方法赋值，执行。。。
3.6 jdbc的代码和共用接口(第三方持久层需要实现)
jdbc
public static final String INSERT_SQL = "insert into data_exchange_record (operator,className,json) values (?,?,?)";
public static final String DELETE_SQL = "delete from data_exchange_record where id = ?";
@Autowired
private JdbcTemplate jdbcTemplate;
/**
* 插入
* @param sql
* @param message
* @return
*/
public Boolean execInsert(String sql,MessageWrapper message) {
try {
return jdbcTemplate.update(sql, message.getOperator(),message.getClassName(),message.getJson()) > 0;
}catch(Exception e ) {
log.error("插入数据失败:",e);
return false;
}
}
/**
* 删除
* @param sql
* @param t
* @return
*/
public Boolean execDelete(String sql,T t) {
return jdbcTemplate.update(sql, t) > 0;
}
/**
* 执行sql
* @param sql
* @return
*/
public Boolean execSql(String sql) {
jdbcTemplate.execute(sql);
return false;
}
公用service接口，第三方持久层需实现
/**
* 数据同步数据层接口
* @author fly
*
* @param
*/
public interface CommonDefinitionService {
/**
* 插入
* @param t
* @return
*/
Boolean insert(T t);
/**
* 更新
* @param t
* @return
*/
Boolean update(T t);
/**
* 删除
* @param t
* @return
*/
Boolean delete(T t);
}
3.7 starter实现
------------定义自动配置类
@Log4j2
@Configuration
@ConditionalOnClass({RabbitMQProperties.class})
@EnableConfigurationProperties(RabbitMQProperties.class)
@ConditionalOnProperty(prefix = "data.exchange", value = "enabled", havingValue = "true")
@PropertySource(value = {"classpath:rabbit.properties"})
@ComponentScan("com.data.exchange")
public class DataExchangeAutoConfiguration{
@PostConstruct
public void init() {
log.info("*********启动[{}]自动配置*********",DataExchangeAutoConfiguration.class);
}
}
然后定义resources/META/spring.factories
org.springframework.boot.autoconfigure.EnableAutoConfiguration=com.data.exchange.config.DataExchangeAutoConfiguration
客户端到这里就差不多完毕了。写的匆忙，后续继续补充，代码后续上传到github。