在今天的互联网+时代,数据存储与查询是每个互联网项目重中之重的问题,如何能够保障系统的响应速率,使之系统不随着数据量的增多而响应缓慢,在很大程度上需要依赖数据库的存储设计。为了解决这个问题,许多人提出了分库分表的思路,确实分库分表是当前能够快速解决系统不随着数据量的增多而响应缓慢的问题。但是分库分表又引入了一个新的问题,那就是数据的查询,怎么查?怎么定位数据在那个库,那张表?现在的业务层面的代码是否需要改动等。
基于上面提出的这些问题,本文虽然不能解决所有问题,但是可以帮助大家解决绝大部分问题。需要的伙伴可以在本文的基础之上进行改进,以符合你自己的业务需求。
为了满足部分小伙伴的急切的需求,本文先贴出github源码地址:基于mybatis实现的分表查询的插件(点我)
(1)独立性
本插件是基于mybatis插件来实现的,可插拔,不影响业务层,引入即可用。
(2)易用性
基于注解形式,方便易用,只需要在需要分表的mapper类上加上@ShardTable注解即可,不用修改业务层代码。
(1)利用mybatis插件接口,拦截所有查询方法;
(2)解析识别是否启用分表功能;
(3)如果启用分表,判断分表规则;
(4)根据分表规则获得当前主表的所有分表;
(5)根据分表规则和传入参数,定位数据所在分表;
(6)解析sql获取聚合函数,排序,分页,分类等参数;
(7)替换sql,分次查询获取返回结果,并根据解析sql获取的聚合函数,排序,分类,分页处理从分表获取的数据,并返回。
(1)插件目录一览表
(2)ShardTable、ShardInterceptor、ShardingHandle、ShardingRule、ResultHandle代码讲解
1)ShardTable
package com.frank.sharding.annotation;
import static java.lang.annotation.ElementType.TYPE;
import static java.lang.annotation.RetentionPolicy.RUNTIME;
import java.lang.annotation.Documented;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;
import com.frank.sharding.rule.ShardRuleType;
/**
* @description 分表注解器,目前只支持按照日期分表,分为当前表(最近一个月数据),历史表(最近2-3月数据),归档表(3个月以前的数据)
* @author fengsheng
* @since 2019年1月8日
* @date 2019年1月8日
*
*/
@Documented
@Retention(RUNTIME)
@Target(TYPE)
public @interface ShardTable {
/**
* @Description: 指定表名
* @since 2019年1月8日
* @date 2019年1月8日
* @return
*/
public String table();
/**
* @Description: 指定列
* @since 2019年1月8日
* @date 2019年1月8日
* @return
*/
public String column();
/**
* @Description: 分表规则 ,默认按照日期拆分
* @since 2019年1月8日
* @date 2019年1月8日
* @return
*/
public int rule() default ShardRuleType.DATE;
}
2)ShardInterceptor
分表拦截器类,拦截所有查询,解析并过滤未启用分表的查询。使用该分表插件时,需要将本拦截器添加到xml配置或注入到配置中。
package com.frank.sharding.interceptor;
import java.util.Properties;
import org.apache.ibatis.executor.Executor;
import org.apache.ibatis.mapping.MappedStatement;
import org.apache.ibatis.plugin.Interceptor;
import org.apache.ibatis.plugin.Intercepts;
import org.apache.ibatis.plugin.Invocation;
import org.apache.ibatis.plugin.Plugin;
import org.apache.ibatis.plugin.Signature;
import org.apache.ibatis.session.ResultHandler;
import org.apache.ibatis.session.RowBounds;
import org.springframework.core.annotation.Order;
import com.frank.sharding.annotation.ShardTable;
import com.frank.sharding.handle.ShardingHandle;
/**
* @description mybatis 提交sql前拦截器,主要针对分表数据进行处理,采用的是多表查询合并
* @author fengsheng
* @since 2019年1月23日
* @date 2019年1月23日
*/
@Intercepts({ @Signature(type = Executor.class, method = "query", args = { MappedStatement.class, Object.class,
RowBounds.class, ResultHandler.class }) })
@Order(Integer.MAX_VALUE)
public class ShardInterceptor implements Interceptor {
@Override
@SuppressWarnings({ "unchecked" })
public Object intercept(Invocation invocation) throws Throwable {
Object[] objects = invocation.getArgs();
MappedStatement statement = (MappedStatement) objects[0];
//通过反射获取@ShardTable注解,如果为NULL则当前查询为采用分表,直接跳过
String mapperPath = statement.getId();
mapperPath = mapperPath.substring(0, mapperPath.lastIndexOf("."));
ShardTable table = Class.forName(mapperPath).getAnnotation(ShardTable.class);
if (table != null) {
Executor executor = (Executor) invocation.getTarget();
ShardingHandle shardingHandle = new ShardingHandle(statement,executor,table,objects[1],(RowBounds)objects[2],(ResultHandler
(3)ShardingHandle
主要是解析分包规则,调用不同分表规则处理器。
package com.frank.sharding.handle;
import java.util.LinkedList;
import java.util.List;
import org.apache.ibatis.executor.Executor;
import org.apache.ibatis.mapping.MappedStatement;
import org.apache.ibatis.session.ResultHandler;
import org.apache.ibatis.session.RowBounds;
import com.frank.sharding.annotation.ShardTable;
import com.frank.sharding.rule.DateRule;
import com.frank.sharding.rule.ParityRule;
import com.frank.sharding.rule.ShardRuleType;
import com.frank.sharding.rule.TotalRule;
/**
* 分表处理,根据不同规则调用不同的处理器
* @author Administrator
*
*/
public class ShardingHandle {
MappedStatement statement;
/**
* 执行器
*/
Executor executor;
/**
* 分表注解
*/
ShardTable table;
/**
* 传入的参数
*/
Object parameter;
/**
* 起始行
*/
RowBounds rowBounds;
/**
* 结果处理类
*/
ResultHandler
(4)ShardingRule
分表规则抽象类,该类的具体实现有三个DateRule(日期拆分),ParityRule(奇偶拆分),TotalRule(单表总数拆分)。
package com.frank.sharding.rule;
import java.util.List;
import org.apache.ibatis.executor.Executor;
import org.apache.ibatis.mapping.MappedStatement;
import org.apache.ibatis.mapping.ParameterMap;
import org.apache.ibatis.mapping.SqlSource;
import org.apache.ibatis.session.ResultHandler;
import org.apache.ibatis.session.RowBounds;
import com.frank.sharding.annotation.ShardTable;
public abstract class ShardingRule {
MappedStatement statement;
Executor executor;
ShardTable table;
Object parameter;
RowBounds rowBounds;
ResultHandler resultHandler;
public ShardingRule(MappedStatement statement, Executor executor, ShardTable table, Object parameter,
RowBounds rowBounds, ResultHandler resultHandler) {
this.statement = statement;
this.executor = executor;
this.table = table;
this.parameter = parameter;
this.rowBounds = rowBounds;
this.resultHandler = resultHandler;
}
/**
* 子类需实现该方法
* @return
* @throws Exception
*/
public abstract List handle() throws Exception;
/**构造新的MappedStatement
* @param ms
* @param newSqlSource
* @return
*/
public MappedStatement mappedStatement(MappedStatement ms, SqlSource newSqlSource) {
MappedStatement.Builder builder = new MappedStatement.Builder(ms.getConfiguration(), ms.getId(), newSqlSource,
ms.getSqlCommandType());
builder.resource(ms.getResource());
builder.fetchSize(ms.getFetchSize());
builder.statementType(ms.getStatementType());
builder.keyGenerator(ms.getKeyGenerator());
if (ms.getKeyProperties() != null && ms.getKeyProperties().length != 0) {
StringBuilder keyProperties = new StringBuilder();
for (String keyProperty : ms.getKeyProperties()) {
keyProperties.append(keyProperty).append(",");
}
keyProperties.delete(keyProperties.length() - 1, keyProperties.length());
builder.keyProperty(keyProperties.toString());
}
builder.timeout(ms.getTimeout());
ParameterMap parameterMap = ms.getParameterMap();
builder.parameterMap(parameterMap);
builder.resultMaps(ms.getResultMaps());
builder.resultSetType(ms.getResultSetType());
builder.cache(ms.getCache());
builder.flushCacheRequired(ms.isFlushCacheRequired());
builder.useCache(ms.isUseCache());
return builder.build();
}
}
(5)ResultHandle
该类是对分表查询结果处理,并返回用户所需要的数据。
package com.frank.sharding.handle;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import org.apache.ibatis.executor.Executor;
import org.apache.ibatis.mapping.BoundSql;
import org.apache.ibatis.mapping.MappedStatement;
import org.apache.ibatis.mapping.ParameterMap;
import org.apache.ibatis.mapping.ParameterMapping;
import org.apache.ibatis.mapping.ResultMap;
import org.apache.ibatis.mapping.ResultMapping;
import org.apache.ibatis.mapping.SqlSource;
import org.apache.ibatis.reflection.DefaultReflectorFactory;
import org.apache.ibatis.reflection.MetaObject;
import org.apache.ibatis.reflection.factory.DefaultObjectFactory;
import org.apache.ibatis.reflection.wrapper.DefaultObjectWrapperFactory;
import org.apache.ibatis.session.ResultHandler;
import org.apache.ibatis.session.RowBounds;
import org.springframework.util.StringUtils;
import com.frank.sharding.annotation.ShardTable;
import com.frank.sharding.config.BoundSqlSqlSource;
import com.frank.sharding.exception.SqlResolveException;
import com.frank.sharding.util.ObjectUtil;
import com.frank.sharding.util.PageableUtil;
import com.frank.sharding.util.ReflectUtil;
import com.frank.sharding.util.SqlResolve;
/**
* @description 分表结果处理
* @author fengsheng
* @since 2019年1月30日
* @date 2019年1月30日
*/
public class ResultHandle {
private static final String SUM = "sum";
private static final String MIN = "min";
private static final String MAX = "max";
private static final String AVG = "avg";
private static final String COUNT = "count";
/**
* @Description: 对同一张表进行了水平拆分,数据存在多张表中,该方法会对传入的失SQL进行替换去所有分表中查询符合条件的数据
* @since 2019年1月30日
* @date 2019年1月30日
* @param statement
* @param executor
* @param parameter
* @param rowBounds
* @param resultHandler
* @param table
* 原表
* @param tableNames
* 分表(不含原表)
* @return
* @throws Exception
*/
public List shardQuery(MappedStatement statement, Executor executor, Object parameter, RowBounds rowBounds,
ResultHandler resultHandler, ShardTable table, String... tableNames) throws Exception {
List resultMaps = statement.getResultMaps();
ResultMap resultMap = resultMaps.get(0);
List resultMappings = resultMap.getIdResultMappings();
String sql = statement.getBoundSql(parameter).getSql();
BoundSql boundSql = statement.getBoundSql(parameter);
List mappings = boundSql.getParameterMappings();
Map paramMap = ObjectUtil.objectToMap(parameter);
SqlResolve resolve = new SqlResolve(sql);
Map aggs = resolve.getAggregateFunction();
List groupBys = resolve.getGroupBy();
Map orderBys = resolve.getOrderBy();
Map limits = resolve.getLimit(mappings, paramMap);
// 去除分页参数
boolean isRemove = false;
if (!limits.isEmpty()) {
isRemove = true;
int end = sql.indexOf("limit") < 0 ? sql.indexOf("LIMIT") : sql.indexOf("limit");
sql = sql.substring(0, end);
}
/**
* 1)判断是否有聚合函数; 1.1)聚合函数类型 1.2)是否有分组函数group by * having * 2)判断是否有分页参数;
* 3)判断是否有排序参数;
*/
List data = new LinkedList<>();
for (int i = 0; i < tableNames.length; i++) {
String mdSql = sql.replace(table.table(), tableNames[i]);
MappedStatement newStatement = null;
if (isRemove) {
if (mappings.size() > 0) {
List newmappings = mappings.subList(0, mappings.size() - 2);
BoundSql mdBoundSql = new BoundSql(statement.getConfiguration(), mdSql, newmappings, parameter);
newStatement = mappedStatement(statement, new BoundSqlSqlSource(mdBoundSql));
}
} else {
newStatement = mappedStatement(statement, new BoundSqlSqlSource(statement.getBoundSql(parameter)));
MetaObject msObject = MetaObject.forObject(newStatement, new DefaultObjectFactory(),
new DefaultObjectWrapperFactory(), new DefaultReflectorFactory());
msObject.setValue("sqlSource.boundSql.sql", mdSql);
}
List querys = executor.query(newStatement, parameter, rowBounds, resultHandler);
data.addAll(querys);
}
Map> groupDatas = new LinkedHashMap<>();
for (Object value : data) {
String key = "";
for (String groupBy : groupBys) {
String property = null;
for (ResultMapping mapping : resultMappings) {
if (mapping.getColumn().toLowerCase().equals(groupBy)) {
property = mapping.getProperty();
break;
}
}
if (!StringUtils.isEmpty(property)) {
Object filedValue = ReflectUtil.getFieldValue(value, property);
if (null != filedValue) {
key += String.valueOf(filedValue);
}
}
}
List list = groupDatas.get(key);
if (null == list) {
list = new LinkedList<>();
}
list.add(value);
groupDatas.put(key, list);
}
List results = new ArrayList();
// 处理聚合函数和分组,合并值情形
if (!aggs.isEmpty()) {
data.clear();
groupDatas.forEach((gpro, datas) -> {
if (resultMap.getType().getName().equals(Integer.class.getName())) {
aggs.forEach((column, fun) -> {
switch (fun) {
case COUNT:
int count = 0;
for (Object object : datas) {
Integer re = (Integer) object;
count += re;
}
data.add(count);
break;
case MIN:
int min = 0;
for (int i = 0; i < datas.size(); i++) {
Integer re = (Integer) datas.get(i);
if (i == 0) {
min = re;
} else if (min > re) {
min = re;
}
}
data.add(min);
break;
case SUM:
int sum = 0;
for (Object object : datas) {
Integer re = (Integer) object;
sum += re;
}
data.add(sum);
break;
case AVG:
throw new SqlResolveException("采用分表策略的sql不能使用数据库函数avg求平均值,只能查询出总和和总数,在代码里计算平均值。");
case MAX:
int max = 0;
for (int i = 0; i < datas.size(); i++) {
Integer re = (Integer) datas.get(i);
if (i == 0) {
max = re;
} else if (max < re) {
max = re;
}
}
data.add(max);
break;
default:
break;
}
});
} else if (resultMap.getType().getName().equals(Long.class.getName())) {
aggs.forEach((column, fun) -> {
switch (fun) {
case COUNT:
long count = 0;
for (Object object : datas) {
Long re = (Long) object;
count += re;
}
data.add(count);
break;
case MIN:
long min = 0;
for (int i = 0; i < datas.size(); i++) {
Long re = (Long) datas.get(i);
if (i == 0) {
min = re;
} else if (min > re) {
min = re;
}
}
data.add(min);
break;
case SUM:
long sum = 0;
for (Object object : datas) {
Long re = (Long) object;
sum += re;
}
data.add(sum);
break;
case AVG:
throw new SqlResolveException("采用分表策略的sql不能使用数据库函数avg求平均值,只能查询出总和和总数,在代码里计算平均值。");
case MAX:
long max = 0;
for (int i = 0; i < datas.size(); i++) {
Long re = (Long) datas.get(i);
if (i == 0) {
max = re;
} else if (max < re) {
max = re;
}
}
data.add(max);
break;
default:
break;
}
});
} else if (resultMap.getType().getName().equals(Double.class.getName())) {
aggs.forEach((column, fun) -> {
switch (fun) {
case COUNT:
double count = 0;
for (Object object : datas) {
Double re = (Double) object;
count += re;
}
data.add(count);
break;
case MIN:
double min = 0;
for (int i = 0; i < datas.size(); i++) {
Double re = (Double) datas.get(i);
if (i == 0) {
min = re;
} else if (min > re) {
min = re;
}
}
data.add(min);
break;
case SUM:
double sum = 0;
for (Object object : datas) {
Double re = (Double) object;
sum += re;
}
data.add(sum);
break;
case AVG:
throw new SqlResolveException("采用分表策略的sql不能使用数据库函数avg求平均值,只能查询出总和和总数,在代码里计算平均值。");
case MAX:
double max = 0;
for (int i = 0; i < datas.size(); i++) {
Double re = (Double) datas.get(i);
if (i == 0) {
max = re;
} else if (max < re) {
max = re;
}
}
data.add(max);
break;
default:
break;
}
});
} else if (!resultMap.getType().getName().startsWith("java.lang")) {
Map maps = new LinkedHashMap<>();
aggs.forEach((column, fun) -> {
String property = null;
Class propertyType = null;
for (ResultMapping mapping : resultMappings) {
if (mapping.getColumn().toLowerCase().equals(column)) {
property = mapping.getProperty();
propertyType = mapping.getJavaType();
break;
}
}
if (!StringUtils.isEmpty(property)) {
if (propertyType.getName().equals(Integer.class.getName())) {
Integer reInteger = 0;
for (int i = 0; i < datas.size(); i++) {
Object object = datas.get(i);
Integer pvalue = (Integer) ReflectUtil.getFieldValue(object, property);
switch (fun) {
case COUNT:
reInteger += pvalue;
break;
case MIN:
if (i == 0) {
reInteger = pvalue;
} else if (reInteger > pvalue) {
reInteger = pvalue;
}
break;
case SUM:
reInteger += pvalue;
break;
case AVG:
throw new SqlResolveException(
"采用分表策略的sql不能使用数据库函数avg求平均值,只能查询出总和和总数,在代码里计算平均值。");
case MAX:
if (i == 0) {
reInteger = pvalue;
} else if (reInteger < pvalue) {
reInteger = pvalue;
}
break;
default:
break;
}
}
maps.put(property, reInteger);
} else if (propertyType.getName().equals(Long.class.getName())) {
Long rLong = 0L;
for (int i = 0; i < datas.size(); i++) {
Object object = datas.get(i);
Long pvalue = (Long) ReflectUtil.getFieldValue(object, property);
switch (fun) {
case COUNT:
rLong += pvalue;
break;
case MIN:
if (i == 0) {
rLong = pvalue;
} else if (rLong > pvalue) {
rLong = pvalue;
}
break;
case SUM:
rLong += pvalue;
break;
case AVG:
throw new SqlResolveException(
"采用分表策略的sql不能使用数据库函数avg求平均值,只能查询出总和和总数,在代码里计算平均值。");
case MAX:
if (i == 0) {
rLong = pvalue;
} else if (rLong < pvalue) {
rLong = pvalue;
}
break;
default:
break;
}
}
maps.put(property, rLong);
} else if (propertyType.getName().equals(Double.class.getName())) {
Double rDouble = 0.0;
for (int i = 0; i < datas.size(); i++) {
Object object = datas.get(i);
Double pvalue = (Double) ReflectUtil.getFieldValue(object, property);
switch (fun) {
case COUNT:
rDouble += pvalue;
break;
case MIN:
if (i == 0) {
rDouble = pvalue;
} else if (rDouble > pvalue) {
rDouble = pvalue;
}
break;
case SUM:
rDouble += pvalue;
break;
case AVG:
throw new SqlResolveException(
"采用分表策略的sql不能使用数据库函数avg求平均值,只能查询出总和和总数,在代码里计算平均值。");
case MAX:
if (i == 0) {
rDouble = pvalue;
} else if (rDouble < pvalue) {
rDouble = pvalue;
}
break;
default:
break;
}
}
maps.put(property, rDouble);
}
}
});
Object reObject = datas.get(0);
maps.forEach((property, value) -> {
ReflectUtil.setFieldValue(reObject, property, String.valueOf(value));
});
data.add(reObject);
}
});
}
/**
* 排序
*/
if (!orderBys.isEmpty()) {
for (String orderColumn : orderBys.keySet()) {
ResultMapping mapping = null;
for (ResultMapping rm : resultMappings) {
if (rm.getColumn().trim().equals(orderColumn)) {
mapping = rm;
break;
}
}
if (null != mapping) {
if (orderBys.get(orderColumn).equalsIgnoreCase("desc")) {
PageableUtil.sort(mapping.getProperty(), false, data);
} else {
PageableUtil.sort(mapping.getProperty(), true, data);
}
}
}
}
/**
* 分页
*/
if (!limits.isEmpty()) {
results = PageableUtil.page(limits.get("start"), limits.get("start") + limits.get("end"), data);
} else {
results = data;
}
/**
* 最终核验处理结果,防止查询报错
*/
if (results.size() > 1) {
String mappId = statement.getId();
Class mappClass = Class.forName(mappId.substring(0, mappId.lastIndexOf(".")));
Method[] methods = mappClass.getMethods();
for (Method method : methods) {
if (method.getName().equals(mappId.substring(mappId.lastIndexOf(".") + 1))) {
Class reClass = method.getReturnType();
String reType = reClass.getName();
if (!reType.substring(0, reType.lastIndexOf(".") < 0 ? 0 : reType.lastIndexOf(".")).equals(
"java.util")) {
results = results.subList(0, 1);
break;
}
}
}
}
return results;
}
/**
* @Description: 构建 MappedStatement
* @since 2019年2月19日
* @date 2019年2月19日
* @param ms
* @param newSqlSource
* @return
*/
private MappedStatement mappedStatement(MappedStatement ms, SqlSource newSqlSource) {
MappedStatement.Builder builder = new MappedStatement.Builder(ms.getConfiguration(), ms.getId(), newSqlSource,
ms.getSqlCommandType());
builder.resource(ms.getResource());
builder.fetchSize(ms.getFetchSize());
builder.statementType(ms.getStatementType());
builder.keyGenerator(ms.getKeyGenerator());
if (ms.getKeyProperties() != null && ms.getKeyProperties().length != 0) {
StringBuilder keyProperties = new StringBuilder();
for (String keyProperty : ms.getKeyProperties()) {
keyProperties.append(keyProperty).append(",");
}
keyProperties.delete(keyProperties.length() - 1, keyProperties.length());
builder.keyProperty(keyProperties.toString());
}
builder.timeout(ms.getTimeout());
ParameterMap parameterMap = ms.getParameterMap();
builder.parameterMap(parameterMap);
builder.resultMaps(ms.getResultMaps());
builder.resultSetType(ms.getResultSetType());
builder.cache(ms.getCache());
builder.flushCacheRequired(ms.isFlushCacheRequired());
builder.useCache(ms.isUseCache());
return builder.build();
}
}
(1)基于XML配置拦截器
(2)基于spring boot注解配置拦截器
package com.frank.sharding.config;
import org.apache.ibatis.session.SqlSessionFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import com.frank.sharding.interceptor.ShardInterceptor;
@Configuration
public class ShardInterceptorConfig {
@Bean
public ShardInterceptor shardInterceptor(SqlSessionFactory sqlSessionFactory) {
ShardInterceptor shardInterceptor = new ShardInterceptor();
sqlSessionFactory.getConfiguration().addInterceptor(shardInterceptor);
return shardInterceptor;
}
}
(3)在项目中使用@ShardTable
package com.frank.ssm.demo1.mapper;
import org.apache.ibatis.annotations.Param;
import com.frank.ssm.demo1.entity.UserInfoEntity;
import com.frank.ssm.demo1.util.ShardTable;
@ShardTable(table="user_info",column="user_id")
public interface UserInfoMapper {
public UserInfoEntity findByUserId(@Param("userId")int userId);
}
到此本分表插件的介绍就完了!
特别说明:ParityRule,TotalRule两个类未有实现具体方法,有需要的小伙伴可以自行实现哦,如果你只需要根据日期进行拆分表的话,那么恭喜你,直接去github下载源码用maven编译成jar包引入就可以用了。
本文源码github地址:基于mybatis实现的分表查询的插件(点我)
如果在使用时不明白的小伙伴可以在文末评论哦!