分库分表-ShardingSphere-ShardingJDBC
分片和分区的区别
| Sharding 分片 | Partition分区 | |
|---|---|---|
| 存储依赖 | 可跨越DB 可跨越物理机器 | 可跨越表空间，不能跨DB存储 |
| 数据划分 | 常见为时间、范围、面向服务等 | 范围、Hash、列表、混合分区等 |
| 存储方式 | 分布式 | 集中式 |
| 扩展性 | Scale Out (水平扩展) | Scale Up（垂直扩展） |
| 可用性 | 无单点 | 存在单点（DB本身） |
| 价格 | 低廉 | 适中（DAS）甚至昂贵（SAN） |
| 应用场景 | Web 2.0网站 | 多数传统应用 |
ShardingSphere
version:4.x
Apache ShardingSphere 是一套开源的分布式数据库中间件解决方案组成的生态圈，它由 JDBC、Proxy 和 Sidecar（规划中）这 3 款相互独立，却又能够混合部署配合使用的产品组成。 它们均提供标准化的数据分片、分布式事务和数据库治理功能，可适用于如 Java 同构、异构语言、云原生等各种多样化的应用场景。
Sharding-JDBC
依赖
<dependency>
<groupId>org.apache.shardingspheregroupId>
<artifactId>sharding-jdbc-spring-boot-starterartifactId>
<version>4.0.0version>
dependency>
库：test1 schema
-- ----------------------------
-- Table structure for user_1
-- ----------------------------
DROP TABLE IF EXISTS `user_1`;
CREATE TABLE `user_1` (
`id` bigint(20) NOT NULL,
`uuid` bigint(20) NULL DEFAULT NULL,
`name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Compact;
-- ----------------------------
-- Table structure for user_2
-- ----------------------------
DROP TABLE IF EXISTS `user_2`;
CREATE TABLE `user_2` (
`id` bigint(20) NOT NULL,
`uuid` bigint(20) NULL DEFAULT NULL,
`name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Compact;
库：test2 schema
-- ----------------------------
-- Table structure for user_1
-- ----------------------------
DROP TABLE IF EXISTS `user_1`;
CREATE TABLE `user_1` (
`id` bigint(20) NOT NULL,
`uuid` bigint(20) NULL DEFAULT NULL,
`name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Compact;
-- ----------------------------
-- Table structure for user_2
-- ----------------------------
DROP TABLE IF EXISTS `user_2`;
CREATE TABLE `user_2` (
`id` bigint(20) NOT NULL,
`uuid` bigint(20) NULL DEFAULT NULL,
`name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Compact;
SET FOREIGN_KEY_CHECKS = 1;
配置文件
spring:
shardingsphere:
#数据源
datasource:
names: test1,test2
test1:
type: com.zaxxer.hikari.HikariDataSource
driver-class-name: com.mysql.cj.jdbc.Driver
jdbc-url: jdbc:mysql://localhost:3306/test_1
username: root
password: abc123
test2:
type: com.zaxxer.hikari.HikariDataSource
driver-class-name: com.mysql.cj.jdbc.Driver
jdbc-url: jdbc:mysql://localhost:3306/test_2
username: root
password: abc123
#分片
sharding:
tables:
user:
#由数据源名 + 表名组成，以小数点分隔。多个表以逗号分隔，支持inline表达式。
#缺省表示使用已知数据源与逻辑表名称生成数据节点，用于广播表（即每个库中都需要一个同样的表用于关联查询，多为字典表）或只分库不分表且所有库的表结构完全一致的情况
actual-data-nodes: test$->{1..2}.user_$->{1..2}
key-generator:
#自增列名称，缺省表示不使用自增主键生成器
column: id
#自增列值生成器类型，缺省表示使用默认自增列值生成器。可使用用户自定义的列值生成器或选择内置类型：SNOWFLAKE/UUID
type: SNOWFLAKE
table-strategy:
inline:
#分表列名称
sharding-column: id
#分表算法行表达式，需符合groovy语法
#偶数进表1，奇数进表2
algorithm-expression: user_$->{id%2+1}
database-strategy:
inline:
#分片列名称
sharding-column: uuid
#分片算法行表达式，需符合groovy语法
#偶数进库1，奇数进库2
algorithm-expression: test$->{uuid%2+1}
props:
sql:
show: true
模型
@Data
public class User {
private Long id;
private Long uuid;
private String name;
}
测试
插入10个用户数据
@Test
void add() {
for (int i = 0; i < 10; i++) {
User user=new User();
user.setUuid(111111111111112L);
user.setName(i+"号用户");
userDao.insert(user);
}
}
部分日志验证配置逻辑：
id=1277564054063390721(奇数)
uuid=0(偶数)
用户数据插入到了test1库的user_2表
2020-06-29 19:26:21.799 INFO 18452 --- [ main] ShardingSphere-SQL : Rule Type: sharding
2020-06-29 19:26:21.799 INFO 18452 --- [ main] ShardingSphere-SQL : Logic SQL: INSERT INTO user ( id,
uuid,
name ) VALUES ( ?,
?,
? )
2020-06-29 19:26:21.799 INFO 18452 --- [ main] ShardingSphere-SQL : SQLStatement: InsertSQLStatementContext(super=CommonSQLStatementContext(sqlStatement=org.apache.shardingsphere.sql.parser.sql.statement.dml.InsertStatement@5fd18419, tablesContext=TablesContext(tables=[Table(name=user, alias=Optional.absent())], schema=Optional.absent())), columnNames=[id, uuid, name], insertValueContexts=[InsertValueContext(parametersCount=3, valueExpressions=[ParameterMarkerExpressionSegment(startIndex=48, stopIndex=48, parameterMarkerIndex=0), ParameterMarkerExpressionSegment(startIndex=51, stopIndex=51, parameterMarkerIndex=1), ParameterMarkerExpressionSegment(startIndex=54, stopIndex=54, parameterMarkerIndex=2)], parameters=[1277564054063390721, 0, 0号用户])])
2020-06-29 19:26:21.800 INFO 18452 --- [ main] ShardingSphere-SQL : Actual SQL: test1 ::: INSERT INTO user_2 ( id,
uuid,
name ) VALUES (?, ?, ?) ::: [1277564054063390721, 0, 0号用户]
id=1277564056995209217(奇数)
uuid=1(奇数)
用户数据插入到了test2库的user_2表
2020-06-29 19:26:21.838 INFO 18452 --- [ main] ShardingSphere-SQL : Rule Type: sharding
2020-06-29 19:26:21.838 INFO 18452 --- [ main] ShardingSphere-SQL : Logic SQL: INSERT INTO user ( id,
uuid,
name ) VALUES ( ?,
?,
? )
2020-06-29 19:26:21.839 INFO 18452 --- [ main] ShardingSphere-SQL : SQLStatement: InsertSQLStatementContext(super=CommonSQLStatementContext(sqlStatement=org.apache.shardingsphere.sql.parser.sql.statement.dml.InsertStatement@5fd18419, tablesContext=TablesContext(tables=[Table(name=user, alias=Optional.absent())], schema=Optional.absent())), columnNames=[id, uuid, name], insertValueContexts=[InsertValueContext(parametersCount=3, valueExpressions=[ParameterMarkerExpressionSegment(startIndex=48, stopIndex=48, parameterMarkerIndex=0), ParameterMarkerExpressionSegment(startIndex=51, stopIndex=51, parameterMarkerIndex=1), ParameterMarkerExpressionSegment(startIndex=54, stopIndex=54, parameterMarkerIndex=2)], parameters=[1277564056995209217, 1, 1号用户])])
2020-06-29 19:26:21.839 INFO 18452 --- [ main] ShardingSphere-SQL : Actual SQL: test2 ::: INSERT INTO user_2 ( id,
uuid,
name ) VALUES (?, ?, ?) ::: [1277564056995209217, 1, 1号用户]
id=1277564057037152258(偶数)
uuid=3(奇数)
用户数据插入到了test2库的user_1表
2020-06-29 19:26:21.848 INFO 18452 --- [ main] ShardingSphere-SQL : Rule Type: sharding
2020-06-29 19:26:21.848 INFO 18452 --- [ main] ShardingSphere-SQL : Logic SQL: INSERT INTO user ( id,
uuid,
name ) VALUES ( ?,
?,
? )
2020-06-29 19:26:21.848 INFO 18452 --- [ main] ShardingSphere-SQL : SQLStatement: InsertSQLStatementContext(super=CommonSQLStatementContext(sqlStatement=org.apache.shardingsphere.sql.parser.sql.statement.dml.InsertStatement@5fd18419, tablesContext=TablesContext(tables=[Table(name=user, alias=Optional.absent())], schema=Optional.absent())), columnNames=[id, uuid, name], insertValueContexts=[InsertValueContext(parametersCount=3, valueExpressions=[ParameterMarkerExpressionSegment(startIndex=48, stopIndex=48, parameterMarkerIndex=0), ParameterMarkerExpressionSegment(startIndex=51, stopIndex=51, parameterMarkerIndex=1), ParameterMarkerExpressionSegment(startIndex=54, stopIndex=54, parameterMarkerIndex=2)], parameters=[1277564057037152258, 3, 3号用户])])
2020-06-29 19:26:21.848 INFO 18452 --- [ main] ShardingSphere-SQL : Actual SQL: test2 ::: INSERT INTO user_1 ( id,
uuid,
name ) VALUES (?, ?, ?) ::: [1277564057037152258, 3, 3号用户]
查询
@Test
void get() {
QueryWrapper<User> userQueryWrapper=new QueryWrapper<>();
userQueryWrapper.eq("id",1277564057016180737L);
User user = userDao.selectOne(userQueryWrapper);
System.out.println(user);
}
由于id=1277564057016180737(奇数)可能在tes1，也可能在test2，但是肯定在表user_2，所以
2020-06-29 19:34:52.485 INFO 20160 --- [ main] ShardingSphere-SQL : Rule Type: sharding
2020-06-29 19:34:52.485 INFO 20160 --- [ main] ShardingSphere-SQL : Logic SQL: SELECT id,uuid,name FROM user
WHERE (id = ?)
2020-06-29 19:34:52.485 INFO 20160 --- [ main] ShardingSphere-SQL : SQLStatement: SelectSQLStatementContext(super=CommonSQLStatementContext(sqlStatement=org.apache.shardingsphere.sql.parser.sql.statement.dml.SelectStatement@7f012923, tablesContext=TablesContext(tables=[Table(name=user, alias=Optional.absent())], schema=Optional.absent())), projectionsContext=ProjectionsContext(startIndex=8, stopIndex=19, distinctRow=false, projections=[ColumnProjection(owner=null, name=id, alias=Optional.absent()), ColumnProjection(owner=null, name=uuid, alias=Optional.absent()), ColumnProjection(owner=null, name=name, alias=Optional.absent())], columnLabels=[id, uuid, name]), groupByContext=org.apache.shardingsphere.sql.parser.relation.segment.select.groupby.GroupByContext@601f264d, orderByContext=org.apache.shardingsphere.sql.parser.relation.segment.select.orderby.OrderByContext@4403bff8, paginationContext=org.apache.shardingsphere.sql.parser.relation.segment.select.pagination.PaginationContext@261275ae, containsSubquery=false)
2020-06-29 19:34:52.486 INFO 20160 --- [ main] ShardingSphere-SQL : Actual SQL: test1 ::: SELECT id,uuid,name FROM user_2
WHERE (id = ?) ::: [1277564057016180737]
2020-06-29 19:34:52.486 INFO 20160 --- [ main] ShardingSphere-SQL : Actual SQL: test2 ::: SELECT id,uuid,name FROM user_2
WHERE (id = ?) ::: [1277564057016180737]
User(id=1277564057016180737, uuid=2, name=2号用户)
总结
- 分片是一个很笼统的概念，具体到可以通过分库和分表来实现
- 分库分表主要是按照一定转发规则将数据进行合理分配存储
个人公众号
