Spring boot+mybatis+mysql+ sharding-jdbc实现分库分表学习demo

项目初期版本使用当当sharding-jdbc进行数据库的分库分表操作,后续项目存储更换为自研弹性数据库。总结学习当当数据库中间件sharding-jdbc,并完成demo,可运行,下面列出demo代码说明:

建表sql

/*
Navicat MySQL Data Transfer 在Navicat中直接运行本sql文件创建表
* 两个库:test_msg1   包含两个表: t_order_0   t_order_1
*         test_msg2  包含两个表: t_order_0   t_order_1
*  建表sql如下
*/

SET FOREIGN_KEY_CHECKS=0;

-- ----------------------------
-- Table structure for `users`
-- ----------------------------
DROP TABLE IF EXISTS `t_order_0`;
CREATE TABLE `t_order_0` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键id',
  `order_id` varchar(32) DEFAULT NULL COMMENT '顺序编号',
  `user_id` varchar(32) DEFAULT NULL COMMENT '用户编号',
  `userName` varchar(32) DEFAULT NULL COMMENT '用户名',
  `passWord` varchar(32) DEFAULT NULL COMMENT '密码',
  `user_sex` varchar(32) DEFAULT NULL,
  `nick_name` varchar(32) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=28 DEFAULT CHARSET=utf8;

pom文件:



    4.0.0

    com.chun
    spring-boot-mybatis-sharding-jdbc
    1.0-SNAPSHOT

    
        org.springframework.boot
        spring-boot-starter-parent
        1.4.2.RELEASE
         
    

    
        UTF-8
        1.8
    

    
        
            org.springframework.boot
            spring-boot-starter
        
        
            org.springframework.boot
            spring-boot-starter-test
            test
        
        
            org.springframework.boot
            spring-boot-starter-web
        
        
            org.mybatis.spring.boot
            mybatis-spring-boot-starter
            1.1.1
        
        
            mysql
            mysql-connector-java
        
        
            org.springframework.boot
            spring-boot-devtools
            true
        

        
            org.projectlombok
            lombok
        

        
        
            com.dangdang
            sharding-jdbc-core
            1.5.4
        

        
            
            
            
        
    

    
        
            
                org.springframework.boot
                spring-boot-maven-plugin
                
                    true
                
            
        
    

分库分表最主要有几个配置:

  1. 有多少个数据源
  2. 每张表的逻辑表名和所有物理表名
  3. 用什么列进行分库以及分库算法
  4. 用什么列进行分表以及分表算法
    分为两个库:test_msg1 , test_msg2
    每个库都包含两个表: t_order_0 , t_order_1
    使用user_id作为分库列;
    使用order_id作为分表列;

配置文件:

application.properties

配置数据源相关数据属性。

mybatis.config-locations=classpath:mybatis/mybatis-config.xml

#datasource
spring.devtools.remote.restart.enabled=false

#data source1
spring.datasource.test1.driverClassName=com.mysql.jdbc.Driver
spring.datasource.test1.url=jdbc:mysql://127.0.0.1:3306/test_msg1
spring.datasource.test1.username=root
spring.datasource.test1.password=root

#data source2
spring.datasource.test2.driverClassName=com.mysql.jdbc.Driver
spring.datasource.test2.url=jdbc:mysql://127.0.0.1:3306/test_msg2
spring.datasource.test2.username=root
spring.datasource.test2.password=root

Application

Application.java
package com.chun;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.transaction.annotation.EnableTransactionManagement;

/**
 * Created by Kane on 2018/1/17.
 */
@SpringBootApplication
@EnableAutoConfiguration(exclude={DataSourceAutoConfiguration.class}) //排除DataSourceConfiguratrion
@EnableTransactionManagement(proxyTargetClass = true)   //开启事物管理功能
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

Entity 定义实体类

UserEntity.java
package com.chun.entity;

import com.chun.enums.UserSexEnum;
import java.io.Serializable;

/**
 * Created by Kane on 2018/1/17.
 */
public class UserEntity implements Serializable {

    private static final long serialVersionUID = 1L;
    private Long id;
    private Long order_id;
    private Long user_id;
    private String userName;
    private String passWord;
    private UserSexEnum userSex;
    private String nickName;

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }


    public Long getOrder_id() {
        return order_id;
    }

    public void setOrder_id(Long order_id) {
        this.order_id = order_id;
    }

    public Long getUser_id() {
        return user_id;
    }

    public void setUser_id(Long user_id) {
        this.user_id = user_id;
    }

    public String getUserName() {
        return userName;
    }

    public void setUserName(String userName) {
        this.userName = userName;
    }

    public String getPassWord() {
        return passWord;
    }

    public void setPassWord(String passWord) {
        this.passWord = passWord;
    }

    public UserSexEnum getUserSex() {
        return userSex;
    }

    public void setUserSex(UserSexEnum userSex) {
        this.userSex = userSex;
    }

    public String getNickName() {
        return nickName;
    }

    public void setNickName(String nickName) {
        this.nickName = nickName;
    }

}
UserSexEnum.java
package com.chun.enums;

/**
 * Created by Kane on 2018/1/17.
 */
public enum UserSexEnum {
    MAN, WOMAN
}

Service层

User1Service.java
package com.chun.service;

import com.chun.entity.UserEntity;
import com.chun.mapper.User1Mapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;

/**
 * Created by Kane on 2018/1/17.
 */
@Slf4j
@Service
public class User1Service {

    @Autowired
    private User1Mapper user1Mapper;

    public List getUsers() {
        List users=user1Mapper.getAll();
        return users;
    }

//    @Transactional(value="test1TransactionManager",rollbackFor = Exception.class,timeout=36000)  //说明针对Exception异常也进行回滚,如果不标注,则Spring 默认只有抛出 RuntimeException才会回滚事务
    public void updateTransactional(UserEntity user) {
        try{
            user1Mapper.insert(user);
            log.error(String.valueOf(user));
        }catch(Exception e){
            log.error("find exception!");
            throw e;   // 事物方法中,如果使用trycatch捕获异常后,需要将异常抛出,否则事物不回滚。
        }

    }
}

Mapper层

User1Mapper.java
package com.chun.mapper;

import com.chun.entity.UserEntity;

import java.util.List;

/**
 * Created by Kane on 2018/1/17.
 */
public interface User1Mapper {

    List getAll();

    void update(UserEntity user);

}

数据源配置及Mybatis配置:

配置多个数据源,数据源的名称最好要有一定的规则,方便配置分库的计算规则;
配置数据源规则,即将多个数据源交给sharding-jdbc管理,并且可以设置默认的数据源,当表没有配置分库规则时会使用默认的数据源;
分库:
user_id % 2 = 0的数据存储到test_msg1 ,为1的数据存储到test_msg0
分表:
order_id % 2 = 0的数据存储到 t_order_0,为1的数据存储到t_order_1

DataSourceConfig.java
package com.chun;


import com.chun.strategy.ModuloDatabaseShardingAlgorithm;
import com.chun.strategy.ModuloTableShardingAlgorithm;
import com.dangdang.ddframe.rdb.sharding.api.ShardingDataSourceFactory;
import com.dangdang.ddframe.rdb.sharding.api.rule.BindingTableRule;
import com.dangdang.ddframe.rdb.sharding.api.rule.DataSourceRule;
import com.dangdang.ddframe.rdb.sharding.api.rule.ShardingRule;
import com.dangdang.ddframe.rdb.sharding.api.rule.TableRule;
import com.dangdang.ddframe.rdb.sharding.api.strategy.database.DatabaseShardingStrategy;
import com.dangdang.ddframe.rdb.sharding.api.strategy.table.TableShardingStrategy;
import org.apache.ibatis.session.SqlSessionFactory;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.SqlSessionTemplate;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;

import javax.sql.DataSource;
import java.sql.SQLException;
import java.util.*;

/**
 *  数据源及分表配置
 * Created by Kane on 2018/1/17.
 */
@Configuration
@MapperScan(basePackages = "com.chun.mapper", sqlSessionTemplateRef  = "test1SqlSessionTemplate")
public class DataSourceConfig {


    /**
     * 配置数据源0,数据源的名称最好要有一定的规则,方便配置分库的计算规则
     * @return
     */
    @Bean(name="dataSource0")
    @ConfigurationProperties(prefix = "spring.datasource.test1")
    public DataSource dataSource0(){
        return DataSourceBuilder.create().build();
    }
    /**
     * 配置数据源1,数据源的名称最好要有一定的规则,方便配置分库的计算规则
     * @return
     */
    @Bean(name="dataSource1")
    @ConfigurationProperties(prefix = "spring.datasource.test2")
    public DataSource dataSource1(){
        return DataSourceBuilder.create().build();
    }

    /**
     * 配置数据源规则,即将多个数据源交给sharding-jdbc管理,并且可以设置默认的数据源,
     * 当表没有配置分库规则时会使用默认的数据源
     * @param dataSource0
     * @param dataSource1
     * @return
     */
    @Bean
    public DataSourceRule dataSourceRule(@Qualifier("dataSource0") DataSource dataSource0,
                                         @Qualifier("dataSource1") DataSource dataSource1){
        Map dataSourceMap = new HashMap<>(); //设置分库映射
        dataSourceMap.put("dataSource0", dataSource0);
        dataSourceMap.put("dataSource1", dataSource1);
        return new DataSourceRule(dataSourceMap, "dataSource0"); //设置默认库,两个库以上时必须设置默认库。默认库的数据源名称必须是dataSourceMap的key之一
    }

    /**
     * 配置数据源策略和表策略,具体策略需要自己实现
     * @param dataSourceRule
     * @return
     */
    @Bean
    public ShardingRule shardingRule(DataSourceRule dataSourceRule){
        //具体分库分表策略
        TableRule orderTableRule = TableRule.builder("t_order")
                .actualTables(Arrays.asList("t_order_0", "t_order_1"))
                .tableShardingStrategy(new TableShardingStrategy("order_id", new ModuloTableShardingAlgorithm()))
                .dataSourceRule(dataSourceRule)
                .build();

        //绑定表策略,在查询时会使用主表策略计算路由的数据源,因此需要约定绑定表策略的表的规则需要一致,可以一定程度提高效率
        List bindingTableRules = new ArrayList();
        bindingTableRules.add(new BindingTableRule(Arrays.asList(orderTableRule)));
        return ShardingRule.builder()
                .dataSourceRule(dataSourceRule)
                .tableRules(Arrays.asList(orderTableRule))
                .bindingTableRules(bindingTableRules)
                .databaseShardingStrategy(new DatabaseShardingStrategy("user_id", new ModuloDatabaseShardingAlgorithm()))
                .tableShardingStrategy(new TableShardingStrategy("order_id", new ModuloTableShardingAlgorithm()))
                .build();
    }

    /**
     * 创建sharding-jdbc的数据源DataSource,MybatisAutoConfiguration会使用此数据源
     * @param shardingRule
     * @return
     * @throws SQLException
     */
    @Bean(name="dataSource")
    public DataSource shardingDataSource(ShardingRule shardingRule) throws SQLException {
        return ShardingDataSourceFactory.createDataSource(shardingRule);
    }

    /**
     * 需要手动配置事务管理器
     * @param dataSource
     * @return
     */
    @Bean
    public DataSourceTransactionManager transactitonManager(@Qualifier("dataSource") DataSource dataSource){
        return new DataSourceTransactionManager(dataSource);
    }

    @Bean(name = "test1SqlSessionFactory")
    @Primary
    public SqlSessionFactory testSqlSessionFactory(@Qualifier("dataSource") DataSource dataSource) throws Exception {
        SqlSessionFactoryBean bean = new SqlSessionFactoryBean();
        bean.setDataSource(dataSource);
        bean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath:mybatis/mapper/*.xml"));
        return bean.getObject();
    }

    @Bean(name = "test1SqlSessionTemplate")
    @Primary
    public SqlSessionTemplate testSqlSessionTemplate(@Qualifier("test1SqlSessionFactory") SqlSessionFactory sqlSessionFactory) throws Exception {
        return new SqlSessionTemplate(sqlSessionFactory);
    }
}

分库策略实现:

ModuloDatabaseShardingAlgorithm.java
package com.chun.strategy;

import com.dangdang.ddframe.rdb.sharding.api.ShardingValue;
import com.dangdang.ddframe.rdb.sharding.api.strategy.database.SingleKeyDatabaseShardingAlgorithm;
import com.google.common.collect.Range;

import java.util.Collection;
import java.util.LinkedHashSet;

/**
 * 分库策略的简单实现
 * Created by Kane on 2018/1/22.
 */
public class ModuloDatabaseShardingAlgorithm implements SingleKeyDatabaseShardingAlgorithm {

    @Override
    public String doEqualSharding(Collection databaseNames, ShardingValue shardingValue) {
        for (String each : databaseNames) {
            if (each.endsWith(Long.parseLong(shardingValue.getValue().toString()) % 2 + "")) {
                return each;
            }
        }
        throw new IllegalArgumentException();
    }

    @Override
    public Collection doInSharding(Collection databaseNames, ShardingValue shardingValue) {
        Collection result = new LinkedHashSet<>(databaseNames.size());
        for (Long value : shardingValue.getValues()) {
            for (String tableName : databaseNames) {
                if (tableName.endsWith(value % 2 + "")) {
                    result.add(tableName);
                }
            }
        }
        return result;
    }

    @Override
    public Collection doBetweenSharding(Collection databaseNames, ShardingValue shardingValue) {
        Collection result = new LinkedHashSet<>(databaseNames.size());
        Range range = (Range) shardingValue.getValueRange();
        for (Long i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) {
            for (String each : databaseNames) {
                if (each.endsWith(i % 2 + "")) {
                    result.add(each);
                }
            }
        }
        return result;
    }
}

debug方法doEqualSharding会发现:

 

Spring boot+mybatis+mysql+ sharding-jdbc实现分库分表学习demo_第1张图片

分库代码.png

Spring boot+mybatis+mysql+ sharding-jdbc实现分库分表学习demo_第2张图片

doEqualSharding参数.png

根据user_id的值返回路由的库的名称。
分库:
user_id % 2 = 0的数据存储到test_msg1 ,为1的数据存储到test_msg0,

分表策略的基本实现

分表逻辑类需要实现SingleKeyTableShardingAlgorithm接口的三个方法doBetweenSharding、doEqualSharding、doInSharding

ModuloTableShardingAlgorithm.java
package com.chun.strategy;

import com.dangdang.ddframe.rdb.sharding.api.ShardingValue;
import com.dangdang.ddframe.rdb.sharding.api.strategy.table.SingleKeyTableShardingAlgorithm;
import com.google.common.collect.Range;

import java.util.Collection;
import java.util.LinkedHashSet;

/**
 * 分表策略的基本实现
 * Created by Kane on 2018/1/22.
 */
public class ModuloTableShardingAlgorithm implements SingleKeyTableShardingAlgorithm {

    @Override
    public String doEqualSharding(Collection tableNames, ShardingValue shardingValue) {
        for (String each : tableNames) {
            if (each.endsWith(shardingValue.getValue() % 2 + "")) {
                return each;
            }
        }
        throw new IllegalArgumentException();
    }

    @Override
    public Collection doInSharding(Collection tableNames, ShardingValue shardingValue) {
        Collection result = new LinkedHashSet<>(tableNames.size());
        for (Long value : shardingValue.getValues()) {
            for (String tableName : tableNames) {
                if (tableName.endsWith(value % 2 + "")) {
                    result.add(tableName);
                }
            }
        }
        return result;
    }

    @Override
    public Collection doBetweenSharding(Collection tableNames, ShardingValue shardingValue) {
        Collection result = new LinkedHashSet<>(tableNames.size());
        Range range = (Range) shardingValue.getValueRange();
        for (Long i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) {
            for (String each : tableNames) {
                if (each.endsWith(i % 2 + "")) {
                    result.add(each);
                }
            }
        }
        return result;
    }
}

Controller层

UserController.java
package com.chun.web;

import com.chun.entity.UserEntity;
import com.chun.enums.UserSexEnum;
import com.chun.service.User1Service;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

/**
 * Created by Kane on 2018/1/17.
 */
@Service
@RestController
public class UserController {

    @Autowired
    private User1Service user1Service;

    @RequestMapping("/getUsers")
    public List getUsers() {
        List users=user1Service.getUsers();
        return users;
    }

    //测试
    @RequestMapping(value="update1")
    public String updateTransactional(@RequestParam(value = "id") Long id,
                                    @RequestParam(value = "user_id") Long user_id,
                                    @RequestParam(value = "order_id") Long order_id,
                                    @RequestParam(value = "nickName") String nickName,
                                    @RequestParam(value = "passWord") String passWord,
                                    @RequestParam(value = "userName") String userName
                                    ) {
        UserEntity user2 = new UserEntity();
        user2.setId(id);
        user2.setUser_id(user_id);
        user2.setOrder_id(order_id);
        user2.setNickName(nickName);
        user2.setPassWord(passWord);
        user2.setUserName(userName);
        user2.setUserSex(UserSexEnum.WOMAN);
        user1Service.updateTransactional(user2);
        return "test";
    }
}

Mybatis 配置文件

resources/mybatis/mapper/UserMapper.xml



    
        
        
        
        
        
        
        
    

    
        id, userName, passWord, user_sex, nick_name
    

    

    
        INSERT INTO
        t_order
        (order_id,user_id,userName,passWord,user_sex)
        VALUES
        (#{order_id},#{user_id},#{userName}, #{passWord}, #{userSex})
    


resources/mybatis/mybatis-config.xml



    
        
        
        
        
        
        
    

 

在对mysql数据库进行SHARDING的设计中,基中非常重要的就是选择SHARD KEY和SHARD KEY如何去分布。
选择SHARD KEY的一个基本原则就是能使数据根据业务类型尽量分散在多个SHARDING服务器中.

   第一种方法是选定某个栏位作为SHARD KEY,根据SHARD KEY的特点,可以选择基于hash,range和list进行SHARDING.
比如现在有3个SHARDING节点,根据SHARD KEY的值,分别hash在这三个SHARDING节点中.range表示对于
某个SHARD KEY范围内的值会存放到一个SHARDING节点上.以上几种shared分布式,相信对于了解分区的
朋友来说一定十分的熟悉.例如有个订单表,表结构如下:

create table order_item
(id integer,
 uid integer,
 amount integer)

alter table order_item add primary key(id);
  
  应用程序生成一个id号,然后再根据id做一个hash操作,比如id为12302,通过hash函数,结果为1,数据在1节点中.
  
  这种SHARD KEY的分布方式有一个很不好的地方就是当需要增加或是减少shared节点的情况很不方
便,需要对以前的所有数据进行重新分布.大数据量的情况下这个过程将会变得非常的耗时.


  另一种方法通过通过建立一个字典表,比如建立一个服务器列表,然后应用程序通过随机函数从表中随机选择
SHARDING节点进行处理,比如

shard id      ip

1            172.28.10.11
2            172.28.10.12
3            172.28.10.13

由应用程序随机产生1-3的随机数,再决定使用哪个服务器进行处理.这种方式的好处是增加SHARDING节点和删除
SHARDING节点都非常的方便.不足的地方是这种方式没有SHARD KEY,数据完全是随机的处理.但是可以通过在redis或是
memcached进行一些优化,比如跟踪原来打算做SHARD KEY的栏位值是分别存放在哪些节点中。

  通过在redis或是memcached跟踪id和对应的存放服务器,比如id为12302是存放在2服务器中。当应用程序需要
查询某个id的记录时,先从缓存中查出来该id对应的服务器是第几号就可以了,而不再通过SHARD KEY的规则自己算出哪个SHAREDING
节点,然后再去访问对应的服务器.



参考文献:
https://www.jianshu.com/p/74c02a2a89de

 

你可能感兴趣的:(数据库事务,分布式)