Springboot使用application.properties集成Sharding-JDBC 分库分表-------（ 2019年成功版）

当你们看到这博客说明我是真实存在的，这是第二次做分享，有什么不好的地方，可使劲的抽打润肉的脸蛋，在sharding-jdbc之前希望大家都先把官网的配置说明看下，尤其是行表达式的配置和主从数据库注入分表分库关系，相信大家都是迷恋技术的人不会那么偷懒，下面是我踩过的坑

官网地址：https://shardingsphere.apache.org/document/current/cn/features/sharding/other-features/inline-expression/

数据库的名字myorder和myuser

CREATE TABLE `user_0` (
  `uid` bigint(20) NOT NULL,
  `name` varchar(64) DEFAULT NULL COMMENT '名称',
  `sex` tinyint(1) DEFAULT NULL COMMENT '性别',
  `phone` varchar(32) DEFAULT NULL COMMENT '电话',
  `email` varchar(32) DEFAULT NULL COMMENT '邮箱',
  `password` varchar(32) DEFAULT NULL COMMENT '密码',
  PRIMARY KEY (`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `user_1` (
  `uid` bigint(20) NOT NULL,
  `name` varchar(64) DEFAULT NULL COMMENT '名称',
  `sex` tinyint(1) DEFAULT NULL COMMENT '性别',
  `phone` varchar(32) DEFAULT NULL COMMENT '电话',
  `email` varchar(32) DEFAULT NULL COMMENT '邮箱',
  `password` varchar(32) DEFAULT NULL COMMENT '密码',
  PRIMARY KEY (`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `user_2` (
  `uid` bigint(20) NOT NULL,
  `name` varchar(64) DEFAULT NULL COMMENT '名称',
  `sex` tinyint(1) DEFAULT NULL COMMENT '性别',
  `phone` varchar(32) DEFAULT NULL COMMENT '电话',
  `email` varchar(32) DEFAULT NULL COMMENT '邮箱',
  `password` varchar(32) DEFAULT NULL COMMENT '密码',
  PRIMARY KEY (`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

我使用的工具是IDEA

pom.xml  需要注意的是sharding -jdbc 3.0.0的版本会自动检测表结构，据说高版本不会有

	4.0.0
	
		org.springframework.boot
		spring-boot-starter-parent
		2.1.5.RELEASE
		 
	
	com.example
	zdm_store_pay
	0.0.1-SNAPSHOT
	war
	zdm_store_pay
	Demo project for Spring Boot

	
		1.8
		Greenwich.SR1
	

	
		
			org.springframework.boot
			spring-boot-starter-web
		
		
			org.springframework.cloud
			spring-cloud-starter-netflix-eureka-client
		
		
			org.springframework.boot
			spring-boot-starter-test
			test
		
		
		
			io.shardingsphere
			sharding-jdbc-spring-boot-starter
			3.0.0
		
		
			io.shardingsphere
			sharding-jdbc-spring-namespace
			3.0.0
		
		
			com.alibaba
			druid-spring-boot-starter
			1.1.9
		
		
			com.alibaba
			druid
			1.1.12
		
		
		
			org.springframework.boot
			spring-boot-starter-log4j2
			2.1.1.RELEASE
		
		
		
			com.alibaba
			fastjson
			1.2.56
		
		
		
			org.mybatis.spring.boot
			mybatis-spring-boot-starter
			1.3.2
		
		
		
			mysql
			mysql-connector-java
		

        
            org.apache.commons
            commons-lang3
            3.1
        
	

	
		
			
				org.springframework.cloud
				spring-cloud-dependencies
				${spring-cloud.version}
				pom
				import
			
		
	

	
		
			
				org.springframework.boot
				spring-boot-maven-plugin
			
		
	

application.properties

在这配置文件中需要注意 这段配置

sharding.jdbc.config.sharding.tables.user.actual-data-nodes=ds_$->{0..1}.user_$->{0..2}

在这段配置中ds_并不是从 sharding.jdbc.datasource.names=master0,mster1中获取   搜过一些文章都是这样配置导致我纳闷了很久也是最初没看官网配置说明出现的情况 如果把maters->{0..1}替换ds_->{0..1} 就会报以下错误

Caused by: io.shardingsphere.core.exception.ShardingException: Cannot find data source in sharding rule, invalid actual data node is: 'master0.user_0'

ds_$->{0..1}指的是主从数据库的设置  因为我这里没有设置从数据库具体配置可参考以下链接

小东哥：https://www.52lhd.com/article/21

https://www.fangzhipeng.com/db/2019/07/01/shardingjdbc-dbs-tbls-ms.html

#主数据库
sharding.jdbc.config.sharding.master-slave-rules.ds_0.masterDataSourceName=master0
# 从数据库0
sharding.jdbc.config.sharding.master-slave-rules.ds_0.slave-data-source-names=master0

#主数据库
sharding.jdbc.config.sharding.master-slave-rules.ds_1.masterDataSourceName=master1
# 从数据库0
sharding.jdbc.config.sharding.master-slave-rules.ds_1.slave-data-source-names=master1

当在配置中 遇到  actual-data-nodes没有时出现的是 actualDataNodes 是一样的用意

sharding.jdbc.config.sharding.tables.user.actual-data-nodes 

sharding.jdbc.config.sharding.tables.user.actualDataNodes

server.port=9888
spring.application.name=pay
eureka.client.service-url.defaultZone=http://zdm:zdm123@eureka的ip地址:9880/eureka/
eureka.instance.prefer-ip-address=true
eureka.instance.instance-id=${spring.cloud.client.ip-address}:${server.port}

mybatis.config-location=classpath:mabaties/mybatis-config.xml

###############################################################################

sharding.jdbc.datasource.names=master0,master1
#主数据源
sharding.jdbc.datasource.master0.type=com.alibaba.druid.pool.DruidDataSource
sharding.jdbc.datasource.master0.driver-class-name=com.mysql.cj.jdbc.Driver
sharding.jdbc.datasource.master0.url=jdbc:mysql://172.16.20.61:3306/myuser?characterEncoding=utf-8&serverTimezone=UTC
# 主数据库用户名 密码
sharding.jdbc.datasource.master0.username=root
sharding.jdbc.datasource.master0.password=123456

# 从数据源
sharding.jdbc.datasource.master1.type=com.alibaba.druid.pool.DruidDataSource
sharding.jdbc.datasource.master1.driver-class-name=com.mysql.cj.jdbc.Driver
sharding.jdbc.datasource.master1.url=jdbc:mysql://172.16.20.61:3306/myorder?characterEncoding=utf-8&serverTimezone=UTC
# 从数据库用户名 密码
sharding.jdbc.datasource.master1.username=root
sharding.jdbc.datasource.master1.password=123456


###########################################################################################
sharding.jdbc.config.masterslave.load-balance-algorithm-type=round_robin
sharding.jdbc.config.masterslave.name=datasource

# 分库策略  分库的维度 定义在那张表 的ID来进行分库 例如：tables.t_clientinfo
sharding.jdbc.config.sharding.default-database-strategy.inline.sharding-column=sex
sharding.jdbc.config.sharding.default-database-strategy.inline.algorithm-expression=ds_${sex %2}

# 水平拆分的数据库（表） 配置分库 + 分表策略 行表达式分片策略 0..1
sharding.jdbc.config.sharding.tables.user.actual-data-nodes=ds_$->{0..1}.user_$->{0..2}
sharding.jdbc.config.sharding.tables.user.table-strategy.inline.sharding-column=sex
sharding.jdbc.config.sharding.tables.user.table-strategy.inline.algorithm-expression=user_${sex % 3}
#主键
sharding.jdbc.config.sharding.tables.user.key-generator-column-name=uid

sharding.jdbc.config.sharding.tables.user.logicTable=user
#主数据库
sharding.jdbc.config.sharding.master-slave-rules.ds_0.masterDataSourceName=master0
# 从数据库0
sharding.jdbc.config.sharding.master-slave-rules.ds_0.slave-data-source-names=master0

#主数据库
sharding.jdbc.config.sharding.master-slave-rules.ds_1.masterDataSourceName=master1
# 从数据库0
sharding.jdbc.config.sharding.master-slave-rules.ds_1.slave-data-source-names=master1

spring.main.allow-bean-definition-overriding=true
sharding.jdbc.config.props.sql.show=true

配置说明

• 在上面的配置中，其中sharding.jdbc.datasource部分是配置数据库的信息，配置了3个数据库。
• sharding.jdbc.config.sharding.master-slave-rules.ds_0.master-data-source-name配置的是ds_0区的的主库名称，同理ds_1。
• sharding.jdbc.config.sharding.master-slave-rules.ds_0.slave-data-source-names配置的是ds_0区的的从库名称，同理ds_1。
• sharding.jdbc.config.sharding.default-database-strategy.inline.sharding-column配置的分库的字段，本案例是根据id进行分。
• sharding.jdbc.config.sharding.default-database-strategy.inline.algorithm-expression配置的分库的逻辑，根据id%2进行分。
• sharding.jdbc.config.sharding.tables.user.actual-data-nodes配置的是user表在真实数据库中的位置，ds_$->{0..1}.user_$->{0..4}表示 数据在ds_0和ds_1中的user_0、user_1、user_2、user_3、user_4中。
• sharding.jdbc.config.sharding.tables.user.table-strategy.inline.sharding-column，配置user表数据切分的字段
• sharding.jdbc.config.sharding.tables.user.table-strategy.inline.algorithm-expression=user_$->{id % 5}，配置user表数据切分的策略。
• sharding.jdbc.config.sharding.tables.user.key-generator-column-name=id 自动生成id。
• sharding.jdbc.config.props.sql.show=true  打印sql

 

项目结构

UserMapper.xml

    
        
        
        
        
        
        

    

    
        INSERT INTO user(uid,name,phone,email,sex,password) VALUES(#{uid},#{name},#{phone},#{email},#{sex},#{password})
    

    
        SELECT  * FROM  user
    

以下配置已经没什么好的价值，仅供参考

mabatis-config.xml

雪花算法生成主键，我也不知雪花算法的原理，随便在网上找了一个拿过发现可以用，就先凑合着用下 

package com.example.zdm_store_pay.model;


public class Test {
    //雪花算法生成主键
    /**
     * 起始的时间戳
     */
    private final static long START_STMP = 1480166465631L;

    /**
     * 每一部分占用的位数
     */
    private final static long SEQUENCE_BIT = 12; //序列号占用的位数
    private final static long MACHINE_BIT = 5;   //机器标识占用的位数
    private final static long DATACENTER_BIT = 5;//数据中心占用的位数

    /**
     * 每一部分的最大值
     */
    private final static long MAX_DATACENTER_NUM = -1L ^ (-1L << DATACENTER_BIT);
    private final static long MAX_MACHINE_NUM = -1L ^ (-1L << MACHINE_BIT);
    private final static long MAX_SEQUENCE = -1L ^ (-1L << SEQUENCE_BIT);

    /**
     * 每一部分向左的位移
     */
    private final static long MACHINE_LEFT = SEQUENCE_BIT;
    private final static long DATACENTER_LEFT = SEQUENCE_BIT + MACHINE_BIT;
    private final static long TIMESTMP_LEFT = DATACENTER_LEFT + DATACENTER_BIT;

    private long datacenterId;  //数据中心
    private long machineId;     //机器标识
    private long sequence = 0L; //序列号
    private long lastStmp = -1L;//上一次时间戳

    public Test(long datacenterId, long machineId) {
        if (datacenterId > MAX_DATACENTER_NUM || datacenterId < 0) {
            throw new IllegalArgumentException("datacenterId can't be greater than MAX_DATACENTER_NUM or less than 0");
        }
        if (machineId > MAX_MACHINE_NUM || machineId < 0) {
            throw new IllegalArgumentException("machineId can't be greater than MAX_MACHINE_NUM or less than 0");
        }
        this.datacenterId = datacenterId;
        this.machineId = machineId;
    }

    /**
     * 产生下一个ID
     *
     * @return
     */
    public synchronized long nextId() {
        long currStmp = getNewstmp();
        if (currStmp < lastStmp) {
            throw new RuntimeException("Clock moved backwards.  Refusing to generate id");
        }

        if (currStmp == lastStmp) {
            //相同毫秒内，序列号自增
            sequence = (sequence + 1) & MAX_SEQUENCE;
            //同一毫秒的序列数已经达到最大
            if (sequence == 0L) {
                currStmp = getNextMill();
            }
        } else {
            //不同毫秒内，序列号置为0
            sequence = 0L;
        }

        lastStmp = currStmp;

        return (currStmp - START_STMP) << TIMESTMP_LEFT //时间戳部分
                | datacenterId << DATACENTER_LEFT       //数据中心部分
                | machineId << MACHINE_LEFT             //机器标识部分
                | sequence;                             //序列号部分
    }

    private long getNextMill() {
        long mill = getNewstmp();
        while (mill <= lastStmp) {
            mill = getNewstmp();
        }
        return mill;
    }

    private long getNewstmp() {
        return System.currentTimeMillis();
    }

    public static void main(String[] args) {
        Test snowFlake = new Test(2, 3);

        long start = System.currentTimeMillis();
        for (int i = 0; i < 10; i++) {
            System.out.println(snowFlake.nextId());
        }

        System.out.println(System.currentTimeMillis() - start);


    }
}

UserController.java 

package com.example.zdm_store_pay.controller;

import com.example.zdm_store_pay.model.Test;
import com.example.zdm_store_pay.model.UserVo;
import com.example.zdm_store_pay.service.impl.UserService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Scope;
import org.springframework.web.bind.annotation.*;

import java.util.Date;
import java.util.Map;

@RestController
@Scope("prototype")
@RequestMapping("/pay")
public class PayController {
    @Autowired
    private UserService userService;
    @GetMapping("/getPayString")
    public String getString(){
        System.out.println("打印进入了geteway   路由转发  成功！ pay");
        return "geteway   路由转发  成功！ pay";
    }

    /***
     *
     */
    @RequestMapping(value = "/saves",method = RequestMethod.POST)
    public  Object saves() throws Exception {
        //Map resultMap = userServiceI.addUser(uid,name,phone);
        Test snowFlake = new Test(2, 3);
        for (int i = 0; i <10 ; i++) {
            UserVo user=new UserVo();
            user.setUid(snowFlake.nextId());
            user.setName("test"+i);
            user.setSex(i);
            user.setPhone("11111111"+i);
            user.setEmail("xxxxx");
            user.setPassword("eeeeeeeeeeee"+i);
            userService.addUser(user);
            System.out.println("执行完  次数 = "+i);
        }
        return null;
    }
    /**
     * 查询数据时会聚合
     *
     * */
    @RequestMapping(value = "/selectALL",method = RequestMethod.GET)
    public  Object selectALL() throws Exception {
        Map resultMap =userService.selectAll();

        return resultMap;
    }

}

User

package com.example.zdm_store_pay.model;

import java.util.Date;

public class UserVo {
    private Long uid;
    private String name;
    private String phone;
    private String email;
    private Integer sex;
    private String password;


    public Long getUid() {
        return uid;
    }

    public void setUid(Long uid) {
        this.uid = uid;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getPhone() {
        return phone;
    }

    public void setPhone(String phone) {
        this.phone = phone;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public String getPassword() {
        return password;
    }

    public void setPassword(String password) {
        this.password = password;
    }

    public Integer getSex() {
        return sex;
    }

    public void setSex(Integer sex) {
        this.sex = sex;
    }

}

UserServiceImpl

package com.example.zdm_store_pay.service.impl;

import com.example.zdm_store_pay.mapper.UserMapper;
import com.example.zdm_store_pay.model.UserVo;
import org.apache.catalina.User;
import org.springframework.stereotype.Service;

import javax.annotation.Resource;

@Service
public class UserServiceImpl implements UserService {
    @Resource
    UserMapper userRepository;
    @Override
    public Integer addUser(UserVo user) {
        return userRepository.addUser(user);
    }

    @Override
    public Map selectAll() {
        Map resultMap = new HashMap<>();
        resultMap.put("list",userRepository.selectAll());
        return resultMap;
    }
}

我踩的坑

Caused by: java.util.concurrent.ExecutionException: java.sql.SQLSyntaxErrorException: Unknown table 't_user' in information_schema
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader.loadActualTableMetaDataList(TableMetaDataLoader.java:126) ~[sharding-core-3.0.0.M2.jar:na]
	... 88 common frames omitted
Caused by: java.sql.SQLSyntaxErrorException: Unknown table 't_user' in information_schema
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.DatabaseMetaData$7.forEach(DatabaseMetaData.java:2980) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.DatabaseMetaData$7.forEach(DatabaseMetaData.java:2968) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.IterateBlock.doForAll(IterateBlock.java:56) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.DatabaseMetaData.getPrimaryKeys(DatabaseMetaData.java:3021) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader.getPrimaryKeys(TableMetaDataLoader.java:106) ~[sharding-core-3.0.0.M2.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader.getColumnMetaDataList(TableMetaDataLoader.java:93) ~[sharding-core-3.0.0.M2.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader.load(TableMetaDataLoader.java:82) ~[sharding-core-3.0.0.M2.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader.load(TableMetaDataLoader.java:75) ~[sharding-core-3.0.0.M2.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader.access$000(TableMetaDataLoader.java:49) ~[sharding-core-3.0.0.M2.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader$1.call(TableMetaDataLoader.java:121) ~[sharding-core-3.0.0.M2.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader$1.call(TableMetaDataLoader.java:117) ~[sharding-core-3.0.0.M2.jar:na]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]

Caused by: java.util.concurrent.ExecutionException: java.sql.SQLSyntaxErrorException: Unknown table 'columns_priv' in information_schema
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
	at io.shardingsphere.core.metadata.table.executor.TableMetaDataLoader.loadActualTableMetaDataList(TableMetaDataLoader.java:126) ~[sharding-core-3.0.0.M2.jar:na]
	... 88 common frames omitted

Caused by: com.mysql.cj.exceptions.InvalidConnectionAttributeException: The server time zone value '�й���׼ʱ��' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the serverTimezone configuration property) to use a more specifc time zone value if you want to utilize time zone support.
	at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]

这是我集成sharding-jdbc 踩坑时搜到的一些资料链接

https://blog.csdn.net/l1028386804/article/details/79368021

https://blog.csdn.net/qq_34936541/article/details/93816813

https://blog.csdn.net/forezp/article/details/94174114

西风颂推荐：https://blog.csdn.net/CSDN_526654341/article/details/99677348

小东哥 https://www.52lhd.com/

sharding-jdbc 按月份分表 https://www.jianshu.com/p/014d78a792fc

 

Sharding-jdbc设置defaultDatasource无效问题解决&源码分析思路 https://blog.csdn.net/wk52525/article/details/89506399#result

解决sharding-sphere强制扫描表结构的方法  http://cache.baiducontent.com/c?m=9d78d513d99212e819bac2690d67872d0e55f0743da4c7150e8fce5f92144c413071e2cb66784344859e203216ae391da8aa6132601421c78cc8ff5ddccbc3756fdf3035000b863605d269f88117728366cb07b5b81897adf04584afa2928507139c08592f97f1fb0d4a459d2db64f6cbfb2de084b504afcab7272fe2a6a34da6605a1478ea2693d5bc0ac8a005bc35d813c4e8afc75f66543eb44b9181e2742a14bb50b57232d&p=996cc71a85cc43ed46bd9b7d0d13c4&newp=9070c54ad5c344f91ea6c7710f4497231610db2151d4d501298ffe0cc4241a1a1a3aecbf22231504d3ce7f6001a9495be0f43d723d0034f1f689df08d2ecce7e79ce6c&user=baidu&fm=sc&query=Caused+by%3A+io%2Eshardingsphere%2Ecore%2Eexception%2EShardingException&qid=9fa23f56000c0b36&p1=5

shardingjdbc文章https://blog.csdn.net/forezp/article/list/2?

这是我微信可方便一起探讨  Beenoneself