将存放在一台数据库服务器中的数据,按照特定方式进行拆分,
分散存放到多台数据库服务器中,以达到分散单台服务器负载的效果
水平分割(横向切分)
将表记录按行切分,分散存储到多个数据库服务器表中。
垂直分割(纵向切分)
按业务类型分类,将存储在一个库里的表分别存储到不同的数据库器里
(提供数据分库分表存储的服务软件(中间件))
mycat是基于Java的分布式数据库系统中间件,为高并发环境的分布式存储提供解决方案 ;
适合数据大量写入的存储需求;
支持MySQL、Oracle、Sqlserver、Mongodb等;
提供数据读写分离服务;
提供数据分片服务;
基于阿里巴巴Cobar进行研发的开源软件。
1) 枚举法 sharding-by-intfile
2)固定分片 rule1
3)范围约定 auto-sharding-long
4)求模法 mod-long
5)日期列分区法 sharding-by-date
6)通配取模 sharding-by-pattern
7)ASCII码求模通配 sharding-by-prefixpattern
8)编程指定 sharding-by-substring
9)字符串拆分hash解析 sharding-by-stringhash
10)一致性hash sharding-by-murmur
**分片服务器的工作过程;**当mycat收到一个SQL命令时
第一步:解析SQL命令涉及到的表
第二步:根据表使用分片规则的计算结果(看对表的配置,如果有分片规则,则获取SQL命令里分片字段的值,并匹配分片函数,获得分片列表)
第三步:将SQL命令发往对应的数据库服务器去执行
第四步:最后收集和处理所有分片结果数据,并返回到客户端
[root@maxscale56 ~]# yum -y install java-1.8.0-openjdk.x86_64
[root@maxscale56 ~]# tar -xf Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz
[root@maxscale56 ~]# mv mycat/ /usr/local/
[root@maxscale56 ~]# ls /usr/local/mycat/
bin catlet conf lib logs version.txt
]# ls /usr/local/mycat
bin # mycat命令
catlet # 扩展功能
conf # 配置文件
.txt 和 ..properties 结尾的是 分片规则配置文件
.xml 结尾的是 mycat服务配置文件
lib # mycat使用的jar包
logs # mycat启动日志和运行日志
version.txt # mycat软件 说明文件
XML 扩展标记语言 的格式特点 有开始有结束 标签有单标签和双 标签
注释符号 可以注释1行 也可以注释多行重要配置文件说明:
server.xml //设置连接账号及逻辑库
schema.xml //设置数据分片存储的表
rule.xml //分片规则
其他文件 //分片规则配置文件
#定义客户端连接56主机时使用的用户名密码 及 逻辑库名(虚拟库名)
host56]# vim /usr/local/mycat/conf/server.xml (使用默认配置即可)
<user name="root"> 用户名
<property name="password">123456</property> 密码
<property name="schemas">TESTDB</property> 虚拟库名
</user>
<user name="user"> 用户名
<property name="password">user</property> 密码
<property name="schemas">TESTDB</property> 虚拟库名
<property name="readOnly">true</property> 只读访问权限
</user>
:wq
设置分片存储数据的表 schema.xml 文件格式说明
]# vim /usr/local/mycat/conf/schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
#定义分片存储数据的表
<schema ....>
<table name="表名" dataNode="数据库要存储在几台服务器里" rule="分片规则名" / >
<table ...>
....
</table>
</schema>
#定义数据库服务器主机名
<dataNode name="第几台数据库服务器" dataHost="主机名" database="库名" />
#定义数据库服务器ip地址
<dataHost ..... >
......
</dataHost>
</mycat:schema>
修改前schema.xml的准备工作
]# cp /usr/local/mycat/conf/schema.xml /root/ #备份源文件
[root@maxscale56 ~]# sed -i '56,77d' /usr/local/mycat/conf/schema.xml #删除无用的行
[root@maxscale56 ~]# sed -i '39,42d' /usr/local/mycat/conf/schema.xml #删除无用的行
修改schema.xml文件的如下内容
]#vim /usr/local/mycat/conf/schema.xml
<table name="goods" primaryKey="ID" type="global" dataNode="dn1,dn2,dn3" />
<table name="employee" primaryKey="ID" dataNode="dn1,dn2,dn3" rule="sharding-by-intfile" />
<table name="customer" primaryKey="ID" dataNode="dn1,dn2,dn3" rule="sharding-by-intfile">
#定义3台数据库服务器的主机名
<dataNode name="dn1" dataHost="mysql53" database="db1" /> 第一台
<dataNode name="dn2" dataHost="mysql54" database="db2" /> 第二台
<dataNode name="dn3" dataHost="mysql55" database="db3" /> 第三台
#指定名称叫mysql53主机的ip地址
<dataHost name="mysql53" ...... >
......
<writeHost host="hostM1" url="192.168.4.53:3306" user="zhu" password="123456">
</writeHost>
</dataHost>
#指定名称叫mysql54主机的ip地址
<dataHost name="mysql54" ...... >
......
<writeHost host="hostM2" url="192.168.4.54:3306" user="zhu" password="123456">
</writeHost>
</dataHost>
#指定名称叫mysql55主机的ip地址
<dataHost name="mysql55" ...... >
......
<writeHost host="hostM3" url="192.168.4.55:3306" user="zhu" password="123456">
</writeHost>
</dataHost>
:wq
schema.xml 配置后的内容
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
<schema name="TESTDB" checkSQLschema="false" sqlMaxLimit="100">
<table name="travelrecord" dataNode="dn1,dn2,dn3" rule="auto-sharding-long" />
<table name="company" primaryKey="ID" type="global" dataNode="dn1,dn2,dn3" />
<table name="goods" primaryKey="ID" type="global" dataNode="dn1,dn2,dn3" />
<table name="hotnews" primaryKey="ID" autoIncrement="true" dataNode="dn1,dn2,dn3"
rule="mod-long" />
<table name="employee" primaryKey="ID" dataNode="dn1,dn2,dn3"
rule="sharding-by-intfile" />
<table name="customer" primaryKey="ID" dataNode="dn1,dn2,dn3"
rule="sharding-by-intfile">
<childTable name="orders" primaryKey="ID" joinKey="customer_id"
parentKey="id">
<childTable name="order_items" joinKey="order_id"
parentKey="id" />
</childTable>
<childTable name="customer_addr" primaryKey="ID" joinKey="customer_id"
parentKey="id" />
</table>
</schema>
<dataNode name="dn1" dataHost="mysql53" database="db1" />
<dataNode name="dn2" dataHost="mysql54" database="db2" />
<dataNode name="dn3" dataHost="mysql55" database="db3" />
<dataHost name="mysql53" maxCon="1000" minCon="10" balance="0"
writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
<heartbeat>select user()</heartbeat>
<writeHost host="hostM1" url="192.168.4.53:3306" user="zhu" password="123456">
</writeHost>
</dataHost>
<dataHost name="mysql54" maxCon="1000" minCon="10" balance="0"
writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
<heartbeat>select user()</heartbeat>
<writeHost host="hostM2" url="192.168.4.54:3306" user="zhu" password="123456">
</writeHost>
</dataHost>
<dataHost name="mysql55" maxCon="1000" minCon="10" balance="0"
writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
<heartbeat>select user()</heartbeat>
<writeHost host="hostM3" url="192.168.4.55:3306" user="zhu" password="123456">
</writeHost>
</dataHost>
</mycat:schema>
添加用户zhu:
[root@host55 mysql]# mysql -uroot -p123456 -e 'grant all on *.* to zhu@"%" identified by "123456"'
[root@host54 mysql]# mysql -uroot -p123456 -e 'grant all on *.* to zhu@"%" identified by "123456"'
[root@host53mysql]# mysql -uroot -p123456 -e 'grant all on *.* to zhu@"%" identified by "123456"'
#创建存储数据的库
[root@host53 ~]# mysql -uroot -p123456 -e 'create database db1'
[root@host54 ~]# mysql -uroot -p123456 -e 'create database db2'
[root@host55 ~]# mysql -uroot -p123456 -e 'create database db3'
1)验证数据库服务器的授权用户
[root@host56 ~]# which mysql || yum -y install mariadb
[root@host56 ~]# mysql -h192.168.4.53 -uzhu -p123456
[root@host56 ~]# mysql -h192.168.4.54 -uzhu -p123456
[root@host56 ~]# mysql -h192.168.4.55 -uzhu -p123456
2) 定义主机名
[root@host56 ~]# hostname host56
[root@host56 ~]# echo host56 > /etc/hostname
3) 56虚拟机的内存不能小于1G
[root@host56 mycat]# /usr/local/mycat/bin/mycat start
Starting Mycat-server...
[root@host56 mycat]#
查看日志 文件
[root@host56 mycat]# ls /usr/local/mycat/logs/
mycat.log mycat.pid wrapper.log
[root@host56 mycat]#
查看端口号
[root@host56 mycat]# netstat -utnlp | grep 8066
tcp6 0 0 :::8066 :::* LISTEN 1808/java
[root@host56 mycat]# /usr/local/mycat/bin/mycat stop
Stopping Mycat-server...
Stopped Mycat-server.
[root@host56 mycat]# netstat -utnlp | grep 8066 服务停了端口没有了
[root@host56 mycat]# ls /usr/local/mycat/logs/ pid 号文件也没了
mycat.log wrapper.log
[root@host56 mycat]#
查看日志文件获取报错信息
vim /usr/local/mycat/logs/wrapper.log
在客户端连接mycat服务器查看库和表
[root@host50 ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456
mysql> show databases;
+----------+
| DATABASE |
+----------+
| TESTDB |
+----------+
1 row in set (0.00 sec)
mysql> use TESTDB;
Database changed
mysql>
mysql> show tables;
+------------------+
| Tables in TESTDB |
+------------------+
| company |
| customer |
| customer_addr |
| employee |
| goods |
| hotnews |
| orders |
| order_items |
| travelrecord |
+------------------+
9 rows in set (0.00 sec)
mysql>
#这些表示不存在的 ,要存储数据 需要根据 表使用的分片规则建表后 才能存储数据
mysql> select * from goods;
ERROR 1146 (42S02): Table 'db3.goods' doesn't exist
mysql>
mysql> exit;
Bye
常用分片规则的工作过程(支持的分片规则共有10种)
(1) 枚举分片规则 (sharding-by-intfile) :
分片字段的值 必须在 分片规则配置文件定义的值里 选择。
(2) 求模分片规则 (mod-long):
根据分片字段值 与 指定数字的取余计算结果 存储数据
mod-long分片规则工作过程: 根据分片字段值与设定的数字求模结果存储数据
当余数是 0 数据存储到 dn1
当余数是 1 数据存储到 dn2
当余数是 2 数据存储到 dn3
配置思路:
1 确定使用分片规则的表叫什么名字名
(查看schema.xml 文件里 )
2 确定分片字段的表头名
(查看rule.xml文件里的 )
3 确定分片规则使用的配置文件,定义分片字段的值
(查看rule.xml文件里的 )
4 修改分片规则的配置文件,定义分片字段的值
5 重启mycat服务
6 根据分片规则建表存储数据
7 在数据库服务器本机查看存储的数据
验证sharding-by-intfile工作过程
1.找使用sharding-by-intfile分片规则的表名
]# vim /usr/local/mycat/conf/schem.xml
<table name="employee" primaryKey="ID" dataNode="dn1,dn2,dn3" rule="sharding-by-intfile" />
2.创建employee分片字段名 sharding_id
]# vim /usr/local/mycat/conf/rule.xml
<tableRule name="sharding-by-intfile"> 分片规则名
<rule>
<columns>sharding_id</columns> 分片字段名
<algorithm>hash-int</algorithm> 算法名 (存储数据的计算规则)
</rule>
</tableRule>
3.找到sharding-by-intfile分片规则配置文件partition-hash-int.txt
<function name="hash-int" 算法名
class="io.mycat.route.function.PartitionByFileMap"> 算法调用的执行程序
<property name="mapFile">partition-hash-int.txt</property> 配置文件名
</function>
4.编辑 sharding-by-intfile分片规则配置文件partition-hash-int.txt 设置 分片字段的值
host56]# vim /usr/local/mycat/conf/partition-hash-int.txt
分片字段值=第几台数据库服务器
10000=0 # 0对应dn1 mysql53 192.168.4.53 db1
10010=1 # 1对应dn2 mysql54 192.168.4.54 db2
10020=2 # 2对应dn3 mysql54 192.168.4.54 db3
:wq
[root@host56 conf]# /usr/local/mycat/bin/mycat restart
Starting Mycat-server...
[root@host56 conf]#
6.根据分片规则建表并存储数据
[root@host50 ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456 #连接mycat服务
mysql> show databases;
+----------+
| DATABASE |
+----------+
| TESTDB |
+----------+
1 row in set (0.01 sec)
mysql> use TESTDB; 建表或存储数据都有进入到 TESTDB库下执行
mysql> show tables;
+------------------+
| Tables in TESTDB |
+------------------+
| company |
| customer |
| customer_addr |
| employee |
| goods |
| hotnews |
| orders |
| order_items |
| travelrecord |
+------------------+
9 rows in set (0.00 sec)
mysql> create table employee(
-> sharding_id int ,
-> name char(10), addr char(20) );
Query OK, 0 rows affected (1.34 sec)
mysql> insert into employee(sharding_id,name,addr) values(10000,"bob","bj");
Query OK, 1 row affected (0.05 sec)
mysql> insert into employee(sharding_id,name,addr) values(10000,"bob2","bj");
Query OK, 1 row affected (0.06 sec)
mysql> insert into employee(sharding_id,name,addr) values(10000,"bob3","bj");
Query OK, 1 row affected (0.03 sec)
mysql> insert into employee(sharding_id,name,addr) values(10010,"tom","bj");
Query OK, 1 row affected (0.04 sec)
mysql> insert into employee(sharding_id,name,addr) values(10020,"ABC","bj");
Query OK, 1 row affected (0.03 sec)
mysql> insert into employee(sharding_id,name,addr) values(10020,"ABC","bj");
mysql> insert into employee(sharding_id,name,addr)
values(10040,"tom","bj");
ERROR 1064 (HY000): can't find any valid datanode :EMPLOYEE -> SHARDING_ID -> 10040
超出分片字段配置文件指报错。
mysql> select * from empoyee;
在数据库服务器本机查看数据
[root@host53 ~]# mysql -uroot -p123456 -e 'select * from db1.employee'
[root@host54 ~]# mysql -uroot -p123456 -e 'select * from db2.employee'
[root@host55 ~]# mysql -uroot -p123456 -e 'select * from db3.employee'
3、例子
3.1 创建表
必须根据表使用的分片规则建表,具体操作如下:
1.确定使用分片规则的表叫什么名字名 (查看schema.xml 文件里
)
[root@maxscale56 ~]# vim /usr/local/mycat/conf/schema.xml
<table name="hotnews" primaryKey="ID" autoIncrement="true" dataNode="dn1,dn2,dn3"
rule="mod-long" />
2.确定分片字段的表头名 (查看rule.xml文件里的 )
[root@maxscale56 ~]# vim /usr/local/mycat/conf/rule.xml
<tableRule name="mod-long"> #分片规则名
<rule>
<columns>id</columns> #分片字段名
<algorithm>mod-long</algorithm> #算法
</rule>
</tableRule>
3.确定分片规则使用的配置文件,定义分片字段的值 (查看rule.xml文件里的 )
[root@maxscale56 ~]# vim /usr/local/mycat/conf/rule.xml
<function name="mod-long" class="io.mycat.route.function.PartitionByMod">
<!-- how many data nodes -->
<property name="count">3</property> #定义分片字段 做求模计算的数字(是数据库服务器的台数)
</function>
4.重启mycat服务 (配置文件没有做过修改 ,服务不需要重启)
5.创建表并存储数据
[root@host50 ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456
mysql> use TESTDB;
mysql> create table hotnews(id int , title char(30),comment varchar(150) , worker char(3) );
mysql> insert into hotnews(id,title,comment,worker)values(9,"a","a","a");
mysql> insert into hotnews(id,title,comment,worker)values(10,"b","a","a");
mysql> insert into hotnews(id,title,comment,worker)values(11,"b","a","a");
mysql> insert into hotnews(id,title,comment,worker)values(0,"b","a","a");
mysql> insert into hotnews(id,title,comment,worker)values(1000,"d","a","a");
mysql> select * from hotnews;
+------+-------+---------+--------+
| id | title | comment | worker |
+------+-------+---------+--------+
| 9 | a | a | a |
| 0 | d | a | a |
| 11 | c | a | a |
| 10 | b | a | a |
| 1000 | d | a | a |
+------+-------+---------+--------+
5 rows in set (0.00 sec)
6.在数据库服务器查看数据
[root@host53 ~]# mysql -uroot -p123456 -e 'select * from db1.hotnews'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+-------+---------+--------+
| id | title | comment | worker |
+------+-------+---------+--------+
| 9 | a | a | a |
| 0 | d | a | a |
+------+-------+---------+--------+
[root@host53 ~]#
[root@host54 ~]# mysql -uroot -p123456 -e 'select * from db2.hotnews'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+-------+---------+--------+
| id | title | comment | worker |
+------+-------+---------+--------+
| 10 | b | a | a |
| 1000 | d | a | a |
+------+-------+---------+--------+
[root@host54 ~]#
[root@host55 ~]# mysql -uroot -p123456 -e 'select * from db3.hotnews'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+-------+---------+--------+
| id | title | comment | worker |
+------+-------+---------+--------+
| 11 | c | a | a |
+------+-------+---------+--------+
[root@host55 ~]#
3.2 添加新库新表
在56主机添加新库GAMEDB 新表名 user 和 salary
注意在 schema.xml 文件里 表名必须唯一 不可以重复
1.定义库名
[root@host56 conf]# vim /usr/local/mycat/conf/server.xml
<user name="root">
<property name="password">123456</property>
<property name="schemas">TESTDB,GAMEDB</property> #定义库名
</user>
<user name="user">
<property name="password">user</property>
<property name="schemas">TESTDB,GAMEDB</property>
<property name="readOnly">true</property>
</user>
:wq
2.定义表名
]# vim /usr/local/mycat/conf/schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/"> #在此行下方添加如下行 其他行不能动
<schema name="GAMEDB" checkSQLschema="false" sqlMaxLimit="100"> #定义库名
<table name="user" dataNode="dn1,dn2,dn3" rule="mod-long" /> #定义表名
<table name="salary" dataNode="dn1,dn2,dn3" type="global" /> #定义表名
</schema>
........
:wq
3.重启服务
[root@host56 conf]# /usr/local/mycat/bin/mycat restart
[root@host56 conf]# netstat -utnlp | grep 8066
tcp6 0 0 :::8066 :::* LISTEN 3301/java
[root@host56 conf]#
4.测试新库新表的配置
[root@host50 ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456
mysql> show databases;
+----------+
| DATABASE |
+----------+
| GAMEDB |
| TESTDB |
+----------+
mysql> use GAMEDB;
mysql> show tables;
+------------------+
| Tables in GAMEDB |
+------------------+
| salary |
| user |
+------------------+
3.3 向新添加的表里存储数据,验证分库分表存储
具体操作如下:
1.创建user表并存储数据
[root@maxscale56 ~]# vi /usr/local/mycat/conf/schema.xml
<table name="user" dataNode="dn1,dn2,dn3" rule="mod-long" />
#使用mod-long分片规则的表,必须有分片字段表头名必须叫id
[root@host50 ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456
mysql》 use GAMEDB;
mysql> create table user( id int , username char(10) , password char(6));
mysql> insert into user(id,username,password) values(17,"a","123456"),(17,"b","654321"),(17,"c","121123");
mysql> insert into user(id,username,password) values(4,"a","123456"),(4,"b","654321"),(4,"c","121123");
mysql> insert into user(id,username,password) values(9,"a","123456"),(9,"b","654321"),(9,"c","121123");
#在数据库服务器本机查看数据
[root@host53 ~]# mysql -uroot -p123456 -e 'select * from db1.user'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+----------+----------+
| id | username | password |
+------+----------+----------+
| 9 | a | 123456 |
| 9 | b | 654321 |
| 9 | c | 121123 |
+------+----------+----------+
[root@host53 ~]#
[root@host54 ~]# mysql -uroot -p123456 -e 'select * from db2.user'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+----------+----------+
| id | username | password |
+------+----------+----------+
| 4 | a | 123456 |
| 4 | b | 654321 |
| 4 | c | 121123 |
+------+----------+----------+
[root@host54 ~]#
[root@host55 ~]# mysql -uroot -p123456 -e 'select * from db3.user'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+----------+----------+
| id | username | password |
+------+----------+----------+
| 17 | a | 123456 |
| 17 | b | 654321 |
| 17 | c | 121123 |
+------+----------+----------+
[root@host55 ~]#
2.没有使用分片规则 所以建表时,表头名没有限制
mysql> create table salary(name char(10),pay int);
mysql> insert into salary (name,pay) values("a",10);
mysql> insert into salary (name,pay) values("b",20);
mysql> select * from salary;
+------+------+
| name | pay |
+------+------+
| a | 10 |
| b | 20 |
+------+------+
2 rows in set (0.09 sec)
3.在数据库服务器本机查看数据
[root@host53 ~]# mysql -uroot -p123456 -e 'select * from db1.salary'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+------+
| name | pay |
+------+------+
| a | 10 |
| b | 20 |
+------+------+
[root@host52 ~]# mysql -uroot -p123456 -e 'select * from db2.salary'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+------+
| name | pay |
+------+------+
| a | 10 |
| b | 20 |
+------+------+
[root@host55 ~]# mysql -uroot -p123456 -e 'select * from db3.salary'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+------+
| name | pay |
+------+------+
| a | 10 |
| b | 20 |
+------+------+
3.在数据库服务器本机查看数据
```shell
[root@host53 ~]# mysql -uroot -p123456 -e 'select * from db1.salary'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+------+
| name | pay |
+------+------+
| a | 10 |
| b | 20 |
+------+------+
[root@host52 ~]# mysql -uroot -p123456 -e 'select * from db2.salary'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+------+
| name | pay |
+------+------+
| a | 10 |
| b | 20 |
+------+------+
[root@host55 ~]# mysql -uroot -p123456 -e 'select * from db3.salary'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+------+
| name | pay |
+------+------+
| a | 10 |
| b | 20 |
+------+------+
你可能感兴趣的:(12,RDBMS关系型数据库管理系统,数据库)