版本:
MySQL-5.7.32
前言:
对于业务繁忙的数据库来说,在运行了一定时间后,往往会产生一些数据量较大的表,特别是对于每天新增数据较多的日志表或者流水表,大表对于日常的运维非常的不方便,特别是数据的清理、迁移,表的访问性能也会随着数据量的增大而受到影响,因此,对于大表我们需要进行优化拆分,通常拆分的方案有
所以,通常选择分区表改造方案的主要原因都是为了避免应用层面的改造,对应用层面透明,以及方便日常的运维,前提是表具备改造分区条件。
改造分区前期条件:
参考文档
MySQL官方文档
1 根据业务的场景以及数据的分布,确认是否有匹配的分区表类型以及分区列
2 应用涉及的sql,需要90%以上的操作都包含分区列,按分区操作,如果sql没有包含分区条件,扫描全分区,性能会出现下降。
查询表sql操作历史
select db,query,exec_count
from sys.x$statement_analysis
where lower(query) like '%%'order by exec_count;
3 主键必须包含分区键
4 分区键表达式只支持部分函数,存储过程,不支持二级制操作符以及/
5 分区不支持外键
查询表外键
select *
from information_schema.KEY_COLUMN_USAGE
where constraint_schema ='' and REFERENCED_TABLE_SCHEMA is not null\G
6 不支持查询缓存
7 5.7版本单个表分区最大支持8192个,并且会话第一次访问分区表,都需要打开全部的分区表,所以避免建立过多的分区
8 数据库最大文件打开数open_files_limit要设置足够大以满足表,分区的打开数量
9 数据库大文件large_files_support设置为on
10 分区列支持null值(对于rang分区,null值总小于任何的一个非null值,即存放在最左边的分区;对于list分区需要显示指定null值条件),但从数据管理以及规范来看,不建议分区列存放null值,并且如果表有主键,则分区列不能为null值,因为分区列需要作为主键的一部分,不能为null
12 MySQL目前没有自动分区功能,所以需要监控分区的使用情况,通过job自动或者定时手动添加新分区
13 确定数据保留期限,定期归档分区数据
分区改造案例:
以下是一张数据量为766万的大表xxxx_user. xxxx_tab,计划将其改造为范围分区,按月存放。
select table_schema,table_name,table_rows,data_length/1024/1024/1024
from information_schema.tables
where table_name='xxxx_tab';
表结构
CREATE TABLE `xxxx_tab` (
`ROLE_SEQ` bigint(20) NOT NULL ,
`PRD_ID` varchar(64) NOT NULL ,
`MAKE_RIGHT` varchar(1) DEFAULT '0' ,
`CHECK_RIGHT` varchar(1) DEFAULT '0' ,
`AUTH_RIGHT` varchar(1) DEFAULT '0' ,
`AUTH_GROUP` varchar(4) DEFAULT NULL ,
`RELEASE_RIGHT` varchar(1) DEFAULT '0' ,
`CREATE_USER_SEQ` bigint(20) DEFAULT NULL ,
`CREATE_DEPT_SEQ` bigint(20) DEFAULT NULL ,
`CREATE_TIME` datetime DEFAULT NULL ,
`UPDATE_USER_SEQ` bigint(20) DEFAULT NULL ,
`UPDATE_DEPT_SEQ` bigint(20) DEFAULT NULL ,
`UPDATE_TIME` datetime DEFAULT NULL ,
PRIMARY KEY (`PRD_ID`,`ROLE_SEQ`),
KEY `xxxx_tab_IDX01` (`ROLE_SEQ`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
分区列create_time日期最大值,最小值,根据这个范围按月创建分区
select max(CREATE_TIME),min(CREATE_TIME)
from xxxx_tab;
分区列null值,对于存在的null值,需要应用对null数据进行处理,并且程序上需要确保数据写入not null
select count(*)
from xxxx_tab
where create_time is null;
主键重建添加分区列
alter table xxxx_tab drop primary key,add primary key (`PRD_ID`,`ROLE_SEQ`,`CREATE_TIME`);
由于主键没有包含分区列,需要重建主键添加分区列,对于主键重建我采取的是官方的online ddl功能,这种ddl操作会造成主从延时,但是不会产生大量的binlog,对于主从实时性要求高的,可以采用第三方的在线工具pt-osc,gh-ost
表转化为分区表
采用pt-osc在线将表转化为分区表,对于partition by 官方是不支持online ddl的,所以需要采用第三方的在线工具
./pt-online-schema-change --user=xxx --password=xxx --charset=utf8 D=xxxx_user,t=xxxx_tab --alter "PARTITION BY RANGE COLUMNS(CREATE_TIME)
(PARTITION p200001 VALUES LESS THAN ('2000-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p200101 VALUES LESS THAN ('2001-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p201707 VALUES LESS THAN ('2017-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p201708 VALUES LESS THAN ('2017-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p201709 VALUES LESS THAN ('2017-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p201710 VALUES LESS THAN ('2017-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p201711 VALUES LESS THAN ('2017-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p201712 VALUES LESS THAN ('2018-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p201801 VALUES LESS THAN ('2018-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p201802 VALUES LESS THAN ('2018-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p201803 VALUES LESS THAN ('2018-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p201804 VALUES LESS THAN ('2018-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p201805 VALUES LESS THAN ('2018-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p201806 VALUES LESS THAN ('2018-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p201807 VALUES LESS THAN ('2018-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p201808 VALUES LESS THAN ('2018-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p201809 VALUES LESS THAN ('2018-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p201810 VALUES LESS THAN ('2018-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p201811 VALUES LESS THAN ('2018-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p201812 VALUES LESS THAN ('2019-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p201901 VALUES LESS THAN ('2019-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p201902 VALUES LESS THAN ('2019-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p201903 VALUES LESS THAN ('2019-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p201904 VALUES LESS THAN ('2019-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p201905 VALUES LESS THAN ('2019-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p201906 VALUES LESS THAN ('2019-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p201907 VALUES LESS THAN ('2019-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p201908 VALUES LESS THAN ('2019-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p201909 VALUES LESS THAN ('2019-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p201910 VALUES LESS THAN ('2019-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p201911 VALUES LESS THAN ('2019-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p201912 VALUES LESS THAN ('2020-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202001 VALUES LESS THAN ('2020-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202002 VALUES LESS THAN ('2020-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202003 VALUES LESS THAN ('2020-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202004 VALUES LESS THAN ('2020-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202005 VALUES LESS THAN ('2020-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202006 VALUES LESS THAN ('2020-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202007 VALUES LESS THAN ('2020-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202008 VALUES LESS THAN ('2020-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202009 VALUES LESS THAN ('2020-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202010 VALUES LESS THAN ('2020-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202011 VALUES LESS THAN ('2020-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202012 VALUES LESS THAN ('2021-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202101 VALUES LESS THAN ('2021-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202102 VALUES LESS THAN ('2021-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202103 VALUES LESS THAN ('2021-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202104 VALUES LESS THAN ('2021-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202105 VALUES LESS THAN ('2021-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202106 VALUES LESS THAN ('2021-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202107 VALUES LESS THAN ('2021-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202108 VALUES LESS THAN ('2021-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202109 VALUES LESS THAN ('2021-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202110 VALUES LESS THAN ('2021-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202111 VALUES LESS THAN ('2021-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202112 VALUES LESS THAN ('2022-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202201 VALUES LESS THAN ('2022-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202202 VALUES LESS THAN ('2022-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202203 VALUES LESS THAN ('2022-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202204 VALUES LESS THAN ('2022-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202205 VALUES LESS THAN ('2022-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202206 VALUES LESS THAN ('2022-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202207 VALUES LESS THAN ('2022-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202208 VALUES LESS THAN ('2022-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202209 VALUES LESS THAN ('2022-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202210 VALUES LESS THAN ('2022-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202211 VALUES LESS THAN ('2022-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202212 VALUES LESS THAN ('2023-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202301 VALUES LESS THAN ('2023-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202302 VALUES LESS THAN ('2023-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202303 VALUES LESS THAN ('2023-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202304 VALUES LESS THAN ('2023-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202305 VALUES LESS THAN ('2023-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202306 VALUES LESS THAN ('2023-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202307 VALUES LESS THAN ('2023-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202308 VALUES LESS THAN ('2023-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202309 VALUES LESS THAN ('2023-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202310 VALUES LESS THAN ('2023-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202311 VALUES LESS THAN ('2023-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202312 VALUES LESS THAN ('2024-01-01 00:00:00') ENGINE = InnoDB,
PARTITION pmax VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB)" --recursion-method hosts --max-lag 600 --nodrop-old-table --print --statistics --execute
分区后表模型
CREATE TABLE `xxxx_tab` (
`ROLE_SEQ` bigint(20) NOT NULL ,
`PRD_ID` varchar(64) NOT NULL ,
`MAKE_RIGHT` varchar(1) DEFAULT '0' ,
`CHECK_RIGHT` varchar(1) DEFAULT '0' ,
`AUTH_RIGHT` varchar(1) DEFAULT '0' ,
`AUTH_GROUP` varchar(4) DEFAULT NULL ,
`RELEASE_RIGHT` varchar(1) DEFAULT '0' ,
`CREATE_USER_SEQ` bigint(20) DEFAULT NULL ,
`CREATE_DEPT_SEQ` bigint(20) DEFAULT NULL ,
`CREATE_TIME` datetime NOT NULL ,
`UPDATE_USER_SEQ` bigint(20) DEFAULT NULL ,
`UPDATE_DEPT_SEQ` bigint(20) DEFAULT NULL ,
`UPDATE_TIME` datetime DEFAULT NULL ,
PRIMARY KEY (`PRD_ID`,`ROLE_SEQ`,`CREATE_TIME`),
KEY `xxxx_tab_IDX01` (`ROLE_SEQ`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
PARTITION BY RANGE COLUMNS(CREATE_TIME)
(PARTITION p200001 VALUES LESS THAN ('2000-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p200101 VALUES LESS THAN ('2001-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p201707 VALUES LESS THAN ('2017-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p201708 VALUES LESS THAN ('2017-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p201709 VALUES LESS THAN ('2017-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p201710 VALUES LESS THAN ('2017-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p201711 VALUES LESS THAN ('2017-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p201712 VALUES LESS THAN ('2018-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p201801 VALUES LESS THAN ('2018-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p201802 VALUES LESS THAN ('2018-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p201803 VALUES LESS THAN ('2018-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p201804 VALUES LESS THAN ('2018-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p201805 VALUES LESS THAN ('2018-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p201806 VALUES LESS THAN ('2018-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p201807 VALUES LESS THAN ('2018-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p201808 VALUES LESS THAN ('2018-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p201809 VALUES LESS THAN ('2018-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p201810 VALUES LESS THAN ('2018-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p201811 VALUES LESS THAN ('2018-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p201812 VALUES LESS THAN ('2019-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p201901 VALUES LESS THAN ('2019-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p201902 VALUES LESS THAN ('2019-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p201903 VALUES LESS THAN ('2019-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p201904 VALUES LESS THAN ('2019-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p201905 VALUES LESS THAN ('2019-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p201906 VALUES LESS THAN ('2019-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p201907 VALUES LESS THAN ('2019-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p201908 VALUES LESS THAN ('2019-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p201909 VALUES LESS THAN ('2019-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p201910 VALUES LESS THAN ('2019-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p201911 VALUES LESS THAN ('2019-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p201912 VALUES LESS THAN ('2020-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202001 VALUES LESS THAN ('2020-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202002 VALUES LESS THAN ('2020-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202003 VALUES LESS THAN ('2020-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202004 VALUES LESS THAN ('2020-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202005 VALUES LESS THAN ('2020-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202006 VALUES LESS THAN ('2020-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202007 VALUES LESS THAN ('2020-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202008 VALUES LESS THAN ('2020-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202009 VALUES LESS THAN ('2020-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202010 VALUES LESS THAN ('2020-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202011 VALUES LESS THAN ('2020-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202012 VALUES LESS THAN ('2021-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202101 VALUES LESS THAN ('2021-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202102 VALUES LESS THAN ('2021-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202103 VALUES LESS THAN ('2021-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202104 VALUES LESS THAN ('2021-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202105 VALUES LESS THAN ('2021-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202106 VALUES LESS THAN ('2021-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202107 VALUES LESS THAN ('2021-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202108 VALUES LESS THAN ('2021-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202109 VALUES LESS THAN ('2021-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202110 VALUES LESS THAN ('2021-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202111 VALUES LESS THAN ('2021-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202112 VALUES LESS THAN ('2022-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202201 VALUES LESS THAN ('2022-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202202 VALUES LESS THAN ('2022-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202203 VALUES LESS THAN ('2022-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202204 VALUES LESS THAN ('2022-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202205 VALUES LESS THAN ('2022-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202206 VALUES LESS THAN ('2022-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202207 VALUES LESS THAN ('2022-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202208 VALUES LESS THAN ('2022-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202209 VALUES LESS THAN ('2022-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202210 VALUES LESS THAN ('2022-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202211 VALUES LESS THAN ('2022-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202212 VALUES LESS THAN ('2023-01-01 00:00:00') ENGINE = InnoDB,
PARTITION p202301 VALUES LESS THAN ('2023-02-01 00:00:00') ENGINE = InnoDB,
PARTITION p202302 VALUES LESS THAN ('2023-03-01 00:00:00') ENGINE = InnoDB,
PARTITION p202303 VALUES LESS THAN ('2023-04-01 00:00:00') ENGINE = InnoDB,
PARTITION p202304 VALUES LESS THAN ('2023-05-01 00:00:00') ENGINE = InnoDB,
PARTITION p202305 VALUES LESS THAN ('2023-06-01 00:00:00') ENGINE = InnoDB,
PARTITION p202306 VALUES LESS THAN ('2023-07-01 00:00:00') ENGINE = InnoDB,
PARTITION p202307 VALUES LESS THAN ('2023-08-01 00:00:00') ENGINE = InnoDB,
PARTITION p202308 VALUES LESS THAN ('2023-09-01 00:00:00') ENGINE = InnoDB,
PARTITION p202309 VALUES LESS THAN ('2023-10-01 00:00:00') ENGINE = InnoDB,
PARTITION p202310 VALUES LESS THAN ('2023-11-01 00:00:00') ENGINE = InnoDB,
PARTITION p202311 VALUES LESS THAN ('2023-12-01 00:00:00') ENGINE = InnoDB,
PARTITION p202312 VALUES LESS THAN ('2024-01-01 00:00:00') ENGINE = InnoDB,
PARTITION pmax VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB)