表按date或datetime类型字段进行分区,对于新的数据,需要我们手动添加分区,和删除旧的分区,维护工作量太大。所以Apache Doris启动了一个后台进程,默认每隔1小时(由fe.conf的dynamic_partition_check_interval_seconds
参数设置)根据当前的日期时间来添加分区和删除分区
动态分区只支持Range分区
表创建于2021-11-05 22:30:00,星期五
mysql>
mysql> create table dynamic_partition(
-> user_id bigint,
-> partition_datetime datetime,
-> send_msg varchar(512)
-> )
-> unique key(user_id, partition_datetime)
-> partition by range(partition_datetime) ()
-> distributed by hash(user_id)
-> properties
-> (
-> 'dynamic_partition.enable' = 'true',
-> 'dynamic_partition.time_unit' = 'week',
-> 'dynamic_partition.start' = '-1',
-> 'dynamic_partition.end' = '2',
-> 'dynamic_partition.prefix' = 'p_week',
-> 'dynamic_partition.buckets' = '10'
-> );
Query OK, 0 rows affected (0.10 sec)
mysql>
mysql> show create table dynamic_partition;
......省略部分......
PARTITION p_week2021_53 VALUES [('2021-12-27 00:00:00'), ('2022-01-03 00:00:00')),
PARTITION p_week2022_02 VALUES [('2022-01-03 00:00:00'), ('2022-01-10 00:00:00')),
......省略部分......
1 row in set (0.00 sec)
mysql>
建表语句执行完的分区结果如下:
PARTITION p_week2021_45 VALUES [('2021-11-01 00:00:00'), ('2021-11-08 00:00:00')),
PARTITION p_week2021_46 VALUES [('2021-11-08 00:00:00'), ('2021-11-15 00:00:00')),
PARTITION p_week2021_47 VALUES [('2021-11-15 00:00:00'), ('2021-11-22 00:00:00'))
查看分区的调度情况
mysql> show dynamic partition tables;
+-------------------+--------+----------+-------+------+--------+---------+----------------+-------------------------+---------+---------------------+---------------------+--------+------------------------+----------------------+------------------------+
| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | ReplicationNum | ReplicaAllocation | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg | ReservedHistoryPeriods |
+-------------------+--------+----------+-------+------+--------+---------+----------------+-------------------------+---------+---------------------+---------------------+--------+------------------------+----------------------+------------------------+
| dynamic_partition | true | WEEK | -1 | 2 | p_week | 10 | 3 | tag.location.default: 3 | MONDAY | 2021-11-05 22:30:00 | 2021-11-05 22:30:00 | NORMAL | NULL | NULL | NULL |
+-------------------+--------+----------+-------+------+--------+---------+----------------+-------------------------+---------+---------------------+---------------------+--------+------------------------+----------------------+------------------------+
1 row in set (0.01 sec)
mysql>
字段说明如下:
查看表的分区
mysql> show partitions from dynamic_partition;
dynamic_partition.start_day_of_week
:取值范围为1-7,默认为1,表示周一为每周的起始点dynamic_partition.start_day_of_month
:取值范围为1-28,默认为1,表示1号为每月的起始点下面是历史分区的参数:
dynamic_partition.create_history_partition:默认为false,即创建表的时候不会创建历史分区,只会创建当前和未来的分区。当置为true时,Doris会自动创建所有分区,当期望创建的分区个数大于max_dynamic_partition_num(默认500)值时,操作将被禁止。当不指定start属性时,该参数不生效
dynamic_partition.history_partition_num:该参数用于指定创建历史分区数量。默认值为-1, 不进行设置,以start属性为准。如果设置了则以当前值为准,但必须小于等于start属性的绝对值
dynamic_partition.hot_partition_num:指定当前和最近的历史分区共多少个分区为热分区,但未来的分区都是热分区。对于热分区,系统会自动设置其storage_medium参数为SSD,并且设置storage_cooldown_time
dynamic_partition.reserved_history_periods:需要保留的历史分区的时间范围,不会被后台的定时任务删除。默认为"NULL"。当dynamic_partition.time_unit设置为"DAY/WEEK/MONTH"时,需要以[yyyy-MM-dd,yyyy-MM-dd],[…,…]格式进行设置,例如'dynamic_partition.reserved_history_periods' = '[2022-06-01,2022-06-15],[2022-07-01,2022-07-15]'
。当dynamic_partition.time_unit设置为"HOUR"时,需要以[yyyy-MM-dd HH:mm:ss,yyyyMM-dd HH:mm:ss],[…,…]的格式来进行设置,例如'reserved_history_periods' = '[2022-07-01 00:00:00,2022-07-01 03:00:00]'
定时检测分区是否符合创建或删除的条件,如果符合,则创建或删除。默认时间为1小时(3600秒)。修改方式有以下几种:
admin set frontend config('dynamic_partition_check_interval_seconds' = '7200');
mysql>
mysql> alter table dynamic_partition set
-> (
-> 'dynamic_partition.time_unit' = 'month',
-> 'dynamic_partition.start' = '-5',
-> 'dynamic_partition.prefix' = 'p_month'
-> );
Query OK, 0 rows affected (0.06 sec)
mysql>
修改后的表分区情况如下:
PARTITION p_week2021_45 VALUES [('2021-11-01 00:00:00'), ('2021-11-08 00:00:00')),
PARTITION p_week2021_46 VALUES [('2021-11-08 00:00:00'), ('2021-11-15 00:00:00')),
PARTITION p_week2021_47 VALUES [('2021-11-15 00:00:00'), ('2021-11-22 00:00:00')),
PARTITION p_month202112 VALUES [('2021-12-01 00:00:00'), ('2022-01-01 00:00:00')),
PARTITION p_month202201 VALUES [('2022-01-01 00:00:00'), ('2022-02-01 00:00:00'))
一个表要么为动态分区表,要么为手动分区表
alter table dynamic_partition set
设置动态分区表的属性,来转化为动态分区表。注意:如果设置了dynamic_partition.start属性,则在分区单位之前的分区将会被自动删除alter table dynamic_partition set ('dynamic_partition.enable' = 'false');
,来转化为手动分区表动态分区使用过程中,如果因为一些意外情况导致dynamic_partition.start和 dynamic_partition.end之间的某些分区丢失,那么当前时间与dynamic_partition.end之间的丢失分区会被重新创建,dynamic_partition.start与当前时间之间的丢失分区不会重新创建