PostgreSQL大表快速添加包含not null和default的字段
一、前言
在需求的不断迭代中，表字段也会增加，有时候在大表增加的字段中，存在包含not null和default的字段，这时候添加表字段就执行相当慢，因为PostgreSQL把表数据全部重写，参考：https://my.oschina.net/Kenyon/blog/99757；
现提供一种修改系统表的方式来快速在大表中增加表字段，避免数据重写，参考：https://my.oschina.net/Suregogo/blog/1605598
二、3个相关的系统表
1. pg_class: 记录所有relation的记录；
- relnatts： 表中用户列的数目（系统列不计算在内）
其他参考： http://www.postgres.cn/docs/9.5/catalog-pg-class.html
2. pg_attribute: 记录表字段的记录
- attrelid： 所属表的oid；
- attname： 字段名
- attnum： 列的编号；
- attnotnull：是否非空约束；
- atthasdef： 是否有默认值；
其他参考： http://www.postgres.cn/docs/9.5/catalog-pg-attribute.html
3. pg_attrdef： 记录字段的默认值信息
- oid： 字段的oid；
- adrelid： 所属表的oid；
- adsrc： 默认值的可视化表示；
其他参考： http://www.postgres.cn/docs/9.5/catalog-pg-attrdef.html
三、测试
1. 创建测试表
CREATE TABLE public.address
(
sid varchar primary key,
country character varying,
province character varying,
detail character varying,
state integer NOT NULL DEFAULT 0
)
2. 假如需求增加一个字段a，默认值是0，如下：
alter table address add column a int default 0;
如果表很大，这句执行语句很慢；
3. 替代方案
3.1 思路
在系统表中参考已经存在的相同类型的字段state，在pg_attribute表中增加一条记录，在pg_class的字段数加1，如有默认值，在pg_attrdef增加一条记录；
3.2 查看当前系统表；
pg_class 表
postgres=# select * from pg_class where relname='address';
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode | reltablespace | relpages | reltuples | relallvisible | reltoastrelid | relhasindex | relisshared | relpersistence | relkind | relnatts | relchecks | relhasoids | relhaspkey | relhasrules | relhastriggers | relhassubclass | relrowsecurity | relforcerowsecurity | relispopulated | relreplident | relfrozenxid | relminmxid | relacl | reloptions
---------+--------------+---------+-----------+----------+-------+-------------+---------------+----------+-----------+---------------+---------------+-------------+-------------+----------------+---------+----------+-----------+------------+------------+-------------+----------------+----------------+----------------+---------------------+----------------+--------------+--------------+------------+--------+------------
address | 2200 | 669725 | 0 | 10 | 0 | 669723 | 0 | 0 | 0 | 0 | 669727 | t | f | p | r | 5 | 0 | f | t | f | f | f | f | f | t | d | 58977486 | 1 | |
(1 row)
pg_attribute 表
postgres=# select * from pg_attribute where attrelid ='address'::regclass order by attnum desc;
attrelid | attname | atttypid | attstattarget | attlen | attnum | attndims | attcacheoff | atttypmod | attbyval | attstorage | attalign | attnotnull | atthasdef | attisdropped | attislocal | attinhcount | attcollation | attacl | attoptions | attfdwoptions
----------+----------+----------+---------------+--------+--------+----------+-------------+-----------+----------+------------+----------+------------+-----------+--------------+------------+-------------+--------------+--------+------------+---------------
669723 | state | 23 | -1 | 4 | 5 | 0 | -1 | -1 | t | p | i | t | t | f | t | 0 | 0 | | |
669723 | detail | 1043 | -1 | -1 | 4 | 0 | -1 | -1 | f | x | i | f | f | f | t | 0 | 100 | | |
669723 | province | 1043 | -1 | -1 | 3 | 0 | -1 | -1 | f | x | i | f | f | f | t | 0 | 100 | | |
669723 | country | 1043 | -1 | -1 | 2 | 0 | -1 | -1 | f | x | i | f | f | f | t | 0 | 100 | | |
669723 | sid | 1043 | -1 | -1 | 1 | 0 | -1 | -1 | f | x | i | t | f | f | t | 0 | 100 | | |
669723 | ctid | 27 | 0 | 6 | -1 | 0 | -1 | -1 | f | p | s | t | f | f | t | 0 | 0 | | |
669723 | xmin | 28 | 0 | 4 | -3 | 0 | -1 | -1 | t | p | i | t | f | f | t | 0 | 0 | | |
669723 | cmin | 29 | 0 | 4 | -4 | 0 | -1 | -1 | t | p | i | t | f | f | t | 0 | 0 | | |
669723 | xmax | 28 | 0 | 4 | -5 | 0 | -1 | -1 | t | p | i | t | f | f | t | 0 | 0 | | |
669723 | cmax | 29 | 0 | 4 | -6 | 0 | -1 | -1 | t | p | i | t | f | f | t | 0 | 0 | | |
669723 | tableoid | 26 | 0 | 4 | -7 | 0 | -1 | -1 | t | p | i | t | f | f | t | 0 | 0 | | |
(11 rows)
pg_attrdef 表
postgres=# select * from pg_attrdef where adrelid ='address'::regclass;
adrelid | adnum | adbin | adsrc
---------+-------+------------------------------------------------------------------------------------------------------------------------------------------------------+-------
669723 | 5 | {CONST :consttype 23 :consttypmod -1 :constcollid 0 :constlen 4 :constbyval true :constisnull false :location 183 :constvalue 4 [ 0 0 0 0 0 0 0 0 ]} | 0
(1 row)
postgres=#
这里attnum=5，指的就是state字段，adsrc就是默认值0；
3.3. 增加字段
- 参考state在pg_attribute中加一个记录，表示字段a；
insert into pg_attribute (
attrelid,attname,atttypid,attstattarget,attlen,attnum,attndims,attcacheoff,atttypmod,attbyval,attstorage,attalign,attnotnull,atthasdef,attisdropped,attislocal,attinhcount,attcollation,attacl,attoptions,attfdwoptions
)
select
attrelid,'a',atttypid,attstattarget,attlen,attnum+1,attndims,attcacheoff,atttypmod,attbyval,attstorage,attalign,'f','t',attisdropped,attislocal,attinhcount,attcollation,attacl,attoptions,attfdwoptions
from pg_attribute where attrelid='address'::regclass and attname='state';
- pg_class字段数量加1
update pg_class set relnatts=relnatts+1 where relname='address';
- pg_attrdef 增加一条记录，表示添加缺省值
with t as(select max(attnum) as maxAttNum from pg_attribute where attrelid='address'::regclass)
insert into pg_attrdef(adrelid,adnum,adbin,adsrc) select adrelid,maxAttNum,adbin,adsrc from pg_attrdef,t where adrelid='address'::regclass and adnum=(select attnum from pg_attribute where attrelid ='address'::regclass and attname='state');
3.4 查看表结构
postgres=# \d+ address
Table "public.address"
Column | Type | Modifiers | Storage | Stats target | Description
----------+-------------------+--------------------+----------+--------------+-------------
sid | character varying | not null | extended | |
country | character varying | | extended | |
province | character varying | | extended | |
detail | character varying | | extended | |
state | integer | not null default 0 | plain | |
a | integer | default 0 | plain | |
Indexes:
"address_pkey" PRIMARY KEY, btree (sid)
可以看到字段a已经加上；
四、编写Function添加字段
4.1 函数
create or replace function func_fast_add_column(tableName varchar,referenceColumn varchar,newColumnName varchar,isNotNull boolean,hasDefaultValue boolean) returns void as $$
--快速向大表中加字段
--tableName：表名
--referenceColumn：新字段的类型，默认值等参考的字段
--newColumnName：新字段名称
--isNotNull：是否非空
--hasDefaultValue：是否有默认值
declare
currentAttNum int; --当前表的字段最大序号
begin
select max(attnum) into currentAttNum from pg_attribute where attrelid=tableName::regclass;
--1. 添加字段属性 --attnotnull 表示是否非空
insert into pg_attribute (attrelid,attname,atttypid,attstattarget,attlen,attnum,attndims,attcacheoff,atttypmod,attbyval,attstorage,attalign,attnotnull,atthasdef,attisdropped,attislocal,attinhcount,attcollation,attacl,attoptions,attfdwoptions )
select attrelid,newColumnName,atttypid,attstattarget,attlen,currentAttNum+1,attndims,attcacheoff,atttypmod,attbyval,attstorage,attalign,isNotNull,hasDefaultValue,attisdropped,attislocal,attinhcount,attcollation,attacl,attoptions,attfdwoptions from pg_attribute where attrelid=tableName::regclass and attname=referenceColumn;
--2. 修改pg_class字段个数
update pg_class set relnatts=relnatts+1 where relname=tableName;
--3. 添加缺省值 adnum:列号
if(hasDefaultValue) then
insert into pg_attrdef(adrelid,adnum,adbin,adsrc) select adrelid,currentAttNum+1,adbin,adsrc from pg_attrdef where adrelid=tableName::regclass and adnum=(select attnum from pg_attribute where attrelid =tableName::regclass and attname=referenceColumn);
end if;
end
$$language plpgsql;
4.2 添加字段b
alter table address column b int not null default 0;
或者
select func_fast_add_column('address','state','b',true,true);
4.3 查看表结构
postgres=# \d+ address
Table "public.address"
Column | Type | Modifiers | Storage | Stats target | Description
----------+-------------------+--------------------+----------+--------------+-------------
sid | character varying | not null | extended | |
country | character varying | | extended | |
province | character varying | | extended | |
detail | character varying | | extended | |
state | integer | not null default 0 | plain | |
a | integer | default 0 | plain | |
b | integer | not null default 0 | plain | |
Indexes:
"address_pkey" PRIMARY KEY, btree (sid)
五、风险
与正常添加字段后有啥区别，有啥风险尚不明确；