在线重定义操作步骤

Oracle有个存储过程，通过在线重定义，可以实现业务表，与临时表进行转换，并且不影响业务的情况下(实际还是存在影响)。

本篇文档根据操作后，进行精简，重点描述操作步骤，减少操作遇到问题的可能性。

一 实施流程

1）前期调研，查询表及相关对象大小，临时表分区创建语法，后期分区表相关索引创建及维护测试；

2）提前创建临时表，并使用在线重定义进行同步操作；

3）与业务人员沟通确认后，最后一次同步，并且完成在线重定义操作；

 

二 具体操作步骤

2.1 前期调研

检查是否能够重定义，当前表无主键，使用rowid
EXEC DBMS_REDEFINITION.CAN_REDEF_TABLE('owner','table_name',DBMS_REDEFINITION.cons_use_rowid);
--如果存在主键，不需要第三个参数
EXEC DBMS_REDEFINITION.CAN_REDEF_TABLE('xxx','xxx');
查询是否存在引用表的其它对象 
SQL> select * from DBA_DEPENDENCIES where REFERENCED_OWNER='xxx' and REFERENCED_NAME='xxx';
no rows selected

查询是否存在单独授权的grant语句 
SQL>select GRANTEE,OWNER,TABLE_NAME,GRANTOR,PRIVILEGE from dba_tab_privs where OWNER='xxx' and TABLE_NAME='xxx';
no rows selected
--备份,重定义后,重新授权
--select 'grant '||PRIVILEGE||' on '||OWNER||'.'||TABLE_NAME||' to '||GRANTEE||';' as "grant_text" from dba_tab_privs
 where OWNER='xxx' and TABLE_NAME='xxx';

查询表属性
SQL>select owner,table_name,compression,compress_for from dba_tables where owner='xx' and table_name='xxx';
SQL>select table_owner,table_name,partition_name,compression,compress_for from dba_tab_partitions where
 table_owner='xxx' and table_name='xxx';

表大小
select owner,segment_name,segment_type,round(sum(bytes)/1024/1024/1024,2) G,tablespace_name from dba_segments
 where segment_name='xxx' group by owner,segment_name,segment_type,tablespace_name;

主键约束
SQL> select constraint_name,table_name,constraint_type from dba_constraints where owner='BSP' and TABLE_NAME
 in('xxx') and constraint_type='P';
SQL>  SELECT * FROM DBA_CONS_COLUMNS WHERE OWNER='xxx' AND CONSTRAINT_NAME='xxx';

表约束信息
SQL> select OWNER,CONSTRAINT_NAME,CONSTRAINT_TYPE,TABLE_NAME,R_OWNER,R_CONSTRAINT_NAME,STATUS,INDEX_OWNER,INDEX_NAME
 from dba_constraints where TABLE_NAME='xxx' and owner='xxx';

查询数据最小值
SQL> select to_char(min(REPORT_TIME),'yyyy-mm-dd') as "date" from xxxx.xxxx;

表的注释信息
SQL>select * from dba_col_comments where owner='BSP' AND TABLE_NAME='table&';

表的索引信息
SQL> SELECT INDEX_OWNER,INDEX_NAME,TABLE_OWNER,TABLE_NAME,COLUMN_NAME,COLUMN_POSITION FROM DBA_IND_COLUMNS
 WHERE TABLE_OWNER='xxx' and TABLE_NAME='xxx' order by 2,5;

SQL> select dbms_metadata.get_ddl('INDEX','xxx','owner') ddl_text from dual;

备份操作
nohup expdp \'/ as sysdba\' directory=dump dumpfile=xxxxx%u.dmp logfile=xxx.log TABLES=xxxP.xxxx
  COMPRESSION=DATA_ONLY CLUSTER=N PARALLEL=4 &

2.2 创建临时表

与开发人员沟通TIME时间字段，每天一个分区，使用11g新特性间隔范围分区
 CREATE TABLE "owner"."temp_table"
   (    "ID"  NOT NULL ENABLE   ) 
        SEGMENT CREATION IMMEDIATE
 COMPRESS LOGGING
  TABLESPACE "xxxx"
  PARTITION BY RANGE (xxx_TIME) INTERVAL (NUMTODSINTERVAL(1, 'day'))
(partition part_t1 values less than(to_date('2018-09-27', 'yyyy-mm-dd')))

2.3 初步同步数据

如果存在主键/唯一约束/索引，一定要删掉！ 因为初步同步数据后，默认创建索引，默认并行度1，非常慢，且需要占用大量temp空间，报错需要重来。
sqlplus / as sysdba <<EOF
set timing on
alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss';
select sysdate from dual;
BEGIN
DBMS_REDEFINITION.START_REDEF_TABLE (
   uname         => 'owner',
   orig_table    => '业务表',
   int_table     => '临时表',
   options_flag  => dbms_redefinition.cons_use_rowid);   --如果是主键，删掉这行参数
END;
/
exit
EOF
[oracle@vplusdb1 ~]$ nohup sh dbms_init.sh &
第一次全量同步，使用Insert append方式写入，500G大表，使用BASIC压缩属性，3小时不到写入完毕，压缩后的数据200G

2.4 增量同步数据(多次)

创建索引

!#/bin/bash
export PATH
export ORACLE_HOME=xxx
sqlplus / as sysdba <<EOF
set timing on
alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss';
select sysdate from dual;
ALTER SESSION  FORCE PARALLEL DDL parallel 4;
alter session set workarea_size_policy = manual;
alter session set sort_area_size=1024000000;
alter session set db_file_multiblock_read_count= 128;
alter session set "_sort_multiblock_read_count"= 128;
alter session set current_schema=xx;
set timing on time on ;
CREATE INDEX "xx"."xx" ON "xx"."xxx" ("Vxx_ID", "Rxx_TIME" DESC) LOCAL  tablespace BSP parallel 8;
alter index "xx"."xxx" parallel 1;
exec dbms_stats.gather_table_stats(ownname => 'xx',tabname => 'xxx',estimate_percent => 0.1,degree => 16,
granularity => 'ALL',cascade => true,no_invalidate => false);    --注意，对临时表收集统计信息
 exit;
 EOF 

 

（小表，几十G，可以忽略本篇的很多内容，因为数据量少，一些特性可以忽略，但是对于几百G的大表，建议多关注文字说明，都是坑)
一定要对临时表创建索引，特别是主键索引或者唯一索引，其次是复合索引，创建索引后，收集统计信息使用no_invalidate => false
(没必要所有的索引都创建，目的是临时表存在主键或者唯一，最次复合索引，再增量同步数据的过程当中，能够根据索引进行nest loop执行计划进行同步操作，
而非HASH JOIN两个大表进行关联，再实际操作中，500G业务表，与200G临时表，使用HASH JOIN 增量同步，导致占用200G TEMP临时表空间，同步效率异常缓慢，
最后创建主键索引，收集统计信息后，再次进行增量同步，执行计划走索引nest looop）
 *增量同步过程中，添加临时表空间，如果SQL有占用temp，会导致增量同步报错终止，再次增量同步就好了
BEGIN
DBMS_REDEFINITION.SYNC_INTERIM_TABLE (
uname       => 'owner', 
orig_table  => '业务表',
int_table   => '临时表');
END;
/

参考收集统计信息参数
https://blog.csdn.net/cpongo3/article/details/88800429

2.5 复制依赖对象

不复制索引、不忽略错误、不复制统计信息。
SET SERVEROUTPUT ON
DECLARE
     error_count pls_integer := 0;
BEGIN
DBMS_REDEFINITION.COPY_TABLE_DEPENDENTS(
   uname               => 'owner',
   orig_table          => '业务表',
   int_table            => '临时表',
   copy_indexes         => 0,
   copy_triggers        => TRUE,
   copy_constraints     => FALSE,
   copy_privileges      => TRUE,
   ignore_errors        => FALSE,
   num_errors           => error_count,
   copy_statistics      => FALSE); 
DBMS_OUTPUT.PUT_LINE('errors := ' || TO_CHAR(error_count));
END;
/ 

 

2.6 异常处理

删除结束之前的重定义过程
官方文档说明
Aborting Online Table Redefinition and Cleaning Up After Errors
In the event that an error is raised during the redefinition process, or if you choose to terminate the redefinition process, 
call ABORT_REDEF_TABLE. This procedure drops temporary logs and tables associated with the redefinition process. 
After this procedure is called, you can drop the interim table and its dependent objects.

If the online redefinition process must be restarted, if you do not first call ABORT_REDEF_TABLE, 
subsequent attempts to redefine the table will fail.
SQL>EXEC DBMS_REDEFINITION.ABORT_REDEF_TABLE('owner','业务表','临时表'); 

 

2.7 监控并调整

SQL>select * from table(dbms_xplan.display_cursor('sqL_id&'))；
SQL>select a.TEMP_SPACE_ALLOCATED/1024/1024/1024,to_char(a.SAMPLE_TIME,'yyyy-mm-dd hh24:mi') as "date",a.inst_id,
a.SESSION_ID,a.SESSION_SERIAL#,b.username,a.event,a.sql_id,a.MACHINE
 from gv$active_session_history a,dba_users b
 where a.user_id=b.user_id and a.SAMPLE_TIME>sysdate-1 and a.sql_id='sqL_id&'
  order by 2,1;
SQL> select file_name,file_id,tablespace_name,bytes/1024/1024/1024,maxbytes/1024/1024/1024 from dba_temp_files;

版本(11.2.0.4)本次使用sql moniter进行监控sql执行进度！！！

SQL>  select INST_ID,sid,serial#,USERNAME,STATUS,MACHINE,SQL_ID,EVENT,(sysdate-LOGON_TIME)*86400 as "s",LAST_CALL_ET from gv$session where sid=xxx and SERIAL#=xxx;

SET LONG 1000000
SET LONGCHUNKSIZE 1000000
SET LINESIZE 1000
SET PAGESIZE 0
SET TRIM ON
SET TRIMSPOOL ON
SET ECHO OFF
SET FEEDBACK OFF
SELECT DBMS_SQLTUNE.report_sql_monitor(sql_id => 'xxx', type => 'TEXT') AS report FROM dual;

查询需要增量数据的类型(dml）以及行数

SQL> select DMLTYPE$$,count(*) from ower.mlog$_业务表 group by DMLTYPE$$ ;(表名称很长通过新建对象获取一下，物化视图日志）

 

2.8 最后一次同步

（上述操作均可以提前操作，但是最后一步的操作，需要和业务确认，最好没有业务或者业务数据量少的时候操作)

正常情况下，执行次操作，同步完成。
EXEC DBMS_REDEFINITION.FINISH_REDEF_TABLE('owner','业务表','临时表');

 

3.1 检测

  1.应用验证功能没有问题；
  2.验证数据没有问题；
  3.验证表约束是否OK
  4.查询表大小对比, 旧表480G 新表208G    近50%压缩比例
  5.索引状态及是否分区
  6.表的压缩属性
  7.表的注释信息同步
  8.临时表空间扩容的数据文件进行删除回收
  9.DB 归档空间，表空间使用情况

SQL> select OWNER,CONSTRAINT_NAME,CONSTRAINT_TYPE,TABLE_NAME,STATUS,INDEX_NAME from DBA_constraints
 where owner='xx' and table_name in('xx','xx');
select owner,segment_name,segment_type,round(sum(bytes)/1024/1024/1024,2) g,TABLESPACE_NAME,count(*)
 from dba_segments where owner='xx' AND SEGMENT_NAME in ('xx','xx')
 group by owner,segment_name,segment_type,TABLESPACE_NAME ;
OWNER  SEGMENT_NAME                     SEGMENT_TYPE                G TABLESPACE  COUNT(*)
------ -------------------------------- ------------------ ---------- ----------------

select owner,segment_name,segment_type,partition_name,round((bytes)/1024/1024,2) MBYTES,TABLESPACE_NAME
 from dba_segments where owner='xxx' AND SEGMENT_NAME in ('xxx')
 order by 5;
OWNER      SEGMENT_NAME                SEGMENT_TYPE       PARTITION_NAME                     MBYTES 
---------- --------------------------- ------------------ ------------------------------ ---------- 

SQL> select table_owner,table_name,partition_name,high_value,tablespace_name from dba_tab_partitions
 where table_owner='xxx' and table_name='xxx' and partition_name='PART_T1'
 PARTITION_NAME         HIGH_VALUE                                                     
----------------------------------------------------------------------------------------------------- 
PART_T1    TO_DATE(' 2018-09-27 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA 

SQL>select OWNER,INDEX_NAME,TABLE_OWNER,TABLE_NAME,STATUS from dba_indexes where table_owner='xx' AND TABLE_NAME='xxx';
OWNER   INDEX_NAME                     TABLE_NAME                     STATUS
------- ------------------------------ ------------- ------------------------------ ----------
IDX_TB_VEHICLE_COM_2NEW        xxx N/A
PK_TB_VEHICLE_COMPONENT_NO_NEW xxx VALID
IDX_TB_VEHICLE_COM_1NEW        xxx N/A
IDX_TB_VEHICLE_COM_3NEW        xxx N/A

SQL>select index_name,STATUS,TABLESPACE_NAME,count(*) from dba_ind_partitions where index_owner='xx'
 and index_name in('xxx','xx','xxx') group by index_name,STATUS,TABLESPACE_NAME;
INDEX_NAME                     STATUS     TABLESPACE_NAME   COUNT(*)
------------------------------ ---------- --------------------------
IDX_TB_VEHICLE_COM_1NEW        USABLE    513
IDX_TB_VEHICLE_COM_3NEW        USABLE    513
IDX_TB_VEHICLE_COM_2NEW        USABLE    513

SQL> select owner,table_name,COMPRESSION,COMPRESS_FOR from dba_tables where owner='xx' and table_name='xxx';
OWNER                          TABLE_NAME                     COMPRESS COMPRESS_FOR
------------------------------ ------------------------------ -------- ------------

SQL> SELECT TABLE_OWNER,TABLE_NAME,COMPRESSION,COMPRESS_FOR,count(*) FROM DBA_TAB_PARTITIONS
 WHERE TABLE_OWNER='xxx' AND TABLE_NAME='xxx' group by TABLE_OWNER,TABLE_NAME,COMPRESSION,COMPRESS_FOR;
--PARTITION_NAME
TABLE_NAME                     COMPRESS COMPRESS_FOR   COUNT(*)
------------------------------ -------- ------------ ----------
xxx                            ENABLED  BASIC               513

select 'comment on column '||owner||'.xxx.'||COLUMN_NAME||' is '''||COMMENTS||''';' as "sql"
 from dba_col_comments where table_name='xxx' and COMMENTS is not null and owner='xxx';
sql----------------------------------------------------------------------
comment on column xx.xxx.ID is 'ID';

SQL> select OBJECT_NAME,OBJECT_ID,DATA_OBJECT_ID,OBJECT_TYPE,STATUS from
 user_objects where created>sysdate-5 and OBJECT_TYPE not in('INDEX PARTITION','INDEX','LOB','TABLE','TABLE PARTITION')
no rows selected

 

3.2 再上述重定义完成后，业务也并未反馈异常，清理历史数据

1）对单个分区进行expdp导出备份

nohup expdp \'/ as sysdba\' directory=dump dumpfile=xx%u.dmp logfile=xxx.log TABLES=owner.table:part_name
COMPRESSION=DATA_ONLY CLUSTER=N PARALLEL=4 &
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 63.05 GB
······
. . exported   37.05 GB 455139131 rows
Master table "SYS"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYS.SYS_EXPORT_TABLE_01 is:
Job "SYS"."SYS_EXPORT_TABLE_01" successfully completed at Tue Nov 26 17:42:07 2019 elapsed 0 00:27:34
使用导出的备份，导出sql，观察如果需要恢复，是记录全表还是只是单纯一个分区信息
 impdp \'/ as sysdba\' directory=dump dumpfile=xxx%u.dmp logfile=xxx.log sqlfile=part_sql.log
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/COMMENT
Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX
Processing object type TABLE_EXPORT/TABLE/INDEX/FUNCTIONAL_INDEX/INDEX
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/FUNCTIONAL_INDEX/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Job "SYS"."SYS_SQL_FILE_FULL_01" successfully completed at Tue Nov 26 21:38:59 2019 elapsed 0 00:01:07
-- new object type path: TABLE_EXPORT/TABLE/TABLE
CREATE TABLE "xxx"."xxx"xxxxxx
 COMPRESS BASIC 
  PARTITION BY RANGE ("xxx") INTERVAL (NUMTODSINTERVAL(1, 'DAY')) TRANSITION ("PART_T1") 
 (PARTITION "PART_T1"  VALUES LESS THAN (TO_DATE(' 2018-09-27 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 
'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION IMMEDIATE 
 PARTITION "SYS_P26401"  VALUES LESS THAN (TO_DATE(' 2018-09-28 00:00:00', 'SYYYY-MM-DD HH24:MI:SS',
 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION IMMEDIATE 
·····  从上述信息，可以发现导出单个分区(从统计的blocks大小可以得到百分百是对分区数据进行导出)但是表结构是全表
2）删除前，检查查询
SQL>select table_owner,table_name,partition_name,high_value,tablespace_name from dba_tab_partitions
 where table_owner='xxx' and table_name='xxx' and partition_name='PART_T1';
TABLE_OWNE  TABLE_NAME               PARTITION_NAME              HIGH_VALUE                        TABLESPACE_NAME
-------------------------------------------------------------------------------------------------------------------------------
 PART_T1      TO_DATE(' 2018-09-27 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA BSP

XXX表    备份并清理xxx<20181001的数据

3）删除操作
vi 20191126_truncate.sh
#!/bin/bash
sqlplus / as sysdba <<EOF 
alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss';
select sysdate from dual;
set timin on 
alter table xx.xxx truncate partition PART_T1 update global indexes;
exit;
EOF 
Elapsed: 01:47:28.50
https://www.cnblogs.com/lvcha001/p/11930072.html


SQL> select sql_text from v$sql where sql_id='723qvrhgxzavn';
SQL_TEXT
--------------------------------------------------------------------------------

alter table xx.xxx truncate partition PART_T1 update global indexes;

-- select distinct round(a.TEMP_SPACE_ALLOCATED/1024/1024/1024,1) as "used_g",
to_char(a.SAMPLE_TIME,'yyyy-mm-dd hh24:mi') as "date",a.inst_id,a.SESSION_ID,
a.SESSION_SERIAL#,b.username,a.event,a.sql_id,a.MACHINE from gv$active_session_history a,
dba_users b where a.user_id=b.user_id and a.SAMPLE_TIME>to_date('20191126 11','yyyymmdd hh24')
 and a.sql_id='723qvrhgxzavn' group by a.TEMP_SPACE_ALLOCATED/1024/1024/1024,to_char(a.SAMPLE_TIME,'yyyy-mm-dd hh24:mi'),
 a.inst_id,a.SESSION_ID,a.SESSION_SERIAL#,b.username,a.event,a.sql_id,a.MACHINE order by 2,1; 
used_g date EVENT SQL_ID 

---------- ---------------- - -------------------------- ------------------- 

0 2019-11-26 18:06 723qvrhgxzavn 

23.8 2019-11-26 18:44 723qvrhgxzavn 

26.1 2019-11-26 19:01 db file sequential read 723qvrhgxzavn 

26.1 2019-11-26 19:01 gc current grant 2-way 723qvrhgxzavn 

26.1 2019-11-26 19:01 direct path read temp 723qvrhgxzavn --占用26g temp 

4）确认 重建完毕。。。查询分区数据信息，发现存在心得错误数据产生！挺好 
SQL> select count(*) from xxx partition(PART_T1); 
COUNT(*)
 ---------- 

38 
SQL>select OWNER,INDEX_NAME,TABLE_OWNER,TABLE_NAME,STATUS from dba_indexes
 where table_owner='xxx' AND TABLE_NAME='xxx'; 
OWNER INDEX_NAME TABLE_OWNER TABLE_NAME STATUS 
------------------------------ ------------------------------ ------------
IDX_TB_VEHICLE_COM_2NEW N/A 
PK_TB_VEHICLE_COMPONENT_NO_NEW VALID 
IDX_TB_VEHICLE_COM_1NEW N/A 
IDX_TB_VEHICLE_COM_3NEW N/A 
select owner,segment_name,segment_type,round(sum(bytes)/1024/1024/1024,2) g,TABLESPACE_NAME,count(*)
 from dba_segments where owner='xx' AND SEGMENT_NAME 
in ('xx','xx') group by owner,segment_name,segment_type,TABLESPACE_NAME ; 
----------------------------------------------------------------------------------- ---------- 
 TABLE            479.141
 TABLE PARTITION 146.91  516 --删除历史分区前，释放了60G数据 
 在线重定义之前得表，
后续进行drop table xxx; or drop table xx purge;