3)dw sqlldr
1@@@@example03 a bit of advanced
guide:
In this example, aim to express how to speed up your sql*loader.
~ disable contstraint,
~ rebuild index after loading data,
~ use position key word
~ loading data to a partition of sales, do not impact other partitions.
~ use direct path loading without logging, speed up but danger.
@@@
@@@<1>check the index, create tbs for new partition of table sh.sales.
@@@
@@@In this case, I use table sales of "sh" schema.
SH@ocp> select index_name, index_type,compression from user_indexes where table_name='SALES';
INDEX_NAME INDEX_TYPE COMPRESS
------------------------------ --------------------------- --------
LOCAL_SALES_IDX BITMAP DISABLED
@@@
SYS@ocp> create tablespace dw_sh_sqlldr datafile size 5M
> autoextend on default
> storage(initial 64k next 64k pctincrease 0 maxextents unlimited);
Tablespace created.
@@@
alter table sh.sales
add partition sales_q1_2012
values less than (to_date('01-02-2012','dd-mm-yyyy'))
pctfree 0 pctused 99
storage (initial 64k next 64k pctincrease 0)
tablespace dw_sh_sqlldr
/
SYS@ocp> /
Table altered.
@@@
@@@<2>disable constraints and trigger, however I leave the check constraint.
@@@
@@@disable all constraints, for speed up.
SYS@ocp> ALTER TABLE sh.sales DISABLE CONSTRAINT SALES_TIME_FK;
Table altered.
SYS@ocp> ALTER TABLE sh.sales DISABLE CONSTRAINT SALES_PROMO_FK;
Table altered.
SYS@ocp> ALTER TABLE sh.sales DISABLE CONSTRAINT SALES_PRODUCT_FK;
Table altered.
SYS@ocp> ALTER TABLE sh.sales DISABLE CONSTRAINT SALES_CUSTOMER_FK;
Table altered.
SYS@ocp> ALTER TABLE sh.sales DISABLE CONSTRAINT SALES_CHANNEL_FK;
Table altered.
@@@
SH@ocp> select CONSTRAINT_NAME,CONSTRAINT_TYPE, status from user_constraints where table_name='SALES';
CONSTRAINT_NAME C STATUS
------------------------------ - --------
SALES_TIME_FK R DISABLED
SALES_PRODUCT_FK R DISABLED
SALES_CUSTOMER_FK R DISABLED
SALES_PROMO_FK R DISABLED
SYS_C005209 C ENABLED
SYS_C005210 C ENABLED
SYS_C005211 C ENABLED
SYS_C005212 C ENABLED
SYS_C005213 C ENABLED
SYS_C005214 C ENABLED
SYS_C005215 C ENABLED
SALES_CHANNEL_FK R DISABLED
@@@
@@@make sure all trigger about this table are disable.
SYS@ocp> SELECT trigger_name, status FROM all_triggers WHERE table_name = 'SALES';
no rows selected
@@@
@@@<3>create datafile to test.
@@@
[oracle@station78 dwdir]$ cat sales_2012.dat
| | | | | ||
123123422-AUG-2010123412345671234561
123123416-AUG-2004123412345671234561
123123415-AUG-2011123412345671234561
123123414-AUG-2011123412345671234561
123123412-AUG-2012123412345671234561
@@@
@@@<4>write control file, and loading data.
@@@
[oracle@station78 dwdir]$ cat control02.ctl
options (direct=true)
unrecoverable load data
infile 'sales_2012.dat' badfile 'sales_2012.bad'
append
into table sales
partition (SALES_Q1_2012)
(
PROD_ID position (1-3) integer external,
CUST_ID position (4-7) integer external,
TIME_ID position (8-18) DATE "DD-MON-YYYY",
CHANNEL_ID position (19-22) integer external,
PROMO_ID position (23-29) integer external,
QUANTITY_SOLD position (30-35) integer external,
AMOUNT_SOLD position (36) integer external
)
[oracle@station78 dwdir]$ sqlldr USERID=sh/sh
> CONTROL=control02.ctl LOG=control02.log DIRECT=true SKIP_INDEX_MAINTENANCE=true;
@@@
@@@<5>inspect result
@@@
[oracle@station78 dwdir]$ pwd
/home/oracle/dwdir
[oracle@station78 dwdir]$ ls
control02.ctl control02.log sales_2012.bad sales_2012.dat
@@@
[oracle@station78 dwdir]$ cat control02.log
SQL*Loader: Release 10.2.0.1.0 - Production on Wed Aug 22 19:54:44 2012
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Control File: control02.ctl
Data File: sales_2012.dat
Bad File: sales_2012.bad
Discard File: none specified
(Allow all discards)
Number to load: ALL
Number to skip: 0
Errors allowed: 50
Continuation: none specified
Path used: Direct
Load is UNRECOVERABLE; invalidation redo is produced.
Table SALES, partition SALES_Q1_2012, loaded from every logical record.
Insert option in effect for this partition: APPEND
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
PROD_ID 1:3 3 CHARACTER
CUST_ID 4:7 4 CHARACTER
TIME_ID 8:18 11 DATE DD-MON-YYYY
CHANNEL_ID 19:22 4 CHARACTER
PROMO_ID 23:29 7 CHARACTER
QUANTITY_SOLD 30:35 6 CHARACTER
AMOUNT_SOLD 36 1 CHARACTER
Record 1: Rejected - Error on table SALES, column PROD_ID.
ORA-01722: invalid number
Record 6: Rejected - Error on table SALES.
ORA-14400: inserted partition key does not map to any partition
The following index(es) on table SALES were processed:
index SH.LOCAL_SALES_IDX partition SALES_Q1_2012 was made unusable due to:
SKIP_INDEX_MAINTENANCE option requested
Table SALES, partition SALES_Q1_2012:
4 Rows successfully loaded.
2 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Date cache:
Max Size: 1000
Entries : 5
Hits : 0
Misses : 0
Bind array size not used in direct path.
Column array rows : 5000
Stream buffer bytes: 256000
Read buffer bytes: 1048576
Total logical records skipped: 0
Total logical records read: 6
Total logical records rejected: 2
Total logical records discarded: 0
Total stream buffers loaded by SQL*Loader main thread: 2
Total stream buffers loaded by SQL*Loader load thread: 0
Run began on Wed Aug 22 19:54:44 2012
Run ended on Wed Aug 22 19:54:44 2012
Elapsed time was: 00:00:00.64
CPU time was: 00:00:00.02
@@@
@@@<6>summary the falut column
@@@
summary:
first line was failed in first column checking,
second line is out of range in date column. The thrid column
is the partition key column.
[oracle@station78 dwdir]$ cat sales_2012.bad
| | | | | ||
123123412-AUG-2012123412345671234561
@@@
@@@there is another problem, the data what I import using sql*load are not
@@@exist in parent keys.
@@@actually, data must be corresponding with parent key.
SH@ocp> ALTER TABLE sh.sales ENABLE CONSTRAINT sales_time_fk ;
ALTER TABLE sh.sales ENABLE CONSTRAINT sales_time_fk
*
ERROR at line 1:
ORA-02298: cannot validate (SH.SALES_TIME_FK) - parent keys not found
@@@
@@@use execptions to save exceptions record.
[oracle@station78 ~]$ cd $ORACLE_HOME
[oracle@station78 db_1]$ find ./ -name utlexcpt.sql
./rdbms/admin/utlexcpt.sql
[oracle@station78 db_1]$ cat ./rdbms/admin/utlexcpt.sql
rem
rem $Header: utlexcpt.sql,v 1.1 1992/10/20 11:57:02 GLUMPKIN Stab $
rem
Rem Copyright (c) 1991 by Oracle Corporation
Rem NAME
Rem except.sql - <one-line expansion of the name>
Rem DESCRIPTION
Rem <short description of component this file declares/defines>
Rem RETURNS
Rem
Rem NOTES
Rem <other useful comments, qualifications, etc.>
Rem MODIFIED (MM/DD/YY)
Rem glumpkin 10/20/92 - Renamed from EXCEPT.SQL
Rem epeeler 07/22/91 - add comma
Rem epeeler 04/30/91 - Creation
create table exceptions(row_id rowid,
owner varchar2(30),
table_name varchar2(30),
constraint varchar2(30));
@@@
[oracle@station78 ~]$ echo $ORACLE_HOME
/u01/app/oracle/product/10.2.0/db_1
[oracle@station78 ~]$ sqlplus sh/sh
SH@ocp> @/u01/app/oracle/product/10.2.0/db_1/rdbms/admin/utlexcpt.sql
Table created.
@@@
SH@ocp> ALTER TABLE sales ENABLE CONSTRAINTS sales_channel_fk EXCEPTIONS INTO exceptions;
ALTER TABLE sales ENABLE CONSTRAINTS sales_channel_fk EXCEPTIONS INTO exceptions
*
ERROR at line 1:
ORA-02298: cannot validate (SH.SALES_CHANNEL_FK) - parent keys not found
@@@
@@@find the data using rowid where store in exceptions which are not match parent key.
@@@(-.-)actually, all I input data do not match.
@@@now find the dismatch data, you could remove them.
SH@ocp> select * from exceptions;
ROW_ID OWNER TABLE_NAME CONSTRAINT
------------------ ------------------------------ ------------------------------ ------------------------------
AAANiJAAPAAAAAMAAA SH SALES SALES_CHANNEL_FK
AAANiJAAPAAAAAMAAB SH SALES SALES_CHANNEL_FK
AAANiJAAPAAAAAMAAC SH SALES SALES_CHANNEL_FK
AAANiJAAPAAAAAMAAD SH SALES SALES_CHANNEL_FK
SH@ocp> select * from sales where rowid in (select row_id from exceptions);
PROD_ID CUST_ID TIME_ID CHANNEL_ID PROMO_ID QUANTITY_SOLD AMOUNT_SOLD
---------- ---------- ------------------ ---------- ---------- ------------- -----------
123 1234 22-AUG-10 1234 1234567 123456 1
123 1234 16-AUG-04 1234 1234567 123456 1
123 1234 15-AUG-11 1234 1234567 123456 1
123 1234 14-AUG-11 1234 1234567 123456 1
@@@
@@@if you think it is not impact on your db. you could stand these data.
SH@ocp> ALTER TABLE sales ENABLE NOVALIDATE CONSTRAINTS sales_channel_fk;
Table altered.
SH@ocp> ALTER TABLE sales ENABLE NOVALIDATE CONSTRAINTS sales_customer_fk;
Table altered.
SH@ocp> ALTER TABLE sales ENABLE NOVALIDATE CONSTRAINTS sales_product_fk;
Table altered.
SH@ocp> ALTER TABLE sales ENABLE NOVALIDATE CONSTRAINTS sales_promo_fk;
Table altered.
SH@ocp> ALTER TABLE sales ENABLE NOVALIDATE CONSTRAINTS sales_time_fk;
@@@
@@@add RELY for query write
SH@ocp> ALTER TABLE sales MODIFY CONSTRAINTS sales_channel_fk RELY;
Table altered.
SH@ocp> ALTER TABLE sales MODIFY CONSTRAINTS sales_product_fk RELY;
Table altered.
SH@ocp> ALTER TABLE sales MODIFY CONSTRAINTS sales_promo_fk RELY;
Table altered.
SH@ocp> ALTER TABLE sales MODIFY CONSTRAINTS sales_product_fk RELY;
Table altered.
SH@ocp> ALTER TABLE sales MODIFY CONSTRAINTS sales_time_fk RELY;
Table altered.
@@@
SH@ocp> select CONSTRAINT_NAME,CONSTRAINT_TYPE, status,rely from user_constraints where table_name='SALES';
CONSTRAINT_NAME C STATUS RELY
------------------------------ - -------- ----
SALES_CHANNEL_FK R ENABLED RELY
SALES_TIME_FK R ENABLED RELY
SALES_PRODUCT_FK R ENABLED RELY
SALES_CUSTOMER_FK R ENABLED RELY
SALES_PROMO_FK R ENABLED RELY
SYS_C005209 C ENABLED
SYS_C005210 C ENABLED
SYS_C005211 C ENABLED
SYS_C005212 C ENABLED
SYS_C005213 C ENABLED
SYS_C005214 C ENABLED
SYS_C005215 C ENABLED
2@@@@maintance partition index
@@@Note: N/A means the index was partitioned.
@@@check the unusable index after loading data to your db.
SH@ocp> select index_name, status from user_indexes where table_name='SALES';
INDEX_NAME STATUS
------------------------------ --------
LOCAL_SALES_IDX N/A
SH@ocp> select index_name, partition_name, status from user_ind_partitions where status != 'USABLE';
INDEX_NAME PARTITION_NAME STATUS
------------------------------ ------------------------------ --------
LOCAL_SALES_IDX SALES_Q1_2012 UNUSABLE
SH@ocp> ALTER INDEX local_sales_idx REBUILD PARTITION sales_q1_2012;
Index altered.
SH@ocp> select index_name, partition_name, status from user_ind_partitions where status != 'USABLE';
no rows selected