【问题记录】逻辑备库 警告 长时间运行 事务

下午检查下备库的alter log，一直报如下警告，该备库是逻辑备库。

Mon Aug  5 16:29:17 2013

WARNING: the following transaction makes no progress
WARNING: in the last 30 seconds for the given message!
WARNING: xid = 
0x0003.025.0013b492 cscn = 9532709322703, message# = 11, slavid = 1

knacrb: no offending session found (not ITL pressure)

对于此警告，官方文档说明如下：

One of the primary causes for long-running transactions in a SQL Apply environment is because of Full Table Scans. Additionally, long-running transactions could be the result of SQL statements being replicated to the standby database, such as when creating or rebuilding an index.

http://docs.oracle.com/cd/B28359_01/server.111/b28294/troubleshooting.htm

1、NOT ITL(Interested Transaction List) PRESSURE

--关于Interested Transaction List 参见Dave的blog

Identifying Long-Running Transactions

If SQL Apply is executing a single SQL statementfor a long period of time, then a warning message similar to the following is reported in the alert log of the SQL Apply instance:

Mon Feb 17 14:40:15 2003
WARNING: the following transaction makes no progress
WARNING: in the last 30 seconds for the given message!
WARNING: xid =
0x0016.007.000017b6 cscn = 1550349, message# = 28, slavid = 1
knacrb: no offending session found (not ITL pressure)

Note the following about the warning message:

• This warning is similar to the warning message returned for interested transaction list (ITL) pressure, with the exception being the last line that begins with knacrb. The final line indicates:

  • A Full Table Scan may be occurring

  • This issue has nothing to do with interested transaction list (ITL) pressure

• This warning message is reported only if a single statement takes more than 30 seconds to execute.

It may not be possible to determine the SQL statement being executed by the long-running statement, but the following SQL statement may help in identifying the database objects on which SQL Apply is operating: 

SQL> SELECT SAS.SERVER_ID
  2       , SS.OWNER
  3       , SS.OBJECT_NAME
  4       , SS.STATISTIC_NAME
  5       , SS.VALUE
  6    FROM V$SEGMENT_STATISTICS SS
  7       , V$LOCK L
  8       , V$STREAMS_APPLY_SERVER SAS
  9   WHERE SAS.SERVER_ID = &SLAVE_ID
 10     AND L.SID = SAS.SID
 11     AND L.TYPE = 'TM'
 12     AND SS.OBJ# = L.ID1;

--断定是哪个object在执行这个长时间运行的事务。

Additionally, you can issue the following SQL statement to identify the SQL statement that has resulted in a large number of disk reads being issued per execution:

SQL> SELECT SUBSTR(SQL_TEXT,1,40)
  2       , DISK_READS
  3       , EXECUTIONS
  4       , DISK_READS/EXECUTIONS
  5       , HASH_VALUE
  6       , ADDRESS
  7    FROM V$SQLAREA
  8   WHERE DISK_READS/GREATEST(EXECUTIONS,1) > 1
  9     AND ROWNUM < 10
 10   ORDER BY DISK_READS/GREATEST(EXECUTIONS,1) DESC

--查看执行大量disk_read 磁盘读操作的SQL语句

Oracle recommends that all tables have primary key constraintsdefined, which automatically means that the column is defined as NOT NULL. For any table where a primary-key constraint cannot be defined, an index should be defined on an appropriate column that is defined as NOT NULL. If a suitable column does not exist on the table, then the table should be reviewed and, if possible, skipped by SQL Apply. 

--Oracle建议所有表应该有主键约束即该列有NOT NULL约束，如果没有主键也应该在有NOT NULL约束的列上加索引。如果表中不存在合适的列（即以上两点都不能满足），则应该重新审视该表，可能的话跳过SQL Apply。

The following steps describe how to skip all DML statements issued against the FTS table on the SCOTT schema:

1. Stop SQL Apply:

   SQL> ALTER DATABASE STOP LOGICAL STANDBY APPLY;
   Database altered
   

2. Configure the skip procedure for the SCOTT.FTS table for all DML transactions:

   SQL> EXECUTE DBMS_LOGSTDBY.SKIP(stmt => 'DML' , -
        schema_name => 'SCOTT' , -
        object_name => 'FTS');
   PL/SQL procedure successfully completed
   

3. Start SQL Apply:

   SQL> ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;
   Database altered

2、Troubleshooting ITL Pressure

Interested transaction list (ITL) pressure is reported in the alert log of the SQL Apply instance. Example A-3 shows an example of the warning messages.

Example A-3 Warning Messages Reported for ITL Pressure

Tue Apr 22 15:50:42 2003
WARNING: the following transaction makes no progress
WARNING: in the last 30 seconds for the given message!
WARNING: xid =
0x0006.005.000029fa cscn = 2152982, message# = 2, slavid = 17

Real-Time Analysis

The messages shown in Example A-3 indicate that the SQL Apply process (slavid) #17 has not made any progress in the last 30 seconds. To determine the SQL statement being issued by the Apply process, issue the following query:

SQL> SELECT SA.SQL_TEXT
  2    FROM V$SQLAREA SA
  3       , V$SESSION S
  4       , V$STREAMS_APPLY_SERVER SAS
  5   WHERE SAS.SERVER_ID = &SLAVEID
  6     AND S.SID = SAS.SID
  7   AND SA.ADDRESS = S.SQL_ADDRESS
SQL_TEXT
------------------------------------------------------------
insert into "APP"."LOAD_TAB_1" p("PK","TEXT")values(:1,:2)

--查看Apply进程正在进行的SQL语句

An alternative method to identifying ITL pressure is to query the V$LOCK view, as shown in the following example. Any session that has a request value of 4 on a TX lock, is waiting for an ITL to become available.

SQL> SELECT SID,TYPE,ID1,ID2,LMODE,REQUEST
  2    FROM V$LOCK
  3   WHERE TYPE = 'TX'

SID        TY ID1        ID2        LMODE      REQUEST
---------- -- ---------- ---------- ---------- ----------
         8 TX     327688         48          6          0
        10 TX     327688         48          0          4

--查看V$LOCK视图也能辨别ITL pressure。任何session有request为4的TX锁都在等待ITL的可用。

In this example, SID 10 is waiting for the TX lock held by SID 8.

Post-Incident Review

Pressure for a segment's ITL is unlikely to last for an extended period of time. In addition, ITL pressure that lasts for less than 30 seconds will not be reported in the standby databases alert log. Therefore, to determine which objects have been subjected to ITL pressure, issue the following statement:

SQL> SELECT OWNER, OBJECT_NAME, OBJECT_TYPE
  2    FROM V$SEGMENT_STATISTICS
  3   WHERE STATISTIC_NAME = 'ITL WAITS'
  4     AND VALUE > 0
  5   ORDER BY VALUE

--查看所有成为过ITL pressure的object

This statement reports all database segments that have had ITL pressure at some time since the instance was last started.

Note:

This SQL statement is not limited to a logical standby databases in the Data Guard environment. It is applicable to any Oracle database.

Resolving ITL Pressure

To increase the INITRANS integer for a particular database object, it is necessary to first stop SQL Apply.

--增大表的INITRANS

See Also:

Oracle Database SQL Language Reference for more information about specifying the  INITRANS integer, which it the initial number of concurrent transaction entries allocated within each data block allocated to the database object

The following example shows the necessary steps to increase the INITRANS for table load_tab_1 in the schema app.

1. Stop SQL Apply:

   SQL> ALTER DATABASE STOP LOGICAL STANDBY APPLY;
   Database altered.
   

2. Temporarily bypass the database guard:

   SQL> ALTER SESSION DISABLE GUARD;
   Session altered.
   

3. Increase the INITRANS on the standby database. For example:

   SQL> ALTER TABLE APP.LOAD_TAB_1 INITRANS 30;
   Table altered
   

4. Reenable the database guard:

   SQL> ALTER SESSION ENABLE GUARD;
   Session altered
   

5. Start SQL Apply:

   SQL> ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;
   Database altered.
   

Also, consider modifying the database object on the primary database, so in the event of a switchover, the error should not occur on the new standby database.