Discovering Bad SQL

--it comes from TOAD WORLD (http://www.toadworld.com/KNOWLEDGE/KnowledgeXpertforOracle/tabid/648/TopicID/BDSQL/Default.aspx)

There are basically three scenarios that you will encounter:

• You will need to discover what caused a performance problem that occurred at some time in the immediate past
• You need to find out why the overall system is performing poorly at this time
• You need to find out why a particular process is performing poorly at this time.

Successful discovery of the first problem is usually dependent on whether or not you had your Statspack, Oracle 10g's Automated Workload Repository (AWR) data captures or, or third-party data collection running, although the dynamic performance views might be helpful as well. Successful resolution of the second and third scenarios will require a process or methodology that uses the V$ dynamic performance views to discover bad SQL, and then to explain why the SQL performs poorly. So, let's look at usingat using the V$ views to discover current bad SQL.

Discovering the Cause of Poor Performance

Ok, so your database is performing poorly all of a sudden, and no one knows any particular reason why. We can call this scenario troubleshooting with no information. These types of problems can easily be due to one or more rouge SQL or processes. Regardless, the DBA is usually tasked with discovering the offensive processes and bad SQL, given only the database name and no further information.

The V$ Views and Queries

Since you don’t already know which process or processes are consuming system resources, you need to discover this information quickly. Here are some basic things to look for:

• locking/blocking/waiting
• high CPU utilization
• undo or rollback issues
• wait issues
• IO issues
• Top-n SQL

Locking Issues

Before determining if you have poorly tuned SQL, you need to determine if instead you have a locking issue. Run catblock.sql immediately after the database has been built and the basic packages installed, then query DBA_BLOCKERS and DBA_WAITERS to discover sessions that are blocking other sessions:

SQL> select * from dba_blockers;
HOLDING_SESSION
---------------
             12

If no rows are returned, we eliminate locking as a performance problem, and start to investigate SQL. If there is blocking, then you need to determine what is causing it and deal with the issue. Blocking can be a troublesome problem in it's own right. Querying DBA_BLOCKERS can take a long time to return rows of data. Many DBA's prefer to query V$LOCK instead. The downside to the V$LOCK view is that it is much harder to read and interpret the data. The following query can be used against V$LOCK.

SELECT sid,type,id1,id2,lmode,request
FROM v$lock;

A session will be waiting on a lock to be released if the REQUEST column is non-zero. That session is waiting for a certain type of lock. To find the session that is holding that lock, look for the same ID1 and ID2 values for the same TYPE in another session.

In addition to the database views, Oracle's Enterprise Manager can be used to detect blocking. The following screenshot shows on user blocking another:

In this case, SCOTT is blocked by SYSTEM. Selecting one of the radio buttons and pressing the Kill Session button will terminate one of these sessions. There are many links to be able to drill down for further information.

CPU Issues

As the DBA, you should have access to the database server, and OS commands that you can use to discover the process id (PID) of the top CPU consumers:

• Unix: top, topas, ps – ef | grep ora
• Windows: NT Task Manager

You can relate the system processes to the database sessions in the v$process table. For Unix, the PID is the same as the SPID column in v$process. For Windows, the PID is the same as the first part of the PROCESS column in v$session. Here is an example of finding the sid and it’s associated process on an Oracle NT Database.

-- Run on NT
Select sid, serial#, process
from v$session
where username is not NULL;
SID           SERIAL# PROCESS
---------- ---------- -------
         9         33 315:338
        10        258 346:345
        12          7 352:351

On Oracle for UNIX the query would look like this:

Select a.sid, a.serial#, b.spid
from v$session a, v$process b
where username is not NULL
and a.paddr=b.paddr;
SID           SERIAL# PROCESS
---------- ---------- -------
         9         33 11332
        10        258 11322
        12          7 12211

Select a.sid, a.serial#, b.spid
from v$session a, v$process b
where username is not NULL
and a.paddr=b.paddr;
SID           SERIAL# PROCESS
---------- ---------- -------
         9         33 11332
        10        258 11322
        12          7 12211

Also, the CPU_TIME column of the V$SQLAREA view indicates how much CPU time has been used by the query. One thing to keep in mind is that CPU time is cumulative for all executions of this particular SQL. The following query looks at the SQL statement that has consumed the most CPU time, providing us with a tuning candidate:

select fetches, executions, a.parsing_user_id, a.cpu_time, a.sql_text
from v$sqlarea a, dba_users b
where cpu_time = (select max(cpu_time) from v$sql)
and a.parsing_user_id=b.user_id
and b.username not in ('SYS','SYSTEM')
/

FETCHES    EXECUTIONS CPU_TIME
---------- ---------- --------
SQL_TEXT
----------------------------------------------------------------
      2296       1232    11134
select * from emp where empno not in (select empno from bad_emp);

Here we see that a SQL statement has 2296 fetches for 1232 executions and a large amount of CPU time. We would probably want to investigate this situation further and determine if there is some tuning that can be done for this statement.

Undo or Rollback Issues

If a transaction is consuming large amounts of rollback, you’ll be able to identify it with this query:

select a.sid, a.serial#,
       substr(b.owner,1,8) "schema",
       substr(b.object_name,1,20) "object Name ",
       substr(b.object_type,1,10) "type ",
       substr(c.segment_name,1,8) "rbs ",
       substr(d.used_urec,1,12) "# of records "
from   v$locked_object l,
       dba_objects b,
       dba_rollback_segs c,
       v$transaction d,
       v$session a
where  l.object_id = b.object_id and
       l.xidusn = c.segment_id and
       l.xidusn = d.xidusn and
       l.xidslot = d.xidslot and
       d.addr = a.taddr
/
SID      SERIAL# schema   object Name        type       rbs      # of records
----- ---------- -------- ------------------ ---------- -------- ------------
   10        258 HR       EMPLOYEES          TABLE      _SYSSMU2 12

In this case SID 10 is accessing the HR.EMPLOYEES table. Only 12 records are being accessed, not a large amount at all, so we probably wouldn’t worry about this session.

Session Wait Issues

You can also determine what sessions are in a particular wait event, and for how long.

select sid, event, seconds_in_wait
from v$session_wait
where sid in (select sid from v$session where username is not NULL)
/
SID   EVENT                                    SECONDS_IN_WAIT
----- ---------------------------------------- ---------------
    9 null event                                             0
   10 SQL*Net message from client                          778
   12 SQL*Net message from client                         3119
   22 Buffer Busy Wait                                     222

In this case, session 22 is experiencing buffer busy waits, and has experienced 222 seconds of them. This would certainly be something we would want to investigate quickly.

I/O Issues

The easiest way to find potential IO issues is to query the V$SESSIO view, sorted by the PHYSICAL_READS column:

select * from v$sess_io
where sid in (select sid from v$session where username is not NULL)
order by physical_reads
/
SID   BLOCK_GETS CONSISTENT_GETS PHYSICAL_READS BLOCK_CHANGES CONSISTENT_CHANGES
----- ---------- --------------- -------------- ------------- ------------------
   10         21              65              1            35                  0
   12          3              64              2             4                  0
    9        121          212893           2399             8                  0

In this listing, we see that session SID 9 has a large number of consistent gets. You would want to determine just what this session is up to, and try to tune the SQL appropriately.

Top-N SQL

Query the V$SQL view to determine the top-N queries by resource utilization. In the following query, we sort by CPU utilization:

SELECT * FROM
(SELECT cpu_time, elapsed_time, disk_reads, buffer_gets, rows_processed, executions,
sql_text from v$sql order by cpu_time desc)
WHERE ROWNUM < 6
/
CPU_TIME   ELAPSED_TIME DISK_READS BUFFER_GETS ROWS_PROCESSED EXECUTIONS
---------- ------------ ---------- ----------- -------------- ----------
SQL_TEXT
----------------------------------------------------------------
  52843750     64212467        150      228183          11858        539
select t.schema, t.name, t.flags, q.name from system.aq$_queue_tables t, sys.aq$_queue_table_affinities aft, system.aq$_queues q where aft.table_objno = t.objno and aft.owner_instance = :1 and q.table_objno = t.objno and q.usage = 0 and bitand(t.flags, 4+16+32+64+128+256) = 0 for update of t.name, aft.table_objno skip locked

  15265625     17625255          1       83258          11858      11858
select t.name, (select owner_instance from sys.aq$_queue_table_affinities where table_objno = t.objno) from system.aq$_queue_tables t where t.name = :1 and t.schema = :2 for update skip locked
......

This type of query is extremely useful at determining the actual SQL that contributes to poor performance. In the case of this output, there would not be a great deal we could do as this all appears to be recursive SQL associated with advanced queuing. However, it might be that if we are not using advanced queuing we might disable the feature and eliminate this SQL from executing.