In this Document
|
Pro-Active Problem Avoidance and Diagnostic Collection |
|
Cache Buffers Chains Latch Contention |
|
Connection Timeouts: Errors ORA-3135/ORA-3136 |
|
Deadlocks: Error ORA-00060 |
|
High Numbers of Child Cursors/High Cursor Version Counts |
|
SQL Issue Diagnosis Using DBMS_SQLDIAG |
|
'WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!' |
APPLIES TO:
Oracle Database - Enterprise Edition - Version 10.1.0.2 and later
Oracle Database - Personal Edition - Version 10.1.0.2 and later
Oracle Database - Standard Edition - Version 10.1.0.2 and later
Information in this document applies to any platform.
PURPOSE
This article provides a central point for Performance Troubleshooting advice.
You can access categorised Troubleshooting information via the following:
Document 1543445.2 Troubleshooting Assistant: Troubleshooting Performance Issues
Document 1542678.2 Troubleshooting Assistant: SQL Performance Issues
This articles supersedes :
Document 106285.1 TROUBLESHOOTING GUIDE: Common Performance Tuning Issues (ARCHIVED)
Best Practices
Pro-Active Problem Avoidance and Diagnostic Collection
Although some problems may be unforeseen, in many cases problems may be avoidable if signs are detected early enough. Additionally, if an issue does occur, it is no use collecting information about that issue after the event. For information on suggested preparations to avoid issues and in case diagnostics are required, see:
Document 1482811.1 Best Practices: Proactively Avoiding Database and Query Performance Issues
Document 1477599.1 Best Practices Around Data Collection For Performance Issues
TROUBLESHOOTING STEPS
General Performance
The following articles can help with the interpretation of diagnostics to troubleshoot various issues:
Document 390374.1 Oracle Performance Diagnostic Guide (OPDG)
Document 1362329.1 How To Investigate Slow Database Performance Issues
AWR Generation
To troubleshoot the failure to collect AWR snapshots or reports, See:
Document 1301503.1 Troubleshooting: AWR Snapshot Collection issues
Document 1363422.1 Automatic Workload Repository (AWR) Reports - Start Point
AWR Interpretation
How to troubleshoot performance issues using AWR snapshots or reports, See:
Document 1359094.1 FAQ: How to Use AWR reports to Diagnose Database Performance Issues
Buffer Busy Contention
To troubleshoot issues where Buffers have become Busy due to concurrent access, see:
Document 155971.1 Resolving Intense and "Random" Buffer Busy Wait Performance Problems:
Cache Buffers Chains Latch Contention
To troubleshoot issues caused by contention due to multiple sessions waiting to read the same block, see:
Document 1342917.1 Troubleshooting 'latch: cache buffers chains' Wait Contention
Configuration Parameters
Database parameters are designed so that the default is appropriate for the majority of cases. The following article discusses Optimizer Specific Parameters:
Document 1507909.1 What does it Mean if Disabling an Optimizer Specific Parameter Avoids a Performance Problem Detected After an Upgrade?
Connection Timeouts: Errors ORA-3135/ORA-3136
To troubleshoot issues caused by Connection Timeouts when errors such as ORA-3135 or ORA-3136 are seen, see:
Document 730066.1 Troubleshooting ORA-3135/ORA-3136 Connection Timeouts Errors - Database Diagnostics
Deadlocks: Error ORA-00060
For issues related to Deadlocks,refer to :
Document 60.1 Troubleshooting Assistant: Oracle Database ORA-00060 Errors on Single Instance (Non-RAC) Diagnosing Using Deadlock Graphs in ORA-00060 Trace Files
Document 1509919.1 Master Note for Database Error ORA-00060
Disk I/O
For issues related to Disk Input/Ouput (I/O),refer to :
Document 223117.1 Troubleshooting I/O-related waits
Document 1275596.1 How to Tell if the IO of the Database is Slow
Enqueue/Locks
For locking issues refer to :
Document 1392319.1 Master Note: Locks, Enqueues and Deadlocks
Document 15476.1 FAQ about Detecting and Resolving Locking Conflicts
High Numbers of Child Cursors/High Cursor Version Counts
When a SQL statement is executed, Oracle will try to match that statement up with an existing identical statement that has already been executed and attempt to reuse the stored parsed representation of it (which is stored within a cursor structure in the library cache). If the existing information in the cursor for that statement cannot be used, a new version of that statement will be created and used instead. If lots of versions of a particular statement are created, Oracle has to search through the versions to determine which (if any) can be shared. In extreme circumstances, this can cause mutex contention and other issue that degrade the database performance. In order to troubleshoot these issues and determine the cause, see:
Document 296377.1 Troubleshooting: High Version Count Issues
Hung database
To troubleshoot cases where the database appears to have hung, refer to :
Document 1378583.1 Troubleshooting Database Hang Issues
Document 452358.1 How to Collect Diagnostics for Database Hanging Issues
Latches
To troubleshoot issues where there is contention on 'cache buffers chains' latches see:
Document 1342917.1 Troubleshooting 'latch: cache buffers chains' Wait Contention
Library Cache
For issues related to contention in the Library Cache
Document 62143.1 Troubleshooting: Tuning the Shared Pool and Tuning Library Cache Latch Contention
Log File Sync
To troubleshoot issues where 'log file sync' is a primary cause of session wait activity, see:
Document 1376916.1 Troubleshooting: "Log File Sync" Waits
Mutex Contention
For issues related to Mutex contention, refer to :
Document 1377998.1 Troubleshooting: Waits for Mutex Type Events
Document 1349387.1 Troubleshooting 'cursor: pin S wait on X' waits
Document 1357946.1 Troubleshooting 'library cache: mutex X' waits.
Document 1356828.1 FAQ: 'cursor: mutex ..' / 'cursor: pin ..' / 'library cache: mutex ..' Type Wait Events
Operating System (OS)
To troubleshoot Operating System (OS) related issues see:
Document 352648.1 How to Diagnose High CPU Usage Problems to the Module Level
Document 148176.1 Diagnosing hardware configuration induced performance problems
Paging/Memory
If heavy paging activity is seen and the database appears to be running more slowly than normal, refer to :
Document 1195174.1 Heavy Paging Causes Slow Database Performance
Slow Database
To troubleshoot occasions where the database appears to be running more slowly than normal, refer to :
Document 1362329.1 How To Investigate Slow Database Performance Issues
SQL Issue Diagnosis Using DBMS_SQLDIAG
An increasing number of issues can be detected, diagnosed and avoided using the DBMS_SQLDIAG package. For details see:
Document 1509192.1 How To Use DBMS_SQLDIAG To Diagnose Various Query Issues
SQL Tuning
For issues with individual queries, refer to the following troubleshooting documents:
Document 742112.1 * Query Performance Degradation - Recommended Actions
Document 745216.1 * Query Performance Degradation - Upgrade Related - Recommended Actions
Document 179668.1 * TROUBLESHOOTING: Tuning Slow Running Queries
Document 33089.1 * TROUBLESHOOTING: Possible Causes of Poor SQL Performance
Document 372431.1 * TROUBLESHOOTING: Tuning a New Query
Document 122812.1 * TROUBLESHOOTING: Tuning Queries That Cannot be Modified
Document 163563.1 * TROUBLESHOOTING: Advanced Query Tuning
Document 1386036.1 How To Diagnose Issues Where Plans Are Different From Different Clients
SYSAUX Tablespace Issues
Document 1399365.1 Troubleshooting Issues with the SYSAUX Tablespace
TKProf Runtime Errors
Document 1383275.1 Troubleshooting TKProf Runtime Errors
Upgrade
To troubleshoot issues encountered after upgrading a database see:
Document 1528847.1 Troubleshooting: Avoiding and Resolving Database Performance Related Issues After Upgrade
Document 160089.1 TROUBLESHOOTING: Server Upgrade Results in Slow Query Performance
For advice on how to avoid such issues before upgrading, see:
Document 1482811.1 Best Practices: Proactively Avoiding Database and Query Performance Issues
Document 785351.1 Oracle 11gR2 Upgrade Companion
Document 1392633.1 Things to Consider before upgrading to 11.2.0.3 to avoid performance or wrong results
Document 1320966.1 Things to Consider before upgrading to 11.2.0.2 to avoid performance or wrong results
Virtual Circuit Waits
Document 1415999.1 Troubleshooting: Virtual Circuit Waits
'WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!'
To troubleshoot issues when the database detects that a waiter has waited for a resource for longer than a particular threshold and he message "WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!" appears in the alert log, See:
Document 278316.1 Troubleshooting: "WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!"
REFERENCES
NOTE:1383275.1 - Troubleshooting TKProf Runtime Errors
NOTE:1386036.1 - How To Diagnose Issues Where Plans Are Different From Different Clients
NOTE:1392319.1 - Master Note: Locks, Enqueues and Deadlocks (ORA-00060)
NOTE:1392633.1 - Things to Consider Before Upgrading to 11.2.0.3 to Avoid Poor Performance or Wrong Results
NOTE:1399365.1 - Troubleshooting Issues with SYSAUX
NOTE:1195174.1 - Heavy Paging Causes Slow Database Performance
NOTE:1509192.1 - How To Use DBMS_SQLDIAG To Diagnose Various Query Issues
NOTE:1482811.1 - Best Practices: Proactively Avoiding Database and Query Performance Issues
NOTE:1477599.1 - Best Practices: Proactive Data Collection for Performance Issues
NOTE:1507909.1 - What does it Mean if Disabling an Optimizer Specific Parameter Avoids a Performance Problem Detected After an Upgrade?
NOTE:1376916.1 - Troubleshooting: "log file sync" Waits
NOTE:223117.1 - Troubleshooting I/O-related waits
NOTE:296377.1 - Troubleshooting: High Version Count Issues
NOTE:1377998.1 - Troubleshooting: Waits for Mutex Type Events
NOTE:1378583.1 - Troubleshooting Database Hang Issues
NOTE:1275596.1 - How to Tell if the I/O of the Database is Slow
NOTE:1301503.1 - Troubleshooting: AWR Snapshot Collection issues
NOTE:1320966.1 - Things to Consider Before Upgrading to 11.2.0.2 to Avoid Poor Performance or Wrong Results
NOTE:1342917.1 - Troubleshooting 'latch: cache buffers chains' Wait Contention
NOTE:106285.1 - Legacy Troubleshooting Guide: Common Performance Tuning Issues
NOTE:1349387.1 - Troubleshooting 'cursor: pin S wait on X' waits.
NOTE:1356828.1 - FAQ: 'cursor: mutex ..' / 'cursor: pin ..' / 'library cache: mutex ..' Type Wait Events
NOTE:390374.1 - Oracle Performance Diagnostic Guide (OPDG)
NOTE:452358.1 - How to Collect Diagnostics for Database Hanging Issues
NOTE:122812.1 - * TROUBLESHOOTING: Tuning Queries That Cannot be Modified (10g and below)
NOTE:163563.1 - * TROUBLESHOOTING: Advanced Query Tuning
NOTE:179668.1 - TROUBLESHOOTING: Tuning Slow Running Queries
NOTE:1362329.1 - How To Investigate Slow or Hanging Database Performance Issues
NOTE:1363422.1 - Automatic Workload Repository (AWR) Reports - Start Point
NOTE:61552.1 - Troubleshooting Oracle Database Hanging Issues for versions from 7 to 9--Exhaustive.
NOTE:62143.1 - Troubleshooting: Tuning the Shared Pool and Tuning Library Cache Latch Contention
NOTE:730066.1 - Troubleshooting ORA-3135/ORA-3136 Connection Timeouts Errors - Database Diagnostics
NOTE:33089.1 - * TROUBLESHOOTING: Possible Causes of Poor SQL Performance
NOTE:742112.1 - Troubleshooting Query Performance Degradation - Recommended Actions
NOTE:745216.1 - Troubleshooting Upgrade Related Query Performance Degradation - Recommended Actions
NOTE:352648.1 - How to Diagnose High CPU Usage Problems to the Module Level
NOTE:372431.1 - * TROUBLESHOOTING: Tuning a New Query
NOTE:1415999.1 - Troubleshooting: Virtual Circuit Waits
NOTE:15476.1 - FAQ: Detecting and Resolving Locking Conflicts and Ora-00060 errors
NOTE:155971.1 - Resolving Intense and "Random" Buffer Busy Wait Performance Problems
NOTE:160089.1 - Troubleshooting a Server Upgrade Resulting in Slow Query Performance
NOTE:1509919.1 - Master Note for Database Error ORA-00060 "deadlock detected while waiting for resource"
NOTE:1528847.1 - Troubleshooting: Avoiding and Resolving Database Performance Related Issues After Upgrade
NOTE:1357946.1 - Troubleshooting 'library cache: mutex X' waits.