Troubleshooting 'library cache: mutex X' waits. (文档 ID 1357946.1)
In this Document
| Purpose |
| Troubleshooting Steps |
| What is a 'library cache: mutex X' wait? |
| What causes 'library cache: mutex X' wait? |
| How to diagnose the cause. |
| How to Examine the Diagnostics. |
| Potential Solutions |
| References |
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.1.0.6 and laterInformation in this document applies to any platform.
***Checked for relevance on 10-June-2013***
PURPOSE
The purpose of the note is to help customers troubleshoot contention for wait event 'library cache: mutex X'.
TROUBLESHOOTING STEPS
What is a 'library cache: mutex X' wait?
Library cache: mutex X is similar to a library cache wait in earlier versions. 'Library cache: mutex X' may be caused by many issues. So it is important to find the correct cause; so that the right solution can be implemented.
What causes 'library cache: mutex X' wait?
- Frequent Hard Parses - If the frequency of Hard Parsing is extremely high, then contention can occur on this pin.
- High Version Counts - When Version counts become excessive, a long chain of versions needs to be examined and this can lead to contention on this event
- Invalidations and reloads
- Known Bugs
How to diagnose the cause.
1. Check to see if anything has changed:
a. increased load
b. any change in the application, os, or middle tier
c. any os change
2. Is there a trend to library cache: mutex X:
a. is there a certain time of the day when this wait is seen?
b. does something trigger this wait?
3. During the time of the issue, run AWR and ADDM. Also obtain the baseline to compare the load, parameter changes, and any other differences.
To gather this it is suggested to run AWR and ADDM for half an hour to an hour interval as follows:
SQL>@$ORACLE_HOME/rdbms/admin/addmrpt.sql
4. Sometimes system state dump is necessary to match known issues. For example, if there is no obvious candidate SQL in AWR, capturing holder or waiter processes in systemstate allows you to focus in on potential problems. Run system state when processes appear hung on 'library cache: mutex X':
(a) Non-Rac
sqlplus "/ as sysdba"
oradebug setmypid
oradebug unlimit
oradebug dump systemstate 266
wait 90 seconds
oradebug dump systemstate 266
wait 90 seconds
oradebug dump systemstate 266
quit
(b) RAC
oradebug setmypid
oradebug unlimit
oradebug setinst all
oradebug -g all hanganalyze 4
oradebug -g all dump systemstate 266
quit
5. Errorstacks: Another way to obtain process information is with errorstack. Assuming you can identify a blocker, taking errorstacks will provide much the same information as systemstates but with a much reduced disk footprint for trace. Once the ospid of the blocker has been found, an errorstack can be generated:
SQL> oradebug setospid <p.spid from above>
oradebug dump errorstack 3
<< wait 1min>>
oradebug dump errorstack 3
<< wait 1min>>
oradebug dump errorstack 3
exit
In particular, the stack from the resultant trace can be used to match known issues.
The system state and errorstacks are not easily readable; so a Service Request may need to be opened to read the files.
6. Sometimes it is not feasible to run system state dump, as it may be resource intensive. So the following sql can also be ran in interval:
from v$session s, v$sql t
where s.event like '%mutex%'
and t.sql_id = s.sql_id
Check to see what sessions are waiting on.
7. In 11g RAC, there is another less resource intensive tool that can be used when compared with taking system state dumps:
How to Examine the Diagnostics.
1. Normally, the top wait event will be the library cache: mutex X in the problematic AWR:
2. First look for high parsing and high version counts from AWR.
Click on *SQL Statistics under Main Report of AWR:
Then, under SQL Statistics click on 'SQL ordered by Parse Calls' and 'SQL ordered by Version Count' to view that information:
Check for high parse calls.
Check to see if there is high parse calls to execute. Ideally, there should be less parse to executions. Notice there is as many parses as executes, indicating cursors are not used well in the application. Once the cursor is opened and parsed, it should be kept open. Check with the application developer on how to keep the cursor opened to re execute the sqls.
Next, check the version count of the sql:
From this list, investigate the SQLs with the high version count. What are the reasons that these statements are not shared? Can this be addressed?
Check V$SQL_SHARED_CURSOR to see the potential reason for the high version count using:
Document 296377.1Troubleshooting: High Version Count Issues
Potential Solutions
1. Check for high hard parsing, as this can cause can reloads in the sql area. Check the hard parse under the load profile:
This load shows 26.3 hard parses per second, indicating high hard parsing. Check to see if the application is sharing the sql. If application is mostly using literals, see if the sqls can be shared by using bind variables. Furthermore, review the 'Over Parsing' section of the following note:
Also check for high reloads in the sql area:
If there is a high number of reloads, then look to see if cursors are being shared efficiently. If they are then check to see if the shared pool or sga_target is large enough. Remember that inefficient sharing means that the library cache will fill up with non-reuseable cursors which may then cause reuseable ones to be flushed out. These will cause reloads when they are re-executed.
If sharing is efficient and the shared pool is too small, then shareable SQL statements will age out and hard parses will be higher. In most cases however this is not the case and the problem is inefficient sharing.
The following note is helpful in tuning the shared pool:
2. Check for invalidations under Library Cache Activity. If the invalidation has high number, then check ddl's performed during the time such as truncate, drop, grants, dbms_stats, etc.
3. Check the following note for relevant bugs under 'Known Bugs' and relevant version:
4. For 11g, make sure cursor_sharing is not similar, as it has been deprecated. This may also cause mutex waits:
5. If the database has been migrated from 10g to 11g and mutex performance issue surfaces, please consider the 11.2.0.2.2 psu + fix for Unpublished Bug 12431716. Many mutex fixes are already included in this patch:
REFERENCES
NOTE:438755.1 - High SQL Version Counts - Script to determine reason(s)NOTE:1169017.1 - ANNOUNCEMENT: Deprecating the cursor_sharing = 'SIMILAR' setting
NOTE:1291879.1 - Oracle Database Patch Set Update 11.2.0.2.2 Known Issues
NOTE:296377.1 - Troubleshooting: High Version Count Issues
NOTE:33089.1 - * TROUBLESHOOTING: Possible Causes of Poor SQL Performance
NOTE:459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes
NOTE:62143.1 - Troubleshooting: Tuning the Shared Pool and Tuning Library Cache Latch Contention
NOTE:727400.1 - WAITEVENT: "library cache: mutex X"
- Snap1.bmp(196.68 KB)
- Snap11.bmp(671.54 KB)
- Snap2.bmp(181.04 KB)
- Snap3.bmp(191.65 KB)
- Snap5.bmp(389.55 KB)
- Snap8.bmp(452.13 KB)
- Snap9.bmp(485.99 KB)
- Oracle Database Products > Oracle Database > Oracle Database > Oracle Database - Enterprise Edition > RDBMS > Database Level Performance Issues (not SQL Tuning)