Chapter 2. Waits

Chapter 2. Waits
In an Oracle instance many processes (or threads of a single process) work
together. To work together, they must communicate, and one of main ways that
they communicate is via semaphores. A semaphore is a signal. It is somewhat like
a railway signal that tells trains whether to stop and wait, and when to go. Oracle
server processes often need to stop and wait:
?? Sometimes because a resource is not available
?? Sometimes because they have no work to do
?? Sometimes because they need to wait for another server process to
perform a prerequisite task
Semaphores allow Oracle server processes to stop and wait, and then to be notified
when they should resume processing.
2.1 Semaphores
There is a semaphore for every Oracle server process. Processes wait on their
semaphore when they need to wait for a resource, or need work to do, or need
work to be done. When the resource has been freed, or when there is work to do,
or when the prerequisite work has been done, then their semaphore is posted as a
signal to stop waiting.
For example, LGWR (the Log Writer process) may be waiting on its semaphore
for work to do, while a user process may be copying redo information into the
redo log buffer. When the user commits, LGWR must write the redo and commit
marker to the log file while the user waits. To achieve this, the user process posts
LGWR's semaphore to signal that it can stop waiting for work to do, as some
work is now available. The user process then waits on its own semaphore. When
the log file I/O has completed, LGWR posts the semaphore of the user process to
signal that it can now begin its next transaction, because the commit operation
has completed. LGWR then waits on its own semaphore again, because it has no
more work to do.
For another example, process A may need to update a row, but find that process
B has not yet committed an earlier update to the same row. Process A must wait
for process B to commit. To achieve this, process A will wait on its semaphore.
When process B commits, it will post process A's semaphore to signal that it can
now proceed with its update.
2.1.1 Semaphore Facilities
Semaphores are an operating system facility. When an Oracle process is waiting
on its semaphore, the operating system will not schedule it to run on a CPU. In
operating system terms, it is blocked, not runnable. When the semaphore is
posted, the operating system status of the process is changed from blocked to
runnable, and the process will be scheduled to run as soon as possible.
Some operating systems support more than one type of semaphore. System V
semaphores are the most common. The semaphore data structures for System V
semaphores form a fixed array in kernel memory sized by the SEMMNS kernel
parameter. To post a semaphore or wait on a semaphore, processes must use the
semop( ) system call. Because they are implemented in the operating system
kernel, System V semaphores suffer from unnecessarily high system call context
switch overheads and poor scalability due to serialization requirements for access
to the kernel data structures.
For better performance and scalability, an alternative set of semaphore
operations is supported on several operating systems. These are implemented in a
pseudo device driver, called a post-wait driver. The data structures for these
semaphores reside in user memory, rather than kernel memory, and can
therefore be manipulated by the pseudo device driver running in user context.
This reduces the number of system call context switches, and improves
scalability, but it is operating system specific.
The POSIX real-time extensions subcommittee has identified the need for a
standards-compliant user memory semaphore facility. The POSIX.1b standard
(formerly POSIX.4) defines both the interface and implementation requirements
for such a semaphore facility that is elegant and efficient, not to mention
portable. POSIX.1b semaphores are now available on many operating systems.
Which semaphore facility Oracle uses is operating system and release specific. If
your Oracle installation guide has instructions about setting the SEMMNS kernel
parameter, that means System V semaphores will be used by default.
Unfortunately, this is still the case on a large number of operating systems.
Incidentally, the prevalent recommendation to set SEMMNS to 200, without
regard for the projected number of Oracle processes, or the requirements of other
system and application software, is ill-conceived. You must allow one semaphore
for each Oracle server process, in addition to other requirements, as explained
more fully in Table 2.1.
You should also be aware that on some platforms each Oracle instance requires
its semaphores to be allocated in a single semaphore set. So the SEMMNI
parameter need only allow one semaphore identifier per instance, and SEMMSL
(if defined) must be no less than the largest PROCESSES parameter that might be
required for any instance. This is necessary to enable vector posts. Vector posts
may be used, mainly by the key background processes, LGWR and DBWn, to post
multiple waiting processes in a single semaphore operation. The use of vector
posts is dependent on the setting of the _USE_VECTOR_POSTS parameter.

Hidden Parameters
Parameters that begin with an underscore, such as
_USE_VECTOR_POSTS, are hidden parameters. You will not find
them in V$PARAMETER, or see them with the SHOW PARAMETERS
command, because they are hidden. You certainly will not find them
explained in the Oracle documentation, because they are
undocumented. You can, however, get their descriptions with the APT
script hidden_parameters.sql and check their values with the script
all_parameters.sql .
Some hidden parameters are operating system specific. Some are
needed only in unusual recovery situations. Some are used to disable
or enable new features. And many are related to obscure performance
issues. As with all undocumented features, hidden parameters may
disappear or change in a future release. You should therefore use them
as a last resort, and only after checking with Oracle Support, and
documenting the issues fully for your successor.

Further, if the SEMMNU kernel parameter is defined for your operating system,
it should be greater than the projected number of concurrent semaphore
operations system-wide. For systems with many semaphore client processes, the
default may be inadequate. If so, semaphore operations will fail intermittently at
periods of peak activity and return the ORA-7264 or ORA-7265 errors. To avoid
this, the SEMMNU parameter must be at least equal to the number of CPUs plus
the peak length of the CPU run queues.

-----------------------------------------------------------------------------------------------------------------------------------------

Table 2.1. System V Semaphore Parameters
Parameter Description
SEMMNS:
The number of semaphores in the system. In addition to the
requirements of the operating system and other software, you should
allow at least one semaphore for each Oracle server process—that is,
the sum of the setting of the PROCESSES parameter for all instances
on the system. If the semaphore clients are not always shut down and
started up in strict sequence, then an extra allowance at least equal to
the largest single requirement is recommended.
Further, the kernel parameter controlling the maximum number of
simultaneous processes owned by a single named user (often MAXUP)
should be at least equal to the SEMMNS setting, with an allowance for
other administrative processes owned by the "oracle" user that do not
require semaphores. However, this parameter should not be so large
as to allow the risk of another user creating so many processes that the
kernel process table would be completely filled. Therefore, the kernel
parameter controlling the maximum number of simultaneous
processes for all users (often NPROC) should be at least three times
the value of SEMMNS.
SEMMSL:
The size limit for a single semaphore set. This parameter is not defined
on some operating systems. Where it is defined, and where Oracle
requires all the semaphores for an instance to be allocated in a single
semaphore set, this parameter must be at least equal to the largest
PROCESSES parameter required for any instance.
SEMMNI:
The number of semaphore set identifiers in the system. In addition to
the requirements of the operating system and other software, you
should allow one identifier per instance, or more if the SEMMSL
parameter is set such that multiple semaphore sets will be required for
any instance.
SEMMNU:
The number of semaphore undo structures in the system. Undo
structures are used to recover the kernel semaphore data structures in
the event of the unexpected death of a process during a semaphore
operation. SEMMNU should be greater than the peak number of
running and runnable processes.

If Oracle uses System V semaphores on your operating system by default, but also
supports the use of a post-wait driver, then you should use the post-wait driver
instead. This normally involves setting the USE_POST_WAIT_DRIVER
parameter to TRUE, and it is sometimes necessary to set the
POST_WAIT_DEVICE parameter as well. Please consult your Oracle installation
guide, because the instructions are operating system and release dependent.

The semaphore parameters are operating system kernel
parameters and cannot be set in the Oracle initialization
parameter file (INIT.ORA).

If your installation guide makes no mention of setting kernel semaphore
parameters or of a post-wait driver, the selection and configuration of the
semaphore facility for your operating system is automatic.
2.1.2 Scheduling Latencies
When a process is posted, its operating system status is changed from blocked to
runnable. However, that does not mean it will be scheduled to run on a CPU
immediately. It must wait at least until the operating system's process scheduler
is next run, and possibly longer if there are higher priority processes waiting to
run. The delay from when a process is posted until it begins running is called the
scheduling latency. Scheduling latencies contribute to Oracle response times, as
illustrated in Figure 2.1, and so minimizing scheduling latencies is an important
part of performance tuning.

Many operating system scheduling algorithms adjust the execution priority of
processes in proportion to the amount of CPU time that they have consumed
recently. In very busy Oracle environments, this has the unfortunate effect of
degrading the execution priority of key background processes, such as LGWR,
DBWn, LCKn, and LMDn. This causes an increase in scheduling latencies for
those processes, and can in the extreme make the entire instance bottleneck on
the services of the affected background processes.

Some operating systems support multiple scheduling algorithms. Where possible,
you should choose a scheduling algorithm that does not degrade the execution
priority of processes in this way. Failing that, your operating system may provide
a priority fixing facility. If the execution priority of a process is fixed, it will not
degrade. In some cases, priority fixing is available to all users, and Oracle uses it
automatically. In other cases, it is only available to the system administrator, and
specially privileged users. If so, the "oracle" user must be granted this privilege,
or the system administrator must start the Oracle instance from a fixed priority
command shell, so that all Oracle processes will run with fixed priority.
Where priority fixing is not available, you may be able to obtain equivalent relief
from the priority degradation mechanism by artificially raising the execution
priority of the key background processes, or even running them in the real-time
priority class. You may feel reluctant to do this, on the basis that Oracle has often
recommended that all Oracle processes should run at the same priority. The
rationale for this recommendation is to prevent the possibility of a low-priority
process holding a critical resource but being unable to free it because of CPU
starvation, while other high-priority processes try repeatedly to obtain that
resource. However, this rationale scarcely applies to raising the priority of the
background processes. These processes will soon sleep if the resources they
require are not available, and beyond that will only consume CPU time in
proportion to the amount of work being done by the rest of the instance. So, there
is no risk of CPU starvation for other Oracle processes.

2.1.3 Timeouts
Oracle server processes are never willing to wait indefinitely, lest they never be
posted and wait forever. Fortunately, semaphore waits can be interrupted. So
before an Oracle process begins to wait on its semaphore, it arranges for its sleep
to be interrupted by setting an alarm clock, or timeout. If the process is posted, it
switches the alarm clock off and then continues processing. However, if the
timeout expires, the wait is interrupted by a SIGALRM signal. The process then
has the opportunity to reassess the situation and decide whether it wants to
continue to wait.
For example, a process waiting for an enqueue lock may perform deadlock
detection when its wait times out. If a deadlock is discovered, the statement will
be rolled back and an exception will be raised, but if not, the process will set a
new timeout and will begin to wait on its semaphore again.
It sometimes happens that a process is posted very shortly before its timeout is
due to expire, and the alarm goes off just as the process is trying to switch it off. In
this case, the Oracle process concerned will write a message to its trace file:
Ignoring SIGALRM
If you find some trace files with this message, it is nothing to be alarmed about. It
merely tells you that waiting processes are sometimes not being posted as quickly
as you might wish, and that is something you ought to be aware of anyway from
the wait statistics.
2.2 Wait Statistics
The Oracle wait statistics are pure gold—but not to be overvalued. Many types of
performance problems are easy to identify from the wait statistics. If Oracle is
waiting extensively for resources such as latches, free cache buffers, enqueue
locks, and so on, then the wait statistics can both identify and quantify the
problem. With experience, you may also be able to use the wait statistics to
identify network and disk performance problems. The wait statistics also provide
valuable feedback on attempts to resolve such problems.
But if your application is doing more parsing, or more disk I/O than necessary for
its workload, then the wait statistics cannot help you. They will appear to give
your instance a clean bill of health, and rightly so. The wait statistics are only able
to reveal inefficiencies at the database server level and below. So they are silent
about application-level performance problems that increase the load on the
database server but do not cause it to work inefficiently.
However, you should already have addressed all the application performance
issues before considering database server tuning in detail. If so, the wait statistics
can have full value for database server tuning. But they can only have full value if
the waits are timed.
2.2.1 Timed Statistics
Waits are timed if and only if the TIMED_STATISTICS parameter is set to TRUE.
Let me endorse what others have said before, that the overhead of timed statistics
is negligible. If you need to convince yourself, use the SET TIMING ON command
in SQL*Plus to measure the elapsed time of a benchmark query. Use an otherwise
idle system and take ten or more measurements with and without timed
statistics. You will be hard pressed to discern any significant difference.
Without timed statistics, Oracle records the reason for each wait before it begins
to wait, and when the wait is over, it records whether it timed out. But with timed
statistics enabled, Oracle checks the time just before and after each wait, and also
records the time waited. The time waited is recorded in hundredths of a second—
that is, centiseconds.
2.2.2 Wait Types
V$SYSTEM_EVENT shows the total number of waits and timeouts, and the total
waiting time recorded for each type of event, accumulated for all processes over
the life of the instance. It is normal to order the events waited for in descending
order of the total time waited, as an indicator of the potential severity of each
type of wait.
However, the total time waited is really only meaningful for those that indicate
waiting for resources. If processes have been waiting because they have no work
to do, then the time waited is immaterial. If they have been waiting for routine
operations, such as disk I/O, then the total time waited will depend on the
workload. In such cases, the average time waited is much more interesting than
the total time waited.
This classification of wait types into idle waits, routine waits, and resource waits
is vital to a correct understanding of the wait statistics. Accordingly, APT has
separate scripts for resource waits and routine waits, and ignores idle waits
altogether. The routine_waits.sql script shows only the average time waited for
each type of routine wait. The resource_waits.sql script (see Example 2.1) shows
the types of resources waited for in descending order of the total time waited, but
also shows the average time waited.
Example 2.1. Sample Output from resource_waits.sql
SQL> @resource_waits
---------------------------------------- ----------- ------------
write complete waits 3816218 212.02
buffer busy waits 1395921 21.79
enqueue 503217 529.15
log file switch completion 144263 90.11
latch free 31173 0.61
free buffer waits 19352 302.38
row cache lock 876 73.00
library cache pin 131 18.71
library cache load lock 29 2.64
non-routine log file syncs 0 2.32

The average time waited reported by resource_waits.sql is not what you might
expect. Because of timeouts, a single logical wait for a resource may be reported
as a series of distinct waits, each of which would have timed out, except the last.
The number of logical waits is approximately the number of times the waiting
process was posted to end its wait—that is, the number of distinct waits, minus
the number of waits that timed out. The average time waited for each logical wait
is a better indication of the time taken to resolve resource waits, than the average
time for each component wait. Therefore, that is what this script reports for all
resource waits except latch free waits. It is normal for latch free waits to time out,
because latch wait posting is the exception, not the rule. Also, apart from latch
contention, it is normal for the latch to be obtained after a timeout. So the
average time waited for each distinct wait is a better indication of the duration of
latch free waits.

2.2.3 Session Waits
V$SESSION_EVENT shows the wait statistics for each live session. Although
waits affect processes rather than sessions, they are recorded against sessions
because sessions can migrate between processes (as in Multi-Threaded Server
configurations). The cumulative session wait statistics have two main uses.
First, if a particular user reports an episode of poor performance, then the wait
statistics for that session can be examined to diagnose that user's problem. The
APT script called session_times.sql (see Example 2.2) shows the waiting time
accumulated by the session for each type of event waited for, together with the
amount of CPU time consumed by that session. This makes it easy to see whether
the session has been working or waiting, and if it has been waiting, what it has
been waiting for.
Example 2.2. Sample Output from session_times.sql
SQL> @session_times
Enter SID: 29
EVENT
TIME_WAITED
---------------------------------------------------------------- ------
-----
SQL*Net message from client
2954196
CPU used by this session
1657275
db file sequential read
246759
write complete waits
139698
buffer busy waits
61832
log file sync
32601
enqueue
9576
log file switch completion
3530
SQL*Net message to client
2214
db file scattered read
1879
SQL*Net more data to client
952
SQL*Net more data from client
908
latch free
840
free buffer waits
100
buffer deadlock
57
row cache lock
1
SQL*Net break/reset to client
0
Second, if there has been extensive waiting for a particular type of resource, then
the session wait statistics can be used to determine which of the sessions that are
still connected have contributed to or been affected by the problem. The APT
script for this job is called resource_waiters.sql . It shows the breakdown by
session of the waiting time for the resource type in question. The total waiting
time for sessions that are no longer active is also shown. For example, if there
have been a large number of buffer busy waits, then looking at the session wait
statistics may reveal whether the problem has been widespread, or confined to
just a few sessions.

2.2.4 Wait Parameters
The wait statistics are very useful because they tell you which sessions have been
waiting, and which types of resources they have been waiting for. They may have
been waiting for latches, database blocks, enqueue locks, or other resource types.
Knowing which type can direct your tuning efforts. But the wait parameters are
even more valuable than the wait statistics. They can tell you exactly which
resource—which latch, which database block, or which enqueue lock—is being
waited for. The wait statistics merely put you in the right neighborhood, but the
wait parameters can focus your attention on the right spot.
Unfortunately, the wait parameters are hard to catch. They can be seen fleetingly
in V$SESSION_WAIT . This view shows the wait parameters for the current or
most recent wait for each session, as well as the duration of the wait, if known.
However, querying V$SESSION_WAIT takes a long time relative to the length of
most waits. If you query this view twice in quick succession and look at the SEQ#
column, which is incremented for each distinct wait, it is not uncommon to notice
that many event waits have been missed in each active session between the two
queries. It is also rather expensive to query V$SESSION_WAIT repeatedly in
quick succession, and so it is of limited usefulness for watching wait parameters.
Fortunately, the wait parameters can also be seen in trace files produced by the
new DBMS_SUPPORT package, or by the underlying event 10046. This trace is
the same as that produced by the SQL_TRACE facility but also includes a line for
each wait, including the wait parameters.
For example, if there appears to be a problem with buffer busy waits, then you
can enable this trace for a while in the most heavily affected sessions with the
APT script trace_waits.sql . It is then just a matter of extracting the buffer busy
wait lines from the trace files, and examining the wait parameters to find the file
and block numbers of the blocks being waited for. In the case of buffer busy
waits, the file and block numbers are parameters p1 and p2. This is illustrated in
Example 2.3.
Example 2.3. Sample Dialog from trace_waits.sql
SQL> @trace_waits
the top N sessions affected by waits for a particular resource.
Select sessions waiting for: buffer busy waits
Number of sessions to trace: 5
Seconds to leave tracing on: 900
Tracing ... Please wait ...
PL/SQL procedure successfully completed.
SQL> exit
$ cd udump
$ grep 'buffer busy waits' ora_*.trc |
> sed -e 's/.*p1=/ file /' -e 's/ p2=/ block /' -e 's/ p3.*//' |
> sort |
> sort -nr |
> head -5
42 file 2 block 1036
12 file 24 block 3
10 file 2 block 1252
7 file 2 block 112
6 file 7 block 5122
$

The meaning of the wait parameters for each type of wait event is visible in
V$EVENT_NAME and is documented in an appendix to the Oracle8i Reference
guide. However, this is a particularly weak section of the Oracle documentation.
Much of the information is enigmatic, out-of-date, or inaccurate. Because the
wait parameters are so vital to advanced performance tuning, this book explains
the meaning of the wait parameters for each wait event discussed.

2.3 Reference
This section contains a quick reference to the parameters, events, statistics, and
APT scripts mentioned in Chapter 2.
2.3.1 Parameters
Parameter Description
_USE_VECTOR_POSTS
Vector posts enable multiple waiting processes to be posted
in a single semaphore operation.
POST_WAIT_DEVICE
The post-wait driver is a pseudo device driver. Its functions
are invoked when operations are performed against a device
special file of that device type. Where this parameter is used,
it specifies the path to the device file for the post-wait driver.
TIMED_STATISTICS
Should be set to TRUE whenever timing information may be
required for tuning purposes, which is always.
USE_POST_WAIT_DRIVER
If this parameter exists, it should be set to TRUE in order to
use the post-wait driver, instead of regular semaphore
operations.

2.3.2 Events
Event Description
10046
This is the event used to implement the DBMS_SUPPORT trace, which is a
superset of Oracle's SQL_TRACE facility. At level 4, bind calls are included in the
trace output; at level 8, wait events are included, which is the default level for
DBMS_SUPPORT; and at level 12, both binds and waits are included. See the
excellent Oracle Note 39817.1 for a detailed explanation of the raw information in
the trace file.
2.3.3 Statistics
Statistic Source Description
total_waits
V$SYSTEM_EVENT
V$SYSTEM_EVENT
The number of distinct waits.

total_timeouts
V$SYSTEM_EVENT
V$SESSION_EVENT
The number of waits that timed out instead
of being posted.
logical_waits
total_waits -
total_timeouts
A logical wait is a series of distinct waits for
the same event. Each component wait times
out, except the last, which is posted.
time_waited
V$SESSION_EVENT
V$SESSION_EVENT
The total time waited.
average_wait
V$SYSTEM_EVENT
V$SESSION_EVENT
The average time for each distinct wait.
average_logical
time_waited /
logical_waits
The average time for each logical wait.
max_wait V$SESSION_EVENT
The longest component wait by the session
for the event.

2.3.4 APT Scripts
Script Description
resource_waiters.sql
Shows which sessions have waited for a particular resource
type, and for how long.
resource_waits.sql
Shows all the resources waited for, and the total waiting time,
over the life of the instance, in order of severity.
routine_waits.sql Reports the average time waited for each routine wait.
session_times.sql
Shows how much time a particular session has used working
or waiting, and what is has been waiting for.
trace_waits.sql
Enables the DBMS_SUPPORT trace (event 10046, level 8) for
a period in the sessions most affected by a particular type of
resource wait. Used to sample the wait parameters, in order
to diagnose performance problems.