Wait Event Enhancements in Oracle 10g

by Terry Sutton and Roger Schrag Database Specialists, Inc. About Database Specialists, Inc.
Database Specialists, Inc. provides remote DBA services and onsite database support for your mission critical Oracle systems. Since 1995, we have been providing Oracle database consulting in Solaris, HP-UX, Linux, AIX, and Windows environments. We are DBAs, speakers, educators, and authors. Our team is continually recognized by Oracle, at national conferences and by leading trade publications. Learn more about our remote DBA, database tuning, and consulting services. Or, call us at 415-344-0500 or 888-648-0500.

Introduction
The wait event interface has continued to be an invaluable tool for DBAs as it oers both breadth and depth in the information it provides to aid in troubleshooting and boosting system performance. Throughout this paper we will assume the reader is familiar with wait event concepts and the wait event interface in Oracle. In particular, this paper is designed for DBAs who have experience using the wait event facility in Oracle 9i or earlier and want to learn what enhancements have been made in Oracle 10g. (Those new to this area of Oracle technology might want to rst read our paper entitled, Interpreting Wait Events to Boost System Performance available for free download at http://www.dbspecialists.com/presentations.html#wait_events.) There are still signicant gaps in the documentation as of Oracle 10g release 10.1.0.3, making it that much harder to learn what has changed in Oracle 10g with respect to wait events and the wait event interface. For example, the Oracle 10g Database Reference manual (part number B10755-01) still provides the wait events list from Oracle 9i. For this reason, you might nd the Appendix listed at the end of this paper to be useful. It lists all of the wait event names for Oracle 10g release 10.1.0.3, along with the parameter names for each. Although introduced in Oracle 7, not much changed in the wait event interface through and including Oracle 9i. Oracle 7.3 had just 106 wait events, while that number has increased to over 400 in Oracle 9i. Also, Oracle 9i TKPROF reports include wait event information and Oracle 9i v$ views show wait times in microseconds. But these are minor enhancements, really. Oracle 10g, on the other hand, brings more signicant change to the wait event interface than we have seen in years. Now there are over 800 wait events and names are more descriptive, wait events are categorized into classes, several v$ views have been added, helpful columns have been added to existing v$ views, built-in statistics collection by Active Session History and the Automatic Workload Repository has been introduced, a new time model concept for looking at how sessions spend their time has appeared, and improvements have been made to the session tracing facility.

Wait Event Enhancements in Oracle 10g

In this section, we will introduce what we see as the top dozen areas in which wait events and the wait event interface have been enhanced in Oracle 10g. Weve listed these enhancements in no particular order.

More Descriptive Wait Event Names

Prior to Oracle 10g, some wait event names were quite vague and not very useful without looking at the parameter values for a specic occurrence of the event. For example, an enqueue wait could indicate various situations ranging from contention for a row in a table to waiting on a user-dened lock. Wait event names in Oracle 10g are more descriptive in the areas of latches, enqueues, and buer busy waits. There is still a latch free wait event in Oracle 10g, but there are also 26 more specic latch-related events. These cover the most common latches that experience contention. In the past, if we saw a session waiting on the latch free event, we would have output like the following:
SQL> SELECT event, state, p1, p2, p3 2 FROM v$session_wait 3 WHERE sid = 162; EVENT STATE P1 ------------- ------- ----------latch free WAITING 15113593728 P2 P3 ------ ----97 5

We would then have to query v$event_name to determine the meaning of the parameters p1, p2, and p3:

1 of 32

SQL> SELECT * FROM v$event_name WHERE name = 'latch free'; EVENT# NAME PARAMETER1 PARAMETER2 PARAMETER3 ------ ---------- --------------- --------------- --------------3 latch free address number tries

And, seeing that p2 is the latch number, we would need to query v$latch to nd out which latch was being waited upon:
SQL> SELECT name 2 FROM v$latch 3 WHERE latch# = 97; NAME -------------------cache buffers chains

In Oracle 10g we simply see:

SQL> SELECT event, state 2 FROM v$session_wait 3 WHERE sid = 162; EVENT STATE ------------------------------ ------latch: cache buffers chains WAITING

The descriptive event name saves us two steps in determining which latch is causing the wait. The more detailed description enables one to more quickly drill down to the root cause of the wait. The names of enqueue-related wait events have also been made more descriptive in Oracle 10g. There is no longer an enqueue wait eventit has been replaced by 184 events with more detailed names. In the past, if sessions were waiting on enqueues, we would have to decode the type of lock from the p1 parameter:
SQL> SELECT event, state, seconds_in_wait siw 2 FROM v$session_wait 3 WHERE sid = 96; EVENT STATE SIW ----------------------------------- ------------------- ---------enqueue WAITING 24 SQL> SELECT sid, 2 CHR (BITAND (p1,-16777216) / 16777215) || 3 CHR (BITAND (p1, 16711680) / 65535) enq, 4 DECODE (CHR (BITAND (p1,-16777216) / 16777215) || 5 CHR (BITAND (p1, 16711680) / 65535), 6 'TX', 'Transaction (RBS)', 7 'TM', 'DML Transaction', 8 'TS', 'Tablespace and Temp Seg', 9 'TT', 'Temporary Table', 10 'ST', 'Space Mgt (e.g., uet$, fet$)', 11 'UL', 'User Defined', 12 CHR (BITAND (p1,-16777216) / 16777215) || 13 CHR (BITAND (p1, 16711680) / 65535)) enqueue_name, 14 DECODE (BITAND (p1, 65535), 1, 'Null', 2, 'Sub-Share', 15 3, 'Sub-Exclusive', 4, 'Share', 5, 'Share/Sub-Exclusive', 16 6, 'Exclusive', 'Other') lock_mode 17 FROM v$session_wait 18 WHERE sid = 96; SID ENQ ENQUEUE_NAME LOCK_MODE ----- ---- ------------------------------ ---------96 TX Transaction (RBS) Exclusive

In Oracle 10g we get more information directly from the enqueue name:

2 of 32

SQL> SELECT event, state, seconds_in_wait siw 2 FROM v$session_wait 3 WHERE sid = 143; EVENT STATE SIW ----------------------------------- ------------------- ---------enq: TX - row lock contention WAITING 495

Additionally, more information is available in some cases from the p1, p2, and p3 parameters, whose meanings vary with the dierent enqueue-related wait events. (See the Appendix for a listing of all wait events and their associated parameter meanings.) In addition to latch and enqueue waits, there are a few other wait events that have more descriptive names in Oracle 10g. However, the state of the Oracle 10g documentation as of this writing makes it hard to enumerate all of them. One example of another descriptive name change has to do with buer busy waits. The situation where one session is waiting for another session to read in a desired data block from disk (reason code 130) has been given the more descriptive wait event name read by other session.

Wait Event Classes

In Oracle 10g wait events are classied into categories which can help the DBA to more easily determine the likely root cause of the wait. The categories are: Administrative Idle Application Cluster Commit Concurrency Conguration Network Scheduler System I/O User I/O Other

While nearly 70% of the wait events are in the Other category (557 out of 811 in release 10.1.0.3), the most frequently encountered ones are in wait classes with helpful names. Lets look at the wait class designations of some enqueue events as an example:
SQL> 2 3 4 5 SELECT FROM WHERE AND ORDER BY wait_class, name v$event_name name LIKE 'enq%' wait_class <> 'Other' wait_class; NAME ---------------------------------------enq: TW - contention enq: DB - contention enq: PW - flush prewarm buffers enq: RO - contention enq: RO - fast object reuse enq: TM - contention enq: TX - row lock contention enq: UL - contention enq: TX - index contention enq: ST - contention enq: TX - allocate ITL entry enq: SQ - contention enq: HW - contention

WAIT_CLASS -----------------------------Administrative Administrative Application Application Application Application Application Application Concurrency Configuration Configuration Configuration Configuration

We see that TX enqueues (row locks) and TM enqueues (table locks) are in the Application class, which makes sense since these wait events generally occur because of application behavior. Meanwhile, ST (space management), HW (high-water mark extension), and SQ (sequence number) enqueues are in the Conguration class, as these can usually be alleviated by changes in object and database settings. The User I/O class includes the db le scattered read, db le sequential read, direct path read, and direct path write events as one might expect, while the System I/O class includes many waits related to reading and writing of redo logs and archive logs. The Commit class has one member, log le sync, as that wait is caused by commits. And the Idle class is made up of various wait events which have traditionally been considered idle events, such

3 of 32

as SQL*Net message from client. It should be noted that sometimes such idle events can actually be symptoms of the root cause of poor performance, so they should not be disregarded without consideration. In general, the addition of wait classes helps direct the DBA more quickly toward the root cause of performance problems.

v$ View Enhancements
In Oracle 10g there are quite a few new v$ views that pertain to wait events, and helpful new columns have been added to existing v$ views. We will discuss several of the enhancements in this section, although some new v$ views are part of major new Oracle 10g functionality and will be discussed in separate sections later on. v$event_name Three columns have been added to the v$event_name view in Oracle 10g: wait_class_id, wait_class#, and wait_class. These columns show which wait class the wait event is part of. We saw in the previous section how this new information might be used. The columns in v$event_name now are:
SQL> DESCRIBE v$event_name Name Null? ----------------------------------------- -------EVENT# EVENT_ID NAME PARAMETER1 PARAMETER2 PARAMETER3 WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS

Type ---------------------------NUMBER NUMBER VARCHAR2(64) VARCHAR2(64) VARCHAR2(64) VARCHAR2(64) NUMBER NUMBER VARCHAR2(64)

v$sql and v$sqlarea The v$sql and v$sqlarea views have six new columns in Oracle 10g that relate to wait events: application_wait_time concurrency_wait_time cluster_wait_time user_io_wait_time plsql_exec_time java_exec_time These columns are designed to identify the amount of time a SQL statement spends in PL/SQL or Java code execution, or waiting in four specic wait classes. The Oracle release 10.1.0.3 documentation provides almost no information concerning these new columns, though they could be extremely important in diagnosing performance problems. In the absence of documentation, we will use an example to demonstrate the behavior and value of these new columns. Suppose we create a table called testtab with about a million rows. We then run the following statement from one session without committing, and then run the same statement from another session:
SQL> UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000;

Obviously the second session will wait on an enqueue wait event until the rst session either commits or rolls back. After a while we roll back the rst session and then the second session. Next, in a third session, we run the following statement:
SQL> UPDATE testtab SET numcol = numcol + 1;

The instance were using has a small buer cache, so quite a bit of physical I/O is caused by the statement. After the UPDATE completes we look at v$sqlarea:

4 of 32

SQL> SELECT sql_id, application_wait_time appl, concurrency_wait_time concurr, 2 user_io_wait_time user_io 3 FROM v$sqlarea 4 WHERE sql_text LIKE 'UPDATE testtab SET numcol%'; SQL_ID APPL CONCURR USER_IO ------------- --------- --------- ----------038m56cp4am0c 178500000 0 20000 fd5mxhdbf09ny 0 10000 105040000 SQL> SELECT sql_id, sql_text 2 FROM v$sqlarea 3 WHERE sql_id IN ('fd5mxhdbf09ny','038m56cp4am0c'); SQL_ID ------------038m56cp4am0c fd5mxhdbf09ny SQL_TEXT ------------------------------------------------------------UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000 UPDATE testtab SET numcol = numcol + 1

So we see that the rst statement (locking rows) spent 178.5 seconds (178,500,000 microseconds) waiting on events in the Application wait class and 0.02 seconds waiting on events in the User I/O wait class. If we recall from the discussion of wait classes, the TX enqueue for row contention is in the Application wait class, and data le reads are in the User I/O wait class. The second statement, which required more disk reads, shows 105 seconds of User I/O waits and a very small amount of concurrency waits. While the currently available documentation from Oracle does not provide much information about these new columns in v$sql and v$sqlarea, they appear to hold much promise for diagnosing query performance problems. v$session_wait_history Up through Oracle 9i, the v$ views show us only the most recent wait event for each session. Even though wait times can accumulate to greatly slow down a process, many waits are very short (from a human perspective) individually. So, its often dicult to grab information on a wait event as it is happening. The v$session_wait_history view, new in Oracle 10g, helps by showing the last ten wait events each session has experienced As an example, the following query shows the ten most recent wait events for session 154:
SQL> 2 3 4 SELECT FROM WHERE ORDER BY sid, seq#, event, wait_time, p1, p2, p3 v$session_wait_history sid = 154 seq#;

SID SEQ# EVENT WAIT_TIME P1 P2 P3 --- ---- ------------------------ ---------- ------ ------ -----154 1 db file sequential read 28 4 3547 1 154 2 log buffer space 18 0 0 0 154 3 log buffer space 36 0 0 0 154 4 db file sequential read 0 4 3559 1 154 5 db file sequential read 0 4 1272 1 154 6 db file sequential read 0 4 3555 1 154 7 log buffer space 9 0 0 0 154 8 db file sequential read 0 4 3551 1 154 9 db file sequential read 6 4 1268 1 154 10 log buffer space 8 0 0 0

The seq# column is supposed to show the chronological sequence of the wait events, with 1 being the most recent wait event in the session. On our release 10.1.0.3 databases on Solaris, the seq# column behaves dierentlymaking it dicult to tell which wait event is the most recent. In any case, note that this seq# value diers from the seq# column in v$session, which is incremented for each wait experienced by the session throughout the life of the session. In the above query, we see that the sessions most recent waits alternated between single-block disk reads and log buer space. This makes sense, since the SQL that the session was performing looked like:
INSERT SELECT FROM WHERE INTO table1 (column1, column2) column1, column2 table2 ...

5 of 32

From this list of recent waits, we can also drill down to get more detail. The p1 and p2 values for db le sequential read indicate the le and block numbers being read, so we can quickly determine what segment was being read. The columns in v$session_wait_history are:
SQL> DESCRIBE v$session_wait_history Name Null? ----------------------------------------- -------SID SEQ# EVENT# EVENT P1TEXT P1 P2TEXT P2 P3TEXT P3 WAIT_TIME WAIT_COUNT

Type ---------------------------NUMBER NUMBER NUMBER VARCHAR2(64) VARCHAR2(64) NUMBER VARCHAR2(64) NUMBER VARCHAR2(64) NUMBER NUMBER NUMBER

v$session The v$session view has been enhanced in Oracle 10g with several new columns which are quite useful. The wait event columns from v$session_wait have been added to v$session. In previous releases of Oracle, to get more detailed information about a session experiencing waits (such as what SQL the waiting session is executing), we had to join v$session_wait with v$session, as in:
SQL> 2 3 4 5 SELECT s.sid, w.state, w.event, w.seconds_in_wait siw, s.sql_address, s.sql_hash_value hash_value, w.p1, w.p2, w.p3 FROM v$session s, v$session_wait w WHERE s.sid = w.sid AND s.sid = 154;

In Oracle 10g we can get all of this information from v$session:

SQL> SELECT sid, state, event, seconds_in_wait siw, 2 sql_address, sql_hash_value hash_value, p1, p2, p3 3 FROM v$session 4 WHERE sid = 154; SID STATE EVENT SIW SQL_ADDRESS HASH_VALUE P1 P2 P3 --- ------- ----------------------- --- ---------------- ---------- --- ---- --154 WAITING db file sequential read 1 000000038551E820 3625097388 4 9813 1

Two additional columns have been added to v$session that are helpful for wait event analysis: blocking_session and blocking_session_status. The blocking_session column contains the session id (SID) of the holder of the resource that the waiting session is waiting for. The blocking_session_status column indicates the validity of the contents of the blocking_session column. If blocking_session_status is VALID, a valid SID is present in the blocking_session column. In the past, if we saw that a session was waiting for a resource, we would have to do additional queries to determine who was holding the resource. If the resource was an enqueue lock, we would have to query v$lock (sometimes a very expensive query) to determine who was holding the lock. Now we can nd out who is holding the lock much more quickly:
SQL> SELECT sid, blocking_session, username, 2 blocking_session_status status 3 FROM v$session 4 WHERE blocking_session_status = 'VALID'; SID BLOCKING_SESSION USERNAME STATUS --- ---------------- -------- ----------154 157 TSUTTON VALID

If we combine this with the wait event information now available in v$session, we see:

6 of 32

SQL> SELECT sid, blocking_session, username, 2 event, seconds_in_wait siw 3 FROM v$session 4 WHERE blocking_session_status = 'VALID'; SID BLOCKING_SESSION USERNAME EVENT SIW --- ---------------- -------- ------------------------------ --154 157 TSUTTON enq: TX - row lock contention 318

v$event_histogram The v$system_event view shows the number of waits, total time waited, and average wait time for a given wait event name (system-wide since instance startup). However, this aggregation can cloud the picture, because a small number of long waits can skew the data. For example, consider the following query from v$system_event:
SQL> SELECT event, total_waits, time_waited, average_wait 2 FROM v$system_event 3 WHERE event = 'enq: TX - row lock contention'; EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT ----------------------------- ----------- ----------- -----------enq: TX - row lock contention 17218 2101966 122

We see that there have been 17,218 waits and that the average wait time was 1.22 seconds, but we have no idea how the wait times are distributed. Were all of these waits roughly the same length? Were most of them under one second long and a few really long waits threw o the average? We cant tell from v$system_event. However, in Oracle 10g we can look at v$event_histogram for a more complete picture:
SQL> SELECT event, wait_time_milli, wait_count 2 FROM v$event_histogram 3 WHERE event = 'enq: TX - row lock contention'; EVENT WAIT_TIME_MILLI WAIT_COUNT ----------------------------- --------------- ---------enq: TX - row lock contention 1 833 enq: TX - row lock contention 2 635 enq: TX - row lock contention 4 372 enq: TX - row lock contention 8 395 enq: TX - row lock contention 16 781 enq: TX - row lock contention 32 3729 enq: TX - row lock contention 64 3050 enq: TX - row lock contention 128 410 enq: TX - row lock contention 256 47 enq: TX - row lock contention 512 46 enq: TX - row lock contention 1024 37 enq: TX - row lock contention 2048 3 enq: TX - row lock contention 4096 6880

We see that very few of the waits were anywhere near 1.22 seconds. Nearly 60% of the waits were less than 0.128 seconds (with most of those between 16 milliseconds and 64 milliseconds), and most of the remaining waits were between 2.048 seconds and 4.096 seconds (at which point some timed out and started new waits). In this way, the v$event_histogram view gives us a more accurate picture of wait times summarized by event name. The columns of v$event_histogram are:
SQL> DESCRIBE v$event_histogram Name Null? ----------------------------------------- -------EVENT# EVENT WAIT_TIME_MILLI WAIT_COUNT

Type ---------------------------NUMBER VARCHAR2(64) NUMBER NUMBER

v$system_wait_class and v$session_wait_class The two new views v$system_wait_class and v$session_wait_class enable us to get system and session wait information summarized by wait classes. This gives us a higher level view of what is happening in the system or

7 of 32

session, rather than focusing on individual events. The views are roughly equivalent to the views v$system_event and v$session_event, except that they roll up the events by wait class. The wait times are expressed in centiseconds since instance startup for v$system_wait_class and centiseconds since session connection for v$session_wait_class. The following queries show how much time (in centiseconds) has been spent on waits in each class across the system since instance start and for one specic session since that session began:
SQL> SELECT wait_class, time_waited 2 FROM v$system_wait_class 3 ORDER BY time_waited DESC; WAIT_CLASS TIME_WAITED ------------- ----------Idle 777450022 System I/O 1261584 User I/O 116667 Configuration 116481 Application 72301 Other 12432 Commit 3496 Concurrency 319 Network 1 SQL> 2 3 4 SELECT FROM WHERE ORDER BY wait_class, time_waited v$session_wait_class sid = 154 time_waited DESC;

WAIT_CLASS TIME_WAITED ------------- ----------Idle 612453 User I/O 1500 Configuration 28 Commit 11 Other 0 Application 0 Network 0

Since the wait times shown in these views are aggregations since system or session startup, these views are best used by taking samples and comparing the results to determine waits over a period of time. For instance, you could get data for the entire instance at time T1:
DROP TABLE swc_snap; CREATE TABLE swc_snap AS SELECT wait_class, total_waits, time_waited FROM v$system_wait_class;

And then, at time T2 a while later, get a summary of waits between T1 and T2:
SELECT FROM WHERE AND ORDER BY a.wait_class, (a.time_waited - b.time_waited) tm_waited v$system_wait_class a, swc_snap b a.wait_class = b.wait_class a.total_waits > NVL (b.total_waits, 0) tm_waited DESC;

WAIT_CLASS TM_WAITED --------------- ---------Idle 255767 Application 171 System I/O 156 User I/O 44 Other 21 Commit 13 Network 1

Active Session History

8 of 32

In previous releases of Oracle, the detailed information displayed in v$session_wait could prove extremely helpful in diagnosing performance problemsif you queried the view at the right time. The v$session_wait_history view in Oracle 10g makes it a little easier to catch detailed information by preserving the last ten waits for each session. But what if you want detailed information about a sessions waits for a period further back in time? This is where the Active Session History feature of Oracle 10gASH for shortcomes in handy. ASH makes detailed information about a sampling of waits encountered by all sessions available to us for a very long time. In Oracle 10g, a new background daemon process called MMNL queries v$session once each second and stores information about all active sessions in a circular buer in memory accessible by a new view called v$active_session_history. How far back you can look at sessions in this view depends on session activity and how much memory Oracle allocated for ASH. Oracles goal is to keep at least a few hours of session data available in this view. The v$active_session_history view includes much of the detailed wait event information shown in v$session:
SQL> DESCRIBE v$active_session_history Name Null? ----------------------------------------- -------SAMPLE_ID SAMPLE_TIME SESSION_ID SESSION_SERIAL# USER_ID SQL_ID SQL_CHILD_NUMBER SQL_PLAN_HASH_VALUE SQL_OPCODE SERVICE_HASH SESSION_TYPE SESSION_STATE QC_SESSION_ID QC_INSTANCE_ID EVENT EVENT_ID EVENT# SEQ# P1 P2 P3 WAIT_TIME TIME_WAITED CURRENT_OBJ# CURRENT_FILE# CURRENT_BLOCK# PROGRAM MODULE ACTION CLIENT_ID

Type ---------------------------NUMBER TIMESTAMP(3) NUMBER NUMBER NUMBER VARCHAR2(13) NUMBER NUMBER NUMBER NUMBER VARCHAR2(10) VARCHAR2(7) NUMBER NUMBER VARCHAR2(64) NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER VARCHAR2(48) VARCHAR2(48) VARCHAR2(32) VARCHAR2(64)

As you can see, v$active_session_history captures the essential wait-related data from v$session. It also captures useful information about the SQL statement currently being executed, as well as current object number, le, and block being accessed. When a wait that was sampled by ASH completes, Oracle lls in the time_waited column for the row in v$active_session_history with the actual duration of the wait. The Automatic Workload Repository, which we will discuss in the next section, writes data from v$active_session_history to disk at regular intervals, preserving one sample every ten seconds from each active session. So, active session information remains accessiblealthough with less detaileven after the data has aged out of v$active_session_history. Because ASH is always on, you always have access to detailed information about waits encountered in sessions within the last few hours. This means that if a user complains about a performance problem, you may be able to query v$active_session_history and gain insight into the problem without having to initiate an extended SQL trace or start a close watch of v$session while they reproduce the problem. An important thing to keep in mind about v$active_session_history, however, is that it is populated by sampling v$session once each second. A session may encounter many dierent waits during a one second period, but only the one wait that was in progress when ASH collected its sample will be recorded in v$active_session_history. For this reason, ASH is valuable for general aggregate queries but not for precise counting of individual events

9 of 32

or determining minimum or maximum wait times. Statistically speaking, the data collected by ASH is probably more accurate over a larger time interval and/or number of sessions. For example, you might query v$active_session_history to see what percentage of time over the last two hours a particular group of sessions spent waiting on disk reads. However, using this view to determine how many disk read waits a session encountered in the last minute probably will not yield very accurate results. Even though ASH data is only a sampling of active sessions, the information can prove to be quite useful. For example, the following query shows that sessions running the ARXENV application over the last two hours encountered a great deal of row-level lock contention:
SQL> 2 3 4 5 6 7 DECODE (session_state, 'WAITING', event, NULL) event, session_state, COUNT(*), SUM (time_waited) time_waited FROM v$active_session_history WHERE module = 'ARXENV' AND sample_time > SYSDATE - (2/24) GROUP BY DECODE (session_state, 'WAITING', event, NULL), session_state; SELECT

EVENT SESSION_STATE COUNT(*) TIME_WAITED ------------------------------ ------------- -------- ----------ON CPU 124 0 log file sync WAITING 2 52686 db file scattered read WAITING 2 28254 db file sequential read WAITING 1 6059 control file sequential read WAITING 1 9206 SQL*Net break/reset to client WAITING 1 9140 enq: TX - row lock contention WAITING 922 930864016

In addition to running queries against the v$active_session_history view, you can use Enterprise Manager to run reports that will display ASH data. Although ASH runs on all Oracle 10g databases by default, you are not allowed to query the v$active_session_history view (or run the corresponding reports in Enterprise Manager) unless you have purchased the Diagnostic Pack.

Automatic Workload Repository

Oracle 10g includes another signicant facility relevant to the wait event interface. Its known as the Automatic Workload Repositoryor AWR for short. AWR is basically a next-generation Statspack. By default, AWR collects an hourly snapshot of database performance information, storing the data in tables in the SYS schema. AWR is congured automatically when you create an Oracle 10g database. You can call the dbms_workload_repository package to collect a snapshot on demand, purge a snapshot or range of snapshots, or change the snapshot interval or retention period. (By default snapshots are collected at the top of each hour and are purged after seven days.) AWR collects the same type of data that Statspack doesincluding system-level statistics, resource-intensive SQL, and of course instance-wide wait event information. AWR also collects data that is new for Oracle 10g, such as time model statistics (which we will discuss in the next section). As an aside, the Oracle 10g version of Statspack also collects a lot of this new information, including time model statistics. You can generate an AWR report of database activity between two snapshots by running the awrrpt.sql script in the rdbms/admin directory under $ORACLE_HOME. This script oers reports formatted as plain text or HTML. The reports will look familiar if you have used Statspack before. You can use Enterprise Manager to generate AWR reports as well. AWR oers many benets over Statspack. For one, it is more tightly integrated into the Oracle kernel, reducing resource requirements and overhead when collecting snapshots. AWR snapshots also include ASH data from v$active_session_history, providing session-level information to complement the system-level data collection familiar to Statspack users. Data collected by AWR is made easily accessible via views with names that start DBA_HIST. This enables you to write your own reports that extract just the data you need to address a specic situation, if for some reason you dont nd what you need in the standard AWR report. For example, the following query displays the two most recent snapshot IDs:

10 of 32

SQL> SELECT snap_id, begin_interval_time, end_interval_time 2 FROM ( 3 SELECT snap_id, begin_interval_time, end_interval_time 4 FROM dba_hist_snapshot 5 ORDER BY end_interval_time DESC 6 ) 7 WHERE ROWNUM <= 2; SNAP_ID ---------362 361 BEGIN_INTERVAL_TIME ------------------------10-MAR-05 04.00.02.018 PM 10-MAR-05 03.00.25.885 PM END_INTERVAL_TIME ------------------------10-MAR-05 05.00.36.581 PM 10-MAR-05 04.00.02.018 PM

Just like ASH, AWR runs on all Oracle 10g databases by default. Also like ASH, you are not allowed to query the AWR views (or run AWR reports) unless you have licensed the Diagnostic Pack. Because AWR consumes system resources when collecting snapshots and uses up storage in the SYSAUX tablespace, you may want to disable the collection of AWR snapshots if you are not licensed to use AWR. This may be done by using the dbms_workload_repository package. If AWR is not available to you, Statspack is still a good way to go in Oracle 10g.

Time Model Statistics

Oracle 10g introduces a new concept called Time Model Statistics. This information provides yet another way to see how time is spent, and with greater detail than was available previously. The v$sys_time_model view shows time model statistics for the entire system since instance startup, while the v$sess_time_model view shows time model statistics for each session since session start. The columns in these two views are as follows:
SQL> DESCRIBE v$sys_time_model Name Null? ---------------------------------------- -------STAT_ID STAT_NAME VALUE SQL> DESCRIBE v$sess_time_model Name Null? ---------------------------------------- -------SID STAT_ID STAT_NAME VALUE

Type --------------------------NUMBER VARCHAR2(64) NUMBER

Type --------------------------NUMBER NUMBER VARCHAR2(64) NUMBER

A sample query from v$sys_time_model shows the following:

SQL> SELECT stat_name, value / 1000000 seconds 2 FROM v$sys_time_model 3 ORDER BY seconds DESC; STAT_NAME SECONDS ------------------------------------------------ ---------DB time 80970.190 sql execute elapsed time 75057.271 DB CPU 44448.628 background elapsed time 29333.160 PL/SQL execution elapsed time 8824.538 background cpu time 5170.311 parse time elapsed 1270.147 hard parse elapsed time 838.068 PL/SQL compilation elapsed time 176.731 sequence load elapsed time 112.334 connection management call elapsed time 44.644 failed parse elapsed time 11.946 hard parse (sharing criteria) elapsed time 5.579 hard parse (bind mismatch) elapsed time 4.610 failed parse (out of shared memory) elapsed time 0.000 Java execution elapsed time 0.000 inbound PL/SQL rpc elapsed time 0.000

This query shows us a lot more information about how Oracle sessions have spent their time (categorically) than

11 of 32

v$sysstat and v$sesstat do. Of course, we have to know how to interpret this information before we can put it to work for us. Values in these views are shown in microseconds, and they do not include background processes unless background appears in the statistic name. The DB time statistic shows elapsed time spent on database calls (user processes only). This amounts to time spent on the CPU or waiting on non-idle wait events. For a description of the other time model statistics, see the v$sess_time_model view listing in the Oracle 10g Database Reference manual. From this query, among many other useful facts, we can see that no time has been spent executing Java, very little time has been spent hard parsing or compiling PL/SQL, background processes have used about 10% of the CPU time, and about 11% of the elapsed time for user sessions was spent on PL/SQL execution.

Tracing Facility Improvements

The extended SQL trace facility, also known as event 10046, allows us to capture in a trace le detailed information about every wait event encountered by a database session. This feature has been available in Oracle for a long time. However, Oracle 10g oers some helpful improvements in this area. Enabling extended SQL trace has always been a bit of a nuisance. In earlier releases of Oracle, you had to use a clumsy ALTER SESSION SET EVENTS statement orworse yetcall the undocumented dbms_system.set_ev procedure to set the 10046 event in another users session. In Oracle 8i the dbms_support package was introduced to make this step easier, but the package was missing from many releases of Oracle and usually was not installed by default. Oracle 10g introduces the new dbms_monitor package. This package, among many other things, makes it very easy to turn extended SQL trace on and o in any Oracle session. With one easy to remember call, you can turn extended SQL trace on or o, with wait events and/or bind variables captured in the trace le:
SQL> DESCRIBE dbms_monitor ... PROCEDURE SESSION_TRACE_DISABLE Argument Name -----------------------------SESSION_ID SERIAL_NUM PROCEDURE SESSION_TRACE_ENABLE Argument Name -----------------------------SESSION_ID SERIAL_NUM WAITS BINDS

Type ----------------------BINARY_INTEGER BINARY_INTEGER Type ----------------------BINARY_INTEGER BINARY_INTEGER BOOLEAN BOOLEAN

In/Out -----IN IN In/Out -----IN IN IN IN

Default? -------DEFAULT DEFAULT Default? -------DEFAULT DEFAULT DEFAULT DEFAULT

If the session_id parameter is not specied or set to NULL, your own session will be traced. Thus, the following two statements should be equivalent:
ALTER SESSION SET events '10046 trace name context forever, level 12'; EXECUTE dbms_monitor.session_trace_enable (waits=>TRUE, binds=>TRUE);

In Oracle 9i and earlier, extended SQL trace was easy to use if your application connected to the Oracle database via a dedicated server connection. If the shared server architecture was used, each shared server process that serviced a request from the session being traced would write its data to a separate trace le. Furthermore, tracing sessions in a connection pool environment became dicult because one Oracle session could actually process requests for many dierent end-user sessions. The dbms_monitor package in Oracle 10g addresses this problem. Instead of enabling extended SQL trace for a specic Oracle session, you can enable it for a specic client identier or combination of service, module, and action. Any time any Oracle session has the specied client identier or combination of service, module, and action, the session will be traced. Sessions can set or clear their client identier at will by calling the dbms_session package, and they can set their module and action by calling the dbms_application_info package. Suppose a web-based application uses a pool of 30 database connections to serve user requests and maintains a current_sessions table to keep track of the state of each end-user session. When a user clicks a button in their browser window, the application server receives the HTTP request and hands it o to an application server process. The application server process grabs a free database connection from the pool and accesses the database as necessary to service the request. It is likely that subsequent requests from the same user will be

12 of 32

processed using dierent database connections. In Oracle 9i and earlier it would have been very dicult to trace the one users database activity in this environment. The users database accesses are spread among multiple Oracle server processes in the connection pool. Moreover, each of those Oracle processes is handling requests from many dierent end users. With dbms_monitor in Oracle 10g, the application could be modied in a way to make extended SQL trace a whole lot easier. We mentioned that the application uses the current_sessions table to maintain state for each end user session. Each time the application server grabs a database connection from the pool, it could set the client identier for the Oracle session to the session_id from the current_sessions table before doing any database access for that end user session. Then the application could clear the client identier before returning the database connection to the pool. The Oracle calls could look like this:
EXECUTE dbms_session.set_identifier ('session_id174837492748'); ...do the work for this end user session... EXECUTE dbms_session.clear_identifier

To trace this end user session, we could now call the dbms_monitor package like this:
SQL> EXECUTE dbms_monitor.client_id_trace_enable > ('session_id174837492748', waits=>TRUE, binds=>TRUE);

This call to dbms_monitor will cause each Oracle process to write extended SQL trace data to a trace le for all calls that occur while the client identier for the session is set to the specied value. However, each Oracle process will write to its own trace le. This will cause the trace data to be split over multiple les. To address this problem, Oracle 10g provides a new command-line utility called trcsess. The trcsess utility reads multiple trace les and consolidates entries from the various les that meet the specied criteria into one trace le that can be processed by TKPROF. To consolidate the trace data for our current example, we could use the following commands:
$ cd $ORACLE_BASE/admin/$ORACLE_SID/udump $ trcsess output=/home/rschrag/case1403/case1403-trial1.trc \ clientid=session_id174837492748

In this way the dbms_monitor package and trcsess utility in Oracle 10g make it a lot easier to collect extended SQL trace data from an end users session when connection pooling or other session aggregation techniques are used by the application server tier.

Conclusion
Oracle 10g includes many enhancements to the wait event interface that should make performance management using wait event methodologies easier than ever. Some enhancements, such as descriptive wait event names, wait classes, and the session_trace_enable procedure in the dbms_monitor package, are simple conveniences that make our jobs easier. These enhancements dont give us any information or power that we didnt have before. Other enhancements, however, such as time model statistics, v$event_histogram, and new columns in v$sql and v$sqlarea, provide us with helpful timing and wait information that was previously unavailable. Although as of this writing documentation on these new features is incomplete and discussion on Metalink is surprisingly sparse, many of these enhancements will likely prove quite valuable to the Oracle DBA responsible for performance management of a complex Oracle system.

About the Authors

Terry Sutton, OCP , has been an Oracle DBA for eleven years, and has worked in the information technology area for 18 years. Since 2000, Terry has been a Senior Sta Consultant at Database Specialists, performing assignments ranging from production database administration to emergency troubleshooting with a particular focus on Oracle database performance tuning. You may contact Terry by email at tsutton@dbspecialists.com. Roger Schrag, OCP , has been an Oracle DBA and application architect for over 15 years. He began his career at Oracle Corporation on the Oracle Financials development team. In 1995, he founded Database Specialists, Inc., a boutique consulting rm specializing in Oracle database technology, remote administration, and performance

13 of 32

tuning. Since that time, Roger has focused his expertise in the area of performance optimization. Roger is a frequent speaker at Oracle OpenWorld and the International Oracle Users Group (IOUG) Live conferences, where he has frequently been voted in the Top 10% of speakers. Roger has been an Oracle Masters Class instructor for the IOUG and is Director of Conference Programming for the Northern California Oracle Users Group (NoCOUG). He can be reached at rschrag@dbspecialists.com. Still Looking for Help on this Subject?
Get a Consultation We would be happy to talk with you about our services and how our senior-level database team might help you. Call Database Specialists at 415-344-0500 or 888-648-0500 or fill out a free consultation request form. Complimentary Newsletter If you'd like to receive our complimentary monthly newsletter with database tips and new white paper announcements, sign up for The Specialist.

Appendix: Oracle 10g Wait Event Names

Name alter rbs oine alter system set dispatcher AQ Deallocate Wait AQ Proxy Cleanup Wait ARCH random i/o ARCH sequential i/o ARCH wait for archivelog lock ARCH wait for ow-control ARCH wait for net re-connect ARCH wait for netserver detach ARCH wait for netserver init 1 ARCH wait for netserver init 2 ARCH wait for netserver start ARCH wait for pending I/Os ARCH wait for process death 1 ARCH wait for process death 2 ARCH wait for process death 3 ARCH wait for process death 4 ARCH wait for process death 5 ARCH wait for process start 1 ARCH wait for process start 2 ARCH wait for process start 3 ARCH wait for process start 4 ARCH wait on ATTACH ARCH wait on c/f tx acquire 1 ARCH wait on c/f tx acquire 2 ARCH wait on DETACH ARCH wait on SENDREQ ASM background running ASM background starting ASM background timer ASM db client exists ASM mount : wait for heartbeat ASM PST query : wait for [PM][grp][0] grant Wait Class Administrative Administrative waited Conguration Idle System I/O System I/O Other Network Network Network Network Network Network System I/O Other Other Other Other Other Other Other Other Other Network Other Other Network Network Other Other Idle Other Administrative Cluster P1 P2 P3

14 of 32

Backup: sbtbackup Backup: sbtclose Backup: sbtclose2 Backup: sbtcommand Backup: sbtend Backup: sbterror Backup: sbtinfo Backup: sbtinfo2 Backup: sbtinit Backup: sbtinit2 Backup: sbtopen Backup: sbtpcbackup Backup: sbtpccancel Backup: sbtpccommit Backup: sbtpcend Backup: sbtpcquerybackup Backup: sbtpcqueryrestore Backup: sbtpcrestore Backup: sbtpcstart Backup: sbtpcstatus Backup: sbtpcvalidate Backup: sbtread Backup: sbtread2 Backup: sbtremove Backup: sbtremove2 Backup: sbtrestore Backup: sbtwrite Backup: sbtwrite2 BFILE check if exists BFILE check if open BFILE closure BFILE get length BFILE get name object BFILE get path object BFILE internal seek BFILE open BFILE read block change tracking buer space buer busy buer busy waits buer deadlock buer dirty disabled buer exterminate buer freelistbusy buer invalidation wait buer latch buer pool resize buer read retry buer rememberlist busy buer resize

Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Administrative Other Other Other Other Other Other Other Other User I/O Other Other Concurrency Other Other Other Other Other Other User I/O Other Other group# le# dba group# le# group# group# latch addr le# group# block# obj# obj# chain# current size block# obj# block# new size buf_ptr block# block# obj# block# block# class#

class*10+mode ag

Administrative buer pool id

15 of 32

buer write wait buer writeList full CGS skgxn join retry CGS wait for IPC msg change tracking le parallel write change tracking le synchronous write check CPU wait times checkpoint advanced checkpoint completed class slave wait Cluster stablization wait Cluster Suspension wait contacting SCN server or SCN lock master control le heartbeat control le parallel write control le sequential read control le single write cr request retry Data Guard broker: wait upon ORA-12850 error db le parallel read db le parallel write db le scattered read db le sequential read db le single write DBFG waiting for reply DBMS_LDAP: LDAP operation debugger command dedicated server timer DFS db le lock DFS lock handle DIAG dummy wait direct path read direct path read temp direct path write direct path write temp dispatcher listen timer dispatcher shutdown dispatcher timer DLM lock cancel DLM lock cvt S DLM lock cvt X DLM lock esc DLM lock esc X DLM lock open DLM lock open S

Other Other Other Other Other

group# group# retry count blocks block# block#

obj# obj#

block# block#

requests blocks blocks

change tracking le synchronous read Other Other Other Other Conguration Other Other Other Cluster Other System I/O System I/O System I/O Other Other User I/O System I/O User I/O User I/O User I/O Other Other Other Network Other Other Other User I/O User I/O User I/O User I/O Network Other Idle Other Other Other Other Other Other Other

group# slave id

les le# le# le# waiting for retrying the query to mask ORA-12850 error les requests le# le# le#

block# block# block# block#

requests blocks blocks

blocks interrupt block# block# block#

requests timeout blocks blocks blocks

wait event le# type|mode le number le number le number le number sleep time waited sleep time le group group group group group group obj# obj# obj# obj# obj# obj# block# block# block# block# block# block# id1 rst dba rst dba rst dba rst dba id2 block cnt block cnt block cnt block cnt

16 of 32

DLM lock open X DLM recovery lock convert DLM recovery lock open dma prepare busy dupl. cluster key enq: AD - allocate AU enq: AD - deallocate AU enq: AF - task serialization enq: AG - contention enq: AO - contention enq: AS - contention enq: AT - contention enq: AW - AW generation lock enq: AW - AW state lock enq: AW - AW$ table lock enq: AW - user access for AW enq: BR - le shrink enq: BR - proxy-copy enq: CF - contention enq: CI - contention enq: CL - compare labels enq: CL - drop label enq: CM - gate enq: CM - instance enq: CT - change stream ownership enq: CT - CTWR process start/stop enq: CT - global space management enq: CT - local space management enq: CT - reading enq: CT - state enq: CT - state change gate 1 enq: CT - state change gate 2 enq: CU - contention enq: DB - contention enq: DD - contention enq: DF - contention enq: DG - contention enq: DL - contention enq: DM - contention enq: DN - contention enq: DP - contention enq: DR - contention enq: DS - contention enq: DT - contention enq: DV - contention enq: DX - contention enq: FA - access le

Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other

group group group group dba name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode

obj# obj# obj# obj#

block# block# block# block#

group and disk AU number number group and disk AU number number task id workspace # workspace # 0 0 operation operation operation operation operation operation 0 opcode object # object # disk group # disk group # operation operation operation operation operation operation operation operation handle EnqMode disk group 0 disk group object # type 0 0 0 0 0 object # transaction entry # disk group number 0 generation object # 0 0 workspace # workspace # workspace # workspace # le # le # operation type 0 0 type type operation parm operation parm operation parm operation parm operation parm operation parm operation parm operation parm handle 0 type le # type 0 type 0 0 0 0 0 0 0 le number

Administrative name|mode

17 of 32

enq: FB - contention enq: FC - open an ACD thread enq: FC - recover an ACD thread enq: FD - Flashback coordinator enq: FD - Flashback on/o enq: FD - Marker generation enq: FG - FG redo generation enq race

Other Other Other Other Other Other

name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode

tablespace # disk group disk group Internal Internal Internal Internal disk group disk group disk group Log # Log # 0 disk group 0 disk group disk group 0 disk group tablespace # object # object # table space # 0 0 object # pool # 0 0 0 object # 0 view object # 0 service ID service ID service ID service ID service ID service ID service ID service ID service ID service ID service ID service ID service ID service ID 0

dba thread thread Internal Internal Internal Internal type type type zero zero 0 unused type thread thread 0 0 dba hash value 0 block 0 0 0 0 0/1 0/1 type 0 0 0 0 queue type queue type queue type queue type queue type queue type queue type queue type queue type queue type queue type queue type queue type queue type redo thread

enq: FD - Tablespace ashback on/o Other Other

enq: FG - LGWR redo generation enq Other race enq: FG - serialize ACD relocate enq: FL - Flashback database log enq: FL - Flashback db command enq: FM - contention enq: FR - contention enq: FS - contention enq: FT - allow LGWR writes enq: FT - disable LGWR writes enq: FU - contention enq: HD - contention enq: HP - contention enq: HQ - contention enq: HV - contention enq: HW - contention enq: IA - contention enq: ID - contention enq: IL - contention enq: IM - contention for blr enq: IR - contention enq: IR - contention2 enq: IS - contention enq: IT - contention enq: JD - contention enq: JI - contention enq: JQ - contention enq: JS - contention enq: JS - coord post lock enq: JS - coord rcv lock enq: JS - global wdw lock enq: JS - job chain evaluate lock enq: JS - job recov lock enq: JS - job run lock - synchronize enq: JS - q mem clnup lck enq: JS - queue lock enq: JS - running job cnt lock enq: JS - running job cnt lock2 enq: JS - running job cnt lock3 enq: JS - slave enq get lock1 enq: JS - slave enq get lock2 enq: KK - context Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other

Conguration name|mode

18 of 32

enq: KM - contention enq: KP - contention enq: KT - contention enq: MD - contention enq: MH - contention enq: ML - contention enq: MN - contention enq: MR - contention enq: MS - contention enq: MW - contention enq: OC - contention enq: OL - contention enq: OQ - xsoq*histrecb enq: OQ - xsoqhiAlloc enq: OQ - xsoqhiClose enq: OQ - xsoqhiFlush enq: OQ - xsoqhistrecb enq: PD - contention enq: PE - contention enq: PF - contention enq: PG - contention enq: PH - contention enq: PI - contention enq: PL - contention enq: PR - contention enq: PS - contention enq: PT - contention enq: PV - syncshut enq: PV - syncstart enq: PW - ush prewarm buers enq: PW - perwarm status in dbw0 enq: RB - contention enq: RF - atomicity enq: RF - new AI enq: RF - synch: per-SGA Broker metadata enq: RF - synchronization: chief enq: RF - synchronization: critical ai enq: RN - contention enq: RO - contention enq: RO - fast object reuse enq: RP - contention enq: RS - le delete enq: RS - persist alert level enq: RS - prevent aging list update enq: RS - prevent le delete enq: RS - read alert level

Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Application Other Other Other Other Other

name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode

type 0 plan # master object # 0 0 session ID 0 or le # master object # Schedule Id 1 hash value resource id resource id resource id resource id resource id parno 0 0 0 operation 0 0 instance disk group # 0 0 0 0 disk group

type 0 0 0 0 0 0 type 0 0 2 0 0 0 0 0 0 0 0 0 0 serial # 0 0 slave ID type 0 0 0 0 0

property name key hash

lock operation lock value lock operation lock value lock operation lock value lock operation lock value lock operation lock value lock operation lock value lock operation lock value thread number log number 1 1 le # record type record type record type record type record type 0 0 1 or block record id record id record id record id record id

enq: RF - synchronization: aifo master Other Other Other Other Application Application Other Other Other Other Other Other

enq: RF - synchronization: HC master Other

19 of 32

enq: RS - record reuse enq: RS - write alert level enq: RT - contention enq: SB - contention enq: SF - contention enq: SH - contention enq: SI - contention enq: SK - contention enq: SQ - contention enq: SR - contention enq: SS - contention enq: ST - contention enq: SU - contention enq: SW - contention enq: TA - contention enq: TB - SQL Tuning Base Cache Load enq: TB - SQL Tuning Base Cache Update enq: TC - contention enq: TC - contention2 enq: TD - KTF dump entries enq: TE - KTF broadcast enq: TF - contention enq: TL - contention enq: TM - contention enq: TO - contention enq: TQ - DDL contention enq: TQ - INI contention enq: TQ - TM contention enq: TS - contention enq: TT - contention enq: TW - contention enq: TX - allocate ITL entry enq: TX - contention enq: TX - index contention enq: TX - row lock contention enq: UL - contention enq: US - contention enq: WA - contention enq: WF - contention enq: WL - contention enq: WP - contention enq: XH - contention enq: XR - database force logging enq: XR - quiesce database enq: XY - contention extent map load/unlock

Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Application Other Other Other Other Other Other

name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode

record type record type redo thread 0 0 0 object # tablespace # object # operation tablespace # 0 table space # 0 operation 1 1 checkpoint ID checkpoint ID 0 0 tablespace # 0 object # object # QT_OBJ# QT_OBJ# QT_OBJ# tablespace ID tablespace ID 0

record id record id type 0 0 0 0 dba 0 sequence # / apply # dba 0 0 0 undo segment # / other 2 2 0 0 0 0 relative le # 0 table/partition 1 0 0 0 dba operation operation

Conguration name|mode

Conguration name|mode

Administrative name|mode Conguration name|mode Other Concurrency Application Application Other Other Other Other Other Other Other Other Other Other name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode name|mode group

usn<<16 | slot sequence usn<<16 | slot sequence usn<<16 | slot sequence usn<<16 | slot sequence id undo segment # 0 0 log # / thread id # 0 0 operation operation id1 le 0 0 0 0 sequence # 0 0 0 0 id2 extent

20 of 32

ashback buf free by RVWR ashback free VI log ashback log switch Flow Control Event foreground creation: start foreground creation: wait free buer waits free global transaction table entry free process state object gc assume gc block recovery request gc buer busy gc claim gc cr block 2-way gc cr block 3-way gc cr block busy gc cr block congested gc cr block unknown gc cr cancel gc cr disk read gc cr disk request gc cr failure gc cr grant 2-way gc cr grant busy gc cr grant congested gc cr grant unknown gc cr multi block request gc cr request gc current block 2-way gc current block 3-way gc current block busy gc current block congested gc current block unknown gc current cancel gc current grant 2-way gc current grant busy gc current grant congested gc current grant unknown gc current multi block request gc current request gc current retry gc current split gc domain validation gc freelist gc prepare gc quiesce wait gc recovery free gc recovery quiesce gc remaster gcs ddet enter server mode

Other Other Other Other Other Other Conguration le# Other Other Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Other le# le# block# block# id# id# le le# le# block# block# class# class# le# block# class# le le le# le# block# block# class# id# tries block# set-id#

21 of 32

gcs domain validation gcs drm freeze begin gcs drm freeze in enter server mode gcs enter server mode gcs log ush sync gcs remastering wait for read latch gcs remastering wait for write latch gcs remote message gcs to be enabled ges cached resource cleanup ges cancel ges cgs registration ges enter server mode ges generic event ges global resource directory to be frozen ges inquiry response ges lmd and pmon to attach ges LMD suspend for testing event ges LMD to inherit communication channels ges LMD to shutdown ges lmd/lmses to freeze in rcfg mrcvr ges lmd/lmses to unfreeze in rcfg mrcvr ges LMON for send queues ges LMON to get to FTDONE ges LMON to join CGS group

Other Other Other Other Other Other Other Idle Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other

cluinc

rcvinc

waittime

poll

event

waittime

poll

event

gcs resource directory to be unfrozen Other waittime

type|mode

id1

id2

ges master to get established for SCN Other op ges performance test completion ges pmon to exit ges process with outstanding i/o ges reconguration to start ges remote message ges resource cleanout during enqueue open ges resource cleanout during enqueue open-cvt ges reusing os pid ges user error ges wait for lmon to be ready ges1 LMON to wake up LMD - mrcvr ges2 LMON to wake up LMD - mrcvr Other Other Other Idle Idle Other Other waittime loop p3 pid

ges resource directory to be unfrozen Other Other Other Other Other Other pid error count

ges2 LMON to wake up lms - mrcvr 2 Other ges2 LMON to wake up lms - mrcvr 3 Other ges2 proc latch in rm latch get 1 ges2 proc latch in rm latch get 2 global cache busy Other Other Other group le# block#

22 of 32

global enqueue expand wait HS message to agent i/o slave wait imm op inactive session inactive transaction branch index (re)build online cleanup index (re)build online merge index (re)build online start index block split instance state change io done IPC busy async request IPC send completion sync IPC wait for name service busy IPC waiting for OSD resources job scheduler coordinator slave wait jobq slave wait JS coord start wait JS external job JS kgl get object wait JS kill job wait kcbzps kcrrrcp kdic_do_merge kfcl: instance recovery kfk: async disk IO kgltwait

Other Idle Other Other Other Other msg ptr msg ptr session# branch# waited waited mode mode mode level value wait wait wait childdba waited

GV$: slave acquisition retry wait time Other

Administrative object Administrative object Administrative object Other Other System I/O Other Other Other Other Other Idle Other Idle Other Other Other Other Other Other System I/O Other group count send count rootdba layer msg ptr

obj# intr

block# timeout

kjbdomalc allocate recovery domain Other retry kjbdrmcvtq lmon drm quiesce: ping completion kjbopen wait for recovery domain attach KJC: Wait for msg sends to complete kjctssqmg: quick message send wait kjudomatt wait for recovery domain attach kjudomdet wait for recovery domain detach kjxgrtest kkdlgon kkdlhpon kkdlsipon kksfbc child completion kksfbc research kkshgnc reloop kksscl hash split knlqdeq Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other msg dest|rcvr mtype

kjctcisnd: Queue/Send client message Other

23 of 32

knlWaitForStartup knpc_anq_AwaitNonemptyQueue knpsmai ksbcic ksbsrv ksdxexeother ksdxexeotherwait ksfd: async disk IO ksfd: b/fob latch ksim generic wait event ksqded ksv slave avail wait ksxr poll remote instances ktfbtgex ktm: instance recovery ktsambl kttm2d Kupp process shutdown kupp process wait kxfxse kxfxsp L1 validation latch activity latch free latch: cache buer handles latch: cache buers chains latch: cache buers lru chain latch: checkpoint queue latch latch: enqueue hash chains latch: gcs resource hash latch: ges resource hash list latch: In memory undo latch latch: KCL gc element parent latch latch: latch wait list latch: library cache latch: library cache lock latch: library cache pin latch: messages latch: MQL Tracking Latch latch: object queue header heap latch: object queue header operation latch: parallel query alloc buer latch: redo allocation latch: redo copy latch: redo writing latch: row cache objects latch: session allocation latch: shared pool

Other Other Other Other Other Other Other System I/O Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Other Concurrency Other Other Other Other Other Concurrency Other Other Concurrency Concurrency Concurrency Other Concurrency Other Other Other Other kxfxse debug wait: stalling for slave 0 kxfxsp debug wait: stalling for slave 0 seghdr address address address address address address address address address address address address address address address address address address address address address l1bmb number number number number number number number number number number number number number number number number number number number number number number number number number number process# tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries tries nalive sleeptime loop tsn undo segment# where wait_count count intr timeout

knpc_acwm_AwaitChangedWaterMark Other

Conguration address Conguration address Other Other address address

Conguration address

24 of 32

latch: undo global data latch: virtual circuit queues LGWR random i/o LGWR sequential i/o LGWR simulation latency wait LGWR wait for redo copy LGWR wait on ATTACH LGWR wait on DETACH LGWR wait on full LNS buer LGWR wait on LNS LGWR wait on SENDREQ LGWR-LNS wait on channel library cache load lock library cache lock library cache pin library cache revalidation listen endpoint status LNS simulation latency wait LNS wait for LGWR redo LNS wait on ATTACH LNS wait on DETACH LNS wait on LGWR LNS wait on SENDREQ local write wait lock close lock deadlock retry lock escalate retry lock release pending lock remastering log buer space log le parallel write log le sequential read log le single write log le switch (archiving needed) log le switch (checkpoint incomplete) log le switch (clearing log le) log le switch completion log le sync log switch/archive log write(even) log write(odd) master exit master wait MMON (Lite) shutdown MMON slave messages MRP wait on archivelog archival MRP wait on archivelog arrival MRP wait on archivelog delay MRP wait on process death

Other Other System I/O System I/O Other Other Network Network Other Network Network Other Concurrency Concurrency Concurrency Other Other Other Other Network Network Network Network User I/O Other Other Other Other Cluster Conguration System I/O System I/O System I/O Conguration Conguration Conguration Conguration Commit Other Other Other Other Other Other Other Other Other Other

address address

number number

tries tries

copy latch #

object address handle address handle address end-point#

lock address lock address pin address status

100*mask+namespace 100*mode+namespace 100*mode+namespace

le# group

block# lms#

group

le#

block#

les log# log#

blocks block# block#

requests blocks blocks

buer# group# group# alive slaves process# waited

Conguration thread#

25 of 32

MRP wait on process restart MRP wait on process start MRP wait on startup clear MRP wait on state change MRP wait on state n_a MRP wait on state reset

Other Other Other Other Other Other

multiple dbwriter suspend/resume for Administrative le oine name-service call wait no free buers no free locks null event OLAP Aggregate Client Deq OLAP Aggregate Client Enq OLAP Aggregate Master Deq OLAP Aggregate Master Enq OLAP Null PQ Reason OLAP Parallel Temp Grew OLAP Parallel Temp Grow Request OLAP Parallel Temp Grow Wait OLAP Parallel Type Deq opishd parallel recovery coordinator waits for cleanup of slaves pending global transaction(s) pi renounce write complete pipe get pipe put PL/SQL lock timer pmon timer process shutdown process startup PX create server PX Deq Credit: free buer PX Deq Credit: need buer PX Deq Credit: send blkd PX Deq: Execute Reply PX Deq: Execution Msg PX Deq: Index Merge Close PX Deq: Index Merge Execute PX Deq: Index Merge Reply PX Deq: Join ACK PX Deq: kdcph_mai PX Deq: kdcphc_ack PX Deq: Msg Fragment PX Deq: OLAP Update Close PX Deq: OLAP Update Execute PX Deq: OLAP Update Reply PX Deq: Par Recov Change Vector PX Deq: Par Recov Execute PX Deq: Par Recov Reply Other Other Other Other Other Other Other Other Other Other Other Other Other Other Idle Other Cluster Idle Concurrency Idle Idle Other Other Other Other Idle Other Idle Idle Idle Idle Idle Idle Idle Idle Idle Other Other Other Idle Idle Idle scans le# handle address handle address duration duration type type nservers process# process# sleeptime waited waited enqueue qref qref qref block# buer length record length timeout timeout sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes waittime group# obj# block#

sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes kdcph_mai kdcphc_ack sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes

26 of 32

PX Deq: Parse Reply PX Deq: reap credit PX Deq: Signal ACK PX Deq: Table Q Close PX Deq: Table Q Get Keys PX Deq: Table Q Normal PX Deq: Table Q qref PX Deq: Table Q Sample PX Deq: Test for msg PX Deq: Txn Recovery Reply PX Deq: Txn Recovery Start PX Deque wait PX Idle Wait PX Nsq: PQ descriptor query PX Nsq: PQ load info query PX qref latch PX Send Wait PX server shutdown PX signal server PX slave connection PX slave release qerex_gdml queue messages Queue Monitor IPC wait Queue Monitor Shutdown Wait Queue Monitor Slave Wait Queue Monitor Task Wait Queue Monitor Wait queue slave messages rdbms ipc message rdbms ipc message block rdbms ipc reply read by other session recovery area: computing applied logs recovery area: computing backed up les recovery area: computing dropped les recovery area: computing identical les recovery area: computing obsolete les recovery read refresh controlle command reliable message Replication Dequeue resmgr:become active resmgr:cpu quantum resmgr:internal state change resmgr:internal state cleanup

Idle Other Other Other Other Idle Other Idle Other Idle Idle Idle Idle Other Other Other Other Other Other Other Other Other Idle Idle Idle Idle Other Idle Other Idle Other Other User I/O Other Other Other Other Other System I/O Administrative Other Other Scheduler Scheduler Concurrency Concurrency

sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes sleeptime/senderid passes

function nalive serial

sleeptime sleeptime error

qref loop nbusy

queue id

process#

wait time

timeout from_process le# timeout block# class#

channel context location location location location

channel handle broadcast message

sleeptime/senderid passes

27 of 32

resmgr:sessions to exit retry contact SCN lock master rfc_open_retry

Concurrency Cluster Other

location DMON waiting to retry conguration le open DRCX waiting for site to delete metadata wait for INSV to shutdown wait for INSV to start NSVx to defer delete response message post to DMON NSVx metadata le close wait NSVx metadata le write wait NSVx to defer message post to DMON wait for NSVx to shutdown wait for NSVx to start letting site register with its local listener before connect ret retrying connection for sending to remote DRCX DMON waiting on the last gasp event DMON phase deferral wait wait for DMON to shutdown DMON waiting to timeout an operation PMON delete state object stall waiting for database to be opened reconnecting back to new primary site during standby viability c waiting for standby database to be mounted

r_drcx_site_del r_insv_shut r_insv_start

Other Other Other

r_nsv_deldef

Other

r_nsv_md_close r_nsv_md_write r_nsv_postdef r_nsv_shut r_nsv_start

Other Other Other Other Other

r_recon1

Other

r_recon2

Other

rfm_dmon_last_gasp rfm_dmon_pdefer rfm_dmon_shut rfm_dmon_timeout_op rfm_pmon_dso_stall rfrdb_dbop

Other Other Other Other Other Other

rfrdb_recon1

Other

rfrdb_recon2

Other

28 of 32

rfrdb_try235

Other

waiting for retrying the query to mask ORA-235 error waiting for logical apply engine to initialize checking for logical apply engine run-down progress waiting for new primary to initialize tables waiting for v$logstdby_stats view to be initialized waiting to reconnect to primary that is in BUILD_UP RSM notier: waiting for sql latch on db close RSM notier: waiting for sql latch on db open wait for subscriber count to become nonzero wait for RSMx processes to shutdown wait for RSMx to attach to state object wait for RSMx processes to start RSM stall due to event RSM_STALL wait for subscriber count to become zero waiting for MRP0 to stop while bringing physical apply engine of waiting for MRP0 to start while bringing physical apply engine o waiting for retrying potential dataloss calculation before switc

rfrla_lapp1

Other

rfrla_lapp2

Other

rfrla_lapp3

Other

rfrla_lapp4

Other

rfrla_lapp5

Other

rfrm_dbcl

Other

rfrm_dbop

Other

rfrm_nonzero_sub_count

Other

rfrm_rsm_shut

Other

rfrm_rsm_so_attach rfrm_rsm_start rfrm_stall rfrm_zero_sub_count

Other Other Other Other

rfrpa_mrpdn

Other

rfrpa_mrpup

Other

rfrxpt_pdl

Other

29 of 32

rfrxptarcurlog RFS announce RFS attach RFS close RFS create RFS detach RFS dispatch RFS ping RFS random i/o RFS register RFS sequential i/o RFS write rollback operations active rollback operations block full row cache lock row cache read RVWR wait for ashback copy scginq AST call secondary event select wait simulated log write delay single-task message slave exit slave shutdown wait slave TJ process wait smon timer sort segment request SQL*Net break/reset to client SQL*Net break/reset to dblink SQL*Net message from client SQL*Net message from dblink SQL*Net message to client SQL*Net message to dblink SQL*Net more data from client SQL*Net more data from dblink SQL*Net more data to client SQL*Net more data to dblink

Other Other Other Other Other Other Other Other System I/O Other System I/O System I/O Other Other Concurrency Concurrency Other Other Other Other Other Idle Other Other Other Idle Conguration Application Application Idle Idle Network Network Network Network Network Network

waiting for logical apply engine to nish initialization

operation count max operations cache id cache id copy latch # event # wait time mode address request times

nalive

sleeptime

loop

sleep time driver id driver id driver id driver id driver id driver id driver id driver id driver id driver id

failed break? break? #bytes #bytes #bytes #bytes #bytes #bytes #bytes #bytes

statement suspended, wait error to be Idle cleared STREAMS apply coord waiting for slave message STREAMS apply slave idle wait STREAMS apply slave waiting for coord message STREAMS capture process lter callback wait for ruleset STREAMS capture process waiting for archive log Streams: Wait for inter instance ack Idle Idle Application Idle Other

STREAMS fetch slave waiting for txns Idle Other

30 of 32

Streams: Wating for DDL to apply switch logle command switch undo - oine SWRF RWM Auto Capture Event SWRF Wait on Flushing Sync ASM rebalance test long ops timer in sksawat trace continue trace unfreeze trace writer ush trace writer I/O transaction txn to complete unbound tx undo segment extension undo segment recovery undo segment tx slot unspecied wait event virtual circuit status wait active processes wait for a paralle reco to abort wait for a undo record wait for activate message wait for another txn - rollback to savepoint wait for another txn - txn abort wait for another txn - undo rcv abort wait for assert messages to be sent wait for change Wait for Dictionary Build to lock all tables wait for EMON to die wait for EMON to process ntfns wait for EMON to spawn wait for FMON to come up wait for Logical Standby Apply shutdown wait for master scn wait for message ack wait for MTTR advisory state object wait for possible quiesce nish wait for record update wait for resize request completion wait for rr lock release wait for scn ack wait for SGA component shrink wait for sga_target resize Wait for shrink lock Wait for shrink lock2

Application Administrative Administrative Other Other Other Other Other Other Other Other Other Other Other Other

sleep time

delay time

undo seg#|slot#

wrap#

count

Conguration segment# Other Other Idle Other Other Other Idle Other Other Other Other Other Other Other Conguration Other Other Other Other Other Other Administrative Other Other Other Other Other Other Other Other object_id object_id lock_mode lock_mode pending_nd component id scnwrp current size scnbas target size waittime startscn ackscn circuit# status segment# tx ags Conguration segment#

wait for membership synchronization Other

31 of 32

wait for split-brain resolution wait for sync ack Wait for Table Lock wait for tmc2 to complete wait for transaction Wait for TT enqueue

Other Other Application Other Idle Other tsn channel context channel context cluinc address address channel handle channel handle count pending_insts number number process# tries cluinc pending_nd

wait for stopper event to be increased Other

wait for unread message on broadcast Idle channel wait for unread message on multiple broadcast channels wait for verication ack wait for votes wait list latch activity wait list latch free Wait on stby instance close waiting for low memory condition to be resolved waiting for subscribers to catch up waiting to get CAS latch waiting to get RM CAS latch wakeup blocked enqueuers wakeup event for builder wakeup event for preparer wakeup event for reader wakeup time manager write complete waits writes stopped by instance recovery or database suspension Idle Other Other Other Other Other Idle Idle Other Other Other Idle Idle Idle Idle

Conguration le# Other by thread#

block# our thread#

Copyright 2005 Database Specialists, Inc. http://www.dbspecialists.com

32 of 32