Sei sulla pagina 1di 5

AWR report generation taking long time

One of the recent issue we encountered where in AWR report generation was taking
long time. We could find what the actual reason was but yet involved Oracle
support to get additional inputs which definitely helped. So without wasting much
time, lets get started.
Environment:
Database Version: Oracle version 11.2.0.2 Ent. Edition 64 bit
Operating System: Oracle Solaris 10 SPARC64
Findings:
In our case generating AWR report for 1 hour interval was taking about 90 to 120
minutes. Going by theory AWR data gets stored in SYSAUX tablespace, we looked
into the SYSAUX tablespace size and it was 115 GB approx.
We looked further into the AWR snapshot interval and retention period settings
and this was set to 10 minutes & 6 months respectively:
SQL> col snap_interval format a25
SQL> col RETENTION format a25
SQL> col MOST_RECENT_PURGE_TIME format a30
SQL> set lines 500
SQL> select snap_interval, retention, most_recent_purge_time from
sys.wrm$_wr_control;
SNAP_INTERVAL
----------------+00000 00:10:00.0

RETENTION
----------------+00182 12:00:00.0

MOST_RECENT_PURGE_TIME
-------------------------03-JUL-14 11.09.18.993 PM

Note: Default AWR snapshot interval is 1 hour and retention period is 7 days.
This explained the reason for the size of SYSAUX tablespace to be so high.
Further researching on Metalink found Doc ID 287679.1 which discusses the same
issue.
Using below query we could confirm that SYSAUX tablespace was mostly being used
by AWR & related components:
SQL> select occupant_name, occupant_desc, space_usage_kbytes from
v$sysaux_occupants where occupant_name like '%AWR%';
OCCUPANT_NAME OCCUPANT_DESC
------------- ---------------------------------------------------SM/AWR
Server Manageability - Automatic Workload Repository

SPACE_USAGE_KBYTES
----------------109505984

Orphaned ASH row count were as below:


SQL> SELECT count(1) Orphaned_ASH_Rows from wrh$_active_session_history a where
not exists (select 1 from wrm$_snapshot where snap_id = a.snap_id and dbid =
a.dbid and instance_number = a.instance_number);
Page 1

ORPHANED_ASH_ROWS
----------------108005585
Note: As we can see above we had considerable ORPHANED_ASH_ROWS. If we would have
deleted them in one go the amount of UNDO tablespace required and the time taken
could have been considerable. We could have deleted the rows in batches & then
committed, which would have helped with UNDO tablespace requirement but the time
taken would have been more.
Resolution Steps:
Below was the action plan we implemented which helped freeing up considerable
space in SYSAUX tablespace (approx. 75 GB) and resolved the AWR report generation
issue. All the steps can be performed online without any outage:
1. Set snapshot interval to 60 minutes and retention period to 31 days (i.e.
44640 minutes) based on the current requirement using below query:
SQL> execute dbms_workload_repository.modify_snapshot_settings (interval => 60,
retention => 44640);
PL/SQL procedure successfully completed.
Note: Initially above query was failing. Refer errors encountered section for
more details.
Verified the change using below query:
SQL> select snap_interval, retention, most_recent_purge_time from
sys.wrm$_wr_control;
SNAP_INTERVAL
----------------+00000 01:00:00.0

RETENTION
----------------+00031 00:00:00.0

MOST_RECENT_PURGE_TIME
-------------------------09-JUL-14 11.16.54.518 PM

2. Identify the minimum snap id that should be retained from the last 31 days:
SQL> select min(snap_id) from wrm$_snapshot where begin_interval_time > (sysdate -31);
MIN(SNAP_ID)
-----------313244

3. Took and export of below tables:


WRH$_ACTIVE_SESSION_HISTORY
WRH$_LATCH_MISSES_SUMMARY
WRH$_LATCH
WRH$_SYSMETRIC_SUMMARY
WRH$_SYSMETRIC_HISTORY
WRH$_EVENT_HISTOGRAM

Page 2

4. Truncated all the tables mentioned in step 3.


Note: Its not recommended you play with WRH$ tables. In our case we did consult
Oracle and based on the discussion implemented these steps.
5. Imported all the dumps taken in step 3 back in respective tables.
6. Once this was done the size occupied by AWR in SYSAUX was reduced drastically
along with the number of ORPHANED_ASH_ROWS count:
SQL> select occupant_name, occupant_desc, space_usage_kbytes from
v$sysaux_occupants where occupant_name like '%AWR%';
OCCUPANT_NAME OCCUPANT_DESC
------------- ---------------------------------------------------SM/AWR
Server Manageability - Automatic Workload Repository

SPACE_USAGE_KBYTES
----------------28648704

SQL> SELECT count(1) Orphaned_ASH_Rows from wrh$_active_session_history a where


not exists (select 1 from wrm$_snapshot where snap_id = a.snap_id and dbid =
a.dbid and instance_number = a.instance_number);
ORPHANED_ASH_ROWS
-----------------130126
Note: Commands used in step 3, 4 & 5 were as mentioned in attached file below:

Export_Truncate_Im
port.txt

7. Now deleted the orphaned rows using below query:


SQL> DELETE FROM wrh$_active_session_history a WHERE NOT EXISTS (SELECT 1 FROM
wrm$_snapshot WHERE snap_id = a.snap_id AND dbid = a.dbid AND instance_number =
a.instance_number);
130126 rows deleted.
SQL> commit;
Commit complete.
SQL> alter table wrh$_active_session_history move;
alter table wrh$_active_session_history move
*
ERROR at line 1:
ORA-14511: cannot perform operation on a partitioned object
The segment is partitioned :
Note: As per the discussion with Oracle we could ignore above errors
Page 3

SQL> select PARTITION_NAME, SEGMENT_NAME from dba_segments where segment_name


='WRH$_ACTIVE_SESSION_HISTORY' order by 1;
PARTITION_NAME
--------------WRH$_ACTIVE_2106565951_290608
WRH$_ACTIVE_SES_MXDB_MXSN

SEGMENT_NAME
----------------WRH$_ACTIVE_SESSION_HISTORY
WRH$_ACTIVE_SESSION_HISTORY

SQL> select min(snap_id) low , max(snap_id) high from dba_hist_snapshot where


END_INTERVAL_TIME<=sysdate-31;
LOW
HIGH
---------- ---------313449
313459
SQL> EXEC DBMS_WORKLOAD_REPOSITORY.DROP_SNAPSHOT_RANGE(low_snap_id=>313449,
high_snap_id=>313459);
PL/SQL procedure successfully completed.
8. We now tested generating AWR report which was taking approximately 5 to 6
minutes.
9. To further diagnose we enabled 10046 traces at session level using below set
of commands:
SQL>alter session set tracefile_identifier='10046_trace';
SQL>alter session set timed_statistics = true;
SQL>alter session set statistics_level=all;
SQL>alter session set max_dump_file_size = unlimited;
SQL>alter session set events '10046 trace name context forever,level 12';
Reproduce the issue by generating the AWR reports.
SQL>select * from dual;
SQL>alter session set events '10046 trace name context off';
Analyzing the traces we found that most of the time was spent on queries from
WRH$_SEG_STAT. This is because the data was very high in this table due to very
less snap interval that was set previously. So 5 minutes for generating the AWR
report the best we can achieve.
Errors Encountered:
1. Modifying AWR snapshot settings was failing as indicated:
SQL> execute dbms_workload_repository.modify_snapshot_settings (interval => 60,
retention => 44640);
ORA-13541:
(2678400)
ORA-06512:
ORA-06512:
ORA-06512:

system moving window baseline size (7862400) greater than retention


at "SYS.DBMS_WORKLOAD_REPOSITORY", line 174
at "SYS.DBMS_WORKLOAD_REPOSITORY", line 222
at line 1

Note: 2678400 (in seconds) is the retention period we are trying to set.
Page 4

44640 * 60 = 2678400
The current moving window can be get from DBA_HIST_BASELINE:
SQL> SELECT moving_window_size FROM dba_hist_baseline WHERE baseline_type =
'MOVING_WINDOW';
MOVING_WINDOW_SIZE
-----------------91
SQL> exec DBMS_WORKLOAD_REPOSITORY.modify_baseline_window_size(window_size=>31);
PL/SQL procedure successfully completed.
Now we were able to modify the AWR snapshot settings successfully:
SQL> execute dbms_workload_repository.modify_snapshot_settings (interval => 60,
retention => 44640);
PL/SQL procedure successfully completed.
2. Another issue we identified was ORA-12751 which basically indicates auto-purge
slave action faced lot of timeout errors. Check *m00*.trc present in
user_dump_destination for below errors to confirm this:
ORA-12751: cpu time or run time policy violation
There are known bugs where maintenance job responsible for statistics purging
fails and below are the recommended patches if you are running Oracle version
11.2.02 or 11.2.0.3 to get this fixed:
Patch 13901201 - WRH$_SQL_PLAN NOT PURGED LEADING TO EXCESSIVE SYSAUX TABLESPACE GROWTH
Patch 14373728 - NOT PURGING OLD STATISTICS FROM SYSAUX TABLESPACE
Patch 14084247 - STBH: ORA-01555 DUE TO WRH$_ACTIVE_SESSION_HISTORY NOT PURGED

Happy Reading..!!!

Page 5