GATHER TABLE STATS IN PARALLEL Author JP Vijaykumar OORAle DBA Date Feb 14th 2011 In the recent years,

the oracle databases' sizes are increasing exponentially. Gathering stats on all the tables in oracle databases is a time consuming proces s. Gathering stats with estimate 30 percent on very large database takes few hours. How to reduce the process time of gathering tables' stats? ----------------------------------------------------------------------------------------------*************PARALLELIZING GATHER TABLE STATS PROCESS BASED ON THE ORDER OF THE TABLE_NAMES***** 01 GATHERING TABLE STATS IN PARALLEL BASED ON ALPHABETICAL ORDER OR THE TAB LE_NAMES --THIS SCRIPT GATHERs STATS ON SCHEMA TABLES IN PARALLEL BASED ON THE ALPHABETIC AL ORDER OF THE --TABLES IN THE SCHEMA. IN THIS PROCESS, THE SCRIPT RUNS 4 PROCESS IN PARALLEL T O GATHER STATS ON --THE SCHEMA TABLES. AT THE END, MANUALLY CHECK THE LOGS FOR ANY "ORA-" ERRORS A ND FIX THEM. oracle@linux01:/apps/oracle/scripts/jp> cat parallel_tabl_stats1.sh #!/bin/ksh export ORAINSTANCE=$1 export ORADBNAME=`echo ${ORAINSTANCE}|awk '{print substr($0,1,length($0)-1)}'` export OWNER=$2 export OPTION=$3 export OSLOGFILE=/apps/oracle/scripts/jp/${ORADBNAME}_${OWNER}_${OPTION}.log >${OSLOGFILE} export SCHEMAPWD=`cat /apps/oracle/scripts/password.dat|grep ${ORADBNAME}|grep $ {OWNER}|awk '{print $2}'` echo $ORADBNAME $ORAINSTANCE $OWNER $SCHEMAPWD . /apps/oracle/scripts/setenv ${ORAINSTANCE} ${ORACLE_HOME}/bin/sqlplus -s / >>${OSLOGFILE}<<EOF1 connect ${OWNER}/${SCHEMAPWD} show user select name,(select username from user_users) analyzed_schema,to_char(sysdate,'y yyy-mm-dd hh24:mi:ss') start_time from v\$database; set serverout on size 1000000 timing on declare v_part number:=$3; v_string varchar2(200):='AEFJKOPZ'; v_begin char(1); v_end char(1); v_num number; v_owner varchar2(20); begin execute immediate 'select username from user_users' into v_owner; if (v_part = 1) then

execute immediate 'select substr('''||v_string||''',1,1),substr('''||v_string||' '',2,1) from dual' into v_begin,v_end; elsif (v_part = 2) then execute immediate 'select substr('''||v_string||''',3,1),substr('''||v_string||' '',4,1) from dual' into v_begin,v_end; elsif (v_part = 3) then execute immediate 'select substr('''||v_string||''',5,1),substr('''||v_string||' '',6,1) from dual' into v_begin,v_end; else execute immediate 'select substr('''||v_string||''',7,1),substr('''||v_string||' '',8,1) from dual' into v_begin,v_end; end if; for c1 in (select table_name from user_tables where substr(table_name,1,1) betwe en v_begin and v_end order by 1) loop begin dbms_stats.gather_table_stats(ownname=>v_owner, tabname=>c1.table_name,estimate_ percent=>30, cascade=>TRUE,method_opt=>'FOR ALL INDEXED COLUMNS SIZE AUTO',degre e=>4); exception when others then dbms_output.put_line(c1.table_name||' '||sqlerrm); end; end loop; end; / select name,(select username from user_users) analyzed_schema,to_char(sysdate,'y yyy-mm-dd hh24:mi:ss') end_time from v\$database; quit EOF1 exit ----------------------------------------------------------------------------------------------linux01_oracle> cat parallel_tabl_stats_run1.sh #!/bin/ksh export INST_NAME=$1 export SCHEMA=$2 if [[ $# -ne 2 ]]; then echo "usage parallel_tabl_stats_run.sh INSTANCE_NAME SCHEMA" exit else parallel_tabl_stats1.sh ${INST_NAME} ${SCHEMA} 1 & parallel_tabl_stats1.sh ${INST_NAME} ${SCHEMA} 2 & parallel_tabl_stats1.sh ${INST_NAME} ${SCHEMA} 3 & parallel_tabl_stats1.sh ${INST_NAME} ${SCHEMA} 4 & fi exit ----------------------------------------------------------------------------------------------linux01024_oracle> ls -ltr |tail -6 -rwx------ 1 oracle users 2824 2011-01-06 21:21 parallel_tabl_stats1.sh

-rwx------ 1 oracle -rw-r--r-- 1 oracle -rw-r--r-- 1 oracle -rw-r--r-- 1 oracle -rw-r--r-- 1 oracle linux01_oracle>

users users users users users

190 412 412 412 412

2011-01-06 2011-01-06 2011-01-06 2011-01-06 2011-01-06

21:49 21:51 21:52 21:55 21:57

parallel_tabl_stats_run1.sh PROD_SCOTT_3.log PROD_SCOTT_2.log PROD_SCOTT_1.log PROD_SCOTT_4.log

-----------------------------------------------------------------------------------------------linux01_oracle> cat PROD_SCOTT_1.log START TIME: 01-12-2011 01:01:14 END TIME: 01-12-2011 01:01:39 PL/SQL procedure successfully completed. Elapsed: 00:00:24.44 ----------------------------------------------------------------------------------------------****************PARALLELIZING GATHER TABLE STATS PROCESS BASED ON NUMBER OF BLOC KS USED********* 02 GATHERING TABLE STATS IN PARALLEL BASED ON THE NUMBER OF BLOCKS ACQUIRED BY THE TABLES --PARALLELIZING THE GATHER SCHEMA TABLES' STATS BASED ON ALPHABETICAL ORDER MAY NOT BE EFFICIENT --IF THE LARGEST TABLES ARE CREATED WITH TABLE_NAMES STARTING WITH FEW ALPHABETS . --THE FOLLOWING SCRIPT GATHERS TABLE' STATS BASED ON THE NUMBER OF BLOCKS TABLE IS USING. --IN THIS SCRIPT, I HAD INTRODUCED AN OPTION TO PARALLELIZING THE NUMBER OF PARA LLEL PROCESSES --TO REDUCE THE EXECUTION TIME. IN THIS TEST RUN, I CHOSE 10 PARALLEL PROCESSES TO GATHER TABLE STATS. --CHOOSE THE NUMBER FOR PARALLEL PROCESSES, BASED ON THE CPUS ON YOUR SYSTEM AND --THE NUMBER OF PARALLEL PROCESSES THAT CAN BE SAFELY SUPPORTED ON YOUR SYSTEM. ----------------------------------------------------------------------------------------------oracle@linux01/apps/oracle/scripts/jp> cat parallel_tabl_stats2.sh #!/bin/ksh export ORAINSTANCE=$1 export ORADBNAME=`echo ${ORAINSTANCE}|awk '{print substr($0,1,length($0)-1)}'` export OWNER=$2 export OPTION=$3 export OSLOGFILE=/apps/oracle/scripts/jp/${ORADBNAME}_${OWNER}_${OPTION}.log >${OSLOGFILE} export SCHEMAPWD=`cat /apps/oracle/scripts/password.dat|grep ${ORADBNAME}|grep $ {OWNER}|awk '{print $2}'` echo $ORADBNAME $ORAINSTANCE $OWNER $SCHEMAPWD . /apps/oracle/scripts/setenv ${ORAINSTANCE} ${ORACLE_HOME}/bin/sqlplus -s / >>${OSLOGFILE}<<EOF1 connect ${OWNER}/${SCHEMAPWD}

show user select name,(select username from user_users) analyzed_schema,to_char(sysdate,'y yyy-mm-dd hh24:mi:ss') start_time from v\$database; set serverout on size 1000000 timing on declare v_opt number:=$3; v_max number:=$4; v_num number:=0; v_tot number:=0; begin for c1 in (select owner,segment_name,blocks from dba_segments where owner in (se lect username from user_users) --SYS_CONTEXT ('USERENV', 'SESSION_USER') and segment_type='TABLE' order by 2 desc ) loop begin v_num:=v_num+1; if ( v_num = v_opt and v_num = v_max ) then --dbms_output.put_line(c1.segment_name||' '||c1.blocks); dbms_stats.gather_table_stats(ownname=>c1.owner, tabname=>c1.segment_name,estima te_percent=>30, cascade=>TRUE,method_opt=>'FOR ALL INDEXED COLUMNS SIZE AUTO',degree=>4); v_tot:=v_tot+c1.blocks; v_num:=0; elsif ( v_num = v_opt and v_num <> v_max ) then --dbms_output.put_line(c1.segment_name||' '||c1.blocks); dbms_stats.gather_table_stats(ownname=>c1.owner, tabname=>c1.segment_name,estima te_percent=>30, cascade=>TRUE,method_opt=>'FOR ALL INDEXED COLUMNS SIZE AUTO',degre e=>4); v_tot:=v_tot+c1.blocks; elsif ( v_num <> v_opt and v_num = v_max ) then v_num:=0; end if; exception when others then dbms_output.put_line(c1.owner||' '||c1.segment_name||' '||sqlerrm); end; end loop; dbms_output.put_line(v_tot); end; / select name,(select username from user_users) analyzed_schema,to_char(sysdate,'y yyy-mm-dd hh24:mi:ss') end_time from v\$database; quit EOF1 exit ----------------------------------------------------------------------------------------------oracle@linux01:/apps/oracle/scripts/jp> cat parallel_tabl_stats_run2.sh #!/bin/ksh export INST_NAME=$1 export SCHEMA=$2 export NUM_PROC=$3 export CNT=1; if [[ $# -ne 3 ]]; then

echo "usage parallel_tabl_stats_run.sh INSTANCE_NAME SCHEMA NUM_PARALLEL_PROCESS ES" exit else while [[ $CNT -le $NUM_PROC ]] do #echo $INST_NAME $SCHEMA $CNT $NUM_PROC parallel_tabl_stats2.sh ${INST_NAME} ${SCHEMA} $CNT $NUM_PROC & CNT=`expr $CNT + 1` done fi exit ----------------------------------------------------------------------------------------------oracle@linux01:/apps/oracle/scripts/jp> parallel_tabl_stats_run2.sh PROD1 SCOTT 10 oracle@linux01:/apps/oracle/scripts/jp> PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger PROD PROD1 SCOTT tiger oracle@linux01:/apps/oracle/scripts/jp> ls -ltr|tail -10 -rw-r--r-- 1 oracle users 417 2011-02-04 19:50 PROD_SCOTT_10.log -rw-r--r-- 1 oracle users 418 2011-02-04 19:50 PROD_SCOTT_2.log -rw-r--r-- 1 oracle users 418 2011-02-04 19:50 PROD_SCOTT_5.log -rw-r--r-- 1 oracle users 418 2011-02-04 19:50 PROD_SCOTT_7.log -rw-r--r-- 1 oracle users 418 2011-02-04 19:50 PROD_SCOTT_3.log -rw-r--r-- 1 oracle users 418 2011-02-04 19:51 PROD_SCOTT_8.log -rw-r--r-- 1 oracle users 418 2011-02-04 19:51 PROD_SCOTT_4.log -rw-r--r-- 1 oracle users 494 2011-02-04 19:51 PROD_SCOTT_1.log -rw-r--r-- 1 oracle users 418 2011-02-04 19:51 PROD_SCOTT_9.log -rw-r--r-- 1 oracle users 419 2011-02-04 19:51 PROD_SCOTT_6.log Happy scripting. ******************************************************************************** **** ******************************************************************************** **** GATHER STATS IN PARALLEL Author JP Vijaykumar Oracle DBA Written Nov 08th 2013 01 The script takes three arguements db_name, schema_name, num_processes 02 This script was tested on AIX, Oracle 11G db. 03 Some of the conventions are specific to the test db. You need to modify the environment settings, as required on your systems. 04 The script was tested with 1 process - 100 proceses. 05 The script writes the output to a single log and email the log once all the processes completed. The script also writes the details of all tables/partitions analyzed into a s chema table.

06 All the shell scripts are consolidated into one script and a pl/sql script to handle parallel processing is introduced. cat parallel_tbl_stats_run.sh #!/bin/ksh ############################################# parallel_tbl_stats(){ export PASSWD=`/apps/oracle/scripts/getpss.sh $DBNAME $SCHEMA` echo $DBNAME $SCHEMA . /apps/oracle/scripts/go2 ${DBNAME} sqlplus -s / >> ${LOGNAM} <<EOF1 connect ${SCHEMA}/${PASSWD} set serverout on size 1000000 timing on declare v_opt number:=$OPTION; v_max number:=$NUMPRC; v_num number:=0; v_dgr number:=64; --v_dgr varchar2(30):=DBMS_STATS.AUTO_DEGREE; v_mth varchar2(100):='FOR ALL COLUMNS SIZE AUTO'; v_pct varchar2(30):=DBMS_STATS.AUTO_SAMPLE_SIZE; v_inv boolean:=TRUE;--FALSE --no_invalidate option v_tab varchar2(30):='GATHER_TBL_STATS_P'; --v_tot number:=0; begin execute immediate 'alter session set nls_date_format='''||'mm-dd-yyyy hh24:mi:ss '||''' '; if (v_opt = 1) then begin execute immediate 'drop table '||v_tab; exception when others then --dbms_output.put_line(v_tab||' '||sqlerrm); null; end; execute immediate 'create table '||v_tab||' as select '''|| 'STARTSTARTSTARTSTARTSTARTSTART'||''' table_name,'''||'STARTSTARTSTARTSTARTSTART START'||''' partition_name, SYSDATE run_date, '''||'N'||''' flag from dual'; execute immediate 'grant select on '||v_tab||' to public'; end if; for c1 in ( select table_name,partition_name from ( select table_name,null partition_name from user_tables where partitioned='NO' union all select p.table_name,p.partition_name from user_tab_partitions p, (select table_name, case when partitioning_type='RANGE' and partition_count > 2 then partition_count -1 else 1 end part_count from user_part_tables) t where p.table_name = t.table_name and p.partition_position >= t.part_count) where table_name not in (select job_name from user_datapump_jobs) and table_name not in (select distinct table_name from user_tab_statistics whe

re stattype_locked IS NOT NULL) and table_name not in (select table_name from user_tables where temporary='Y' ) ) loop begin v_num:=v_num+1; if ( v_num = v_opt and v_num = v_max ) then if (c1.partition_name is null) then dbms_stats.delete_table_stats(ownname=>'',tabname=>c1.table_name); dbms_stats.gather_table_stats(ownname=>'', tabname=>c1.table_name,estimate_perce nt=>v_pct, cascade=>TRUE,method_opt=>v_mth,degree=>v_dgr,no_invalidate=>v_inv); else dbms_stats.delete_table_stats(ownname=>'',tabname=>c1.table_name,partname=>c1.pa rtition_name); dbms_stats.gather_table_stats(ownname=>'', tabname=>c1.table_name,granularity => 'PARTITION', partname=>c1.partition_name,estimate_percent=>v_pct,cascade=>TRUE,method_opt=>v_ mth,degree=>v_dgr,no_invalidate=>v_inv); end if; execute immediate 'insert into '||v_tab||'(table_name,partition_name,run_date) v alues('''|| c1.table_name||''','''||c1.partition_name||''', to_char(sysdate,'''||'mm -dd-yyyy hh24:mi:ss'||'''))'; commit; v_num:=0; elsif ( v_num = v_opt and v_num <> v_max ) then if (c1.partition_name is null) then dbms_stats.delete_table_stats(ownname=>'',tabname=>c1.table_name); dbms_stats.gather_table_stats(ownname=>'', tabname=>c1.table_name,estimate_perce nt=>v_pct, cascade=>TRUE,method_opt=>v_mth,degree=>v_dgr,no_invalidate=>v_inv); else dbms_stats.delete_table_stats(ownname=>'',tabname=>c1.table_name,partname=>c1.pa rtition_name); dbms_stats.gather_table_stats(ownname=>'', tabname=>c1.table_name,granularity => 'PARTITION', partname=>c1.partition_name,estimate_percent=>v_pct,cascade=>TRUE,method_opt=>v_ mth,degree=>v_dgr,no_invalidate=>v_inv); end if; execute immediate 'insert into '||v_tab||'(table_name,partition_name,run_date) v alues('''|| c1.table_name||''','''||c1.partition_name||''', to_char(sysdate,'''||'mm -dd-yyyy hh24:mi:ss'||'''))'; commit; elsif ( v_num <> v_opt and v_num = v_max ) then v_num:=0; end if; exception when others then dbms_output.put_line(c1.table_name||' '||sqlerrm); end; end loop; end; /

quit EOF1 } ############################################# if [[ $# -ne 3 ]]; then echo "usage parallel_tbl_stats_run.sh DB_NAME SCHEMA_NAME NUM_PARALLEL_PROCESSES " exit fi cd /apps/oracle/scripts/jp export export export export export export DBNAME=$1 SCHEMA=$2 NUMPRC=$3 MAILST="jp[.]vijaykumar[@]gmail[.]com" LOGNAM=${DBNAME}_${SCHEMA}_stats_p.log OPTION=1;

> $LOGNAM while [[ $OPTION -le $NUMPRC ]] do #echo $DBNAME $SCHEMA $OPTION $NUMPRC #parallel_tabl_stats.sh ${DBNAME} ${SCHEMA} $OPTION $NUMPRC & parallel_tbl_stats ${DBNAME} ${SCHEMA} $OPTION $NUMPRC & OPTION=`expr $OPTION + 1` done wait uuencode $LOGNAM $LOGNAM|mailx -s "parallel tbl stats in $DBNAME $SCHEMA complet ed" $MAILST exit /* --TO CAPTURE TABLE_NAME,PARTITION_NAME,BYTES AND SORT THE CURSOR LOOP BASED ON A TABLE/PARTITION'S SIZE select table_name,p.partition_name,s.bytes from ( select table_name,null partition_name from user_tables where partitioned='NO' union all select p.table_name,p.partition_name from user_tab_partitions p, (select table_name, case when partitioning_type='RANGE' and partition_count > 2 then partition_count -1 else 1 end part_count from user_part_tables) t where p.table_name = t.table_name and p.partition_position >= t.part_count) p, user_segments s where table_name not in (select job_name from user_datapump_jobs) and table_name not in (select distinct table_name from user_tab_statistics whe re stattype_locked IS NOT NULL) and table_name not in (select table_name from user_tables where temporary='Y' ) and table_name = segment_name and nvl(p.partition_name,'XXX') = nvl(s.partition_name,'XXX') order by 3 --TO CHECK THE LAST_ANALYZED TIME FOR THE TABLES & PARTITIONS FROM USER_TAB_STAT ISTICS VIEW select s.table_name,s.partition_name,to_char(s.last_analyzed,'mm-dd-yyyy hh24:mi :ss') last_analyzed from (

select table_name,null partition_name from user_tables where partitioned='NO' union all select p.table_name,p.partition_name from user_tab_partitions p, (select table_name, case when partitioning_type='RANGE' and partition_count > 2 then partition_count -1 else 1 end part_count from user_part_tables) t where p.table_name = t.table_name and p.partition_position >= t.part_count) p, user_tab_statistics s where p.table_name not in (select job_name from user_datapump_jobs) and p.table_name not in (select distinct table_name from user_tab_statistics w here stattype_locked IS NOT NULL) and p.table_name not in (select table_name from user_tables where temporary='Y ' ) and p.table_name = s.table_name and nvl(p.partition_name,'XXX') = nvl(s.partition_name,'XXX') order by 3 */