Sei sulla pagina 1di 20

RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

ARCHIVE GAP

An archive gap means, a set of archived redo logs that could not be transmitted to the
Standby site from the Primary database. Mostly this problem occurs the network connectivity
becomes unavailable between the Primary and Standby site. When the network is available
again Data Guard resumes redo data transmission from the Primary to Standby site. Oracle
DG provides 2 methods for GAP resolution. They are AUTOMATIC and FAL.

Automatic Gap Resolution & FAL Gap Resolution

Lets consider an extended n/w failure occurred between the Primary and Standby machines
which causes the Standby is very far behind the Primary database, then an RMAN INCREMENTAL
BACKUP can be used to roll the Standby database forward faster than redo log apply.

When the archived logs are missing on the Standby database, simply we can ship missing logs
from the Primary to Standby database; (If missing logs are very less count e.g. (below 15).
We need to register all Shipped logs in the Standby database so that Gap can be resolved.

In this article I will demonstrate how to resolve archive log gaps using following methods.

Manually resolving a Gap (by Shipping logs from Primary to Standby)


To Roll Forward a Standby database using RMAN Incremental backup.

DISASTER RECOVERY CONFIGURATION

Primary Database 192.168.222.131 DEV Database Server


Standby Database 192.168.222.132 UAT Database Server

Primary database db_unique_name crms


Standby database db_unique_name stbycrms

MANUALLY RESOLVING GAPS

# On Primary database

SYS> select name, db_unique_name, database_role from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE


--------- ------------------------------ ----------------
CRMS crms PRIMARY

# On Standby database

SYS> select name, db_unique_name, database_role from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE


--------- ------------------------------ ---------------- ------------
CRMS stbycrms PHYSICAL STANDBY MOUNTED

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

# On Primary database

SYS> select thread#, max(sequence#) from v$archived_log group by thread#;

THREAD# MAX(SEQUENCE#)
---------- --------------
1 551

# On Standby database

SYS> select thread#, max(sequence#) from v$archived_log


where applied='YES' group by thread#;

THREAD# MAX(SEQUENCE#)
---------- --------------
1 551

# Standby database is waiting for SEQ# 552

SYS> select process, sequence#, status from v$managed_standby;

PROCESS SEQUENCE# STATUS


--------- ---------- ------------
ARCH 551 CLOSING
ARCH 558 CLOSING
ARCH 0 CONNECTED
ARCH 0 CONNECTED
RFS 0 IDLE
RFS 0 IDLE
RFS 559 IDLE
RFS 0 IDLE
MRP0 552 WAIT_FOR_GAP

9 rows selected.

# Snippet of Standby alert log

$ cd /u01/app/oracle/diag/rdbms/stbycrms/stbycrms/trace
$ tail -f alert_stbycrms.log

Fetching gap sequence in thread 1, gap sequence 552-556


..
...
FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 552-556
DBID 1613387466 branch 913081878
FAL[client]: All defined FAL servers have been attempted.

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

DETECTING GAPS

Oracle Data Guard provide us with a simple view (v$archive_gap) to detect a gap.

On Standby database

SYS> select * from v$archive_gap;

THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#


---------- ------------- --------------
1 552 556

Output of Standby database is currently missing log files from sequence# 552 to 556; the
Standby database is 5 logs behind the Primary database. ORACLE NOTE: Refer BUG #10072528
V$ARCHIVE_GAP may not detect archive gap when Physical Standby is open read only.

FIND ARCHIVE LOGS AT PRIMARY SERVER

After identifying a gap (as shown above), as a DBA you need to query the primary database
to locate the archived redo logs on the primary database. I have configured the local
archive destination on primary is LOG_ARCHIVE_DEST_1.

# On Primary database (SEQUENCE 552 to 556)

SYS> select name


FROM v$archived_log
WHERE thread# = 1
AND dest_id = 1
AND sequence# BETWEEN 552 and 556;

NAME
--------------------------------------------------------------------------------------
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_552_cnvodqq5_.arc
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_553_cnvohcy1_.arc
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_554_cnvokk6z_.arc
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_555_cnvom1l4_.arc
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_556_cnvomxkz_.arc

Copy the above redo log files to the Physical Standby database and register them using the
ALTER DATABASE REGISTER LOGFILE ... SQL statement on the Physical Standby database.

COPY ARCHIVELOG FILES TO STANDBY

$ ls /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/

o1_mf_1_553_cnvohcy1_.arc o1_mf_1_555_cnvom1l4_.arc o1_mf_1_552_cnvodqq5_.arc


o1_mf_1_554_cnvokk6z_.arc o1_mf_1_556_cnvomxkz_.arc

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

COPY ARCHIVE LOG FILES TO STANDBY

# Example to transfer archive logs to Standby Server

$ scp log_file_n.arc oracle@standby:/archive_log_location/log_file_n.arc

$ scp o1_mf_1_552_cnvodqq5_.arc o1_mf_1_553_cnvohcy1_.arc o1_mf_1_554_cnvokk6z_.arc


o1_mf_1_555_cnvom1l4_.arc o1_mf_1_556_cnvomxkz_.arc oracle@192.168.222.134:/u01/app/oracle/
flash_recovery_area/STBYCRMS/archivelog/2016_06_01/
oracle@192.168.222.134's password:
o1_mf_1_552_cnvodqq5_.arc 100% 29KB 29.0KB/s 00:00
o1_mf_1_553_cnvohcy1_.arc 100% 73KB 72.5KB/s 00:00
o1_mf_1_554_cnvokk6z_.arc 100% 179KB 179.0KB/s 00:00
o1_mf_1_555_cnvom1l4_.arc 100% 61KB 60.5KB/s 00:00
o1_mf_1_556_cnvomxkz_.arc 100% 19KB 19.0KB/s 00:00

As per above example you need to transfer all archive logs to the Standby Server.

REGISTER LOGFILES AT STANDBY SERVER

SQL> alter database register logfile '/directory path/filename';

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive


log/2016_06_01/o1_mf_1_552_cnvodqq5_.arc';

Database altered.

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive


log/2016_06_01/ o1_mf_1_553_cnvohcy1_.arc';

Database altered.
.
SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive
log/2016_06_01/o1_mf_1_554_cnvokk6z_.arc';

Database altered.

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive


log/2016_06_01/o1_mf_1_555_cnvom1l4_.arc';

Database altered.

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive


log/2016_06_01/o1_mf_1_556_cnvomxkz_.arc';

Database altered.

Recovery process would start automatically or else stop the managed recovery process (MRP)
and re-start it once again; thats it!

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

WHY WE NEED TO REGISTER ARCHIVE LOGS AT STANDBY ?

On Primary to Standby Server, archive logs were not transferred by the log transfer service,
(but through SCP) so the managed recovery process will not have any info about these logs.
Then manually transferred logs need to be registered with the managed recovery process
before they applied by the log apply service.

CHECK WITH FOLLOWING QUERIES

SQL> archive log list;


SQL> select max(sequence#) from v$archived_log;
SQL> select sequence#, archived, applied from v$archived_log
where applied='YES' order by 1;

This is simple way to handle manually Resolving Gaps.

ROLL FORWARD PHYSICAL STANDBY USING RMAN INCREMENTAL BACKUP

The Standby database lags far behind from the Primary database then i noticed that there
is a huge sequence mismatch between Primary & Standby database. There could be some reasons.

Network unavailable between Primary and Standby database.


Archive logs deleted/corrupted on Primary database.

In cases where a Physical Standby database is out of Sync with Primary database. In case
of archive logs are missing/corrupt we have to rebuild the Standby from scratch. If the
database size in terabytes again rebuilding the Standby database could be a tedious job;
but we a solution to resolve this kind of issues.

As a DBA you can go for an RMAN Incremental backup to sync a Physical Standby with the
Primary database; using the command RMAN BACKUP INCREMENTAL FROM SCN create a backup on
the Primary database that starts at the standby databases current SCN, which can then be
used to roll the Standby database forward in time.

Please assume bunch of archive logs are deleted/corrupted on the Primary database Server
before they are transferring to the Standby database Server. In this case, I demonstrate
an efficient way to Sync Standby with Primary (an alternative to rebuild the Standby DB)!

DISASTER RECOVERY

11g DATABASE SERVER FOR PRIMARY 192.168.222.133 DB_UNIQUE_NAME CRMS


11g DATABASE SERVER FOR STANDBY 192.168.222.134 DB_UNIQUE_NAME STBYCRMS

Oracle Database Software Version is 11.2.0.1.0 on OEL(5.5)

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

# On Primary database

SYS> select name, db_unique_name, database_role from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE


--------- ------------------------------ ----------------
CRMS crms PRIMARY

# On Standby database

SYS> select name, db_unique_name, database_role, open_mode from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE


--------- ------------------------------ ---------------- --------------------
CRMS stbycrms PHYSICAL STANDBY MOUNTED

# On Primary database (Capture

SYS> select thread#, max(sequence#) from v$archived_log group by thread#;

THREAD# MAX(SEQUENCE#)
---------- --------------
1 653

# On Standby database

SYS> select thread#, max(sequence#) from v$archived_log


where applied='YES' group by thread#;

THREAD# MAX(SEQUENCE#)
---------- --------------
1 558

Note the difference, On Standby last applied SEQUENCE# 558 but on Primary SEQUENCE# 653

# Finding Archive Gap SEQ# at Standby database

SYS> select * from v$archive_gap;

THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#


---------- ------------- --------------
1 559 644

SYS> @archive_gap.sql

DB_NAME HOSTNAME LOG_ARCHIVED LOG_APPLIED APPLIED_TIME LOG_GAP


---------- -------------- ------------ ----------- -------------- -------
CRMS OEL5 653 558 01-JUN/04:33 95

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

CANCEL MANAGED RECOVERY PROCEES

# On Standby database

SYS> alter database recover managed standby database cancel;


Database altered.

GET LAST SCN OF THE STANDBY DATABASE

Last SCN of the STANDBY, will be used for the RMAN Incremental backup at Primary DB Server.

# On the Standby database Server (Checking last SCN of the Standby)

SYS> select current_scn from v$database;

CURRENT_SCN
-----------
3473651

# On the Primary database Server (Capture current_scn at primary)

SYS> select current_scn from v$database;

CURRENT_SCN
-----------
4538393

Note the SCN DIFFERENCE 4538393 3473651 = 1064742 (Now the Standby is lag behind).
But I want to know how long the Standby database is far behind in terms of (hours/days).
To know that I used the scn_to_timestamp function to translate the SCN to timestamp.

# On Primary database

SYS> select scn_to_timestamp(4538393) from dual;

SCN_TO_TIMESTAMP(4538393)
---------------------------------------
01-JUN-16 11.12.52.000000000 PM

# I execute following SQL Statement at Primary database (NOT workable at Standby)

SYS> select scn_to_timestamp(3473651) from dual;

SCN_TO_TIMESTAMP(3473651)
--------------------------------------------------
01-JUN-16 04.32.44.000000000 AM

The Standby database far behind more than 18 hours from the Primary database.
NOTE: Query NOT WORKABLE in OPEN READ ONLY MODE or MOUNTED MODE.

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

CHECKING ARCHIVE LOGS AT PRIMARY

$ ls /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/

o1_mf_1_645_cnxo02c4_.arc o1_mf_1_647_cnxo9ndy_.arc o1_mf_1_649_cnxoc4d6_.arc


o1_mf_1_651_cnxpl4fd_.arc o1_mf_1_653_cnxpyh0c_.arc o1_mf_1_646_cnxo2329_.arc
o1_mf_1_648_cnxob5cl_.arc o1_mf_1_650_cnxp29x7_.arc o1_mf_1_652_cnxpyd51_.arc

To Sync Standby with Primary we need archive logs SEQUENCE# 559 to SEQUENCE# 644. But these
archives are NOT found at Primary Site; then only choice is RMAN Incremental backup.

# Shutdown the Standby database

SYS> shutdown immediate;


...

On the Primary, take an Incremental backup from the SCN# (3473651) of the Standby database.
The last recorded SCN# (3473651) of the Standby database.

# Connect target as Primary database

$ rman target sys/passwd@CRMSDB


..
connected to target database: CRMS (DBID=1613387466)

RMAN> backup incremental from SCN 3473651 database format '/u03/rman-bkp/stby_%U'


TAG='FOR_STBY';
Starting backup at 02-JUN-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=52 device type=DISK
backup will be obsolete on date 09-JUN-16
archived logs will not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00003 name=/u01/app/oracle/oradata/crms/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/crms/users01.dbf
input datafile file number=00001 name=/u01/app/oracle/oradata/crms/system01.dbf
input datafile file number=00002 name=/u01/app/oracle/oradata/crms/sysaux01.dbf
input datafile file number=00005 name=/u01/app/oracle/oradata/crms/example01.dbf
input datafile file number=00007 name=/u01/app/oracle/oradata/crms/users02.dbf
input datafile file number=00006 name=/u01/app/oracle/oradata/crms/users03.dbf
input datafile file number=00008 name=/u01/app/oracle/oradata/crms/mytbs01.dbf
channel ORA_DISK_1: starting piece 1 at 02-JUN-16
channel ORA_DISK_1: finished piece 1 at 02-JUN-16

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

piece handle=/u03/rman-bkp/stby_03r73ftc_1_1 tag=FOR_STBY comment=NONE


channel ORA_DISK_1: backup set complete, elapsed time: 00:13:13
using channel ORA_DISK_1
backup will be obsolete on date 09-JUN-16
archived logs will not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 02-JUN-16
channel ORA_DISK_1: finished piece 1 at 02-JUN-16
piece handle=/u03/rman-bkp/stby_04r73gm6_1_1 tag=FOR_STBY comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:04
Finished backup at 02-JUN-16

CREATE A NEW STANDBY CONTROLFILE

On the Primary database, create a new Standby control file.


We can use (SQL*Plus or RMAN) to create a Standby control file; Anyone method is enough.

SYS> alter database create standby controlfile as '/home/oracle/stdb_ctrl.ctl';


Database altered. Or

# Connect target as Primary & Create control file backup for Standby database

RMAN> backup current controlfile for standby format '/u03/rman-bkp/stdb_ctrl.ctl';

Starting backup at 02-JUN-16


using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 02-JUN-16
channel ORA_DISK_1: finished piece 1 at 02-JUN-16
piece handle=/u03/rman-bkp/stdb_ctrl.ctl tag=TAG20160602T024937 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 02-JUN-16

LISTING CREATED BACKUP SETS

$ ls -l /u03/rman-bkp/
total 4707832
-rw-r----- 1 oracle oinstall 4792238080 Jun 2 02:42 stby_03r73ftc_1_1
-rw-r----- 1 oracle oinstall 11927552 Jun 2 02:42 stby_04r73gm6_1_1
-rw-r----- 1 oracle oinstall 11927552 Jun 2 02:49 stdb_ctrl.ctl

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

TRANSFER BACKUP SETS TO STANDBY SERVER

Using OS utility to transfer these backups to the Standby Server from Primary Server.

# SCP all backup sets to the Standby Server at Primary Server

$ scp * oracle@192.168.222.134:/u03/bkp/
oracle@192.168.222.134's password: ******

stby_03r73ftc_1_1 100% 4570MB 2.4MB/s 31:43


stby_04r73gm6_1_1 100% 11MB 5.7MB/s 00:02
stdb_ctrl.ctl 100% 11MB 2.8MB/s 00:04

VERIFY RECEIVED BACKUP SETS ON THE STANDBY SERVER

$ ls -l /u03/bkp/
total 4707832
-rw-r----- 1 oracle oinstall 4792238080 Jun 2 05:40 stby_03r73ftc_1_1
-rw-r----- 1 oracle oinstall 11927552 Jun 2 05:40 stby_04r73gm6_1_1
-rw-r----- 1 oracle oinstall 11927552 Jun 2 05:40 stdb_ctrl.ctl

Note the location of all data files & Control file(s) at the Standby Database Server.

SYS> startup mount;


SYS> spool datafile_names.txt

SYS> set lines 100;


SYS> column name format a100;

SYS> show parameter control_files;


SYS> select count(*) from v$datafile;
SYS> select file#, name from v$datafile order by file#;

SYS> spool off


SYS> shut immediate;
SYS> ! cat datafile_names.txt

RELOCATE EXISTING CONTROL FILES

# Renamed old control files at Standby Server

$ cd /u01/app/oracle/flash_recovery_area/stbycrms/
$ mv control02.ctl /tmp/control02.ctl.bkp
$ cd /u01/app/oracle/oradata/stbycrms/
$ mv control01.ctl /tmp/control01.ctl.bkp

NOTE: If you do not want rename control files, you can remove these files at OS level.

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

BRINGUP THE STANDBY INSTANCE NOMOUNT PHASE

$ export ORACLE_SID=stbycrms
$ rman target /

connected to target database (not started)

RMAN> startup nomount;

Oracle instance started

Total System Global Area 912306176 bytes

Fixed Size 1340244 bytes


Variable Size 536874156 bytes
Database Buffers 369098752 bytes
Redo Buffers 4993024 bytes

# Restore Standby control files from the backup


# Replacing newly created controlfile with old files

RMAN> restore standby controlfile from '/u03/bkp/stdb_ctrl.ctl';

Starting restore at 02-JUN-16


allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=19 device type=DISK

channel ORA_DISK_1: restoring control file


channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
output file name=/u01/app/oracle/oradata/stbycrms/control01.ctl
output file name=/u01/app/oracle/flash_recovery_area/stbycrms/control02.ctl
Finished restore at 02-JUN-16

SHUTDOWN THE STANDBY DATABASE & STARTUP MOUNT

RMAN> shutdown immediate;

Oracle instance shut down

# Startup the database with new controlfile

RMAN> startup mount;


connected to target database (not started)

Oracle instance started


database mounted

Total System Global Area 912306176 bytes


..
...

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

CATALOG THE INCREMENTAL BACKUP FILES AT STANDBY

RMAN does not know about these backup sets. We must catalog the new backup sets on the
Standby database. I.e. when we register these backup files to the Standby database control
file with RMAN CATALOG command, the Standby control file gets updated about these backup
files. It helps to recover the Standby using these RMAN backup files.

# REGISTERING BACKUP SETS

RMAN> catalog start with '/u03/bkp';

Starting implicit crosscheck backup at 02-JUN-16


allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 device type=DISK
Crosschecked 3 objects
Finished implicit crosscheck backup at 02-JUN-16

Starting implicit crosscheck copy at 02-JUN-16


using channel ORA_DISK_1
Crosschecked 2 objects
Finished implicit crosscheck copy at 02-JUN-16

searching for all files in the recovery area


cataloging files...
cataloging done

List of Cataloged Files


=======================

File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_646_cnxof4nl_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_647_cnxocnkr_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_652_cnxpz1lz_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_645_cnxodgj5_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_651_cnxodqq5_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_650_cnxp48lh_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_649_cnxoc5oq_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_653_cnxpzpvs_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_648_cnxobwyc_.arc

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

searching for all files that match the pattern /u03/bkp

List of Files Unknown to the Database


=====================================

File Name: /u03/bkp/stby_04r73gm6_1_1


File Name: /u03/bkp/stby_03r73ftc_1_1
File Name: /u03/bkp/stdb_ctrl.ctl

Do you really want to catalog the above files (enter YES or NO)? Y
cataloging files...
cataloging done

List of Cataloged Files


=======================
File Name: /u03/bkp/stby_04r73gm6_1_1
File Name: /u03/bkp/stby_03r73ftc_1_1
File Name: /u03/bkp/stdb_ctrl.ctl

APPLY INCREMENTAL BACKUP TO THE STANDBY DATABASE

The recovery process will use cataloged incremental backup sets because we have registered.
This backup taken for Physical Standby DB Sync; noredo key word is required. Check here

# Recover the database with Catalog incremental backup sets

RMAN> recover database noredo;

Starting recover at 02-JUN-16


using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/02/2016 05:55:09
RMAN-06094: datafile 6 must be restored

If data files have been added to the Primary database during the archive log gap time, they
were not included in the incremental backup sets. We need to register newly created files
to the Standby database. We can find newly created files using CURRENT_SCN of the Standby.

# Execute the following query at Primary DB - (Last SCN of the Standby database)

SYS> select file#, name from v$datafile where creation_change# > 3473651;

FILE# NAME
---------- --------------------------------------------
6 /u01/app/oracle/oradata/crms/users03.dbf

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

Newly created files should be restored at Standby before recovery operation.

RMAN> run
{restore datafile 6;}

Starting restore at 02-JUN-16


using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00006 to
/u01/app/oracle/oradata/stbycrms/users03.dbf
channel ORA_DISK_1: reading from backup piece /u03/bkp/stby_03r73ftc_1_1
channel ORA_DISK_1: piece handle=/u03/bkp/stby_03r73ftc_1_1 tag=FOR_STBY
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:08
Finished restore at 02-JUN-16

# Recover the Standby database using Incremental backup Sets

RMAN> recover database noredo;

Starting recover at 02-JUN-16


using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001:
/u01/app/oracle/oradata/stbycrms/system01.dbf
destination for restore of datafile 00002:
/u01/app/oracle/oradata/stbycrms/sysaux01.dbf
destination for restore of datafile 00003:
/u01/app/oracle/oradata/stbycrms/undotbs01.dbf
destination for restore of datafile 00004:
/u01/app/oracle/oradata/stbycrms/users01.dbf
destination for restore of datafile 00005:
/u01/app/oracle/oradata/stbycrms/example01.dbf
destination for restore of datafile 00007:
/u01/app/oracle/oradata/stbycrms/users02.dbf
destination for restore of datafile 00008:
/u01/app/oracle/oradata/stbycrms/mytbs01.dbf
channel ORA_DISK_1: reading from backup piece /u03/bkp/stby_03r73ftc_1_1
channel ORA_DISK_1: piece handle=/u03/bkp/stby_03r73ftc_1_1 tag=FOR_STBY
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:42:18

Finished recover at 02-JUN-16

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

All changed blocks has been captured in the incremental backup and also updated at the
standby database, thus brings the Standby database up to date with the primary database.

# Start MRP at Standby database

SYS> alter database recover managed standby database disconnect;


Database altered.

# Check CURRENT_SCN of the Standby Database & Primary database

SYS> select current_scn from v$database;


...

Check the SCNs in Primary and Standby it should be close to each other.

ARCHIVE LOG SEQUENCE DETAILS AT PRIMARY & STANDBY DATABASE

# Maximum generated LOG SEQUENCE# at Primary

SYS> select thread#, max(sequence#) from v$archived_log group bythread#;

THREAD# MAX(SEQUENCE#)
---------- --------------
1 773

# Maximum archive log SEQUENCE# applied at Standby

SYS> select thread#, max(sequence#) from v$archived_log


where applied='YES' group by thread#;

THREAD# MAX(SEQUENCE#)
---------- --------------
1 773

SYS> select process, status, thread#, sequence#, block#, blocks from v$managed_standby;
...

Wow! Successfully Standby database Synced with Primary database.


Check alert.log and status of processes if any problems;

REFERENCE DOCS :

https://docs.oracle.com/cd/E11882_01/server.112/e41134/rman.htm#SBYDB00759
http://docs.oracle.com/cd/B19306_01/backup.102/b14191/rcmdupdb.htm#sthref955
https://web.stanford.edu/dept/itss/docs/oracle/10gR2/backup.102/b14191/rcmdupdb008.htm

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

BELOW ARE THE STEPS I FOLLOWED FOR A SUCCESSFUL RECOVERY

# Check the archive log gap SEQUENCE#


SQL> select * from v$archive_gap;

# Find last SCN of the Standby database


SQL> select current_scn from v$database;

# Cancel the recovery


SQL> alter database recover managed standby database cancel;

# Shutdown the Standby


SQL> shutdown immediate;

# Backup the Source (Primary) using from SCN


RMAN> backup incremental from SCN XXXXX database format '/bkp_location/stby_U%'

# Backup current control file of the Source (Primary database)


RMAN> backup current controlfile for standby format '/bkp_location/stby_ctrl.ctl';

# SCP backup Sets to the target(Standby) Server from Primary


$ scp /bkp_location/* oracle@standby:/standby_location/

# Startup Standby database at NOMOUNT mode


RMAN> startup nomount;

# Restore control files from the backup


RMAN> restore standby controlfile from '/standby _location';

# Shutdown the Standby database


RMAN> shutdown immediate;

# Startup the Standby database at MOUNT


RMAN> startup mount;

# Register backup sets by a process called cataloging


RMAN> catalog startwith '/standby _location';

# Perform Recover
RMAN> recover database noredo;

# Restart MRP at standby


SQL> alter database recover managed standby database disconnect from session;

# Check SCN of the Primary and Standby database


SQL> select current_scn from v$database;

These are Steps I have done to roll the Standby database forward in time. Thats it!

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

METHODS OF GAP RESOLUTION

Automatic Gap Resolution


FAL (Fetch Archive Log) Gap Resolution
Manual Gap Resolution
Roll forward using RMAN Incremental Backup (Physical Standby)

AUTOMATIC GAP RESOLUTION

If connectivity is lost between the Primary and Standby databases (due to network problems),
redo data being generated on the primary database cannot be sent to the Standby database.
Once a connection is reestablished, the missing archived redo log files are automatically
detected by Data Guard, which then automatically transmits the missing archived redo log
files to the Standby databases. The Standby databases are synchronized with the Primary
database, without manual intervention by the DBA. How this happens? Lets dig into deep.

Automatic Gap Resolution is performed automatically by the Log Transport Services.


Basically the currently transferred Redo is compared with the last received.

On Primary ARCH/LGWR to transfer archived redo logs to the Standby.


On the Standby RFS process receives archive redo log file.

RFS process Compares the Sequence number of currently being archived file with the sequence
number of previously received archived redo file; (if currently being archived redo file
sequence# is greater than the last sequence# received plus one, there is a gap). An archived
redo log file is uniquely identified by its sequence number and thread number.

o1_mf_1_648_cnxob5cl_.arc Last archived log file.


o1_mf_1_652_cnxpyd51_.arc Currently being archived file.

Now 3 files are missing. There is a gap between both files, then RFS automatically requests
the missing redo log sequence# from the Primary DB again via the ARCH-RFS Heartbeat Ping.
The archiver of the Primary then retransmits these missing archived redo files.

o1_mf_1_649_cnxoc4d6_.arc, o1_mf_1_650_cnxp29x7_.arc, o1_mf_1_651_cnxpl4fd_.arc

This type of Gap Resolution is using the Service defined in log_archive_dest_n on the
Primary database Serving this Standby database.

SYS> show parameter log_archive_dest_2;

NAME TYPE VALUE


----------------------- ----------- --------------------------------------------
log_archive_dest_2 string Service=stby_crmsdb LGWR ASYNC
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)
DB_UNIQUE_NAME=stbycrms

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

The archiver process of the Primary database polls the Standby databases every minute -
(Referred to as heartbeat) to see if there is a gap in the sequence of archived redo logs.
If a gap is detected, the ARCH process sends the missing archived redo log files to the
Standby databases that reported the gap. Once a gap is resolved then the Transport Process
(ARCH/LGWR) is notified about the Resolution of the Gap.

FAL GAP RESOLUTION

As I said above, RFS receives an archive log on the Standby database, then the archive logs
are registered in the Standby Control file with name and location. Missing log files are
typically detected by the Log Apply Services (MRP) on the Standby database. If the archived
redo log file is missing/corrupted for any reason (eg. it got deleted). FAL is required to
resolve a Gap, to obtain a new copy of the corrupted or deleted file.

WHY ESPECIALLY FAL IS REQUIRED ?

Since MRP has NO direct communications (LOG TRANSPORT SERVICES) of the primary database;
it must use the FAL_SERVER and FAL_CLIENT initialization parameters to resolve the gap.
Both of these parameters must be set in the Standby Initialization parameter file.

The FAL_CLIENT parameter is no longer required since Oracle 11g R2.


In earlier releases, you set the FAL_CLIENT parameter on the Standby database.

FAL_SERVER AND FAL_CLIENT

FAL_CLIENT and FAL_SERVER need to be defined in the initialization parameter file of the
Standby database(s). They mainly used for GAP RESOLUTION through FAL background process.

Using FAL parameters the MRP in the Physical Standby database checks automatically and also
Resolving gaps at the time redo is applied.

FAL_SERVER = NET_SERVICE_NAME of Primary database.

From where the missing archive log(s) Should be requested.

FAL_CLIENT = NET_SERVICE_NAME of Standby database.

The destination where the FAL_SERVER database Should send the archived log(s).

SYS> show parameter fal; # On Standby database

NAME TYPE VALUE


------------------------------------ ----------- ------------------------------
fal_client string STBY_CRMSDB
fal_server string CRMSDB

In earlier release when you set FAL_CLIENT parameter on the Standby database, the Primary
database (FAL_SERVER) uses Standby database (NET SERVICE NAME) to connect the Standby DB.

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

The Parameter log_archive_dest_n on FAL_SERVER database which is pointing to the Standby


database to resolve the Gap Resolution..

# On Primary database

SYS> show parameter log_Archive_dest_2;

NAME TYPE VALUE


--------------------- ----------- ------------------------------
log_archive_dest_2 string service=stby_crmsdb LGWR ASYNC
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)
DB_UNIQUE_NAME=stbycrms

FAL_SERVER: An Oracle NET SERVICE NAME (TNS-Alias or Connect Descriptor). This parameter
must be configured on the Standby database System that points from where the missing archive
log(s) Should be requested. Its recommended to set FAL_SERVER on the Primary DB with the
value of the Standby database NET SERVICE NAME thus helps when you do Switch over event.

Once the Log Apply Services (MRP) detect an ARCHIVE GAP, then it sends a FAL Request to
the FAL_SERVER; Once communication has been established with the Primary database, it
passes the Sequence number of archived files (which are causing for the archive gap) to
retransmit from the archiver process of the Primary database; and additionally it passes
the service name defined by the FAL_CLIENT parameter to the Primary ARCH process.

Fetching gap sequence in thread 1, gap sequence 1076-1077


FAL[client]: Trying FAL server: CRMSDB

An ARCH Process on the FAL_SERVER tries to pick up the requested Sequences from that
database and sends them to the FAL_CLIENT. I.e. The Primary database ARCH process Ships the
requested archived logs to the remote archive destination of corresponding service name.

# SNIPPET OF STANDBY ALERT.LOG

RFS[54]: Opened log for thread 1 sequence 1076 dbid 1613387466 branch 913081878
Media Recovery Log
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_04/o1_mf_1_1076_co5qn3sd_.arc

RFS[55]: Opened log for thread 1 sequence 1077 dbid 1613387466 branch 913081878
Media Recovery Log
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_04/o1_mf_1_1077_co5qn4db_.arc

In order to successfully complete a Gap Request the requested archive log Sequence(s) must
be available on the FAL_SERVER database.

When you have multiple Physical Standby databases, the FAL mechanism can automatically
retrieve missing archived redo log files from another Physical Standby database.

Oracle DBA Technology Explored by Gunasekaran , Thiyagu


RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

IF THE FAL_SERVER IS NOT ABLE TO RESOLVE THE GAP?

The FAL-Request fails and a corresponding Message will be put in the ALERT.LOG of the
Standby database. Example taken for some other SEQUENCE (1078 1081).

FAL[client]: Failed to request gap sequence


GAP - thread 1 sequence 1078-1081
DBID 1613387466 branch 913081878
FAL[client]: All defined FAL servers have been attempted.
-------------------------------------------------------------

Every minute the Primary database polls its Standby databases to see if there are gaps in
the sequence of archived redo log files.

The FAL (Client) requests to transfer archived redo log files automatically.
The FAL (Server) services the FAL requests coming from the FAL Client.
A separate FAL server is created for each incoming FAL client.

FAL is available since Oracle 9.2.0 for Physical Standby database and Oracle 10.1.0 for
Logical Standby databases. In 11g, if you not set FAL_CLIENT parameter, the Primary database
will obtain service name from related LOG_ARCHIVE_DEST_n parameter.

REFERENCE LINKS

http://flylib.com/books/en/1.145.1.66/1/

https://docs.oracle.com/cd/B19306_01/server.102/b14239/log_transport.htm#i1268294

Oracle Note: Data Guard Gap Detection and Resolution Possibilities [ID 1537316.1]

Oracle DBA Technology Explored by Gunasekaran , Thiyagu

Potrebbero piacerti anche