General Backup and Recovery questions

[edit] Why and when should I backup my database?

Backup and recovery is one of the most important aspects of a DBAs job. If you lose your
company's data, you could very well lose your job. Hardware and software can always be
replaced, but your data may be irreplaceable!

Normally one would schedule a hierarchy of daily, weekly and monthly backups, however
consult with your users before deciding on a backup schedule. Backup frequency normally
depends on the following factors:

 Rate of data change/ transaction rate

 Database availability/ Can you shutdown for cold backups?
 Criticality of the data/ Value of the data to the company
 Read-only tablespace needs backing up just once right after you make it read-only
 If you are running in archivelog mode you can backup parts of a database over an
extended cycle of days
 If archive logging is enabled one needs to backup archived log files timeously to
prevent database freezes
 Etc.

Carefully plan backup retention periods. Ensure enough backup media (tapes) are available
and that old backups are expired in-time to make media available for new backups. Off-site
vaulting is also highly recommended.

Frequently test your ability to recover and document all possible scenarios. Remember, it's
the little things that will get you. Most failed recoveries are a result of organizational errors
and miscommunication.

[edit] What strategies are available for backing-up an Oracle database?

The following methods are valid for backing-up an Oracle database:

 Export/Import - Exports are "logical" database backups in that they extract logical
definitions and data from the database to a file. See the Import/ Export FAQ for more
details.

 Cold or Off-line Backups - shut the database down and backup up ALL data, log,
and control files.
  Hot or On-line Backups - If the database is available and in ARCHIVELOG mode, set
the tablespaces into backup mode and backup their files. Also remember to backup the
control files and archived redo log files.

 RMAN Backups - while the database is off-line or on-line, use the "rman" utility to
backup the database.

It is advisable to use more than one of these methods to backup your database. For
example, if you choose to do on-line database backups, also cover yourself by doing
database exports. Also test ALL backup and recovery scenarios carefully. It is better to be
safe than sorry.

Regardless of your strategy, also remember to backup all required software libraries,
parameter files, password files, etc. If your database is in ARCHIVELOG mode, you also need
to backup archived log files.

[edit] What is the difference between online and offline backups?

A hot (or on-line) backup is a backup performed while the database is open and available for
use (read and write activity). Except for Oracle exports, one can only do on-line backups
when the database is ARCHIVELOG mode.

A cold (or off-line) backup is a backup performed while the database is off-line and
unavailable to its users. Cold backups can be taken regardless if the database is in
ARCHIVELOG or NOARCHIVELOG mode.

It is easier to restore from off-line backups as no recovery (from archived logs) would be
required to make the database consistent. Nevertheless, on-line backups are less disruptive
and doesn't require database downtime.

Point-in-time recovery (regardless if you do on-line or off-line backups) is only available

when the database is in ARCHIVELOG mode.

[edit] What is the difference between restoring and recovering?

Restoring involves copying backup files from secondary storage (backup media) to disk. This
can be done to replace damaged files or to copy/move a database to a new location.

Recovery is the process of applying redo logs to the database to roll it forward. One can roll-
forward until a specific point-in-time (before the disaster occurred), or roll-forward until the
last transaction recorded in the log files.
SQL> connect SYS as SYSDBA
SQL> RECOVER DATABASE UNTIL TIME '2001-03-06:16:00:00' USING BACKUP CONTROLFILE;
RMAN> run {
set until time to_date('04-Aug-2004 00:00:00', 'DD-MON-YYYY HH24:MI:SS');
restore database;
recover database;
}

[edit] My database is down and I cannot restore. What now?

This is probably not the appropriate time to be sarcastic, but, recovery without backups are
not supported. You know that you should have tested your recovery strategy, and that you
should always backup a corrupted database before attempting to restore/recover it.

Nevertheless, Oracle Consulting can sometimes extract data from an offline database using
a utility called DUL (Disk UnLoad - Life is DUL without it!). This utility reads data in the data
files and unloads it into SQL*Loader or export dump files. Hopefully you'll then be able to
load the data into a working database.

Note that DUL does not care about rollback segments, corrupted blocks, etc, and can thus
not guarantee that the data is not logically corrupt. It is intended as an absolute last resort
and will most likely cost your company a lot of money!

DUDE (Database Unloading by Data Extraction) is another non-Oracle utility that can be
used to extract data from a dead database. More info about DUDE is available at
http://www.ora600.nl/.

[edit] How does one backup a database using the export utility?

Oracle exports are "logical" database backups (not physical) as they extract data and logical
definitions from the database into a file. Other backup strategies normally back-up the
physical data files.

One of the advantages of exports is that one can selectively re-import tables, however one
cannot roll-forward from an restored export. To completely restore a database from an
export file one practically needs to recreate the entire database.

Always do full system level exports (FULL=YES). Full exports include more information about
the database in the export file than user level exports. For more information about the
Oracle export and import utilities, see the Import/ Export FAQ.

[edit] How does one put a database into ARCHIVELOG mode?

The main reason for running in archivelog mode is that one can provide 24-hour availability
and guarantee complete data recoverability. It is also necessary to enable ARCHIVELOG
mode before one can start to use on-line database backups.

Issue the following commands to put a database into ARCHVELOG mode:

SQL> CONNECT sys AS SYSDBA

SQL> STARTUP MOUNT EXCLUSIVE;
SQL> ALTER DATABASE ARCHIVELOG;
SQL> ARCHIVE LOG START;
SQL> ALTER DATABASE OPEN;

Alternatively, add the above commands into your database's startup command script, and
bounce the database.

The following parameters needs to be set for databases in ARCHIVELOG mode:

log_archive_start = TRUE
log_archive_dest_1 = 'LOCATION=/arch_dir_name'
log_archive_dest_state_1 = ENABLE
log_archive_format = %d_%t_%s.arc

NOTE 1: Remember to take a baseline database backup right after enabling archivelog
mode. Without it one would not be able to recover. Also, implement an archivelog backup to
prevent the archive log directory from filling-up.

NOTE 2:' ARCHIVELOG mode was introduced with Oracle 6, and is essential for database
point-in-time recovery. Archiving can be used in combination with on-line and off-line
database backups.

NOTE 3: You may want to set the following INIT.ORA parameters when enabling
ARCHIVELOG mode: log_archive_start=TRUE, log_archive_dest=..., and
log_archive_format=...

NOTE 4: You can change the archive log destination of a database on-line with the ARCHIVE
LOG START TO 'directory'; statement. This statement is often used to switch archiving
between a set of directories.

NOTE 5: When running Oracle Real Application Clusters (RAC), you need to shut down all
nodes before changing the database to ARCHIVELOG mode. See the RAC FAQ for more
details.
[edit] I've lost an archived/online REDO LOG file, can I get my DB back?

The following INIT.ORA/SPFILE parameter can be used if your current redologs are corrupted
or blown away. It may also be handy if you do database recovery and one of the archived
log files are missing and cannot be restored.

NOTE: Caution is advised when enabling this parameter as you might end-up losing your
entire database. Please contact Oracle Support before using it.

_allow_resetlogs_corruption = true

This should allow you to open the database. However, after using this parameter your
database will be inconsistent (some committed transactions may be lost or partially
applied).

Steps:

 Do a "SHUTDOWN NORMAL" of the database

 Set the above parameter
 Do a "STARTUP MOUNT" and "ALTER DATABASE OPEN RESETLOGS;"
 If the database asks for recovery, use an UNTIL CANCEL type recovery and apply all
available archive and on-line redo logs, then issue CANCEL and reissue the "ALTER
DATABASE OPEN RESETLOGS;" command.
 Wait a couple of minutes for Oracle to sort itself out
 Do a "SHUTDOWN NORMAL"
 Remove the above parameter!
 Do a database "STARTUP" and check your ALERT.LOG file for errors.
 Extract the data and rebuild the entire database

[edit] User managed backup and recovery

This section deals with user managed, or non-RMAN backups.

[edit] How does one do off-line database backups?

Shut down the database from sqlplus or server manager. Backup all files to secondary
storage (eg. tapes). Ensure that you backup all data files, all control files and all log files.
When completed, restart your database.

Do the following queries to get a list of all files that needs to be backed up:
select name from sys.v_$datafile;
select member from sys.v_$logfile;
select name from sys.v_$controlfile;[/code]

Sometimes Oracle takes forever to shutdown with the "immediate" option. As workaround to
this problem, shutdown using these commands:

alter system checkpoint;

shutdown abort
startup restrict
shutdown immediate

Note that if you database is in ARCHIVELOG mode, one can still use archived log files to roll
forward from an off-line backup. If you cannot take your database down for a cold (off-line)
backup at a convenient time, switch your database into ARCHIVELOG mode and perform hot
(on-line) backups.

[edit] How does one do on-line database backups?

Each tablespace that needs to be backed-up must be switched into backup mode before
copying the files out to secondary storage (tapes). Look at this simple example.

ALTER TABLESPACE xyz BEGIN BACKUP;

! cp xyfFile1 /backupDir/
ALTER TABLESPACE xyz END BACKUP;

It is better to backup tablespace for tablespace than to put all tablespaces in backup mode.
Backing them up separately incurs less overhead. When done, remember to backup your
control files. Look at this example:

ALTER SYSTEM SWITCH LOGFILE; -- Force log switch to update control file headers
ALTER DATABASE BACKUP CONTROLFILE TO '/backupDir/control.dbf';

NOTE: Do not run on-line backups during peak processing periods. Oracle will write complete
database blocks instead of the normal deltas to redo log files while in backup mode. This will
lead to excessive database archiving and even database freezes.

[edit] My database was terminated while in BACKUP MODE, do I need to recover?

If a database was terminated while one of its tablespaces was in BACKUP MODE (ALTER
TABLESPACE xyz BEGIN BACKUP;), it will tell you that media recovery is required when you
try to restart the database. The DBA is then required to recover the database and apply all
archived logs to the database. However, from Oracle 7.2, one can simply take the individual
datafiles out of backup mode and restart the database.

ALTER DATABASE DATAFILE '/path/filename' END BACKUP;

One can select from V$BACKUP to see which datafiles are in backup mode. This normally
saves a significant amount of database down time. See script end_backup2.sql in the Scripts
section of this site.

From Oracle9i onwards, the following command can be used to take all of the datafiles out
of hotbackup mode:

ALTER DATABASE END BACKUP;

This command must be issued when the database is mounted, but not yet opened.

[edit] Does Oracle write to data files in begin/hot backup mode?

When a tablespace is in backup mode, Oracle will stop updating its file headers, but will
continue to write to the data files.

When in backup mode, Oracle will write complete changed blocks to the redo log files.
Normally only deltas (change vectors) are logged to the redo logs. This is done to enable
reconstruction of a block if only half of it was backed up (split blocks). Because of this, one
should notice increased log activity and archiving during on-line backups.

To solve this problem, simply switch to RMAN backups.

[edit] RMAN backup and recovery

This section deals with RMAN backups:

[edit] What is RMAN and how does one use it?

Recovery Manager (or RMAN) is an Oracle provided utility for backing-up, restoring and
recovering Oracle Databases. RMAN ships with the database server and doesn't require a
separate installation. The RMAN executable is located in your ORACLE_HOME/bin directory.

In fact RMAN, is just a Pro*C application that translates commands to a PL/SQL interface. The
PL/SQL calls are stallically linked into the Oracle kernel, and does not require the database
to be opened (mapped from the ?/rdbms/admin/recover.bsq file).
RMAN can do off-line and on-line database backups. It cannot, however, write directly to
tape, but various 3rd-party tools (like Veritas, Omiback, etc) can integrate with RMAN to
handle tape library management.

RMAN can be operated from Oracle Enterprise Manager, or from command line. Here are the
command line arguments:

Argument Value Description

-----------------------------------------------------------------------------
target quoted-string connect-string for target database
catalog quoted-string connect-string for recovery catalog
nocatalog none if specified, then no recovery catalog
cmdfile quoted-string name of input command file
log quoted-string name of output message log file
trace quoted-string name of output debugging message log file
append none if specified, log is opened in append mode
debug optional-args activate debugging
msgno none show RMAN-nnnn prefix for all messages
send quoted-string send a command to the media manager
pipe string building block for pipe names
timeout integer number of seconds to wait for pipe input
-----------------------------------------------------------------------------

Here is an example:

[oracle@localhost oracle]$ rman

Recovery Manager: Release 10.1.0.2.0 - Production
Copyright (c) 1995, 2004, Oracle. All rights reserved.

RMAN> connect target;

connected to target database: ORCL (DBID=1058957020)

RMAN> backup database;

...

[edit] How does one backup and restore a database using RMAN?

The biggest advantage of RMAN is that it only backup used space in the database. RMAN
doesn't put tablespaces in backup mode, saving on redo generation overhead. RMAN will re-
read database blocks until it gets a consistent image of it. Look at this simple backup
example.
rman target sys/*** nocatalog
run {
allocate channel t1 type disk;
backup
format '/app/oracle/backup/%d_t%t_s%s_p%p'
(database);
release channel t1;
}

Example RMAN restore:

rman target sys/*** nocatalog

run {
allocate channel t1 type disk;
# set until time 'Aug 07 2000 :51';
restore tablespace users;
recover tablespace users;
release channel t1;
}

The examples above are extremely simplistic and only useful for illustrating basic concepts.
By default Oracle uses the database controlfiles to store information about backups.
Normally one would rather setup a RMAN catalog database to store RMAN metadata in. Read
the Oracle Backup and Recovery Guide before implementing any RMAN backups.

Note: RMAN cannot write image copies directly to tape. One needs to use a third-party
media manager that integrates with RMAN to backup directly to tape. Alternatively one can
backup to disk and then manually copy the backups to tape.

[edit] How does one backup and restore archived log files?

One can backup archived log files using RMAN or any operating system backup utility.
Remember to delete files after backing them up to prevent the archive log directory from
filling up. If the archive log directory becomes full, your database will hang! Look at this
simple RMAN backup scripts:

RMAN> run {
2> allocate channel dev1 type disk;
3> backup
4> format '/app/oracle/archback/log_%t_%sp%p'
5> (archivelog all delete input);
6> release channel dev1;
7> }

The "delete input" clause will delete the archived logs as they as backed-up.

List all archivelog backups for the past 24 hours:

RMAN> LIST BACKUP OF ARCHIVELOG FROM TIME 'sysdate-1';

Here is a restore example:

RMAN> run {
2> allocate channel dev1 type disk;
3> restore (archivelog low logseq 78311 high logseq 78340 thread 1 all);
4> release channel dev1;
5> }

[edit] How does one create a RMAN recovery catalog?

Start by creating a database schema (usually called rman). Assign an appropriate

tablespace to it and grant it the recovery_catalog_owner role. Look at this example:

sqlplus sys
SQL> create user rman identified by rman;
SQL> alter user rman default tablespace tools temporary tablespace temp;
SQL> alter user rman quota unlimited on tools;
SQL> grant connect, resource, recovery_catalog_owner to rman;
SQL> exit;

Next, log in to rman and create the catalog schema. Prior to Oracle 8i this was done by
running the catrman.sql script.

rman catalog rman/rman

RMAN> create catalog tablespace tools;
RMAN> exit;

You can now continue by registering your databases in the catalog. Look at this example:

rman catalog rman/rman target backdba/backdba

RMAN> register database;
One can also use the "upgrade catalog;" command to upgrade to a new RMAN release, or
the "drop catalog;" command to remove an RMAN catalog. These commands need to be
entered twice to confirm the operation.

[edit] How does one integrate RMAN with third-party Media Managers?

The following Media Management Software Vendors have integrated their media
management software with RMAN (Oracle Recovery Manager):

 Veritas NetBackup - http://www.veritas.com/

 EMC Data Manager (EDM) - http://www.emc.com/
 HP OMNIBack/ DataProtector - http://www.hp.com/
 IBM's Tivoli Storage Manager (formerly ADSM) - http://www.tivoli.com/storage/
 EMC Networker - http://www.emc.com/
 BrightStor ARCserve Backup - http://www.ca.com/us/data-loss-prevention.aspx
 Sterling Software's SAMS:Alexandria (formerly from Spectralogic) -
http://www.sterling.com/sams/
 SUN's Solstice Backup - http://www.sun.com/software/whitepapers/backup-n-storage/
 CommVault Galaxy - http://www.commvault.com/
 etc...

The above Media Management Vendors will provide first line technical support (and
installation guides) for their respective products.

A complete list of supported Media Management Vendors can be found at:

http://www.oracle.com/technology/deploy/availability/htdocs/bsp.htm

When allocating channels one can specify Media Management spesific parameters. Here are
some examples:

Netbackup on Solaris:

allocate channel t1 type 'SBT_TAPE'

PARMS='SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so.1';

Netbackup on Windows:

allocate channel t1 type 'SBT_TAPE' send "NB_ORA_CLIENT=client_machine_name";

Omniback/ DataProtector on HP-UX:

allocate channel t1 type 'SBT_TAPE' PARMS='SBT_LIBRARY=

/opt/omni/lib/libob2oracle8_64bit.sl';

or:

allocate channel 'dev_1' type 'sbt_tape' parms

'ENV=OB2BARTYPE=Oracle8,OB2APPNAME=orcl,OB2BARLIST=machinename_orcl_archlogs)'
;

[edit] How does one clone/duplicate a database with RMAN?

The first step to clone or duplicate a database with RMAN is to create a new INIT.ORA and
password file (use the orapwd utility) on the machine you need to clone the database to.
Review all parameters and make the required changed. For example, set the DB_NAME
parameter to the new database's name.

Secondly, you need to change your environment variables, and do a STARTUP NOMOUNT
from sqlplus. This database is referred to as the AUXILIARY in the script below.

Lastly, write a RMAN script like this to do the cloning, and call it with "rman cmdfile
dupdb.rcv":

connect target sys/secure@origdb

connect catalog rman/rman@catdb
connect auxiliary /

run {
set newname for datafile 1 to '/ORADATA/u01/system01.dbf';
set newname for datafile 2 to '/ORADATA/u02/undotbs01.dbf';
set newname for datafile 3 to '/ORADATA/u03/users01.dbf';
set newname for datafile 4 to '/ORADATA/u03/indx01.dbf';
set newname for datafile 5 to '/ORADATA/u02/example01.dbf';

allocate auxiliary channel dupdb1 type disk;

set until sequence 2 thread 1;

duplicate target database to dupdb

logfile
GROUP 1 ('/ORADATA/u02/redo01.log') SIZE 200k REUSE,
GROUP 2 ('/ORADATA/u03/redo02.log') SIZE 200k REUSE;
}
The above script will connect to the "target" (database that will be cloned), the recovery
catalog (to get backup info), and the auxiliary database (new duplicate DB). Previous
backups will be restored and the database recovered to the "set until time" specified in the
script.

Notes: the "set newname" commands are only required if your datafile names will different
from the target database.

The newly cloned DB will have its own unique DBID.

[edit] Can one restore RMAN backups without a CONTROLFILE and RECOVERY
CATALOG?

Details of RMAN backups are stored in the database control files and optionally a Recovery
Catalog. If both these are gone, RMAN cannot restore the database. In such a situation one
must extract a control file (or other files) from the backup pieces written out when the last
backup was taken. Let's look at an example:

Let's take a backup (partial in our case for ilustrative purposes):

$ rman target / nocatalog

Recovery Manager: Release 10.1.0.2.0 - 64bit Production
Copyright (c) 1995, 2004, Oracle. All rights reserved.

connected to target database: ORCL (DBID=1046662649)

using target database controlfile instead of recovery catalog

RMAN> backup datafile 1;

Starting backup at 20-AUG-04

allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=146 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00001 name=/oradata/orcl/system01.dbf
channel ORA_DISK_1: starting piece 1 at 20-AUG-04
channel ORA_DISK_1: finished piece 1 at 20-AUG-04
piece handle=
/
flash_recovery_area/ORCL/backupset/2004_08_20/o1_mf_nnndf_TAG20040820T153256_0lcz
d9tf_.bkp comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:45
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current controlfile in backupset
including current SPFILE in backupset
channel ORA_DISK_1: starting piece 1 at 20-AUG-04
channel ORA_DISK_1: finished piece 1 at 20-AUG-04
piece handle=
/
flash_recovery_area/ORCL/backupset/2004_08_20/o1_mf_ncsnf_TAG20040820T153256_0lczf
rx8_.bkp comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:04
Finished backup at 20-AUG-04[/code]

Now, let's destroy one of the control files:

SQL> show parameters CONTROL_FILES

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /oradata/orcl/control01.ctl,
/oradata/orcl/control02.ctl,
/oradata/orcl/control03.ctl
SQL> shutdown abort;
ORACLE instance shut down.
SQL> ! mv /oradata/orcl/control01.ctl /tmp/control01.ctl</pre>

Now, let's see if we can restore it. First we need to start the databaase in NOMOUNT mode:

SQL> startup NOMOUNT

ORACLE instance started.

Total System Global Area 289406976 bytes

Fixed Size 1301536 bytes
Variable Size 262677472 bytes
Database Buffers 25165824 bytes
Redo Buffers 262144 bytes</pre>

Now, from SQL*Plus, run the following PL/SQL block to restore the file:

DECLARE
v_devtype VARCHAR2(100);
v_done BOOLEAN;
v_maxPieces NUMBER;

TYPE t_pieceName IS TABLE OF varchar2(255) INDEX BY binary_integer;

v_pieceName t_pieceName;
BEGIN
 -- Define the backup pieces... (names from the RMAN Log file)
v_pieceName(1) :=
'/flash_recovery_area/ORCL/backupset/2004_08_20/o1_mf_ncsnf_TAG20040820T153256_
0lczfrx8_.bkp';
v_pieceName(2) :=
'/flash_recovery_area/ORCL/backupset/2004_08_20/o1_mf_nnndf_TAG20040820T153256_
0lczd9tf_.bkp';
v_maxPieces := 2;

-- Allocate a channel... (Use type=>null for DISK, type=>'sbt_tape' for TAPE)

v_devtype := DBMS_BACKUP_RESTORE.deviceAllocate(type=>NULL, ident=>'d1');

-- Restore the first Control File...

DBMS_BACKUP_RESTORE.restoreSetDataFile;

-- CFNAME mist be the exact path and filename of a controlfile taht was backed-up
DBMS_BACKUP_RESTORE.restoreControlFileTo(cfname=>'/app/oracle/oradata/orcl/control0
1.ctl');

dbms_output.put_line('Start restoring '||v_maxPieces||' pieces.');

FOR i IN 1..v_maxPieces LOOP
dbms_output.put_line('Restoring from piece '||v_pieceName(i));
DBMS_BACKUP_RESTORE.restoreBackupPiece(handle=>v_pieceName(i), done=>v_done,
params=>null);
exit when v_done;
END LOOP;

-- Deallocate the channel...

DBMS_BACKUP_RESTORE.deviceDeAllocate('d1');
EXCEPTION
WHEN OTHERS THEN
DBMS_BACKUP_RESTORE.deviceDeAllocate;
RAISE;
END;
/

Let's see if the controlfile was restored:

SQL> ! ls -l /oradata/orcl/control01.ctl
-rw-r----- 1 oracle dba 3096576 Aug 20 16:45 /oradata/orcl/control01.ctl[/code]

We should now be able to MOUNT the database and continue recovery...

SQL> ! cp /oradata/orcl/control01.ctl /oradata/orcl/control02.ctl

SQL> ! cp /oradata/orcl/control01.ctl /oradata/orcl/control03.ctl

SQL> alter database mount;

SQL> recover database using backup controlfile;

ORA-00279: change 7917452 generated at 08/20/2004 16:40:59 needed for thread 1
ORA-00289: suggestion :
/flash_recovery_area/ORCL/archivelog/2004_08_20/o1_mf_1_671_%u_.arc
ORA-00280: change 7917452 for thread 1 is in sequence #671

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

/oradata/orcl/redo02.log
Log applied.
Media recovery complete.

Database altered.

SQL> alter database open resetlogs;

Database altered.

See Metalink Note 60545.1 for detailed examples.