How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix

systems [ID 1062983.1]

Modified 10-AUG-2011

Type HOWTO

Status PUBLISHED

In this Document
Goal
Solution
References

Applies to:
Oracle Server - Enterprise Edition - Version: 11.2.0.1.0 to 11.2.0.2 - Release: 11.2 to 11.2
Information in this document applies to any platform.

Goal
It is not possible to directly restore a manual or automatic OCR backup if the OCR is located in
an ASM disk group. This is caused by the fact that the command 'ocrconfig -restore' requires
ASM to be up & running in order to restore an OCR backup to an ASM disk group. However, for
ASM to be available, the CRS stack must have been successfully started. For the restore to
succeed, the OCR also must not be in use (r/w), i.e. no CRS daemon must be running while the
OCR is being restored.
A description of the general procedure to restore the OCR can be found in the documentation,
this document explains how to recover from a complete loss of the ASM disk group that held the
OCR and Voting files in a 11gR2 Grid environment.

Solution
When using an ASM disk group for CRS there are typically 3 different types of files located in
the disk group that potentially need to be restored/recreated:
the Oracle Cluster Registry file (OCR)
the Voting file(s)
the shared SPFILE for the ASM instances
The following example assumes that the OCR was located in a single disk group used
exclusively for CRS. The disk group has just one disk using external redundancy.
Since the CRS disk group has been lost the CRS stack will not be available on any node.
The following settings used in the example would need to be replaced according to the actual
configuration:
GRID user:
GRID home:
ASM disk group name for OCR:
ASM/ASMLIB disk name:
Linux device name for ASM disk:
Cluster name:
Nodes:

oragrid
/u01/app/11.2.0/grid ($CRS_HOME)
CRS
ASMD40
/dev/sdh1
rac_cluster1
racnode1, racnode2

1. Locate the latest automatic OCR backup

When using a non-shared CRS home, automatic OCR backups can be located on any node of
the cluster, consequently all nodes need to be checked for the most recent backup:
$ ls -lrt $CRS_HOME/cdata/rac_cluster1/
-rw------- 1 root root 7331840 Mar 10 18:52
-rw------- 1 root root 7651328 Mar 26 01:33
-rw------- 1 root root 7651328 Mar 29 01:33
-rw------- 1 root root 7651328 Mar 30 01:33
-rw------- 1 root root 7651328 Mar 30 01:33
-rw------- 1 root root 7651328 Mar 30 05:33
-rw------- 1 root root 7651328 Mar 30 09:33

week.ocr
week_.ocr
day.ocr
day_.ocr
backup02.ocr
backup01.ocr
backup00.ocr

2. Start the CRS stack in exclusive mode

On the node that has the most recent OCR backup, log on as root and start CRS in exclusive
mode, this mode will allow ASM to start & stay up without the presence of a Voting disk and
without the CRS daemon process (crsd.bin) running.
11.2.0.1:
# $CRS_HOME/bin/crsctl start crs -excl
...
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded

Please note:
This document assumes that the CRS diskgroup was completely lost, in which case the CRS
daemon (resource ora.crsd) will terminate again due to the inaccessibility of the OCR - even if
above message indicates that the start succeeded.
If this is not the case - i.e. if the CRS diskgroup is still present (but corrupt or incorrect) the CRS
daemon needs to be shutdown manually using:
# $CRS_HOME/bin/crsctl stop res ora.crsd -init

otherwise the subsequent OCR restore will fail.

11.2.0.2:
# $CRS_HOME/bin/crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
...
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'auw2k3'
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.drivers.acfs' on 'racnode1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded

IMPORTANT:
A new option '-nocrs' has been introduced with 11.2.0.2, which prevents the start of the
ora.crsd resource. It is vital that this option is specified, otherwise the failure to start the ora.crsd
resource will tear down ora.cluster_interconnect.haip, which in turn will cause ASM to crash.
3. Label the CRS disk for ASMLIB use
If using ASMLIB the disk to be used for the CRS disk group needs to stamped first, as user root
do:
# /usr/sbin/oracleasm createdisk ASMD40 /dev/sdh1
Writing disk header: done
Instantiating disk: done

4. Create the CRS diskgroup via sqlplus

The disk group can now be (re-)created via sqlplus from the grid user. The compatible.asm
attribute must be set to 11.2 in order for the disk group to be used by CRS:
$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Tue Mar 30 11:47:24 2010
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup CRS external redundancy disk 'ORCL:ASMD40' attribute
'COMPATIBLE.ASM' = '11.2';
Diskgroup created.
SQL> exit

5. Restore the latest OCR backup

Now that the CRS disk group is created & mounted the OCR can be restored - must be done as
the root user:
# cd $CRS_HOME/cdata/rac_cluster1/
# $CRS_HOME/bin/ocrconfig -restore backup00.ocr

6. Start the CRS daemon on the current node (11.2.0.1 only !)

Now that the OCR has been restored the CRS daemon can be started, this is needed to
recreate the Voting file. Skip this step for 11.2.0.2.0.
# crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded

7. Recreate the Voting file

The Voting file needs to be initialized in the CRS disk group:
# $CRS_HOME/bin/crsctl replace votedisk +CRS
Successful addition of voting disk 00caa5b9c0f54f3abf5bd2a2609f09a9.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced

8. Recreate the SPFILE for ASM (optional)

Please note:
If you are
- not using an SPFILE for ASM
- not using a shared SPFILE for ASM
- using a shared SPFILE not stored in ASM (e.g. on cluster file system)
this step possibly should be skipped.
Also use extra care in regards to the asm_diskstring parameter as it impacts the discovery of
the voting disks.
Please verify the previous settings using the ASM alert log.
Prepare a pfile (e.g. /tmp/asm_pfile.ora) with the ASM startup parameters - these may vary from
the example below. If in doubt consult the ASM alert log as the ASM instance startup should list
all non-default parameter values. Please note the last startup of ASM (in step 2 via CRS start)

will not have used an SPFILE, so a startup prior to the loss of the CRS disk group would need to
be located.
*.asm_power_limit=1
*.diagnostic_dest='/u01/app/oragrid'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'

Now the SPFILE can be created using this PFILE:

$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Tue Mar 30 11:52:39 2010
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create spfile='+CRS' from pfile='/tmp/asm_pfile.ora';
File created.
SQL> exit

9. Shutdown CRS
Since CRS is running in exclusive mode, it needs to be shutdown to allow CRS to run on all
nodes again. Use of the force (-f) option may be required:
# $CRS_HOME/bin/crsctl stop crs -f
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on
'auw2k3' has completed
CRS-4133: Oracle High Availability Services has been stopped.

10. Rescan ASM disks

If using ASMLIB rescan all ASM disks on each node as the root user:
# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASMD40"

11. Start CRS

As the root user submit the CRS startup on all cluster nodes:
# $GRID_HOME/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

12. Verify CRS

To verify that CRS is fully functional again:
# $GRID_HOME/bin/crsctl check cluster -all
**************************************************************
racnode1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racnode2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

# $GRID_HOME/bin/crsctl status resource -t

...

References

Related

Products

Oracle Database Products > Oracle Database > Oracle Database > Oracle Server Enterprise Edition

Keywords
11GR2; ASM; CRS; DISKGROUP; OCR; RESTORE
Errors
CRS-4537; CRS-2672; CRS-4266; CRS-4529; CRS-2676; CRS-4533; CRS-2793; CRS4133; CRS-4123

How to Copy asm files between remote ASM instances using ASMCMD command [ID
785580.1]
Modified 14-APR-2011

Type HOWTO

Status PUBLISHED

In this Document
Goal
Solution
References

Applies to:
Oracle Server - Enterprise Edition - Version: 11.1.0.6 to 11.2.0.2 - Release: 11.1 to 11.2
Information in this document applies to any platform.
Oracle Server Enterprise Edition - Version: 11.1 to 11.1
***Checked for relevance on 14-Jan-2011***

Goal
The purpose of this document is to provide information about the Copy the asm files between
remote instances and local instances using ASMCMD.

Solution
11g new feature,you can use asmcmd to copy files between remote instances
Enables you to copy files between ASM disk groups on local instances and remote instances.
You can also use this command to copy files from ASM disk groups to the operating system.
cp -ifr <source file name> <user_name>@<host_name>.<Port Number>.<SID>:<targer

path>/<target file name>

user_name@host_name.<Port Number>.<SID>
The user_name, host_name, and SID are required. The default port number is 1521.
Example :asmcmd>cp -ifr +DATA/RAC/PARAMETERFILE/spfile.257.678975489
sys@stgrac1.1521.+ASM2:+FRA/RAC/ARCHIVELOG/spfile

Troubleshooting-ASMCMD remote copy

asmcmd remote copy works through listener connection.
ASMCMD remote connection can fail with below generic error.
ASMCMD-08202: internal error: [asmcmdshare_error_msg_05] [8201]
[8201] means unable to connect remote ASM Instance.
It could be due to following reason.
* not able to reach remote host.
* Remote host listener is down.
* Remote ASM Instance is not registered with listener and running non-default
port.
* sysasm remote connection does not work.
* Incorrect password given for sys user.
* Remote ASM Instance password file missing.

We need to enable additional tracing for asmcmd connection to get a exact failure message.
++ set DBI_TRACE environment variable for asmcmd perl tracing.
export DBI_TRACE=1
++ Now connect using asmcmd and re-produce the issue.
Example 1:asmcmd>cp +data/spfileorcl.ora.289.686235413 sys@stgrac1.1521.+ASM1:+test
-> DBI->connect(dbi:Oracle:host=stgrac1;port=1521;sid=+ASM1, sys, ****, HASH (0x8b2b044))
connect using '(DESCRIPTION=(ADDRESS=(HOST=stgrac1)(PROTOCOL=tcp)(PORT=1521))
(CONNECT_DATA=(SID=+ASM1)))'
ERROR: '1031' 'ORA-01031: insufficient privileges
(DBD ERROR: OCISessionBegin)'
<- DESTROY= undef at DBI.pm line 591
DBI connect('host=stgrac1;port=1521;sid=+ASM1','sys',...) failed: ORA-01031: insufficient
privileges (DBD ERROR: OCISessionBegin)
KK FROM HERE A
ASMCMD-08202: internal error: [asmcmdshare_error_msg_05] [8201]

Here we can see that asmcmd copy failed due to ORA-01031.

ASMCMD uses SYSASM by default if -a option is not used.
Here the problem is sysasm privelege was not given to sys user on remote ASM Instance.
Given the SYSASM privilege to SYS ( or the user trying to connect ). When you grant a system
privilege, the password file is updated.
SQL> grant sysasm to sys;
Grant succeeded.
SQL> select * from v$pwfile_users;
USERNAME SYSDB SYSOP SYSAS
------------------------------ ----- ----- ----SYS TRUE TRUE TRUE
Now the remote asmcmd copy works fine
For more detals,please go through the below notes
Note.730067.1 - Troubleshooting ORA-1031 Insufficient Privilege
Note.578796.1 - ORA-01031 While Connecting as SYSASM
Example 2:ASMCMD> cp -ifr thread_2_seq_5.264.678983423 sys@bderac2vip.1521.+ASM2:+FRA/RAC/ARCHIVELOG/
Enter password: ***
ASMCMD-08016: copy source>'+FRA/RAC/ARCHIVELOG/2009_02_16/thread_2_seq_5.264.678983423' and target>'+FRA/RAC/ARCHIVELOG/thread_2_seq_5.264.678983423' failed
ORA-17628: Oracle error 19505 returned by remote Oracle server
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 258
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
ASMCMD>
Solution:The cp command failed because the target ASM file name was not specified or File name
should not contain the file number/incarnation.We can not copy OMF form files without
specifying file name
cp -ifr thread_2_seq_5.264.678983423 sys@bderac2vip.1521.+ASM2:+FRA/RAC/ARCHIVELOG/thread_2_seq_5
The file number/incarnation will be created automatically during the copy.

References
BUG:8274003 - ASMCMD REMOTE COPY BETWEEN ASM INSTANCES IS NOT WORKING
NOTE:451900.1 - ASMCMD - New commands in 11gR1
NOTE:578796.1 - ORA-01031, ORA-01017 While Connecting as SYSASM
NOTE:730067.1 - Troubleshooting ORA-1031 Insufficient Privilege While Connecting As
SYSDBA
http://download.oracle.com/docs/cd/B28359_01/server.111/b31107/asm_util.htm

Related

Products

Oracle Database Products > Oracle Database > Oracle Database > Oracle Server Enterprise Edition

Keywords
ASM; ASMCMD; FILES
Errors
ORA-12538; ORA-17628; ORA-6512; ORA-1031; ERROR 19505

How to move ASM database files from one diskgroup to another ? [ID 330103.1]
Modified 03-DEC-2010

Type HOWTO

Status PUBLISHED

In this Document
Goal
Solution
References

Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.2.0.2 - Release: 10.2 to 11.2
Information in this document applies to any platform.
***Checked for relevance on 03-Dec-2010***

Goal
MOVING ASM DATABASE FILES FROM ONE DISKGROUP TO ANOTHER:

Solution

The preferred way of doing the file movement amoung ASM DISKGROUPS is using RMAN.
RMAN is critical to Automatic Storage Management and is responsible for tracking the ASM
filenames and for deleting obsolete ASM files. Since ASM files cannot be accessed through
normal operating system interfaces, RMAN is the preferred means of copying ASM file.

The steps involved in moving a datafile from a diskgroup to another is as given below.
1) Identify the data file to be moved.
2) Identify the diskgroup on to which the file has to be moved.
3) Take the file offline.

4) Copy the file to new diskgroup using Either RMAN or DBMS_FILE_TRANSFER.

5) Rename the file to point to new location.
6) Recover the file.
7) Bring the file online.
8) Verify the new file locations.
9) Delete the file from its original location.
1) Identify the data file to be moved.
---------------------------------------In database instance
SQL:ORCL> SELECT FILE_NAME FROM DBA_DATA_FILES:
+ASMDSK2/orcl/datafile/users.256.565313879 <======= Move this to
ASDSK1.
+ASMDSK1/orcl/sysaux01.dbf
+ASMDSK1/orcl/undotbs01.dbf
+ASMDSK1/orcl/system01.dbf
2) Identify the diskgroup on to which the file has to be moved.
-------------------------------------------------------------In ASM instance
SQL:ASM> SELECT GROUP_NUMBER, NAME FROM V$ASM_DISKGROUP;
GROUP_NUMBER NAME
------------ --------1 ASMDSK1
2 ASMDSK2

3) Take the file offline.

-------------------------SQL:ORCL> ALTER DATABASE DATAFILE
'+ASMDSK2/orcl/datafile/users.256.565313879' OFFLINE;

4)Now Copy the file from Source diskgroup ASMDSK1 to target Diskgroup ASMDSK2.
-------------------------------------------------------------------------------------------Either
4. a) DBMS_FILE_TRANSFER package or
4. b) RMAN
can be used for this step.
( The step 5 to step 8 is based on the filenames from method b).
4.a).Using DBMS_FILE_TRANSFER package
SQL:ORCL>create or replace directory orcl1 as '+asmdsk1/orcl/datafile';
SQL:ASM> Alter disgroup asmdsk2 add directory '+asmdsk2/test';
SQL:ORCL> create or replace directory orcl2 as '+asmdsk2/test';

SQL:ORCL>
BEGIN
DBMS_FILE_TRANSFER.COPY_FILE(
source_directory_object => 'ORCL1',
source_file_name => 'users.259.565359071',
destination_directory_object => 'ORCL2',
destination_file_name => 'USERS01.DBF');
END;
Database altered.
4 b).Using RMAN copy the file to new diskgroup.
$ rman target system@orcl10
target database Password:
connected to target database: ORCL (DBID=1089529226)
RMAN>
RMAN> COPY DATAFILE '+ASMDSK2/orcl/datafile/users.256.565313879' TO
'+ASMDSK1';
Starting backup at 03-AUG-05
using target database controlfile instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=146 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00004 name=+ASMDSK2/orcl/datafile/users.256.565313879
output filename=+ASMDSK1/orcl/datafile/users.259.565359071
tag=TAG20050803T12110
9 recid=2 stamp=565359071
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 03-AUG-05
5) Rename the file to point to new location.
------------------------------------------If you have used DBMS_FILE_TRANSFER (method 4 a)) use the following command to
rename:
SQL:ORCL> ALTER DATABASE RENAME FILE
'+ASMDSK2/orcl/datafile/users.256.565313879' TO
'+ASMDSK1/orcl/datafile/users.259.565359071'
Database altered.
If you have used RMAN (method 4 b) use the following option of RMAN
RMAN run {
set newname for datafile
'+ASMDSK2/orcl/datafile/users.256.565313879'
to '+ASMDSK1/orcl/datafile/users.259.565359071'
;
switch datafile all;
}
6) Recover the file.
------------------SQL:ORCL> RECOVER DATAFILE '+ASMDSK1/orcl/datafile/users.259.565359071'
Media recovery complete.
7) Bring the file online.
-----------------------

SQL:ORCL>ALTER DATABASE DATAFILE

'+ASMDSK1/orcl/datafile/users.259.565359071' ONLINE
Database altered.
8) Verify the new file location.
--------------------------------SQL:ORCL> SELECT FILE_NAME FROM DBA_DATA_FILES;
FILE_NAME
------------------------------------------------------------------------------+ASMDSK1/orcl/datafile/users.259.565359071
+ASMDSK1/orcl/sysaux01.dbf
+ASMDSK1/orcl/undotbs01.dbf
+ASMDSK1/orcl/system01.dbf
9) Delete the file from its original location either per SQLPLUS or per ASMCMD:
e.g.: SQL:ASM> ALTER DISKGROUP ASMDSK2 DROP FILE users.256.565313879;
or: ASMCMD> rm -rf <filename>
Note:
Most Automatic Storage Management files do not need to be manually deleted because, as
Oracle managed files, they are removed automatically when they are no longer needed.
However, if you need to drop an Oracle Managed File (OMF) manually you should use the
fully qualified filename if you reference the file. Otherwise you will get an error (e.g. ORA15177).
e.g.: SQL:ASM> ALTER DISKGROUP ASMDSK2 DROP FILE
'+ASMDSK2/orcl/datafile/users.256.565313879';
Note: The steps provided above assume that the database is open and in Archivelog mode.
Besides these steps are not appropriated for system or sysaux datafiles. For System and
Sysaux an approach similar to the one given below can be used:
1. Create a Copy of datafile in target Diskgroup:
RMAN> backup as copy tablespace system format '<New DG>';
RMAN> backup as copy tablespace sysaux format '<New DG>';
2. Then shutdown the database and restart to a mounted state
RMAN> shutdown immediate;
RMAN> startup mount;
3. switch the datafiles to the copy
RMAN> switch tablespace system to copy;
RMAN> switch tablespace sysaux to copy;
4. Recover the changes made to these tablespaces;
RMAN> recover database;

References

NOTE:330414.1 - How to Find the free space in ASM Diskgroups?

Related

Products

Oracle Database Products > Oracle Database > Oracle Database > Oracle Server Enterprise Edition

Keywords
TABLESPACE; FILES; ASM; DATA FILE; DISK GROUP; LOCATION MOVE
Errors
ORA-15177