MetroMirror For Oracle With V7000

Disaster Recovery solution for Oracle 11gR2 database using Metro Mirror and Global Mirror features of IBM
Storwize V7000
Proof of concept
Mayur Shetty
IBM Systems and Technology Group ISV Enablement March 2011

Copyright IBM Corporation, 2011.
Copyright IBM Corporation, 2011. All Rights Reserved
Table of contents
1 Abstract.................................................................................................................................. 1 2 Introduction ........................................................................................................................... 1
2.1 Overview ........................................................................................................................................... 1 2.2 Assumptions .................................................................................................................................... 1 2.3 Intended audience ........................................................................................................................... 1
3 IBM Storwize V7000 technology stack ................................................................................. 1

3.1 IBM Storwize V7000 overview......................................................................................................... 2 3.1.1 IBM Storwize V7000 functionalities............................................................................................. 2 3.1.2 IBM Storwize V7000 terminology ................................................................................................ 4
4 IBM Storwize V7000 and Remote Copy................................................................................ 5

4.1 Remote Copy overview ................................................................................................................... 5 4.2 Remote Copy concepts................................................................................................................... 5 4.3 Remote Copy mechanism to copy data ........................................................................................ 6 4.3.1 Metro Mirror ............................................................................................................ 6 4.3.2 Global Mirror ........................................................................................................... 7 4.4 Remote Copy and consistency group states................................................................................ 8
5 Configuration and setup ..................................................................................................... 10

5.1 Lab test environment .................................................................................................................... 10 5.2 Lab configuration........................................................................................................................... 12 5.3 IBM Storwize V7000 setup for Remote Copy .............................................................................. 13 5.4 Power Systems server setup for Remote Copy.......................................................................... 35 5.5 Oracle clone server setup............................................................................................................. 37
6 Oracle database replication................................................................................................ 38

6.1 Full database replication............................................................................................................... 38 6.1.1 Setup for full database remote copy .................................................................. 38
7 Summary.............................................................................................................................. 83 Resources............................................................................................................................... 84 Trademarks and special notices ........................................................................................... 85
Disaster Recovery solution for Oracle 11gR2 database using Metro Mirror and Global Mirror features of IBM Storwize V7000 Copyright IBM Corporation, 2011. All Rights Reserved
1 Abstract
This white paper demonstrates the Metro Mirror and Global Mirror features of the IBM Storwize V7000 for a disaster-recovery solution of the Oracle 11gR2 database. The paper also explains the ease with which Metro Mirroring and Global Mirroring can be configured using the IBM Storwize V7000 graphical user interface (GUI).
2 Introduction
This section provides an overview on the various topics explained in this paper, the assumptions made, and the intended audience that are likely to benefit from this paper.
2.1 Overview
The remote copy feature is a flexible data-mirroring technology that allows replication between volumes on two or more disk storage systems, which can be used for data backup and disaster recovery. The copy services layer sits above and operates independently of the function or characteristics of the underlying disk subsystems used to provide storage resources to an IBM Storwize V7000 storage system. Metro Mirror supports synchronous mirroring of logical volumes between two or more IBM Storwize V7000 models that can be located up to 300 km (190 miles) from each other. It is a synchronous copy solution where write operations are completed on both copies (local and remote site) before they are considered to be completed. Global Copy copies data asynchronously between two or more IBM Storwize V7000 models that are located approximately 8000 km (5000 miles) apart. With Global Mirror, the host I/O completes locally and the changed data is sent to the remote site later. This is designed to maintain a consistent recoverable copy of data at the remote site which lags behind the local site.
2.2 Assumptions
The team assumes that the best practices that are documented in the Implementing the IBM Storwize V7000 (see http://www.redbooks.ibm.com/redpieces/abstracts/sg247938.html?Open). IBM Redbooks title holds good for Oracle databases, too.
2.3 Intended audience

The intended audience of this paper is any technical lead, system administrator, storage administrator, or an Oracle database administrator (DBA) who is planning on deploying the IBM Storwize V7000 for running Oracle database. This paper provides a starting point for Remote Copy configuration, as well as provides an idea on how you can use the IBM Storwize V7000 Metro Mirror and Global Mirror features for disaster recovery of Oracle database.
3 IBM Storwize V7000 technology stack

This section includes concepts related to the IBM Storwize V7000.
3.1 IBM Storwize V7000 overview

The IBM Storwize V7000 solution provides a modular storage system that includes the capability to virtualize external storage area network (SAN) attached storage as well as its own internal storage. IBM Storwize V7000 provides a number of preset configuration options which are aimed at simplifying the implementation process. It also provides an active-active solution, and is a clustered, scalable, midrange storage, as well as an external virtualization device. Included with IBM Storwize V7000 is a simple and easy to use graphical user interface (GUI)I that allows storage to be deployed quickly and efficiently. The GUI runs on the IBM Storwize V7000 system so there is no need for a separate console. The IBM Storwize V7000 solution consists of these components: Control enclosure: A hardware unit that includes the chassis, node canisters, drives, and energy sources that include batteries. Expansion enclosure: A hardware unit that includes expansion canisters, drives, and energy sources that do not include batteries.
Within each enclosure are two canisters that are hardware units that includes the node hardware, fabric and service interfaces, and serial-attached SCSI (SAS) expansion ports, which determine the role of the enclosure
3.1.1 IBM Storwize V7000 functionalities

The following functionalities and features are available with the IBM Storwize V7000: Thin provisioning: With thin provisioning, applications can grow dynamically, but consume only the space that they are actually using. Without thin provisioning, preallocated space is reserved irrespective of whether the application uses it or not. Volume Mirroring: Volume Mirroring provides a single volume image to the attached host systems while maintaining pointers to two copies of data in separate storage pools. Copies can be on completely separate disk storage systems that are being virtualized. IBM FlashCopy: FlashCopy creates instant application copies that can be used for backup or application testing. FlashCopy makes better use of space with incremental copy where only changed blocks are copied. Metro Mirror: Metro Mirror provides synchronous remote mirroring function up to approximately 300 km (190 miles) between sites. As the host I/O completes only after the data is cached at both locations, performance requirements may limit the practical distance. Metro Mirror is designed to provide fully-synchronized copies at both sites with zero data loss after the initial copy is completed. Global Mirror: Global Mirror provides long-distance asynchronous remote mirroring function up to approximately 8000 km (5000 miles) between sites. With Global Mirror, the host I/O is completed locally, and the changed data is sent to the remote site later. This is designed to maintain a consistent, recoverable copy of data at the remote site which lags behind the local site. Data migration: A data-migration wizard can be used to import external storage system into the IBM Storwize V7000.
IBM Easy Tier: Provides a mechanism to seamlessly migrate hot spots to a higher performing storage pool within the IBM Storwize V7000 solution. This can be to internal drives within IBM Storwize V7000 or to external storage systems that are virtualized by IBM Storwize V7000.
Figure 1 shows how you can use IBM Storwize V7000 to virtualize your storage.
Figure 1: IBM Storwize V7000 virtualizing external and internal storage
3.1.2 IBM Storwize V7000 terminology

Table 1 lists the Storwize V7000 terminology used in the paper. For an entire list of the Storwize V7000 terminology, refer to the Implementing the IBM Storwize V7000 Redbooks title (see http://www.redbooks.ibm.com/redpieces/abstracts/sg247938.html?Open). IBM Storwize V7000 term Control enclosure Definition A hardware unit that includes the chassis, node canisters, drives, and energy sources that include batteries. A hardware unit that includes expansion canisters, drives, and energy sources that do not include batteries. A hardware unit that includes the node hardware, fabric and service interfaces, and serial-attached SCSI (SAS) expansion ports. The process of controlling in which hosts have access to specific volumes within a cluster. Array managed disks and drives that are held in enclosures and nodes that are part of the cluster. A component of a storage pool that is managed by a cluster. An MDisk is either part of a RAID array of internal storage or a Small Computer System Interface (SCSI) logical unit (LU) for external storage. An MDisk is not visible to a host system on the SAN. A collection of storage capacity that provides the capacity requirements for a volume. The ability to define a storage unit (full system, storage pool, or volume) with a logical capacity size that is larger than the physical capacity assigned to that storage unit. A discrete unit of storage on disk, tape, or other data recording medium that supports some form of identifier and parameter list, such as a volume label or input/output control Each MDisk is divided into segments of equal size called extents. When a volume is created from a storage pool, the volume is allocated based on the number of extents required to satisfy the capacity requirements for the volume.
Expansion enclosure
Node canister
Host mapping Internal storage Managed disk (MDisk)
Storage pool Thin provisioning or thin provisioned
Volume
Extent
Table 1: IBM Storwize V7000 terminology
4 IBM Storwize V7000 and Remote Copy

This section gives an overview of the Remote Copy feature in the IBM Storwize V7000 and mechanism in which Metro Mirror and Global Mirror move the data between the remote volumes of the IBM Storwize V7000.
4.1 Remote Copy overview

The general application of remote copy seeks to maintain two copies of data separated by distance. The remote copy can be maintained in one of the two modes: synchronous or asynchronous. With the IBM Storwize V7000, Metro Mirror and Global Mirror are the IBM branded terms for the functions that are synchronous remote copy and asynchronous remote copy, respectively. Synchronous remote copy ensures that updates are committed at both the primary and the secondary before the application considers the updates complete; therefore, the secondary is fully up-to-date if it is needed in a failover. However, the application is fully exposed to the latency and bandwidth limitations of the communication link to the secondary. In asynchronous remote copy the application is provided acknowledgement that the write is complete prior to the write being committed at the secondary. Hence, on a failover, certain updates (data) might be missing at the secondary.
4.2 Remote Copy concepts

This section talks about the Remote Copy concepts that are used in this white paper. Remote copy Partnership: A Partnership is created by connecting two IBM Storwize V7000 storage systems separated by a distance. After the partnership creation has been configured on both systems, further communication between the node canisters in each of the storage systems is established and maintained by the SAN network. All inter-cluster communication goes through the Fiber Channel network. Partnership state must be Fully Configured on both IBM Storwize V7000s to make it fully functional. Remote copy Relationship: A Remote Copy relationship is a relationship between two individual volumes of the same size. These volumes are called a master (source) volume and an auxiliary (target) volume. Typically, the master volume contains the production copy of the data and is the volume that the application normally accesses. The auxiliary volume typically contains a backup copy of the data and is used for disaster recovery. The master and auxiliary volumes are defined when the relationship is created, and these attributes never change. However, either volume can operate in the primary or secondary role as necessary. The primary volume contains a valid copy of the application data and receives updates from the host application, analogous to a source volume. The secondary volume receives a copy of any updates to the primary volume, because these updates are all transmitted across the mirror link. Therefore, the secondary volume is analogous to a continuously updated target volume. When a relationship is created, the master volume is assigned the role of primary volume and the auxiliary volume is assigned the role of secondary volume. Therefore, the initial copying direction is from master to auxiliary. When the relationship is in a consistent state, you can reverse the copy direction. The two volumes in a relationship must be the same size. The Remote Copy relationship is supported to be established on the volumes within one IBM
Storwize V7000 storage system, which called intra-cluster relationship, or in different IBM Storwize V7000 storage systems, which called inter-cluster relationship. Usage of Remote Copy target volumes as Remote Copy source volumes is not allowed. A FlashCopy target volume cannot be used as Remote Copy source volume. Remote copy Consistency Group: A Consistency Group is a logical entity that groups copy relationships. Grouping the relationships ensures that the relationships are managed in unison and the data within the group is in a Consistent state.
4.3 Remote Copy mechanism to copy data

There are two different mechanisms based on which data can be copied between source volumes and target volumes: Metro Mirror Global Mirror
4.3.1 Metro Mirror

Metro Mirror is a type of remote copy that creates a synchronous copy of data from a master volume to an auxiliary volume. With synchronous copies, host applications write to the master volume but do not receive confirmation that the write operation has completed until the data is written to the auxiliary volume. This ensures that both the volumes have identical data when the copy completes. After the initial copy completes, the Metro Mirror function maintains a fully synchronized copy of the source data at the target site at all times. Figure 2 illustrates how a write to the master volume is mirrored to the cache of the auxiliary volume before an acknowledgement of the write is sent back to the host that issued the write. This process ensures that the auxiliary is synchronized in real time, in case it is needed in a failover situation. The Metro Mirror function supports copy operations between volumes that are separated by distances up to 300 km (190 miles). For disaster recovery purposes, Metro Mirror provides the simplest way to maintain an identical copy on both the primary and secondary volumes.
Figure 2. Write between the Master volume and the Auxiliary volume in a Metro Mirror Relationship
4.3.2 Global Mirror

The Global Mirror provides an asynchronous copy, which means that the secondary volume is not an exact match of the primary volume at every point in time. Global Mirror provides long distance asynchronous remote mirroring function up to approximately 8000 km (5000 miles) between sites. The Global Mirror function provides the same function as Metro Mirror Remote Copy without requiring the hosts to wait for the full round-trip delay of the long distance link. In Global Mirror write operations are completed on the primary site and the write acknowledgement is sent to the host before it is received at the secondary site. An update of this write operation is sent to the secondary site at a later stage, which provides the capability to perform remote copy over distances exceeding the limitations of synchronous remote copy. Figure 3shows that a write operation to the master volume is acknowledged back to the host issuing the write before the write operation is mirrored to the cache for the auxiliary volume.
Figure 3. Write between the Master volume and the Auxiliary volume in a Global Mirror Relationship
4.4 Remote Copy and consistency group states

Stand-alone Remote Copy relationships and consistency groups share a common configuration and state model. All of the relationships in a nonempty consistency group have the same state as the consistency group. These states apply to both the relationships and the consistency groups, except Empty state is only for consistency groups (see Table 2). State InconsistentStopped Description The primary volumes are accessible for read and write I/O operations, but the secondary volumes are not accessible for either. A copy process must be started to make the secondary volumes consistent. The primary volumes are accessible for read and write I/O operations, but the secondary volumes are not accessible for either. This state indicates a copy process is ongoing from the primary to the secondary volume. The secondary volumes contain a consistent
InconsistentCopying
ConsistentStopped
image, but it might be out-of-date with respect to the primary volumes. This state can occur when a relationship was in the ConsistentSynchronized state and experiences an error that forces a freeze of the consistency group or the Remote Copy relationship. ConsistentSynchronized The primary volumes are accessible for read and write I/O operations. The secondary volumes are accessible for read-only I/O operations. Both the primary volumes and the secondary volumes are operating in the primary role. Consequently the volumes are accessible for write I/O operations. The volumes in this half of the consistency group are all operating in the primary role and can accept read or write I/O operations. The volumes in this half of the consistency group are all operating in the secondary role and cannot accept read or write I/O operations. The volumes in this half of the consistency group are all operating in the secondary role and can accept read I/O operations but not write I/O operations. The consistency group does not contain any relationships.
Idling
IdlingDisconnected
InconsistentDisconnected
ConsistentDisconnected
Empty
Table 2: IBM Storwize V7000 states
5 Configuration and setup

This section talks about the configuration of various components that were involved in testing. It includes: lab configuration, IBM Storwize V7000 setup, and IBM Power Systems server setup.
5.1 Lab test environment

See Table 3, Table 4and Table 5 for the lab test environment. Production Machine Model IBM,8233-E8B (IBM Power Systems 750) isvp17 IBM POWER7 24 576 MB Standby IBM,8233-E8B
Node name Processor RAM

Table 3. Oracle host node information
isvp18 PowerPC_POWER7 24 576 MB
Production OS version Oracle version Metro Mirror database name Global Mirror database name ORACLE_HOME Metro Mirror Filesystems IBM AIX 7.1 SP2 11gR2 test_mm
Standby AIX 7.1 SP2 11gR2 test_mm
test_gm
test_gm
/u01/app/oracle/product/11.2.0/dbhome_1 Data files: /oradata Control files: /oraarch Redo logs: /oralog
/u01/app/oracle/product/11.2.0/dbhome_1 Data files: /oradata Control files: /oraarch Redo logs: /oralog Data files: /gm_oradata Control files: /gm_oraarch Redo logs: /gm_oralog
Global Mirror Filesystems
Data files: /gm_oradata Control files: /gm_oraarch Redo logs: /gm_oralog
10
Table 4. Oracle database information
Primary Storage Storage Software version Storage System name Metro Mirror Volumes IBM Storwize V7000 6.1.0.0 (build 49.0.1010130001) isv7k4.storage oracle_mmtest1, oracle_mmtest2, oracle_mmtest3 oracle_gmtest1, oracle_gmtest2,oracle_gmtest3
Secondary IBM Storwize V7000 6.1.0.0 (build 49.0.1010130001) isv7kd10.storage oracle_dr_mmtest1, oracle_dr_mmtest2,oracle_dr_mmtest3
Global Mirror Volumes
oracle_dr_gmtest1, oracle_dr_gmtest2,oracle_dr_gmtest3
Table 5. IBM Storwize V7000 information
11
5.2 Lab configuration

Figure 4 shows the lab configuration for the Remote Copy scenario.
IBM Storwize V7000
Remote Copy
IBM Storwize V7000
Fibre Channel switch
Power Systems server
Power Systems server
Ethernet
Figure 4. Lab configuration for Remote Copy scenarios
12
5.3 IBM Storwize V7000 setup for Remote Copy

A Partnership is created between the two IBM Storwize V7000 storage systems that are to be used for the Disaster Recovery solution. In the IBM Storwize V7000 GUI, on the Partnership panel, you can manage the partnership between the two storage systems. To access the Partnership panel, select the Copy Services function and click Partnership as shown in the Figure 5.
Figure 5. Fully configured Partnership between the IBM Storwize V7000 storage systems
Before creating the Partnership between the two IBM Storwize V7000 storage systems, check the zoning and the system status, also make sure that the storage can see each other. After that create the partnership by selecting the appropriate remote storage systems, and define the available bandwidth between the two systems. The bandwidth that is entered here is used by the background copy process between the clusters in the system. To set the background copy bandwidth, optimally consider the primary storage, intercluster link bandwidth, and the secondary storage to avoid overloading them so as to affect the foreground I/O latency. After the partnership definition is completed on the first IBM Storwize V7000, the Partnership of the first panel displays Partially Configured: Local. Perform the partnership creation on the remote storage system, at which point the partnership pane will display Fully Configured. The next step would be to create the remote copy relationship between the two IBM Storwize V7000 storage systems. The Figure below shows the remote copy panel.
13
Figure 6. Remote Copy panel of the IBM Storwize V7000
To create a new Remote Copy relationship, in the right of the Remote Copy panel, click New Relationship. The wizard will then guide us thorough the creation of the Remote Copy relationship creation process. Based on your requirement, select Metro Mirror (synchronous replication) or Global Mirror (asynchronous replication) as shown in Figure 7, which shows the creation of a new Metro Mirror relationship.
Figure 7. New Metro Mirror Relationship creation pop up window
In the next step as shown in Figure 8, select the Remote Copy auxiliary storage system, which is remote.
14
Figure 8. Select Metro Mirror partner
The Remote Copy master and auxiliary volumes are specified as shown in Figure 9. Both the master and auxiliary volumes must be of the same size to be used in a Remote Copy relationship.
Figure 9. Select Master and Auxiliary volumes for Metro Mirror relationship
Figure 10 shows that you have created a Metro Mirror relationships between Master volume oracle_mmtest1 and Auxiliary volume oracle_dr_mmtest1, Master volume oracle_mmtest2 and Auxiliary volume oracle_dr_mmtest2, and Master volume oracle_mmtest3 and Auxiliary volume oracle_dr_mmtest3.
15
Figure 10. The three volumes paired to be used in a Metro Mirror relationship
In the next step as shown in Figure 11, you are asked if the volumes are already synchronized. As the data in the Master volumes and the Auxiliary volumes are not identical, selected No.
Figure 11. Metro Mirror volumes are not synchronized
The next step shows that you have differentiated your initial copy of the Master volumes from the Auxiliary volumes to a later time. Click Finish.
16
Figure 12. No initial copy between Metro Mirror volumes
Figure 13 shows that the new relationships that you created are now in the Inconsistent Stopped state.
Figure 13. Metro Mirror relationship creation completes
Next create a new Consistency Group to add the Metro Mirror relationships that you created. Figure 14 shows the creation of the new Consistency Group v7k_mm_OrclDR
17
Figure 14. Enter the name of the new Consistency Group for the Metro Mirror relationships
As shown in Figure 15, the wizard indicates that the auxiliary volumes are on a remote storage system.
Figure 15. Auxiliary volumes on a remote IBM Storwize V7000
Create an empty Metro Mirror consistency group as shown Figure 16, and then click Finish.
18
Figure 16. Creating an empty consistency group
The Figure 17 below shows that a new Consistency group v7k_mm_OrclDR has been created
Figure 17. Empty consistency group created
In the next Figure 18 you have highlighted the Metro Mirror volume relationships that you had created earlier, and are adding it to a Consistency Group.
19
Figure 18. Adding Metro Mirror relationships to consistency group
As shown in Figure 19, move the three Metro Mirror relationships to the Consistency Group v7k_mm_OrclDR., and then click Add to Consistency Group.
Figure 19. Name of the consistency group to move the Metro Mirror relationships
Figure 20 shows the actual commands that are run to add the 3 relationships to the Consistency Group v7k_mm_OrclDR.
20
Figure 20. Command for changing remote copy relationships
As shown in Figure 21, the three relationships have been added to the Consistency Group v7k_mmOrclDR and the Consistency Group is in the Inconsistent Stopped State.
Figure 21. Consistency group and Metro Mirror relationships in Inconsistent Stopped state
As shown in Figure 22, go to Consistency Group v7k_mm_OrclDR, and from the Actions drop down list, start the consistency group to start the initial copy of the Master volumes to the Auxiliary volumes.
21
Figure 22. Starting the Consistency Group
As shown in Figure 23, the background command that is run to start the Remote Copy of the Consistency Group.
Figure 23. Command to start remote copy consistency group
As shown in Figure 24, you see that the three relationships in the v7k_mm_OrclDR Consistency Group are in the Inconsistent Copying state.
22
Figure 24. The relationships and the consistency groups in Inconsistent Copying state
As shown in Figure 25, the bottom right-hand side of the GUI with three Running Tasks, indicates the percentage of copy that has completed between the master volumes and the auxiliary volumes.
Figure 25. Percentage of copying that has completed between the volumes
23
After the three Remote-Copy operations are completed the Metro Mirror volume relationships reaches a Consistent Synchronized state. You can see that the Consistency Group v7k_mm_OrclDR is also is Consistent Synchronized state as shown in the Figure 26.
Figure 26. The relationships and the consistent groups in Consistent Synchronized state
Global Mirror
The Figures 27 shows the creation of the Global Mirror relationship. To create a Global Mirror relationships go to the Copy Services on the left panel of the GUI, and select Remote Copy. This takes you to the panel show in the Figure below. Click New Relationship and select Global Mirror.
24
Figure 27. New Global Mirror relationship creation pop up window
As shown in Figure 28, you selected the Global Mirror relationship with auxiliary volumes on a remote IBM Storwize V7000 called isv7kd10
Figure 28. Global Mirror auxiliary volumes on a remote IBM Storwize V7000
The Figure 29 shows that the Master volumes oracle_gmtest1, oracle_gmtest2, and oracle_gmtest3 on IBM Storwize V7000 have been selected for a Global Mirror relationship with Auxiliary volumes oracle_dr_gmtest1, oracle_dr_gmtest2, and oracle_dr_gmtest3 respectively.
25
Figure 29. Volumes paired for a Global Mirror relationship
The Figure 30 below indicates that the Master volumes and the Auxiliary volumes in the Global Mirror relationships have not yet been synchronized.
Figure 30. Global Mirror volumes are not synchronized
As shown in Figure 31, for the initial copy of the Master volumes to the Auxiliary volumes at a later time, click the Finish button.
26
Figure 31. No initial copy between the Global Mirror volumes
As shown in Figure 32, the actual background command that is run to create the Global Mirror relationships.
Figure 32. Command to create the Global Mirror relationships
Figure 33 below shows that three Global relationships that have been created. The Global Mirror relationships are currently in Inconsistent Stopped state.
27
Figure 33. The Global Mirror relationships in Inconsistent Stopped state
Figure 34 below shows the creation of the Consistency Group for the Global Mirror relationships. To create a Global Mirror Consistency Group go to Copy Services on the left panel of the GUI, and select Remote Copy. This takes you to the screen show in the Figure below. Click on the New Consistency Group button and it pop up a screen where you can enter the name of new Consistency Group that you want to create. The Consistency Group that you plan on creating for the Global Mirror relationships is called v7k_gm_OrclDR
Figure 34. Consistency Group for the Global Mirror relationships
Indicate that the auxiliary volumes are located on a remote IBM Storwize V7000 called isv7kd10 as shown in Figure 35.
28
Figure 35. Auxiliary Global Mirror volumes on a remote IBM Storwize V7000
Indicate that you do not want to add the relationships to the group at this point as shown in the Figure 36.
Figure 36. No initial copy between the Global Mirror volumes
Figure 37 shows the actual command that is run in the background to create the Remote Copy consistency group.
29
Figure 37. Command to create the remote copy Consistency Group
Figure 38 shows that an empty consistency group called v7k_gm_OrclDR has been created.
Figure 38. Empty Global Mirror Consistency group created
Next, in the left panel, click the Not in a Group icon. This will show the Global Mirror relationships that you had created earlier. Select all the relationships, and right-click and select Add to Consistency Group as shown in the Figure 39.
30
Figure 39. Adding Global Mirror relationships to Consistency Group
The next panel shows the pop-up window where you select the name of the Consistency Group that you want to add your relationships. Figure 40 shows that you have selected Consistency Group v7k_gm_OrclDR to add the three relationships.
Figure 40. Consistency Group to move the Global Mirror relationships
Figure 41 shows the actual command that gets run in the background to add the 3 Global Mirror relationships to the consistency group v7k_gm_OrclDR
31
Figure 41. Command for changing remote copy relationships
Figure 42 show that the Consistency Group now has the 3 Global Mirror relationships in it, and that the state of the relationships and the consistency group is Inconsistent Stopped.
Figure 42. Consistency Group and Global Mirror relationships in Inconsistent Stopped state
To start the Consistency Group click on the Actions drop down menu, and select Start as shown in the Figure 43.
32
Figure 43. Starting the Global Mirror Consistency Group
Figure 44 shows the actual background command that gets run to start the Remote copy consistency group.
Figure 44. Command to start remote copy consistency group
Figure 45 shows that remote copy of the 3 relationships in the consistency group. During the copy process the state of the Global Mirror Relationships and the Consistency Group is Inconsistent Copying.
33
Figure 45. Global Mirror relationships and the Consistency Groups in Inconsistent Copying state
After the completion of the remote copy operation, the relationships and the consistency group are in Consistent Synchronized state as shown in the Figure 46.
Figure 46. Global Mirror relationships and the Consistent Groups in Consistent Synchronized state
34
5.4 Power Systems server setup for Remote Copy

This session talks about the setup of the Power Systems server for the Remote Copy of Oracle database. The example below talks about the setup for Metro Mirror remote copy. The setup for Global Mirror remote copy is also very similar. On the source system, create volumes groups test1, test2, and test3 on the physical disks hdisk1, hdisk2, and hdisk3. isvp17> lspv hdisk0 hdisk1 hdisk2 hdisk3 00f65d51a5aa3cf1 00f65d51bfba4e2e 00f65d5108ff9043 00f65d5108ffefb6 rootvg test1 test2 test3 active active active active
Create three 100 GB file systems /oradata , /oralog , and /oraarch on the volume groups test1, test2 , and test3 respectively. On the target system you also need to create the volume groups and the file systems, but before that you have to Stop the Consistency Group v7k_mm_OrclDR as it is in a Consistent Synchronized state, which provides only read access to the auxiliary volumes. The Remote Copy relationship in the Consistency Group can be stopped by selecting Stop in the Actions drop down list on the top of the Remote Copy panel, as shown in Figure 47
Figure 47. Stopping the Consistency Group
In the next step, allow read/write access to secondary volumes by selecting the checkbox, as shown in Figure 48, and click Stop Consistency Group to proceed.
35
Figure 48. Allowing read-write access to the secondary volumes
Figure 49 shows the actual command that is being executed to stop the Consistency Group.
Figure 49. Command to stop Consistency Group
Figure 50 shows that the relationships between the Master Volumes and the Auxiliary Volumes are now in Idling state.
36
Figure 50. Consistency Group and relationships in Idling state
On the target system you can now create the volume groups on the three physical volumes. isvp18> lspv hdisk0 hdisk1 hdisk2 hdisk3 00f65d52a5abd459 00f65d5108ff9043 00f65d5108ffefb6 00f65d51bfba4e2e rootvg test_copy1 test_copy2 test_copy3 active active active active
Just as on the source, create the /oradata , /oralog and the /oraarch file systems on the target system.
5.5 Oracle clone server setup

There are four important files to copy or edit on the Clone or Disaster Recovery Oracle database nodes to migrate the database from the primary: init<db_instance>.ora, orapw<db_instance>, oratab file and tnsnames.ora: Copy the init<db_instance>.ora file found at /u01/app/oracle/product/11.2.0/dbhome_1/dbs/init<db_instance>.ora to the target machine. Copy the orapw<db_name> file found at /u01/app/oracle/product/11.2.0/dbhome_1/dbs/orapw<db_name>, which is the database password file. Check the oratab file found at /etc/oratab and add the following: <db_name>:/u01/app/oracle/product/11.2.0/dbhome_1:N Copy the tnsnames.ora file from the primary nodes subdirectory
37
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/. Then, edit it by replacing the original hostnode names with the Clone or Disaster Recovery Oracle database host name. Create subdirectories for database logs and trace files under (that is,/u01/app/oracle/admin/<db_name> adump, bdump, cdump, dpdump, hdump, pfile and udump).
6 Oracle database replication

A disaster recovery solution for Oracle database can be provided either by replicating the entire database, or by replicating just the online redo logs of the database.
6.1 Full database replication

In this mode all the physical data of the database which are stored on the volumes of IBM Storwize V7000 are replicated using either the Remote Copy method of Metro Mirroring or Global Mirroring.
6.1.1 Setup for full database remote copy

After completing the setup of the IBM Storwize V7000 storage systems at the source and the target sites, followed by the setup of the Power servers, and Oracle database clone, you are now ready for the full database replication using Metro Mirror or Global Mirror. Metro Mirror:
On the source system, the database uses the /oradata, /oralog, and /oraarch file systems that were created on the IBM Storwize V7000 volumes for the data files, log files, and control files respectively. The Oracle database on the source system can now be mounted and opened for production purposes as the source volumes always have read write access. In the Figure 51 below of the source storage system, you can see that the State of the relationships between the Master Volumes and the Auxiliary volumes is Idling. Go to Copy Services -> Remote Copy Consistency Group v7k_mmOrclDR and click the Actions drop-down menu and click Start. This starts the copy of the data from the Source to the Auxiliary volumes of the IBM Storwize V7000 on the target site.
38
Figure 51. Start the Consistency Group for Metro Mirror
Figure 52 shows that when you start the remote copy consistency group, you have to select the primary volumes for the remote-copy. The volumes on the production system are the source volumes for this scenario.
Figure 52. Select source volumes for Metro Mirror copy
Figure 53 below shows the actual background command that is run to start the remote copy of the consistency group.
39
Figure 53. Command to start the Consistency Group
After the completion of the data copy onto the Auxiliary volumes of the target systems the State of the consistency group changes to Consistent Synchronized as shown in the Figure 54.
Figure 54. Consistency Group and Metro Mirror relationships in Consistent Synchronized state
During the Remote Copy of the data, the Auxiliary volumes on the target storage system are in read-only mode. To mount /oradata, /oralog, and /oraarch on the target host, you need to first stop the relationship by going to the IBM Storwize V7000 GUI and click Copy Services -> Remote Copy Consistency Group v7k_mmOrclDR and selecting the Actions drop down menu and then click Stop as shown in Figure 55.
40
Figure 55. Stop the Metro Mirror Consistency Group
Follow that by providing read-write access to the target volumes as shown in the Figures 56.
Figure 56. Allowing read-write access to the secondary Metro Mirror volumes
Figure 57 shows the background command that is executed while stopping the Metro Mirror consistency group.
41
Figure 57. Command to stop remote copy consistency group
After the Metro Mirror Consistency Group stops, the Consistency Group changes to Idling state as shown in Figure 58. Because you had given read-write access to the Auxiliary volumes on the target system, you can now mount the /oradata, /oralog and /oraarch filesystems on the target system, and then mount and open the Oracle database for client access.
Figure 58. Metro Mirror Consistency Group and the relationships are in Idling state
Global Mirror:
42
On the source system the database uses the /gm_oradata, /gm_oralog, and /gm_oraarch file systems that were created earlier on IBM Storwize V7000 volumes for the data files, log files, and control files respectively. The Oracle database on the source system can now be mounted and opened for production purposes as the source volumes always have read write access.
In the Figure 59 below of the source storage system, you can see that the State of the relationships between the Master Volumes and the Auxiliary volumes is Idling. Go to Copy Services -> Remote Copy Consistency Group v7k_gmOrclDR and click the Actions drop-down menu and click Start. This starts the copy of the data from the Source to the Auxiliary volumes of the IBM Storwize V7000 on the target site.
Figure 59. Start the Consistency Group for Global Mirror
The Figure 60 shows that when you start the Global Mirror Consistency Group, you have to select the primary volumes for the remote-copy. The volumes on the production system are the source volumes for this scenario.
43
Figure 60. Select source volumes for Global Mirror copy
Figure 61 shows the actual background command that is executed to start the remote copy of the consistency group.
Figure 61. Command to start the Consistency Group
After the completion of the data copy onto the Auxiliary volumes of the target systems the State of the Global Mirror Consistency Group changes to Consistent Synchronized as shown in Figure 62.
44
Figure 62. Consistency Group and Global Mirror relationships in Consistent Synchronized state
During the Remote Copy of the data the Auxiliary volumes on the target storage system are in read-only mode. To mount /gm_oradata, /gm_oralog, and /gm_oraarch on the target host we need to first stop the relationship by going to the IBM Storwize V7000 GUI Copy Services -> Remote Copy Consistency Group v7k_gmOrclDR and clicking on the Actions drop down menu and then press Stop as shown in Figure 63.
Figure 63. Stop the Global Mirror Consistency Group
Follow that by providing read-write access to the target volumes as shown in Figure 64.
45
Figure 64. Allowing read-write access to the secondary Global Mirror volumes
Figure 65 shows the background command that is executed while stopping the Global Mirror consistency group.
Figure 65. Command that is executed to stop remote copy consistency group
After the Global Mirror Consistency group stops, the Consistency Group changes to Idling state as shown in Figure 66. Because you had earlier given read-write access to the Auxiliary volumes on the target system you can now mount the /gm_oradata, /gm_oralog and /gm_oraarch filesystems on the target system, and then mount and open the Oracle database for client access.
46
Figure 66. Global Mirror Consistency Group and the relationships are in Idling state
6.1.2 Full database failover for testing and backup purposes

In this scenario the target site database is to be used for backup, test, and other development purposes without interrupting the ongoing remote copy between the Master volumes on the production system and the Auxiliary volumes on the target system. Complete the steps specified in section 6.1.1 "Setup for full database remote copy" and make sure that the data copy is available at the target site. Metro Mirror:
Figure 67 shows the creation of the Consistency Group FC_mm_OrclDR on the target IBM Storwize V7000 storage system. This new Consistency Group will hold all the FlashCopy relationships on the target storage system. The target remote copy volumes oracle_dr_mmtest1, oracle_dr_mmtest2, and oracle_dr_mmtest3 will be cloned onto volumes FC_oracle_dr_mmtest1, FC_oracle_dr_mmtest2, and FC_oracle_dr_mmtest3 on the same target storage system using FlashCopy.
47
Figure 67. Consistency Group for FlashCopy Mapping
Figure 68 shows the new FlashCopy Mapping between the auxiliary Metro Mirror volumes and new volumes that were create for cloning the data.
Figure 68. FlashCopy mappings between the volumes
Figure 69 below shows that you have selected the data to be Cloned using FlashCopy. This creates an exact replica of the source volumes when the remote copy is in progress on the source.
Figure 69. Selecting Preset Clone for the FlashCopy mapping
48
Figure 70 below shows that the FlashCopy mappings between the volumes is added to the Consistency Group FC_mm_OrclDR
Figure 70. Adding FlashCopy Mapping to the Flash Copy Consistency Group
As shown in Figure 71, the Consistency Group FC_mm_OrclDR is in Idle state.
Figure 71. FlashCopy Consistency Group is in Idle state
Figure 72 shows that the FlashCopy Consistency Group can be started by clicking on the Action drop down menu and then pressing Start.
49
Figure 72. Start FlashCopy Consistency Group
Figure 73 shows the FlashCopy in progress between the auxiliary volumes on the target system and the new volumes that were created for cloning the data. It is important to note that while the clone is accessing the remote copy target volumes, it will not be updated with the latest updates happening to the Metro Mirror target volumes.
Figure 73. FlashCopy in progress
After the completion of the FlashCopy the relationship between the FlashCopy source and the target volumes is deleted automatically. You now have an exact replica of the Metro Mirror target volumes, which you can use to create the Oracle database for testing or backup purposes. Here you describe a method of accessing the FlashCopy target volumes on a single AIX host while the source volumes is still active on the same server. While using the same host to work with source and
50
target volumes, use the recreatevg command to overcome the problem of duplicated LVM data structures and identifiers caused by a disk duplication process such as FlashCopy. It is used to recreate an AIX Volume Group (VG) on a set of target volumes that are copied from a set of source volumes belonging to a specific VG. The command will allocate new physical volume identifiers (PVIDs) for the member disks and a new Volume Group identifier (VGID) to the Volume Group. The command also provides options to rename the logical volumes with a prefix you specify, and options to rename labels to specify different mount points for filesystems. After the completion of the cloning using FlashCopy the target system will see that the new disks hdisk7, hdisk8, and hdisk9 have the same volume groups as hdisk1, hdisk2, and hdisk3 respectively. isvp18> varyoffvg test_cp1 isvp18> varyoffvg test_cp2 isvp18> varyoffvg test_cp3 isvp18> lspv hdisk0 hdisk2 hdisk3 hdisk4 hdisk5 hdisk1 hdisk6 hdisk7 hdisk8 hdisk9 00f65d52a5abd459 00f65d5108ffefb6 00f65d51bfba4e2e 00f65d51c465f8eb 00f65d52ce125f70 00f65d5108ff9043 00f65d51c465f8eb 00f65d5108ff9043 00f65d5108ffefb6 00f65d51bfba4e2e rootvg test_cp2 test_cp3 metro swap test_cp1 metro test_cp1 test_cp2 test_cp3 active active
The chdev command below will clear the source LVM data structures from the FlashCopy clones. isvp18> chdev -l hdisk7 -a pv=clear hdisk7 changed isvp18> chdev -l hdisk8 -a pv=clear hdisk8 changed isvp18> chdev -l hdisk9 -a pv=clear hdisk9 changed isvp18> lspv hdisk0 hdisk2 00f65d52a5abd459 00f65d5108ffefb6 rootvg test_cp2 active
51
hdisk3 hdisk4 hdisk5 hdisk1 hdisk6 hdisk7 hdisk8 hdisk9 isvp18>
00f65d51bfba4e2e 00f65d51c465f8eb 00f65d52ce125f70 00f65d5108ff9043 00f65d51c465f8eb none none none None None None
test_cp3 metro swap test_cp1 metro active
The recreatevg command recreates volume groups that already exist on a specified set of disks with the names fc_test_cp1, fc_test_cp2, and fc_test_cp3 respectively. The -L option change the labels of the logical volumes on the volume group being create to /backup. The -Y causes the logical volumes on the volume group being recreated to be renamed with the bkup prefix. isvp18> recreatevg -y fc_test_cp1 -L /backup -Y bkup hdisk7 fc_test_cp1 isvp18> isvp18> recreatevg -y fc_test_cp2 -L /backup -Y bkup hdisk8 fc_test_cp2 isvp18> recreatevg -y fc_test_cp3 -L /backup -Y bkup hdisk9 isvp18> lspv hdisk0 hdisk2 hdisk3 hdisk4 hdisk5 hdisk1 hdisk6 hdisk7 hdisk8 hdisk9 00f65d52a5abd459 00f65d5108ffefb6 00f65d51bfba4e2e 00f65d51c465f8eb 00f65d52ce125f70 00f65d5108ff9043 00f65d5226de2253 00f65d5226bdf18a 00f65d5226c876f6 00f65d5226ca8c57 rootvg test_cp2 test_cp3 metro swap test_cp1 fc_metro fc_test_cp1 fc_test_cp2 fc_test_cp3 active active active active active active active active active
52
The /etc/filesystems will have the entries as shown below for the filesystems /backup/oradata, /backup/oralog, and /backup/oraarch respectively /backup/oralog: dev vfs log mount check options account = /dev/bkuplv02 = jfs = /dev/bkuploglv02 = true = false = rw = false
/backup/oraarch: dev vfs log mount check options account = /dev/bkuplv03 = jfs = /dev/bkuploglv03 = true = false = rw = false
/backup/oradata: dev vfs log mount check options account = /dev/bkuplv01 = jfs = /dev/bkuploglv00 = true = false = rw = false
You then mount the three file systems and create sym links to them as shown below using the ln command. isvp18> mount /backup/oradata
53
isvp18> mount /backup/oralog isvp18> mount /backup/oraarch isvp18> isvp18> ln -s /backup/oradata/testmm /oradata isvp18> ln -s /backup/oralog/testmm /oralog isvp18> ln -s /backup/oraarch/testmm /oraarch isvp18> ls -l /oradata lrwxrwxrwx 1 root system 15 Feb 14 23:25 /oradata -> /backup/oradata/testmm
Log in as user oracle and startup the Oracle database using the file systems created on the FlashCopy clones. isvp18> su - oracle $ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Mon Feb 14 23:26:52 2011
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup; ORACLE instance started.
Total System Global Area 1.0289E+10 bytes Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. SQL>
2215712 bytes 5268046048 bytes 4999610368 bytes 18743296 bytes
54
The database is now accessible for test, backup, or other development purposes. It is important to note that while the clone is accessing the remote copy target volumes, it will not be updated with the latest updates happening to the Metro or Global mirror target volumes.
Global Mirror
Figure 74 shows the creation of the Consistency Group FC_gm_OrclDR on the target IBM Storwize V7000 storage system. This new Consistency Group will hold all the FlashCopy relationships on the target storage system. The target remote copy volumes oracle_dr_mgtest1, oracle_dr_gmtest2, and oracle_dr_gmtest3 will be cloned onto volumes FC_oracle_dr_gmtest1, FC_oracle_dr_gmtest2, and FC_oracle_dr_gmtest3 on the same target storage system using FlashCopy.
Figure 74. Consistency Group for FlashCopy Mapping
Figure 75 shows the new FlashCopy Mapping between the auxiliary Global Mirror volumes and new volumes that were create for cloning the data.
55
Figure 75. FlashCopy mappings between the volumes
Figure 76 shows that you have selected the data to be Cloned using FlashCopy. This creates an exact replica of the source volumes when the remote copy is in progress on the source.
Figure 76. Selecting Preset Clone for the FlashCopy mapping
Figure 77 below shows that the FlashCopy mappings between the volumes is added to the Consistency Group FC_gm_OrclDR
56
Figure 77. Adding FlashCopy Mapping to the Flash Copy Consistency Group
As shown in Figure 78, the Consistency Group FC_mm_OrclDR is in Idle state.
Figure 78. FlashCopy Consistency Group is in Idle state
Figure 79 shows that the FlashCopy Consistency Group can be started by clicking on the Action drop down menu and then pressing Start.
57
Figure 79. Start FlashCopy Consistency Group
Figure 80 shows the FlashCopy in progress between the auxiliary volumes on the target system and the new volumes that were created for cloning the data. It is important to note that while the clone is accessing the remote copy target volumes, it will not be updated with the latest updates happening to the Global Mirror target volumes.
Figure 80. FlashCopy in progress
After the completion of the FlashCopy the relationship between the FlashCopy source and the target volumes is deleted automatically. You now have an exact replica of the Global Mirror target volumes, which you can use to create the Oracle database for testing or backup purposes. Here you describe a method of accessing the FlashCopy target volumes on a single AIX host while the source volumes is still active on the same server. While using the same host to work with source and
58
target volumes, use the recreatevg command to overcome the problem of duplicated LVM data structures and identifiers caused by a disk duplication process such as FlashCopy. It is used to recreate an AIX Volume Group (VG) on a set of target volumes that are copied from a set of source volumes belonging to a specific VG. The command will allocate new physical volume identifiers (PVIDs) for the member disks and a new Volume Group identifier (VGID) to the Volume Group. The command also provides options to rename the logical volumes with a prefix you specify, and options to rename labels to specify different mount points for filesystems. After the completion of the cloning using FlashCopy the target system will see that the new disks hdisk15, hdisk16, and hdisk17 have the same volume groups as hdisk11, hdisk12, and hdisk13 respectively. The chdev command below will clear the source LVM data structures from the FlashCopy clones. isvp18> chdev -l hdisk16 -a pv=clear hdisk16 changed isvp18> chdev -l hdisk17 -a pv=clear hdisk17 changed The recreatevg command recreates volume groups that already exist on a specified set of disks with the names fc_testgm_cp1, fc_testgm_cp2, and fc_testgm_cp3 respectively. The -L option change the labels of the logical volumes on the volume group being create to /backup. The -Y causes the logical volumes on the volume group being recreated to be renamed with the bkup prefix. isvp18> recreatevg -y fc_testgm_cp1 -L /backup -Y bkup hdisk15 fc_testgm_cp1 isvp18> recreatevg -y fc_testgm_cp2 -L /backup -Y bkup hdisk16 fc_testgm_cp2 isvp18> recreatevg -y fc_testgm_cp3 -L /backup -Y bkup hdisk17 fc_testgm_cp3
The /etc/filesystems will have the entries as shown below for the filesystems /backup/gm_oradata, /backup/gm_oralog, and /backup/gm_oraarch respectively cat /etc/filesystems /backup/gm_oradata: dev vfs log mount check options = /dev/bkuplv04 = jfs = /dev/lv05 = true = false = rw
59
account
= false
/backup/gm_oralog: dev vfs log mount check options account = /dev/bkuplv05 = jfs = /dev/bkuploglv05 = true = false = rw = false
/backup/gm_oraarch: dev vfs log mount check options account isvp18> isvp18> mount /backup/gm_oradata Replaying log for /dev/bkuplv04. isvp18> mount /backup/gm_oralog Replaying log for /dev/bkuplv05. isvp18> mount /backup/gm_oraarch Replaying log for /dev/bkuplv06. = /dev/bkuplv06 = jfs = /dev/bkuploglv06 = true = false = rw = false
You then mount the three file systems and create sym links to them as shown below using the ln command. isvp18> ln -s /backup/gm_oradata/testgm /gm_oradata isvp18> ln -s /backup/gm_oralog/testgm /gm_oralog isvp18> ln -s /backup/gm_oraarch/testgm /gm_oraarch
60
Log in as user oracle and start the Oracle database using the file systems created on the FlashCopy clones. isvp18> su - oracle $ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 3 00:44:29 2011
Total System Global Area 1.0289E+10 bytes Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. SQL> The database is now accessible for test, backup, or other development purposes. It is important to note that while the clone is accessing the remote copy target volumes, it will not be updated with the latest updates happening to the Metro or Global mirror target volumes. 2215712 bytes 5268046048 bytes 4999610368 bytes 18743296 bytes
6.1.3 Full database failover to the target site

In this scenario, the source site becomes inaccessible and the target site takes over as the production site. Complete the steps specified in section 6.1.1 "Setup for full database remote copy", and make sure that the data copy available at the target site. Metro Mirror:
61
At the target site, you observe that the Metro Mirror Copy Consistency Group is in Consistent Synchronized state as shown in Figure 81.
Figure 81. Metro Mirror Consistency Group and the relationships in Consistent Synchronized state
Stop the Metro Mirror Consistency Group on the target by clicking on the Action drop down menu and select Stop as shown in Figure 82.
Figure 82. Stopping the Metro Mirror Consistency Group
Provide read-write access to the Auxiliary Metro Mirror volumes as shown in Figure 83.
62
Figure 83. Allowing read-write access to the secondary Metro Mirror volumes
Figure 84 shows the actual background command that gets executed to stop the Metro Mirror Consistency Group.
Figure 84. Command to stop the Consistency Group
Delete the Metro Mirror relationships by highlighting all the Metro Mirror relationships, and then right click the mouse button and move the mouse to Delete Relationships. You will see the pop-up window as shown in Figure 85 which will verify the number of relationships that are to be deleted.
63
Figure 85. Deleting the Metro Mirror relationships
The Metro Mirror Auxiliary volumes are now accessible by the target system with read/write access. On the target host isvp18 mount the /oradata, /oralog, and /oraarch file systems on the volume. You can now proceed to Start and open the database on the target system for client access. isvp18> mount /oradata Replaying log for /dev/lv01. isvp18> mount /oralog Replaying log for /dev/fslv01. isvp18> mount /oraarch Replaying log for /dev/fslv02. isvp18> su - oracle $ echo $ORACLE_SID test_mm $ sqlplus / as sysdba
64
Total System Global Area 1.0289E+10 bytes Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. 2215712 bytes 5268046048 bytes 4999610368 bytes 18743296 bytes
Global Mirror:
At the target site, you observe that the Global Mirror Copy Consistency Group is in Consistent Synchronized state as shown in Figure 86.
Figure 86. Global Mirror Consistency Group and the relationships in Consistent Synchronized state
Stop the Global Mirror Consistency Group on the target by clicking on the Action drop down menu and select Stop as shown in the Figure , and provide read-write access to the Auxiliary volumes as shown in Figure 87.
65
Figure 87. Stop Global Mirror Consistency Group
Provide read-write access to the Auxiliary Metro Mirror volumes as shown in Figure 88.
Figure 88. Allowing read-write access to the secondary Global Mirror volumes
Figure 89 shows the actual background command that gets executed to stop the Global Mirror Consistency Group.
66
Figure 89. Command to stop the Consistency Group
Delete the Global Mirror relationships by highlighting all the Global Mirror relationships, and then right click the mouse button and move the mouse to Delete Relationships. You will see the pop-up window shown in the Figure 90 will verify the number of relationships that are to be deleted.
Figure 90. Deleting the Metro Mirror relationships
67
The Global Mirror Auxiliary volumes are now accessible by the target system with read/write access. On the target host isvp18 mount the /oradata, /oralog, and /oraarch file systems on the volume. You can now proceed to Start and open the database on the target system for client access. isvp18> mount /gm_oradata Replaying log for /dev/lv03. isvp18> mount /gm_oralog Replaying log for /dev/lv04. isvp18> mount /gm_oraarch Replaying log for /dev/lv05. isvp18> su - oracle $ export ORACLE_SID=testgm $ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Feb 24 11:19:21 2011
Total System Global Area 1.0289E+10 bytes Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. SQL> 2215712 bytes 5268046048 bytes 4999610368 bytes 18743296 bytes
6.1.4 Full database failover with role reversal
68
In this scenario, both the source and target volumes are accessible but the roles of the source and target for the consistency group are reversed. The target site becomes the active production database site and the initial production site becomes the new target site. The scenario could be performed in a situation where the roles of the source database and target database need to be reversed for a quick failover. Complete the steps specified in section 6.1.1 "Setup for full database remote copy" before starting this scenario, and make sure the replica is available at the target site. Metro Mirror:
Figure 91 below shows the target site IBM Storwize V7000 GUI shows that Consistency Group with the Metro Mirror copy relationships is in Consistency Synchronized state. The Master volumes are accessible for read and write I/O operations, while the Auxiliary volumes are accessible for read-only I/O operations. To reverse the roles of the Master volumes and Auxiliary volumes click on the Actions drop down menu of the v7k_mm_OrclDR Consistency Group, and then select Switch.
Figure 91. Switching the Metro Mirror copy relationships
As shown in Figure 92, a warning message will be displayed saying that the write access will be
disabled to the Metro Mirror Master volumes, and the write access with enabled for the Metro Mirror Auxiliary volumes. Click OK and proceed with the switch.
69
Figure 92. Warning message for switching Metro Mirror copy relationship
Figure 93 shows the actual background command that is executed to perform the switch between the Metro Mirror Master volumes and the Auxiliary volumes.
Figure 93. Command to switch the Metro Mirror copy consistency group
70
Figure 94 shows that the switch has completed. This is indicated by a switch icon next to the Consistent Synchronized State message of the Metro Mirror relationships.
Figure 94. Switch tag on the Metro Mirror relationships
Mount the three file systems that have the control files, data files, and log files of your Oracle database isvp18> mount /oradata Replaying log for /dev/lv01. isvp18> mount /oralog Replaying log for /dev/fslv01. isvp18> mount /oraarch Replaying log for /dev/fslv02.
Log in as user oracle and start the Oracle database using the file systems created on the Auxiliary volumes isvp18> su - oracle $ sqlplus / as sysdba
71
Global Mirror:
Figure 95 below of the target site IBM Storwize V7000 GUI shows that Consistency Group with the Global Mirror copy relationships is in Consistency Synchronized state. The Master volumes are accessible for read and write I/O operations, while the Auxiliary volumes are accessible for read-only I/O operations. To reverse the roles of the Master volumes and Auxiliary volumes click on the Actions drop down menu of the v7k_gm_OrclDR Consistency Group, and then select Switch.
Figure 95. Switching the Global Mirror copy relationships
72
As shown in Figure 96, a warning message will be displayed saying that the write access will be disabled to the Metro Global Master volumes, and the write access with enabled for the Global Mirror Auxiliary volumes. Click OK and proceed with the switch.
Figure 96. Warning message for switching Global Mirror copy relationship
Figure 97 shows the actual background command that is executed to perform the switch between the Global Mirror Master volumes and the Auxiliary volumes.
Figure 97. Command to switch the Global Mirror copy consistency group
73
Figure 98 shows that the switch has completed. This is indicated by a switch icon next to the Consistent Synchronized State message of the Global Mirror relationships.
Figure 98. Switch tag on the Global Mirror relationships
Mount the three file systems that have the control files, data files, and log files of your Oracle database
isvp18> mount /gm_oradata isvp18> mount /gm_oralog isvp18> mount /gm_oraarch
Log in as user oracle and start the Oracle database using the file systems created on the Auxiliary volumes.
$ export ORACLE_SID=testgm $ sqlplus / as sysdba
SQL> SQL> startup;
74
ORACLE instance started.
Total System Global Area 1.0289E+10 bytes Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. 2215712 bytes 5268046048 bytes 4999610368 bytes 18743296 bytes
6.1.5 Failback to the source site

In this scenario the source site which was down becomes available again. After a failover the initial target site had become the new source site for the database. Move the source back to the original site, and restore the standby duty to the target site. Metro Mirror:
On the target site shutdown the Oracle database. This needs to be done as the volumes on the target site have only ready access after the switch. Click on the Action drop down menu and select Switch as shown in Figure 99. isvp18> su - oracle $ echo $ORACLE_SID test_mm $ sqlplus / as sysdba
SQL> shutdown abort; ORACLE instance shut down. SQL> quit
75
isvp18> umount /oradata isvp18> umount /oralog isvp18> umount /oraarch
Figure 99.: Switch the Metro Mirror Consistency Group
A warning message will appear as shown below in Figure 100. Click OK to continue with the switch.
Figure 101 shows the background command that is executed to perform the switch of roles between the auxiliary volumes and the master volumes.
76
Figure 101. Command to perform the switch
As shown in Figure 102, after the switch has been completed, you see that the Switch icon is no more there next to the Consistent Synchronized State message of the volumes.
Figure 102. Metro Mirror Consistency Group after the failback
Mount the /oradata, /oralog , and /oraarch file systems on the on the production system and use it to mount and open the Oracle database for client access. isvp17> mount /oradata isvp17> mount /oralog
77
isvp17> mount /oraarch isvp17> su - oracle $ sqlplus / as sysdba
Global Mirror:
On the target site shutdown the Oracle database. This needs to be done as the volumes on the target site have only ready access after the switch. Click on the Action drop down menu and select Switch as shown in Figure 103. isvp18> su - oracle $ echo $ORACLE_SID test_gm $ sqlplus / as sysdba
78
SQL> shutdown abort; ORACLE instance shut down. SQL> quit
isvp18> umount /oradata isvp18> umount /oralog isvp18> umount /oraarch
Figure 103. Switch the Global Mirror Consistency Group
A warning message will appear as shown below in Figure 104. Click OK to continue with the switch.
79
Figure 105 below shows the background command that is executed to perform the switch of roles between the auxiliary volumes and the master volumes.
Figure 105. Command to perform the switch
As shown in Figure 106, after the switch has been completed, you see that the Switch icon is no more there next to the Consistent Synchronized State message of the volumes.
80
Figure 106. Global Mirror Consistency Group after the failback
You can now mount the /gm_oradata, /gm_oralog , and /gm_oraarch file systems on the on the production system and use it to mount and open the Oracle database for client access. isvp17> mount /gm_oradata isvp17> mount /gm_oralog isvp17> mount /gm_oraarch isvp17> su - oracle $ export ORACLE_SID=testgm $ sqlplus / as sysdba
Total System Global Area 1.0289E+10 bytes Fixed Size Variable Size 2215712 bytes 5268046048 bytes
81
Database Buffers Redo Buffers Database mounted. Database opened. SQL>
4999610368 bytes 18743296 bytes
82
7 Summary
This paper explains a method for using IBM Storwize V7000 Remote Copy Services to provide a disaster recovery solution for an Oracle 11gR2 database when the database uses file systems for its control, data, and redo log files. Also included is a detailed description of the infrastructure that is required to implement this architecture, along with detailed steps for implementing backup and recovery by using both FlashCopy, for local backups and cloning, and Metro Mirror and Global Mirror, for remote cloning or disaster recovery.
83
Resources
The following websites provide useful references to supplement the information contained in this paper: Deploying Oracle 11g RAC Release 2 with IBM Storwize V7000 on Red Hat Enterprise Linux ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101772 Practice guide: Backup and restore of native Oracle Database solutions using IBM Tivoli FlashCopy Manager Version 1.0 ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101771 IBM SAN Volume Controller Performance Configuration Guidelines for Implementing Oracle Databases with Automatic Storage Management (ASM) ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101481 IBM Business Partner support and resources ibm.com/partnerworld/systems Storwize V7000 on PartnerWorld ibm.com/partnerworld/wps/pub/overview/HW26Z IBM Publications Center www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi?CTY=US IBM Redbooks ibm.com/redbooks IBM developerWorks ibm.com/developerWorks
84
Trademarks and special notices

Copyright IBM Corporation 2011. All rights Reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC. DIA Data Integrity Assurance, Storewiz, Storwize, and the Storwize logo are trademarks or registered trademarks of Storwize, Inc., an IBM Company. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction.
85
Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with you customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk.
86

MetroMirror For Oracle With V7000

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

MetroMirror For Oracle With V7000

Caricato da

Copyright:

Formati disponibili

Disaster Recovery solution for Oracle 11gR2 database using Metro Mirror and Global Mirror features of IBM

IBM Systems and Technology Group ISV Enablement March 2011

Copyright IBM Corporation, 2011. All Rights Reserved

3 IBM Storwize V7000 technology stack ................................................................................. 1

4 IBM Storwize V7000 and Remote Copy................................................................................ 5

5 Configuration and setup ..................................................................................................... 10

6 Oracle database replication................................................................................................ 38

2.3 Intended audience

3 IBM Storwize V7000 technology stack

3.1 IBM Storwize V7000 overview

3.1.1 IBM Storwize V7000 functionalities

Figure 1: IBM Storwize V7000 virtualizing external and internal storage

3.1.2 IBM Storwize V7000 terminology

Host mapping Internal storage Managed disk (MDisk)

Storage pool Thin provisioning or thin provisioned

Table 1: IBM Storwize V7000 terminology

4 IBM Storwize V7000 and Remote Copy

4.1 Remote Copy overview

4.2 Remote Copy concepts

4.3 Remote Copy mechanism to copy data

4.3.1 Metro Mirror

4.3.2 Global Mirror

4.4 Remote Copy and consistency group states

Table 2: IBM Storwize V7000 states

5 Configuration and setup

5.1 Lab test environment

Node name Processor RAM

isvp18 PowerPC_POWER7 24 576 MB

Standby AIX 7.1 SP2 11gR2 test_mm

Global Mirror Filesystems

Data files: /gm_oradata Control files: /gm_oraarch Redo logs: /gm_oralog

Table 4. Oracle database information

Global Mirror Volumes

Table 5. IBM Storwize V7000 information

5.2 Lab configuration

IBM Storwize V7000

IBM Storwize V7000

Fibre Channel switch

Power Systems server

Power Systems server

Figure 4. Lab configuration for Remote Copy scenarios

5.3 IBM Storwize V7000 setup for Remote Copy

Figure 6. Remote Copy panel of the IBM Storwize V7000

Figure 7. New Metro Mirror Relationship creation pop up window

Figure 8. Select Metro Mirror partner

Figure 11. Metro Mirror volumes are not synchronized

Figure 12. No initial copy between Metro Mirror volumes

Figure 13. Metro Mirror relationship creation completes

Figure 15. Auxiliary volumes on a remote IBM Storwize V7000

Figure 16. Creating an empty consistency group

Figure 17. Empty consistency group created

Figure 18. Adding Metro Mirror relationships to consistency group

Figure 20. Command for changing remote copy relationships

Figure 22. Starting the Consistency Group

Figure 23. Command to start remote copy consistency group

Figure 27. New Global Mirror relationship creation pop up window

Figure 29. Volumes paired for a Global Mirror relationship

Figure 30. Global Mirror volumes are not synchronized

Figure 31. No initial copy between the Global Mirror volumes

Figure 32. Command to create the Global Mirror relationships

Figure 33. The Global Mirror relationships in Inconsistent Stopped state

Figure 34. Consistency Group for the Global Mirror relationships

Figure 36. No initial copy between the Global Mirror volumes