Green Plum 300-012-943

Configuring EMC Greenplum Data Computing Appliance Using SAN Mirror and EMC Symmetrix VMAX for Disaster
Recovery
Configuration Guide
P/N 300-012-943 REV A01
EMC Corporation Corporate Headquarters: Hopkinton, MA 01748-9103

1-508-435-1000 www.EMC.com
Copyright 2011 EMC Corporation. All rights reserved. Published August, 2011 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners.
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Contents
Preface.............................................................................................................................. 5 Chapter 1 Introducing the SAN Mirror Solution

Introduction ....................................................................................... Business need ............................................................................. EMC Greenplum Data Computing Appliance (DCA) ......... Terminology ............................................................................... Using a SAN Mirror solution .......................................................... Overview..................................................................................... How SAN Mirror increases the capacity of the DCA........... How EMC Greenplum DCA mirrors data............................. Mirroring data to the SAN with Greenplum SAN Mirror .. Components of a SAN Mirror configuration................................ EMC Greenplum DCA.............................................................. EMC Symmetrix VMAX storage system ................................ Considerations and best practices .................................................. Performance................................................................................ Data integrity.............................................................................. 10 10 11 12 13 13 15 15 16 18 18 21 25 25 26
Chapter 2
Setup and Operation of SAN with VMAX

Introduction to the SAN Mirror test configuration...................... Detailed list of tested components ................................................. EMC Greenplum DCAs ............................................................ VMAX.......................................................................................... Prerequisites....................................................................................... Configuring for SAN Mirror ........................................................... Configuring a new system ....................................................... Preparing the VMAX for SAN Mirror attachment ............... 30 31 31 32 34 35 35 36
Configuring the local DCA for the Symmetrix ..................... Configuring the remote DCA .................................................. Operating a SAN Mirror configuration.................................. Summary............................................................................................
38 42 42 47
Appendix A
Discovering DCA Internal Server WWNs

Discovering DCA internal server WWNs .................................... 50
Appendix B
Verifying DCA Volume Sizes

Verifying DCA volume sizes .......................................................... 52
Appendix C
Creating VMAX Volumes for DCA Mirror Database

Creating VMAX volumes for DCA mirror database .................. 54 Assigning volumes to the DCA segment servers ....................... 56
Appendix D
Creating the SRDF Configuration

Creating the SRDF configuration .................................................. 58
Appendix E
Configuring DCA Internal Switch for SAN Mirror

Configuring DCA internal switch for SAN Mirror ..................... 60
Appendix F
Automating DCA Internal SAN Configuration

Script to automate DCA internal SAN configuration ................ 68
Appendix G
Creating SNAPs on Remote Symmetrix for Disaster Recovery

Creating SNAPs on remote Symmetrix for disaster recovery ... 76 Creating a single SNAP of R2 standard devices ......................... 77 Creating multiple SNAPS of R2 standard devices ...................... 91
Appendix H
Mounting and Checking a Symmetrix SNAP of the Greenplum Database

Mounting and checking Symmetrix SNAP of Greenplum database ........................................................................................... 104
Appendix I
Recovering a DCA Using a SNAP Session and SRDF

Recovering a DCA using a SNAP session and SRDF ............... 126
Appendix J
Starting EMC Greenplum Database from Mirror Segments

Starting EMC Greenplum database from mirror segments ..... 152
Preface
This guide documents how to configure an EMC Greenplum Data Computing Appliance (DCA) to attach to an EMC Symmetrix VMAX system, where the primary data copy resides on the DCA and the VMAX is used as a mirror. The facilities of the VMAX storage array can then be used to provide remote replicas for disaster recovery and create point-in-time copies to allow quick recovery from equipment or data center outages. This document specifies the steps to move the data manually. The appendices provide detailed, step-by-step instructions and output from commands, taken from EMC's engineering lab tests. Forthcoming tools from Greenplum will help customers automate the movement of data to external storage. As part of an effort to improve and enhance the performance and capabilities of its product line, EMC from time to time releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all revisions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes. If a product does not function properly or does not function as described in this document, contact your EMC representative. Audience This guide is intended for EMC Field personnel who are implementing or considering a SAN Mirror solution for the Greenplum DCA. It will also be useful for customers and partners to gain understanding of the benefits and requirements of SAN Mirror implementations.
Preface
Related documentation
More information is available about each of the components discussed in this document. A partial list is shown next. Documents are available at http://powerlink.emc.com. Related documents include:

White Paper: EMC Greenplum Data Computing Appliance Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery White Paper: EMC Greenplum Data Computing Appliance: High Performance for Data Warehousing and Business intelligence - An Architectural Overview White Paper: EMC Greenplum Database 4.0 - Critical Mass Innovation Greenplum Database 4.1 Administrator Guide (describes the database operation in great detail) Greenplum Database 4.1 Installation Guide Greenplum DCA and DIA 1.0.3.x Getting Started Guide Symmetrix VMAX Series Product Guide Symmetrix Remote Data Facility (SRDF) Product Guide Symmetrix TimeFinder Product Guide Solutions Enabler 7.3 Documentation Set PowerPath Family Product Guide PowerPath for Linux Installation and Administration Guide 5.5 A03
Conventions used in this guide
EMC uses the following conventions for notes, cautions, and warnings.
Note: A note presents information that is important, but not hazard-related.
IMPORTANT An important notice contains information essential to operation of the software.
Preface
Typographical conventions EMC uses the following type style conventions in this document:
Normal Used in running (nonprocedural) text for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) Names of resources, attributes, pools, Boolean expressions, buttons, DQL statements, keywords, clauses, environment variables, filenames, functions, utilities URLs, pathnames, filenames, directory names, computer names, links, groups, service keys, file systems, notifications Used in running (nonprocedural) text for: Names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system call, man pages Used in procedures for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) What user specifically selects, clicks, presses, or types
Bold
Italic
Used in all text (including procedures) for: Full titles of publications referenced in text Emphasis (for example a new term) Variables Used for: System output, such as an error message or script URLs, complete paths, filenames, prompts, and syntax when shown outside of running text Used for: Specific user input (such as commands) Used in procedures for: Variables on command line User input variables Angle brackets enclose parameter or variable values supplied by the user Square brackets enclose optional values Vertical bar indicates alternate selections - the bar means or Braces indicate content that you must specify (that is, x or y or z) Ellipses indicate nonessential information omitted from the example
Courier
Courier bold
Courier italic
<> [] | {} ...
Preface
Where to get help
EMC support, product, and licensing information can be obtained as follows. Product information For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at:
http://Powerlink.EMC.com
Technical support For technical support, go to EMC Customer Service on Powerlink. To open a service request through Powerlink, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or to answer any questions about your account. Your comments Your suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Please send your opinion of this guide to:
techpub_comments@EMC.com
Invisible Body Tag
1
Introducing the SAN Mirror Solution
This chapter provides the following information.

Introduction ........................................................................................ Using a SAN Mirror solution ........................................................... Components of a SAN Mirror configuration ................................. Considerations and best practices ...................................................
10 13 18 25
Introduction
This section provides a brief introduction to the following:

Business need on page 10 EMC Greenplum Data Computing Appliance (DCA) on page 11 Terminology on page 12
Business need
Data warehouses are a critical tool for making business decisions. As data warehouse and business intelligence systems (DW/BI) continue to grow, they have faced some difficult challenges:
DW/BI infrastructures are growing exponentially over time and often require many terabytes or even petabytes of high-performance, protected storage. DW/BI systems must be continually available. High performance update, backup and disaster recovery are necessary to prevent and recover from outages.
Currently, data warehouse environments are demanding a more comprehensive strategy for data protection, security, and high availability than ever before, based on the business impact of data unavailability. Data recovery options must align with application and business requirements to yield the highest availability. To integrate the EMC Greenplum Data Computing Appliance (DCA) into larger data centers, some customers require compatibility with advanced storage and software infrastructures, such as the EMC Symmetrix VMAX in order to achieve the highest levels of fault tolerance and to provide industry-leading disaster recovery. To facilitate this, EMC engineered a solution that integrates the DCA with the VMAX, where the VMAX provides local and remote replicas of the data for disaster recovery and point-in-time snapshots. This allows customers to recover data warehouse/business intelligence (DW/BI) functionality quickly in the face of hardware or software failure, or even total site loss.
10
EMC Greenplum Data Computing Appliance (DCA)

The EMC Greenplum Data Computing Appliance (DCA) is an industry-leading, purpose-built data warehouse appliance that helps organizations make educated strategic and operational decisions. The DCA addresses essential business requirements, delivering predictable performance, scalability and functionality. This eliminates the complexity and unpredictability of highly customized, in-house solutions. The DCA is a self-contained system built of industry-standard, commodity components. It includes both processor, storage, and software resources needed to implement large-scale DW/BI infrastructures. Data in the DCA is fully protected and the system is able to ride through multiple component failures non-disruptively. For more information, refer to How EMC Greenplum DCA mirrors data on page 15. By connecting the DCA to industry-leading SAN storage, such as the VMAX system, customers can get all the performance and functionality of the DCA, combined with robust remote replication and point-in-time copies provided by external storage. This capability is called SAN Mirror, discussed further in Using a SAN Mirror solution on page 13.
Introduction
11
Terminology
The following list key terms used in this document. Term Definition
Business Intelligence The effective use of information assets to (BI) improve the profitability, productivity, or efficiency of a business. IT professionals use this term to refer to the business applications and tools that enable such information usage. The source of information is frequently the data warehouse. CNA Data Warehouse (DW) Converged Network Adapter The process of organizing and managing information assets of an enterprise. IT professionals often refer to the physical stored data content in some databases managed by database management software as the data warehouse. They refer to the applications that manipulate the data stored in such databases as DW applications. The DCA is a purpose-built, highly scalable, parallel DW appliance that architecturally integrates database, compute, storage, and network into an enterprise-class, easy-to-implement system. A type of distributed computing architecture where tens to hundreds of processors team up to work concurrently to solve large computational problems
Data Computing Appliance (DCA)
Massively Parallel Processing (MPP)
12
Using a SAN Mirror solution

This section contains the following information:

Overview on page 13 How SAN Mirror increases the capacity of the DCA on page 15 How EMC Greenplum DCA mirrors data on page 15 Mirroring data to the SAN with Greenplum SAN Mirror on page 16
Overview
The EMC Greenplum DCA maintains two copies of customer data and normally handles all data replication and protection tasks internally to the appliance. This generally achieves the highest level of performance for DW/BI tasks. In a SAN Mirror solution, the second copy of the data is moved to SAN-based storage. The DCA retains the primary copy of the data in order to maximize query performance. The SAN Mirror copy is updated with writes, but it is not read unless a primary database segment becomes inaccessible. By keeping the mirrored copy on the SAN, customers can use storage facilities such as EMC TimeFinder and SRDF to create remote copies or point-in-time images for backup and disaster recovery. Figure 1 and Figure 2 on page 14 compare a stand-alone DCA configuration to different SAN Mirror configurations.
13
Figure 1
Stand-alone DCA versus SAN Mirror configuration
Figure 2
SAN Mirror configuration with remote replication
14
How SAN Mirror increases the capacity of the DCA

A SAN Mirror configuration significantly increases the DCA's internal capacity. Whereas a standard DCA configuration stores both primary and mirrored data internal to the appliance, a SAN Mirror configuration moves the mirrored data to the SAN, which frees the other half of the machine's internal capacity to be used as additional primary storage.
How EMC Greenplum DCA mirrors data

The EMC Greenplum DCA is a fully redundant system that maintains two copies of data to ensure seamless operation if one or more components in the system fail. The system is able to process large amounts of data by distributing the load across all of the servers in the DCA. An EMC Greenplum Database is actually a loosely-coupled array of individual, highly-customized PostreSQL databases, working together to present a single database image. The master is the entry point to the Greenplum Database system. The master host contains all the metadata required to distribute transactions across the system, but it does not hold any user data. It coordinates the work with the other database instances in the system, the segments, which handle data processing and storage. When a Greenplum Database system is deployed, there is the option to mirror the segments. Mirroring allows the database to remain operational if a segment instance or segment host goes down, as long as a copy of the data is accessible in another running segment. Primary segment instances replicate to mirror instances at the sub-file level. In order to guarantee that both primary and mirror segments present the same crash-consistent database image, primary segments will wait for mirror segments to synchronize data during transaction commits and periodic database checkpoints. On segment failure, the remaining copy goes into change tracking mode and saves a list of changes made to its data. Once the underlying segment failure is resolved, an online recovery process will refresh the recovered segment either by copying the changed data, or by executing a full copy to the failed segment.
15
In the EMC Greenplum DCA, Six primary segment instances and six mirror segment instances run on each segment server. The segments use internal DCA storage. The primary and mirror segments run on separate servers in the DCA. A high-level picture of the mirrored configuration is shown in Figure 3.
Figure 3
DCA mirroring
Mirroring data to the SAN with Greenplum SAN Mirror

In a SAN Mirror configuration, storage for both master instances and all the mirrored segments is moved from internal storage to external SAN storage. A SAN Mirror configuration allows the Greenplum database to benefit from advanced storage capabilities such as SRDF and TimeFinder. While the DCA will get highest performance from internal storage, the additional functionality of the SAN Mirror configuration makes it appealing for certain use cases.
16
A conceptual picture of a SAN Mirror configuration is shown in Figure 4.
Figure 4
DCA SAN Mirror configuration
17
Components of a SAN Mirror configuration

This section briefly describes each component of the SAN Mirror solution. It is assumed that the reader is familiar with all or most of these products. Further details can be found in documents listed in Related documentation on page 6. The components used in this solution are listed below and further described in this section:
EMC Greenplum DCA on page 18 EMC PowerPath is loaded on all segment servers. PowerPath is additional, separately chargeable software. Trunking is enabled on internal EMC Connectrix MP-8000B switches to provide a large, load-balanced pipe between the DCA and the SAN. Trunking is an additional, separately chargeable software function.
EMC Symmetrix VMAX storage system on page 21 EMC VMAX system, with four storage engines VP EMC Virtual Provisioning TimeFinder/SNAP for point-in-time copies SRDF for remote replication Solutions Enabler Symmetrix Management Console (SMC)
EMC Greenplum DCA

EMC Greenplum Data Computing Appliance (DCA) is a self-contained data warehouse solution that integrates all of the database software, servers and switches necessary to perform big data analytics. The EMC Greenplum DCA is a turn-key data warehouse solution that provides extremely high query and loading performance for analyzing large data sets. The EMC Greenplum DCA integrates the Greenplum database with compute, storage and network and is delivered racked and ready for immediate data loading and query execution.
18
The EMC Greenplum DCA runs the Greenplum Database relational database management system (RDBMS) software. The EMC Greenplum Database enables large data queries and analysis. The DCA is offered in quarter-rack, half-rack, full-rack, and multiple-rack appliance configurations to achieve maximum flexibility and scalability for organizations faced with terabyte to petabyte scale data analyses. A full rack DCA configuration is shown in Figure 5.
Figure 5
Greenplum DCA components
PowerPath
PowerPath is host-based multipath software that provides high availability and load balancing across multiple Host Bus Adapter/Converged Network Adapter (HBA/CNA) ports. In a PowerPath configuration, each LUN is assigned to multiple HBA ports on a single server. PowerPath uses sophisticated algorithms to deliver I/O optimally down each path, to ensure the highest overall system performance. If a path fails, PowerPath continues to load balance down the remaining paths to each device, enabling the server to continue operation. PowerPath automatically re-enables failed paths when the failures are corrected.
19
PowerPath load balancing is an additional, separately-chargeable software product. Additional capabilities for data migration and data encryption can also be purchased for PowerPath. While Linux Multi-Path I/O (MPIO) also delivers primitive load balancing and path failure protection, it is more difficult to manage than PowerPath. MPIO was not explored as an option for the SAN Mirror solution. Trunking software The EMC Connectrix trunking feature optimizes the use of bandwidth by allowing a group of inter-switch links (ISLs) to merge into a single logical link. Trunking is automatically implemented for any eligible ISLs after you install the Connectrix ISL Trunking license. The license must be installed on each switch that participates in trunking. The trunking license should be installed on both Connectrix MP-8000B switches within the DCA, and also on the SAN switches to which the DCA is connected. In a fabric with numerous switches, you can increase the bandwidth between switches by enabling multiple physical ports to appear as a single port. Enabling multiple physical ports form a trunking group where the traffic is distributed dynamically and in order at the frame level, thus achieving greater performance with fewer inter-switch links. Trunking groups are based on the user port number with contiguous eight ports as one group, such as 0-7, 8-15, and 16-23. Figure 6 shows how trunking can result in more throughput by distributing data over four ISLs with no congestion. In a fabric that does not have trunking capability, some paths would be congested and other paths under-utilized.
Figure 6
Connectrix fabric trunking avoids SAN bottlenecks
20
EMC Symmetrix VMAX storage system

The EMC Symmetrix VMAX system is the industry's premier high-end storage array. It enables massive consolidation and incremental scalability, providing all the benefits of tiered storage in a single platform, as well as the flexibility to address rapidly changing business requirements. For the most extreme and demanding storage environments, Symmetrix VMAX is a powerful solution that is remarkably easy to manage. Symmetrix VMAX systems are ideal for the most extreme application demands, where large-scale consolidation is required by the most information-intensive enterprises. Virtual Provisioning EMC Virtual Provisioning greatly simplifies data layout. Virtual Provisioning lets users make large volumes available to attached hosts without reserving large, unused capacities. Capacity is allocated as it is actually written. Data is transparently striped across all the resources of the Virtual Pool, which ensures the best performance for all applications. Virtual Provisioning automatically reclaims "zeroed" capacity and returns it to the free capacity pool. This makes optimal use of storage resources by allocating only the capacity that is actually in use. As the system grows, it can be upgraded non-disruptively and data is transparently redistributed across all storage resources to ensure optimal performance of the system. TimeFinder/Snap EMC TimeFinder and Symmetrix Remote Data Facility (SRDF) are the most powerful suites of local and remote storage replication solutions available in the industry, enabling business continuance volumes for parallel processing activities like backup, testing and development and local restore, as well as remotely replicated copies to guard against primary site disasters and outages. In fact, the TimeFinder and SRDF families are the most widely deployed set of local and remote replication solutions in the industry and are installed in tens of thousands of the most demanding environments worldwide. SRDF on page 22 provides more details on SRDF and which mode was used for this solution. The TimeFinder family is a set of Symmetrix local replication solutions designed to non-disruptively create point-in-time copies of critical data. You can configure backup sessions, initiate copies, and terminate TimeFinder operations using EMC Solutions Enabler software, described below.
21
TimeFinder local replication solutions include TimeFinder/Clone and TimeFinder/Snap. TimeFinder/Clone creates full-device and extent-level point-in-time copies while TimeFinder/Snap creates pointer-based logical copies that consume less storage space on physical drives. TimeFinder/Snap was used for the SAN Mirror solution described in this document. TimeFinder/Snap gives maximum flexibility with the lowest capacity requirement for point-in-time copies. In an SRDF/A configuration, it is essential to enable Write Pacing on the VMAX before taking Snaps on the R2 VMAX. Write Pacing became available with EMC Enginuity v5875 release; it protects SRDF/A sessions by slowing down host writes if the host write bandwidth is greater than the R2 VMAX can support. Since this could impact the performance of ongoing data warehouse loads, it is strongly advised to engage a Symmetrix Performance Guru when planning the SAN Mirror configuration.
Note: Symmetrix Performance Gurus can be located on the SPEED website, accessible to EMC customers, partners, and field personnel, at http://speed.corp.emc.com , Tools > Guru List. If you do not have access to this list, contact your local EMC Sales Representative.
SRDF
The Symmetrix Remote Data Facility (SRDF) is a business continuance solution that maintains a mirror image of data at the device level in Symmetrix arrays located in physically separate sites. There are three modes of SRDF:

Synchronous Asynchronous Adaptive copy
Asynchronous mode (SRDF/A) was used in this solution, as it provides a high level of consistency with minimal impact to system performance. The local and remote systems were immediately adjacent to each-other, so the effective distance was 0 km. For longer distances, it is critical to correctly size the links between data centers to support the expected write workload. SRDF/A creates a dependent-write consistent, point-in-time image on the target (R2) device, which is a short period of time behind the source (R1) device. Managed in sessions, SRDF/A transfers data in
22
cycles or delta sets to ensure that data at the R2 site is dependent-write consistent. The Symmetrix system acknowledges all writes to the R1 devices in exactly the same way as other non-R1 local devices. Host writes accumulate on the R1 side until the cycle time is reached, and are then transferred to the R2 Symmetrix in one delta set. The writes are then destaged to the R2 devices to create a permanent, consistent image of the R1 data. Because writes are transferred in cycles, writes can be coalesced, and duplicate writes to the same tracks require only a single transfer across the links. This ensures optimal link bandwidth utilization. EMC Enginuity Consistency Assist (ECA) ensures that snaps and remote replicas are write-order consistent across multiple devices and even across multiple Symmetrix VMAX systems. This is a required capability for SAN Mirror, to ensure recoverability of database on the remote Symmetrix system. Solutions Enabler The EMC Solutions Enabler kit contains all the base management software that provides a host with SYMCLI commands and APIs to configure the Symmetrix and to control Symmetrix operations for SRDF and TimeFinder. SYMCLI resides on the host system to monitor and control operations on Symmetrix storage arrays. SYMCLI commands are invoked from the host operating system via command line or scripts. SYMCLI commands invoke low-level channel communications to specialized gatekeeper devices on the Symmetrix. SYMCLI is required to control TimeFinder and SRDF operations from the DCA. Because of this, EMC Solutions Enabler should be loaded on the database master servers in the DCA. Symmetrix Management Console (SMC) The EMC Symmetrix Management Console (SMC) is an intuitive, browser-based user interface that configures and manages Symmetrix system arrays. SMC presents the functionality of the Solutions Enabler SYMCLI (command line interface) in a browser interface. Use Symmetrix Management Console to:
Manage Symmetrix Access Controls, user accounts, and permission roles Discover Symmetrix arrays
23
Perform configuration operations (create devices, map and mask devices, set Symmetrix system attributes, set device attributes, set port flags, create SAVE device pools) Manage devices (change device configuration, set device status, reserve devices, duplicate devices, create/dissolve meta devices) Perform and monitor replication operations (TimeFinder/Snap, TimeFinder/Clone, SRDF, Open Replicator) Configure and manage Fully Automated Storage Tiering (FAST) Monitor alerts Monitor an application's performance
Symmetrix Management Console is preinstalled on the Symmetrix VMAX system Service Processor and can also run on a data center host. Licenses are available individually, or as part of a Symmetrix VMAX Management Integration bundle.
24
Considerations and best practices

This section contains things to consider and best practices for the following:

Performance on page 25 Data integrity on page 26
Performance
Connecting the DCA to SAN-based storage delivers many manageability and functionality benefits, and it is important to configure the storage properly to ensure system performance. It is essential to configure enough engines, ports and disks to support the expected read and write workloads. A full DCA can ingest up to 13.4 TB/hr of new data (almost 4 GB/s), and can scan at up to 23.5 GB/s. A system configured for this maximum load may require one or more dedicated VMAX systems. However, very few customers will push a DCA to its bandwidth limits. In many cases, an existing Symmetrix might be used to support the needs of the data warehouse. As long as the Symmetrix configuration has enough spare bandwidth, the data warehouse workload can co-exist with other production data. The optimal Symmetrix configuration will be a function of the customer's actual expected workload. In SRDF configurations, it is critical to size the connection between VMAX systems properly. The number and speed of links will determine how much write data the system can sustain. Too few connections may significantly impact write performance. Please consult a SymmetrixPerformance Guru when planning a SAN Mirror configuration, to ensure that the VMAX will support the required bandwidth.
Note: Symmetrix Performance Gurus can be located on the SPEED website, accessible to EMC customers, partners, and field personnel, at http://speed.corp.emc.com , Tools > Guru List.. If you do not have access to this list, contact your local EMC Sales Representative.
The DCA contains EMC Connectrix MP-8000B 8000 switches to support internal communication between servers. In a SAN Mirror configuration, the Fibre ports on the switches are configured to attach
25
to a Fibre Channel SAN. Customers should purchase optional SAN Trunking software for the switches to provide the greatest possible throughput. This requires an additional software license for the Connectrix switches. PowerPath should be loaded on each internal server to provide the greatest possible bandwidth and resiliency for the SAN Mirror configuration. PowerPath is a chargeable feature. Standard Linux MPIO can also be used, but may be much more difficult to manage as PowerPath is aware of the Symmetrix devices in the VMAX. Configuring and managing MPIO for the DCA is outside of the scope of this document.
Data integrity
A SAN Mirror configuration moves the "mirror" database segment storage to the SAN. In normal operation, the mirror segments are only used for writes to ensure resiliency in case a primary segment becomes unavailable. Query activity uses the primary segments on internal DCA disk, since that provides the highest performance. For a particular segment of data, Greenplum database mirroring relies on the primary instance process to send writes over the IP interconnect to a mirror instance process on a different segment server host. If a segment server host becomes unavailable, primary instances on that host will be taken out of operation and their corresponding mirrors activated on other segment hosts. Additionally, mirror instances on that host will no longer be able to receive replication updates and write them to SAN storage.
26
Since the SAN copy of the database is taken from the mirror, the SAN copy of the database will no longer be consistent. This is illustrated in Figure 7.
Figure 7
Segment failure creates inconsistent remote image
To ensure there is always a restartable copy of the database if a failure occurs, TimeFinder/Snap is used to create "gold copy" images periodically on the remote VMAX. The gold copy will be slightly behind the SRDF image, but could be brought to currency by re-applying updates to the remote database. It is necessary to verify that there are no segment failures and that the system is operating normally when taking a snap, to ensure the gold copy is consistent. If a mirror segment is inoperable for any reason, no snap copy should be taken. This will ensure a recent, consistent gold copy on the remote VMAX. Procedures for making and verifying a gold copy are documented in Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery, and in Appendix H, Mounting and Checking a Symmetrix SNAP of the Greenplum Database.
27
28
Invisible Body Tag
2
This chapter provides the following information on the setup and operation of SAN Mirror with VMAX. This chapter details the test system that was used and gives configuration details for both the EMC/Greenplum Data Computing Appliance (DCA) and the EMC Symmetrix VMAX system.

Introduction to the SAN Mirror test configuration....................... Detailed list of tested components .................................................. Prerequisites........................................................................................ Configuring for SAN Mirror ............................................................ Summary .............................................................................................
30 31 34 35 47
29
Introduction to the SAN Mirror test configuration

For this test, a full rack configuration was configured into two separate logical "half-rack" DCAs, each with eight segment servers. The "standby master" was configured as the primary master of the second logical DCA. Each DCA was connected to the SAN, and its "SAN Mirror" was located on a different Symmetrix VMAX on the SAN. The primary VMAX held the mirror copy of the primary DCA's data. The VMAX was configured to use Virtual Provisioning, and 2.7 TB volumes were created to match the volumes on the primary DCA. Those Symmetrix volumes were replicated by SRDF/A to a second, "remote" Symmetrix, which held copies of the data for disaster recovery. A 7 TB database, similar to what is used for TPC-H benchmarks, was created for testing. Figure 8 shows the high-level layout of the tested solution. Customers will typically implement this solution using separate DCAs, in separate racks.
Figure 8
SAN Mirror test configuration architecture
30
Detailed list of tested components

The test configuration used for this document is detailed in the Table 1 through Table 5, as follows:

EMC Greenplum DCAs on page 31 VMAX on page 32
EMC Greenplum DCAs

Table 1 and Table 2 list the Greenplum DCA components and software.
Table 1
Greenplum DCA components Component Master server Segment servers Interconnection Switch Administration Switch Quantity 1 (primary, no standby configured) 8 (GP100, half-rack configuration) 2 (EMC Connectrix MP-8000B) 1 (Allied Telesis)
Table 2
Greenplum DCA software Software EMC Greenplum-db PowerPath Solutions Enabler Version 4.1.1.1 5.5 7.2.1.0 Comments EMC Greenplum database software Strongly advised for multipathing and load balancing Enables the DCA to issue TimeFinder and SRDF commands to the VMAX
31
VMAX
Table 3, Table 4, and Table 5 list the VMAX components and software.
Table 3
Primary VMAX hardware components Hardware 4-Engine VMAX Fibre Channel directors RF (Remote Fibre) directors 15k RPM Fibre Channel Disk 8 Gb/s Fibre 8 Gb/s connection for SRDF replication 600 GB capacity, 15k RPM, configured RAID 5 7+1 and used with Virtual Provisioning Specifications Quantity 1 24 8 192
Table 4
Remote VMAX hardware component Hardware 4-Engine VMAX Fibre Channel directors RF (Remote Fibre) directors 15k RPM Fibre Channel Disk 8 Gb/s Fibre 8 Gb/s connection for SRDF replication 600 GB capacity, 15k RPM, enough to accommodate snaps and remote replicas Configured RAID 5 7+1 and used with Virtual Provisioning Specifications Quantity 1 24 8 470
32
Table 5
VMAX software Software Enginuity 5875.198 Virtual Provisioning Specifications Base Enginuity Operating Environment EMC Virtual Provisioning allocates only the storage that was actually used, and stripes wide across all disks Not used for this paper Create asynchronous remote replicas (this test used zero-distance SRDF/A) Not used for this paper Create pointer-based gold copy of database volumes Graphical managing and monitoring interface for Symmetrix VMAX systems Quantity 2 2
SRDF/S SRDF/A TimeFinder/Clone TimeFinder/Snap Symmetrix Management Solution
2 2 2 2 2
33
Prerequisites
It is assumed that the following steps have been taken prior to configuring the SAN Mirror solution, and thus this solution will not detail the installation, setup and configuration of the components listed here. Further information can be located in the documents listed inRelated documentation on page 6.
The EMC/Greenplum DCA should be installed and configured in the data center. Specific information on the initial configuration of the EMC/Greenplum hardware and software is described in the Greenplum Database 4.1 Installation Guide. The installation guide discusses the installation process for the DCA servers, switches and hosts. This guide also discusses the basic configuration process for the Greenplum database. EMC PowerPath should already be installed on the DCA segment servers and database masters. PowerPath is not included with the DCA and should be installed by EMC Professional Services on each server in the DCA. EMC Solutions Enabler should already be installed on the database masters. Solutions Enabler is not included with the DCA and should be installed by EMC Professional Services on the master and the standby master. VMAX systems should be installed and configured in the local data center, and also in the remote data center if SRDF is used to replicate data between data centers. SRDF should be installed, configured and replicating between the VMAX systems. The data center SAN should be installed and operational, with the VMAX systems attached to it.
34
Configuring for SAN Mirror

This section describes the steps necessary to implement SAN Mirror on an EMC/Greenplum DCA. Scripts and sample output are provided in the appendices referenced within each subsection. The following information is provided:

Configuring a new system on page 35 Preparing the VMAX for SAN Mirror attachment on page 36 Configuring the local DCA for the Symmetrix on page 38 Configuring the remote DCA on page 42 Operating a SAN Mirror configuration on page 42
Configuring a new system

The steps needed to configure a new system are shown below. Each of them is discussed in detail after the high-level list. For the initial release of SAN Mirror, the steps are manual. Future releases of the DCA and of this document will provide tools to help automate this process. Steps required to configure a new DCA for SAN Mirror are listed below: 1. Prepare the VMAX for SAN Mirror attachment. a. Create Symmetrix volumes for DCA master and mirror databases. b. Make volumes visible to the DCA Master and Segment servers. c. Create SRDF mirrors of the Symmetrix volumes. d. Create TimeFinder/Snap or Clone volumes on R2 VMAX. 2. Prepare the Greenplum DCA for SAN Mirror. a. Load PowerPath on each server in the local and remote DCAs. b. Install EMC Solutions Enabler on the master and standby master servers. c. Configure internal Connectrix MP-8000B switches for SAN attachment.
35
d. Zone Fibre Channel over Ethernet (FCoE) to the Symmetrix. e. Ensure Symmetrix volumes are seen by the DCA servers. f. Format and mount Symmetrix volumes on mirror locations. 3. Move master databases to VMAX storage. 4. Create a test database utilizing the VMAX SAN Mirror configuration. 5. Test failover to remote DCA to run directly off R2 volumes. a. Split SRDF links. b. Mount R2 volumes on Remote DCA. c. Recover remote DCA internal volumes from R2 image.
Preparing the VMAX for SAN Mirror attachment

SYMCLI commands were used to configure the VMAX systems for this document since the commands are easily scripted to provide repeatable operations. The EMC Symmetrix Management Console (SMC) could also have been used for one-time setup or to produce a template of failover operations for later scripting. This section contains the following information:
Creating volumes for DCA master and mirror databases on page 36 Configuring SRDF copies on page 37 Snaps on page 38 Clones on page 38
Creating volumes for DCA master and mirror databases Each segment server supports two, 2.7 TB database LUNs. In normal operation, one of these LUNs is used for a mirror database instance, and the other is used for a primary instance. The master server supports a single, 2.1 TB LUN. To implement SAN Mirror, LUNs of the same size should be created on the VMAX and made visible to the servers. The mirror LUN may or may not be used as an additional primary in a SAN Mirror configuration. The LUNs can be verified by issuing omreport storage vdisk on the segment servers. By issuing the gpssh -f gpssh.txt omreport storage
36
vdisk command, it is possible to see the exact LUN configuration on each segment server. Omreport output from the DCA and a script to create Symmetrix volumes are shown in Appendix C, Creating VMAX Volumes for DCA Mirror Database. Configuring SRDF copies To have a consistent, remote replica of the database, all of the DCA mirror volumes are replicated to the remote VMAX. This includes all volumes from the master server and each segment server. To ensure consistency during SRDF operations, all of the SAN Mirror volumes must be part of the same SRDF group. Configurations that span SRDF groups or Symmetrix VMAX systems must use SRDF Consistency Groups, which is beyond the scope of this paper. See EMC SRDF/A and SRDF/A Multi-Session Consistency on Unix and Windows for more information about SRDF consistency groups. If the primary site is lost for any reason, the database can be brought up on a remote DCA attached to the remote VMAX. The procedure to do this is described in Appendix J, Starting EMC Greenplum Database from Mirror Segments, (such as, on the R2 VMAX). The test system used SRDF/A to replicate data from the primary to the remote VMAX System. When using SRDF, it is important to ensure there is enough bandwidth between the systems to handle the expected write workload. A Symmetrix Performance Guru should be consulted to verify the configuration has adequate bandwidth.
To use TimeFinder/Snap on the remote VMAX, the Symmetrix must be configured for SRDF Device Write Pacing. With write pacing enabled, the primary Symmetrix will slow down host writes if the primary DCA's write workload is greater than the remote VMAX's ability to destaged and switch cycles. This preserves SRDF/A's ability to keep propagate data to the remote VMAX. Sample SYMCLI commands and output from the SRDF configuration are shown in Appendix D, Creating the SRDF Configuration.
37
Snaps The test configuration used TimeFinder/Snap to create point-in-time copies of the data on the remote VMAX. Snaps provide a flexible mechanism to make instant, consistent copies of the database volumes. As hosts write data, the original data is preserved in a "SAVE" pool, and pointers for the Snap volumes are changed to point to this preserved data. Only changed data is tracked by TimeFinder/Snap. This results in the lowest capacity overhead when creating multiple Snaps of the data. Taking frequent Snaps allows the database to be quickly rolled back to a known good state if data corruption is discovered. The SAVE pool must be sized to accommodate the expected capacity and bandwidth of changed data. It is considered best practice to configure SAVE devices across as many physical disks as possible in order to optimize the system's performance. Sample commands and output are shown in Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery. Clones TimeFinder/Clone is a facility that creates full copies of Symmetrix volumes. Clones are very useful when making copies of data for mounting and querying on a separate host, as the clone volumes can be placed on separate physical drives. By separating the clones from the source data, users can work on the clones without impacting production. The tested configuration used TimeFinder/Snap instead of clones because multiple point-in-time copies of the data are required. Clones would require significantly more storage since each clone is a full copy of the source volume.
Configuring the local DCA for the Symmetrix

In order to attach a DCA to the SAN, it is necessary to change the configuration of the internal Connectrix switches, and to discover the WWNs of each server's Converged Network Adapters (CNAs). Once the DCA servers can recognize SAN storage, it is possible to move the master databases and mirrored segments to the SAN. This section contains the following information:
Discovering the DCA server WWNs on page 39
38
Enabling FCoE on the internal EMC Connectrix MP-8000B Switches on page 39 Changing mirror locations on the DCA (if needed) on page 39 Moving master database for existing DCA on page 41 Moving segment database mirrors for existing DCA on page 41
Discovering the DCA server WWNs Each master server and segment server contains a Brocade 1020 CNA for Fibre Channel Over Ethernet (FCOE) connectivity to the SAN. The internal Connectrix MP-8000B switch strips the Ethernet headers when forwarding traffic to the SAN. Thus, issuing "bcu port -list | grep fcoe" on each DCA server will display the adapters' WWNs. The output of the command on the test harness is shown in Appendix A, Discovering DCA Internal Server WWNs. Enabling FCoE on the internal EMC Connectrix MP-8000B Switches FCoE must be enabled on the DCA to connect the it to external SAN storage. Each CNA in each server will then show up as a separate initiator to be zoned on the SAN. The DCA is configured with two, EMC Connectrix MP-8000B switches. Each switch contains eight Fibre ports for SAN connection. FCoE must be enabled individually on each of the MP-8000Bs internal Ethernet ports in order for the internal servers to appear on the SAN. The procedure to accomplish this manually is shown in Appendix E, Configuring DCA Internal Switch for SAN Mirror. The procedure can be automated using Expect, which is a standard Linux utility on the DCA. A sample script is shown in Appendix F, Automating DCA Internal SAN Configuration. Changing mirror locations on the DCA (if needed) Users specify where the DCA stores primary and mirrored data segments at installation time when gpinitsystem is run. The default is to create multiple data directories (/data1 and /data2), where each /data directory contains ./primary and ./mirror subdirectories. See The Greenplum Database Installation Guide, Chapter 3, for more information about default configurations. In a SAN Mirror configuration, internal DCA storage should be mounted on the ./primary directories. SAN storage should be mounted on the ./san_mirror directories.
39
Storage for the master and standby master servers must also be moved to the VMAX to enable failover to the remote VMAX. Moving an operational master to the VMAX will require a short period of downtime as the database requires restart during these operations.
IMPORTANT For the purposes of this paper, the SAN Mirror storage was mounted directly on the ./mirror directories. An upcoming release of the Greenplum database for DCA will streamline and automate the SAN Mirror installation process. Default mount points for SAN Mirror disk will be ./san_mirror. A high-level diagram of the recommended configuration is shown in Figure 9.
Figure 9
SAN Mirror mount points on VMAX
40
If the initial database was created without mirrors. the gpaddmirrors utility can be run to dynamically add SAN Mirrors to the existing database. More information on gpaddmirrors can be found in the Greenplum Database 4.1 Administrator's Guide. Moving master database for existing DCA Downtime is required to move the master database to VMAX storage. The test configuration did not use a standby master, so downtime had to be taken to manually copy data from the /data/master directory to the VMAX, and to mount the VMAX volume on the /data/master directory. For existing DCAs with mirrored master databases, it is possible to remove and reinitialize the standby master server on VMAX storage. When this process is complete, the database can be failed over to the standby master while the primary master server is moved to VMAX. This greatly reduces the downtime required to implement a SAN Mirror configuration. The Greenplum command gpinitstandby can be used to change between the master server and the standby master. Moving segment database mirrors for existing DCA If SAN Mirror is being added to a pre-existing DCA configuration, where both the mirrors and the primaries reside on the DCA, the gpmovemirrors utility can be used to move the existing mirrors to external SAN storage. This is an online process for the segment servers. Gpmovemirrors ships with Greenplum database release 4.1.1.1. When using gpmovemirrors, create an input file that specifies the host address, port and system file space location of the current mirror, and the host address, port, replication port, and system file space location of the new mirror. The input file should be a plain text file with the format:
[<filespace1_fsname>[:<filespace2_fsname>:...] <old_address>:<port>:<system_filespace_location> [<new_address:port>:<replication_port>:<system_filespace_location>[:<fselocat ion>:...]]
For example:
sdw1-1:50000:/data1/mirror/gpseg11 sdw1-1:50000:51000:/data1/san_mirror/gpseg11
The host address, ports and system filespace locations can be found in the gp_segment_configuratoin and pg_filespace tables. For more information, refer to the Greenplum Database 4.1 Administrator's Guide.
41
IMPORTANT It is critical that all of the mirrors reside on the VMAX to ensure a recoverable database on the remote DCA.
Configuring the remote DCA

The mount points on the remote DCA should be the same as on the primary DCA, with internal DCA storage on the ./primary directory for each /data directory, and VMAX storage mounted on the ./san_mirror directory. For the purposes of this document, VMAX volumes were mounted directly on the ./mirror directory. This provided the simplest configuration for failover and backup. Configuring the primary and remote sides identically will allow the database to recover on the remote side as explained in Appendix J, Starting EMC Greenplum Database from Mirror Segments, (such as, on the R2 VMAX).
Operating a SAN Mirror configuration

The external VMAX storage is not read from in normal operation. Read requests are satisfied by primary database segments, whose data reside on internal DCA storage. By replicating the mirrored segments with SRDF, a copy of the data is propagated to a remote site. The database can then be brought up on the mirrored segments at the remote site if the primary site fails. However, SRDF alone does not guard against the possibility data corruption. To provide an extra level of protection, TimeFinder/Snap is used to take periodic copies of the database at the remote site. This lets system administrators fall back to the last known good copy of the database if the database becomes unusable for any reason. By default, the DCA will work from the primary copies of the data on the remote DCA. To bring TimeFinder and SRDF copies up on the remote DCA, the remote database must be configured to use the SAN Mirror devices. This requires marking the primary segments as "down" on the remote DCA prior to starting the database. Once the database is successfully started, the gprecoverseg utility can be used to resynchronize the primary segments with the SAN Mirror copies.
42
Two tables within the GP database contain the configuration and location of the GP segment databases. These tables are critical to SAN Mirror operations that bring the database up on a different DCA. They are gp_segment_configuration and pg_filespace_entry. The gp_segment_configuration table maintains information about what database is located on which segment server, as well as what role each segment database has (that is, primary or mirror). In order to configure the GP database to use only the mirror segments, the gp_segment_configuration table space needs to be modified. The pg_filespace_entry table maintains the filesystem and directory location of each of the segment databases. It is important for manageability to have consistent, easy-to-understand directory structures for the primary and mirror instances. The procedure to bring up the database on the remote VMAX and to copy data from the remote VMAX to the remote DCA is detailed in Appendix J, Starting EMC Greenplum Database from Mirror Segments, (such as, on the R2 VMAX). The following are further described in this section:
Using TimeFinder/SNAP to create point-in-time images on the remote VMAX on page 43 Recovering a corrupt database using TimeFinder/SNAP and SRDF on page 44 Failover on page 45 Failback on page 46
Using TimeFinder/SNAP to create point-in-time images on the remote VMAX In normal operation a DCA will serve user requests from the primary copies of the data and will write updates to the SAN Mirror copies to ensure consistency and resiliency of the database. In a SAN Mirror configuration, additional TimeFinder/Snap images are created on the remote DCA to provide quick recovery in the event of data corruption. The procedure outlined in Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery and Appendix H, Mounting and Checking a Symmetrix SNAP of the Greenplum Database can be scripted and run at regular intervals to ensure easy recoverability of the database. The frequency with which copies are made is a function of the customer's business requirements.
43
Typical steps in normal operation are shown below: 1. Data is continually replicated to the remote VMAX using SRDF/A. 2. TimeFinder/Snap is used to take periodic copies of the data on the remote VMAX, as long as all of the DCAs primary and mirrored segments are operational. See Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery. 3. Each Snap image is verified on the remote DCA. The steps listed below are detailed in Appendix H, Mounting and Checking a Symmetrix SNAP of the Greenplum Database. a. Verify the current master server; use that master image to obtain all current configuration data. b. Mount the Snap volumes to the remote DCA. c. Verify that all segment servers were operational and reporting correct status. d. Run gpcheckcat to verify that the database is correct. 4. Older Snaps may be deleted, if desired, once the new Snap is verified. Recovering a corrupt database using TimeFinder/SNAP and SRDF If the database becomes corrupted for any reason, it can be restored to a previous snapped image. Since the snaps are taken on the remote VMAX, the procedure to restore the primary site is as shown below. Details are shown in Appendix I, Recovering a DCA Using a SNAP Session and SRDF. 1. Stop the primary database and unmount volumes on primary DCA. 2. Split the SRDF devices so the snap can be restored to the R2 volumes. 3. Restore the Snaps to the R2 volumes, then terminate the Snap restore session. 4. Restore the SRDF R2 devices to the R1 devices, to restore the R1 image to a known good point. 5. Remount the R1 devices on the primary DCA
44
6. Start the database in maintenance mode on the primary DCA and mark the primary segments as "down" so the database will come up on the mirrored segments. 7. Restart the database normally on the primary DCA. 8. Type gprecoverseg -F to recover the primary volumes from the mirrors. 9. Verify that all segments have synchronized by typing gprecoverseg -m. 10. Either restart the database or type gprecoverseg -r to move primary activity to internal primary volumes. Failover The database can be brought up on the remote DCA by running a series of SRDF and Greenplum commands. The step-by-step procedure to do this is detailed in Appendix J, Starting EMC Greenplum Database from Mirror Segments, (such as, on the R2 VMAX), and is summarized below: 1. Use SYMCLI commands to fail over the storage to the remote site. 2. Mount the R2 volumes to the remote DCA and start the database in maintenance mode. 3. Mark the remote DCA's primary segments as "down" and the mirrored segments as "up" so the database will come up on the mirrored (SAN) segments. 4. Restart the database normally. 5. Type gprecoverseg -F to recover the remote DCA's primary (internal) storage from mirrored (SAN) storage. 6. Verify that all segments have synchronized by typing gprecoverseg -m. 7. Either restart the database or type gprecoverseg -r to move primary activity to internal primary volumes. 8. When the primary site is restored, begin synchronizing from the remote VMAX to the primary VMAX.
45
Failback Failback from the remote to the primary site is simply the reverse of the failover process. A high-level description is listed below. 1. Ensure the primary DCA has unmounted the mirrored segments from the primary VMAX. 2. Synchronize the primary VMAX from the remote VMAX. This can be done while the database is running on the remote VMAX. 3. Use SYMCLI or the Symmetrix Management Console to issue SRDF commands to fail the database back to the primary site. 4. Mount the R1 volumes on the primary DCA and start the database in maintenance mode 5. Mark the primary DCA's primary segments as "down" and the mirrored segments as "up" so the database will come up on the mirrored (SAN) segments. 6. Restart the database normally. 7. Type gprecoverseg -F to recover the DCA's primary (internal) volumes from the mirrored (SAN) segments.
8. Verify that all segments have synchronized by typing gprecoverseg -m. 9. Either restart the database or type gprecoverseg -r to move primary activity to internal primary volumes.
46
Summary
The EMC Greenplum database offers industry-leading capability for "big data" analysis, allowing customers to make precise business decisions more quickly. The EMC Greenplum Data Compute Appliance (DCA) is a self-contained system, designed to produce optimal performance using commodity components. It contains all the hardware and software necessary and is a complete data warehouse appliance. By using SAN Mirror to place the DCA mirrored databases on external storage, users leverage the capabilities of external storage to augment the performance and simplicity of the DCA. The EMC Symmetrix VMAX brings robust multipathing, replication and data management to the Greenplum database, to facilitate disaster recovery and to create gold copies of the database that guard against corruption of the data warehouse. The procedure to place SAN copies on external storage is documented here as a manual, step-by-step process. Following these procedures enables SAN Mirror to run on external VMAX arrays. Future releases of the Greenplum database will include tools to simplify SAN Mirror implementation.
Summary
47
48
Invisible Body Tag
A
This appendix contains the following information:
Discovering DCA internal server WWNs ...................................... 50
49
Discovering DCA internal server WWNs

By issuing the bcu port -list command on each server, the WWNs on each segment server and the master server are displayed. These should be used to create mapping and masking files for the Symmetrix. Issue the command below to display all the WWNs on the DCA. The sed command was used to remove colons so that it is easier to cut and paste WWNs into Symmetrix configuration scripts.
gpssh -f gpssh.txt 'bcu port --list |grep fcoe' | sort -k1 | sed -e 's/://g'
Figure 10 shows a sample output:
Figure 10
Sample output
50
Invisible Body Tag
B
Verifying DCA volume sizes ............................................................ 52
51
Verifying DCA volume sizes

The following command was used to inspect the volume sizes on the segment servers:
gpssh -f gpssh.txt omreport storage vdisk
The following is sample output from one server:

[root@mdw ~]# gpssh -f gpssh.txt omreport storage vdisk [sdw4] List of Virtual Disks in the System [sdw4] [sdw4] Controller PERC H700 Integrated (Slot 4) [sdw4] ID : 0 [sdw4] Status : Ok [sdw4] Name : Virtual Disk 0 [sdw4] State : Ready [sdw4] Hot Spare Policy violated : Not Assigned [sdw4] Virtual Disk Bad Blocks : No [sdw4] Secured : Not Applicable [sdw4] Progress : Not Applicable [sdw4] Layout : RAID-5 [sdw4] Size : 48.01 GB (51548651520 bytes) [sdw4] Device Name : /dev/sda [sdw4] Bus Protocol : SAS [sdw4] Media : HDD [sdw4] Read Policy : Adaptive Read Ahead [sdw4] Write Policy : Write Back [sdw4] Cache Policy : Not Applicable [sdw4] Stripe Element Size : 128 KB [sdw4] Disk Cache Policy : Disabled [sdw4] [sdw4] ID : 1 [sdw4] Status : Ok [sdw4] Name : Virtual Disk 1 [sdw4] State : Ready [sdw4] Hot Spare Policy violated : Not Assigned [sdw4] Virtual Disk Bad Blocks : No [sdw4] Secured : Not Applicable [sdw4] Progress : Not Applicable [sdw4] Layout : RAID-5 [sdw4] Size : 2,743.86 GB (2946202337280 bytes) [sdw4] Device Name : /dev/sdb [sdw4] Bus Protocol : SAS [sdw4] Media : HDD [sdw4] Read Policy : Adaptive Read Ahead [sdw4] Write Policy : Write Back [sdw4] Cache Policy : Not Applicable [sdw4] Stripe Element Size : 128 KB [sdw4] Disk Cache Policy : Disabled [sdw4] Data cut due to length of output ?
52
Invisible Body Tag
C

Creating VMAX volumes for DCA mirror database .................... 54 Assigning volumes to the DCA segment servers.......................... 56
53
Creating VMAX volumes for DCA mirror database

The following command was used to create Symmetrix Virtual (thin) devices, to assign to the DCA. The specified volume size was 2.7 TB, to match the DCA. Actual capacity will only be allocated as it is required.
[root@mdw ~]# symconfigure -sid 4467 -cmd "create dev count=18, size=2700GB, emulation=fba, binding to pool=DCA_R1_Pool, config=tdev, meta_config=striped;" commit
The following is sample output:

Execute a symconfigure operation for symmetrix '000192604467' (y/[n]) ? y A Configuration Change operation is in progress. Please wait... Establishing a configuration change session...............Established. Processing symmetrix 000192604467 Performing Access checks..................................Allowed. Checking Device Reservations..............................Allowed. Initiating COMMIT of configuration changes................Started. Committing configuration changes..........................Queued. COMMIT requesting required resources......................Obtained. Step 014 of 050 steps.....................................Executing. Step 040 of 177 steps.....................................Executing. Step 044 of 177 steps.....................................Executing. Step 044 of 177 steps.....................................Executing. Step 081 of 177 steps.....................................Executing. Step 084 of 177 steps.....................................Executing. Step 088 of 177 steps.....................................Executing. Step 091 of 177 steps.....................................Executing. Step 093 of 177 steps.....................................Executing. Step 093 of 177 steps.....................................Executing. Step 107 of 177 steps.....................................Executing. Step 109 of 177 steps.....................................Executing. Step 109 of 177 steps.....................................Executing. Step 128 of 177 steps.....................................Executing. Step 139 of 177 steps.....................................Executing. Step 139 of 177 steps.....................................Executing. Step 139 of 177 steps.....................................Executing. Step 139 of 177 steps.....................................Executing. Step 139 of 177 steps.....................................Executing. Step 139 of 177 steps.....................................Executing. Local: COMMIT............................................Done. Binding devices...........................................Done. New symdevs: 025B:0266 [Striped meta, head 025B, member size 262668 cyl] New symdevs: 0267:0272 [Striped meta, head 0267, member size 262668 cyl] New symdevs: 0273:027E [Striped meta, head 0273, member size 262668 cyl]
54
New symdevs: 027F:028A [Striped meta, head 027F, member size 262668 New symdevs: 028B:0296 [Striped meta, head 028B, member size 262668 New symdevs: 0297:02A2 [Striped meta, head 0297, member_size 262668 New symdevs: 02A3:02AE [Striped meta, head 02A3, member size 262668 New symdevs: 02AF:02BA [Striped meta, head 02AF, member size 262668 New symdevs: 02BB:02C6 [Striped meta, head 02BB, member size 262668 New symdevs: 02C7:02D2 [Striped meta, head 02C7, member size 262668 New symdevs: 02D3:02DE [Striped meta, head 02D3, member size 262668 New symdevs: 02DF:02EA [Striped meta, head 02DF, member size 262668 New symdevs: 02EB:02F6 [Striped meta, head 02EB, member size 262668 New symdevs: 02F7:0302 [Striped meta, head 02F7, member size 262668 New symdevs: 0303:030E [Striped meta, head 0303, member size 262668 New symdevs: 030F:031A [Striped meta, head 030F, member size 262668 New symdevs: 031B:0326 [Striped meta, head 031B, member size 262668 New symdevs: 0327:0332 [Striped meta, head 0327, member size 262668 Terminating the configuration change session..............Done. The configuration change session has successfully completed. ?
cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl]
Creating VMAX volumes for DCA mirror database

55
Assigning volumes to the DCA segment servers

The following symaccess commands were used to create initiator groups for the master and segment servers in the DCA, port groups for the Symmetrix, and to assign devices to the servers. WWNs from the output above were cut and pasted into the script.
symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid -sid 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 4467 create create create create create create create create create -name -name -name -name -name -name -name -name -name MDW1 SDW1 SDW2 SDW3 SDW4 SDW5 SDW6 SDW7 SDW8 -type -type -type -type -type -type -type -type -type storage storage storage storage storage storage storage storage storage devs devs devs devs devs devs devs devs devs 25B,267 273,27F 28B,297 2A3,2AF 2BB,2C7 2D3,2DF 2EB,2F7 303,30F 31B,327
create -name MDW1 -type initiator -wwn 1000000533482f24 -type initiator add -name MDW1 -wwn 1000000533482f25 create -name SDW1 -type initiator -wwn 100000053348159a -type initiator add -name SDW1 -wwn 100000053348159b create -name SDW2 -type initiator -wwn 10000005334817e8 -type initiator add -name SDW2 -wwn 10000005334817e9 create -name SDW3 -type initiator -wwn 10000005334819b8 -type initiator add -name SDW3 -wwn 10000005334819b9 create -name SDW4 -type initiator -wwn 100000053348146c -type initiator add -name SDW4 -wwn 100000053348146D create -name SDW5 -type initiator -wwn 1000000533482554 -type initiator add -name SDW5 -wwn 1000000533482555 create -name SDW6 -type initiator -wwn 1000000533481f58 -type initiator add -name SDW6 -wwn 1000000533481f59 create -name SDW7 -type initiator -wwn 100000053348256c -type initiator add -name SDW7 -wwn 100000053348256D create -name SDW8 -type initiator -wwn 1000000533481f48 -type initiator add -name SDW8 -wwn 1000000533481f49
symaccess -sid 4467 create -name DCA -type port -dirport 6g:0,8g:0,10g:0,12g:0,5f:0,7f:0,9f:0,11f:0 symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess -sid -sid -sid -sid -sid -sid -sid -sid -sid 4467 4467 4467 4467 4467 4467 4467 4467 4467 create create create create create create create create create view view view view view view view view view -name -name -name -name -name -name -name -name -name MDW1 SDW1 SDW2 SDW3 SDW4 SDW5 SDW6 SDW7 SDW8 -sg -pg -pg -pg -pg -pg -pg -pg -pg MDW1 -ig MDW1 -pg DCA DCA -ig SDW1 -sg SDW1 DCA -ig SDW2 -sg SDW2 DCA -ig SDW3 -sg SDW3 DCA -ig SDW4 -sg SDW4 DCA -ig SDW5 -sg SDW5 DCA -ig SDW6 -sg SDW6 DCA -ig SDW7 -sg SDW7 DCA -ig SDW8 -sg SDW8
56
Invisible Body Tag
D
Creating the SRDF configuration..................................................... 58
57
Creating the SRDF configuration

The SRDF configuration between VMAX systems was created with the following command:
symrdf addgrp -label dca_rdfg -rdfg 1 -sid 04467 -dir 5h 04452 -remote_dir 5h symrdf modifygrp -label dca_rdfg -sid 04467 -add -dir symrdf modifygrp -label dca_rdfg -sid 04467 -add -dir symrdf modifygrp -label dca_rdfg -sid 04467 -add -dir symrdf modifygrp -label dca_rdfg -sid 04467 -add -dir symrdf modifygrp -label dca_rdfg -sid 04467 -add -dir symrdf modifygrp -label dca_rdfg -sid 04467 -add -dir symrdf modifygrp -label dca_rdfg -sid 04467 -add -dir -remote_rdfg 1 -remote_sid 6h -remote_dir 6h 7h -remote_dir 7h 8h -remote_dir 8h 9h -remote_dir 9h 10h -remote_dir 10h 11h -remote_dir 11h 12h -remote_dir 12h
symrdf -file rdf_pairs.txt -sid 04467 -rdfg 1 -noprompt -v createpair -type r1 -establish -rdf_mode sync -g DCA_rdfg
The file rdf_pairs.txt contained the following lines, listing the device pairs to create:
025B0405 02670411 0273041D 027F0429 028B0435 02970441 02A3044D 02AF0459 02BB0465 02C70471 02D3047D 02DF0489 02EB0495 02F704A1 030304AD 030F04B9 031B04C5 032704D1
58
Invisible Body Tag
E
Configuring DCA internal switch for SAN Mirror....................... 60
59
Configuring DCA internal switch for SAN Mirror

The DCA uses dual Connectrix MP-8000B switches for internal communication. Every segment server has a 10 Gb Ethernet attachment to each of the switches. Each switch also has four, 8 Gb Fibre Channel ports to connect to an external SAN. Normally, these ports are not used. In a SAN Mirror configuration, they must be configured to connect to the Symmetrix VMAX arrays. The number of ports required is a function of the SAN Mirror workload. Consult a Symmetrix Performance Guru to help determine the correct number of ports.
This section details the procedure to configure the switches. The next section provides a script that could be used to automate the switch configuration. 1. Log in to switch 1 on the DCA. ssh 172.28.0.170 username = admin password = changeme (default password) 2. If connecting the switch to an existing fabric make sure to change the domain ID of the switch to one that is not already assigned in the fabric. The default ID for the switches is 1. 3. Enter into the configuration shell. cmsh 4. Enter into config mode. configure terminal (short hand = conf t) 5. Set the protocol to spanning-tree Rapid Spanning Tree (rstp) and the bridge priority to a high number so the switch will not become the root switch when plugged into the fabric. protocol spanning-tree rstp bridge-priority 8192
60
Sample command screen output:

i-sw-1(config)# protocol spanning-tree rs ftp i-sw-1(conf-rstp)# bridge-priority 8192 i-sw-1(conf-rstp)# exit
i-sw-1(config)# 6. Create a CEE-MAP for the FCoE protocol and set the priority-group table weights. priority-group-table 1 weight 40 pfc priority-group-table 1 weight 60 priority-table 2 2 2 1 2 2 2 15.0 exit Sample command screen output:
i-sw-1(config)#cee-map fcoe-vlan i-sw-1(conf-ceemap)#priority-group-table 1 weight 40 pfc i-sw-1(conf-ceemap)#priority-group-table 2 weight 60 i-sw-1(conf-ceemap)#priority-table 2 2 2 1 2 2 2 15.0 i-sw-1(conf-ceemap)#exit i-sw-1(config)#
7. Set up the lldp protocol. Protocol lldp advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv exit Sample command screen output:
i-sw-1(config)#protocol lldp i-sw-1(conf-lldp)#advertise dcbx-fcoe-app-tlv i-sw-1(conf-lldp)#advertise dcbx-fcoe-logical-link-tlv i-sw-1(conf-lldp)#exit i-sw-1(config)#
8. Set up a vlan for the FCoE traffic.

Note: Only one FCoE vlan is allowed per switch.
The vlan number can be any number other than 1 or 199. The default setting on the DCA is to route network traffic thru vlan 199 on switch 1, and 299 on switch 2. For this environment, FCOE traffic is routed thru vlan 1002 on switch 1 and 2002 on switch 2. interface vlan 1002 (on switch one; should be 2002 on switch 2) description FCoE VLAN fcf forward
61
exit Sample command output screen capture: i-sw-1(config)#interface vlan 1002 (on switch one; should be 2002 on switch 2) i-sw-1(conf-if-vl-1002)#description FCoE VLAN i-sw-1(conf-if-vl-1002)#fcf forward i-sw-1(conf-if-vl-1002)#exit i-sw-1(config)# 9. Setup vlan rules for FCoE traffic and add the rules to a group for assignment to the interfaces. Group 2 is assigned to the Ethernet vlan rules configured on the switch by default at the factory. vlan classifier rule 1 proto fip encap ethv2 vlan classifier rule 2 proto fcoe encap ethv2 vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 Sample command output screen capture:
i-sw-1(config)#vlan i-sw-1(config)#vlan i-sw-1(config)#vlan i-sw-1(config)#vlan classifier classifier classifier classifier rule 1 proto fip encap ethv2 rule 2 proto fcoe encap ethv2 group 1 add rule 1 group 1 add rule 2
i-sw-1(config)# 10. Configure each of the connected network ports. For a full rack ports 0/0 0/17 are used. For a rack ports 0/0 0/7 & 0/16 & 0/17 are used. Ports 0/16 abd 0/17 are for the master and standby servers. Ports 0/0 0/15 are for the segment servers. Repeat the below commands for each interface incrementing the int te 0/# for each port. int te 0/0 switchport switchport mode converged vlan classifier activate group 2 vlan 199 vlan classifier activate group 1 vlan 1002 no shutdown lldp fcoe-priority-bits 0x8 spanning-tree edgeport spanning-tree edgeport bpdu-guard cee fcoe-vlan exit Sample command output screen capture:
i-sw-1(config)#int te 0/0 i-sw-1(conf-if-te-0/0)#switchport
62
i-sw-1(conf-if-te-0/0)#switchport mode converged i-sw-1(conf-if-te-0/0)#vlan classifier activate group 2 vlan 299 i-sw-1(conf-if-te-0/0)#vlan classifier activate group 1 vlan 2002 i-sw-1(conf-if-te-0/0)#no shutdown i-sw-1(conf-if-te-0/0)#lldp fcoe-priority-bits 0x8 i-sw-1(conf-if-te-0/0)#spanning-tree edgeport i-sw-1(conf-if-te-0/0)#spanning-tree edgeport bpdu-guard i-sw-1(conf-if-te-0/0)#cee fcoe-vlan i-sw-1(conf-if-te-0/0)#exit
11. Exit out of the configuration terminal.

i-sw-1(config)# exit i-sw-1#
12. Create a copy of the startup-config file so that you have the default configuration in the event that you have to revert back.
copy startup-config orig-startup-config
13. Copy the running-config to the startup-config so that you dont lose the settings on a reboot or power outage. Copy running-config startup-config. Sample command output screen capture:
i-sw-1# copy startup-config orig-startup-config Building configuration... i-sw-1# copy running-config startup-config Overwrite the startup config file (y/n): y Building configuration... i-sw-1# dir Contents of flash:// -rw-r----4589 Wed Apr 27 18:47:25 2011 i-sw-1#
orig-startup-config
14. Exit out of the cee shell back into the switch shell
Exit
Sample screen capture:

i-sw-1# i-sw-1#exit i-sw-1:admin>
15. To validate that the changes made were correct check the fcoe port logins to make sure you have the proper number of WWNs logging into the switch.
fcoe loginshow

63
Sample command output screen capture are shown below:

i-sw-1:admin> fcoe --loginshow ================================================================================ Port Te port Device WWN Device MAC Session MAC ================================================================================ 8 Te 0/0 10:00:00:05:33:48:15:9b 00:05:33:48:15:9b 0e:fc:00:03:08:01 9 Te 0/1 10:00:00:05:33:48:17:e9 00:05:33:48:17:e9 0e:fc:00:03:09:01 10 Te 0/2 10:00:00:05:33:48:19:b9 00:05:33:48:19:b9 0e:fc:00:03:0a:01 11 Te 0/3 10:00:00:05:33:48:14:6d 00:05:33:48:14:6d 0e:fc:00:03:0b:01 12 Te 0/4 10:00:00:05:33:48:25:55 00:05:33:48:25:55 0e:fc:00:03:0c:01 13 Te 0/5 10:00:00:05:33:48:1f:59 00:05:33:48:1f:59 0e:fc:00:03:0d:01 14 Te 0/6 10:00:00:05:33:48:25:6d 00:05:33:48:25:6d 0e:fc:00:03:0e:01 15 Te 0/7 10:00:00:05:33:48:1f:49 00:05:33:48:1f:49 0e:fc:00:03:0f:01 16 Te 0/8 10:00:00:05:33:26:b0:b3 00:05:33:26:b0:b3 0e:fc:00:03:10:01 17 Te 0/9 10:00:00:05:33:64:03:19 00:05:33:64:03:19 0e:fc:00:03:11:01 18 Te 0/10 10:00:00:05:33:48:e3:15 00:05:33:48:e3:15 0e:fc:00:03:12:01 19 Te 0/11 10:00:00:05:33:48:e7:73 00:05:33:48:e7:73 0e:fc:00:03:13:01 20 Te 0/12 10:00:00:05:33:48:e7:93 00:05:33:48:e7:93 0e:fc:00:03:14:01 21 Te 0/13 10:00:00:05:33:48:19:15 00:05:33:48:19:15 0e:fc:00:03:15:01 22 Te 0/14 10:00:00:05:33:48:1f:25 00:05:33:48:1f:25 0e:fc:00:03:16:01 23 Te 0/15 10:00:00:05:33:48:1e:dd 00:05:33:48:1e:dd 0e:fc:00:03:17:01 i-sw-1:admin>
16. You can also check each of the nodes and make sure that the fcoe port link status is up and running using gpssh with a host file containing the nodes in the DCA.
gpssh -f seg_srvr_list 'bcu port --list |grep fcoe'
64
Sample command output screen capture:

[root@mdw /]# gpssh -f seg_srvr_list 'bcu port --list |grep fcoe' [sdw16] fcoe 10:00:00:05:33:48:1e:dc 021701 Linkup [sdw16] fcoe 10:00:00:05:33:48:1e:dd 031701 Linkup [sdw14] fcoe 10:00:00:05:33:48:19:14 021501 Linkup [sdw14] fcoe 10:00:00:05:33:48:19:15 031501 Linkup [sdw15] fcoe 10:00:00:05:33:48:1f:24 021601 Linkup [sdw15] fcoe 10:00:00:05:33:48:1f:25 031601 Linkup [sdw12] fcoe 10:00:00:05:33:48:e7:72 021301 Linkup [sdw12] fcoe 10:00:00:05:33:48:e7:73 031301 Linkup [sdw13] fcoe 10:00:00:05:33:48:e7:92 021401 Linkup [sdw13] fcoe 10:00:00:05:33:48:e7:93 031401 Linkup [sdw10] fcoe 10:00:00:05:33:64:03:18 021101 Linkup [sdw10] fcoe 10:00:00:05:33:64:03:19 031101 Linkup [sdw11] fcoe 10:00:00:05:33:48:e3:14 021201 Linkup [sdw11] fcoe 10:00:00:05:33:48:e3:15 031201 Linkup [ sdw4] fcoe 10:00:00:05:33:48:14:6c 020b01 Linkup [ sdw4] fcoe 10:00:00:05:33:48:14:6d 030b01 Linkup [ sdw5] fcoe 10:00:00:05:33:48:25:54 020c01 Linkup [ sdw5] fcoe 10:00:00:05:33:48:25:55 030c01 Linkup [ sdw6] fcoe 10:00:00:05:33:48:1f:58 020d01 Linkup [ sdw6] fcoe 10:00:00:05:33:48:1f:59 030d01 Linkup [ sdw7] fcoe 10:00:00:05:33:48:25:6c 020e01 Linkup [ sdw7] fcoe 10:00:00:05:33:48:25:6d 030e01 Linkup [ sdw1] fcoe 10:00:00:05:33:48:15:9a 020801 Linkup [ sdw1] fcoe 10:00:00:05:33:48:15:9b 030801 Linkup [ sdw2] fcoe 10:00:00:05:33:48:17:e8 020901 Linkup [ sdw2] fcoe 10:00:00:05:33:48:17:e9 030901 Linkup [ sdw3] fcoe 10:00:00:05:33:48:19:b8 020a01 Linkup [ sdw3] fcoe 10:00:00:05:33:48:19:b9 030a01 Linkup [ sdw8] fcoe 10:00:00:05:33:48:1f:48 020f01 Linkup [ sdw8] fcoe 10:00:00:05:33:48:1f:49 030f01 Linkup [ sdw9] fcoe 10:00:00:05:33:26:b0:b2 021001 Linkup [ sdw9] fcoe 10:00:00:05:33:26:b0:b3 031001 Linkup [root@mdw greenplum-db]#
17. Repeat the switch steps for switch 2 on the DCA. The IP is 172.28.0.180. Make sure to use a different vlan number for the FCoE vlan such as 2002 to keep them separate from switch 1. Sample command output for creating vlan on switch 2:
i-sw-2(config)#interface vlan 2002 i-sw-2(conf-if-vl-1002)#description FCoE VLAN i-sw-2(conf-if-vl-1002)#fcf forward i-sw-2(conf-if-vl-1002)#exit i-sw-2(config)#
Once the FCoE ports are logged into the switch you can alias and zone them to the SAN storage as a normal fiber port is zoned.
65
66
Invisible Body Tag
F
Script to automate DCA internal SAN configuration................... 68
67
Script to automate DCA internal SAN configuration

The DCA segment servers and database servers are pre-loaded with the "Expect" programming language, which is able to automate repetitive typing tasks. The following script, written in Expect, worked in the lab environment. It automates the steps outlined in Appendix E, Configuring DCA Internal Switch for SAN Mirror, and can be run on the database master to automatically set the internal switches for FCOE SAN attachment. It is presented "as-is", with no guarantees of correctness. Error checking in the script is primitive.
#!/usr/bin/expect -f # # Script to set up internal switches in DCA for FCOE # Facilitates SAN mirror attachment # Note: minimal error checking is done for this first revision
# The "switch" variable is the last octet of the internal IP address of the # MP-8000B switch. The DCA has two switches: 172.28.0.170 and 172.28.0.180 # Both switches need to be set up with an FCOE VLAN, and ports need to route to them set switch "170" while { $switch < "181" } { puts "\r\rswitch $switch\r" if {$switch == "170"} { puts "THIS IS THE FIRST SWITCH\r" set enet_vlanid "199" set fcoe_vlanid "1002" } elseif {$switch == "180"} { puts "THIS IS THE SECOND SWITCH\r" set enet_vlanid "299" set fcoe_vlanid "2002" } puts "\r\rLogging into switch at 172.28.0.$switch\r" spawn ssh "admin\@172.28.0.$switch" expect { timeout abort failed abort "could not reach" abort "password:" } send "changeme\r" expect { timeout abort failed abort "Permission denied" abort ":admin>"
68
} sleep 1 send "cmsh\r" expect { timeout abort failed abort "Error:" abort "#" } sleep 1 puts "\r\rSetting switch protocol to Rapid Spanning Tree, and setting priority\r" send "conf t\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "protocol spanning-tree rstp\r" expect { timeout abort failed abort "Error:" abort "(conf-rstp)#" } send "bridge-priority 8192\r" expect { timeout abort failed abort "Error:" abort "(conf-rstp)#" } puts "\r\rCreating CEE Map for FCOE protocol\r" send "cee-map fcoe-vlan\r" expect { timeout abort failed abort "Error:" abort "(conf-ceemap)#" } send "priority-group-table 1 weight 40 pfc\r" expect { timeout abort failed abort "Error:" abort "(conf-ceemap)#" } send "priority-group-table 2 weight 60\r" expect { timeout abort failed abort
69
"Error:" abort "(conf-ceemap)#" } send "priority-table 2 2 2 1 2 2 2 15.0\r" expect { timeout abort failed abort "Error:" abort "(conf-ceemap)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } puts "\r\rSetting up lldp protocol\r" send "protocol lldp\r" expect { timeout abort failed abort "Error:" abort "(conf-lldp)#" } send "advertise dcbx-fcoe-app-tlv\r" expect { timeout abort failed abort "Error:" abort "(conf-lldp)#" } send "advertise dcbx-fcoe-logical-link-tlv\r" expect { timeout abort failed abort "Error:" abort "(conf-lldp)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } puts "\r\rSetting up vlan for FCOE traffic\r" send "interface vlan $fcoe_vlanid\r" expect { timeout abort failed abort "Error:" abort
70
"(conf-if-vl-$fcoe_vlanid)#" } send "description FCOE VLAN\r" expect { timeout abort failed abort "Error:" abort "(conf-if-vl-$fcoe_vlanid)#" } send "fcf forward\r" expect { timeout abort failed abort "Error:" abort "(conf-if-vl-$fcoe_vlanid)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } puts "\r\rSetting up vlan rules for FCOE traffic\r" send "vlan classifier rule 1 proto fip encap ethv2\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "vlan classifier rule 2 proto fcoe encap ethv2\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "vlan classifier group 1 add rule 1\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "vlan classifier group 1 add rule 2\r" expect { timeout abort failed abort "Error:" abort "(config)#"
71
} # # # # Loop to set each of the configured network ports for FCOE For full rack, ports 0-17 are used For half rack, ports 0-7 and 16-17 are used This script sets ALL ports to FCOE puts "\r\rSetting up individual ports for FCOE...\r" for {set i 0} {$i < 18} {incr i 1} { puts "Port $i...\r" send "int te 0/$i\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "switchport\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "switchport mode converged\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } # Note ENET vlan id is set to 199 on switch .170, 299 on switch .180 send "vlan classifier activate group 2 vlan $enet_vlanid\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } # Note FCOE vlan id is set to 1002 on switch .170, 2002 on switch .180 send "vlan classifier activate group 1 vlan $fcoe_vlanid\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "no shutdown\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#"
72
} send "lldp fcoe-priority-bits 0x8\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "spanning-tree edgeport\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "spanning-tree edgeport bpdu-guard\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "cee fcoe-vlan\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } } # End FOR PORT loop send "exit\r" expect { timeout abort failed abort "Error:" abort "#" } puts "\r\rSaving original config as orig-startup-config-before-fcoe\r" send "copy startup-config orig-startup-config-before-fcoe\r" expect { timeout abort failed abort "Error:" abort
73
"Overwrite it" {send "y\r"; exp_continue} "Building configuration" } sleep 2 puts "\r\rCopying running config to startup-config\r" send "Copy running-config startup-config\r" expect { timeout abort failed abort "Error:" abort "Overwrite the startup config file (y/n):" } send "y\r" expect { timeout abort failed abort "Error:" abort "Building configuration" } sleep 2 send "exit\r" expect { timeout abort failed abort "Error:" abort ":admin" } puts "\r\rIMPORTANT NOTE: PLEASE LOG BACK IN TO SWITCHES AND ISSUE THE FOLLOWING TO\r" puts " VERIFY THAT PORTS HAVE LOGGED IN: fcoe --loginshow\r\r" send "exit\r" set switch [expr $switch + 10]; # END WHILE SWITCH loop } ?
74
Invisible Body Tag
G

Creating SNAPs on remote Symmetrix for disaster recovery ..... 76 Creating a single SNAP of R2 standard devices............................ 77 Creating multiple SNAPS of R2 standard devices ........................ 91
75
Creating SNAPs on remote Symmetrix for disaster recovery

In order to create snaps of the R2 devices on the Symmetrix, the Symmetrix must be configured with the proper amount of snap save devices and the devices are assigned to a snap pool. A proper amount of virtual devices configured with same size and layout as the R2 standard devices that the snap is conducted on.
Note: For the purpose of these procedures, the R2 devices were configured as thin virtual striped meta volumes so the virtual devices (VDEVs) required must also be striped meta volumes of the same size.
76
Creating a single SNAP of R2 standard devices

To create a single SNAP of R2 standard devices, complete the following steps: 1. Verify that the Symmetrix where the snap will be conducted has the proper amount of snap save devices configured and assigned to a snap pool.
symsnap list -sid <symm serial #> -savedevs
Sample output of command:

[root@smdw ~]# symsnap list -sid 452 -savedevs Symmetrix ID: 000192604452 S N A P S A V E D E V I C E S --------------------------------------------------------------------Device SaveDevice Total Used Free Full Sym Emulation Pool Name Tracks Tracks Tracks (%) --------------------------------------------------------------------0339 FBA R2_SNAP_POOL 3939540 0 3939540 0 033A FBA R2_SNAP_POOL 3939540 0 3939540 0 033B FBA R2_SNAP_POOL 3939540 0 3939540 0 033C FBA R2_SNAP_POOL 3939540 0 3939540 0 033D FBA R2_SNAP_POOL 3939540 0 3939540 0 033E FBA R2_SNAP_POOL 3939540 0 3939540 0 033F FBA R2_SNAP_POOL 3939540 0 3939540 0 0340 FBA R2_SNAP_POOL 3939540 0 3939540 0 0341 FBA R2_SNAP_POOL 3939540 0 3939540 0 0342 FBA R2_SNAP_POOL 3939540 0 3939540 0 0343 FBA R2_SNAP_POOL 3939540 0 3939540 0 0344 FBA R2_SNAP_POOL 3939540 0 3939540 0 0345 FBA R2_SNAP_POOL 3939540 0 3939540 0 .. Data cut due to output length 03CE FBA R2_SNAP_POOL 3939540 0 3939540 0 03CF FBA R2_SNAP_POOL 3939540 0 3939540 0 Total Tracks MB(s) [root@smdw ~]# --------- --------- --------594870540 0 594870540 37179428 0 37179428 ---0

77
2. To list available save device pools (snap pools) and verify the state of the poo, make sure the pool that will be used is Enabled.
symsnap -sid <symm serial #> -pools list

[root@smdw ~]# symsnap -sid 452 -pools list Symmetrix ID: 000192604452 S A V E D E V I C E P O O L S -------------------------------------------------------------------------------Dev Total Enabled Used Free Full Pool Session Pool Name Emul Tracks Tracks Tracks Tracks (%) State Status -------------------------------------------------------------------------------DEFAULT_POOL FBA 0 0 0 0 0 Disabled Inactive DEFAULT_POOL 3390 0 0 0 0 0 Disabled Inactive DEFAULT_POOL 3380 0 0 0 0 0 Disabled Inactive DEFAULT_POOL AS400 0 0 0 0 0 Disabled Inactive R2_SNAP_POOL FBA 594870540 594870540 0 594870540 0 Enabled Inactive [root@smdw ~]#
3. Verify that the Symmetrix where the snap will be conducted has the proper amount of virtual devices configured and that they have the same layout as the R2 standard volumes.
symdev list -sid <symm serial #> -vdev -meta

[root@mdw gpadmin]# symdev list -sid 452 -vdev -meta Symmetrix ID: 000192604452 Device Name Device Meta Information ---------------------------- ---------------------- --------------------------Sym Physical Config Attr Config Stripe # of Cap Size Devs (MB) ---------------------------- ---------------------- --------------------------04DF 04EB 04F7 0503 050F 051B 0527 0533 053F 054B 0557 0563 Not Not Not Not Not Not Not Not Not Not Not Not Visible Visible Visible Visible Visible Visible Visible Visible Visible Visible Visible Visible VDEV VDEV VDEV VDEV VDEV VDEV VDEV VDEV VDEV VDEV VDEV VDEV N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 12 12 12 12 12 12 12 12 12 12 12 12 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015
78
. Data cut due to output length 071F Not Visible VDEV 072B Not Visible VDEV 0737 Not Visible VDEV 0743 Not Visible VDEV 074F Not Visible VDEV 075B Not Visible VDEV
N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd
Striped Striped Striped Striped Striped Striped
960k 960k 960k 960k 960k 960k
12 12 12 12 12 12
2955015 2955015 2955015 2955015 2955015 2955015
4. Create a device group to maintain the R2 standard devices and the virtual devices that will be used for the snap session.
symdg create <device group name> -type RDF2

[root@smdw ~]# symdg create DCA2880_RDF2_SNAP_1 -type RDF2 [root@smdw ~]#
5. Add all of the standard R2 devices to the newly created device group.
symdg -g <device group name> addall -sid <symm serial #> -devs <symdev>:<symdev>

[root@smdw ~]# symdg -g DCA2880_RDF2_SNAP_1 addall -sid 452 -devs 0411:04D1 [root@smdw ~]#
6. Verify the devices were added to the newly created device group.
symdg show <device group name>

[root@smdw config]# symdg show DCA2880_RDF2_SNAP_1 Group Name: DCA2880_RDF2_SNAP_1 : : : : : : : RDF2 (RDFA) No Yes 000192604452 Mon Jun 27 11:16:49 2011 EMC Corp SYMCLI 17 0 0 0 0 0 0 0
79
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number Number Number Number Number Number Number Number of of of of of of of of
STD Devices in Group : Associated GK's : Locally-associated BCV's : Locally-associated VDEV's : Locally-associated TGT's : Remotely-associated VDEV's(STD RDF): Remotely-associated BCV's (STD RDF): Remotely-associated TGT's(TGT RDF) :
Number of Number of Number of Number of Number of Number of Composite
Remotely-associated BCV's (BCV RDF): 0 Remotely-assoc'd RBCV's (RBCV RDF) : 0 Remotely-assoc'd BCV's (Hop-2 BCV) : 0 Remotely-assoc'd VDEV's(Hop-2 VDEV): 0 Remotely-assoc'd TGT's (Hop-2 TGT) : 0 Composite Groups : 0 Group Names : N/A
Standard (STD) Devices (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015 DEV015 N/A 04B9 RDF2+TDEV (M) RW 2955015 DEV016 N/A 04C5 RDF2+TDEV (M) RW 2955015 DEV017 N/A 04D1 RDF2+TDEV (M) RW 2955015 } Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode : : : : False False False False
: Normal : False
80
RDF RDF RDF RDF
Mode Adaptive Copy Adaptive Copy Write Pending State Adaptive Copy Skew (Tracks)
: : : :
Synchronous Disabled N/A 32767
RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: Disabled : : : : Fibre Disabled Enabled Enabled (RW) (RW) (NR)
: Ready
Device RA Status : Ready Device Link Status : Not Ready Time of Last Device Link Status Change : N/A Device Suspend State Device Consistency State Device Consistency Exempt State RDF R2 Not Ready If Invalid Device Write Pacing Exempt State Effective Write Pacing Exempt State Device RDF State Remote Device RDF State RDF Pair State ( R1 <=\=> R2 ) : : : : : : Offline Enabled Disabled Disabled Disabled Disabled
: Ready : Ready : Split : : 106 0
(RW) (RW)
Number of R1 Invalid Tracks Number of R2 Invalid Tracks RDFA Information: { Session Number Cycle Number Number of Devices in the Session Session Status Consistency Exempt Devices Write Pacing Exempt Devices Session Consistency State Minimum Cycle Time Average Cycle Time Duration of Last cycle Session Priority
: : : : : : : : : : :
0 0 17 Inactive No No N/A 00:00:15 00:00:00 00:00:00 33
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:39:24 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True
81
R1 Side Percent Cache In Use R2 Side Percent Cache In Use Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]#
: 0 : 0 : : : : 00:00:00 0 0 0
7. Add the virtual devices that will be used for the snaps created against the R2 devices to the device group.
symdg -g <device group name> addall -sid <symm serial #> -devs <symm dev>:<symm dev> -vdev

[root@smdw config]# symdg -g DCA2880_RDF2_SNAP_1 addall -sid 452 -devs 04DF:059F -vdev [root@smdw config]#
8. Verify the virtual devices were added to the device group.


[root@smdw config]# symdg show DCA2880_RDF2_SNAP_1 Group Name: DCA2880_RDF2_SNAP_1 : : : : : : : RDF2 (RDFA) No Yes 000192604452 Mon Jun 27 11:16:49 2011 EMC Corp SYMCLI 17 0 0 17 0 0 0 0 0 0 0 0 0
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number Number Number Number Number Number Number Number Number Number Number Number Number of of of of of of of of of of of of of
STD Devices in Group : Associated GK's : Locally-associated BCV's : Locally-associated VDEV's : Locally-associated TGT's : Remotely-associated VDEV's(STD RDF): Remotely-associated BCV's (STD RDF): Remotely-associated TGT's(TGT RDF) : Remotely-associated BCV's (BCV RDF): Remotely-assoc'd RBCV's (RBCV RDF) : Remotely-assoc'd BCV's (Hop-2 BCV) : Remotely-assoc'd VDEV's(Hop-2 VDEV): Remotely-assoc'd TGT's (Hop-2 TGT) :
82
Number of Composite Groups Composite Group Names Standard (STD) Devices (17): {
: 0 : N/A
-------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015 DEV015 N/A 04B9 RDF2+TDEV (M) RW 2955015 DEV016 N/A 04C5 RDF2+TDEV (M) RW 2955015 DEV017 N/A 04D1 RDF2+TDEV (M) RW 2955015 } VDEV Devices Locally-associated (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------VDEV001 N/A 04DF VDEV (M) NR 2955015 VDEV002 N/A 04EB VDEV (M) NR 2955015 VDEV003 N/A 04F7 VDEV (M) NR 2955015 VDEV004 N/A 0503 VDEV (M) NR 2955015 VDEV005 N/A 050F VDEV (M) NR 2955015 VDEV006 N/A 051B VDEV (M) NR 2955015 VDEV007 N/A 0527 VDEV (M) NR 2955015 VDEV008 N/A 0533 VDEV (M) NR 2955015 VDEV009 N/A 053F VDEV (M) NR 2955015 VDEV010 N/A 054B VDEV (M) NR 2955015 VDEV011 N/A 0557 VDEV (M) NR 2955015 VDEV012 N/A 0563 VDEV (M) NR 2955015 VDEV013 N/A 056F VDEV (M) NR 2955015 VDEV014 N/A 057B VDEV (M) NR 2955015
83
VDEV015 VDEV016 VDEV017 }
N/A N/A N/A
0587 VDEV 0593 VDEV 059F VDEV
(M) NR (M) NR (M) NR
2955015 2955015 2955015
Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode RDF RDF RDF RDF Mode Adaptive Copy Adaptive Copy Write Pending State Adaptive Copy Skew (Tracks) : : : : False False False False
: Normal : False : : : : Synchronous Disabled N/A 32767
RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: Disabled : : : : Fibre Disabled Enabled Enabled (RW) (RW) (NR)
: Ready
: Ready : Ready : Split
(RW) (RW)
84
: :
106 0
: : : : : : : : : : :
0 0 17 Inactive No No N/A 00:00:15 00:00:00 00:00:00 33
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:46:58 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True R1 Side Percent Cache In Use : 0 R2 Side Percent Cache In Use : 0 Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]# : : : : 00:00:00 0 0 0
9. Create a snap using the created device group.

symsnap -g <device group name> create -svp <save device pool name> -noprompt
If the save device pool is the 'DEFAULT_POOL' then the -svp <save device pool name> parameters can be dropped as the 'DEFAULT_POOL' name is the default.

85

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_1 create -svp R2_SNAP_POOL -noprompt 'Create' operation execution is in progress for device group 'DCA2880_RDF2_SNAP_1'. Please wait... 'Create' operation successfully executed for device group 'DCA2880_RDF2_SNAP_1'. [root@smdw ~]#
10. Query the device group to verify the snap was created.
symsnap -g <device group name> query

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_1 query Device Group (DG) Name: DCA2880_RDF2_SNAP_1 DG's Type : RDF2 DG's Symmetrix ID : 000192604452
Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s) Legend:
86
0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1
47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264
VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
04DF 04EB 04F7 0503 050F 051B 0527 0533 053F 054B 0557 0563 056F 057B 0587 0593 059F
X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Created Created Created Created Created Created Created Created Created Created Created Created Created Created Created Created Created
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
(G): X . (D): X M
The Target device is associated with this group, The Target device is not associated with this group. The Target device has one or more inactive duplicates. The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates.
= = = =
[root@smdw ~]#
11. Activate the snap so it is available for mounting on a host or set of hosts. This puts the virtual devices in a "copy on write" state so that any changes to the R2 that the snap was taken of or the virtual devices used for the snap are tracked. To make sure that all devices are activated at the same time,use the consistent option with the activate command.
symsnap -g <device group name> activate -consistent -noprompt

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_1 activate -consistent -noprompt 'Activate' operation execution is in progress for device group 'DCA2880_RDF2_SNAP_1'. Please wait... 'Activate' operation successfully executed for device group 'DCA2880_RDF2_SNAP_1'. [root@smdw ~]#
12. Query the device group to verify the snap was activated. The devices should now be in a "CopyOnWrite" state.


87
Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s) Legend: (G): X . (D): X M The Target device is associated with this group, The Target device is not associated with this group. The Target device has one or more inactive duplicates. The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates. = = = = 0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264 VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017 04DF 04EB 04F7 0503 050F 051B 0527 0533 053F 054B 0557 0563 056F 057B 0587 0593 059F X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[root@smdw ~]#
13. Mount the snap to another host or set of hosts make sure the virtual devices used for the snap are visible to the host either using symaccess commands or another means. Once the devices are assigned to the host(s) performs a rescan on the hosts and mounts them to the appropriate locations. 14. Recreate the snap session so it picks up all changes from the R2 and starts tracking changes from this new point.
88
Make sure if the snap was mounted that it is unmounted before recreating the snap.
symsnap -g <device group name> recreate -noprompt

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_1 recreate -noprompt 'Recreate' operation execution is in progress for device group 'DCA2880_RDF2_SNAP_1'. Please wait... 'Recreate' operation successfully initiated for device group 'DCA2880_RDF2_SNAP_1'. [root@smdw ~]#
15. Query the device group to verify the snap was recreated.

Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017 04DF 04EB 04F7 0503 050F 051B 0527 0533 053F 054B 0557 0563 056F 057B 0587 0593 059F X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

89
Total Track(s) MB(s) Legend: (G): X . (D): X M
-------803764080 50235264
---------0 0
= = = =
[root@smdw ~]#
16. Terminate the snap session.

symsnap -g <device group name> terminate

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_1 terminate Execute 'Terminate' operation for device group 'DCA2880_RDF2_SNAP_1' (y/[n]) ? y 'Terminate' operation execution is in progress for device group 'DCA2880_RDF2_SNAP_1'. Please wait... 'Terminate' operation successfully executed for device group 'DCA2880_RDF2_SNAP_1'. [root@smdw ~]#
17. Query the device group to verify the snap was destroyed.

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_1 query The Source device and the Target device do not form a Copy session
Device group 'DCA2880_RDF2_SNAP_1' does not have any devices that are Snap source devices [root@smdw ~]#
90
Creating multiple SNAPS of R2 standard devices

To create multiple SNAPs of R2 standard devices, complete the following steps: 1. Modify the SYMAPI options file and enable two SYMAPI parameters that allow the ability to control multiple snaps/clones independently. The options file is located in /var/symapi/config/options on the host(s) where Solutions Enabler is loaded. Make sure to modify the options file where the snap operations are executed from. a. Enable SYMAPI_COMMAND_SCOPE When set to enable, this parameter limits actions performed on a specific device or composite group to that specific group and takes no action on paired TimeFinder devices outside the group. For easy maintenance of multiple snaps the instructions will place the source devices into two separate groups and each group will maintain their own virtual devices. Without this option enabled, any snap creation, activation or termination will occur against both groups. Rather than having two separate point-in-time snaps of the data for two separate time periods, with this option disabled the snap would create two copies of the data for the same time period. Example of parameter enabled: SYMAPI_COMMAND_SCOPE = ENABLE b. Enable SYMAPI_ALLOW_DEV_IN_MULT_GRPS When set to enable, this parameter allows a device (source, virtual, clone, bcv) to exist in more than one device or composite group. Example of parameter enabled: SYMAPI_ALLOW_DEV_IN_MULT_GRPS = ENABLE 2. If there is a potential for more than 16 snaps being created against a single source, modify the root users .bash_profile file and add an additional SYMCLI environment variable SYMCLI_MULTI_VIRTUAL_SNAP and set it to ENABLED.

91
With this environment variable set, Solutions Enabler will allow up to 128 snaps for a single source. The root users .bash_profile file is located in root's home directory. Typically, /root/.bash_profile a. Enable SYMCLI_MULTI_VIRTUAL_SNAP by adding the following lines in the .bash_profile file of the user that will conduct the snap operations. Typically root. SYMCLI_MULTI_VIRTUAL_SNAP=ENABLED export SYMCLI_MULTI_VIRTUAL_SNAP b. Source the .bash_profile file after saving the changes to pick up the new environment parameters. source .bash_profile 3. Verify that the Symmetrix where the snap will be conducted has the proper amount of snap save devices configured for the number of snap sessions that will be created and the save devices are assigned to a snap pool.
symsnap list -sid <symm serial #> -savedevs

[root@smdw ~]# symsnap list -sid 452 -savedevs Symmetrix ID: 000192604452 S N A P S A V E D E V I C E S --------------------------------------------------------------------Device SaveDevice Total Used Free Full Sym Emulation Pool Name Tracks Tracks Tracks (%) --------------------------------------------------------------------0339 FBA R2_SNAP_POOL 3939540 0 3939540 0 033A FBA R2_SNAP_POOL 3939540 0 3939540 0 033B FBA R2_SNAP_POOL 3939540 0 3939540 0 033C FBA R2_SNAP_POOL 3939540 0 3939540 0 033D FBA R2_SNAP_POOL 3939540 0 3939540 0 033E FBA R2_SNAP_POOL 3939540 0 3939540 0 033F FBA R2_SNAP_POOL 3939540 0 3939540 0 0340 FBA R2_SNAP_POOL 3939540 0 3939540 0 0341 FBA R2_SNAP_POOL 3939540 0 3939540 0 0342 FBA R2_SNAP_POOL 3939540 0 3939540 0 0343 FBA R2_SNAP_POOL 3939540 0 3939540 0 0344 FBA R2_SNAP_POOL 3939540 0 3939540 0 0345 FBA R2_SNAP_POOL 3939540 0 3939540 0 .. Data cut due to output length 03CE FBA R2_SNAP_POOL 3939540 0 3939540 0 03CF FBA R2_SNAP_POOL 3939540 0 3939540 0
92
Total Tracks MB(s) [root@smdw ~]#
--------- --------- --------594870540 0 594870540 37179428 0 37179428
---0
4. To list available save device pools (snap pools) and verify the state of the pool, make sure the pool that will be used is Enabled.
symsnap -sid <symm serial #> -pools list

[root@smdw ~]# symsnap -sid 452 -pools list Symmetrix ID: 000192604452
S A V E D E V I C E P O O L S -------------------------------------------------------------------------------Dev Total Enabled Used Free Full Pool Session Pool Name Emul Tracks Tracks Tracks Tracks (%) State Status -------------------------------------------------------------------------------DEFAULT_POOL FBA 0 0 0 0 0 Disabled Inactive DEFAULT_POOL 3390 0 0 0 0 0 Disabled Inactive DEFAULT_POOL 3380 0 0 0 0 0 Disabled Inactive DEFAULT_POOL AS400 0 0 0 0 0 Disabled Inactive R2_SNAP_POOL FBA 594870540 594870540 0 594870540 0 Enabled Inactive [root@smdw ~]#
5. Verify that the Symmetrix where the snap will be conducted has the proper amount of virtual devices configured for the number of snap sessions that will be created and that they have the same layout as the R2 standard volumes.
symdev list -sid <symm serial #> -vdev -meta

[root@mdw gpadmin]# symdev list -sid 452 -vdev -meta Symmetrix ID: 000192604452
Device Name Device Meta Information ---------------------------- ---------------------- --------------------------Sym Physical Config Attr Config Stripe # of Cap Size Devs (MB) ---------------------------- ---------------------- --------------------------04DF Not Visible 04EB Not Visible VDEV VDEV N/Asst'd Striped N/Asst'd Striped 960k 960k 12 12 2955015 2955015

93
04F7 Not Visible VDEV 0503 Not Visible VDEV 050F Not Visible VDEV 051B Not Visible VDEV 0527 Not Visible VDEV 0533 Not Visible VDEV 053F Not Visible VDEV 054B Not Visible VDEV 0557 Not Visible VDEV 0563 Not Visible VDEV . Data cut due to output length 071F Not Visible VDEV 072B Not Visible VDEV 0737 Not Visible VDEV 0743 Not Visible VDEV 074F Not Visible VDEV 075B Not Visible VDEV [root@mdw gpadmin]#
N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd
Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped
960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k
12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015
6. Create a device group for the first snap session.


7. Add the R2 standard devices to device group 1 that was just created.
symdg -g <device group name> addall -sid <symm serial #> -devs <start dev>:<end dev>

8. Add the virtual devices to the device group 1 that will be used for snap session 1.
symdg -g <device group name> addall -sid 452 -devs <start dev>:<end dev> -vdev

[root@smdw config]# symdg -g DCA2880_RDF2_SNAP_2 addall -sid 452 -devs 05AB:066B -vdev [root@smdw config]#
9. Display the device group to verify the standard devices and the virtual devices were correctly added to the group.
94

[root@smdw config]# symdg show DCA2880_RDF2_SNAP_1 Group Name: DCA2880_RDF2_SNAP_1 : : : : : : : RDF2 (RDFA) No Yes 000192604452 Mon Jun 27 11:16:49 2011 EMC Corp SYMCLI
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Composite
STD Devices in Group : 17 Associated GK's : 0 Locally-associated BCV's : 0 Locally-associated VDEV's : 17 Locally-associated TGT's : 0 Remotely-associated VDEV's(STD RDF): 0 Remotely-associated BCV's (STD RDF): 0 Remotely-associated TGT's(TGT RDF) : 0 Remotely-associated BCV's (BCV RDF): 0 Remotely-assoc'd RBCV's (RBCV RDF) : 0 Remotely-assoc'd BCV's (Hop-2 BCV) : 0 Remotely-assoc'd VDEV's(Hop-2 VDEV): 0 Remotely-assoc'd TGT's (Hop-2 TGT) : 0 Composite Groups : 0 Group Names : N/A
Standard (STD) Devices (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015
95
DEV015 DEV016 DEV017 }
N/A N/A N/A
04B9 RDF2+TDEV 04C5 RDF2+TDEV 04D1 RDF2+TDEV
(M) RW (M) RW (M) RW
2955015 2955015 2955015
VDEV Devices Locally-associated (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------VDEV001 N/A 04DF VDEV (M) NR 2955015 VDEV002 N/A 04EB VDEV (M) NR 2955015 VDEV003 N/A 04F7 VDEV (M) NR 2955015 VDEV004 N/A 0503 VDEV (M) NR 2955015 VDEV005 N/A 050F VDEV (M) NR 2955015 VDEV006 N/A 051B VDEV (M) NR 2955015 VDEV007 N/A 0527 VDEV (M) NR 2955015 VDEV008 N/A 0533 VDEV (M) NR 2955015 VDEV009 N/A 053F VDEV (M) NR 2955015 VDEV010 N/A 054B VDEV (M) NR 2955015 VDEV011 N/A 0557 VDEV (M) NR 2955015 VDEV012 N/A 0563 VDEV (M) NR 2955015 VDEV013 N/A 056F VDEV (M) NR 2955015 VDEV014 N/A 057B VDEV (M) NR 2955015 VDEV015 N/A 0587 VDEV (M) NR 2955015 VDEV016 N/A 0593 VDEV (M) NR 2955015 VDEV017 N/A 059F VDEV (M) NR 2955015 } Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode RDF Mode RDF Adaptive Copy RDF Adaptive Copy Write Pending State : : : : False False False False
: Normal : False : Synchronous : Disabled : N/A
96
RDF Adaptive Copy Skew (Tracks) RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: 32767 : Disabled : : : : Fibre Disabled Enabled Enabled (RW) (RW) (NR)
: Ready
(RW) (RW)
: : : : : : : : : : :
0 0 17 Inactive No No N/A 00:00:15 00:00:00 00:00:00 33
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:46:58 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True R1 Side Percent Cache In Use : 0 R2 Side Percent Cache In Use : 0

97
Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]#
: : : :
00:00:00 0 0 0
10. Create a second device group for the second snap session.

11. )Add the same R2 standard devices that were added to device group 1 to device group 2 that was just created.
IMPORTANT If the SYMAPI environment parameters were not enabled from step 1 of this section this step will fail since by default a device can only be in one device group at a time by default.
symdg -g <device group name> addall -sid <symm serial #> -devs <start dev>:<end dev>

12. Add the virtual devices to device group 2 that will be used for the second snap session.
Note: These should not be the same virtual devices added to device group 1 in step 8 of this section.
symdg -g <device group name> addall -sid <symm serial #> -devs <start dev>:<end dev> -vdev

[root@smdw config]# symdg -g DCA2880_RDF2_SNAP_1 addall -sid 452 -devs 04DF:059F -vdev [root@smdw config]#
13. Display the device group to verify the R2 standard devices and virtual devices were added correctly to the group
98

[root@smdw config]# symdg show DCA2880_RDF2_SNAP_2 Group Name: DCA2880_RDF2_SNAP_2 : : : : : : : RDF2 (RDFA) No Yes 000192604452 Mon Jun 27 11:17:07 2011 EMC Corp SYMCLI
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Composite
STD Devices in Group : 17 Associated GK's : 0 Locally-associated BCV's : 0 Locally-associated VDEV's : 17 Locally-associated TGT's : 0 Remotely-associated VDEV's(STD RDF): 0 Remotely-associated BCV's (STD RDF): 0 Remotely-associated TGT's(TGT RDF) : 0 Remotely-associated BCV's (BCV RDF): 0 Remotely-assoc'd RBCV's (RBCV RDF) : 0 Remotely-assoc'd BCV's (Hop-2 BCV) : 0 Remotely-assoc'd VDEV's(Hop-2 VDEV): 0 Remotely-assoc'd TGT's (Hop-2 TGT) : 0 Composite Groups : 0 Group Names : N/A
Standard (STD) Devices (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015
99
DEV015 DEV016 DEV017 }
N/A N/A N/A
04B9 RDF2+TDEV 04C5 RDF2+TDEV 04D1 RDF2+TDEV
(M) RW (M) RW (M) RW
2955015 2955015 2955015
VDEV Devices Locally-associated (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------VDEV001 N/A 05AB VDEV (M) NR 2955015 VDEV002 N/A 05B7 VDEV (M) NR 2955015 VDEV003 N/A 05C3 VDEV (M) NR 2955015 VDEV004 N/A 05CF VDEV (M) NR 2955015 VDEV005 N/A 05DB VDEV (M) NR 2955015 VDEV006 N/A 05E7 VDEV (M) NR 2955015 VDEV007 N/A 05F3 VDEV (M) NR 2955015 VDEV008 N/A 05FF VDEV (M) NR 2955015 VDEV009 N/A 060B VDEV (M) NR 2955015 VDEV010 N/A 0617 VDEV (M) NR 2955015 VDEV011 N/A 0623 VDEV (M) NR 2955015 VDEV012 N/A 062F VDEV (M) NR 2955015 VDEV013 N/A 063B VDEV (M) NR 2955015 VDEV014 N/A 0647 VDEV (M) NR 2955015 VDEV015 N/A 0653 VDEV (M) NR 2955015 VDEV016 N/A 065F VDEV (M) NR 2955015 VDEV017 N/A 066B VDEV (M) NR 2955015 } Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode RDF Mode RDF Adaptive Copy RDF Adaptive Copy Write Pending State : : : : False False False False
: Normal : False : Synchronous : Disabled : N/A
100
RDF Adaptive Copy Skew (Tracks) RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: 32767 : Disabled : : : : Fibre Disabled Enabled Enabled (RW) (RW) (NR)
: Ready
(RW) (RW)
: : : : : : : : : : :
0 0 17 Inactive No No N/A 00:00:15 00:00:00 00:00:00 33
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:47:05 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True R1 Side Percent Cache In Use : 0 R2 Side Percent Cache In Use : 0

101
Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]#
: : : :
00:00:00 0 0 0
14. Repeat steps 10 - 13 for any additional snap sessions that are needed. 15. Follow steps 9 - 17 from Creating a single SNAP of R2 standard devices on page 77 to perform snap operations on any of the newly created device groups. Each device group will be treated independently from the others, so it is possible to generate rolling snaps of the Greenplum database to maintain multiple point-in-time copies.
102
Invisible Body Tag
H
Mounting and checking Symmetrix SNAP of Greenplum database ..... 104
103
Mounting and checking Symmetrix SNAP of Greenplum database

This procedure assumes that TimeFinder/Snap is configured and that a snap has been taken of the primary DCA's volumes with symsnap create. or detailed information about using TimeFinder/Snap, see Creating SNAPs from the R2 Volumes for Disaster Recovery documentation, located on EMC Powerlink. To mount and check a Symmetrix SNAP of the Greenplum database, complete the following steps: 1. Activate the snap that was created.
symsnap -g <device group name> activate -consistent
Sample of command output:

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_2 activate -consistent Execute 'Activate' operation for device group 'DCA2880_RDF2_SNAP_2' (y/[n]) ? y 'Activate' operation execution is in progress for device group 'DCA2880_RDF2_SNAP_2'. Please wait... 'Activate' operation successfully executed for device group 'DCA2880_RDF2_SNAP_2'. [root@smdw ~]#
2. Query the device group to verify the snap was activated. Devices should report that they are in "CopyOnWrite" state.

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_2 query Device Group (DG) Name: DCA2880_RDF2_SNAP_2 DG's Type : RDF2 DG's Symmetrix ID : 000192604452 Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ----
104
DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s)
0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1
47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264
VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
05AB 05B7 05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B
X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Legend: (G): X = The Target device is associated with this group, . = The Target device is not associated with this group. (D): X = The Target device has one or more inactive duplicates. M = The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates.
3. Assign the SNAP virtual devices to the appropriate storage groups for the DCA nodes where the devices need to be mounted. a. Gather information from each of the DCA nodes to identify which system is using which R1 device so that they can be mapped to the R2 and subsequently to the virtual device used for the SNAP.
gpssh -f <host file> 'powermt display dev=all |egrep "Pseudo|Logical"'
b. Run a query against the SRDF device group to get the device R1/R2 mapping to compare with the SNAP device group to identify how the devices are paired
symrdf cg <composite group> query
c. Run a query against the SNAP device group to map the R2 to virtual device used for the snap
symsnap g <device group name> query

105
d. Add the devices to the appropriate storage group based on the mapping identified in steps a c.
symaccess sid <symm serial #> -type storage name <sg name> add devs <symdev>,<symdev>

[root@mdw ~]# gpssh -f gp-hosts_1-8 'powermt display dev=all |egrep "Pseudo|Logical"' [sdw4] Pseudo name=emcpowerf [sdw4] Logical device ID=02BB [sdw4] Pseudo name=emcpowere [sdw4] Logical device ID=02C7 [sdw5] Pseudo name=emcpowerf [sdw5] Logical device ID=02D3 [sdw5] Pseudo name=emcpowere [sdw5] Logical device ID=02DF [sdw6] Pseudo name=emcpowerf [sdw6] Logical device ID=02EB [sdw6] Pseudo name=emcpowere [sdw6] Logical device ID=02F7 [sdw7] Pseudo name=emcpowerf [sdw7] Logical device ID=0303 [sdw7] Pseudo name=emcpowere [sdw7] Logical device ID=030F [sdw1] Pseudo name=emcpowerf [sdw1] Logical device ID=0273 [sdw1] Pseudo name=emcpowere [sdw1] Logical device ID=027F [sdw2] Pseudo name=emcpowerf [sdw2] Logical device ID=028B [sdw2] Pseudo name=emcpowere [sdw2] Logical device ID=0297 [sdw3] Pseudo name=emcpowerf [sdw3] Logical device ID=02A3 [sdw3] Pseudo name=emcpowere [sdw3] Logical device ID=02AF [sdw8] Pseudo name=emcpowerf [sdw8] Logical device ID=031B [sdw8] Pseudo name=emcpowere [sdw8] Logical device ID=0327 [ mdw] Pseudo name=emcpowerd [ mdw] Logical device ID=0267 [root@mdw ~]# [root@mdw ~]# symrdf query -cg DCA_R1_CG Composite Group Name Composite Group Type Number of Symmetrix Units Number of RDF (RA) Groups RDF Consistency Mode : : : : : DCA_R1_CG RDF1 1 1 SYNC
106
Symmetrix ID Remote Symmetrix ID RDF (RA) Group Number
: 000192604467 : 000192604452 : 1 (00)
(Microcode Version: 5875) (Microcode Version: 5875)
Source (R1) View Target (R2) View MODES -------------------------------- ------------------------- ----ST LI ST Standard A N A Logical Sym T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks MDAE -------------------------------- -- ----------------------- ----DEV4467* 0267 RW 0 0 RW 0411 WD 0 0 S... X DEV4467* 0273 RW 0 0 RW 041D WD 0 0 S... X DEV4467* 027F RW 0 0 RW 0429 WD 0 0 S... X DEV4467* 028B RW 0 0 RW 0435 WD 0 0 S... X DEV4467* 0297 RW 0 0 RW 0441 WD 0 0 S... X DEV4467* 02A3 RW 0 0 RW 044D WD 0 0 S... X DEV4467* 02AF RW 0 0 RW 0459 WD 0 0 S... X DEV4467* 02BB RW 0 0 RW 0465 WD 0 0 S... X DEV4467* 02C7 RW 0 0 RW 0471 WD 0 0 S... X DEV4467* 02D3 RW 0 0 RW 047D WD 0 0 S... X DEV4467* 02DF RW 0 0 RW 0489 WD 0 0 S... X DEV4467* 02EB RW 0 0 RW 0495 WD 0 0 S... X DEV4467* 02F7 RW 0 0 RW 04A1 WD 0 0 S... X DEV4467* 0303 RW 0 0 RW 04AD WD 0 0 S... X DEV4467* 030F RW 0 0 RW 04B9 WD 0 0 S... X DEV4467* 031B RW 0 0 RW 04C5 WD 0 0 S... X DEV4467* 0327 RW 0 0 RW 04D1 WD 0 0 S... X Total Track(s) MBs Legend for MODES: M(ode of Operation) : D(omino) : A(daptive Copy) : (Consistency) E(xempt): Legend for STATES: Cons(istency State) Susp(end State) A X D X = = = = Async, S = Enabled, . Disk Mode, Enabled, . ------- ------0 0 0.0 0.0 ------- ------0 0 0.0 0.0
STATES -----C S o u n s RDF Pair s p STATE -----Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized
Sync, E = Semi-sync, C = Adaptive Copy = Disabled W = WP Mode, . = ACp off = Disabled, M = Mixed, - = N/A
: X = Enabled, . = Disabled, M = Mixed, - = N/A : X = Online, . = Offline, P = Offline Pending, - = N/A
[root@mdw ~]# [root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_2 query
Device Group (DG) Name: DCA2880_RDF2_SNAP_2 DG's Type : RDF2 DG's Symmetrix ID : 000192604452
107
Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s) 0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264 VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017 05AB 05B7 05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Legend: (G): X = The Target device is associated with this group, . = The Target device is not associated with this group. (D): X = The Target device has one or more inactive duplicates. M = The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates. [root@smdw ~]# [root@smdw ~]# symaccess -sid 452 list Symmetrix ID : 000192604452 Group Name Type -------------------------------- --------mdw1_ig Initiator sdw1_ig Initiator sdw2_ig Initiator sdw3_ig Initiator sdw4_ig Initiator sdw5_ig Initiator sdw6_ig Initiator
108
sdw7_ig sdw8_ig DCA_PG mdw1_sg sdw1_sg sdw2_sg sdw3_sg sdw4_sg sdw5_sg sdw6_sg sdw7_sg sdw8_sg
Initiator Initiator Port Storage Storage Storage Storage Storage Storage Storage Storage Storage
[root@smdw ~]# symaccess -sid 452 -type storage -name mdw1_sg add devs 05AB [root@smdw ~]# symaccess -sid 452 -type storage -name sdw1_sg add devs 05B7,05C3 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw2_sg add devs 05CF,05DB [root@smdw ~]# symaccess -sid 452 -type storage -name sdw3_sg add devs 05E7,05F3 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw4_sg add devs 05FF,060B [root@smdw ~]# symaccess -sid 452 -type storage -name sdw5_sg add devs 0617,0623 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw6_sg add devs 062F,063B [root@smdw ~]# symaccess -sid 452 -type storage -name sdw7_sg add devs 0647,0653 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw8_sg add devs 065F,066B [root@smdw ~]#
4. Force the new DCA nodes where the SNAP will be mounted to probe their buses to find the newly added devices. The command provided when used with gpssh will successfully probe all required buses on the DCA. Some errors will occur but can be ignored because the host buses 3 and 4 do not exist on the segment servers. They only exist on the master and standby master. For i in 0 1 2 3 4, do
echo - - - > /sys/class/scsi_host/host$i/scan; done

[root@smdw ~]# gpssh -f my_gp-hosts_1-8 'for i in 0 1 /sys/class/scsi_host/host$i/scan; done' [sdw8] bash: /sys/class/scsi_host/host3/scan: No such [sdw8] bash: /sys/class/scsi_host/host4/scan: No such [sdw6] bash: /sys/class/scsi_host/host3/scan: No such [sdw6] bash: /sys/class/scsi_host/host4/scan: No such [sdw7] bash: /sys/class/scsi_host/host3/scan: No such [sdw7] bash: /sys/class/scsi_host/host4/scan: No such [sdw4] bash: /sys/class/scsi_host/host3/scan: No such [sdw4] bash: /sys/class/scsi_host/host4/scan: No such [sdw5] bash: /sys/class/scsi_host/host3/scan: No such [sdw5] bash: /sys/class/scsi_host/host4/scan: No such [sdw2] bash: /sys/class/scsi_host/host3/scan: No such 2 3 4; do echo - - - > file file file file file file file file file file file or or or or or or or or or or or directory directory directory directory directory directory directory directory directory directory directory
109
[sdw2] bash: /sys/class/scsi_host/host4/scan: [sdw3] bash: /sys/class/scsi_host/host3/scan: [sdw3] bash: /sys/class/scsi_host/host4/scan: [ mdw] [sdw1] bash: /sys/class/scsi_host/host3/scan: [sdw1] bash: /sys/class/scsi_host/host4/scan: [root@smdw ~]#
No such file or directory No such file or directory No such file or directory No such file or directory No such file or directory
5. Issue a powermt config to each of the segment servers to generate power devices for the newly found devices.
powermt config

[root@smdw ~]# gpssh -f my_gp-hosts_1-8 'powermt config' [sdw8] [sdw6] [sdw7] [sdw4] [sdw5] [sdw2] [sdw3] [ mdw] [sdw1] [root@smdw ~]#
6. Verify that the newly added devices are now configured with power devices.
Note: If more than one set of snap devices were assigned to the host, multiple power devices will be seen. powermt display dev=all |egrep "Pseudo|Logical"

[root@smdw ~]# gpssh -f my_gp-hosts_1-8 'powermt display dev=all|egrep "Pseudo|Logical"' [sdw8] Pseudo name=emcpowerh [sdw8] Logical device ID=04C5 [sdw8] Pseudo name=emcpowerg [sdw8] Logical device ID=04D1 [sdw8] Pseudo name=emcpowerf [sdw8] Logical device ID=0593 [sdw8] Pseudo name=emcpowere [sdw8] Logical device ID=059F [sdw8] Pseudo name=emcpowerj [sdw8] Logical device ID=065F [sdw8] Pseudo name=emcpoweri [sdw8] Logical device ID=066B
110
-- Data cut due to output length [root@smdw ~]#
7. Save the power path configuration.

powermt save

[root@smdw ~]# gpssh -f my_gp-hosts_1-8 'powermt save' [sdw8] [sdw6] [sdw7] [sdw4] [sdw5] [sdw2] [sdw3] [ mdw] [sdw1] [root@smdw ~]#
8. Mount the virtual device containing the master database on the master server.
mount /dev/<emcpower device> <mountpoint> -o noatime,inode64,allocsize=16m

[root@smdw ~]# mount /dev/emcpowere /data -o noatime,inode64,allocsize=16m [root@smdw ~]#
9. Mount the appropriate power device(s) on each of the segment servers.

gpssh -h <segment server> 'mount /<emcpower device> <mountpoint> -o noatime,inode64,allocsize=16m; mount /<emcpower device> <mountpoint> -o noatime,inode64,allocsize=16m'

[root@smdw ~]# gpssh -h sdw1 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw1] [root@smdw ~]# gpssh -h sdw2 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw2] [root@smdw ~]# gpssh -h sdw3 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw3] [root@smdw ~]# gpssh -h sdw4 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' /dev/emcpowerj /mirror1 -o /dev/emcpowerk /mirror2 -o
/dev/emcpoweri /mirror1 -o /dev/emcpowerj /mirror2 -o

111
[sdw4] [root@smdw ~]# gpssh -h sdw5 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw5] [root@smdw ~]# gpssh -h sdw6 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw6] [root@smdw ~]# gpssh -h sdw7 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw7] [root@smdw ~]# gpssh -h sdw8 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw8] [root@smdw ~]#
10. Verify that the SNAP devices were mounted successfully.

gpssh -f <host file> 'df -h |egrep "<mountloc>|<mountloc>"'

[root@smdw ~]# gpssh -f my_gp-hosts_1-8 'df -h |egrep "data|mirror"' [sdw8] /dev/sdb 2.7T 204G 2.5T 8% /data1 [sdw8] /dev/sdd 2.7T 204G 2.5T 8% /data2 [sdw8] /dev/emcpoweri 2.9T 2.6G 2.9T 1% /mirror1 [sdw8] /dev/emcpowerj 2.9T 2.6G 2.9T 1% /mirror2 [sdw6] /dev/sdb 2.7T 204G 2.5T 8% /data1 [sdw6] /dev/sdd 2.7T 204G 2.5T 8% /data2 [sdw6] /dev/emcpoweri 2.9T 2.6G 2.9T 1% /mirror1 [sdw6] /dev/emcpowerj 2.9T 2.6G 2.9T 1% /mirror2 [sdw7] /dev/sdb 2.7T 204G 2.5T 8% /data1 [sdw7] /dev/sdd 2.7T 204G 2.5T 8% /data2 [sdw7] /dev/emcpoweri 2.9T 2.6G 2.9T 1% /mirror1 [sdw7] /dev/emcpowerj 2.9T 2.6G 2.9T 1% /mirror2 [sdw4] /dev/sdb 2.7T 204G 2.5T 8% /data1 [sdw4] /dev/sdd 2.7T 204G 2.5T 8% /data2 [sdw4] /dev/emcpoweri 2.9T 2.6G 2.9T 1% /mirror1 [sdw4] /dev/emcpowerj 2.9T 2.6G 2.9T 1% /mirror2 [sdw5] /dev/sdb 2.7T 204G 2.5T 8% /data1 [sdw5] /dev/sdd 2.7T 204G 2.5T 8% /data2 [sdw5] /dev/emcpoweri 2.9T 2.6G 2.9T 1% /mirror1 [sdw5] /dev/emcpowerj 2.9T 2.6G 2.9T 1% /mirror2 [sdw2] /dev/sdb 2.7T 204G 2.5T 8% /data1 [sdw2] /dev/sdd 2.7T 204G 2.5T 8% /data2 [sdw2] /dev/emcpoweri 2.9T 2.6G 2.9T 1% /mirror1 [sdw2] /dev/emcpowerj 2.9T 2.6G 2.9T 1% /mirror2 [sdw3] /dev/sdb 2.7T 204G 2.5T 8% /data1 [sdw3] /dev/sdd 2.7T 204G 2.5T 8% /data2 [sdw3] /dev/emcpoweri 2.9T 2.6G 2.9T 1% /mirror1
112
[sdw3] /dev/emcpowerj [ mdw] /dev/emcpowere [sdw1] /dev/sdb [sdw1] /dev/sdd [sdw1] /dev/emcpowerj [sdw1] /dev/emcpowerk [root@smdw ~]#
2.9T 2.9T 2.7T 2.7T 2.9T 2.9T
2.6G 263M 204G 204G 2.6G 2.6G
2.9T 2.9T 2.5T 2.5T 2.9T 2.9T
1% 1% 8% 8% 1% 1%
/mirror2 /data /data1 /data2 /mirror1 /mirror2
11. Switch to the gpadmin user on the new master server

su - gpadmin

[root@smdw ~]# su - gpadmin [gpadmin@smdw ~]$
12. Start the database in maintenance mode.

gpstart -m

gpadmin@smdw ~]$ gpstart -m 20110630:06:35:54:gpstart:smdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20110630:06:35:54:gpstart:smdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110630:06:35:55:gpstart:smdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110630:06:35:55:gpstart:smdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201101130' 20110630:06:35:55:gpstart:smdw:gpadmin-[WARNING]:-postmaster.pid file exists on Master, checking if recovery startup required 20110630:06:35:55:gpstart:smdw:gpadmin-[INFO]:-Commencing recovery startup checks 20110630:06:35:55:gpstart:smdw:gpadmin-[INFO]:-No socket connection or lock file in /tmp found for port=5432 20110630:06:35:55:gpstart:smdw:gpadmin-[INFO]:-No Master instance process, entering recovery startup mode 20110630:06:35:55:gpstart:smdw:gpadmin-[INFO]:-Clearing Master instance pid file 20110630:06:35:55:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20110630:06:35:56:gpstart:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110630:06:35:56:gpstart:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110630:06:35:57:gpstart:smdw:gpadmin-[INFO]:-Commencing forced instance shutdown 20110630:06:35:58:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20110630:06:35:59:gpstart:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110630:06:35:59:gpstart:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110630:06:36:00:gpstart:smdw:gpadmin-[INFO]:-Master Started... [gpadmin@smdw ~]$

113
13. Connect to the database using the utility role.

PGOPTIONS='-c gp_session_role=utility' psql <database name>

[gpadmin@smdw ~]$ PGOPTIONS='-c gp_session_role=utility' psql srdftesting psql (8.2.15) Type "help" for help.
14. Run a query against the gp_segment_configuration and pg_filespace_entry table spaces and verify that all of mirror segments were up at the time of the snap. If any of the mirror segments were down during the snap then the snap does not contain a consistent copy of the database since the R2 device associated with the mirror segment that was down was not updated from the primary segment and therefore does not contain the latest data.
select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | p | p | s | u | sdw1 | /data1/primary/gpseg0 3 | 1 | p | p | s | u | sdw1 | /data1/primary/gpseg1 4 | 2 | p | p | s | u | sdw1 | /data1/primary/gpseg2 5 | 3 | p | p | s | u | sdw1 | /data2/primary/gpseg3 6 | 4 | p | p | s | u | sdw1 | /data2/primary/gpseg4 7 | 5 | p | p | s | u | sdw1 | /data2/primary/gpseg5 8 | 6 | p | p | s | u | sdw2 | /data1/primary/gpseg6 9 | 7 | p | p | s | u | sdw2 | /data1/primary/gpseg7 10 | 8 | p | p | s | u | sdw2 | /data1/primary/gpseg8 11 | 9 | p | p | s | u | sdw2 | /data2/primary/gpseg9 12 | 10 | p | p | s | u | sdw2 | /data2/primary/gpseg10 13 | 11 | p | p | s | u | sdw2 | /data2/primary/gpseg11 14 | 12 | p | p | s | u | sdw3 | /data1/primary/gpseg12 15 | 13 | p | p | s | u | sdw3 | /data1/primary/gpseg13 16 | 14 | p | p | s | u | sdw3 | /data1/primary/gpseg14 17 | 15 | p | p | s | u | sdw3 | /data2/primary/gpseg15 18 | 16 | p | p | s | u | sdw3 | /data2/primary/gpseg16
114
19 |
17 | p
| p
| s
| u
| sdw3
| /data2/primary/gpseg17
-- Data cut due to output length 94 | 95 | 96 | 97 | (97 rows) 44 45 46 47 | | | | m m m m | | | | m m m m | | | | s s s s | | | | u u u u | | | | sdw3 sdw4 sdw5 sdw6 | | | | /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
15. If all of the mirror segments were up and synchronized at the time of the snap, bring the database online on the recovery DCA using only the mirror segments. Update the gp_segment_configuration table space and mark the primary segment roles as 'mirror' and the status as 'down'.
update gp_segment_configuration sc set role='m',status='d' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;

srdftesting=# update gp_segment_configuration sc set role='m',status='d' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/data_/%' and sc.content >= 0; UPDATE 48 srdftesting=#
Sample output of command after running the update statement:

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1 4 | 2 | m | p | s | d | sdw1 | /data1/primary/gpseg2 5 | 3 | m | p | s | d | sdw1 | /data2/primary/gpseg3 6 | 4 | m | p | s | d | sdw1 | /data2/primary/gpseg4 7 | 5 | m | p | s | d | sdw1 | /data2/primary/gpseg5 8 | 6 | m | p | s | d | sdw2 | /data1/primary/gpseg6 9 | 7 | m | p | s | d | sdw2 | /data1/primary/gpseg7 10 | 8 | m | p | s | d | sdw2 | /data1/primary/gpseg8 11 | 9 | m | p | s | d | sdw2 | /data2/primary/gpseg9 12 | 10 | m | p | s | d | sdw2 | /data2/primary/gpseg10 13 | 11 | m | p | s | d | sdw2 | /data2/primary/gpseg11
115
14 15 16 17 18 19
| | | | | |
12 13 14 15 16 17
| | | | | |
m m m m m m
| | | | | |
p p p p p p
| | | | | |
s s s s s s
| | | | | |
d d d d d d
| | | | | |
sdw3 sdw3 sdw3 sdw3 sdw3 sdw3
| | | | | |
/data1/primary/gpseg12 /data1/primary/gpseg13 /data1/primary/gpseg14 /data2/primary/gpseg15 /data2/primary/gpseg16 /data2/primary/gpseg17
-- Data cut due to output length 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | (97 rows) 40 41 42 43 44 45 46 47 | | | | | | | | m m m m m m m m | | | | | | | | m m m m m m m m | | | | | | | | s s s s s s s s | | | | | | | | u u u u u u u u | | | | | | | | sdw4 sdw5 sdw1 sdw2 sdw3 sdw4 sdw5 sdw6 | | | | | | | | /mirror1/mirror/gpseg40 /mirror1/mirror/gpseg41 /mirror2/mirror/gpseg42 /mirror2/mirror/gpseg43 /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
16. Update the gp_segment_configuration table space and set the mirror segments to a role of 'primary' and a mode of 'change tracking'.
update gp_segment_configuration sc set role='p',mode='c' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;

srdftesting=# update gp_segment_configuration sc set role='p',mode='c' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/mirror_/%' and sc.content >= 0; UPDATE 48 srdftesting=#
Sample output of command after running the update statement:

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1 4 | 2 | m | p | s | d | sdw1 | /data1/primary/gpseg2 5 | 3 | m | p | s | d | sdw1 | /data2/primary/gpseg3 6 | 4 | m | p | s | d | sdw1 | /data2/primary/gpseg4 7 | 5 | m | p | s | d | sdw1 | /data2/primary/gpseg5 8 | 6 | m | p | s | d | sdw2 | /data1/primary/gpseg6
116
9 10 11 12 13 14 15 16 17 18 19 20
| | | | | | | | | | | |
7 8 9 10 11 12 13 14 15 16 17 18
| | | | | | | | | | | |
m m m m m m m m m m m m
| | | | | | | | | | | |
p p p p p p p p p p p p
| s | s | s | s | s | s | s | s | s | s | s | s
| d | d | d | d | d | d | d | d | d | d | d | d
| sdw2 | sdw2 | sdw2 | sdw2 | sdw2 | sdw3 | sdw3 | sdw3 | sdw3 | sdw3 | sdw3 | sdw4
| /data1/primary/gpseg7 | /data1/primary/gpseg8 | /data2/primary/gpseg9 | /data2/primary/gpseg10 | /data2/primary/gpseg11 | /data1/primary/gpseg12 | /data1/primary/gpseg13 | /data1/primary/gpseg14 | /data2/primary/gpseg15 | /data2/primary/gpseg16 | /data2/primary/gpseg17 | /data1/primary/gpseg18
-- Data cut due to output length 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | (97 rows) 40 41 42 43 44 45 46 47 | | | | | | | | p p p p p p p p | | | | | | | | m m m m m m m m | | | | | | | | c c c c c c c c | | | | | | | | u u u u u u u u | | | | | | | | sdw4 sdw5 sdw1 sdw2 sdw3 sdw4 sdw5 sdw6 | | | | | | | | /mirror1/mirror/gpseg40 /mirror1/mirror/gpseg41 /mirror2/mirror/gpseg42 /mirror2/mirror/gpseg43 /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
17. Exit out of the psql session and back to the command prompt.
q

srdftesting=# \q [gpadmin@smdw ~]$
18. Stop the database master server so it is removed from maintenance mode.
gpstop -m

[gpadmin@smdw ~]$ gpstop -m 20110701:07:20:41:gpstop:smdw:gpadmin-[INFO]:-Starting gpstop with args: -m 20110701:07:20:41:gpstop:smdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110701:07:20:41:gpstop:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110701:07:20:41:gpstop:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-There are 0 connections to the database
117
20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Master host=mdw 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1 [gpadmin@smdw ~]$
19. Restart the database normally. Watch to make sure that the primary segments were skipped and only the mirror segments are started.
gpstart

20110701:07:20:54:gpstart:smdw:gpadmin-[INFO]:-Starting gpstart with args: 20110701:07:20:54:gpstart:smdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110701:07:20:54:gpstart:smdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110701:07:20:54:gpstart:smdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201101130' 20110701:07:20:54:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20110701:07:20:55:gpstart:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110701:07:20:55:gpstart:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110701:07:20:55:gpstart:smdw:gpadmin-[INFO]:-Master Started... 20110701:07:20:55:gpstart:smdw:gpadmin-[INFO]:-Shutting down master 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg0 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg1 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg2 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg3 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg4 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg5 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data1/primary/gpseg6 <<<<< -- Data cut due to output length 20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw8 directory /data2/primary/gpseg46 <<<<<
118
20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw8 directory /data2/primary/gpseg47 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:--------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master instance parameters 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:--------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Database = template1 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master Port = 5432 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master directory = /data/master/gpseg-1 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Timeout = 600 seconds 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master standby = Off 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Segment instances that will be started 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:Host Datadir Port Role 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw2 /mirror2/mirror/gpseg0 50000 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg1 50001 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw4 /mirror2/mirror/gpseg2 50002 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw5 /mirror1/mirror/gpseg3 50003 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw6 /mirror1/mirror/gpseg4 50004 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw7 /mirror1/mirror/gpseg5 50005 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg6 50000 Primary -- Data cut due to output length 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:50004 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:50005 Primary sdw5 sdw6 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
Continue with Greenplum instance startup Yy|Nn (default=N): > y 20110701:07:21:02:gpstart:smdw:gpadmin-[INFO]:-No standby master configured. skipping... 20110701:07:21:02:gpstart:smdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... ... 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:-Process results... 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:- Successful segment starts = 48 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:- Failed segment starts = 0
119
20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 48 <<<<<<<< 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:-Successfully started 48 of 48 segment instances, skipped 48 other segments 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-There are 48 segment(s) marked down in the database 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20110701:07:21:06:gpstart:smdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20110701:07:21:19:gpstart:smdw:gpadmin-[WARNING]:-Database started but warnings generated <<<<< 20110701:07:21:19:gpstart:smdw:gpadmin-[INFO]:-Check status of database with gpstate utility [gpadmin@smdw ~]$
20. To finish verifying that the SNAP was a valid snap and can be used to conduct a recovery of the primary system, execute gpcheckcat to check all of the catalogs in the database.
/usr/local/greenplum-db/bin/lib/gpcheckcat -p 5432 <database name>

[gpadmin@smdw ~]$ /usr/local/greenplum-db/bin/lib/gpcheckcat -p 5432 srdftesting 20110701:08:41:49:gpcheckcat:default-[INFO]:-master at host localhost, port 5432, user gpadmin, database srdftesting 20110701:08:41:49:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:49:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:49:gpcheckcat:default-[INFO]:-Performing check for database 'srdftesting', version 4.1 20110701:08:41:49:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:49:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:49:gpcheckcat:default-[INFO]:-Checking pg_partition ... 20110701:08:41:49:gpcheckcat:default-[INFO]:-[OK] pg_partition branch integrity 20110701:08:41:49:gpcheckcat:default-[INFO]:-[OK] partition with oids check 20110701:08:41:49:gpcheckcat:default-[INFO]:-[OK] partition distribution policy check 20110701:08:41:49:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:49:gpcheckcat:default-[INFO]:-Performing segment specific consistency checks 20110701:08:41:49:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:49:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:49:gpcheckcat:default-[INFO]:-Checking pg_class.reltoastrelid
120
20110701:08:41:54:gpcheckcat:default-[INFO]:-[OK] pg_class.reltoastrelid 20110701:08:41:54:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:54:gpcheckcat:default-[INFO]:-Checking for leaked temporary schemas 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] temporary schemas 20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking missing schema definitions 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] missing schema definitions 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking constraints on randomly distributed tables 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] randomly distributed tables 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking that unique constraints are only on distribution columns 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] unique constraints 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking table ownership 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] table ownership 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] type ownership 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking indexes 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] relhasindex 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] deprecated indexes 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking Object Dependencies 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] basic object dependencies 20110701:08:41:57:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:57:gpcheckcat:default-[INFO]:-Checking persistent tables 20110701:08:41:57:gpcheckcat:default-[WARNING]:-System requires recovery via gprecoverseg 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_filespace_node <=> pg_filespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_filespace_node <=> gp_global_sequence 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_database_node <=> pg_database 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_database_node <=> pg_tablespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_database_node <=> gp_global_sequence 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_tablespace_node <=> pg_tablespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_tablespace_node <=> pg_filespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_tablespace_node <=> gp_global_sequence 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> pg_tablespace 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> pg_database 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> gp_relation_node
121
20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> pg_class 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> gp_global_sequence 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> filesystem 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] pg_database <=> filesystem 20110701:08:41:58:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:58:gpcheckcat:default-[INFO]:-Performing foreign key and cross database consistency tests 20110701:08:41:58:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:58:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:58:gpcheckcat:default-[INFO]:-Checking foreign keys ... 20110701:08:41:58:gpcheckcat:default-[INFO]:- 28 tables, 102 queries 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggfnoid) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggtransfn) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(agginvtransfn) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggprelimfn) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(agginvprelimfn) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggfinalfn) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggsortop) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggtranstype) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(aminsert) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(ambeginscan) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(amgettuple) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(amgetmulti) -- Data cut due to output length 20110701:08:42:00:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_window(winprefunc) 20110701:08:42:00:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_window(winfinfunc) 20110701:08:42:00:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:42:00:gpcheckcat:default-[INFO]:-Checking cross database consistency ... 20110701:08:42:00:gpcheckcat:default-[INFO]:-[OK] Cross consistency check for gp_version_at_initdb
122
20110701:08:42:01:gpcheckcat:default-[INFO]:-[OK] pg_aggregate 20110701:08:42:01:gpcheckcat:default-[INFO]:-[OK] pg_am 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_amop 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_amproc 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_appendonly 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_appendonly_alter_column 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_attrdef 20110701:08:42:07:gpcheckcat:default-[INFO]:-[OK] pg_attribute -- Data cut due to output length
Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for
20110701:08:42:18:gpcheckcat:default-[INFO]:-[OK] Cross consistency acl check for pg_proc 20110701:08:42:18:gpcheckcat:default-[INFO]:-[OK] Cross consistency acl check for pg_tablespace 20110701:08:42:18:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:42:18:gpcheckcat:default-[INFO]:-Check complete [gpadmin@smdw ~]$
21. Stop the database.

gpstop
22. Unmount all of the devices so that they can be used for a recovery if the checks were all successful or so that the SNAP can be recreated until it is identified as a good snap.
umount <mountpoint>

[root@smdw ~]# [root@smdw ~]# [sdw8] [sdw6] [sdw7] [sdw4] [sdw5] [sdw2] [sdw3] [ mdw] umount: [ mdw] umount: [sdw1] [root@smdw ~]# umount /data gpssh -f my_gp-hosts_1-8 'umount /mirror1;umount /mirror2'
/mirror1: not found /mirror2: not found

123
124
Invisible Body Tag
I
Recovering a DCA using a SNAP session and SRDF ................. 126
125
Recovering a DCA using a SNAP session and SRDF

!
IMPORTANT This procedure assumes that theMounting and checking Symmetrix SNAP of Greenplum database on page 104 was followed to ensure a consistent SNAP on the remote VMAX. An inconsistent SNAP will result in corrupted data on restore. 1. Unmount the R1 Symmetrix devices from the DCA that will be recovered in this process.
umount <mountpoint>

[root@mdw ~]# umount /data [root@mdw ~]# gpssh -f gp-hosts_1-8 'umount /mirror1;umount /mirror2' [sdw4] [sdw5] [sdw6] [sdw7] [sdw1] [sdw2] [sdw3] [sdw8] [ mdw] umount: /mirror1: not found [ mdw] umount: /mirror2: not found [root@mdw ~]#
2. Split the SRDF devices so that the SNAP can be restored to the R2 volumes.
symrdf -cg <composite group name> split -force

[root@mdw ~]# symrdf -cg DCA_R1_CG split -force Execute an RDF 'Split' operation for composite group 'DCA_R1_CG' (y/[n]) ? y An RDF 'Split' operation execution is in progress for composite group 'DCA_R1_CG'. Please wait... Pend I/O on RDF link(s) for device(s) in (4467,001).............Done. Suspend RDF link(s) for device(s) in (4467,001)..................Done. Read/Write Enable device(s) in (4467,001) on RA at target (R2)...Done.
126
The RDF 'Split' operation successfully executed for composite group 'DCA_R1_CG'. [root@mdw ~]#
3. Query the SRDF devices and verify that they are in a split state.
symrdf query -cg <composite group name>

[root@mdw ~]# symrdf query -cg DCA_R1_CG Composite Group Name Composite Group Type Number of Symmetrix Units Number of RDF (RA) Groups RDF Consistency Mode : : : : : DCA_R1_CG RDF1 1 1 SYNC
Symmetrix ID Remote Symmetrix ID RDF (RA) Group Number Source (R1) View -------------------------------ST Standard A Logical Sym T R1 Inv R2 Inv Device Dev E Tracks Tracks -------------------------------DEV4467* 0267 RW 0 0 DEV4467* 0273 RW 0 0 DEV4467* 027F RW 0 0 DEV4467* 028B RW 0 0 DEV4467* 0297 RW 0 0 DEV4467* 02A3 RW 0 0 DEV4467* 02AF RW 0 0 DEV4467* 02BB RW 0 0 DEV4467* 02C7 RW 0 0 DEV4467* 02D3 RW 0 0 DEV4467* 02DF RW 0 0 DEV4467* 02EB RW 0 0 DEV4467* 02F7 RW 0 0 DEV4467* 0303 RW 0 0 DEV4467* 030F RW 0 0 DEV4467* 031B RW 0 0 DEV4467* 0327 RW 0 0 Total Track(s) MBs ------- ------0 0 0.0 0.0
: 000192604467 : 000192604452 : 1 (00)
(Microcode Version: 5875) (Microcode Version: 5875)
Target (R2) View ------------------------LI ST N A K T R1 Inv R2 Inv S Dev E Tracks Tracks -- ----------------------NR 0411 RW 2462 0 NR 041D RW 5128 0 NR 0429 RW 5085 0 NR 0435 RW 5159 0 NR 0441 RW 5049 0 NR 044D RW 5036 0 NR 0459 RW 5033 0 NR 0465 RW 5146 0 NR 0471 RW 5019 0 NR 047D RW 5017 0 NR 0489 RW 4998 0 NR 0495 RW 5170 0 NR 04A1 RW 5155 0 NR 04AD RW 5088 0 NR 04B9 RW 4963 0 NR 04C5 RW 5122 0 NR 04D1 RW 5046 0 ------- ------0 0 0.0 0.0
MODES STATES ----- ------ ----C S o u n s RDF Pair MDAE s p STATE ----- ------ ----S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split

127
Legend for MODES: M(ode of Operation) : D(omino) : A(daptive Copy) : (Consistency) E(xempt): Legend for STATES: Cons(istency State) Susp(end State) [root@mdw ~]# : X = Enabled, . = Disabled, M = Mixed, - = N/A : X = Online, . = Offline, P = Offline Pending, - = N/A A X D X = = = = Async, S = Enabled, . Disk Mode, Enabled, . Sync, E = Semi-sync, C = Adaptive Copy = Disabled W = WP Mode, . = ACp off = Disabled, M = Mixed, - = N/A
4. Execute a restore of the SNAP device group to restore the SNAP to the R2 devices.
symsnap -g <device group name> restore

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_2 restore Execute 'Incremental Restore' operation for device group 'DCA2880_RDF2_SNAP_2' (y/[n]) ? y 'Incremental Restore' operation execution is in progress for device group 'DCA2880_RDF2_SNAP_2'. Please wait... 'Incremental Restore' operation successfully initiated for device group 'DCA2880_RDF2_SNAP_2'. [root@smdw ~]#
5. Query the SNAP device group to verify the SNAP devices are in the restored state.
symsnap -g <device group name> query -restore

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_2 query -restore Device Group (DG) Name: DCA2880_RDF2_SNAP_2 DG's Type : RDF2 DG's Symmetrix ID : 000192604452 Source Device ----------------------Protected Logical Sym Tracks ----------------------DEV001 0411 0 DEV002 041D 0 Target Device Virtual Device State Copy ---------------------- -------------- -----------Indirect Logical Sym Tracks Logical Sym SRC <=> TGT ---------------------- -------------- -----------VDEV001 05AB 0 VDEV001 05AB Restored VDEV002 05B7 0 VDEV002 05B7 Restored
---(%) ---100 100
128
DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017
0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -------0 0.0
VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ------0 0.0
VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B
Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Total Track(s) MB(s) Legend: (G): X . (D): X M
= = = =
[root@smdw ~]#
6. Terminate the SNAP restore session but, leave the SNAP session active in the event something happens with the restore and the SNAP session is needed again.
symsnap -g <device group name> terminate -restored

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_2 terminate -restored Execute 'Terminate' operation for device group 'DCA2880_RDF2_SNAP_2' (y/[n]) ? y 'Terminate' operation execution is in progress for device group 'DCA2880_RDF2_SNAP_2'. Please wait... 'Terminate' operation successfully executed for device group 'DCA2880_RDF2_SNAP_2'. [root@smdw ~]#
129
7. Query the SNAP device group and verify that it is back into the active state and the devices are marked with a status of "CopyOnWrite".

[root@smdw ~]# symsnap -g DCA2880_RDF2_SNAP_2 query Device Group (DG) Name: DCA2880_RDF2_SNAP_2 DG's Type : RDF2 DG's Symmetrix ID : 000192604452 Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s) Legend: (G): X . (D): X M The Target device is associated with this group, The Target device is not associated with this group. The Target device has one or more inactive duplicates. The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates. = = = = 0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1 47277826 47275407 47275351 47275359 47275323 47275407 47275340 47275304 47275335 47275415 47275386 47275312 47275295 47275429 47275376 47275399 47275388 -------803683652 50230228 VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017 05AB 05B7 05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. 2414 4833 4889 4881 4917 4833 4900 4936 4905 4825 4854 4928 4945 4811 4864 4841 4852 ---------80428 5027 CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[root@smdw ~]#
130
8. Query the SRDF composite group to verify that there are tracks to update from the R2 to the R1 devices.

[root@mdw ~]# symrdf query -cg DCA_R1_CG Composite Group Name Composite Group Type Number of Symmetrix Units Number of RDF (RA) Groups RDF Consistency Mode Symmetrix ID Remote Symmetrix ID RDF (RA) Group Number Source (R1) View -------------------------------ST Standard A Logical Sym T R1 Inv R2 Inv Device Dev E Tracks Tracks -------------------------------DEV4467* 0267 RW 0 0 DEV4467* 0273 RW 0 0 DEV4467* 027F RW 0 0 DEV4467* 028B RW 0 0 DEV4467* 0297 RW 0 0 DEV4467* 02A3 RW 0 0 DEV4467* 02AF RW 0 0 DEV4467* 02BB RW 0 0 DEV4467* 02C7 RW 0 0 DEV4467* 02D3 RW 0 0 DEV4467* 02DF RW 0 0 DEV4467* 02EB RW 0 0 DEV4467* 02F7 RW 0 0 DEV4467* 0303 RW 0 0 DEV4467* 030F RW 0 0 DEV4467* 031B RW 0 0 DEV4467* 0327 RW 0 0 Total Track(s) MBs Legend for MODES: M(ode of Operation) D(omino) : A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy : X = Enabled, . = Disabled
131
: : : : :
DCA_R1_CG RDF1 1 1 SYNC : 000192604467 : 000192604452 : 1 (00) (Microcode Version: 5875) (Microcode Version: 5875)
Target (R2) View ------------------------LI ST N A K T R1 Inv R2 Inv S Dev E Tracks Tracks -- ----------------------NR 0411 RW 2462 0 NR 041D RW 5128 0 NR 0429 RW 5085 0 NR 0435 RW 5159 0 NR 0441 RW 5049 0 NR 044D RW 5036 0 NR 0459 RW 5033 0 NR 0465 RW 5146 0 NR 0471 RW 5019 0 NR 047D RW 5017 0 NR 0489 RW 4998 0 NR 0495 RW 5170 0 NR 04A1 RW 5155 0 NR 04AD RW 5088 0 NR 04B9 RW 4963 0 NR 04C5 RW 5122 0 NR 04D1 RW 5046 0 ------- ------83676 0 5229.8 0.0
MODES STATES ----- -----------C S o u n s RDF Pair MDAE s p STATE ----- -----------S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split
------- ------0 0 0.0 0.0
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off (Consistency) E(xempt): X = Enabled, . = Disabled, M = Mixed, - = N/A Legend for STATES: Cons(istency State) Susp(end State) [root@mdw ~]# : X = Enabled, . = Disabled, M = Mixed, - = N/A : X = Online, . = Offline, P = Offline Pending, - = N/A
9. Issue a restore against the SRDF composite group to restore and sync the R1 devices to the R2 devices using the data on the R2s.
symrdf -cg <composite group name> restore

[root@mdw ~]# symrdf -cg DCA_R1_CG restore Execute an RDF 'Incremental Restore' operation for composite group 'DCA_R1_CG' (y/[n]) ? y An RDF 'Incremental Restore' operation execution is in progress for composite group 'DCA_R1_CG'. Please wait... Write Disable device(s) in (4467,001) on SA at source (R1).......Done. Write Disable device(s) in (4467,001) on RA at target (R2).......Done. Suspend RDF link(s) for device(s) in (4467,001)..................Done. Merge track tables between source and target in (4467,001).......Started. Devices: 0327-0332 in (4467,001)................................ Merged. Devices: 0307-0326 in (4467,001)................................ In Progress. Devices: 02E7-0306 in (4467,001)................................ In Progress. Devices: 02C7-02E6 in (4467,001)................................ In Progress. Devices: 02A7-02C6 in (4467,001)................................ In Progress. Devices: 0287-02A6 in (4467,001)................................ In Progress. Devices: 0267-0286 in (4467,001)................................ In Progress. Devices: 0307-0326 in (4467,001)................................ Merged. Devices: 02E7-0306 in (4467,001)................................ Merged. Devices: 02C7-02E6 in (4467,001)................................ Merged. Devices: 02A7-02C6 in (4467,001)................................ Merged. Devices: 0267-0286 in (4467,001)................................ Merged. Devices: 0287-02A6 in (4467,001)................................ Merged. Merge track tables between source and target in (4467,001).......Done. Resume RDF link(s) for device(s) in (4467,001)...................Started. Resume RDF link(s) for device(s) in (4467,001)...................Done. Read/Write Enable device(s) in (4467,001) on SA at source (R1)...Done. The RDF 'Incremental Restore' operation successfully initiated for composite group 'DCA_R1_CG'. [root@mdw ~]#
132
10. Query the SRDF composite group until the devices are in a "Synchronized" state indicating that all of the data was restored from the R2 to the R1.

[root@mdw ~]# symrdf query -cg DCA_R1_CG Composite Group Name Composite Group Type Number of Symmetrix Units Number of RDF (RA) Groups RDF Consistency Mode Symmetrix ID Remote Symmetrix ID RDF (RA) Group Number : : : : : DCA_R1_CG RDF1 1 1 SYNC : 000192604467 : 000192604452 : 1 (00) (Microcode Version: 5875) (Microcode Version: 5875)
Source (R1) View Target (R2) View MODES STATES -------------------------------- ------------------------- ----- -----------ST LI ST C S Standard A N A o u Logical Sym T R1 Inv R2 Inv K T R1 Inv R2 Inv n s RDF Pair Device Dev E Tracks Tracks S Dev E Tracks Tracks MDAE s p STATE -------------------------------- -- ----------------------- ----- ------ -----DEV4467* 0267 RW 0 0 RW 0411 WD 0 0 S... X Synchronized DEV4467* 0273 RW 0 0 RW 041D WD 0 0 S... X Synchronized DEV4467* 027F RW 0 0 RW 0429 WD 0 0 S... X Synchronized DEV4467* 028B RW 0 0 RW 0435 WD 0 0 S... X Synchronized DEV4467* 0297 RW 0 0 RW 0441 WD 0 0 S... X Synchronized DEV4467* 02A3 RW 0 0 RW 044D WD 0 0 S... X Synchronized DEV4467* 02AF RW 0 0 RW 0459 WD 0 0 S... X Synchronized DEV4467* 02BB RW 0 0 RW 0465 WD 0 0 S... X Synchronized DEV4467* 02C7 RW 0 0 RW 0471 WD 0 0 S... X Synchronized DEV4467* 02D3 RW 0 0 RW 047D WD 0 0 S... X Synchronized DEV4467* 02DF RW 0 0 RW 0489 WD 0 0 S... X Synchronized DEV4467* 02EB RW 0 0 RW 0495 WD 0 0 S... X Synchronized DEV4467* 02F7 RW 0 0 RW 04A1 WD 0 0 S... X Synchronized DEV4467* 0303 RW 0 0 RW 04AD WD 0 0 S... X Synchronized DEV4467* 030F RW 0 0 RW 04B9 WD 0 0 S... X Synchronized DEV4467* 031B RW 0 0 RW 04C5 WD 0 0 S... X Synchronized DEV4467* 0327 RW 0 0 RW 04D1 WD 0 0 S... X Synchronized Total Track(s) MBs Legend for MODES: M(ode of Operation) : A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
133
------- ------0 0 0.0 0.0
------- ------0 0 0.0 0.0
D(omino) : X = Enabled, . = Disabled A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off (Consistency) E(xempt): X = Enabled, . = Disabled, M = Mixed, - = N/A Legend for STATES: Cons(istency State) Susp(end State) [root@mdw ~]# : X = Enabled, . = Disabled, M = Mixed, - = N/A : X = Online, . = Offline, P = Offline Pending, - = N/A
11. Mount the R1 Symmetrix devices back to the DCA that were just restored and unmounted in step 1 of this procedure.
mount <mount point>

[root@mdw ~]# mount /data [root@mdw ~]# gpssh -f gp-hosts_1-8 'mount /mirror1;mount /mirror2' [sdw4] [sdw5] [sdw6] [sdw7] [sdw1] [sdw2] [sdw3] [sdw8] [root@mdw ~]#
12. Switch to the gpadmin user.

su - gpadmin

[root@mdw ~]# su - gpadmin [gpadmin@mdw ~]$
13. Start the database in maintenance mode.

gpstart -m

[gpadmin@mdw ~]$ gpstart -m 20110705:09:01:04:gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20110705:09:01:04:gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110705:09:01:04:gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110705:09:01:04:gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201101130' 20110705:09:01:04:gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
134
20110705:09:01:05:gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110705:09:01:05:gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110705:09:01:05:gpstart:mdw:gpadmin-[INFO]:-Master Started... [gpadmin@mdw ~]$
14. Connect to the database using utility mode


[gpadmin@mdw ~]$ PGOPTIONS='-c gp_session_role=utility' psql srdftesting psql (8.2.15) Type "help" for help. srdftesting=#
15. Verify that the primary segments set to a role of 'm' mirror and a status of 'd' down and that the mirror segments are set to a role of 'p' primary, a mode of 'c' change tracking and a status of 'u' up. The segments should already be set if the SNAP devices were checked properly and were not set back before stopping the database on the DR system and unmounting the volumes.
select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid;

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1 4 | 2 | m | p | s | d | sdw1 | /data1/primary/gpseg2 5 | 3 | m | p | s | d | sdw1 | /data2/primary/gpseg3 6 | 4 | m | p | s | d | sdw1 | /data2/primary/gpseg4 7 | 5 | m | p | s | d | sdw1 | /data2/primary/gpseg5 8 | 6 | m | p | s | d | sdw2 | /data1/primary/gpseg6 9 | 7 | m | p | s | d | sdw2 | /data1/primary/gpseg7 10 | 8 | m | p | s | d | sdw2 | /data1/primary/gpseg8 11 | 9 | m | p | s | d | sdw2 | /data2/primary/gpseg9 12 | 10 | m | p | s | d | sdw2 | /data2/primary/gpseg10 13 | 11 | m | p | s | d | sdw2 | /data2/primary/gpseg11
135
14 | 15 | 16 | 17 | 18 | 19 | 20 | -- Data 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | (97 rows)
12 | m | 13 | m | 14 | m | 15 | m | 16 | m | 17 | m | 18 | m | cut due to 40 41 42 43 44 45 46 47 | | | | | | | | p p p p p p p p | | | | | | | |
p | s p | s p | s p | s p | s p | s p | s output length m m m m m m m m | | | | | | | | c c c c c c c c
| | | | | | |
d d d d d d d
| | | | | | |
sdw3 sdw3 sdw3 sdw3 sdw3 sdw3 sdw4
| | | | | | |
/data1/primary/gpseg12 /data1/primary/gpseg13 /data1/primary/gpseg14 /data2/primary/gpseg15 /data2/primary/gpseg16 /data2/primary/gpseg17 /data1/primary/gpseg18
| | | | | | | |
u u u u u u u u
| | | | | | | |
sdw4 sdw5 sdw1 sdw2 sdw3 sdw4 sdw5 sdw6
| | | | | | | |
/mirror1/mirror/gpseg40 /mirror1/mirror/gpseg41 /mirror2/mirror/gpseg42 /mirror2/mirror/gpseg43 /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
16. If the primary and mirror segments were set correctly skip to step 19. Otherwise, use the SQL statements in steps 17 and 18 to update the configuration so that the database starts up using the mirror segments that were just restored from the SNAP. 17. To set the primary segments to a role of 'm' and a status of 'd' issue the following SQL statement.

Sample output of query command after running the update statement:

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1 4 | 2 | m | p | s | d | sdw1 | /data1/primary/gpseg2
136
5 | 3 | m | 6 | 4 | m | 7 | 5 | m | 8 | 6 | m | 9 | 7 | m | 10 | 8 | m | -- Data cut due to
p | p | p | p | p | p | output length
s s s s s s
| | | | | |
d d d d d d
| | | | | |
sdw1 sdw1 sdw1 sdw2 sdw2 sdw2
| | | | | |
/data2/primary/gpseg3 /data2/primary/gpseg4 /data2/primary/gpseg5 /data1/primary/gpseg6 /data1/primary/gpseg7 /data1/primary/gpseg8
48 | 46 | m | p | s 49 | 47 | m | p | s 50 | 0 | m | m | s 51 | 1 | m | m | s 52 | 2 | m | m | s 53 | 3 | m | m | s 54 | 4 | m | m | s 55 | 5 | m | m | s 56 | 6 | m | m | s 57 | 7 | m | m | s -- Data cut due to output length 96 | 46 | m 97 | 47 | m (97 rows) srdftesting=# | m | m | s | s
| | | | | | | | | |
d d u u u u u u u u
| | | | | | | | | |
sdw8 sdw8 sdw2 sdw3 sdw4 sdw5 sdw6 sdw7 sdw3 sdw4
| | | | | | | | | |
/data2/primary/gpseg46 /data2/primary/gpseg47 /mirror2/mirror/gpseg0 /mirror2/mirror/gpseg1 /mirror2/mirror/gpseg2 /mirror1/mirror/gpseg3 /mirror1/mirror/gpseg4 /mirror1/mirror/gpseg5 /mirror2/mirror/gpseg6 /mirror2/mirror/gpseg7
| u | u
| sdw5 | sdw6
| /mirror1/mirror/gpseg46 | /mirror1/mirror/gpseg47
18. To set the mirror segments to a role of 'p' and a mode of 'c' issue the following SQL statement.

Sample output of query command after running the update statement:

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1
137
4 | 2 | m | 5 | 3 | m | 6 | 4 | m | 7 | 5 | m | 8 | 6 | m | 9 | 7 | m | 10 | 8 | m | -- Data cut due to
p | p | p | p | p | p | p | output length
s s s s s s s
| | | | | | |
d d d d d d d
| | | | | | |
sdw1 sdw1 sdw1 sdw1 sdw2 sdw2 sdw2
| | | | | | |
/data1/primary/gpseg2 /data2/primary/gpseg3 /data2/primary/gpseg4 /data2/primary/gpseg5 /data1/primary/gpseg6 /data1/primary/gpseg7 /data1/primary/gpseg8
49 | 47 | m | p | s 50 | 0 | p | m | c 51 | 1 | p | m | c 52 | 2 | p | m | c 53 | 3 | p | m | c 54 | 4 | p | m | c 55 | 5 | p | m | c 56 | 6 | p | m | c 57 | 7 | p | m | c 58 | 8 | p | m | c -- Data cut due to output length 96 | 46 | p 97 | 47 | p (97 rows) srdftesting=# | m | m | c | c
| | | | | | | | | |
d u u u u u u u u u
| | | | | | | | | |
sdw8 sdw2 sdw3 sdw4 sdw5 sdw6 sdw7 sdw3 sdw4 sdw5
| | | | | | | | | |
/data2/primary/gpseg47 /mirror2/mirror/gpseg0 /mirror2/mirror/gpseg1 /mirror2/mirror/gpseg2 /mirror1/mirror/gpseg3 /mirror1/mirror/gpseg4 /mirror1/mirror/gpseg5 /mirror2/mirror/gpseg6 /mirror2/mirror/gpseg7 /mirror2/mirror/gpseg8
| u | u
| sdw5 | sdw6
| /mirror1/mirror/gpseg46 | /mirror1/mirror/gpseg47
19. Exit out of the psql session and back to the command prompt.
\q

srdftesting=# \q [gpadmin@mdw ~]$
20. Stop the database to remove it from maintenance mode.

gpstop -m

[gpadmin@mdw ~]$ gpstop -m 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -m 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database
138
20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1 [gpadmin@mdw ~]$
21. Restart the database normally. Watch to make sure that the primary segments were skipped and only the mirror segments are started.
gpstart

[gpadmin@mdw ~]$ gpstart 20110705:09:01:40:gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: 20110705:09:01:40:gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110705:09:01:40:gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110705:09:01:40:gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201101130' 20110705:09:01:40:gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20110705:09:01:41:gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110705:09:01:41:gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110705:09:01:42:gpstart:mdw:gpadmin-[INFO]:-Master Started... 20110705:09:01:42:gpstart:mdw:gpadmin-[INFO]:-Shutting down master 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg0 <<<<< 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg1 <<<<< 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg2 <<<<< 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg3 <<<<< 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg4 <<<<< 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg5 <<<<< 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data1/primary/gpseg6 <<<<< -- Data cut due to output length 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw8 directory /data2/primary/gpseg46 <<<<< 20110705:09:01:44:gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw8 directory /data2/primary/gpseg47 <<<<<
139
20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:--------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master instance parameters 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:--------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Database = template1 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master Port = 5432 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master directory = /data/master/gpseg-1 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Timeout = 600 seconds 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master standby = Off 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Segment instances that will be started 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- Host Datadir Port Role 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw2 /mirror2/mirror/gpseg0 50000 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg1 50001 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw4 /mirror2/mirror/gpseg2 50002 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw5 /mirror1/mirror/gpseg3 50003 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw6 /mirror1/mirror/gpseg4 50004 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw7 /mirror1/mirror/gpseg5 50005 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg6 50000 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw4 /mirror2/mirror/gpseg7 50001 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw5 /mirror2/mirror/gpseg8 50002 Primary -- Data cut due to output length 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:50004 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:50005 Primary sdw5 sdw6 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
Continue with Greenplum instance startup Yy|Nn (default=N): > y 20110705:09:01:47:gpstart:mdw:gpadmin-[INFO]:-No standby master configured. skipping... 20110705:09:01:47:gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:-Process results... 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 48
140
20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:- Failed segment starts = 0 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 48 <<<<<<<< 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:-Successfully started 48 of 48 segment instances, skipped 48 other segments 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-******************************* ********************************************* 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-There are 48 segment(s) marked down in the database 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-******************************* ********************************************* 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20110705:09:01:50:gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20110705:09:02:01:gpstart:mdw:gpadmin-[WARNING]:-Database started but warnings generated <<<<< 20110705:09:02:01:gpstart:mdw:gpadmin-[INFO]:-Check status of database with gpstate utility [gpadmin@mdw ~]$
22. Once the database is up and running on the mirror segments that were just restored from the SNAP session, run a full gprecoverseg to rebuild the primary segments with the data from the mirror segments and synchronize the system. A full recovery is necessary since the mirror data was changed by a different DCA and the remote DCA has no knowledge of what changes were made.
IMPORTANT Performing an "incremental" recovery will result in a corrupted database!

Note: The output from gprecoverseg is very long; fifteen lines for each segment. The important thing to verify is that a full recovery is being performed from the mirror to the primary. gprecoverseg -F

141

[gpadmin@mdw ~]$ gprecoverseg -F 20110705:09:03:15:gprecoverseg:mdw:gpadmin-[INFO]:-Starting gprecoverseg with args: -F 20110705:09:03:15:gprecoverseg:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110705:09:03:15:gprecoverseg:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-Greenplum instance recovery parameters 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-Recovery type = Standard 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-Recovery 1 of 48 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Synchronization mode = Full 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance host = sdw1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance address = sdw1-1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance directory = /data1/primary/gpseg0 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance port = 40000 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance replication port = 41000 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance host = sdw2 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance address = sdw2-2 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance directory = /mirror2/mirror/gpseg0 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance port = 50000 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance replication port = 51000 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Target = in-place 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-Recovery 2 of 48 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Synchronization mode = Full 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance host = sdw1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance address = sdw1-1
142
20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance directory = /data1/primary/gpseg1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance port = 40001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance replication port = 41001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance host = sdw3 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance address = sdw3-2 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance directory = /mirror2/mirror/gpseg1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance port = 50001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance replication port = 51001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Target = in-place . Data cut due to output length 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-Recovery 48 of 48 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Synchronization mode = Full 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance host = sdw8 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance address = sdw8-2 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance directory = /data2/primary/gpseg47 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance port = 40005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance replication port = 41005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance host = sdw6 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance address = sdw6-1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance directory = /mirror1/mirror/gpseg47 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance port = 50005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance replication port = 51005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Target = in-place 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------Continue with segment recovery procedure Yy|Nn (default=N): > y
143
20110705:09:03:22:gprecoverseg:mdw:gpadmin-[INFO]:-48 segment(s) to recover 20110705:09:03:22:gprecoverseg:mdw:gpadmin-[INFO]:-Ensuring 48 failed segment(s) are stopped 20110705:09:03:23:gprecoverseg:mdw:gpadmin-[INFO]:-Cleaning files from 48 segment(s) ......... 20110705:09:03:32:gprecoverseg:mdw:gpadmin-[INFO]:-Building template directory 20110705:09:03:33:gprecoverseg:mdw:gpadmin-[INFO]:-Validating remote directories . 20110705:09:03:34:gprecoverseg:mdw:gpadmin-[INFO]:-Copying template directory file . 20110705:09:03:35:gprecoverseg:mdw:gpadmin-[INFO]:-Configuring new segments . 20110705:09:03:36:gprecoverseg:mdw:gpadmin-[INFO]:-Cleaning files . 20110705:09:03:37:gprecoverseg:mdw:gpadmin-[INFO]:-Updating configuration with new mirrors 20110705:09:03:39:gprecoverseg:mdw:gpadmin-[INFO]:-Updating mirrors . 20110705:09:03:40:gprecoverseg:mdw:gpadmin-[INFO]:-Starting mirrors 20110705:09:03:40:gprecoverseg:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Process results... 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Updating configuration to mark mirrors up 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Updating primaries 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Commencing parallel primary conversion of 48 segments, please wait... .. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Process results... 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Done updating primaries 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-***************************** ************************************* 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Updating segments for resynchronization is completed. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Use gpstate -s to check the resynchronization progress. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-***************************** ************************************* [gpadmin@mdw ~]$
23. Periodically, run the gpstate command as reported in the gprecoverseg output to obtain the status of the recovery. When all segments report synchronized, the recovery is complete.
144
24. Note: the output from gpstate -s is very long - fourteen lines for each segment. The important thing to monitor is the "Mirror status", which will report "Resynchronizing" during the recovery process, and "Synchronized" when it is complete.
gpstate -s
Sample output of command when the systems is still resynchronizing:

[gpadmin@mdw ~]$ gpstate -s 20110705:09:06:15:gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -s 20110705:09:06:15:gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110705:09:06:15:gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110705:09:06:15:gpstate:mdw:gpadmin-[INFO]:-Gathering data from segments... ........... 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:--Master Configuration & Status 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:- Master host = mdw 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Master postgres process ID = 2015 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Master data directory = /data/master/gpseg-1 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:- Master port = 5432 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Master current role = dispatch 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:- Greenplum initsystem version = 4.1.1.1 build 1 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Greenplum current version = PostgreSQL 8.2.15 (Greenplum Database 4.1.1.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 12 2011 18:07:08 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Postgres version = 8.2.15 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Master standby = No master standby configured 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:-Segment Instance Status Report 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw2 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Address = sdw2-2 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Datadir = /mirror2/mirror/gpseg0 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Port = 50000 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirroring Info
145
20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Current role = Primary 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Preferred role = Mirror 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirror status = Resynchronizing 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Change Tracking Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Change tracking data size = 199 MB 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Resynchronization Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Resynchronization mode = Full 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Data synchronized = 2.50 GB 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Estimated total data to synchronize = 59.3 GB 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Estimated resync progress with mirror = 4.21% 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Estimated resync end time = 2011-07-05 10:06:38 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Status 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:PID = 8937 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Database status = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw1 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Address = sdw1-1 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Datadir = /data1/primary/gpseg0 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Port = 40000 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirror status = Resynchronizing 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Status 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:PID = 5602 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment status = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
146
. Data cut due to output length 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw8 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Address = sdw8-2 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Datadir = /data2/primary/gpseg47 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Port = 40005 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirror status = Resynchronizing 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Status 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:PID = 28224 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment status = Up [gpadmin@mdw ~]$
Sample output of command when systems are fully synchronized:

[gpadmin@mdw ~]$ gpstate -s 20110705:10:41:33:gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -s 20110705:10:41:33:gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110705:10:41:33:gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110705:10:41:34:gpstate:mdw:gpadmin-[INFO]:-Gathering data from segments... ... 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:--Master Configuration & Status 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:- Master host = mdw 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Master postgres process ID = 2015 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Master data directory = /data/master/gpseg-1 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:- Master port = 5432 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Master current role = dispatch 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:- Greenplum initsystem version = 4.1.1.1 build 1
147
20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Greenplum current version = PostgreSQL 8.2.15 (Greenplum Database 4.1.1.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 12 2011 18:07:08 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Postgres version = 8.2.15 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Master standby = No master standby configured 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:-Segment Instance Status Report 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw2 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Address = sdw2-2 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Datadir = /mirror2/mirror/gpseg0 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Port = 50000 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Current role = Primary 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Preferred role = Mirror 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirror status = Synchronized 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Status 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:PID = 8937 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Database status = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw1 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Address = sdw1-1 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Datadir = /data1/primary/gpseg0 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Port = 40000 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirror status = Synchronized 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Status
148
20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:PID = 5602 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment status = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------. Data cut due to output length 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw8 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Address = sdw8-2 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Datadir = /data2/primary/gpseg47 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Port = 40005 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirror status = Synchronized 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Status 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:PID = 28224 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment status = Up [gpadmin@mdw ~]$
25. Once the database is fully synchronized and all segments are up-to-date, stop and restart the database to switch the primary segments back to the primary role. This will happen automatically when the database is restarted. There is no need to go back into maintenance mode and update the gp_segment_configuration table space manually.
gpstop gpstart

149
150
Invisible Body Tag
J
Starting EMC Greenplum database from mirror segments....... 152
151
Starting EMC Greenplum database from mirror segments

Complete the following steps to start the Greenplum database from the mirror segments, such as on the R2 VMAX.
Note: All commands are issued as gpadmin, or another database user with appropriate permissions, unless otherwise noted.
1. Issue SRDF commands to split the SRDF link from R1 (symrdf failover), and mount SRDF R2 volumes to the remote DCA. 2. Start the Greenplum database in maintenance mode, which starts only the Master Server and nothing else.
gpstart -m
Sample output from command:

[gpadmin@smdw ~]$ gpstart -m 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201101130' 20110616:08:27:34:gpstart:smdw:gpadmin-[WARNING]:-postmaster.pid file exists on Master, checking if recovery startup required 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-Commencing recovery startup checks 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-No socket connection or lock file in /tmp found for port=5432 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-No Master instance process, entering recovery startup mode 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-Clearing Master instance pid file 20110616:08:27:34:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20110616:08:27:35:gpstart:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110616:08:27:35:gpstart:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110616:08:27:36:gpstart:smdw:gpadmin-[INFO]:-Commencing forced instance shutdown 20110616:08:27:38:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20110616:08:27:39:gpstart:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110616:08:27:39:gpstart:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110616:08:27:39:gpstart:smdw:gpadmin-[INFO]:-Master Started... [gpadmin@smdw ~]$
3. Log in to a database on the GP server using utility mode.

152

[gpadmin@smdw ~]$ PGOPTIONS='-c gp_session_role=utility' psql srdftesting psql (8.2.15) Type "help" for help. srdftesting=#
4. If not previously done, alter the users search path so that the user can find and access the gp_toolkit schema where the gp_segment_configuration table space is maintained.
ALTER role gpadmin set search_path=gp_toolkit,information_schema,pg_aoseg,pg_bitmapindex,pg_catalog,pg_ toast,public;
5. To see the current configuration of the GP database, run the following SQL command. This SQL select command will show which database is on which segment server as well as its role, status and location on the segment server.
select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid;

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.fselo cation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+---------------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | p | p | s | u | sdw1 | /data1/primary/gpseg0 3 | 1 | p | p | s | u | sdw1 | /data1/primary/gpseg1 4 | 2 | p | p | s | u | sdw1 | /data1/primary/gpseg2 5 | 3 | p | p | s | u | sdw1 | /data2/primary/gpseg3 6 | 4 | p | p | s | u | sdw1 | /data2/primary/gpseg4 7 | 5 | p | p | s | u | sdw1 | /data2/primary/gpseg5 8 | 6 | p | p | s | u | sdw2 | /data1/primary/gpseg6 9 | 7 | p | p | s | u | sdw2 | /data1/primary/gpseg7 10 | 8 | p | p | s | u | sdw2 | /data1/primary/gpseg8 Data cut due to length of output 88 | 38 | m | m | s | u | sdw2 | /mirror2/mirror/gpseg38 89 | 39 | m | m | s | u | sdw3 | /mirror1/mirror/gpseg39 90 | 40 | m | m | s | u | sdw4 | /mirror1/mirror/gpseg40 91 | 41 | m | m | s | u | sdw5 | /mirror1/mirror/gpseg41 92 | 42 | m | m | s | u | sdw1 | /mirror2/mirror/gpseg42 93 | 43 | m | m | s | u | sdw2 | /mirror2/mirror/gpseg43 94 | 44 | m | m | s | u | sdw3 | /mirror2/mirror/gpseg44 95 | 45 | m | m | s | u | sdw4 | /mirror1/mirror/gpseg45 96 | 46 | m | m | s | u | sdw5 | /mirror1/mirror/gpseg46 97 | 47 | m | m | s | u | sdw6 | /mirror1/mirror/gpseg47 (97 rows) srdftesting=#

153
Note: A role of 'p' indicates the database has the 'primary' role and a role of 'm' indicates the database has the 'mirror' role. A mode of 's' indicates the database is 'Synchronized', a mode of 'c' indicates the database is in 'change tracking' mode because its mirror or primary is down and a mode of 'r' indicates the database is in 'recovery' mode. A status of 'u' indicates a segment database is 'up' and a status of 'd' indicates the database is 'down'.
6. Set the primary databases to the status of down and a role of mirror so that when the GP database is brought online it will not attempt to start the primary database segments, and if a gprecoverseg is issued it will use the mirror segments as the source for the recovery.

Sample output from running select statement after this update statement:
srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1 4 | 2 | m | p | s | d | sdw1 | /data1/primary/gpseg2 5 | 3 | m | p | s | d | sdw1 | /data2/primary/gpseg3 6 | 4 | m | p | s | d | sdw1 | /data2/primary/gpseg4 7 | 5 | m | p | s | d | sdw1 | /data2/primary/gpseg5 8 | 6 | m | p | s | d | sdw2 | /data1/primary/gpseg6 9 | 7 | m | p | s | d | sdw2 | /data1/primary/gpseg7 10 | 8 | m | p | s | d | sdw2 | /data1/primary/gpseg8 Data cut due to length of output 88 | 38 | m | m | s | u | sdw2 | /mirror2/mirror/gpseg38
154
89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | (97 rows)
39 40 41 42 43 44 45 46 47
| | | | | | | | |
m m m m m m m m m
| | | | | | | | |
m m m m m m m m m
| | | | | | | | |
s s s s s s s s s
| | | | | | | | |
u u u u u u u u u
| | | | | | | | |
sdw3 sdw4 sdw5 sdw1 sdw2 sdw3 sdw4 sdw5 sdw6
| | | | | | | | |
/mirror1/mirror/gpseg39 /mirror1/mirror/gpseg40 /mirror1/mirror/gpseg41 /mirror2/mirror/gpseg42 /mirror2/mirror/gpseg43 /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
7. Set the mirror database segments to the role of primary and mode of change tracking so that when the GP database is brought online it will start the mirror segments as the primary segments and keep track of any changes that are made to the segments so a gprecoverseg can be issued.

Sample output from select statement after this update statement:

srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.fselo cation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1 4 | 2 | m | p | s | d | sdw1 | /data1/primary/gpseg2 5 | 3 | m | p | s | d | sdw1 | /data2/primary/gpseg3 6 | 4 | m | p | s | d | sdw1 | /data2/primary/gpseg4 7 | 5 | m | p | s | d | sdw1 | /data2/primary/gpseg5 8 | 6 | m | p | s | d | sdw2 | /data1/primary/gpseg6 9 | 7 | m | p | s | d | sdw2 | /data1/primary/gpseg7 10 | 8 | m | p | s | d | sdw2 | /data1/primary/gpseg8 Data cut due to length of output 88 | 38 | p | m | c | u | sdw2 | /mirror2/mirror/gpseg38 89 | 39 | p | m | c | u | sdw3 | /mirror1/mirror/gpseg39 90 | 40 | p | m | c | u | sdw4 | /mirror1/mirror/gpseg40 91 | 41 | p | m | c | u | sdw5 | /mirror1/mirror/gpseg41 92 | 42 | p | m | c | u | sdw1 | /mirror2/mirror/gpseg42 93 | 43 | p | m | c | u | sdw2 | /mirror2/mirror/gpseg43

155
94 | 44 95 | 45 96 | 46 97 | 47 (97 rows) srdftesting=#
| | | |
p p p p
| | | |
m m m m
| | | |
c c c c
| | | |
u u u u
| | | |
sdw3 sdw4 sdw5 sdw6
| | | |
/mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
8. Quit the psql session to return to the command prompt.

\q

srdftesting=# \q [gpadmin@smdw ~]$
9. Stop the database master server so it is removed from maintenance mode.

gpstop -m

20110616:08:40:43:gpstop:smdw:gpadmin-[INFO]:-Starting gpstop with args: -m 20110616:08:40:43:gpstop:smdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110616:08:40:43:gpstop:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110616:08:40:43:gpstop:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110616:08:40:44:gpstop:smdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110616:08:40:44:gpstop:smdw:gpadmin-[INFO]:-There are 0 connections to the database 20110616:08:40:44:gpstop:smdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20110616:08:40:44:gpstop:smdw:gpadmin-[INFO]:-Master host=mdw 20110616:08:40:44:gpstop:smdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20110616:08:40:44:gpstop:smdw:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1 [gpadmin@smdw ~]$
10. )Start the database as normal. Notice that the primary segments are skipped since they were marked down in the gp_segment_configuration table space and only the mirror segments were started.
gpstart
156

[gpadmin@smdw ~]$ gpstart 20110616:08:40:50:gpstart:smdw:gpadmin-[INFO]:-Starting gpstart with args: 20110616:08:40:50:gpstart:smdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20110616:08:40:50:gpstart:smdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.1.1.1 build 1' 20110616:08:40:50:gpstart:smdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201101130' 20110616:08:40:50:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20110616:08:40:51:gpstart:smdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110616:08:40:51:gpstart:smdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110616:08:40:51:gpstart:smdw:gpadmin-[INFO]:-Master Started... 20110616:08:40:51:gpstart:smdw:gpadmin-[INFO]:-Shutting down master 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg0 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg1 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data1/primary/gpseg2 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg3 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg4 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw1 directory /data2/primary/gpseg5 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data1/primary/gpseg6 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data1/primary/gpseg7 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data1/primary/gpseg8 <<<<< 20110616:08:40:53:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data2/primary/gpseg9 <<<<< Data cut due to length of output 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:--------------------------20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:-Master instance parameters 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:--------------------------20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:-Database = template1 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:-Master Port = 5432 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:-Master directory = /data/master/gpseg-1 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:-Timeout = 600 seconds 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:-Master standby = Off 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:-Segment instances that will be started 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:---------------------------------Starting EMC Greenplum database from mirror segments
157
20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:Port Role 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50000 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50001 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50002 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50003 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50004 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50005 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50000 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50001 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50002 Primary Data cut due to length of output 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50004 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50005 Primary
Host sdw2 sdw3 sdw4 sdw5 sdw6 sdw7 sdw3 sdw4 sdw5
Datadir /mirror2/mirror/gpseg0 /mirror2/mirror/gpseg1 /mirror2/mirror/gpseg2 /mirror1/mirror/gpseg3 /mirror1/mirror/gpseg4 /mirror1/mirror/gpseg5 /mirror2/mirror/gpseg6 /mirror2/mirror/gpseg7 /mirror2/mirror/gpseg8
sdw5 sdw6
/mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
Continue with Greenplum instance startup Yy|Nn (default=N): > y 20110616:08:40:55:gpstart:smdw:gpadmin-[INFO]:-No standby master configured. skipping... 20110616:08:40:55:gpstart:smdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... ... 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:-Process results... 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:Successful segment starts = 48 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:Failed segment starts = 0 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 48 <<<<<<<< 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:-Successfully started 48 of 48 segment instances, skipped 48 other segments 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-There are 48 segment(s) marked down in the database 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
158
20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20110616:08:40:59:gpstart:smdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20110616:08:41:11:gpstart:smdw:gpadmin-[WARNING]:-Database started but warnings generated <<<<< 20110616:08:41:11:gpstart:smdw:gpadmin-[INFO]:-Check status of database with gpstate utility [gpadmin@smdw ~]$
11. To recover the primary segments that were marked down, issue gprecoverseg -F to perform a full recovery from the online mirror segments (SAN) to the offline primary segments (internal to the remote DCA). 12. Once the database is fully synchronized and all segments are up-to-date, stop and restart the database to switch the primary segments back to the primary role. This will happen automatically when the database is restarted. There is no need to go back into maintenance mode and update the gp_segment_configuration table space manually.
gpstop gpstart

159
160

Green Plum 300-012-943

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Green Plum 300-012-943

Caricato da

Copyright:

Formati disponibili

Configuring EMC Greenplum Data Computing Appliance Using SAN Mirror and EMC Symmetrix VMAX for Disaster

EMC Corporation Corporate Headquarters: Hopkinton, MA 01748-9103

Preface.............................................................................................................................. 5 Chapter 1 Introducing the SAN Mirror Solution

Setup and Operation of SAN with VMAX

Discovering DCA Internal Server WWNs

Verifying DCA Volume Sizes

Creating VMAX Volumes for DCA Mirror Database

Creating the SRDF Configuration

Configuring DCA Internal Switch for SAN Mirror

Automating DCA Internal SAN Configuration

Creating SNAPs on Remote Symmetrix for Disaster Recovery

Mounting and Checking a Symmetrix SNAP of the Greenplum Database

Recovering a DCA Using a SNAP Session and SRDF

Starting EMC Greenplum Database from Mirror Segments

Conventions used in this guide

IMPORTANT An important notice contains information essential to operation of the software.

Where to get help

Invisible Body Tag

This chapter provides the following information.

Introducing the SAN Mirror Solution

Introducing the SAN Mirror Solution

Introducing the SAN Mirror Solution

EMC Greenplum Data Computing Appliance (DCA)

Introducing the SAN Mirror Solution

Data Computing Appliance (DCA)

Massively Parallel Processing (MPP)

Introducing the SAN Mirror Solution

Using a SAN Mirror solution

Using a SAN Mirror solution

Introducing the SAN Mirror Solution

Stand-alone DCA versus SAN Mirror configuration

SAN Mirror configuration with remote replication

Introducing the SAN Mirror Solution

How SAN Mirror increases the capacity of the DCA

How EMC Greenplum DCA mirrors data

Using a SAN Mirror solution

Introducing the SAN Mirror Solution

Mirroring data to the SAN with Greenplum SAN Mirror

Introducing the SAN Mirror Solution

A conceptual picture of a SAN Mirror configuration is shown in Figure 4.

DCA SAN Mirror configuration

Using a SAN Mirror solution

Introducing the SAN Mirror Solution

Components of a SAN Mirror configuration

EMC Greenplum DCA

Introducing the SAN Mirror Solution

Greenplum DCA components

Components of a SAN Mirror configuration

Introducing the SAN Mirror Solution

Connectrix fabric trunking avoids SAN bottlenecks

Introducing the SAN Mirror Solution

EMC Symmetrix VMAX storage system

Components of a SAN Mirror configuration

Introducing the SAN Mirror Solution

Synchronous Asynchronous Adaptive copy

Introducing the SAN Mirror Solution

Components of a SAN Mirror configuration

Introducing the SAN Mirror Solution

Introducing the SAN Mirror Solution

Considerations and best practices

Performance on page 25 Data integrity on page 26

Introducing the SAN Mirror Solution

Introducing the SAN Mirror Solution

Segment failure creates inconsistent remote image