Technical Report

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Operating in Cluster-Mode
Reena Gupta, Sandra Moulton, NetApp September 2012 | TR-4078

Executive Summary
This technical report provides best practices and recommendations for the deployment and optimal performance of NetApp Infinite Volume. Infinite Volume is a new innovation in scalable NetApp storage systems that allows customers to add storage as they grow without increasing operational complexity. This document provides functional and operational guidance for Infinite Volume in the Data ONTAP 8.1.x architecture operating in Cluster-Mode.

TABLE OF CONTENTS 1 Introduction ........................................................................................................................................... 3

1.1 1.2 Purpose and Scope ........................................................................................................................................3 Intended Audience ..........................................................................................................................................3

2 3 4

Target Workloads and Use Cases ....................................................................................................... 4 Features of Infinite Volume ................................................................................................................. 5 Infinite Volume Deployment ................................................................................................................ 5
4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 Supported Configuration .................................................................................................................................5 Components of an Infinite Volume ..................................................................................................................6 Aggregate Configuration .................................................................................................................................6 Cluster Configuration ......................................................................................................................................8 Vserver Configuration .....................................................................................................................................9 Configuration of the Infinite Volume ................................................................................................................9 Mounting the Infinite Volume.........................................................................................................................10 Maxfiles and Maxdirsize ................................................................................................................................10 Upgrading and Reverting ..............................................................................................................................11

Infinite Volume Operations ................................................................................................................ 11

5.1 5.2 Managing with System Manager 2.1 .............................................................................................................11 Capacity Management ..................................................................................................................................12

Data Protection ................................................................................................................................... 14

6.1 6.2 6.3 Snapshot Copies...........................................................................................................................................14 Protection for Namespace Constituent .........................................................................................................15 SnapMirror for Infinite Volume ......................................................................................................................16

Storage Efficiency .............................................................................................................................. 19

References ................................................................................................................................................. 20 Version History ......................................................................................................................................... 20

LIST OF TABLES
Table 1) Target use cases for Infinite Volume. ...............................................................................................................4

LIST OF FIGURES
Figure 1) Infinite Volume overview. ............................................................................................................................... 3

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

1 Introduction
NetApp Infinite Volume is a software abstract hosted over Data ONTAP 8.1.x operating in Cluster-Mode that provides scalable capacity. Infinite Volume is similar to a NetApp FlexVol volume in Data ONTAP except that Infinite Volume can scale up to 20PB of storage capacity without any operational complexity involved with the management of multiple smaller volumes. Infinite Volume provides a single mount point accessible over the NFS v3 protocol, and it integrates with NetApp technologies and products such as NetApp Snapshot technology, deduplication, compression, NetApp SnapMirror technology, and OnCommand System Manager. An Infinite Volume is a compound volume that consists of multiple constituent volumes, namely, a single namespace constituent and multiple data constituents. It also includes a namespace redirection layer that provides access to data files in the data constituents because the data files are not directly visible to the client. Figure 1 illustrates Infinite Volume as a single large container with billions of files stored in multiple data constituents.
Figure 1) Infinite Volume overview.

1.1

Purpose and Scope

This document describes the best practices and recommendations for NetApp Infinite Volume, introduced in Data ONTAP 8.1.x operating in Cluster-Mode. This document provides guidance and best practices for the configuration, installation, and operation of Infinite Volume, but it does not include the detailed configuration steps. For the configuration procedure, refer to Data ONTAP 8.1.x operating in ClusterMode documentation. This guide is specific to Infinite Volume and does not cover the general best practices for Data ONTAP operating in Cluster-Mode or FlexVol volumes.

1.2

Intended Audience

This document is for technical audiences such as storage administrators, architects, system engineers, professional services engineers, and field personnel responsible for architecting and deploying NetApp Infinite Volume. This guide assumes knowledge of NetApp storage systems, management of Data ONTAP Cluster-Mode, Data ONTAP replication technologies, and flexible volumes.

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

2 Target Workloads and Use Cases

The Infinite Volume was developed to provide a scalable, cost-effective solution for big content workloads. Specifically, Infinite Volume addresses the requirements for large, unstructured repositories of primary data that are also known as enterprise content repositories. Enterprise content repositories can be subdivided into workloads with similar access patterns and data protection, protocol, and performance requirements. The first version of NetApp Infinite Volume focuses on a subsegment of enterprise content repositories called fixed content repositories. A fixed content repository is characterized by input/output (I/O) patterns in which data is written once and seldom read. Fixed content, by nature, does not change, nor do updates occur. However, because this data is used for normal business operations, the content must be kept online for fast retrieval, rather than moved to secondary storage. Table 1 describes the target use cases for Infinite Volume.
Table 1) Target use cases for Infinite Volume.

Attribute

Product Capability Infinite Volume in Data ONTAP 8.1.x

Target Workload Requirements Fixed Content Repository

Reference rich media Video, images, audio Web content HTML, ASP, PHP, WebApps Maximum price/performance ratio Write once, read often Sequential access, mix of read and write Multi-PB volumes <5GB/sec Snapshot, SnapMirror

File types

Unstructured data over NFSv3

Requirement

Large-scale single-file system with high availability and storage efficiency Optimized for sequential read or mix of random I/O with moderate performance

Online content repository for transcoded media Write once, read seldom Sequential access, primarily read ~5PB volume <5GB/sec Snapshot, SnapMirror

Typical access pattern

Capacity Throughput Data protection

Up to 20PB volume or up to two billion files Up to 5GB/sec in a cluster RAID-DP technology, Snapshot, SnapMirror, NFS-based tape backup

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

3 Features of Infinite Volume

Infinite Volume features can be grouped into the following categories: Manageability: The Infinite Volume manageability features include: A single namespace for the cluster Scalability in capacity and performance Capacity balancing at ingest Simplified setup RAID-DP protection for disks Snapshot copies at the Infinite Volume level SnapMirror at the Infinite Volume level NFS-mounted tape backup Deduplication Compression Thin provisioning Storage failover Cluster-Mode protection Nondisruptive operations

Data protection: Infinite Volume supports the following methods of data protection:

Efficiency: Infinite Volume supports the following NetApp storage efficiency features:

Resiliency: Infinite Volume supports the use of NetApps storage resiliency technologies, such as:

Performance acceleration with Flash Cache

For details about Infinite Volume features, refer to TR-4037: Introduction to NetApp Infinite Volume.

4 Infinite Volume Deployment

The Infinite Volume solution has specific requirements and configurations for deployment in a Data ONTAP 8.1.x Cluster-Mode environment. Its important to understand the supported configuration and deployment steps, because an Infinite Volume is targeted for very specific use cases and workloads. Although Infinite Volume creation may appear very similar to FlexVol volume creation, there are several differences.

4.1

Supported Configuration

An Infinite Volume is a single, large, scalable file system that supports up to 20PB (petabytes) of storage or up to two billion files. Infinite Volume hosted over Cluster-Mode supports the following configuration. FAS/V6280 and FAS/V3270 systems Up to a maximum of 10 nodes or 5 HA pairs in a cluster NFSv3 protocol only Maximum raw capacity of 20PB A maximum of 2 billion files Maximum throughput of 5GB/sec for a 10-node cluster

The Infinite Volume configuration in Cluster-Mode requires a dedicated cluster because it supports one Infinite Volume in a Vserver and one Vserver for the entire cluster, with the exception of the root Vserver.

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

4.2

Components of an Infinite Volume

Aggregate. This is a collection of disk drives configured with RAID-DP. Node. This is the NetApp storage controller participating in Cluster-Mode. Namespace constituent. This is a type of flexible volume that contains the directory hierarchy and file names. The file names are associated with redirector (pointer) files, each of which refers to a data file in a data constituent. Only one active namespace constituent is allowed in an Infinite Volume. Data constituent. This is a type of flexible volume that stores data files. Multiple data constituents are allowed in an Infinite Volume; however, a minimum of two data constituents is required. Vserver with Infinite Volume. This is a virtual server with a special attribute that hosts the Infinite Volume. NFSv3 protocol. This protocol provides access to the Infinite Volume from an NFS client through a single mount point.

An Infinite Volume includes the following components.

4.3

Aggregate Configuration

On a NetApp FAS system, aggregates can be created using RAID 4 or RAID-DP. NetApp highly recommends using RAID-DP for Infinite Volume because it offers excellent resiliency and nondisruptive background firmware updates for the disk drives. For optimal performance and resiliency NetApp recommends configuring RAID groups using RAID-DP in groups of 20 disks (18 for data and 2 for parity). For the Infinite Volume in Cluster-Mode, aggregates are dedicated to data constituents; that is, one data constituent per aggregate. Aggregates can be created by using System Manager or the Data ONTAP command line interface (CLI). The following command creates the aggregate through the CLI. A RAID-DP aggregate will be created by default.
cluster::> aggr create -aggregate <aggregate name> nodes <nodename> -diskcount disktype <SAS|FCAL|SATA|BSAS> <# of disks> -

In System Manager 2.1, when creating the aggregates as part of the Create Vserver wizard and the Vserver with Infinite Volume option is selected, only RAID-DP aggregates are allowed to be created. When creating the aggregates using the Aggregate Create wizard, both RAID 4 and RAID-DP aggregates are allowed to be created. For maximum usable capacity with optimal performance and resiliency, NetApp recommends creating each aggregate as follows: FAS6280: 40 3TB disks configured in 2 RAID-DP groups of 20 disks (18 data + 2 parity) FAS3270: 20 3TB disks configured in 1 RAID-DP group (18 data + 2 parity)

If this configuration is not possible, NetApp recommends creating an aggregate 10% larger than the desired constituent size up to the maximum FlexVol size supported on the platform. Because an aggregate is allowed to hold only one constituent and the constituent cant exceed the maximum supported FlexVol size, NetApp does not recommend creating an aggregate with the maximum allowable capacity of 162TB for the FAS6280 and 105TB for the FAS3270. For the FAS6280, the maximum aggregate size for a constituent should be no larger than 100TB; for the FAS3270 it should be no larger than 68TB. To see the size of your aggregates you can look at the following: Systems Manager, Configure Aggregates portion of wizard, Available Space value Systems Manager Aggregates tab, the Total Space for the aggregate Command line, the value shown for Size from the output of the aggr show command

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

For maximum aggregate throughput, distribute the aggregates evenly across all the available nodes in the cluster. For example, if there are 4 nodes in a cluster and 12 aggregates need to be created, you should create 3 aggregates per node to get maximum throughput from the entire cluster. If there are aggregates of different sizes in a cluster, System Manager selects the largest available aggregate for the namespace constituent and then selects the smallest available aggregate to place the Vserver root volume on. Aggregates can be either created before creating the Vserver with Infinite Volume or as part of the Create Vserver wizard in System Manager. By default the System Manager Vserver with Infinite Volume wizard will create aggregates with a fixed RAID group size of 16. If you want to create a different RAID group size such as the recommended maximum configuration, which includes 20-drive RAID groups, aggregates should be created beforehand using the standard Cluster Administration Aggregate Create tool. After that, the Create Vserver with Infinite Volume wizard should use all the available aggregates to create the Infinite Volume spread across all these aggregates. Best Practices Create aggregates no larger than 100TB on the FAS6280 and 68TB on the FAS3270 to achieve the maximum usable capacity for each individual constituent. Creating aggregates larger than this will result in wasted space. Create aggregates that will be used for the namespace and data constituents as follows (this requires the aggregates to be created using the Cluster Administration Aggregate Create tool instead of having the Create Vserver with Infinite Volume wizard create them): - FAS6280: 40 3TB disks configured in 2 RAID-DP groups of 20 disks (18 data + 2 parity) - FAS3270: 20 3TB disks configured in 1 RAID-DP group (18 data + 2 parity) Make sure the aggregates are created with the RAID type set to RAID-DP. Design the aggregates taking the namespace aggregate and the namespace data protection (DP) mirror aggregate into consideration, and use the disks accordingly. Create the namespace DP mirror aggregate only after the creation of the Infinite Volume. Create an aggregate for the namespace DP mirror that is equal in size to the active namespace aggregate. A minimum of four aggregates is required in the cluster to create an Infinite Volume; one aggregate is required for the Vserver root volume, one for the namespace constituent, and two for the data constituents. Keep the Vserver root aggregate to be the smallest possible three-disk RAID-DP aggregate and allow the System Manager Create Vserver wizard to create this aggregate as part of the Configure Aggregate option. You can rename this aggregate as appropriate for your Vserver. Before creating a Vserver for Infinite Volume, create all aggregates except for the aggregate needed for the namespace DP mirror, or else that aggregate will be consumed during the creation of the Infinite Volume. Verify that the aggregates used for namespace and data constituents are not larger than 100TB on the FAS6280 and 68TB on the FAS3270. If creating an Infinite Volume on a FAS3270, validate that the total Infinite Volume size the wizard is going to create does not exceed ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB). If it does you need to manually decrease the value to no more than (62.9TB * <# data constituents>) 10TB for a successful Infinite Volume creation. For better throughput, distribute the aggregates evenly across all the available nodes in the cluster.

Note: Only one constituent per aggregate is allowed. Infinite Volume supports only 64-bit aggregates. The WAFL file system or RAID performance characteristics of aggregate sizes will impact the overall system performance.

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

4.4

Cluster Configuration

A Data ONTAP cluster is a group of nodes that can function as a single powerful system. Before an administrator decides to create an Infinite Volume in a Data ONTAP cluster, its important to understand the following requirements for a cluster. Infinite Volume requires a dedicated Data ONTAP cluster, because there is only one Infinite Volume supported in one cluster. In Data ONTAP 8.1.x, an Infinite Volume cannot coexist with a FlexVol volume. The only FlexVol volumes that can be present in the cluster are the node root volume and the Vserver root volume. Both of these volumes have their own dedicated aggregates. An Infinite Volume supports a maximum of 10 nodes in a cluster with 2PB raw capacity per node. Apart from the node and Vserver root aggregates an Infinite Volume can span a maximum of: - FAS/V6280 with DS424x: 140 (1 NS + 1 NS Mirror + 138 DCs) o 28 aggregates per HA pair

FAS/V6280 with DS4486: 175 (1 NS + 1 NS Mirror + 173 DCs) o 35 aggregates per HA pair

FAS/V3270 with either DS424X or DS4486: 235 (1 NS + 1 NS Mirror + 233 DCs) o 47 aggregates per HA pair

To deploy an Infinite Volume solution, the first step is to bring up a basic Data ONTAP 8.1.x cluster on either a FAS/V6280 or FAS/V3270 platform. To create a basic cluster, follow the procedure described in Data ONTAP 8.1.x Cluster-Mode documentation. The cluster should have only the node root aggregates and volumes created and should not have any regular Vserver or FlexVol volumes existing in the cluster. Once the cluster is created, you can use System Manager 2.1 to manage this cluster and complete the Infinite Volume configuration. Once a cluster has been set up for an Infinite Volume, it will be restricted for use only with that Infinite Volume. If you want to use this cluster for FlexVol volumes, you must delete the Infinite Volume, remove the Vserver with Infinite Volume, and create a Vserver with FlexVol and flexible volume for Cluster-Mode. Best Practices Dedicate a cluster for Infinite Volume. Deploy the Infinite Volume on a new cluster running Data ONTAP 8.1.x Cluster-Mode, or clean up an existing cluster by removing the Vservers and nonroot FlexVol volumes. If deploying Infinite Volume on an existing cluster, make sure that the existing Vservers and regular FlexVol volumes are removed. The cluster should essentially be a new cluster, with only node root volumes.

Note: Only System Manager 2.1 or the Data ONTAP CLI will support the management of a cluster associated with Infinite Volume. Only one Vserver with Infinite Volume is allowed in a cluster. Only one Infinite Volume is allowed in a Vserver or in a cluster for Data ONTAP 8.1.x.

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

4.5

Vserver Configuration

Infinite Volume resides in a special Vserver in a cluster called Vserver with Infinite Volume with a special attribute only for Infinite Volume. This special Vserver can be created through the Data ONTAP CLI or the System Manager 2.1 Create Vserver wizard by selecting the Vserver with Infinite Volume option. For configuration details, refer to Data ONTAP 8.1.x Cluster-Mode documentation. If any existing Vservers are detected during the creation of the Vserver for Infinite Volume, an error message will be displayed and the creation of the Infinite Volume will fail. Once you have a Vserver with Infinite Volume, you can know that it is the only Vserver in the cluster and you will not be able to create nonroot FlexVol volumes in it. The Create Vserver wizard requires a minimum of three aggregates to create an Infinite Volume: one for the namespace constituent and two for the data constituents. Additionally, it picks the smallest aggregate available in the cluster for the Vserver root volume. Best Practices Make sure that there are no existing Vservers in the cluster and that the cluster is as good as new, with only node root volumes. For the Vserver root aggregate, NetApp recommends allowing System Manager to create the smallest possible three-disk RAID-DP aggregate for the Vserver root volume in the Create Vserver wizard. For better throughput, create multiple data LIFs, at least one for each NIC in the cluster. Reserve some NICs for cluster-peer interfaces.

Note: Only one Vserver with Infinite Volume is allowed in a cluster. In Data ONTAP 8.1.x, the Vserver creation function is enabled only through a product variance request (PVR) process. Contact your local sales representative for details about the PVR. If creating an Infinite Volume on a FAS3270, validate that the total Infinite Volume size the wizard is going to create does not exceed ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB). If it does you will need to manually decrease the value to no more than ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB).

4.6

Configuration of the Infinite Volume

An Infinite Volume is a single large container accessible over the NFSv3 protocol. It can be created through the Data ONTAP CLI or the System Manager 2.1 Create Vserver wizard by selecting the Vserver with Infinite Volume option. For configuration details, refer to Data ONTAP 8.1.x Cluster-Mode documentation. Before creating an Infinite Volume, its important to understand the following requirements. Infinite Volume supports only the NFSv3 protocol. Only one Infinite Volume can be created in a Vserver and a cluster. Only one namespace constituent can be created in an Infinite Volume. Infinite Volume uses only UNIX security.

An Infinite Volume can sometimes be in a mixed state. A mixed state is only applicable to Infinite Volumes and it indicates that one or more constituents of the Infinite Volume are offline. This state can occur when: One of the nodes is offline. One of the aggregates is offline. One of the constituents is offline.

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

4.7

Mounting the Infinite Volume

An Infinite Volume is mounted on an NFSv3 client as a regular volume by using the same mount command. There is only one path or junction point for Infinite Volume: the junction path of the namespace constituent. By default the name for this junction path is /NS when the volume is created using the CLI, but it can be changed to any name. An example of a simple mount command is:
# mount Cluster01-lif1:/NS /mnt/bigvol

The Infinite Volume does not support NFS over UDP and only supports NFS over TCP. Most of the UNIX clients support rsize and wsize (read/write block size) as 64k by default, but on Red Hat Enterprise Linux version 5.3-5.6, Infinite Volume should be mounted with rsize and wsize, specified as 64k.

Export Policy
The Vserver with Infinite Volume and the Infinite Volume are configured with multiple default export policies. System export policies: The following policies protect the systems internals. repos_root_readonly_export_policy: Applicable only for the Vserver root volume. repos_restricted_export_policy: Applicable only for the data constituents. repos_namespace_export_policy: Applicable only for the namespace constituent.

Client access export policies: The following policy controls client access to the Infinite Volume. By default, the client access export policies control a clients read and write access to files in the Infinite Volume. You can add export rules to the export policies to further control client access. The storage system creates the repos_root_readonly_export_policy export policy when you create a Vserver with Infinite Volume and then creates the repos_namespace_export_policy and repos_restricted_export_policy export policies when you create the Infinite Volume.

An Infinite Volume can use only one export policy at a time. You can add export rules to the repos_namespace_export_policy to restrict client access as needed. For more information on the export policies for Infinite Volume, refer to the File Access and Protocol Management Guide for Data ONTAP 8.1.x Cluster-Mode. Best Practices Do not modify the repos_root_readonly_export_policy and repos_restricted_export_policy export policies, because these export policies restrict the client from accessing the system files. The name of the junction path for an Infinite Volume can be only a single element, for example, /NS. It cant be two elements, such as /vol/NS or /NS1/NS2.

4.8

Maxfiles and Maxdirsize

An Infinite Volume is capable of supporting a maximum of two billion files, which is based on the number of redirector files supported in the namespace constituent. The namespace constituent size can grow up to a maximum capacity of 10TB to accommodate the two-billion-file count. In Data ONTAP, a single file can be as large as 16TB in size. The limits imposed on the maximum number of files and the maximum number of subdirectories for a regular FlexVol volume in Cluster-Mode are also applicable to the Infinite Volume. If the namespace constituent size is 10TB or more, adding more disks to the aggregate containing the namespace does not increase the space in the namespace constituent and the maximum number of files. It does help performance by leveraging the additional spindles in the aggregates.

10

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

If the namespace constituent size is less than 10TB, adding more disks to the aggregate containing the namespace (explained later in the document) does not automatically increase the maximum number of files in the namespace constituent. It has to be manually increased at the Infinite Volume level using the CLI. In Data ONTAP 8.1 and higher, there is no limit to the number of subdirectories; a user can continue to create subdirectories as long as the maximum directory size and the maximum number of files limit are not exceeded. For more information on this topic and the commands, refer to the Data ONTAP 8.1.x Logical Storage Management Guide. It is possible that an existing data file cannot be appended to expand the file size even if the file size is less than 16TB and despite there being 16TB or more free space available in an Infinite Volume. The reason is that the Infinite Volume free space is the total sum of the free space of all the data constituents. However, the maximum size to which a file can grow is limited by the free space of the specific data constituent in which the file resides. It might also be difficult to diagnose this problem, because it's hard to determine in which data constituent the file is located. NetApp advises contacting NetApp Global Support for help in such a situation.

4.9

Upgrading and Reverting

The Infinite Volume feature is available only in Data ONTAP 8.1.x Cluster-Mode. An upgrade from previous versions of Data ONTAP may have existing Vservers and FlexVol volumes. After upgrading, to create an Infinite Volume, first remove the existing Vservers and nonroot FlexVol volumes (except for node root volumes). Once the cluster is ready as a new cluster, you can proceed with the new Vserver and Infinite Volume creation. Similarly, a downgrade from Data ONTAP 8.1.x Cluster-Mode to a previous version requires the deletion of the Infinite Volume and the Vserver with the Infinite Volume and the removal of the namespace mirror, if created.

5 Infinite Volume Operations

5.1 Managing with System Manager 2.1

NetApp System Manager 2.1 provides easy manageability of the Infinite Volume feature. System Manager 2.1 has the following built-in wizards to manage the Infinite Volume. Create Vserver. This wizard helps to create a Vserver for the Infinite Volume, to create Infinite Volume, and to enable the Vserver for NFSv3 access with data LIF creation and export policies. SnapMirror Relationship Create. This wizard helps to create a SnapMirror relationship between a source Infinite Volume and a disaster recovery site so that the source Infinite Volume data can be replicated to a SnapMirror destination volume for disaster recovery purposes.

In addition to these wizards, System Manager provides an interface for the management of cluster, aggregates, Snapshot copies, storage efficiency, NFS protocol, export policies, expansion of the volume, space monitoring, and namespace. Best Practice Make sure that the cluster is added to System Manager before starting the management of Infinite Volume or the Vserver. Note: Follow the best practices for individual functions separately as documented in this guide, even if those functions are managed through System Manager.

11

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

Note: NetApp recommends using System Manager 2.1 for managing the Vserver and Infinite Volume. Data ONTAP 8.1.x requires a PVR to get the Vserver enabled for Infinite Volume. Contact your local sales representative for details about the PVR. System Manager allows manageability at the Infinite Volume level, not at the individual constituent level. If there are specific tasks that need to be performed at the constituent level, contact NetApp Global Support for guidance.

5.2

Capacity Management

Auto Capacity Balancing for File Placement

Infinite Volume provides capacity balancing across the nodes as data is ingested. When new files are written to an Infinite Volume, they are distributed evenly across data constituents on a round-robin basis, with preference for data constituents that have the most available space. In general, incoming files are created in the following manner. Data constituents with more than 80% used space generally do not receive incoming files. Data constituents with less than 80% used space receive incoming files in a round-robin fashion. Data constituents that are empty or nearly empty receive most of the incoming files.

Over time, this method of creating files tends to balance space usage across the data constituents. This is an automatic background function that helps to avoid problems created by uneven capacity balancing. If a file cannot be created on a data constituent, Data ONTAP tries to create the file on other data constituents before returning an error to the client. When data constituents become unbalanced due to adding new data constituents, Data ONTAP preferentially places data for new files in empty or underutilized data constituents. This is an automatic process that does not require intervention from the storage administrator or the user. The automatic process, however, does not fix the imbalances caused by file size changes or the deletion of existing files. Best Practices Infinite Volume capacity utilization and AutoSupport events should be carefully monitored. To avoid hot spots, NetApp recommends adding storage space when the data constituent capacity approaches 60%.

Capacity Addition by Adding Disks to the Cluster

You can use the volume show command to check the status of the Infinite Volume and the space usage of the volume.
cluster::> volume show -volume iv1 Vserver Volume Aggregate State Type Size Available Used% --------- ------------ ------------ ---------- ---- ---------- ---------- ----vs1 iv1 online RW 3072TB 205TB 90%

Although NetApp recommends creating aggregates using the recommended configuration documented in the aggregate configuration section, above, for use with Infinite Volume, it is possible to create smaller aggregates. In this scenario, it might be possible to add more disks to existing aggregates up to the maximum size (100TB for FAS6280 and 68TB for FAS3270) to increase the capacity of the Infinite Volume. Before adding disks, check the available space of the aggregates to determine which of the aggregates are running at the same capacity. For example, if there are five aggregates that are 25TB in size running at 60% used and 10 aggregates that are 50TB in size running at 25% used, you should evenly add more

12

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

disks to those five 25TB aggregates. Remember that the maximum recommended aggregate size for a FAS6280 is 100TB and for FAS3270 it is 68TB. Space utilization of the aggregates can be checked by using the aggr show command, disks can be added by using the aggr add-disks command, or aggregate space can be managed through System Manager.
cluster::> aggr show iviv_aggr* Aggregate Size Available Used% State #Vols Nodes RAID Status --------- -------- --------- ----- ------- ------ ---------------- -----------iv_aggr_001 80TB 72TB 10% online 1 beacon-a raid_dp iv_aggr_002 80TB 72TB 10% online 1 beacon-b raid_dp iv_aggr_003 80TB 72TB 10% online 1 beacon-a raid_dp cluster::> aggr add-disks -aggregate iv_aggr_001 -diskcount 14 cluster::> aggr add-disks -aggregate iv_aggr_002 -diskcount 14 cluster::> aggr add-disks -aggregate iv_aggr_003 -diskcount 14

Best Practices When increasing the size of aggregates by adding more disks, always add the same number of disks to the undersized aggregates to evenly increase the capacity. Additional disk drives should be added only to the data constituent aggregates. Adding disks to the namespace constituent will not increase the capacity of the Infinite Volume for storing content. Adding disks to expand an aggregate beyond 100TB on a FAS6280 or 68TB on a FAS3270 will not result in any additional usable space for the Infinite Volume.

Capacity Addition by Adding Aggregates and/or Nodes to the Cluster

Infinite Volume capacity can also be increased by adding new aggregates with the newly installed disks or by adding more nodes with storage to the cluster. New aggregates can be created from System Manager or the CLI by using the aggr create command. Similarly, new nodes can be added to the existing cluster by launching the System Managers advance option for managing a cluster or through the CLI by using the cluster join command. After adding the new aggregates/nodes to the cluster, make sure to use volume resize in System Manager or run the volume modify command on the CLI to expand the volume. The following example shows the volume modify command to increase the size of the Infinite Volume.
cluster::> volume modify -vserver vs1 -volume iv1 -size +2PB

If a SnapMirror relationship was set up for this Infinite Volume, increase the size of the destination volume before increasing the source volume or the next SnapMirror update will fail. Best Practice If the existing nodes in a cluster are close to the maximum number of aggregates per HA pair limit (35 for FAS6280 with DS4486, 28 for FAS6280 with DS424x, 47 for FAS3270 with either drive shelf), NetApp recommends adding more nodes to the cluster and creating the aggregates on new nodes rather than on the existing nodes for better performance.

Expanding the Infinite Volume

To accommodate data growth, the clusters storage capacity can be increased by adding disks to existing aggregates, adding aggregates, or adding more storage nodes to the cluster as described in the previous section. The Infinite Volume needs to be resized or expanded to accommodate the additional capacity added to the system in the following scenarios. When an Infinite Volume is thin provisioned (for example, when a volume much larger than the actual available space from the aggregates is created) and at some point more disks are added to the

13

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

existing aggregates containing data constituents. In this case, there is no need to resize the Infinite Volume unless the actual new capacity of the volume is more than what was provisioned. When the Infinite Volume is thin provisioned and new aggregates are added to existing nodes, or new nodes are added to the cluster. In this case, you must resize the Infinite Volume to create new data constituents on the new aggregates. When the Infinite Volume is thin provisioned and the size of the aggregate containing the namespace constituent is greater than 10TB. If new disks are added to the namespace aggregate, there is no need to resize the Infinite Volume. This is because the namespace constituent has the volume guarantee set to "volume and therefore its size does not increase by the addition of more disks to this aggregate. In this case, adding more disks will result in improved performance for new write and subsequent read requests because of the increased number of spindles in the aggregate.

If the Infinite Volume is thin provisioned and the size of the aggregate containing the namespace constituent is much smaller in size, for example, 5TB, it can still grow to accommodate a 10TB namespace constituent. In that case, you must resize the Infinite Volume to include the increased size of the namespace constituent. Note: If you resize an Infinite Volume on FAS3270 systems, the maximum size is calculated incorrectly in the tool. Dont set this value to larger than ((62.9TB * (<# aggregates in the Infinite Volume> - 2)) + 20TB) if you have a namespace mirror and ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB) if you dont have a namespace mirror for a FAS3270 configuration. Any attempt to resize an Infinite Volume above this maximum will fail. The maximum size is calculated correctly for the FAS6280. Best Practices Before creating an Infinite Volume, make sure the minimum number of aggregates is available in the cluster and assigned to the Vserver. Make sure to use the UNIX security type. Create the Infinite Volume with the volume guarantee set to none for thin provisioning. This is the default setting when creating the volume through System Manager, but it needs to be set when using the CLI. Do not attempt to remove any aggregates from the Infinite Volume without first properly planning for moving the data out of these aggregates, since shrinking an Infinite Volume is not supported.

6 Data Protection
6.1 Snapshot Copies

An Infinite Volume supports Snapshot copies at the Infinite Volume level. Snapshot copy names should not exceed 255 characters. In System Manager, the snapshot create and snapshot restore operations on Infinite Volume can be performed in the same way as with FlexVol volumes. The Snapshot copies of Infinite Volume have an additional property called state. This property has three values: Valid. This state indicates that the Snapshot copies of all constituents are intact. Partial. This state indicates that the Snapshot copies of one or more data constituents are missing. Invalid. This state indicates that the Snapshot copy of the namespace constituent is missing.

If for some reason the namespace constituent is offline or missing, all the Snapshot copies will start indicating an invalid state. Once the namespace constituent is recovered and brought online, the invalid Snapshot copies will return to a valid state. Snapshot copies in the invalid state cannot be restored. Snapshot copies in the partial state cannot be restored in the admin mode. Contact NetApp Global Support if it is necessary to restore a partial Snapshot copy.

14

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

If a namespace resiliency (DP) mirror is configured on a cluster, the total number of Snapshot copies available for an Infinite Volume is 253. This is because the namespace resiliency mirror uses two Snapshot copies for the data protection SnapMirror relationship. The Snapshot copies for Infinite Volume cannot be set for autodelete. However, when you manually delete Snapshot copies of Infinite Volumes, you can run the delete operation in the background. For details on the configuration of Snapshot copies, refer to the Data ONTAP 8.1.x Cluster-Mode Data Protection Guide. Best Practices NetApp does not recommend that a Snapshot policy have a schedule for less than an hour. It is important to reserve enough disk space for Snapshot copies so that the active file system always has space available to create new files or modify existing ones. Do not attempt to rename a Snapshot file, because this is not supported. Do not remove invalid state Snapshot copies without verifying why they became invalid. If the namespace constituent has recovered correctly and invalid Snapshot copies still appear, they can be deleted because they are just occupying space. Infinite Volume must be unmounted before executing the volume snapshot restore command. Infinite Volume must be online to be able to perform any Snapshot operation (create, delete, restore).

6.2

Protection for Namespace Constituent

The namespace constituent in the Infinite Volume keeps the redirector pointer files for every data file on the data constituents. If the namespace constituent is not available at any point in time, the file operations on recently accessed files (read, write, get/set attrs) and so on will continue to work (due to the caching of file handle mappings in memory). Namespace or directory operations (mkdir, ls) and so on will not work. To mitigate risk, you can set up an intracluster data protection (DP) mirror for the namespace constituent on other aggregate(s), preferably on another shelf, and another node in the cluster. This namespace mirror can be periodically updated based on the recovery point objective (RPO) requirements. A SnapMirror license is not required for this special mirror. The process to create a namespace mirror and to recover the data from the namespace constituent requires diagnostic-mode commands and should be undertaken only with the help of NetApp Global Support. Best Practices Make sure a data protection mirror is created for the namespace constituent, preferably on another shelf or another node in the cluster. Make sure an aggregate for the namespace DP mirror is available in the cluster, which is ideally the same size as the active namespace aggregate. If it is not easy to keep the same aggregate configuration for the RAID group size, number, and type of disks for the DP mirror, it can be made just large enough to contain the namespace constituent. If the aggregate for the namespace DP mirror does not have the same configuration as the active namespace aggregate, performance by the new namespace constituent (the former namespace DP mirror) will be degraded.

Note:

15

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

6.3

SnapMirror for Infinite Volume

NetApp SnapMirror replication provides easy recovery from a disaster for the Infinite Volume. The entire Infinite Volume can be replicated to another disaster recovery site using a single SnapMirror relationship without managing individual data constituent SnapMirror relationships. The SnapMirror configuration and recovery can be managed through System Manager 2.1. Note: A Vserver with Infinite Volume is only allowed to be mirrored to another Vserver with Infinite Volume. This means that both the source and the destination volumes must be Infinite Volumes. Only a one-to-one SnapMirror relationship is allowed between Infinite Volumes. The secondary Infinite Volume cluster is dedicated to be the only SnapMirror destination for the primary volume. The SnapMirror destination must be the same system model as the source. In other words if the source Infinite Volume resides on a FAS/V6280, then the SnapMirror destination must also be a FAS/V6280.

Setting Up SnapMirror
The process for setting up the SnapMirror for Infinite Volume and FlexVol volumes is the same. To set up the disaster recovery cluster for the Infinite Volume, perform the following steps in the order listed. 1. Make sure SnapMirror is licensed on the source and destination clusters. 2. Create Vserver and aggregates on the destination cluster identical to those on the source cluster in terms of count and sizes. Verify that the SnapMirror destination cluster contains the same system model as the source. In other words, if the source Infinite Volume resides on a FAS/V6280, then the SnapMirror destination must also be a FAS/V6280. 3. Create a cluster-peer relationship between the source cluster and the destination cluster using the intercluster network interfaces. To set up a data protection mirror between cluster peers, refer to the Data ONTAP 8.1.x Cluster-Mode Data Protection Guide. 4. While creating the destination Vserver, the Create Vserver wizard will create the destination Infinite Volume (without constituents). Once the SnapMirror relationship is established and initialized, the constituents are created in the destination Infinite Volume. 5. If you are creating the SnapMirror relationships through System Manager, you can create a destination Infinite Volume as part of the SnapMirror Create wizard, if it does not already exist. 6. It is important to create the data protection Infinite Volume on the same system model, with the same size volume as the primary site's volume. Also, create the same number of aggregates of the same sizes as the primary site. Otherwise, the snapmirror create operation will fail. 7. Use the snapmirror create and snapmirror initialize commands to initialize the DP Infinite Volume.

SnapMirror Updates and Schedules

The SnapMirror destination Infinite Volumes can be updated on automatic schedules (by adding schedule parameters during a "create" or "modify" operation), or they can be updated manually. To update a destination Infinite Volume manually, run the snapmirror update command. The SnapMirror update schedule depends on the number of incremental changes in the Infinite Volume that need to be replicated and the bandwidth between the sites. NetApp recommends not scheduling SnapMirror updates to less than an hour.

16

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

Failing Over to the Secondary Site

In case the source Infinite Volume becomes unavailable (due to a disaster or an availability issue), the users can be redirected for failover access to the destination Infinite Volume. To do this, perform the following steps. 1. Issue a snapmirror break command on the destination Infinite Volume. 2. Export the destination Infinite Volume for applications to access data. 3. Restart the applications at the destination cluster to connect to the destination Infinite Volume. Best Practices A cluster-peer relationship and an intercluster interface must be created before setting up the data protection SnapMirror relation for Infinite Volume. An identical cluster configuration including the model of nodes, the Vserver, and an identical set of aggregates (same count and size) should be created on the destination as the source, before creating the SnapMirror relation. The destination volume must have space equal to or greater than the source volume at all times, even during configuration changes. This is required so that there is enough space in the destination data constituents to hold the mirrored data. A SnapMirror relationship must be established between the source and destination disks and volumes before executing the snapmirror initialize command. When adding additional aggregates or nodes, consider adding them in equal numbers (that is, the aggregate count should match) on both the source and destination sites. Perform the snapmirror update to initialize the newly added data constituents in the Infinite Volume. Do not set the SnapMirror update schedule to less than an hourly interval. To increase the capacity of an Infinite Volume on the source cluster, first expand the capacity on the destination cluster by the same or a greater amount, and then perform a volume resize operation. Following this, add capacity on the source cluster and then perform volume resize followed by a snapmirror update command to initialize the newly added constituents to the destination volume. Do not attempt snap delete or snap restore on the mirrored destination Infinite Volume, because it is a read-only volume and these functions are not allowed on a read-only state of the volume. Do not attempt the snap delete or snap restore operation on the base Snapshot copy on the source Infinite Volume or else future updates will not take place. The snapmirror resync operation might be required in case this operation was performed. If SnapMirror cannot find a common Snapshot copy, resync will also fail.

Recovering the Primary Site from the Secondary Site

After the client data access has been failed over to the destination Infinite Volume, users can continue to operate in this configuration and make this volume the new primary volume. However, when the original source cluster is available, you may want to resync the original source Infinite Volume with all the new changes that have been applied to the current primary (originally the destination) Infinite Volume. This can be achieved by performing a flip-resync. A flip-resync essentially makes the original source Infinite Volume a mirror of the current primary volume. This enables higher levels of data protection for the primary data (which now resides on the destination volume).

17

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

To perform a flip-resync, execute the following steps. 1. Execute the snapmirror create command to create a flip-side relationship from the new primary to the original primary:
cluster::> snapmirror create -source-path dr-cluster://dr_vs1/dr_iv1 -destination-path cluster://vs1/iv1 -type DP

2. Resync the original source volume iv1 with the current primary repository dr_ncv1:
dr-cluster::> snapmirror resync -source-path dr-cluster://dr_vs1/dr_iv1 -destination-path cluster://vs1/iv1 -type DP

Note: Note:

If the number of available aggregates is not exactly the same at the destination as at the source, the snapmirror resync, initialize, and update operations will not be successful. After the recovery operation, if the output of volume snapmirror show on the destination does not display healthy, check if there are partial state Snapshot copies at the destination. This means the recovery process was not successful. Check the job progress of the snapmirror resync to determine the cause of the unsuccessful completion.

Continued Mirroring to Source Infinite Volume

For business continuance, an administrator may want to continue operation on the flip-side configuration for a while, using scheduled updates. To set up a flip-side relationship, perform the following steps. 1. Delete the original relationship from the destination cluster:
dr-cluster::> snapmirror delete -source-path cluster://vs1/iv1 -destination-path dr-cluster://dr_vs1/dr_iv1

2. If a relationship for the flip-side relationship is not already created, create a new relationship on the source cluster to facilitate flip-side mirroring:
cluster::> snapmirror create -source-path dr-cluster://dr_vs1/dr_iv1 -destination-path cluster://vs1/iv1 -type DP -schedule 5min

3. If a relationship was already created, the schedule can be provided by using a snapmirror modify command:
cluster::> snapmirror modify -destination-path cluster://vs1/iv1 -schedule hourly

Note:

An administrator may choose to operate in the flip-side configuration for an extended period of time and can potentially make this the ongoing mirroring setup.

Reestablishing the Original Relationship After Recovery

When both the clusters are up and running and the recovery points are close to each other, the administrator may want to reestablish the original relationship. This involves performing the following steps. 1. Stop the client access on the new primary Infinite Volume (the original destination Infinite Volume). 2. Allow the next update to bring the secondary Infinite Volume to the same recovery point as the primary Infinite Volume. This can also be achieved by explicitly performing a manual update. 3. Break the secondary Infinite Volume:
cluster::> snapmirror break -destination-path cluster://vs1/iv1

4. Create a new relationship on the destination cluster to facilitate mirroring from the original source if it doesn't already exist:
dr-cluster::> snapmirror create -source-path cluster://vs1/iv1 -destination-path drcluster://dr-vs1/dr_iv1 -type DP -schedule hourly

18

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

5. Perform a flip-resync on the current primary Infinite Volume to reestablish the original relationship:
dr-cluster::> snapmirror resync -source-path cluster://vs1/iv1 -destination-path drcluster://dr-vs1/dr_iv1

6. Resume the client access to the original primary Infinite Volume (iv1). 7. Delete the flip-side relationship from the source cluster:
cluster::> snapmirror break -destination-path cluster://vs1/iv1

The preceding procedure will reestablish the original relationship that existed prior to failover. The original source cluster will start to rehost the primary Infinite Volume. The destination cluster will host the mirror (secondary) Infinite Volume.

Recovering Failed or Missing Constituents on the Primary Site from a Secondary Site
If any of the constituents (including the namespace constituent) have failed or are missing in the primary Infinite Volume, you can use the snapmirror resync operation to restore the primary volume. The resync operation will create new data protection constituents and perform initialization for the missing constituents. For the remaining constituents, the snapmirror resync operation will perform the resync operation at the constituent level, thereby avoiding full initialization of the constituent. The resync operation determines the deleted constituents based on the current Infinite Volume configuration. Note: An appropriate warning message will be displayed before proceeding with the resync operation.

7 Storage Efficiency
NetApp Infinite Volume supports storage efficiency features such as NetApp deduplication, compression, and thin provisioning. Deduplication is performed at the Infinite Volume data constituent level within an aggregate and not across the Infinite Volume. For example, if the same file was stored twice in the Infinite Volume on two different data constituents, the blocks between the data constituents will not be deduplicated. Deduplication will be performed only on duplicate blocks of data within a single data constituent. Deduplication requires space for metadata in both the data constituent and the aggregate. For details, refer to TR-3966: NetApp Data Compression and Deduplication Deployment and Implementation Guide. Similar to deduplication, compression is done at the data constituent level within an aggregate. Compression works in the same way for the data constituent as it does with FlexVol volumes. Both deduplication and compression features are enabled at the Infinite Volume level. They are set to off by default and can be turned on as needed. NetApp thin provisioning technology is an efficient way to provision storage, because the storage is not preallocated upfront. The Infinite Volume can be thin provisioned and more disks can be added to the aggregates in the Vserver as required. Note: The namespace constituent cannot be thin provisioned, deduplicated, or compressed. The namespace constituent is always volume space guaranteed.

Best Practice Schedule deduplication to occur during off-peak hours.

19

Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode

References
The following references were used in this report. TR-3982: Data ONTAP 8.1 Operating in Cluster-Mode: An Introduction http://media.netapp.com/documents/tr-3982.pdf TR-4037: Introduction to NetApp Infinite Volume http://media.netapp.com/documents/tr-4037.pdf Data ONTAP 8.1 Logical Storage Management Guide https://library.netapp.com/ecm/ecm_get_file/ECMP1138167 Data ONTAP 8.1 Cluster-Mode Data Protection Guide https://library.netapp.com/ecm/ecm_get_file/ECMP1136081 Data ONTAP 8.1.1 Cluster-Mode documentation http://support.netapp.com/documentation/docweb/index.html?productID=61476 TR-3966: NetApp Data Compression and Deduplication Deployment and Implementation Guide http://media.netapp.com/documents/tr-3966.pdf

Version History
Version
Version 1.0 Version 1.1

Date
June 2012 September 2012

Document Version History

Initial release Updated to include support for FAS/V3270

Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer's installation in accordance with published specifications. NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customers responsibility and depends on the customers ability to evaluate and integrate them into the customers operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.

20

2012 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, AutoSupport, Data ONTAP, FlexVol, OnCommand, RAID-DP, SnapMirror, Snapshot, and WAFL are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode Group. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. TR-4078-0612