MPR Design Guide

Brocade SilkWorm Multiprotocol Router SAN Design Guide
Version 1.0
Publication Number: 53-0000552-01 Publication Date: May 7, 2004
Copyright 2004, Brocade Communications Systems, Incorporated. ALL RIGHTS RESERVED. Publication Number: 53-0000552-01
Brocade, the Brocade B weave logo, Secure Fabric OS, and SilkWorm are registered trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. FICON is a registered trademark of IBM Corporation in the U.S. and other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. The authors and Brocade Communications Systems, Inc. shall have no liability or responsibility to any person or entity with respect to any loss, cost, liability, or damages arising from the information contained in this book or the computer programs that accompany it. Notice: The product described by this document may contain open source software covered by the GNU General Public License or other open source license agreements. To find-out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd. Export of technical data contained in this document may require an export license from the United States Government.
Brocade Communications Systems, Incorporated

Corporate Headquarters 1745 Technology Drive San Jose, CA 95110 T: (408) 487-8000 F: (408) 487-8101 Email: info@brocade.com
Asia-Pacific Headquarters Suites 2301-02, 23/F, Cityplaza One, 1111 Kings Road, Taikoo Shing, Hong Kong. Email: apac-info@brocade.com
European Headquarters 29, route de lAeroport Case Postale 105 CH-1211 Geneva 15, Switzerland T: +41 22 799 56 40 F: +41 22 799 56 41 Email: europe-info@brocade.com
Document History
The table below lists all versions of Brocade SilkWorm Multiprotocol Router SAN Design Guide. Document version First Publication (Version 1.0) Publication Number Publication Date 53-0000552-01 May 7, 2004
Table of Contents
Chapter 1
Introduction
1-1 1-2 1-2 1-2 1-3
The SilkWorm Multiprotocol Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FC-to-FC Routing Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FCIP Tunneling Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iSCSI Gateway Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Innovative Multiprotocol Router Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 2
Determining An Appropriate MetaSAN Architecture

2-1 2-4 2-4 2-5 2-5 2-5 2-6 2-6
Tactical And Strategic Use Of The Multiprotocol Router . . . . . . . . . . . . . . . . . Key Considerations For Determining A SAN Architecture. . . . . . . . . . . . . . . . Primary Non-technical SAN Architecture Considerations . . . . . . . . . . . . . . . . . . . . . Primary Technical SAN Architecture Considerations . . . . . . . . . . . . . . . . . . . . . . . . How Big Is Big And How Big Is Too Big? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interconnected Fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A High Port Count Director Or A Router? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MetaSAN Architecture Selection FlowChart. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 3
Multiprotocol Router SAN Design Concepts

3-1 3-9 3-9
Components of a MetaSAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Resiliency & Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
MetaSAN Resiliency & Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10 Multiprotocol Router Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-13 Fabric Locality and IFL Oversubscription Ratios . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-13 iSCSI and FCIP Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17 FCIP Link Utilization Calculations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17 iSCSI Fan-in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-18
Scalability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-22 Scalability of the Backbone Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-23 Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-27 Multiprotocol Router Interconnect Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-28 Edge Fabric to Backbone Fabric Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29 Interfacing the Multiprotocol Router iSCSI with Gigabit LANs . . . . . . . . . . . . . . . . 3-31 Multiprotocol Router FCIP Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-33 Multiprotocol Router Connection Layout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-34
ii
Preface Audience
This document is intended for use by Brocade partners and end users. The technical professionals that use this guide may include storage administrators, SAN administrators, system administrators, SAN architects, systems engineers, and SAN operators that are involved with the design, deployment, and management of Brocade SANs and Multiprotocol Router solutions. The reader is expected to have working experience with Brocade products. General computer system level troubleshooting skills are always important when configuring and designing sophisticated enterprise solutions. System administration or storage administration experience is also helpful in understanding this document.
Document Conventions
The formatting and conventions used in this document are designed to help the reader locate and comprehend information quickly. In addition to the information provided in standard text, there are Guidelines, Notes, and Warnings to help focus the reader on important information. The following table describes the formatting conventions that are used in this book: Convention bold text Purpose
italic text
code text
identifies GUI elements identifies keywords/operands identifies text to enter at the GUI or CLI provides emphasis identifies variables identifies paths and internet addresses identifies book titles and cross references identifies commands in line with text identifies CLI output identifies syntax examples
Notes and Guidelines

The following notices appear in this document: Note: A note emphasizes important information or provides a reference to related information.
Guideline: A guideline provides a tip or a recommendation.
Warning:
Warnings alert you to potential damage to hardware, firmware, software, or data.
The red circle with a slash through it (shown below) indicates that a particular action or type of connection is not recommended. While the action or connection may function, there are better ways to perform the action or make the connection.
References
The reference material is available on the Brocade web sites at: www.brocade.com and www.brocadeconnect.com.
SAN Migration Guide (53-0000360-xx) Brocade SilkWorm Design, Deployment, and Management Guide (53-0000366-xx) Designing Next-Generation Fabrics With Brocade Switches (whitepaper http://www.brocade.com) LAN Guidelines For Brocade SilkWorm Switches (53-0000350-0x) Exploring Brocade ISL Trunking (53-0000263-0x) Core Switch PID Format Update Best Practices (53-0001626-0x) Brocade Fabric OS Features Guide (53-0000395-0x) Zoning Implementation Strategies For Brocade San Fabrics (whitepaper http://www.brocade.com) XPath OS Procedures Guide (53-0000607-0x) XPath OS Reference Manual (53-0000605-0x) SilkWorm Multiprotocol Router Hardware Reference Manual (53-0000611-0x) Introducing Brocade Multiprotocol Routing Services (whitepaper http://www.brocade.com) SAN Connectivity Using The Brocade SilkWorm Multiprotocol Router (53-0000487-01) Brocade Multiprotocol Routing Services Solution Guide: Microsoft Cluster Server Over iSCSI (53-0000488-xx) Brocade Multiprotocol Routing Services Solution Guide: Meta SAN Tape Consolidation (53-0000489-xx) Brocade Multiprotocol Routing Services Solution Guide: FCIP Remote Site Connectivity(53-0000498-xx)
ii
Chapter
Introduction
The Brocade SilkWorm Multiprotocol Router significantly impacts the design and architecture of SANs. The SAN architect has a choice to build large fabrics or to interconnect fabrics. The approach taken by the SAN architect to designing a SAN is a function of many variables, including risk tolerance, availability needs, SAN administrator capabilities, organizational structure, pre-existing SAN equipment, application deployment patterns, and the proximity of SAN devices to each other. The interconnection of fabrics can span distances using protocols such as FC-IP or using native Fibre Channel distance connectivity (i.e. Long wavelength laser technology). The SAN can be extended to enable low cost iSCSI initiators to access the SAN via Ethernet. The Multiprotocol Router can be used tactically, for example to migrate data from one SAN to a new SAN. The Multiprotocol Router can be used strategically as a permanent interconnect, for example to share a large tape silo between multiple SANs. With the Multiprotocol Router, several new concepts and terms are introduced, such as inter-fabric links (IFLs) and routed fabrics. Familiar SAN design concepts, such as resiliency and redundancy also apply to fabrics that are interconnected by the Multiprotocol Router. It is helpful in understanding this document to have read and be familiar with Brocade SilkWorm Design, Deployment, and Management Guide (53-0000366-xx), which is a document that identifies and discusses SAN design and architecture. In this document, the high level process for determining an appropriate SAN architecture is presented in a format that identifies benefits and considerations for a particular approach. Then, at a more detailed and technical level, concepts that apply to fabrics that are interconnected by the Multiprotocol Router are discussed. Included in this discussion are specifics as they relate to FC-to-FC routing, FCIP, and iSCSI.
1.1. The SilkWorm Multiprotocol Router

The SilkWorm Multiprotocol Router (Router) increases the functionality, connectivity, and versatility of todays SANs. The features and capabilities of the Multiprotocol Router help to increase Storage Area Network (SAN) connectivity options and improve resource utilization. The Multiprotocol Router is designed to host routing services which include:
Brocade FC-to-FC Routing Service for SAN connectivity Brocade FCIP Tunneling Service for SAN extension over distance Brocade iSCSI Gateway Service for sharing Fibre Channel storage with iSCSI servers
These services provide new options for connecting SANs and extending SAN benefits over multiple networks, to larger SAN sizes, and across longer distances. The primary advantage of this approach is the ability to interconnect devices between SAN fabrics without merging themthereby providing a more secure and flexible storage networking foundation. This level of SAN connectivity gives organizations a better way to reduce or even eliminate disruptions associated with common operational events, including:
Data migration from 1 Gbit/sec storage arrays to 2 Gbit/sec systems, or migration between logically separated environments such as test/development SANs and production SANs Data center consolidation that requires the movement of data between physical locations Storage and application rebalancing between previously isolated SANs
Brocade SilkWorm Multiprotocol Router SAN Design Guidelines
1-1
Introduction
1.1.1. FC-to-FC Routing Service

The FC-to-FC Routing Service enables devices located in separate SAN fabrics to establish communication without requiring the fabrics to merge into a single large SAN. By using this service, organizations can interconnect devices without having to redesign or reconfigure their entire environment. FC-to-FC routing capabilities enable organizations to connect devices in separate SANs without merging the fabrics, making it easier to support equipment from multiple OEMs and multiple firmware revisions. As a result, organizations can better implement secure, selective resource sharing through Logical SANs while improving resource utilization. Organizations can utilize FC-to-FC routing capabilities to consolidate the backup of multiple SANs in a single location.The key benefits of this centralized backup approach are optimizing the value of backup devices to increase staff productivity, reduced management overhead, and leveraging off-peak network activity. FC-to-FC routing capabilities provide key strategic advantages, such as:
Simplifying SAN design, implementation, and management Providing a seamless way to share resources across multiple SANs without the complexity of physically merging those SANs Creating a more unified SAN environment with easier interconnection and support for SANs and SAN resources
When devices on different SAN fabrics are allowed to communicate through the Multiprotocol Router, the connectivity group is known as a Logical SAN (LSAN). LSANs enable selective, secure resource sharing across multiple SAN fabrics and facilitate scalability by:
Minimizing the risk and complexity of large SAN fabrics Right-sizing SANs based on application and business requirements
1.1.2. FCIP Tunneling Service

The FCIP Tunneling Service enables organizations to extend their Fibre Channel SANs over distances that would be impractical or expensive with native Fibre Channel links, or in situations where dedicated circuits would be impractical but in which IP WAN connectivity already exists. Deploying FCIP on the Multiprotocol Router is more flexible and cost-effective than an external gateway. As a result, organizations have a more manageable way to share resources across geographical boundaries and implement reliable business continuance solutions. The Brocade SilkWorm Multiprotocol Router and routing services help extend SAN capabilities across the enterprise. Using the FCIP Tunneling Service in conjunction with the FC-to-FC routing capability enables two fabrics to remain separate rather than merging them into a single fabric that would permit any-to-any connectivity between all devices. This combined FC-to-FC routing and FCIP approach enables organizations to maintain separate fabrics for administrative simplicity and increased securityenabling a more secure distance-connectivity solution for business continuance.
1.1.3. iSCSI Gateway Service

The iSCSI Gateway Service enables organizations to integrate low-cost Ethernet-connected servers into Brocade Fibre Channel SANs by bridging the iSCSI protocol to the Fibre Channel protocol. This capability allows iSCSI servers to leverage shared SAN resources, improving asset utilization and enabling new applications such as centralized backup. This integration greatly reduces the cost of connecting servers to centrally managed storage and helps provide a cost-effective solution to introduce utility computing into the enterprise. Brocades iSCSI implementation provides a standards-based solution that leverages existing Ethernet infrastructure to simplify both implementation and management. It also reduces costs by eliminating the need to purchase Fibre Channel Host Bus Adapters (HBAs) in order for iSCSI servers to access Fibre Channel storage resources in the SAN.
1-2
Introduction
1.1.4. The Innovative Multiprotocol Router Platform

The Multiprotocol Router is based on the SilkWorm Fabric Application Platform, the industrys first open, intelligent switching platform for hosting multiprotocol routing services and storage management applications within the SAN and the metaSAN. Designed to seamlessly integrate into existing Brocade SAN infrastructures, the Multiprotocol Router provides a single point of control and management for multiprotocol routing services. A key aspect of this approach is the unprecedented capability to support any of the multiprotocol routing services on a port-by-port basis. By integrating multiple services in a central platform, the Multiprotocol Router helps provide a flexible foundation for implementing a utility computing infrastructure and efficient Information Life cycle Management (ILM) within a familiar Brocade SAN environment. As a result, it can be an ideal solution for improving operational efficiency while maximizing the value of both current and future SAN investments. The Multiprotocol Router provides 16 ports in a 2U form factor, with each port operating at 1 or 2 Gbit/sec and capable of supporting Fibre Channel routing, iSCSI protocol conversion, or FCIP data traffic. In this way, the Multiprotocol Router provides the high performance required to run storage applications at line-rate speed. Organizations can manage the Multiprotocol Router with the same tools they use to manage other Brocade SAN switches: a standard command line interface, Brocade Web Tools, or Brocade Fabric Manager. In addition, Brocade Advanced Zoning permits only authorized devices and applications to access data, increasing security and control.
1-3
Introduction
1-4
Chapter
With the initial deployment of SANs, customers only had one choice when it came to sharing equipment -- create one large fabric. If multiple SANs evolved within the enterprise based on application rollout or organizational control, there was only one choice to developing a centralized SAN infrastructure -- migrate the SAN islands into a large SAN. Large SANs have benefits, such as any to any connectivity with high performance and a centralized management model. However, managing a single large SAN may not fit with every enterprise since some IT organizations have multiple storage/systems administration groups. Additionally, while large SANs do have benefits of management simplicity and high performance any-to-any connectivity, there is a trade off consideration -- a large SAN also means a large fault domain and much greater impact should a far reaching error in that SAN be encountered. With the SilkWorm Multiprotocol Router, customers now have a choice to interconnect SAN islands, to build large SANs, or a combination of interconnecting SAN islands of varying sizes. The integration of iSCSI and FCIP extends the reach and value of existing SAN investments. Many customers are coming to a junction regarding how their SAN architecture is going to evolve. For some customers, the decisions is easy. They are confronted with multiple SAN islands that span their enterprise. Merging these islands is not an option nor is keeping these islands separated viable. Other customers see the benefits of large SANs and have the appropriate operational structure to migrate to a few large SANs that may use the Multiprotocol Router tactically to migrate data and as a strategic interconnect for the newly create merged and larger fabrics. Fortunately, with Brocade, the SAN architect has the choice to deploy large SAN fabrics in excess of 1200 ports (please reference your support provider for current scalability support limits) and the choice to connect islands without merging them. It is not a Brocade decision regarding the appropriate SAN architecture for a customer, rather it is a customer decision that is primarily a function of the customers tolerance for risk, SAN administration capabilities, and organizational structure. Note that the term risk is used in this chapter and can relate to objective metrics or just a level if comfort. Risk can also relate to availability of a system, as discussed in section 3.2. Availability on page 3-9. Note: Several new terms and concepts, such as metaSAN, fault domain, and fabric locality are utilized in this chapter. A glossary is available in section 3.1. Components of a MetaSAN on page 3-1.
2.1. Tactical And Strategic Use Of The Multiprotocol Router

The Multiprotocol Router is highly functional and can be used tactically, for example, to facilitate a data migration from legacy storage to new storage, as show in Figure 2-1. Once the SANs are connected, it is a straightforward process to migrate the data in a variety of ways, including the creation of a mirror that spans the old and new storage. Once the mirrors are synchronized, the old storage can be disconnected. This approach simplifies the migration process too. Having existing hosts access new storage located in a new SAN is approximately 75% of the migration process. The remaining 25% of the migration process involves reconnecting the host to the new SAN. This approach vastly reduces the risk by enabling a phased migration. Long term use of the Multiprotocol Router to facilitate any-to-any connectivity between SANs can be problematic due to bandwidth issues, additional sharing steps, and the complexity of having the majority of initiators importing their storage between multiple fabrics. Sharing certainly has value and the Multiprotocol Router can be used strategically to share high value items such as tape silos or large storage arrays. Key to the long term use of the Multiprotocol Router is to maintain communication of devices within a fabric (locality) within the fabric as much as possible to optimize use of the connections between the router and fabrics. See section 3.3.1. Fabric Locality and IFL Oversubscription Ratios on page 3-13 for further
2-1
discussion regarding fabric locality. If fabric locality is not employed, eventually a migration of the SAN devices using the Multiprotocol Router to a common SAN is likely to resolve IFL congestion issues and simplify operations. Another use of the Multiprotocol Router is as a standard interconnect for all SANs in the enterprise to enable sharing of resources for urgent needs. Nobody knows who needs what, when, and where all of the time. Having the ability to quickly share data or resources since there exists connectivity between devices has real value.
Sun
Windows 2000
Sun
Windows 2000
Sun
Windows 2000
Sun
Windows 2000
HDS-1 SAN
Fabric A Fabric B
EMC-1 SAN
Fabric A Fabric B
HDS-1 SAN
Fabric A Fabric B
EMC-1 SAN
Fabric A Fabric B
HDS 9910
EMC 4700
HDS 9910
EMC-2 SAN
Fabric A
EMC-2 SAN
Fabric A
Before After
Sun Windows 2000
Multiprotocol Router
Sun
Windows 2000
Figure 2-1
Data Migration using the Multiprotocol Router
2-2
An example of a strategic use of the Multiprotocol Router is the Storage Service Provider (SSP) model. The Multiprotocol Router also plays a role as a foundation element to any organization that wants to deploy a storage service provider model -essentially a utility for storage, as shown in Figure 2-2. Just plug the host into access reliable, backed up storage in line with capacity and performance requirements. Many companies are implementing a storage service provider internally and some are considering offering this service externally as a commercial venture.
Figure 2-2
Storage service provider (SSP) architecture
2-3
2.2. Key Considerations For Determining A SAN Architecture

Surprisingly, several non-technical variables come into play when it comes to choosing a SAN architecture. Having reliable, scalable, and flexible foundation components such as switches and routers is certainly critical. It is frustrating to design an architecture and not be able to implement the architecture because the components do not inter-operate or the implementation involves too much down time. The top non-technical considerations for selecting a SAN architecture are organizational structure, risk tolerance, and the capabilities of SAN Administration personnel. A primary technical consideration for interconnecting SAN islands instead of performing a SAN consolidation relates to fabric interoperability and host downtime.
2.2.1. Primary Non-technical SAN Architecture Considerations

Many enterprise IT projects are funded by sponsor organizations or internal customers. The projects frequently encompass software and the hardware. Often the hardware for a project includes servers, storage, and SANs along with dedicated personnel to run these systems. Other similar situations exist where multiple SANs, multiple owners, and multiple administrative organizations are present within an enterprise. While a long term goal may involve consolidation to fewer, larger SANs, short term needs and resources may require a less costly and simpler solution if sharing, consolidation, or migration is necessary. Under these circumstances interconnecting the SAN islands is an effective approach. Managing a few large SANs with multiple organizations can be quite challenging. The benefits of managing a few large SANs are significant: reduced levels of management, any-to-any performance, and increased utilization of assets. Many companies see these benefits and have created internal storage service provider models along with a single dedicated organization to manage the storage services. Companies that have one or two organizations responsible for managing storage services can benefit from deploying larger SANs. Under these circumstances the Multiprotocol Router may be used tactically to migrate to a consolidated SAN model as well as for sparse connections to enable intermittent sharing of SAN resources between a few large SANs or dedicated sharing of devices such as tape silos. Risk tolerance can be a very subjective metric. For example, a mission critical application environment may be connected to an isolated storage array due to uncertainties of connecting to a shared storage array, which can provide sufficient capacity and capability. While it may not be possible to quantify why this mission critical application can not be connected to a shared storage array, the decision to maintain segregation is made anyway. Objectively, service level agreements may specify that the worst case outage that can be sustained is a loss of 25% computing resource. If there are 1000 ports of FC connected computing resources in an enterprise, mathematically, an appropriate design would involve the creation of four 250-port SANs as opposed to one 1000-port SAN. The larger the fabric, the greater the impact of an outage and the smaller the fabric, the less impact a fabric outage has. For some customers, a 100-port fabric is large enough, while for other customers a 2000-port fabric is as large as they feel comfortable managing. Frequently, risk tolerance is put aside, as the demand for SAN growth and the simplicity of incrementally growing an existing SAN, versus building a new SAN, drive the expansion of existing SANs past the comfort level of the SAN management team. Use and deployment of the Multiprotocol Router is simple and employs the existing skill set of SAN administrators-- switch setup and zoning skills. Using the Multiprotocol Router, it is possible to connect hundreds of devices spread across many different SANs without any major changes to SAN administrator skills or processes. However, the management of a single, large SAN consisting of hundreds of ports involves the use of different processes, more sophisticated troubleshooting skills, tighter integration with enterprise management systems, stricter change control, and deeper experience. If the desire to realize the benefits of large fabrics exists and the SAN administration team has experience with managing fabrics of 100 or more ports, moving to a large fabric architecture can be effective. Taking a large fabric architecture approach with out the appropriate level of experience require careful thought, the development of new processes, and additional training.
2-4
2.2.2. Primary Technical SAN Architecture Considerations

While the desire may exist to consolidate multiple SAN islands, this strategy might not be possible due to technical obstacles. For example, conflicting domain or PID settings may necessitate the reboot of mission critical hosts to accept unique core PID settings or domain numbers. Other technical issues that can arise involve compatibility of SAN device firmware and Fabric OS level support amongst multiple vendors. An effective architecture development process involves the identification of these technical obstacles. Please reference the SAN Migration Guide (53-0000360-xx) to better understand the SAN consolidation process. Under these circumstances, if the immediate need to share devices or enable FCIP or iSCSI extension exists, an architecture that incorporates the Multiprotocol Router at a tactical level in the near term with longer term plans to migrate to a consolidated SAN may be the most appropriate path to take. Using this approach, a phased migration of the SAN islands can occur with the flexibility to account for infrequent and differing maintenance windows for the multiple SAN islands. Finally, if no SAN exists, also known as a green field, merging SANs is a non-issue and the freedom exists to build large SANs or to interconnect SANs from the start with out dealing with existing technical obstacles.
2.3. How Big Is Big And How Big Is Too Big?

The size of a fabric is a relative term. Large for some SAN users is 2000+ ports, while other users consider anything greater than 100 ports to be too large. Brocade currently supports many customers with fabric sizes approaching 1000-ports and Brocade sees the demand to support fabric sizes in excess of 2000-ports. The reality is that the majority of fabric deployments are small fabrics. The size of a large fabric is not practically limited when using Brocade technology. Put another way, Brocade enables the freedom to deploy an architecture that meets the key requirements of an enterprise and should technical obstacles exist, such as zone conflicts or domain ID changes, there is the option of using the Multiprotocol Router to interconnect islands. As discussed, the SAN architecture adopted is likely going to be a function of the customer risk tolerance, SAN personnel capabilities, and organizational structure. In some cases, customers may have built SANs that are deemed too large because of risk or management complexity. Rather than attempting to figure how big they want to build their fabrics, these customers are trying to figure out how to downsize their fabrics. The same key factors come into play when attempting to downsize a fabric: risk, capabilities, and organization structure. The SAN Migration Guide (53-0000360-xx) excellent source for assisting with the development of a fabric downsizing plan. This document goes into great detail regarding the planning steps and how to evolve existing SAN environments in a non-disruptive fashion.
2.4. Interconnected Fabrics

A variety of reasons exist to perpetuate the deployment of smaller SAN islands, including the desire to maintain smaller fabrics for troubleshooting and the desire to maintain separate fabrics by application or department but still share resources. Connecting two or more geographically separate fabrics for disaster recovery (DR) purposes is a common requirement. Such configurations utilize FC (ex. DWDM/CWDM/LWL) or FCIP (ex. WAN technology). Since only a few devices need to be shared across the distance, it is logical to not merge the separate SANs -- especially for fault isolation and performance reasons. DR over distance makes a solid case for interconnecting islands, but keeping fabrics separate for well-known, very selective sharing of resources. As discussed, even if a fabric consolidation approach is desired, it may not be possible due to the need to resolve zoning, naming, PID, and other conflicts. Using Brocade technology, the SAN architect is armed with the flexibility to deploy large fabrics and to interconnect many fabrics of varying size. The technology is not the challenge to developing an effective SAN architecture.
2-5
2.5. A High Port Count Director Or A Router?

A related question to the Large SAN versus interconnect SAN island discussion is whether to deploy a router or a director. Many of the same discussions relating to comfort level, SAN personnel capabilities, and organizational structure apply to the Multiprotocol Router versus director question. If high port count (approximately 128 ports), any-to-any connectivity is needed and there exist a single team responsible for administrating the SAN, using a director is an effective choice for connecting multiple SANs. However, such a choice does require consideration, as it creates a larger fault domain. If there exist multiple teams managing the existing SAN islands or if the sharing bandwidth between SAN islands is low, a router is likely a better choice.
2.6. MetaSAN Architecture Selection FlowChart

The flowchart shown in Figure 2-3 summarizes at a high level the considerations and decision flow for determining an appropriate Multiprotocol Router based SAN architecture. Rarely is an architecture selection a discrete decision of solely taking one approach or another. Instead, most architectures will combine multiple SANs and routing to create an architecture customized for their requirements.
2-6
Is there one team responsible to administrate the SAN?
NO
Interconnect SAN Islands using the multiprotocol router
Yes
Are you comfortable in deploying fabrics greater than 200-ports?
NO
Yes
NO Does your SAN administration team have experience managing fabrics larger than 100-ports?
Yes
Build high port count fabrics
Figure 2-3
metaSAN Architecture Selection Flowchart
2-7
2-8
Chapter
The capabilities of the SilkWorm Multiprotocol Router introduce several new concepts and definitions. A top down approach is taken to defining the components of a metaSAN, with additional detail and new terms defined throughout this section as the subcomponents of the metaSAN are discussed. The Multiprotocol Router has the capability to connect dozens of fabrics and potentially tens of thousands of SAN devices using FC, FCIP, and iSCSI protocols. With the fault isolation enabled by the Multiprotocol Router and a sound SAN architecture, disruptions to large metaSANs can be minimized and even eliminated. The impact of other design concepts such as performance and scalability are magnified as the number of SANs are interconnected by the Multiprotocol Router. Familiar topics to SAN design, such as Resiliency, Redundancy, Locality, Scalability, and Oversubscription Ratios also apply to metaSANs. FC-to-FC routing, FCIP, and iSCSI each have specific considerations and behaviors the are important to understand when developing a metaSAN architecture. Read on as these topics are discussed in more detail, enabling you to develop an effective Router based SAN architecture.
3.1. Components of a MetaSAN

Several definitions follow that are fundamental to understanding a metaSAN in the context of this document. For additional detail regarding the Multiprotocol Router and metaSANs, please reference the white paper Introducing Brocade Multiprotocol Routing Services. Fabrics connected via Fibre Channel Routing Services are known as Edge Fabrics. Edge Fabrics are known by their Edge Fabric ID (FID). The fabric or switch that runs Fibre Channel Routing Services, and that connects Edge Fabrics, is known as the backbone. There can be more than one backbone in a routed fabric. Ports on FC Routers that connect to Edge Fabrics are known as EX_Ports. Fibre Channel links that connect Edge Fabrics to Backbone Fabrics are known as Inter Fabric Links (IFLs), and are analogous to ISLs using within an edge fabric. Backbone Fabric: A backbone fabric can consist of a single Multiprotocol Router or multiple Multiprotocol Routers that connect to each other directly or indirectly through other switches via E_Ports or VE_Ports to form a backbone fabric. (Figure 3-7). Edge Fabric: A Fibre Channel fabric connected to EX_Ports of the Multiprotocol Router. The edge fabric is where hosts and storage are normally attached in a metaSAN. E_Port: A standard Fibre Channel mechanism that enables switches to network with each other (Figure ). EX_Port: The type of port used to connect an FC router to an edge fabric. An EX_Port follows standard E_Port protocols, but does not allow fabric merging across EX_Ports and supports FC-NAT (Figure ). Exported Device: A device that has been mapped between fabrics. A host or storage port in one edge fabric can be exported to any other fabric by using LSAN zoning (Figure 3-5). Fabric Locality: The degree that I/O is confined to a particular fabric. If two devices that need to communicate with each other are located on the same fabric, then these two devices are said to have high locality. If these same devices are located on different fabrics and these two devices need to communicate with each other, then these devices are said to have low fabric locality (see section 3.3.1. Fabric Locality and IFL Oversubscription Ratios on page 3-13). The term locality also applies to switches. Fault Domain: An area that could be affected by a fault or error. A fabric is a fault domain as is a routed fabric. FC-NAT: Fibre Channel Network Address Translation allows devices in different fabrics to communicate when those fabrics have addressing conflicts. This is similar to the hide behind NAT used in firewalls.normally attached in a routed fabric.
3-1
Fibre Channel Fabric (fabric): One or more interconnected switches or domains. A Fibre Channel fabric may also refer to the interconnected switches or domains and attached SAN devices (Figure 3-1). Inter-Fabric Link (IFL): A connection between a router and an edge fabric (Figure 3-2). Logical Storage Area Network (LSAN): A logical network that spans multiple fabrics. The path between devices in an LSAN can be local to an edge fabric or cross one or more FC routers and up to one intermediate backbone fabric (Figure 3-6). MetaSAN: A the collection of SAN devices, switches, edge fabrics, LSANs (logical storage area network), and Multiprotocol Routers that make up a physically connected but logically partitioned storage area network (Figure 3-2). Routed Fabric: Two or more edge fabrics interconnected by one or more backbone fabrics (Figure 3-2). SAN device: A device that is connected to one or more fabrics -- such as a host or storage device (Figure 3-1). Storage Area Network (SAN): One or more related fabrics and the connected SAN devices (Figure 3-1). VE_Port: A term for a port that connects to another switch port as an E_Port via FCIP (Figure ). In Figure 3-1, a storage area network is shown, with fabrics and SAN devices identified. In this case, the SAN is termed a dual fabric SAN. The association of the SAN devices and fabrics is what makes a SAN a SAN.
Fabric
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW24000
SW24000
SW24000
SW24000
SAN
Host Storage
SAN Devices
Tape
Figure 3-1
A Storage Area Network
3-2
A metaSAN is shown in Figure 3-2. Note that the metaSAN consists of two routed fabrics - Routed Fabric A and Routed Fabric B. The metaSAN also consists of two SANs, of which one SAN is circled in blue. Note the association of SAN devices and fabrics that make up each SAN -- visually each SAN is laid out horizontally in Figure 3-2.
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW24000
SW24000
SW24000
SW24000
Interfabric Link (IFL)

Host Storage
Tape
Multiprotocol Router SAN
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW24000
SW24000
SW24000
SW24000
Host Storage
Tape
Routed Fabric A
Routed Fabric B
Figure 3-2
A MetaSAN
3-3
The variety of configurations enabled by the Multiprotocol Router is virtually limitless and bounded by the configurations supported by your SAN support provider. Figure 3-3 provides some context relating to how FC-to-FC routing, FCIP, and iSCSI work together the relationship between E_Ports, EX_Ports, and VE_Ports. A large scale metaSAN is show in Figure 3-4 to establish the scope of connectivity enabled by the Multiprotocol Router and provide context for several of the preceding definitions.
EX-port
FC
Fibre Channel SAN

FC L IF
Fibre Channel SAN

E-port G-port (GigE)
EX-port FC IFL
IF L
Ethernet
VE-port
WA
in k NL
MP Router FC Routing Services

F-port
EX-port
MP Router iSCSI Gateway Services
iSCSI Intitiator
MP Router FCIPTunneling Services
FC
Figure 3-3
How FC-to-FC routing, FCIP, and iSCSI relate in a MetaSAN
3-4
Figure 3-4
A large scale MetaSAN
3-5
3
An example of an exported device is shown in Figure 3-5, where a host from Fabric 2 is exported to Fabric 1 and a storage device from Fabric 1 is exported to Fabric 2. If it is desired that these devices communicate with each other, it is necessary to set up the appropriate LSAN zones. A representation of an LSAN is shown in Figure 3-6.
Figure 3-5
MetaSAN Device Export/Import
3-6
Figure 3-6
MetaSAN LSAN Zone
3-7
3
The concepts of a backbone fabric and use of multiple backbone fabrics to connect edge fabrics are depicted in Figure 3-7. Note that MetaSAN 2 is comprised of two backbone fabrics because there exist no E_Port or VE_Port connection between the two routers. In this case, the routers communicate with each other via the edge fabrics.
Single BackBone Fabric
Dual BackBone Fabric
M e taSAN 1
MetaSAN 2
M e taS A N 3
M e taSAN 4
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW24000
SW24000
SW24000
SW24000
SW24000
SW24000
SW24000
SW24000
Multiprotocol Router Multiprotocol Router Multiprotocol Router M u ltiprotocol Router Multiprotocol Router
SW3900 Multiprotocol Router Multiprotocol Router Multiprotocol Router
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW3900
SW24000
SW24000
SW24000
SW24000
SW24000
SW24000
SW24000
SW24000
Figure 3-7
Backbone Fabric Examples
3-8
3.2. Availability
Critical among the metaSAN architecture design points is availability. A primary use of the Multiprotocol Router is to interconnect multiple fabrics. By virtue of this use, several single points of failure (SPOF) can arise relating to the implementation of a metaSAN. Simply stated, computer systems are dependent upon the availability of data and if data is not available, computing (i.e. banking transactions, manufacturing operations, etc.) doesnt happen. For many enterprises, depending upon a single infrastructure element is unacceptable, so redundancy in the form of redundant routed fabrics is a requirement. However, there are several valid circumstances and risk/benefit scenarios that justify a single routed fabric, such as a a tape consolidation. The document Brocade Multiprotocol Routing Services Solution Guide: Meta SAN Tape Consolidation provides a detailed example of a tape consolidation. Redundant dual fabric SANs have been designed by SAN architects and deployed in enterprises since SANs were introduced. Furthermore, the fabrics utilized in these redundant architectures are normally designed to be resilient and remain operational even when a single failure occurs within one of the fabrics that makes up a redundant dual fabric SAN. For a detailed discussion on how availability relates to SAN design and architecture, please refer to the Brocade SilkWorm Design, Deployment, and Management Guide (53-0000366-xx).
3.2.1. Resiliency & Redundancy

The SAN shown in Figure 3-8 is a dual fabric, redundant SAN. Both Fabric A and Fabric B, which make up the SAN are resilient. For example, if an ISL or switch fails, the fabric is functional and it is not necessary to use a redundant path.
Fa br ic St ill Op er at io na l If
Fabric A
at ru nk or IS Ls fa il
Fabric B
x
Fabric Still Operational If Core Switch #1 fails
Figure 3-8
Fabric Resiliency
3-9
3
However, if an entire fabric fails or part of the path fails, a redundant element needs to be utilized for the SAN to remain operational. For example in Figure 3-9, the SAN remains operational in the event of a failure of the host path, the fabric, or storage path. Many times a path or fabric failure is not a hardware failure but instead a software malfunction or operator error.
A redundant fabric SAN remains operational in the event of a host path failure, fabric failure, or storage path failure.
Host
Host
x
Fabric A
SAN 1
x
Storage
Fabric B
Figure 3-9
SAN Redundancy
3.2.2. MetaSAN Resiliency & Redundancy

The metaSAN shown in Figure 3-10 is a low cost and simple way of interconnecting SANs to share devices. This design may be viable under some circumstances, such as sharing a tape library. However, there exist many drawbacks to this design. First, only a single interfabric link (IFL) connects the SANs to the Multiprotocol Router. Aside from the bandwidth issues that might arise from a single IFL connection, the single IFL is a single point of failure. Adding IFLs between both Fabrics and the Multiprotocol Router increases resiliency and performance. Adding a second Router to the configuration eliminates the first Multiprotocol Router as a single point of failure. The drawback to adding ISLs and a Router is cost. If the primary purpose of the metaSAN is casual connectivity and sharing of devices or a non-mission critical application, a single Multiprotocol Router could be an acceptable design choice. Finally, be wary of connecting the Multiprotocol Router to both fabrics that make up a redundant fabric SAN (i.e connecting both Fabric A and Fabric B to the same router), as this creates a single fault domain. If both fabrics of a redundant fabric SAN are connected to the same router, a worst case fault condition could affect all fabrics in the metaSAN. Warning: Avoid connecting the Multiprotocol Router to both fabrics of a redundant fabric SAN (i.e connecting the same router to both Fabric A and Fabric B), as this creates a single fault domain.
3-10
3
.
SAN 1
Fabric A Fabric B Fabric A
SAN 2
Fabric B
IF L
Single IFL
Si ng le
SAN 3
le IF L
Sing
Fabric A
X
Figure 3-10 Low cost, simple connection scheme for the Multiprotocol Router
3-11
3
The architecture presented in Figure 3-11 is a redundant metaSAN, consisting of two routed fabrics. A catastrophic failure in one routed fabric can occur and the remaining routed fabric can continue to function. Multiple ISLs connect each fabric to the Multiprotocol Router, resulting in some resiliency for the routed fabrics.
Host Storage
Tape
SAN-1 Fabric A
SAN-1 Fabric B
SAN-2 Fabric A
SAN-2 Fabric B
SAN-3 Fabric A
routed fabric A
4 IFLs 2 IFLs
routed fabric B
Figure 3-11 Redundant MetaSAN The SPOF issue of a single Multiprotocol Router per routed fabric can be mitigated and the routed fabrics can become fully resilient, as shown in Figure 3-12 by adding a second Router. Note that each routed fabric in Figure 3-12 is built with two separate backbone fabrics, with each backbone fabric consisting of a single Multiprotocol Router. The issue of a single Multiprotocol Router per routed fabric is not as critical as may be initially thought since local SAN operations can continue in the event of a metaSAN failure. One drawback to the two-routed fabric design is that it is not possible to share devices between the A and B fabrics of each SAN, as these fabrics are purposefully isolated from each other for availability reasons. Note: A redundant metaSAN design, as shown in Figure 3-11 and Figure 3-12 does not permit the sharing of devices between routed fabrics.
Guideline: Use a redundant routed fabric model to create two separate fault domains and a very highly available metaSAN architecture (see Figure 3-12).
Guideline: For the highest availability, utilize at least two backbone fabrics per routed fabric and connect each edge edge fabric to each backbone fabric by at least one IFL, as shown in Figure 3-12.
3-12
Host Storage
Tape
SAN-1 Fabric A
SAN-1 Fabric B
SAN-2 Fabric A
Multiprotocol Router Multiprotocol Router
SAN-2 Fabric B
SAN-3 Fabric A
routed fabric A
4 IFLs 2 IFLs
routed fabric B
Figure 3-12 A highly available redundant and resilient metaSAN architecture
3.3. Multiprotocol Router Performance

The same performance design points, such as ISL oversubscription ratios and locality, that apply to SANs also apply to metaSANs. For a detailed discussion regarding ISL oversubscription ratios and locality, please refer to the Brocade SilkWorm Design, Deployment, and Management Guide (53-0000366-xx). With the Multiprotocol Router, there are new concepts such as IFL oversubscription ratios, iSCSI host fan-in ratios, and fabric locality. FCIP utilizes IP networks. Unlike Fibre Channel, IP networks are subject to higher levels of frame (packet) loss and the latency through multiple hops may also be higher than compared to long distance Fibre Channel links.
3.3.1. Fabric Locality and IFL Oversubscription Ratios

The concept of Fabric locality is shown in Figure 3-13. When host A communicates with Storage B, the I/O is local to SAN-1, Fabric A. This type of traffic pattern is termed a high fabric locality traffic pattern. However, when Host X communicates with Tape Y, the I/O traverses the IFLs that connect the Fabrics to the Multiprotocol Router. This traffic pattern is not local to SAN-1, Fabric A and this is termed a no fabric locality traffic pattern.
3-13
3
.
When Host A communicates with storage B, these devices are said to have high fabric locality Host A Storage B
SAN-1 Fabric A
Host X
When Host X communicates with Tape Y, these devices are said to have no fabric locality
SAN-2 Fabric A
SAN-3 Fabric A
routed fabric A
Figure 3-13 The Concept of Fabric Locality
Tape Y
To calculate the number of IFLs that connect a fabric to a Multiprotocol Router, determine the amount of I/O bandwidth consumed by the devices local to the fabric that communicate with devices remote to that fabric. For example, in Figure 3-13, if multiple hosts perform backup and restore operations to Tape Y and Tape Y is only capable of sustaining 150 MB/sec in bandwidth, it is only necessary to connect SAN-1 Fabric A to a Multiprotocol Router with one IFL, as a single IFL can sustain 200 MB/sec. For resiliency purposes, however, it is recommended to have SAN-1 Fabric A connected to the backbone fabrics by at least two IFLs. Note: An IFL oversubscription ratio is defined as the number of devices local to a fabric that communicate with devices remote to that fabric divided by the number of IFLs that connect that fabric to the Multiprotocol Router.
For example, in Figure 3-14, the hosts in SAN-1 Fabric A perform I/O with the tape connected to SAN-2 Fabric A and the storage connected to SAN-3 Fabric A. The total remote bandwidth for SAN-1 Fabric A is 320 MB/sec and therefore at least two 2 Gbit/sec IFL connections are needed to connect this fabric to the Multiprotocol Router. Another aspect to consider is that the end-to-end bandwidth is sufficient and not just the bandwidth of the fabric connecting to the backbone fabric. Using similar calculations, SAN-3 Fabric A requires two 2 Gbit/sec IFLs and SAN-2 Fabric A requires one 2 Gbit/sec IFL to connect to the Multiprotocol Router. It is recommended that under these circumstances, two IFLs be utilized to connect SAN-2 Fabric A to the Multiprotocol Router for resiliency purposes -- even though the bandwidth requirement is only 100 MB/sec.
3-14
Host Disk I/O = 30 MB/sec Tape I/O = 30 MB/sec
SAN-1 Fabric A
5
Hosts connected to SAN-1 utilize disk and tape connected to SAN-2 and SAN-3. SAN-1 Fabric A total disk I/O = 210 MB/sec SAN-1 Fabric A total tape I/O = 120 MB/sec SAN-1 Fabric A Total Remote Bandwidth = 330 MB/sec SAN-2 Fabric A total disk I/O = 210 MB/sec SAN-2 Fabric A Total Remote Bandwidth = 210 MB/sec SAN-3 Fabric A total disk I/O = 110 MB/sec SAN-3 Fabric A Total Remote Bandwidth = 110 MB/sec
MultiprotocolRouter
Tape SAN-2 Fabric A

MultiprotocolRouter
Storage
SAN-3 Fabric A
routed fabric A
Figure 3-14 Calculating remote bandwidth to determine the number of IFLs to connect the Multiprotocol Router to the an edge fabric A fifteen SAN Device-to-one IFL ratio (15:1) equates to slightly less than 14 MB/sec per SAN device - if all SAN devices are communicating simultaneously. A three SAN Device to one IFL ratio (3:1) equates to slightly more than 66 MB/sec per SAN device - if all SAN devices are communicating simultaneously. If all devices are not communicating simultaneously, higher per device peak bandwidth is possible. Based on current SAN performance data gathered from Brocade customers, 15 MB/sec is an above average bandwidth requirement per device. On average, hosts typically consume less than 15 MB/sec. Many tape and storage devices can easily exceed 15 MB/sec per port. If connecting existing SANs it is possible to utilize tools such as Fabric Manager, Fabric Watch, or the Command Line Interface (CLI) command portperfshow to determine the peak and average bandwidth utilization for devices that will communicate with remote devices. The Brocade SilkWorm Design, Deployment, and Management Guide details the steps necessary to utilize Brocade Fabric Watch to monitor IFL performance. Note: Note that Fabric Watch currently is not implemented on the Router and therefore, it is necessary to monitor the IFL performance from a SilkWorm switch that supports Fabric Watch.
If data is not available to calculate the remote bandwidth requirements, it is recommended to provision enough ports on the remote fabric and the backbone fabric to achieve a 7:1 IFL oversubscription ratio, but connect at a 15:1 IFL oversubscription ratio. Doing so enables the ability to expand bandwidth between an edge fabric and the backbone fabric and if the extra ports are not needed, they can always be utilized to connect other switches or devices. Regardless of IFL oversubscription ratio utilized, it is recommended to monitor the IFL links using Fabric Manager or Fabric Watch to confirm that the fabrics are connected with the appropriate number of IFLs and to identify congestion situations.
3-15
3
Guideline: Utilize at least two 2 Gbit/sec IFLs to connect an edge fabric to the backbone fabric(s).
Guideline: For accurate IFL oversubscription ratios, it is advised to obtain actual performance data. If performance data is unknown or not available, it is recommended to provision enough ports on the remote fabric and the backbone fabric to achieve a 7:1 IFL oversubscription ratio, but connect at a 15:1 IFL oversubscription ratio to attach fabrics to the backbone fabric(s). Utilize Fabric Watch or Fabric Manager to monitor IFL performance. In addition to selective sharing of devices, two primary types of SAN performance profiles are expected to be encountered in the metaSAN environment: fan-in and fan-out performance profiles. In the fan-in profile, many lower performing devices (such as hosts) fan-in to a few SANs that contain a few high performing devices such as storage or tape. The other type of SAN performance profile is termed a fan-out, where a few SANs containing a few high performing devices (such as storage or tape) fan-out to many SANs containing many lower performance devices (such as hosts). These concepts are illustrated in Figure 3-15. Identifying these SAN performance profiles is key to establishing appropriate IFL oversubscription ratios.
Fan-in SAN performance profile
Many Low, to Few High
Fan-out SAN performance profile
Fan-in
Meta SAN Fan-out
Many High, to Few Low
Many High, to Few Low
Figure 3-15 Fan-in and Fan-out SAN Performance profiles relative to a metaSAN
3-16
3.3.2. iSCSI and FCIP Performance

The key variables to obtain when preparing a SAN design that utilizes WAN links are the link latency and packet loss percentage. It is also important to gather performance requirements for devices that utilize iSCSI services or SAN devices that traverse FCIP links to accurately calculate the required bandwidth. FCIP ISL reach is discussed in terms of round-trip-time (RTT) latency, and not distance, due to store-and-forward devices (routers) in the IP path that add delay. Native FC ISLs reach is measured using distance, typically computed using speed-of-light delays (speed of light ~100 us per 10 KM round trip). Note that link congestion may also affect latency due to queuing delays in intermediate routers and switches. Unlike a local FC connection, wide area telecommunications configurations can sustain variable degrees of packet loss. This situation requires the application to retransmit the lost frames. This can cause significant performance impact if the TCP extension equipment isn't designed to handle these retransmits efficiently. WAN bandwidth is frequently much lower than the 2 Gbit/sec bandwidth capabilities of Fibre Channel, for example:
T-1: 1.544 Mbps E-1: 2048 Mbps T-3: 44.736 Mbps OC-3: 155.52 Mbps OC-12: 622.08 Mbps
Please reference your support provider support matrix to determine the current performance capabilities of the Multiprotocol Router.
3.3.3. FCIP Link Utilization Calculations

The cost of providing the FCIP links between Multiprotocol Routers is much higher than just connecting an ISL. In addition to the cost of the port, the cost of the WAN link lease needs to be considered. WAN links costs are a recurring and can be quite substantial. Since the cost of FC ports is much lower, SAN administrators will take a conservative approach and connect more ISLs than might actually be necessary. The logic behind this approach is that the extra ISL port costs are quite small compared to the cost of managing the performance. To calculate an FCIP link requirements, it is necessary to identify the bandwidth requirements of the devices that traverse the FCIP link. Given the high costs of WAN links and the mismatch in performance between Fibre Channel and WAN link speeds, it is not practical to utilize FCIP oversubscription ratios. Note: It is not practical to utilize FCIP oversubscription ratios in the same way IFL or ISL oversubscription ratios are utilized given the high costs of WAN links and the mismatch in performance between Fibre Channel and WAN link speeds.
3-17
3 3.3.4. iSCSI Fan-in

There are two parts to fan-in: iSCSI Host fan-in and iSCSI Port fan-in. iSCSI Host fan-in covers the number of hosts that can access the Fibre Channel fabric using the same iSCSI port on the Multiprotocol Router. iSCSI Port fan-in covers the ratio of Multiprotocol Router iSCSI ports to Fibre Channel interswitch links (ISL). Note: SAN Devices directly connected to the backbone being used for FC-to-FC routing cannot be shared and thus have no visibility to other fabrics. Therefore it is not recommended to connect SAN devices directly to backbone fabrics if these devices are intended to be shared or these devices need access to devices located in other fabrics. This behavior may change in future releases of XPath OS.
To correctly calculate the number of iSCSI hosts to be connected to each Router iSCSI port (also referred to as an iSCSI portal), consider the following:
Assume each iSCSI switch port (or portal) currently supports up to eight iSCSI sessions. (Please reference your support provider support matrix to identify the most current iSCSI Host fan-in ratios and Multiprotocol Router performance capabilities.) Each host can have one or more sessions, depending on how the iSCSI host is set up. A session is created when the iSCSI host connects to available storage. In other words, each individual drive or LUN the iSCSI host connects to is a session. It is recommended, to limit the number of storage devices connected to each host (see Table 3-1). Table 3-1 Sessions Per Host 1 2 4 8 1 2 4 8 1 2 4 8 iSCSI Sessions-to-host # Of Hosts Per Port 8 4 2 1 8 4 2 1 8 4 2 1 # Of Ports 14 14 14 14 13 13 13 13 12 12 12 12 Total Hosts Per Switch 112 56 28 14 104 52 26 13 96 48 24 12
3-18
3
The number of iSCSI hosts per port is dependent on three things:
The number of sessions each host has The total bandwidth of all hosts fanned into the port How much over-subscription is acceptable
Figure 3-16 shows an example of iSCSI Host fan-in.
Host A 150Mbps Host B 100Mbps Host C 50Mbps Host D 25Mbps Host E 25Mbps Host F 25Mbps Host G 10Mbps Host H 5Mbps
J4908A 22 port Gig-T/GBIC
21 6 4 13 5 11 6 2 8 7 13 17 14 18 3 20 12 17 9 22 21 16 15 19 16 12 22 12
hp procurve 1 0 Gig-T/GBIC gl Module J4908A
1 1 7
gl
10/100/1000 Ethernet Switch
Total Bandwidth 390Mbps
Figure 3-16 iSCSI Host Fan-in Figure 3-16 shows eight iSCSI hosts fanned into the switch port using an Ethernet switch. Each iSCSI host has one active session. Next to each iSCSI host is the average bandwidth use for that iSCSI host. The total average bandwidth used by all eight hosts is 390 Mbit/sec. This of course is the average bandwidth. If all of the hosts were to spike simultaneously, a congestion problem would occur. In reality, it is not likely that all eight iSCSI hosts will need to transmit a large amount of data at the same time, so this type of fan-in is more than acceptable. If, on the other hand, a host with a high bandwidth average exists (for example 350 Mbit/sec), most likely congestion with other iSCSI hosts using the same iSCSI portal will be encountered. This may or may not be a problem, depending on what the iSCSI hosts are being used for. For example, if the iSCSI hosts are email servers, the added delay may not be a problem.
3-19
3
In short, over-subscription and congestion will vary for each installation, and must be determined by the designer on a case-by-case basis. Now that it is decided how many hosts will be fanned into each port, it is necessary to understand how many Fibre Channel ISLs are needed between the Multiprotocol Router and the Fibre Channel fabric. To correctly calculate the correct iSCSI ports to Fibre Channel ISL ratio, consider the following:
Assume each iSCSI switch port supports approximately 400 Mbit/sec Each Fibre Channel port is either 1000 Mbit/sec or 2000 Mbit/sec
There are two ways to calculate the iSCSI ports to Fibre Channel ISL ratio. Assume each iSCSI port will use the full 400 Mbit/sec bandwidth and base the ratio on that assumption (see Table 3-2). Table 3-2 shows both oversubscription, meaning congestion could be encounter, and under-subscription, meaning there is bandwidth provisioned that will not be utilized. Another alternative is to use the total average bandwidth calculated during the iSCSI host fan-in design portion. Both will work, and depending on the needs one approach may be preferred over the other. For example, if the design calls for little or no oversubscription on the host fan-in, then use of the full bandwidth option is prudent. On the other hand, if the average iSCSI host fan-in bandwidth per port is low (say less then 100 Mbit/sec), use the estimated values used during the iSCSI host fan-in design to increase the number of ports that can be used as iSCSI ports (see Figure 3-17). Table 3-2 # iSCSI Ports 14 13 12
iSCSI Ports to FC ISL Ratio Times 400 Mbit/sec 5600 Mbit/sec 5200 Mbit/sec 4800 Mbit/sec # 2Gbit/sec FC ISLs 2 3 4 Total ISL Throughput Subscription
4000 Mbit/sec Over 1600 Mbit/sec 6000 Mbit/sec Under 800 Mbit/sec 8000 Mbit/sec Under 3200 Mbit/sec
Figure 3-17 shows fourteen iSCSI ports coming into the Multiprotocol Router and two 2 Gb/sec ISLs going to a SilkWorm 3800 Fibre Channel switch. The hosts at the bottom of the diagram represent a group of iSCSI hosts fanned into one or more iSCSI ports. By using the calculated iSCSI host fan-in the total host bandwidth is 3975, which is below the 4000 Mbit/sec Fibre Channel ISL bandwidth. On the other hand, if the total port bandwidth had been used rather than the calculated host fan-in, we would have appeared to be over-subscribed by 1600 Mbit/sec and would have needed to use one of the iSCSI ports as a third Fibre Channel ISL.
3-20
3
Fibre Channel SAN Storage Fibre Channel SAN Hosts
4 6 10 12
5 11 6
1 3 13 1 7 16 21
21
4 10 6
5 11 6 2 8 12 17 13 1314 1718 3 9 12 15 21
hp procurve Gig-T/GBIC gl Module J4908A

7
2 8
20 22 15 14 12 17 18 3 19
9
16
22 12

7
12
19 2 16 2 1 2 20 22 16
1 1 7 1
1 7
SilkWorm 3800 Dual Fabric
ISL Bandwidth 2000Mbps X 2 Total 4000

10/100/1000 Ethernet Switch
gl
gl
Total Host Bandwidth 3975Mbps
iSCSI iSCSI iSCSI iSCSI iSCSI iSCSI iSCSI iSCSI Group A Group B Group C Group D Group E Group F Group G Group H 775Mbps 475Mbps 500Mbps 275Mbps 625Mbps 425Mbps 575Mbps 325Mbps
Figure 3-17 iSCSI Port Fan-in
Note:
Each iSCSI Group in Figure 3-17 represents eight or more iSCSI hosts accessing Fibre Channel storage through one or more iSCSI ports on the Multiprotocol Router.
3-21
3.4. Scalability
Performance requirements drive the appropriate IFL oversubscription ratios and in turn the number of IFLs that connect a fabric to a routed fabric. The number of IFLs and the number of SANs that require interconnection drive the number of ports required for routing capabilities. In addition, some ports on the Multiprotocol Router may be used for iSCSI or FCIP connections. Note: SAN Devices directly connected to the backbone being used for FC-to-FC routing cannot be shared and thus have no visibility to other fabrics. Therefore it is not recommended to connect SAN devices directly to backbone fabrics if these devices are intended to be shared or these devices need access to devices located in other fabrics. This behavior may change in future releases of XPath OS.
In addition to the backbone fabric port counts, there exist other variables that are part of the scalability equation, including:
Maximum number of domains per edge fabric Maximum number of imported devices per fabric Number of LSAN zones/metaSAN Number of edge fabrics per Routed Fabric Maximum number of hops between edge switches Maximum entries per LSAN zone Maximum local switches per edge fabric Maximum number of local and remote WWNs per edge fabric Number of Fibre Channel routers per metaSAN
Please reference your support provider support matrix to determine if your configuration is supported. The iSCSI functionality of the Multiprotocol router is also subject to certain scalability variables. The key scalability variable for the iSCSI functionality is the number of initiators per port, which is known as the fan-in ratio. In order to achieve a fan-in ratio, the servers (initiators) may be aggregated through an Ethernet switch connected to the Multiprotocol Router on one of the Gigabit Ethernet iSCSI configured ports. It is also possible to connect a server directly to an iSCSI configured port on the Multiprotocol Router, if a fan-in ratio of greater than one is not necessary.
3-22
3 3.4.1. Scalability of the Backbone Fabric

When interconnecting fabrics for FC routing functionality there are primarily two approaches for creating an interconnection. One approach involves parallel connections of Multiprotocol Routers to edge fabrics, as shown in Figure 3-18. The other approach is to interconnect Multiprotocol Routers to form a multi-switch backbone fabric or to interconnect multiple Multiprotocol Routers to Fibre Channel switches to form a multi-switch backbone fabric as shown in Figure 3-19.
Figure 3-18 Simple routed fabric using multiple single switch backbone fabrics
3-23
OR
Figure 3-19 Routed fabric using a single multi-switch backbone fabric To determine the number of ports necessary to create a routed fabric, start by adding up the required IFL ports. Ensure that enough ports are provisioned to enable the addition of IFL ports to provide for increases in performance between an Edge Fabric and the routed fabric. Also account for additional Fabrics that may require interconnection to the routed fabric.
3-24
3
Utilizing two Multiprotocol Routers that are configured as single switch backbone fabrics to connect edge fabrics and connecting each edge fabric to each router by a single IFL, it is possible to connect up to sixteen edge fabrics. Performance requirements may dictate connecting the edge fabrics to the backbone fabric by more than one IFL. Increasing the number of IFLs that connect an edge fabric to the backbone fabric decrease number of available ports in the backbone fabric and therefore reduces the number of edge fabrics that can be attached to the backbone fabric. If the competing needs of performance and connecting edge fabrics drive a requirement for more than 32-ports in the backbone fabric, consider using up to four Multiprotocol Routers configured in as single backbone fabrics. Such a configuration can support up to 16 edge fabric connections, with each edge fabric connected to the backbone fabric with 800 MB/sec of bandwidth (see Figure 3-20). It is important that all edge fabrics connect to each of the Multiprotocol Routers when using a single switch backbone fabric. Doing so enables any to any connectivity between edge fabrics. If these guidelines can not be followed, it is recommended to utilize a multi-switch backbone fabric. Since each router configured in a single switch backbone fabric is a separate backbone, there is the added benefit of redundancy for this approach when compared to a multi-switch backbone fabric, which is considered a single backbone. A recommend limit for a single switch backbone fabric backbone fabric is four Multiprotocol Routers.
Connect up to 16 edge fabrics Each edge fabric connects to the router complex at 800 MB/sec Four backbone fabrics for redundancy
Figure 3-20 Single switch backbone fabric backbone fabric using four Multiprotocol Routers
3-25
Guideline: When calculating the number of ports used for interconnecting Fabrics, in addition to IFL ports, factor in growth in performances and attaching additional Edge Fabrics.
Guideline: When using a single switch backbone fabric architecture, connect all edge fabrics to each router to ensure any to any connectivity between edge fabrics is possible. If it is not possible or desired to follow these guidelines, it is recommended to utilize a multi-switch backbone fabric architecture.
Guideline: If the routed fabric IFL port requirement is greater than 64-ports, consider using a multi-switch backbone fabric architecture (see Figure 3-19), otherwise use a single switch backbone fabric architecture (see Figure 3-18). Under certain circumstances, multi-switch backbone fabrics are necessary due to router port count requirements exceeding 16-ports or when interconnecting geographically separate edge fabrics via Fibre Channel (i.e. DWDM/CWDM/ LWL) or FCIP. In Figure 3-21, there exist two routed fabrics. The edge fabrics are interconnected via FCIP using two Multiprotocol routers configured as multi-switch backbone fabric. Note that while the Routers all interconnect together via the IP WAN cloud, this does not mean that there exits any-to-any connectivity between each router. For two routers to communicate via FCIP, it is necessary to have a tunnel (i.e. a VE-port on each router).
Figure 3-21 FCIP using two multi-switch backbone fabrics
3-26
3.5. Trunking
Exchange-based trunking is an optionally licensed product available on the Multiprotocol Router and is licensed on a per-switch basis. The SilkWorm Multiprotocol Router exchange-based Trunking feature load-balances network traffic across interswitch links (ISLs) to optimally utilize available bandwidth. Specifically, exchange-based trunking allows to effectively load-share traffic across all ISLs. (To be precise, load balancing occurs across all equal-cost paths to a destination domain.) This functionality works in two situations:
Between two Multiprotocol Routers From a Multiprotocol Router and to any other Brocade Fibre Channel switch
Exchange-based trunking is supported at both 1 Gbit/sec and 2 Gbit/sec link speeds. There is no restriction on the port location/numbers that can be part of the trunk. When exchange-based trunking is enabled, all equal-cost paths are used for sending a traffic to the given destination domain. Each ingress port contains a list of egress ports that can be used to reach a destination domain. On receiving a frame at the ingress port, based on a hash algorithm, the following SCSI fields in the FC frame are used to determine which egress port to use (for load balancing purposes) to send the outgoing frame of data:
Source ID (S_ID) Destination ID (D_ID) Exchange ID (OX_ID)
Within any given SCSI exchange, the same path will be used, and there is no change for out-of-order delivery. Exchange-based trunking does not depend on having a compatible switch at the destination. Note that the routing decision for any given frame of data is made at the origin switch. Thus, the choice of which ISL to take always occurs when an Multiprotocol Router or any XPath enabled switch is the source switch and any other switch is the destination switch. If the other switch is not an Multiprotocol Router or XPath enabled switch, the return traffic will not enjoy the same load-balancing behavior. The Multiprotocol Router is compatible with other Brocade SilkWorm switches. However, the Multiprotocol Router is not compatible with the Trunk Groups (groups of four ports) feature of ISL Trunking in Fabric OS 4.x and 3.x SilkWorm switches. When an Multiprotocol Router has multiple ISLs to a non-Multiprotocol Router SilkWorm switch, traffic from the Multiprotocol Router to the non-Multiprotocol Router will be load-balanced using exchange-based trunking. Traffic from the non-Multiprotocol Router to the Multiprotocol Router will be load-balanced at a granularity of initiator device, target device pair (FSPF load balancing); that is, rather than at a data frame by data frame level, load will instead be rebalanced each time an initiator or target device's connection or disconnection impacts the routing behavior of a given ISL path. An example of how exchange based trunking and FSPF load balancing work together is shown in Figure 3-22. As host one communicates to the storage device, all I/O follows the same path. In this case exchanges A (H1A1, H1A2), B (H1B1, H1B2), and C (H1C1, H1C2) all take the same path. This is how FSPF load balancing behaves. The exchanges take different paths once they leave the Multiprotocol Router. Exchanges A (H1A1, H1A2) and C (H1C1, H1C2) take one path, while exchanges B (H1B1, H1B2) takes a different path. In all cases, the exchanges are delivered in order.
3-27
Exchange Based T runking F S P F L o a d B a lancing

S torage
H1A1, H1A2 / H1C1, H1C2
SW24000
H1A1, H1A2 / H1B1, H1B2 / H1C1, H1C2
H1B1, H1B2
SW24000
Host1
D a ta Flow
Figure 3-22 FSPF load balancing and exchange based trunking
3.6. Multiprotocol Router Interconnect Strategies

When connecting an edge fabric to one or more Multiprotocol Routers, there are several connection possibilities. Currently, the predominantly deployed fabric topology in enterprise environments is a Core/Edge fabric topology (see Figure 3-23). Please refer to the Brocade SilkWorm Design, Deployment, and, Management Guide (53-0000366-xx) for details regarding the characteristics of this widely deployed topology and why this topology is effective for scalable, available, and high performance SANs. The interconnect strategies discussed are presented primarily in the context of the Core/Edge topology with an alternative interconnect strategy for no Core/Edge topologies also presented. Connecting iSCSI, FCIP, and FC-to-FC connections are addressed in this section. Note: SAN Devices directly connected to the backbone being used for FC-to-FC routing cannot be shared and thus have no visibility to other fabrics. Therefore it is not recommended to connect SAN devices directly to backbone fabrics if these devices are intended to be shared or these devices need access to devices located in other fabrics. This behavior may change in future releases of XPath OS.
Edge Core
Figure 3-23 Core/Edge Topology
3-28
3 3.6.1. Edge Fabric to Backbone Fabric Connections

When connecting into a Core/Edge-based SAN, options exist to connect to the cores, the edges or both the cores and the edges. Normally there are two or more cores in a core /edge fabric. For simplicity, availability, and uniform performance reasons, it is recommended to connect each of the cores to the Multiprotocol Router, as shown in Figure 3-24. Doing so enables resiliency and enables simple performance tuning and monitoring since all routes between the Fabric and the Multiprotocol Router traverse the cores. Guideline: Connect the backbone fabric to the cores of a Core/Edge fabric.
SW3900
SW3900
SW3900
SW3900
Connect the Router and core switches

SW3900
Figure 3-24 Edge fabric to backbone fabric interconnect for Core/Edge topologies If free ports are unavailable in the core to interconnect the backbone fabric or if the edge fabric uses a topology other than the Core/Edge topology, connect the backbone fabric to switches where devices are exported. This strategy can also be effective for high performance devices that participate in LSANs or when a small number of devices that participate in LSANs are localized to one or two edge fabric switches. Guideline: Connect the backbone fabric to switches where devices are exported if free ports are unavailable in the core or if the edge fabric uses a topology other than the Core/Edge topology.
3-29
3
Exported Device Exported Device
Exported Device
Exported Device
Figure 3-25 Alternative edge fabric to backbone fabric interconnect
3-30
3 3.6.2. Interfacing the Multiprotocol Router iSCSI with Gigabit LANs

The design illustrated in Figure 3-26 separates the iSCSI SAN traffic from the production LAN traffic and equipment and is not affected by production LAN problems. This design will not induce production LAN problems and yields improved fault isolation while possibly eliminating equipment accountability issues.
Note: In this example a dedicated Ethernet LAN is used for iSCSI traffic.
4 106 5 11 14 13 21 15 22 19 16 3 12 9 7 1 20 21 16 12 22 4 10 6 12 5 11 13 14 18 6 13 17 2 128 1 7 7 15 19 16 3 9 20 21 2122 16 12 22
Network Servers
hpprocurve Gig-T/GBIC gl Module J4908A

7
12 2 8
18 6 1 3 17
1 1 7 1 1 7
gl
gl
Production LAN Link iSCSI LAN Link
Dedicated iSCSI LAN
Production LAN
Figure 3-26 Dedicated iSCSI LAN It is recommended to separate iSCSI storage traffic onto its own Ethernet LAN or VLAN so that the amount of traffic on the production LAN is reduced. This also prevents hardware and software problems (such as misbehaving applications or faulty hardware) on either iSCSI or production LAN equipment from interfering with each other, reducing fault isolation time. For example, if a rogue application causes a denial of service problem on the iSCSI LAN, equipment attached only to the production LAN can be ruled out as the cause, and attention can be focused on the devices connected to the iSCSI LAN. An additional area this option can help address is resource accountability: In many companies the storage resources and production LAN resources are managed by different groups, which can cause confusion as to who provides budget for future equipment, or who troubleshoots problems on the network. This design reflects the separation of storage and LAN resources. The design illustrated in Figure 3-27 leverages existing Ethernet equipment and separates the iSCSI SAN traffic from the production LAN traffic. The design could be affected by production LAN problems and may induce production LAN problems. This design may cause equipment ownership issues.
3-31
3
N o te: In this exam p le an E thernet VLAN has been created on two of the production LAN switches to create a dedicated L A N for the iSC S I traffic.
6 13 4 12 10
5 17 13 11 14 13 8
2 8
21 16 22 1 5 20 1 9
21
16
1 2 9 17
22 12
Nerwork Servers
6 10 12
11
13 14 118 7 3
1 5 21
13
1 9 22 16 20
21
2 8
16
1 2 9 17
12 22
1 7
1 1 7
iSCSI VLAN
Production LAN
gl
M u ltiprotocol Routers
10/100/1000 Ethernet Switches

J4908A 2 2 p o r t G i g -T / G B I C
gl
Production LAN Link iSCSI VLAN Link Network Servers
Figure 3-27 Dedicated iSCSI VLAN By using existing Ethernet VLAN infrastructure iSCSI SAN traffic is still separated from the production LAN. This will help reduce the possibility of iSCSI SAN traffic and production LAN traffic from interfering with each other. Nevertheless, hardware or VLAN software problems could still affect both production and storage traffic. The problem of ambiguous resource ownership also comes into play. The design illustrated in Figure 3-28 leverages existing Ethernet equipment and does not separate iSCSI storage traffic from production LAN traffic. This design is subject to production LAN problems (both hardware and software) and has a greater possibility of inducing production LAN problem. This design may cause equipment accountability issues.
3-32
3
Note: In this example the network servers are using the same LAN to access FC targets over iSCSI and service Ethernet clients.
4 6 5 13 10 12
Network Servers
11 17 14 13 18
15 1621 19 22 20
21
16
2 8
12 917
12 22
6 12
10 4 5
13 6
11
13
17
121417 3 18 9
16 20
21 21 22 15 19 22 12

7
16
2 8
1 1 7
Production LAN
1 7 1
gl
Multiprotocol Routers
10/100/1000 Ethernet Switches

gl
Standard Ethernet Link iSCSI Ethernet Link Network Servers

Figure 3-28 iSCSi utilize production LAN While using the production LAN will work (and is supported by Brocade), there is no real benefit from this design. By using the separate LAN/VLAN option, all storage-related traffic--including backup traffic--is removed from the production LAN. Guideline: Brocade recommends keeping the iSCSI traffic on a separate LAN or VLAN.
3.6.3. Multiprotocol Router FCIP Connections

Any port of the Multiprotocol Router is capable of supporting FCIP and for reasons discussed later, it is very valuable to use FCIP with FC-to-FC routing. It is recommended that the Gigabit Ethernet interface used for the FCIP connection be part of a separate LAN or VLAN for many of the reason this recommendation is made for iSCSI connections (see Interfacing the Multiprotocol Router iSCSI with Gigabit LANs on page 3-31). FCIP by itself will merge the two fabrics connected via FCIP. When FCIP is used standalone, the loss of a WAN link will result in a fabric reconfiguration. When FCIP is used in conjunction with FC routing services (see Figure 3-29), the loss of a WAN link will result in no reconfiguration of the edge fabrics. The front domain and the translate domain are kept intact as long as the Router is connected to the SAN. FC-to-FC routing allows dissimilar fabrics (PID, Fabric OS versions, etc.) to inter-operate without disruption to either fabric.
3-33
3
An FCIP connection results in the creation of a VE_Port on the Multiprotocol Router. A VE_Port is essentially an E_Port and many of the rules that apply for connecting Edge fabrics to the Multiprotocol Router also apply to connecting FCIP connected edge fabrics. For redundancy purposes, it is recommended to have two FCIP links per routed fabric and to connect one connection to each backbone -- if redundant backbone are utilized. Where it is practical to have two FCIP links and redundant routed fabrics are utilized, connect to each FCIP link to a backbone fabric in each routed fabric. Guideline: It is recommended that the Gigabit Ethernet interface used for the FCIP connection be part of a separate LAN or VLAN.
Edge F abric 1 EX_Ports a VE-port (GigE + F IP) C 3800 E Fabric 2 dge E ort _P 3800 a
WAN Link E _Port Backbone Fabric
Figure 3-29 Integrate FC-to-FC Routing with FCIP for superior functionality and fault isolation.
3.6.4. Multiprotocol Router Connection Layout

The multiprotocol router is capable of supporting many connection types: FC, iSCSI, and FCIP. There are no performance or availability advantages to connecting a particular connection type to a specific port. However, for operational efficiency, there are benefits to partitioning the Multiprotocol Router ports based on expected usage and growth -- if this information is known, and following this scheme for all router utilized in the data center/ enterprise.
F C IP
FC to FC Routing
Or
F C IP M u ltis w itc h Backbone E -ports F C t o F C R o u tin g
Figure 3-30 Partition router ports based on expected use and growth needs for operational efficiency
3-34

MPR Design Guide

Caricato da

Informazioni sul documento

Descrizione originale:

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

MPR Design Guide

Caricato da

Copyright:

Formati disponibili

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Publication Number: 53-0000552-01 Publication Date: May 7, 2004

Brocade Communications Systems, Incorporated

Determining An Appropriate MetaSAN Architecture

Multiprotocol Router SAN Design Concepts

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Notes and Guidelines

Guideline: A guideline provides a tip or a recommendation.

Warnings alert you to potential damage to hardware, firmware, software, or data.

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Brocade SilkWorm Multiprotocol Router SAN Design Guide

1.1. The SilkWorm Multiprotocol Router

Brocade SilkWorm Multiprotocol Router SAN Design Guidelines

1.1.1. FC-to-FC Routing Service

1.1.2. FCIP Tunneling Service

1.1.3. iSCSI Gateway Service

Brocade SilkWorm Multiprotocol Router SAN Design Guidelines

1.1.4. The Innovative Multiprotocol Router Platform

Brocade SilkWorm Multiprotocol Router SAN Design Guidelines

Brocade SilkWorm Multiprotocol Router SAN Design Guidelines

Determining An Appropriate MetaSAN Architecture

2.1. Tactical And Strategic Use Of The Multiprotocol Router

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Determining An Appropriate MetaSAN Architecture

Data Migration using the Multiprotocol Router

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Determining An Appropriate MetaSAN Architecture

Storage service provider (SSP) architecture

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Determining An Appropriate MetaSAN Architecture

2.2. Key Considerations For Determining A SAN Architecture

2.2.1. Primary Non-technical SAN Architecture Considerations

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Determining An Appropriate MetaSAN Architecture

2.2.2. Primary Technical SAN Architecture Considerations

2.3. How Big Is Big And How Big Is Too Big?

2.4. Interconnected Fabrics

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Determining An Appropriate MetaSAN Architecture

2.5. A High Port Count Director Or A Router?

2.6. MetaSAN Architecture Selection FlowChart

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Determining An Appropriate MetaSAN Architecture

Is there one team responsible to administrate the SAN?

Interconnect SAN Islands using the multiprotocol router

Are you comfortable in deploying fabrics greater than 200-ports?

Build high port count fabrics

metaSAN Architecture Selection Flowchart

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Determining An Appropriate MetaSAN Architecture

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Multiprotocol Router SAN Design Concepts

3.1. Components of a MetaSAN

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Multiprotocol Router SAN Design Concepts

A Storage Area Network

Brocade SilkWorm Multiprotocol Router SAN Design Guide

Multiprotocol Router SAN Design Concepts

Interfabric Link (IFL)

Multiprotocol Router SAN

Brocade SilkWorm Multiprotocol Router SAN Design Guide