Deploy Fibre Channel Over Ethernet (FCoE) Solution With IBM® BladeCenter® and System X® Using The IBM Converged Switch B32 (Brocade 8000)

Deploy Fibre Channel over
Ethernet (FCoE) Solution with

IBM BladeCenter and
System x using the IBM
Converged Switch B32
(Brocade 8000)
Khalid Ansari
kmansari@us.ibm.com
IBM Advanced Technical Support
Copyright International Business Machines Corporation 2009
Revision History
1.0 June 14, 2009
Initial Version
Notices:
This paper is intended to provide information regarding IBM BladeCenter Open Fabric Manager. It
discusses findings based on configurations that were created and tested under laboratory
conditions. These findings may not be realized in all customer environments, and implementation
in such environments may require additional steps, configurations, and performance analysis. The
information herein is provided AS IS with no warranties, express or implied. This information
does not constitute a specification or form part of the warranty for any IBM or non-IBM products.
Information in this document was developed in conjunction with the use of the equipment
specified and is limited in application to those specific hardware and software products and levels.
The information contained in this document has not been submitted to any formal IBM test and is
distributed as is. The use of this information or the implementation of these techniques is a
customer responsibility and depends on the customers ability to evaluate and integrate them into
the customers operational environment. While each item may have been reviewed by IBM for
accuracy in a specific situation, there is no guarantee that the same or similar results will be
obtained elsewhere. Customers attempting to adapt these techniques to their own environments
do so at their own risk.
IBM may not officially support techniques mentioned in this document. For questions regarding
officially supported techniques, please refer to the product documentation, announcement letters,
or contact the IBM Support Line at 1-800-IBM-SERV.
This document makes references to vendor-acquired applications or utilities. It is the customer
responsibility to obtain licenses of these utilities prior to their usage.
Copyright International Business Machines Corporation 2009. All rights reserved.
U.S. Government Users Restricted Rights Use, duplication, or disclosure restricted by GSA ADP
Schedule Contract with IBM Corp.
Table of Contents
Table of Contents ........................................................................................................................3
Executive Overview .................................................................................................................4
How does Fibre Channel Over Ethernet Work? .........................................................................5
Logical Connectivity ..............................................................................................................6
IBM BladeCenter and System x FCoE Product Details .............................................................9
10Gb Pass-Thru Module Features........................................................................................9
QLogic 2-port 10 Gb Converged Network Adapter (CFFh) for IBM BladeCenter (QMI8142) 10
QLogic 2-port 10Gb Converged Network Adapter for IBM System x....................................10
Brocade 2-port 10 Gb Converged Network Adapter for IBM System x .................................11
Brocade 8000 CEE Switch ...................................................................................................11
Implementing Blade Boot from SAN Solution with BOFM Enabled........................................14
Useful Links ........................................................................................................................35
Trademarks ...........................................................................................................................36
Executive Overview
This document illustrates the technical implementation details of Fibre
Channel over Ethernet (FCoE) solution with IBM BladeCenter. The objective of
this document is to demonstrate the configuration steps for blade boot from FC
SAN using the BladeCenter 10Gb Ethernet Pass-Thru Module and the Brocade
B32 Converged Enhanced Ethernet (CEE) solution.
The BladeCenter chassis-based FCoE solution will also leverage
BladeCenter Open Fabric Manager (BOFM) for Ethernet MAC address and Fibre
Channel WWPN virtualization solution to enable blade pre-provisioning, simplify
blade server re-deployment and automatic blade failover using the Advanced
Open Fabric Manager. In addition to address virtualization and automatic blade
failover, BOFM simplifies and automates the configuration tasks such as enable
adapter BIOS and selectable boot device settings performed in the HBA firmware
utility. Traditionally, this requires the administrator to interrupt the blade boot
process and change the default HBA settings on one or more blades individually.
This can be painful and time consuming if the configuration is performed
manually on hundreds of blade servers. With BOFM, you can simply modify the
configuration file and apply the profile to one or more blades in multiple chassis
at the same time.
The following sections list the configuration process for blade boot from
SAN using the Qlogic CNA attached to DS4700 and System x rack server booting
locally, and access IBM DS4700 using the Qlogic CNA.
The FCoE test environment setup includes the following hardware:
IBM BladeCenter H Chassis
IBM HS21XM Blade Server
42C1830 - QLogic 2-port 10Gb CNA (CFFh) for IBM BladeCenter
BladeCenter 10Gb Ethernet Pass-Thru Module
IBM DS4700 Storage Subsystem
Brocade 8000 FCoE Switch
The following figure shows the final topology implemented for blade boot from
SAN environment.
Figure 1: FCoE Physical Topology
How does Fibre Channel Over Ethernet Work?

Fibre Channel over Ethernet (FCoE) enables you to transport Fibre
Channel (FC) protocols and frames over Converged Enhanced Ethernet (CEE)
networks. CEE is an enhanced Ethernet that enables the convergence of various
applications in data centers (LAN, SAN, and HPC) onto a single interconnect
technology.
FCoE provides a method of encapsulating FC traffic over a physical
Ethernet link. FCoE frames use a unique EtherType that enables FCoE traffic and
standard Ethernet traffic to be carried on the same link. FC frames are
encapsulated in an Ethernet packet and sent from one FCoE-aware device across
an Ethernet network to another FCoE-aware device. The FCoE-aware devices
may be FCoE end nodes (ENodes) such as servers, storage arrays, or tape drives
on one end and FCoE forwarders on the other end. FCoE Forwarders (FCFs) are
switches providing FC fabric services and FCoE-to-FC bridging function.
The motivation behind using CEE networks as a transport mechanism for
FC arises from the desire to simplify host protocol stacks and consolidate
network interfaces in data center environments. FC standards allow for building
highly reliable, high-performance fabrics for shared storage, and these

characteristics are what CEE brings to data centers. Therefore, it is logical to
consider transporting FC protocols over a reliable CEE network in such a way that
it is completely transparent to the applications. The underlying CEE fabric is
highly reliable and high performing, the same as the FC SAN.
In FCoE, ENodes discover FCFs and initialize the FCoE connection through the
FCoE Initialization Protocol (FIP). The FIP has a separate EtherType from FCoE.
The FIP includes a discovery phase in which ENodes solicit FCFs and FCFs
respond to the solicitations with advertisements of their own.
At this point, the ENodes know enough about the FCFs to log into them. The
fabric login and fabric discover (FLOGI/FDISC) for a VN-to-VF port connection is
also part of the FIP.
FCoE services include:
FC fabric services for FCoE VN_port devices, which provide access to FCoE
VN_port devices similar to those provided by FC F_port to FC N_port
devices
FCoE-to-FC switching and translation services:
- FCoE servers to/from FC targets
- FCoE targets to/from FC servers
FCoE features include:
FIPsolicited and unsolicited FCF advertisements (per June 10, 2008,

version of T11-FC-BB5)
FCoE FLOGI
VF_ports
N_port ID virtualization (NPIV) on VF_ports (up to 1000 devices per FCF)
NOTE
NPIV is an FC facility allowing multiple N_port IDs to share a single
physical N_port. This allows multiple FC initiators to occupy a single
physical port.
For FCoE ENodes, there is support for only directly-connected links.
Flow isolation
Logical Connectivity
Figure 2 shows a logical view of the Brocade 8000 CEE switch. There are
two independent switches, one for CEE and the other for FC. There are six
embedded FCoE virtual ports bridging the two switches. Each of these port is a
10-Gigabit FCoE port. The Fabric OS has FCoE-specific CLI commands for the
configuration and management of these FCoE ports.
The FCoE ports differ from the usual concept of a port in a Brocade switch
as they are embedded ports and are not directly associated with an external
physical port on the switch. Configuration of the FCoE ports through the regular
Fabric OS FC CLI is disabled; only the Fabric OS FCoE CLI commands can be
used to configure and manage these ports.
The FCoE ports are displayed as ports in some show command displays.
The data displayed for a single FCoE port is the sum of the individual ports
comprising the FCoE port. CEE statistics are not shown for the FCoE ports.
NOTE
In the Fabric OS CLI the FCoE ports can be enabled or disabled; there is no other
configuration required. Each of the embedded FCoE ports is used explicitly for
FCoE VF_port service and provide four MAC addresses and flow isolation for up
to four flows per FCoE port.
Figure 2: FCoE Logical Topology
The FCoE VF_ports provide FC services to FCoE initiators and targets, as

well as an FCoE-to-FC bridging service that allows FCoE initiators to access FC
targets and conversely allows FC targets to access FCoE initiators.
Brocade's implementation of FCoE for the Brocade 8000 CEE switch

provides integral N_port ID Virtualization (NPIV) support. Multiple VN_port
devices can log in to a single FCoE VF_port interface. Up to 1,000 VN_port
devices can log in to a single Brocade 8000 CEE switch. Each of the embedded
FCoE ports supports four logical traffic paths. These four logical traffic paths
share the FCoE port bandwidth. Any single traffic path may use the entire 10Gbps of FCoE port bandwidth if available, but it must share the bandwidth
equally with the other logical path traffic flows if more than one path is active.
The bandwidth available to any single logical traffic path (and therefore any
single FCoE-to-FC traffic flow) is between 2.5 Gbps and 10 Gbps.
While they share the FCoE port bandwidth, the logical traffic paths are
independent from one another in the event of downstream congestion. If the
traffic flowing on one logical path stalls because of congestion, the traffic flowing
on the other logical paths on the same FCoE port is not affected. Note however,
that a logical traffic path is not limited to a single FCoE-to-FC traffic flow. If
multiple traffic flows are sharing the same logical traffic path, congestion
independence between the flows cannot be enforced.
The Qlogic 10Gb FCoE Adapter if installed on the rack server can directly
connect to a top of the rack 10Gb CEE switch. The Qlogic 10Gb FCoE Adapter if
installed on the blade server today can connect to external 10Gb CEE switch via
the 10Gb Pass-thru modules installed in bays 7 and 9 of the BCH chassis. The
following figure shows the logical path from the host to the native FC storage:
Figure 3: Host to Storage logical configuration
At a high-level, the following configuration tasks needs to be completed to

occur in order to establish end-to-end FCoE connectivity:
The host must have FCoE adapter installed and connected to CEE + FCoE
switch
10Gb Ethernet and 8Gb FC device drivers must be installed on the host
The link has to be established from FCoE to the corresponding 10Gb CEE
port
The FC device driver logs into the fabric as a VN_port.
The FC target logs into the fabric as an N port device
The FCoE host and F Target WWPNs are zoned
The FC LUN(s) are mapped to the host
The Host sees the LUNs from the FCoE HBA firmware if boot from SAN
The host sees the LUNs from the OS if booting locally
IBM BladeCenter and System x FCoE Product Details
10Gb Pass-Thru Module Features
14 internal and 14 external 10Gb Copper or Optical ports
10Gb End-to-end unblocked access with no packet drop
Low cost solution for Clients to connect to any Top Of Rack 10Gb or CEE
capable Switch
Part of first FCoE Convergence solution offered on BladeCenter
BC-H & BC-HT Chassis ONLY
14 Internal 14 External Ports
No Configuration necessary
No On-Board Management
QLogic 2-port 10 Gb Converged Network Adapter (CFFh) for IBM

BladeCenter (QMI8142)
Figure 4: Qlogic 2 port 10 Gb CNA
Combo Form Factor (CFFh) PCI Express x8 2.0 adapter
Communication module: QLogic ISP8112
Two ports with XAUI interfaces for PCI-E x8 to the HSSM
Support for up to two CEE HSSMs in BC-H or BC-HT chassis
Support for 10Gb Converged Enhanced Ethernet (CEE)
Support for Fiber Channel over Converged Enhanced Ethernet (FCoCEE)
Full hardware offload for Fibre Channel over Converged Enhanced

Ethernet (FCoCEE) protocol processing
Support for IPv4 and IPv6
Support for SAN boot over CEE, PXE boot, and iSCSI boot
Support for Wake on LAN
Support for BladeCenter Open Fabric Manager for BIOS, UEFI, and FCode
QLogic 2-port 10Gb Converged Network Adapter for IBM System x
PCI Express x8 2.0 Generation 2 compliance
Two SFP+ cages for either SFP+ Fiber SR or SFP+ Active Copper
Standard PCI Express half length card with low profile form factor
Support for both standard PCI-E slot and low profile PCI-E slot
10
Support for Fibre Channel over Converged Enhanced Ethernet (FCoCEE)
Full hardware offload for FC protocol processing
Support for BladeCenter Open Fabric Manager for BIOS, UEFI, and FCode
Brocade 2-port 10 Gb Converged Network Adapter for IBM System x
Compliance with PCI Express x8 2.0 Generation 2 servers
Two SFP+ cages for either SFP+ Fiber SR or SFP+ Active Copper
Standard PCI Express half length card with low profile form factor
Support for both a standard PCI-E slot and a low profile PCI-E slot
Support for Fibre Channel over Converged Enhanced Ethernet (FCoCEE)
Full hardware offload for FCoE protocol processing
Brocade 8000 CEE Switch

Fibre Channel ports
Eight Fibre Channel universal (E, F, M, and FL) ports with 1, 2, 4, and 8
Gbps
CEE ports
24 ports with 10 Gigabit Ethernet
FCoE features
Complete T11 FCoE entity and FCoE bridging
The FCoE translation entity built into the hardware engine provides:
Detection of Fibre Channel encapsulation and redirection of FCoE fabric

login frames
Encapsulation of Fibre Channel frames in FCoE Ethernet packets (FC >
11
FCoE)
Extraction of Fibre Channel frames from FCoE Ethernet packets (FCoE >
FC)
Mapping of Fibre Channel destination Virtual Fabrics and destination

FC_ID to Ethernet Virtual LAN and destination MAC addresses
Fabric-Provided MAC Addresses (FPMAs) enable new Ethernet MAC addresses to

be created using the FC_ID assigned by the fabric
CEE features
Data Center Bridging eXchange (DCBX)
Priority-based Flow Control (PFC) IEEE 802.1Qbb
Enhanced Transmission Selection (ETS) IEEE 802.1Qaz
Performance
Fibre Channel: 1, 2, 4, and 8 Gbps line speed full duplex
CEE: 10 Gbps line speed
ISL Trunking
Frame-based ISL Trunking (optional license) enables up to eight ports

between a pair of switches to be combined into a logical ISL with speeds
of up to 64 Gbps (128 Gbps full duplex) for optimal bandwidth utilization
and load balancing; exchange-based load balancing across ISLs with DPS
(included in Fabric OS)
Link aggregation (10 Gigabit Ethernet)
Link Aggregation Control Protocol (LACP), Brocade-enhanced and 802.3ad

standards-based
Maximum frame size
2112-byte Fibre Channel payload; 9048-byte Ethernet frame
Classes of service
Class 2, Class 3, Class F (inter-switch frames)
Port types
FL_Port, F_Port, M_Port (Mirror Port), E_Port; self-discovery based on

switch type (U_Port); optional port type control
12
Data traffic types
Fabric switches supporting unicast, multicast (255 groups), and broadcast
Media types
Fibre Channel media type: Hot-pluggable, industry-standard Small Form

Factor Pluggable (SFP) and SFP+, LC connector; Short-Wave Laser (SWL)
and Long-Wave Laser (LWL); distance depends on fiber optic cable and
port speed; supports SFP+ (2, 4, and 8 Gbps) and SFP (1, 2, and 4 Gbps)
optical transceivers
CEE media type: Hot-pluggable, Brocade 10 Gigabit Ethernet SFP+

supports any combination of Short-Reach (SR) and Long-Reach (LR)
optical transceivers; Brocade copper twinax cables of one, three, or five
meters
USB
1 USB port for firmware download, support save, and configuration

upload/download
Fibre Channel fabric services
Simple Name Server (SNS), Registered State Change Notification (RSCN),

NTP, RADIUS, LDAP, Reliable Commit Service (RCS), Dynamic Path
Selection (DPS), Enhanced Group Management (EGM), and Web Tools;
optional fabric services include Fabric Watch, ISL Trunking, and Advanced
Performance Monitoring
CEE services
Spanning Tree Protocol (STP, MSTP, RSTP, VLAN Tagging (802.1q), MAC
address learning and aging; native FCoE switching; IEEE 802.3ad Link
Aggregation (LACP); access control lists based on VLAN, source,
destination address, and port; eight priority levels for QoS and 4k VLANs;
Priority-based Flow Control (PFC); Data Center Bridging eXchange
(DCBX)-Capabilities Exchange; Enhanced Transmission Selection (ETS)
Licensing options
Fabric OS 6.1.2_cee includes the following optional features that can be enabled
13
via license keys and are applicable only to the Fibre Channel ports of the Brocade
8000:
Brocade Fibre Channel ISL Trunking
Brocade Advanced Performance Monitoring
Brocade Fabric Watch
Management software
SSH v2, HTTP/HTTPS, SNMP v1/v3, Telnet; SNMP (FE MIB, FC

Management MIB, RMON, and IF-MIB for CEE); Web Tools; Data Center
Fabric Manager (DCFM) Professional and Enterprise; SMI-S; RADIUS
Management access
One 10/100/ Gigabit Ethernet (RJ-45), in-band over Fibre Channel, one
serial port (RJ-45), and one USB port
Implementing Blade Boot from SAN Solution with BOFM Enabled

The following sections illustrate the step by step configuration process to
successfully boot the blade with Qlogic CNA adapter from DS4700 attached to
Brocade 8000 CEE switch. The HS21XM blade server with Qlogic CNA Adapter is
connected to 10Gb Pass-thru module installed in bays 7 and 9 of the BCH
chassis.
Major setup and configuration tasks:
Ensure FCoE HBA has the latest firmware that is also compatible with
BOFM
The Blade BIOS and BMC must be compatible with BOFM
BOFM is enabled on the blade slot
Connect the cables from the 10Gb Pass-thru to 10Gb interface on Brocade
8000
The link status shows UP on the 10Gb Ethernet Interface on Brocade 8000
Configure the Brocade 8000

o CEE map
o Port based VLAN
o CEE Mode
14
o Trunking
Configure and Enable Zoning
Define Host and Storage Partitioning on the Storage
Note: Verify and confirm that BOFM is enabled on the Blade slot and the status is
Normal, from the AMM GUI, select Blade tasks Configuration Open Fabric
Manager and the blade slot. The ports 1 and 2 are on-board 1Gbps Broadcom NICs
and ports 5 & 7 are 10Gbps FCoE ports. Both Ethernet and Fibre Channel ports map to
high-speed switch modules in bays 5 and 7. So, when creating the BOFM configuration,
you must select ports 5 and 7 to enable BOFM on the Qlogic CNA.
The following figure lists the WWPNs of the FCoE HBA ports virtualized by
BOFM and the WWPNs of the storage controller is also applied on the HBA.
Important: Once BOFM is enabled on the blade slot then it will automatically
configures the following tasks:
Enable Qlogic FC Adapter BIOS
Enable Selectable Boot Settings
Configures the FC Target WWPNs and LUN ID
BOFM eliminates the manual configuration tasks listed above. BOFM also
overwrites the adapter BIOS and Selectable Boot Settings configuration
performed manually from the HBA firmware utility.
15
Figure 5: BOFM Fibre Channel Port mapping for Qlogic CNA
1. Verify the Link status on 10Gb Ethernet interface on Brocade 8000 CEE.
From the Brocade Web interface, select Port Admin CEE interfaces as
shown below. The blade is connected to Ethernet interfaces 3 and 14 on
the Brocade 8000 switch via the pass-thru module. The link status on
ports 3 & 14 shows Online
16
Figure 6: Brocade 8000 CEE Interface Configuration and Management GUI
2. Configure the Brocade 8000 CEE

a. Create CEE map and configure QOS as shown below:
Switch:admin>cmsh (change shell to configure Ethernet )
switch# configure terminal
Create a CEE MAP to configure PFC (Priority Flow Control) and ETS (Enhanced
Transmission Selection):
switch(config)# cee-map fcoe-test1
switch(config-ceemap)# priority-group-table 1 weight 40 pfc
switch(config-ceemap)# priority-group-table 2 weight 60
switch(config-ceemap)# priority-table 2 2 2 1 2 2 2 2
switch(config-ceemap)# exit
b. Configure LLDP with DCBX attributes:

switch(config)# protocol lldp
switch(conf-lldp)# advertise dcbx-fcoe-app-tlv
switch(conf-lldp)# advertise dcbx-fcoe-logical-link-tlv
c. Create an FCoE capable VLAN:
17
Switch(config)# interface vlan 1002

Switch(conf-vlan100)# fcf forward
Switch(conf-vlan)# exit
d. Administratively enable the interface:

switch(config)# interface TenGigabitethernet 0/3
switch(conf-if-te-0/3)# no shutdown
Switch(config)# interface TenGigabitethernet 0/14
switch(conf-if-te-0/14)# no shutdown
e. Configure the interfaces as L2 (Configures Trunk Ports):

switch(conf-if-te-0/3)# switchport
switch(conf-if-te-0/3)# switchport mode converged
switch(conf-if-te-0/3)# switchport converged allowed vlan add 1002
switch(conf-if-te-0/14)# switchport
switch(conf-if-te-0/14)# switchport mode converged
switch(conf-if-te-0/14)# switchport converged allowed vlan add 1002
f. Apply CEE MAP to the interfaces:

switch(conf-if-te-0/3)# cee fcoe-test1
switch(conf-if-te-0/3)# exit
switch(conf-if-te-0/14)# cee fcoe-test1
switch(conf-if-te-0/14)# exit
The following commands can be used to verify the configuration from the
Brocade 8000 CLI:
The fcoe loginshow displays the FCoE device WWPNs connected to the
fabric. The WWPNs in blue font belongs to the blade in bay 12 connected to
ports 3 & 14 on the Brocade 8000 switch. The WWPNs listed below should also
be available from the zone configuration menu to add as zone members with FC
target WWPNs.
18
swd77:admin> fcoe --loginshow 8

Number of connected devices: 2
=======================================================================
===================================================
Peer Type
Connect Info
Device WWN
Device MAC
Session MAC
FCoE Port MAC
Te port
=======================================================================
===================================================
FCOE_DEVICE Direct 21:00:00:c0:dd:10:0e:1b 00:c0:dd:10:0e:1b 0e:fc:00:01:08:04 00:05:1e:76:71:00
Te 0/0
FCOE_DEVICE Direct 21:80:00:e0:8b:00:01:1e 00:c0:dd:10:10:99 0e:fc:00:01:08:03 00:05:1e:76:71:03
Te 0/3
swd77:admin>
swd77:admin> fcoe --loginshow 11
Number of connected devices: 1
=======================================================================
===================================================
Peer Type
Connect Info
Device WWN
Device MAC
Session MAC
FCoE Port MAC
Te port
=======================================================================
===================================================
FCOE_DEVICE Direct 21:80:00:e0:8b:00:01:1f 00:c0:dd:10:10:9b 0e:fc:00:01:0b:02 00:05:1e:76:71:0e
Te 0/14
swd77:admin>
The following switchshow output displays native FC devices connected to

ports 0 - 7 and FcoE devices connected to ports 8 13.
swd77:admin> switchshow
switchName:
swd77
switchType:
76.7
switchState: Online
switchMode:
Native
switchRole:
Principal
switchDomain: 1
switchId:
fffc01
switchWwn:
10:00:00:05:1e:76:71:80
zoning:
ON (FCoE_Test)
switchBeacon: OFF
Area Port Media Speed State
Proto
=====================================
0 0 id N4 Online
FC F-Port 50:0a:09:82:87:e9:5d:03
1 1 -- N8 No_Module FC
2 2 id N4 Online
FC F-Port 20:15:00:a0:b8:26:12:10
4 4 id N4 Online
FC F-Port 50:0a:09:82:97:e9:5d:03
6 6 id N4 Online
FC F-Port 20:14:00:a0:b8:26:12:10
8 8 -- 10 Online
FCoE F-Port 3 NPIV public
9 9 -- 10 Online
10 10 -- 10 Online
11 11 -- 10 Online
12 12 -- 10 Online
FCoE F-Port 20:0c:00:05:1e:76:71:80
13 13 -- 10 Online
FCoE F-Port 20:0d:00:05:1e:76:71:80
swd77:admin>
19
The following show running-config output lists the configuration on the

10Gb Ethernet switch, CEE map applied, and FCoE forwarding and converged
mode enabled on the FCoE interfaces.
Note: The show running-config output displayed below is truncated. It does
not list the configuration for all the TenGigabitEthernet interfaces.
swd77#sh running-config
!
no protocol spanning-tree
!
cee-map fcoe-test1
priority-group-table 1 weight 40 pfc
priority-group-table 2 weight 60
priority-table 2 2 2 1 2 2 2 2
!
interface Vlan 1
!
interface Vlan 1002
fcf forward
!
interface TenGigabitEthernet 0/0
switchport
switchport mode converged
switchport converged allowed vlan add 1002
no shutdown
lldp fcoe-priority-bits 0x8
cee fcoe-test1
!
no shutdown
!
no shutdown
!
switchport
no shutdown
cee fcoe-test1
!
switchport
no shutdown
cee fcoe-test1
20
!
no shutdown
!
switchport
no shutdown
cee fcoe-test1
!
shutdown
!
switchport
no shutdown
cee fcoe-test1
!
shutdown
!
protocol lldp
advertise dcbx-fcoe-app-tlv
advertise dcbx-fcoe-logical-link-tlv
!
line console 0
login
line vty 0 31
login
!
end
swd77#
Verify Zone Configuration

The cfgshow output shows the defined and active zoning on the Brocade
8000 switch. The active zoning includes the zone for Blade in slot 12 :
BCH5_BS12p1_DS4700 lists the blade and storage WWPN so that there is a
single path from the host to the storage for initial Microsoft Windows 2003 R2
SP2 OS install. Additional paths can be added in the zoning and on the storage
once the OS install completes successfully on the boot LUN.
Note: The zone configuration information listed is truncated. It does not list the
defined configuration and only lists the active configuration.
swd77:admin> cfgshow
21
Effective configuration:
cfg: FCoE_Test
zone: BCH5_BS12p1_DS4700
20:15:00:a0:b8:26:12:10
21:80:00:e0:8b:00:01:1e
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:10:99
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:10:9b
zone: X3550p1_DS4700
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:0e:19
zone: X3550p2_DS4700
20:14:00:a0:b8:26:12:10
20:15:00:a0:b8:26:12:10
21:00:00:c0:dd:10:0e:1b
swd77:admin>
Note: For instructions to configure zoning, refer to the Brocade FOS guide.
Verify Storage Configuration
From the DS4000 Storage Manager GUI, verify and confirm that the boot LUN is
mapped to the host as shown in figure below:
Figure 7: DS4000 Logical Drive Mapping View
Once the above configuration steps are completed, the blade is now setup to
boot from SAN and ready for OS installation.
22
Installing Windows 2003 R2 SP2 32 bit Operating System

1. Disable the on-board SAS disk controller as shown in figure
Figure 8: Disable on-board disk controller on HS21XM blade server
2. Download the latest device driver for the FCoE adapter and extract to a
floppy diskette
3. The FCoE device driver contents should include the files listed in the
following figure:
23
Figure 9: Windows 2003 Driver Diskette content for Qlogic CNA
4. Setup the boot sequence for blade so that it boots from CD first. From
AMM GUI select Blade Tasks Configuration Boot Sequence Blade.
5. Insert the OS CD1 and power ON the blade. For this test, we will be
installing Windows 2003 R2 SP2 integrated.
Note: The Service Pack 2 integrated is required for successful install of
Win2003 R2 image. Without SP2 integrated, the OS will not load the FCoE
device driver correctly.
6. Once the Windows 2003 R2 SP2 CD is detected, immediately look for F6
prompt and quickly press F6 key.
24
Figure 10: F6 prompt during Windows 2003 Install
7. If F6 key was pressed during the initial setup, then the following prompt
shown in figure below will be displayed that allows you to load the FCoE
driver. Ensure that the USB Floppy diskette is inserted and select S to
specify Additional Device.
Figure 11: Install FCoE driver during Windows Setup process
8. The following menu is displayed after reading the contents of the diskette.
25
The Windows Setup successfully recognizes the driver and is ready to

load. Press Enter key to load the FCoE driver from the diskette.
Figure 12: Windows Setup correctly recognizes the FCoE driver
9. The following Windows Setup menu shows that the FCoE driver was
successfully read from the diskette. Do not remove the diskette as
Windows Setup needs to copy the driver from the diskette. Press Enter to
continue as no additional driver is needed for the blade to boot from SAN.
26
Figure 13: Windows Setup will load the FCoE driver for Qlogic CNA
10. Press F8 key to agree to Windows Licensing Agreement and proceed with
OS installation.
Figure 14: Windows License Agreement screen
11. The following screen shows that the Windows Setup can see the 20GB
27
boot LUN via the Qlogic CNA. Press C to create partition.
Figure 15: Windows 2003 Disk Partition menu
12. The following figure shows that new partition is created on boot LUN ID
=0, bus = 0 on qlfcoe adapter:
Figure 16: Windows 2003 Disk Partition menu
28
13. From the following menu, select the File System and Format type and
press Enter key.
Figure 17: Windows 2003 Setup, Create File system menu
14. The following figure shows the Windows Setup is formatting the boot
LUN:
Figure 18: Windows 2003 File System Format in progress
15. The following figure shows the Windows Setup is copying the FCoE device
29
driver from the USB floppy diskette:
Figure 19: Windows 2003 Setup, copying the FCoE driver
16. The following figure shows that the Windows 2003 R2 SP2 OS installation
process in progress:
Figure 20: Windows 2003 Installing Devices menu
From this point, you can follow the prompts to complete the Windows 2003 OS
30
installation. Remember that the OS was installed on the boot LUN via the single
path. Download and install RDAC driver on the host and add additional paths
from the host to the boot LUN.
Installing Multipath and Failover driver compatible with the Storage

Subsystem in use
1. Download and install the latest MPIO driver available at the DS4000
download site. The MPIO driver to access the DS3K/DS4K/5K storage
subsystem is embedded in the DS4000 storage manager application. For
other storage subsystem, refer to the storage interop matrix and install the
compatible driver.
2. Download the latest Storage Manager application package from the
DS3K/DS4K/5K storage download site.
3. Install the MPIO driver by selecting the Host option as shown in the
following figure:
31
Figure 21: IBM DS Storage Manager Installation Menu
Figure 22: DS4000 Storage Manager Pre-Installation Summary Menu
32
3. The following figure shows that the MPIO driver is successfully installed.
Figure 23: Successful MPIO driver installation on the Host
4. The following figure shows that there are multiple paths from the host to
the boot LUN with MPIO driver:
33
Figure 24: The Windows Device Manager shows the MPIO driver is installed
5. You can perform the failover test by disabling one path at a time from the
host to storage. This can be done by disabling the switch ports to which
the blade or storage ports are connected.
6. Once the path is disabled on host side or the storage side, the OS should
still be available via the alternate path. The MPIO driver automatically
moves the LUN to the available path.
Conclusion
This concludes the configuration process to deploy blade boot from SAN using
the Qlogic 10 Gbps CNA attached to the Brocade 8000 CEE switch and IBM
DS4700 storage subsystem.
34
Useful Links
1.
IBM eServer BladeCenter for Converged Network Adapters Supported

Software
42C1830 - QLogic 2-port 10Gb CNA (CFFh) for IBM BladeCenter (FRU P/N
42C1832, Card P/N 42C1831):
http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/Product
_detail_new.aspx?oemid=394&companyid=6
2.
IBM System x Converged Network Adapters Supported Software

42C1800 - QLogic 10Gb CNA for IBM System x (FRU P/N 42C1802, Card
P/N 42C1801)
http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/Product
_detail_new.aspx?oemid=394&companyid=6
3.
IBM DS Storage Subsystem Supported Software

4.
https://www947.ibm.com/systems/support/supportsite.wss/selectproduct?brandind=5
000028&familyind=0&oldbrand=0&oldfamily=0&oldtype=0&taskind=1&ps
id=bm
QLogic 2-port 10Gb Converged Network Adapter (CFFh) for IBM BladeCenter
Tech Note
http://www.redbooks.ibm.com/abstracts/tips0716.html?Open
5.
Brocade 10Gb CNA for IBM System x

http://www.redbooks.ibm.com/abstracts/tips0718.html?Open
35
Trademarks
IBM, the IBM Logo, BladeCenter, DS4000, eServer, and System x are trademarks of International
Business Machines Corporation in the United States, other countries, or both.
For a complete list of IBM Trademarks, see http://www.ibm.com/legal/copytrade.shtml.
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other
countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
36

Deploy Fibre Channel Over Ethernet (FCoE) Solution With IBM® BladeCenter® and System X® Using The IBM Converged Switch B32 (Brocade 8000)

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Deploy Fibre Channel Over Ethernet (FCoE) Solution With IBM® BladeCenter® and System X® Using The IBM Converged Switch B32 (Brocade 8000)

Caricato da

Copyright:

Formati disponibili

Deploy Fibre Channel over

Ethernet (FCoE) Solution with

Copyright International Business Machines Corporation 2009

Copyright International Business Machines Corporation 2009

Copyright International Business Machines Corporation 2009

IBM BladeCenter H Chassis

IBM HS21XM Blade Server

42C1830 - QLogic 2-port 10Gb CNA (CFFh) for IBM BladeCenter

BladeCenter 10Gb Ethernet Pass-Thru Module

IBM DS4700 Storage Subsystem

Brocade 8000 FCoE Switch

Copyright International Business Machines Corporation 2009

Figure 1: FCoE Physical Topology

How does Fibre Channel Over Ethernet Work?

Copyright International Business Machines Corporation 2009

highly reliable, high-performance fabrics for shared storage, and these

FCoE features include:

FIPsolicited and unsolicited FCF advertisements (per June 10, 2008,

Copyright International Business Machines Corporation 2009

Figure 2: FCoE Logical Topology

The FCoE VF_ports provide FC services to FCoE initiators and targets, as

Copyright International Business Machines Corporation 2009

Brocade's implementation of FCoE for the Brocade 8000 CEE switch

Figure 3: Host to Storage logical configuration

At a high-level, the following configuration tasks needs to be completed to

Copyright International Business Machines Corporation 2009

The FC device driver logs into the fabric as a VN_port.

The FC target logs into the fabric as an N port device

The FCoE host and F Target WWPNs are zoned

The FC LUN(s) are mapped to the host

The host sees the LUNs from the OS if booting locally

IBM BladeCenter and System x FCoE Product Details

10Gb Pass-Thru Module Features

14 internal and 14 external 10Gb Copper or Optical ports

10Gb End-to-end unblocked access with no packet drop

Part of first FCoE Convergence solution offered on BladeCenter

BC-H & BC-HT Chassis ONLY

14 Internal 14 External Ports

Copyright International Business Machines Corporation 2009

QLogic 2-port 10 Gb Converged Network Adapter (CFFh) for IBM

Figure 4: Qlogic 2 port 10 Gb CNA

Combo Form Factor (CFFh) PCI Express x8 2.0 adapter

Communication module: QLogic ISP8112

Two ports with XAUI interfaces for PCI-E x8 to the HSSM

Support for up to two CEE HSSMs in BC-H or BC-HT chassis

Support for 10Gb Converged Enhanced Ethernet (CEE)

Support for Fiber Channel over Converged Enhanced Ethernet (FCoCEE)

Full hardware offload for Fibre Channel over Converged Enhanced

Support for IPv4 and IPv6

Support for Wake on LAN

QLogic 2-port 10Gb Converged Network Adapter for IBM System x

PCI Express x8 2.0 Generation 2 compliance

Copyright International Business Machines Corporation 2009

Support for 10Gb Converged Enhanced Ethernet (CEE)

Support for Fibre Channel over Converged Enhanced Ethernet (FCoCEE)

Full hardware offload for FC protocol processing

Support for IPv4 and IPv6

Brocade 2-port 10 Gb Converged Network Adapter for IBM System x

Compliance with PCI Express x8 2.0 Generation 2 servers

Support for 10Gb Converged Enhanced Ethernet (CEE)

Support for Fibre Channel over Converged Enhanced Ethernet (FCoCEE)

Full hardware offload for FCoE protocol processing

Support for IPv4 and IPv6