1

Brocade®
Product Training
Fibre Channel Layers and
Related Components

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003
Course Objectives 2

After completing this module, attendees should be able to:

‰ Identify FC protocol layers

‰ Link components to FC layers

‰ Link functions to FC layers

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003
Topics 3

‰ Ethernet and FC layer relationships

‰ FC protocol layers
¾ FC 0 – Speeds and Feeds

¾ FC 1 – 8b/10b Encoding

¾ FC 2 – Data Delivery

¾ FC 3 – Common Services

¾ FC 4 – Upper Level Protocols (ULPs)

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003
 4
Ethernet and FC Layer Comparison

TCP/
IP

Ether
net

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

A look at network protocol layers

Ethernet is the most widely used LAN technology today. Originally conceived and
developed by Xerox Corporation, it is specified in the IEEE 802.3 standard. Since its
inception, initial speeds of 10 Mbit/sec have evolved into Fast Ethernet (100BASET),
providing transmission speeds up to 100 Mbit/sec. Gigabit Ethernet provides an even
higher level of backbone support at 1000 Mbit/sec primarily on fiber optic cable. In the
diagram, we illustrate how Ethernet utilizes the OSI model. Although it looks like
Ethernet provides solutions for layers one through four of the OSI model, in fact,
Ethernet by itself only covers the two bottom layers. Ethernet is, in general, accessed
through TCP/IP (or UDP/IP) protocol stack, where the SAN is accessed through a simple
SCSI protocol stack with fewer overheads on the server processor.
Most Ethernet adapters work at the packet level. All higher-level segmentation and
reassembly into IP datagrams, or TCP-level sockets are software driven and require
server CPU intervention. Be aware that TCP/IP often drops packets of data when the
network becomes congested. When this happens, the packet must then be retransmitted
using more bandwidth.
Footnote 1: FC-2 has elements of OSI layer four, so it is not a perfect one-to-one
correlation between the OSI and the Fibre Channel layers. Overall, it is a modular
architecture, 5 layers (FC-0 to FC-4) and each level does not define physical or
programming interfaces between the levels.
Multiple Protocols 5
On Common Fibre Channel Transport

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

FC - 0 and 1 layers specify physical and data link functions needed to physically send
from one port to another.

FC – 0 specifications include information about media connections and cables, it is

sometimes referred to at the feeds and speeds layer. 10 Mbaud speed, although not
depicted, is also in development.

FC – 1 layer contains specifications for 8b/10b encoding, ordered set and link control
communication functions.

FC – 2 specify content and structure of information along with how to control and
manage information delivery. This layer contains basic rules needed for sending data
across network. This includes: (1) how to divide the data into smaller frames, (2) how
much data should be sent at one time before sending more (flow control), and (3) where
the frame should go. It also includes Classes of Services, which define different
implementations that can be selected depending on the application.

FC – 3 defines advanced features such as striping (to transmit one data unit across
multiple links) and multicast (to transmit a single transmission to multiple destinations)
and hunt group (mapping multiple ports to a single node). So, while FC-2 level concerns
itself with the definition of functions with a single port. The FC-3 level deals with
functions that span multiple ports. There are parallel link service functions that many
think reside at this layer because they logically fit. These link service functions utilize all
FC-2 and lower services like an ULP but are not ULPs neither but neither do they reside
at FC-3. I think of them as performing FC-4 like functions in a category parallel to FC-3.

FC – 4 provides mapping of Fibre Channel capabilities to pre-existing protocols, such as

IP or SCSI, or ATM, etc.
 FC-0 Layer: Media Connectors and Cables 6

‰ Serial interface (separate

one-bit transmit and receive
lines)
‰ Media types:
¾ Cables
ƒ Fiber Optic Cables
• Single Mode
• Multi Mode
¾ Connectors

‰ Fiber and Electrical Cable

specifications

FC-0 and FC-1 combined are

referred to as FC signaling
interface

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

FC-0: Physical Interface

The lowest level, FC-0, specifies the physical link. One big purpose of Fibre Channel is to
have the selected protocol operate over various physical media and data rates. This
approach ensures maximum flexibility, allowing existing cable plants and a number of
different technologies to be used to meet a wide variety of system requirements.
Common cables are either copper or optics.

Single Mode optical fiber is used between buildings or sites because of its long distance
data transmission capabilities. Multi Mode Fiber (MMF) is used within a building between
nodes and switches or between floors.

Optical connectors are used for interconnection between devices such as nodes, fibre
channel switches and hubs. These connectors include SFP (Small Form Pluggable), GBIC
(Gigabit Interface Connector) and MIA (Media Interface Adapter).

Serial data received from FC-1 is converted to a signal type associated with the
transmission media and sent out transmission port. Transmission continues as long as
link remains in OFC Active state also called Open-Fiber State.
 FC-0 Fiber Optic Cables 7

Glass Core Glass Core

Glass Cladding Glass Cladding
Coating Coating

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

Fibre Channel links are driven optically or electronically. The optical and electrical links
can be combined in a single system when there is a Fabric or other media-type
converter available. The active part of the optical cable is constructed out of the core
(the optical conduit), surrounded by cladding (to keep the light in the core), and fiber
coating wrapped around the cladding. The optical fiber is very thin and could easily
become damaged. The fiber core can be either 9 micron, 50 micron or 62.5 micron. For
comparison, a human hair is about 75 micron thick.

A
ferrule

Optical FC links consists of two fibers; one for transmitting information, the other for
fiber for receiving information. Each side TX connects to the other sides RX consisting of
a single point-to-point connection. Using a switched fabric a nodes TX is attached to the
Brocade ports RX and the Brocade TX is connected to the nodes RX.
 8
FC-0 Optical Cable Links Supported
50/62.5 micron Multi Mode Fiber

780 nm Shortwave Lasers

850 nm Shortwave Lasers

Cladding (125 µm) Core (50/62.5 µm)

9 micron Single Mode Fiber

1300 nm Longwave Laser Cladding (125 µm) Core (9 µm)

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

Fiber types are available in two types: single mode or multimode. The smaller the
glass core the greater the distance. Multimode offers a core of 50 or 62.5 micron in
shorter distances and have a smaller cost base. Single-mode is used for high speed,
long distance links, while Multimode is utilized in lower cost, intermediate links.
(MMF) is used for short wavelength. It comes in 50 & 62.5 micron cores, with a
cladding of 125 microns. The core is much larger (50/62.5 um) allowing for multiple
modes and paths the light can follow. This propagation method is referred as Modal
Dispersion. The distance under this propagation method is significantly reduced.
Single-mode fiber (SMF) is used for long wavelength. It comes with a 9 micron core,
and a cladding of 125 microns. 9 micron goes 2m – 10km distances. The light travels
along the same path since the diameter of the core is reduced (9 um) to such a
degree it constrains the light.
In Single-mode fiber, the idea is to reduce the core size until the possibility of modal dispersion
is reduced to such a degree that only one mode of photon flight is exhibited. By eliminating the
chromatic dispersion (by utilizing monochromatic laser light sources), great distance and
bandwidths become possible in single-mode fiber. Multi-mode propagation refers to the fact that
the pulses of light can travel down the optical wave guide taking different reflective paths.

Following are the prefix and measurement for the small numbers:
• Deci-meter (dm) = 10-1 of a meter
• Centi-meter (cm) = 10-2 of a meter
• Milli-meter (mm) = 10-3 (One thousandth) of a meter
• Micro-meter (µm) = 10-6 (One millionth) of a meter (µ - Greek word)
• Nano-meter (nm) = 10-9 (One billionth) of a meter
• Pico-meter (pm) = 10-12 (One trillionth) of a meter
• Femto-meter (fm) = 10-15 of a meter
• Atto-meter (am) = 10-18 of a meter
 FC-0 Cables and Connector Distances 9

‰ Fibre Channel does not necessarily mean just fiber cable

‰ Optical and Copper cables are the most common

Media Type Speed Distance (m=meter)

9 micron Single-Mode Fibre 100 MB/s Æ 10 km

(Long Distance) 200 MB/s

50 micron Multi-Mode Fibre 100 MB/s .5 m – 500 m

(Short Distance) 200 MB/s .5 m – 300 m
62.5 micron Multi-Mode Fibre 100 MB/s .5 m – 175 m
(Short Distance) 200 MB/s .5 m – 90 m
Electrical (Copper) 100 MB/s 0 m – 33 m

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

Fibre Channel can be implemented using either fiber optic or copper cabling. Each has its own
advantages and disadvantages. Fiber optic cables are more expensive but they give reliability, distance
(up to 10 kilometers), and ease connectivity. They come in two kinds: single-mode (yielding greater
distance) and multi-mode. Multi-mode is by far the most common today, typically providing 500
meters distance at 1Gbit/sec. Each optical patch in a fiber is called a mode. Fiber pairs come with easy-
to-use push-pull SC (Sieman Connector) or LC (Lucent Connector) connectors at the cable ends.
2Gbit/sec switches typically use the LC connectors. The 1Gbit/sec switches typically use an
SC connector. The 2Gbit/sec switches use an LC connector. Both cables have white tips,
called ferrules, that stick into the SFP (GBIC).

Copper cables are less expensive but suffer from more reliability problems and limited distance. They
are available as Coaxial cable and Twisted Pair with connectors in DB-9 or High Speed Serial Data
Connectors (HSSDCs). The HSSDC is a new connector designed for Fibre Channel, providing for a
screw-less, easy to plug/unplug connection.
FC-0 Fiber Optical Cable Connections 10
The two most common optical connections used in FC involve
LC Æ SFP’s and SC Æ GBIC pairs
LC (Lucent Connectors)
plug into SFPs (Small
Form Factor Pluggable)
receptors
SFP
LC

SC (Siemens Connectors)
plug into GBICs (Gigabit
Interface Connector)
receptors

SC GBIC
Typically - 2Gbit/sec cables use LC connectors to SFPs
while 1Gbit/sec fiber cables use SC connectors to GBICs
© 2003 Brocade Communications Systems,
Incorporated.
Revision0.1_FC101_2003

The SC connector features a molded body and a push-pull locking system and was
designed as a low cost alternative to the ST connector. It is utilized in both Multimode
and Single mode hi-bandwidth applications.

The LC connector, a small-form factor connector, features a ceramic ferrule and looks
like a mini SC connector.

SWL Fiber Optic GBIC/SFP Modules

The SWL fiber optic GBIC/SFP modules, with an SC or LC connector
(respectively), are based on short-wavelength 850 nm lasers supporting 1 or 2
Gbit/sec link speeds. This GBIC/SFP modules support 50/62.5 micron, multi-
mode fiber optic cables 50/125 microns will go up to 500 meters in length @ 1
Gbit/sec and 300 meters in length @ 2 Gbit/sec. Fiber optic cables 62.5/125
microns will reach 175m @ 1 Gbit/sec and 90m @ 2 Gbit/sec.
LWL Fiber Optic GBIC/ SFP Modules
The LWL fiber optic GBIC/SFP modules, with SC or LC connector are based on
long-wavelength 1300nm lasers supporting 1 or 2 Gbit/sec link speeds. This
GBIC module supports 9 micron single-mode fiber optic cables up to 10
kilometers in length with a maximum of five splices.
FC-0 GBICs and SPF’s 11

Copper GBIC Fiber SFP Fiber GBIC

Cable connection: LC Connector SC Connector

‰ Passive Copper (Cu) - Up to 13M (1Gbit/sec)

‰ Active Copper (Cu) - Up to 33M (1Gbit/sec)

‰ Short wavelength (SWL) GBIC / SFP

‰ Long wavelength (LWL) GBIC / SFP

‰ Extended Fabrics - Up to 100K

© 2003 Brocade Communications Systems,
Incorporated.
Revision0.1_FC101_2003

Check OEM Compatibility matrices for a list of supported GBICs and SFPs. Brocade
supported GBIC’s / current vendors include: Agilent, Finisar and IBM.

Passive/Active Copper GBIC Module

The Copper (Cu) GBIC module is based on the High-Speed Serial Data
Connection (HSSDC) interface standards. The GBIC provides a female HSSDC
connector. Passive Copper cables up to 13 meters and active up to 33 meters
have currently been qualified, thereby supporting ANSI X3.230 FC-PH intra-
cabinet requirements. Standard cables with HSSDC-to-DB9 male connectors
are also available.

SWL Fiber Optic GBIC/SFP Modules

The SWL fiber optic GBIC/SFP modules, with an SC or LC connector
(respectively), are based on short-wavelength 850 nm lasers supporting 1 or 2
Gbit/sec link speeds. This GBIC/SFP modules support 50/62.5 micron, multi-
mode fiber optic cables 50/125 microns will go up to 500 meters in length @ 1
Gbit/sec and 300 meters in length @ 2 Gbit/sec. Fiber optic cables 62.5/125
microns will reach 175m @ 1 Gbit/sec and 90m @ 2 Gbit/sec.
LWL Fiber Optic GBIC/ SFP Modules
The LWL fiber optic GBIC/SFP modules, with SC or LC connector are based on
long-wavelength 1300nm lasers supporting 1 or 2 Gbit/sec link speeds. This
GBIC module supports 9 micron single-mode fiber optic cables up to 10
kilometers in length with a maximum of five splices.
Cables and Related Components 12

‰ Types of Optical Cable(s)

MIA
¾ LC to LC – SFP to SFP
¾ LC to SC – SFP to GBIC
¾ SC to SC – GBIC to GBIC
‰ Ethernet Port: RJ45
‰ Serial Port: Straight 9-pin female-female D-sub cable
¾ Pins 2,3,5 are required

Media Interface Adapters (MIAs) are used to convert electrical signals

to optical signals (DB9 to SC)

SC LC
Coupler Coupler

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003
 FC-1 Layer: 13

‰ 8b/10B encode/
decode
‰ Link Control
‰ Ordered Sets

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

FC-1: Byte Encoding

This layer describes the 8-bit/10-bit transmission code that is used to provide balance of
the transmitted bit stream. In addition, the coding provides a mechanism for detection of
the transmission and reception errors. 8-bit/10-bit encoding scheme is selected for its
superior transmission characteristics. This well-balanced code allows for low-cost
component design and provides good transition density for easier clock recovery.

Note: 8-bit/10-bit scheme is used in IBM ESCON as well.

FC-1 level defines the method to encode data prior to transmission. As data is passed
down in octet form, FC1 layer encodes it into a ten bit code group (20% overhead). Each
octet is given a code group name according to the bit arrangement. This will guarantee
certain characteristics about the information that gets sent across the serial link.

Syncing up summary --Æ Serial data comes into a receiver port with 7 bit
comma patterns at the front of each ordered set character. These ordered set
characters act as frame delimiters, primitive signals and primitive sequences.
Receiver ports first bit sync (using these commas) and then count to 10 thus
character syncing. They also count to 40 to word sync. Now frames can flow.
If a loss of sync occurs a Loss-of-Sync procedure takes place (5 invalid
transmission words Æ receiver attempts to re-sync).
FC-1 Ordered Sets 14

Fibre Channel uses transmission word-ordered sets (four transmission

characters) to perform control and signaling functions:

Transmission Word

Ordered Set Data Word

(K28.5,Dxx.y,Dxx.y,Dxx.y) (Dxx.y,Dxx.y,Dxx.y,Dxx.y)

Frame Delimiter Primitive Signal Primitive Sequence

SOF- Start of Frame NOS - Not Operational
Fill Word Non-Fill Word
EOF- End of Frame OLS - Offline
IDLE R_RDY
LR - Link Reset
ARB VC_RDY
LRR - Link Reset Resp.
CLS
LIP - Loop Initialization
OPN
LPB - Loop Port Bypass
LPE - Loop Port Enable

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

Transmission words can be classified into two categories:

1. Words that begin with a special character are called ordered sets. Any
character that starts with a “K” (ex: K28.5) is called a “special character” to
indicate its usage as an ordered set. Ordered sets occur outside of the FC
frame content and include frame delimiters, primitive sequences and signals.
2. Words that begin with an encoded data character are called data words. Data
Words occur within the FC Frame content.
Each transmission character in both the Ordered Sets and Data Words is 10 bits,
so transmission words are 40-bit words.

Ordered Sets (K28.5, Dxx.y, Dxx.y, Dxx.y) include:

Frame Delimiters (SOF, EOF) – to delimit, or identify start and end of frames

Primitive Signals– to indicate events at the sending port. Primitive Signals are
used to indicate events or actions and are normally transmitted once.
Fill Word (Idle, ARB(X), ARB(F0), ARB(FF)– transmitted on a link whenever a
port is operational and has no other specific information to send
Non-Fill Word (R_RDY, VC_RDY, CLS, OPN, DHD, MRK, SYNx,y,z)- signal the
events
Undefined but valid ordered sets are treated as fill words (no explicit action in
necessary.

Primitive Sequences (NOS, OLS, LR, LRR, LIP) and (LPB, LPE) – to control an
optional port bypass circuit (for loop). Primitive Sequences are used to
indicate states or conditions and are normally transmitted continuously until
something causes the current state to change. At least 3 primitive sequences
must be received before the appropriate response can be generated.
FC-1 - Link Control Protocols 15

Link control protocols use ordered sets for low-level link initialization
and control Æ link or arbitrated loop (AL) initialization, reset or failure:
¾ Port State Machine (PSM) provides control signal specifications
for Fabric connections
¾ Loop Port State Machine (LPSM) provides control signal
specifications for arbitrated loop connections

Rx Rx
Nx_Port A Nx_Port B

Tx Tx

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

There are four states of the Port State Machine (PSM) used for Fabric (G_Port)
connections:
1. Active State (AC)
2. Link Reset State (LR)
3. Offline State (OLS)
4. Link Failure State (LF)
Active state (AC) – in the Active state, the port is able to transmit and receive
frames and primitive signals. Certain conditions encountered within the
port may cause the port to exit the active state and perform one of the
following primitive sequence protocols: Link Reset, Offline state, or Link
Failure.
Link Reset state consists of:
• LR1 (Link Reset Transmit)
• LR2 (Link Reset Receive)
• LR3 (Link Reset Response)
Offline state consists of :
• OL1 (Offline Transmit)
• OL2 (Offline Receive)
• OL3 (Wait for Offline)
Link Failure state consists of:
• LF1 (No Operational Receive)
• LF2 (No Operational Transmit)
 FC-2 Layer: Framing Protocol / Flow Control 16

‰ Exchange and
Sequence
Management
‰ Frame Structure
‰ Class of Service
‰ Flow Control
¾ Buffer to
Buffer
¾ End to End

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

FC-2 defines the structure and organization of the information being delivered
and how that delivery is controlled and managed.
Exchange management is the mechanism that two fibre channel ports use to
identify and assign an exchange ID number for a set of related information
units. When the entire stream of data will fit in a single frame (2112 bytes) a
single exchange id is created and a sequence number is assigned. However,
when a stream of data will not fit into a single frame (2112 bytes), data is put
into sequences of frames. Within the exchange ID sequence management is
used to number the sequence segments in the stream of data. Sequence
numbers associated with the exchange will be used at the recipient to re-order
the sequence segments, to re-assemble as a contiguous stream of data. In other
protocols, this is commonly known as fragmentation and re-assembly.
Frame Structure has a start-of-frame delimiter ordered set and ends with an
end-of-frame delimiter set.
Flow control is the process to deliver a frame. When a frame is ready for
transmission, it is sent thru the encoder (8b/10b), to the serializer (sfp/gbic) and
transmitted to the receiver port where it is deserialized, decoded and stored in a
receive buffer.
The receiving port sends to the transmitting port a credit to send another frame
and decrements a credit from the credit value established during the login
session (buffer to buffer credit). When the receiving port moves the buffer to
the next port, the debit is restored. Buffer credits regulate the flow of frames
into and out of the fabric.
When a N_port and a destination N_port communicate, an end-to-end credit is
established. End-to-end credit is established between the pairs and is used to
manage the flow of frames between a specific pair of N_ports and allows the
receiving port to control which source N_ports are allowed to send frames to the
receiver.
 FC – 2 Exchange, Sequence, Frame Management 17

Command 1 Command 2 Command 3

ULPs CMD DATA IN STATUS
(Sequence) (Sequence) (Sequence)

FC-4 Mapping Information Unit Information Unit Information Unit

Single Exchange CMD DATA IN STATUS

Multiple
CMD DATA IN STATUS
Sequences

Port
Frames 1 2 3 4

FC- 0 & 1 3 2 1

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

With Fibre Channel, there are almost no limits on the size of transfers between
applications. Whereas with Ethernet the software is sensitive to a maximum packet size
that can be transmitted (1518 bytes). With Fibre Channel, frame sizes are transparent
to the software because of a logical construct called a "sequence." A frame is not a unit
of transfer but sequence is and it always maps with the Upper Layer Protocol (ULP)
command instruction. Let’s see how it works:
First, all the commands coming from a ULP are mapped into logical constructs
called "information units." An Individual information unit is generally mapped to a
sequence. Related information units, such as those required in an I/O operation, are
mapped as a single exchange. The sequence and exchange structures are general
enough and contain tunable options concerning flow control and error recovery policy.
Sequence - A sequence is a set of one or more related data frames transmitted for a
single operation, flowing in the same direction on the link (unidirectional from one
N_Port to another N_Port). The N_Port that transmits a sequence is referred to as the
"sequence initiator," and the N_Port that receives the sequence is referred to as the
"sequence recipient". A sequence is also the recovery boundary in Fibre Channel. When
an error is detected, Fibre Channel identifies the sequence in error and allows that
sequence to be retransmitted.
Exchange - An exchange is composed of one or more non-concurrent, related sequences
for a single higher-level operation. For example, an operation may consist of several
phases: a command to read some data, followed by the data, then followed by the
completion status of the operation. Each phase of the command, data, and status is a
separate sequence, but they can form a single exchange. Within the single exchange,
only one sequence direction can be active at a time, although sequences for different
Exchanges may be concurrently active. (Fibre Channel multiplexing support). Sequence
initiative is handed over before sequences flow in the other direction. Sequences going
in one direction can be streamed. This just means that a 2nd sequence can be sent in
that direction before waiting for delivery confirmation.
Note: The four Exchange error policies are: (1) Abort, discard multiple sequences; (2)
Abort, discard a single sequence; (3) Process with infinite buffering (this policy is a
special design for specific transfers, such as video data); (4) Discard multiple
sequences with immediate retransmission.
 18
FC – 2 Fibre Channel Frame Format

FRAME
S C E
O HEADER PAYLOAD R O
F C F

4 24 Up to 2112 4 4
2148
Bytes

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

A frame has a header and may have a payload. The header contains control and addressing information
associated with the frame. The payload contains the information being transported by the frame on behalf of the
higher level service or FC-4 upper level protocol. There are many different payload formats, based on the
protocol. The TYPE field (Word 2, bit 31- 24) tells which format to use. The routing control INFO bit (bit 27-24)
determines how to interpret the payload.
Field Definitions
•Routing Control bits (R_CTL) are the first 8 bits of the header. They define the type of frame
and its content or function. The first 4 bits (Bit 31-28) identifies the frame type. The 2nd four bits
“INFO bit” (Bit 27-24) defines the contents of the frame or identify the function of the frame.
•Destination_ID (D_ID)- Port Identifier (PID) or 24 bit address of the recipient. It could also be a
a well-known address like the Name Server FFFFFC … more on this later.
•Class specific Control Field (CS_CTL) – The control necessary for different classes of service.
This field is always zero for classes 2 and 3 per the standards. Classes 1 and 4 use it. Brocade
switches currently only use Classes 2,3, and F. If CS_CTL is something other than zero in a
Brocade port log (a running log extracted from portions of the FC frame displayed with the
portLogDump command), then it is a Brocade internal code called “IU_Status Values”.
•Source_ID (S_ID) - Port identifier (PID) or 24 bit address of the source. It could be a a well-
known address like the Name Server FFFFFC.
•Type – identifies the protocol of the frame content for Data Frames (i.e FC_CT, FCP, IPFC)
•Frame Control (F_CTL) -This field contains miscellaneous control information regarding the frame
such as who owns initiative, first frame of the Exchange, last frame of the Exchange, etc.
•Sequence ID (SEQ_ID) –used to identify and track all of the frames within a sequence between a
source and destination port pair.
•Data Field Control (DF_CTL) – this field indicates if any optional headers are present at the
beginning of the data field of the frame. Optional headers are used for information that may be
required by some applications or protocol mappings.
•Sequence Count (SEQ_CNT) – used to indicate the sequential order of frame transmission within
a sequence or multiple consecutive sequences within the same exchange. This is a counter that
increments as sequences of frames are transmitted.
• Originator_ID (OX_ID) – Exchange ID assigned by the originator port
• Responder_ID (RX_ID) - Responder_ID, optionally assigned by the responder to the Exchange.
• Data Field/Payload – The standards limit the size. The maximum size is 2112 bytes.
FC-2 Class of Service (COS) 19
Associates a set of attributes with sequences and the frames within the sequence to
provide different delivery characteristics.

There are six classes of Fibre Channel service:

Class 1 – connection oriented: port-to-port, with ack (acknowledge),
guaranteed bandwidth, in-order delivery (IOD) A circuit-switching
COS One Æ One
Class 2* – connectionless: end-to-end(EE_Credit)/ ack and buffer to
buffer / receiver ready (BB_Credit/r_rdy)flow control
Class 3* – connectionless: no ack, BB_Credit/r_rdy flow control only -
Errors are handled at higher level
Class 4 – connection oriented with ack: Creates Virtual Circuits with
dedicated fractional bandwidths. Bandwidth and latency handled by
a Quality of Service parameter that is IOD guaranteed
Class 6 – behaves like Class 1 with one Æ many capabilities
Class F* – connectionless: switch-to-switch, with ack and BB_Credit ready

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

The ability to differentiate Classes of Service characteristics will be needed

when discussing Link and Flow control of frames

The Classes of service with an asterisk (*) are the ones that Brocade
switches currently support.

Note 1 - There are other link control signals associated with Classes 1, 2, 4
and 6 such as ack, f_bsy (fabric busy), p_bsy (port busy), f_rjt (frame reject),
lcr (link credit reset), nty (destination port engaged in a class-1 connection),
and other control signals.

Note 2 - Classes of service 2, 3 and F have no bandwidth, deterministic

latency, or IOD guarantees. Other Fabric services can provide this capability.

Additional Notes:
Class 1 – Circuit-switching means that when Fabric receives Start of Frame
connect Class One (SOFc1) signal it starts establishing a connecting to
destination port. There are variations of COS 1: Exclusive connection, Intermix
(allows class 2 and 3 frame transmission when class 1 not in use) , Intermix
Bandwidth Recovery (allows Fabric to hold class 1 frame one frame cycle and
make up that frame cycle in idle cycles).
Class F – Based on a simplified Class 2 model – It is used to communicate
Fabric-related traffic on inter-switch links (ISLs).
 FC-2 Common Classes of Service 20

‰ Class 2/F - Connectionless with ACK

‰ Class 3 - Connectionless without ACK

Bidirectional multiplexed frame

streams without dedicated connections
© 2003 Brocade Communications Systems,
Incorporated.
Revision0.1_FC101_2003

Class 2 Service
ƒMultiplexes frames at the frame boundary
ƒAdaptive routing
9Each frame routed separately by Fabric
9If multiple routes supported, frames might be delivered out of order
ƒConfirmation of delivered frames
ƒLogin with both N_Port and Fabric required
ƒConnectionless service with no turnaround delay to establish connection
ƒN_Port to N_Port flow control
ƒEnd-to-End (EE) delivery confirmation with ACK
ƒBuffer-to-Buffer (BB) Link level flow control
ƒNotification of frame delivery failure

Class 3 Service
ƒMultiplexes frames at the frame boundary
ƒAdaptive routing
9Each frame routed separately by Fabric
9If multiple routes supported, frames might be delivered out of order
ƒUnconfirmed delivery of frames
9Datagram service
ƒLogin with both N_Port and Fabric required
ƒConnectionless service with no turnaround delay to establish connection
ƒBuffer-to-Buffer (BB) Link level flow control
ƒULP recovery from frame delivery failures

Brocade supports Class 2 and 3.

Flow Control 21

‰ Flow Control mechanisms

¾ ACK Acknowledgement
¾ R_RDY Receiver_Ready

‰ Flow control is related to COS

‰ Brocade switches support COS 2, 3 and F
¾ COS 3 uses R_RDY (buffer to buffer or BB_Credit) flow control,
each R_RDY received increments BB_Credit value
¾ COS 2 and F use R_RDY and ACK (end to end or EE_Credit) flow
control, each ACK received increments EE_Credit value
‰ Brocade switches use cut through routing – read S_ID / D_ID pairs
and then shoot out the wire

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

Flow Control also includes mechanisms to communication “busyness”:

F_BSY Fabric Busy
F_RJT Fabric_Reject
P_BSY N_PORT_BUSY
P_RJT N_PORT_REJECT

Flow control definitions:

Acknowledgement (ACK) is used to confirm successful frame delivery and used by
class 2, F and not Class 3. ACK times are established with the RA_TOV value. The
default time used by Brocade is 10 seconds.
Receiver Ready (R_RDY) is used by the receiving port to signal it is ready to receive a
new frame.
Fabric Busy (F_BSY) occurs when the fabric is unable to deliver a frame. Fabric busy
conditions are an unusual event. The time for frame delivery is determined by the Error
Error Detection Time Out Value (E_D_TOV). The default time used by Brocade is 2
seconds
Fabric_Reject (F_RJT) Used to signal that a frame could not be delivered. Typical
reason is for an invalid S_ID
N_Port_Busy (P_BSY) When a N_PORT is unable to accept a valid frame due to a busy
condition, the receiving N_PORT will send a P_BSY to the sender of the frame.
N_PORT_REJECT (P_RJT) Used to signal that a frame could not be delivered. Typical
reason is for an ACK
FC-2 Buffer-to-Buffer (BB) Flow Control 22

‰ Frames are moved from one buffer to another using

Receiver Ready (R_Rdy) primitive signals
‰ Frame flow is always from the source buffer to the
destination buffer
‰ Multiple intermediate buffers may be involved

Tx data Rx Tx data Rx

N_Port A N_Port B
Fabric
Rx Tx Rx Tx

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003
FC-2 BB and End to End (EE) Flow Control 23

‰ Frame flow is controlled by the receiver as a back-pressure

mechanism
‰ Flow control is dependent on class of service (COS), but
most use BB_Credits
¾ BB_Credit and length established during Fabric login
¾ BB_Credits are also exchanged during port login but
ignored unless in a point-to- point topology (two devices,
no switches)
‰ End to end (EE_Credit) flow control is a originator device
port to destination device port flow control mechanism
¾ EE_Credits are establish during node port login
¾ ACKs increments EE_Credit value

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003
FC-2 Flow Control Cont. 24

‰ Flow control is a mechanism to establish the

maximum amount of data that can be sent at any
one time
‰ A throttle-back mechanism to handle congestion
without the need to discard frames
‰ Flow control considerations include:
¾ Bandwidth
¾ Internal Host Buses
¾ Extended Distance

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

Bandwidth Considerations - At high speeds more data buffers are required to

achieve the same link utilization for a given distance.
Internal Host Buses - Internal host Buses (like PCI) can become congested and thus
throttle back receiving data from the Fabric
Extended Distance - Over extended distances, more data buffers are required to
maintain a high throughput
FC-2 Flow Control Cont. 25

Increasing E_Port BB_Credits (points 3 and 4) allows full

bandwidth utilization of extended Fabrics at specified distance

Switch 1 Switch 2

ISL

Points
Point 2 3&4
(F_Port) (E_Ports)

‰ BB_credits are exchanged between points 1 and 2

during Fabric login and are used as throttle-back
mechanisms
Point 1
(N_Port) ‰ BB_credits are also exchanged across E_Ports (points 3
and 4) and are used as both throttle-back and
bandwidth utilization (performance) mechanisms
© 2003 Brocade Communications Systems,
Incorporated.
Revision0.1_FC101_2003

Increase E_Port buffer credits to optimize performance at required distance

The latency of a laser beam going through a fiber is 5nsec/meter (light wave
propagation) or 5μsec/km
10km link round trip latency is 100μsec

A 2KB frame propagation delay at 100MB/sec is 20μsec. So to keep a

10km “pipe” full, the switch would need at least 5 E_Port buffers
(“credits”)
Proportionally, a 2KB frame at 50km needs 25 buffer credits
At high speeds more data buffers are required to achieve the same link utilization for a
given distance
A 2KB frame propagation delay at 200MB/sec is 40μsec and would
need at least 10 E_Port buffers (“credits”)
Proportionally, a 2KB frame at 50km needs 50 buffer credits
FC-2 Flow Control Summary 26

FC flow control mechanisms associated

with the Classes of Service (COS) that
Brocade switches support:

Feature Class 2/ F Class 3

Flow control based on BB and EE (AWK) credit BB credit
Busy/ reject response Yes, by Fabric or destination No, frame discarded
Seq_id assignment 0 to 255 0 to 255
Source sequence complete Final ACK received w/ EOFt Acc to ABTS; or ULP
Destination sequence All data frames received and All data frames received and
complete all ACKs sent EOFt in last frame

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

Note – Due to limited space and non-use of COS 1 and 4 in Brocade SANs at
this time, they will not be discussed in relation to flow control
 FC-3 Layer Common Services 27

‰ Largely unused:
mostly theoretic
‰ Functions span
multiple ports
‰ Common service
advanced features:
¾ Striping

¾ Hunt groups

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

FC-3: Common Services

This layer defines advanced features such as striping (to transmit one data unit across
multiple links) and multicast (to transmit a single transmission to multiple destinations)
and hunt group (mapping multiple ports to a single node). So, while FC-2 level concerns
itself with the definition of functions with a single port. The FC-3 level deals with
functions that span multiple ports.

Fibre Channel provides internal protocols called Basic Link Services and Extended Link
Services. These services would logically reside at FC-3 but are NOT part of the FC-3
standards. Basic link services include low-level functions that are transported as a single
data frame within a sequence. Basic link service examples include: Abort sequence
(ABTS) used to abort a sequence or a frame, BA_ACC or BA_RJT are both ABTS
responses and respectively mean Basic link service accept or reject. Another basic link
frame is called No operation (NOP) which is used to initiate or terminate connections and
sequences. There is also a remove connection (RMC) class of service (COS) 1 frame that
is used to request a COS 1 connections removal. Extended link services (ELS) perform
upper level protocol like functions between two fabric ports, one of them is often a well-
known address like the fabric F_Port, fabric controller or name/directory server.
Common ELS commands include FLOGI (Fabric login), PLOGI (port login), SCR (State
Change Registration), RSCN (Registered State Change Notification), PRLI (Process login)
and more.
 FC-4 Upper Level Protocol (ULP) Mapping 28

‰ SCSI
‰ IP
‰ WAN tunneling
‰ FICON

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

¾SCSI (SCSI-FCP or FCSI-201) - Transport of SCSI commands and data over the Fibre
Channel protocol hierarchy is such a major part of the Fibre Channel usage that a
particular acronym, “FCP”, is used to denote the SCSI over Fibre Channel, it is like the
native protocol for Fibre Channel, so we called it Fibre Channel Protocol.

¾IP(IETF Draft FC_IP or FCSI 202)- Maps the IP protocol over Fibre Channel

¾WAN Tunneling (FC-BB)- defines mapping for ATM or LAN over FC

¾Audio Visual (FC-AV)- Mapping for digital TV standard and MPEG (For example) over FC

¾VIA (FC-VI)- Virtual Interface Architecture mapping, (i.e. clustering)

¾FICON (FC-SB-2) - Mapping FICON over FC. The standard is called FC-SB-2 for Fibre
Channel Single Byte Command 2.
 Fibre Channel Protocol Mappings 29

Upper Level Protocols

(ULPs)
• Support existing protocols
• Transparent to OS
SCSI-3 IP ATM
• No changes + New capabilities

IPI - 3 Command FC Link

FC-4 SCSI - 3 Command
Set Mapping Encapsulation FC - ATM
Set Mapping
(IPI-3 STD) FC - LE

FC-3 Common Services

FC-2 Framing Protocol

FC - AL FC - AL -2
Fibre Channel Physical
FC-1 Encode / Decode & Signaling Interface
8b/10b Encoding
( FC- PH, FC-PH2,
FC-0 Physical Variant FC-PH3 ) Copper, Optical

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003

A large part of the work done in developing the Fibre Channel architecture has been concentrated in
assuring that the architecture could efficiently and naturally operate as a transport or delivery
mechanism for a wide variety of well-established Upper Layer Protocols, or ULPs. Since much of the
investment in current Operating Systems is at the device driver level, the incremental cost in
transferring systems over to a Fibre Channel data communication level decreases if the interfaces can
be made as similar to previously-existing interfaces as possible. This allows new capabilities to be
added with minimal changes to currently-available interface.

The following protocols are currently support:

Storage protocols
- Small Computer System Interface (SCSI) command set
- Single Byte command code set (SBCCS)
- High Performance Parallel Interface (HIPPI)
- Intelligent Peripheral Interface (IPI)
Network protocols
- IP (Internet Protocol)
- ATM Adaptation Layer for Computer Data (AAL5)
- IEEE 802.2
- Link Encapsulation (FC-LW)
Cluster Server protocols
- VI (Virtual Interface)

FC-4, the ULP mappings, some of which are listed, isolate the ULPs from the underlying Fibre Channel
fabric and define interoperability of processes within a ULP. This layer maps the ULP onto the Fibre
Channel transport layers. Fibre Channel is equally adept at transporting both network and channel
information and allows both protocol types to be concurrently transported over the same physical
interface.
Summary 30

‰ FC - 0 and 1 layers specify physical and data link functions needed to

physically send from one port to another
¾ FC-0 defines media connections and cables (speeds and feeds)
¾ FC-1 defines 8b/10b encoding, link control and ordered sets
‰ FC – 2 specify content and structure of information along with how to
control and manage information delivery
¾ FC-2 defines exchange and sequence management; frame structure, COS
and flow control mechanisms

‰ While FC-2 level concerns itself with the definition of functions with a
single port, FC-3 level deals with functions that span multiple ports

‰ FC – 4 provides mapping of Fibre Channel capabilities to pre-existing

protocols, such as IP or SCSI, or ATM, etc.

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003
Additional Information 31

From the resources tab –

¾ UseInternet Links to access SAN ED 101 “SAN Fabric

Foundation” chapter 2 (Fibre Channel Essentials)

Also use the “Fibre Channel Theory Fundamentals

Resource and Reference Material” selection under course
syllabus “SAN Glossary” and “FC Recommended Reading
and Resource List”

© 2003 Brocade Communications Systems,

Incorporated.
Revision0.1_FC101_2003