Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Abstract
A new concept is currently moving through the
Fibre Channel (T11) standards committee called
Fire Channel over Ethernet (FCoE).
The FCoE standard will specify the
encapsulation of Fibre Channel frames into
Ethernet Frames and the amalgamation of these
technologies into a network fabric that can
support Fibre Channel protocols and other
protocols such as TCP/IP, UDP/IP etc.
The tutorial will show the Fundamentals of the
FCoE concept and describe how it might be
exploited in a Data Center environment.
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
Agenda
Introduction
Goals & Requirements
Consolidation
Architecture
Discovery & Link Instantiation
Topologies
Scenarios
Summary
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
Introduction
This presentation provides an overview of Fibre
Channel over Ethernet (FCoE)
One should think about FCoE as placing the FC
protocol on a new physical link
Ethernet links instead of physical FC links
But it is still Fibre Channel
Increasing Performance
10Gbps
Integration of Carrier-grade features
1Gbps
Incremental Protocol Enhancements
100Mbps
Logical Partitioning
10Mbps
Evolution from shared media to dedicated media
Introduction of Ethernet
1973
Goals/Requirements
FCoE Goals/Requirements
(part 1)
FC
FC
FC
Note:
FCF Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocol
Non-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as
STP (Spanning Tree Protocol) and MSTP (Multiple STP)
See additional FSPF and STP explanation in Appendix
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
Applications
UDP
SCSI
SCSI
iSCSI
Fibre Channel
TCP
IP
FCoE
(FCoE VN_Port)
10
FCoE Goals/Requirements
(part 2)
11
Consolidation
12
With CEE
OS3
DB Server
OS2
App Server
OS2
App Server
FC
HBA
OS1
Web Server
Messaging
Ethernet
TCP/IP
FC SAN
E-HBA
(CEE)
MPI
RDMA
Today
IB/Ethernet
Cluster
OS2
App Server
OS3
DB Server
E-HBA
(CEE)
Hyper Visor
(vmWare, Xen, etc)
OS1
Web Server
NIC
TCP acceleration.
MPI, RDMA
over Ethernet
FCoE
Data Center
CEE Network
OS3
DB Server
E-HBA
(CEE)
Data Center
CEE Network
13
Architecture
14
FC-4
FC-3
FC-3
FC-2
FC-2
FC-1
FCoE Mapping
MAC
PHY
FC-0
FC Levels
(Unchanged)
IEEE 802.3
Layers
Ethertype
FCoE
FCoE
Header
FC Header
SCSI Commands/Data
FCS
Ethertype
FIP
FIP
Header
Descriptors
FCS
15
Login Phase
ENodes chose among discovered FCFs Ports
Creates association between ENode Ports and FCF Ports
VN_Port VF_Port Logical FC Link
FCoE
Initialization
Protocol (FIP)
FCoE Protocol
16
31-24
23-16
15-8
7-0
1
Optional IEEE 802.1q
4 Byte
Tag goes here
See Appendix for
Descriptor list items
Descriptor list
varies
In size
2
3
4
Reserved
Descriptor List
Reserved
FIPLength
subcode
Flags
(12 bits)
FIP SubCode
FP SP
Flags List Length
Descriptor
S F
F S S F
P P
Descriptor List
Ethernet frame
n+1
size
Is 64Bytes to 2220Bytes
Ethernet FCS
Capability Bits (SPMA or FPMA)
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
Solicited bit
FCF bit
17
31-24
23-16
15-8
7-0
0
1
Optional IEEE 802.1q
4 Byte
Tag goes here
2
3
Reserved
Reserved
In size
n
n+1
Ethernet frame
n+2
size
Is 64Bytes to 2220Bytes
Reserved
Reserved
6
This field varies
Ver (4b)
(12 bits)
SOF (8 bits)
Encapsulated FC Frame
FC Frame = Minimum 28 Bytes (7 Words)
Maximum 2180 Bytes (545 Words)
(including FC-CRC)
EOF (8 bits)
Reserved
Ethernet FCS
18
FCF Model
Link
End
Point
(LEP)
19
Link
End
Point
(LEP)
20
FC-3 /FC-4s
FC-3 /FC-4s
VN_Port
FC
Entity
VN_Port
FC
Entity
FCoE_LEP
FCoE
Entity
FCoE_LEP
FCoE
Entity
...
FCoE
Controller
Ethernet_Port
MAC Address
of Burnt-in
MAC
In this
model this
is where
FC-2
functions
live
In this model this
is where the
Encapsulation
/De-Encapsulation
functions live
MAC Address of
FCoE_LEP
(VN_Port)
May or may not be
the same as the
FCoE controller
Only one MAC Address is required for the FCoE Controller and the VN_Ports on a
single physical MAC (aka Server Provided MAC Address SPMA)
FCF may chose to specify new MAC addresses for each VN_Port (aka Fabric
Provided MAC Address FPMA)
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
21
Examples of
single MACs
with
connections
to two
different
FCFs
Switch
22
B C
Host PCIe interface
FCoE
ASIC
External
Port
FCoE
function
NIC Function
Lossless
Ethernet
MAC
FC
Function
23
The FCoE controller will perform the FIP functions and will instantiate new
VN_Ports as FCoE Link End Point (LEP)
With the same MAC address as the FCoE Controller (SPMA)
Or with a new MAC address specified by the FCF (FPMA)
A
FCoE Chip
MAC
Address of
Burnt-in
MAC
FCoE Function
FC-3 /FC-4s
FC
Entity
FCoE
Controller
VN_Port
FCoE
Entity
FC
Entity
FC-3 /FC-4s
FCoE_LEP
Etype=
FCoE
or
FIP?
VN_Port
FCoE
FCoE_LEP
Entity
NIC
Ethernet_Port
Lossless Ethernet MAC
MAC
Address of
Burnt-in
MAC
Used to separate HW
based FCoE from other
Ethernet Traffic
Most NICs come with
several Burnt-in MAC
Addresses
24
25
FCoE_LEPs
FCoE_LEP
MAC(H1)
FCF-MAC(A)
FCF A
VF_Port
H1
FCoE_LEPs
FC
fabric
VN_Port
FCF-MAC(B)
FCoE_LEP
VN_Port
FCoE_LEP
MAC(H2)
Lossless
Ethernet
Switch
VF_Port
FCoE_LEPs
FCF B
VF_Port
H2
FCoE_LEPs
VN_Port
VF_Port,
VF_Port_Name
FCoE_LEP
VN_Ports,
VN_Port_Names
Classical
Ethernet
LAN
26
End-Node
Discovery
Multicas
to ALL
FCoE Initialization
Protocol
FCFs
Discovery
Phase
FLOGI
FLOGI ACC with the FCFs
chosen VN_Port MAC address
as a descriptor value
FC Command
(Using the FCF
selected MAC
Address as the SA)
Login Phase
Normal FC
Processing
FC Command responses
(using the commands SA as a DA)
FCOE Protocol
27
FCF
FCoE Initialization
Protocol
FLOGI
FLOGI ACC with the FCFs
chosen VN_Port MAC address
as a descriptor value
Login Phase
FC Command
(Using the FCF
selected MAC
Address as the SA)
Normal FC
Processing
FC Command responses
(using the commands SA as a DA)
See Appendix for more details in Discovery Consideration and Actions
FCOE Protocol
28
Topologies
29
FCoE Switch
FC
FC
Ethernet
FC
30
FCoE Switch
Lossless
Ethernet
Fabric
FC
FC
Lossless Ethernet
Fabric
FC
FC
FCoE
Switch
FCoE
Switch
FCoE
Switch
FCoE
Switch
Lossless Ethernet
FC
FC
Lossless
Ethernet
1
Lossless
Ethernet
2
FCoE
Switch
FC
FC
FCoE
Switch
Lossless
Ethernet
4
Ethernet
FC
31
LAN/WAN
Data Center
Messaging
NAS
- Fibre Channel
LAN/WANs
Note: with multiple Data Centers there may also
be interconnects with DWDM, FCIP/iFCP, etc.
Messaging
NAS
32
Current Fabrics
Remote
Offices
Clustering Network
iSCSI
Storage
Mainframe
Focus: Low
Latency & High
Bandwidth
Ficon
FC
Outfacing
IP Network
FC
Network
FC
FC
(LAN/WAN)
Messaging
NAS
Ficon
Clustering Network
File Storage
Arrays (NAS)
Focus: Protection,
Bandwidth/Congestion
Management
Ficon
Link
Ethernet
Link
33
Other IP Nets
(LAN/WAN)
Messaging
NAS
34
Ethernet
SW
FC
FC & Ethernet
SW Blades
FC
Outfacing
IP Network
DataCenter
Fabric
(LAN/WAN)
Including iSCSI
Gateways
FCoE
Messaging
NAS
Ficon
FC
Ficon Storage
Controller
Ethernet
SW
File Storage
Arrays (NAS)
Business
Campus
with iSCSI
connections
FC Link
Ficon
Link
Ethernet
Link
35
Scenarios
36
Internet
Classical
Ethernet
Lossless
Ethernet
FCoE
Switches
FC
FC
Lossless
Ethernet
FCoE
Switch
FCoE Flows
IP Flows
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
37
Internet
Classical
Ethernet
Lossless
Ethernet
FCoE
Switches
FC
FC
Lossless
Ethernet
FCoE
Switch
FCoE Flows
Invalid FCoE Flows
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
38
Summary
39
Summary
FCoE is a simple, efficient mechanism for
encapsulating Fibre Channel in Ethernet frames
FCoE is being standardized in INCITS Fibre
Channel (T11) technical committee
Target completion is 2H08
40
Thank You!
Q&A / Feedback
Please send any questions or comments on this
presentation to SNIA: tracknetworking@snia.org
Many thanks to the following individuals
for their contributions to this tutorial.
SNIA Education Committee
Claudio DeSanti
Robert Snively
Joe Pelissier
Howard Goldstein
Suresh Vobbilisetty
John Hufferd
Walter Dey
Silvano Gai
42
Appendix
Additional
Requirements
FCoE Relation to ISO
Layers
Flows
Additional Topologies
FSPF and STP
FIP Considerations
and Actions
FIP Descriptors
Pause vs. BB_Credit
Fibre Channel over Ethernet (FCoE)
2008 Storage Networking Industry Association. All Rights Reserved.
43
44
FCoE Goals/Requirements
(part 4)
45
46
Flows
47
Lossless
Ethernet
FCoE
Switch
Ethernet
FC
H2
FCoE
Switch
#2
#1
Lossless
Ethernet
Lossless
Ethernet
#3
#4
FCoE
Switch
H4
H3
H1
FC
FC
S2
Lossless
Ethernet
S1
Destination
& Source
Encapsulated
FC Frame
D_ID
S_ID
Path #1
FCoE-A MAC
FCoE-H2 MAC
FC_ID for S1
FC_ID for H2
Path #2
FCoE-B MAC
FCoE-A MAC
FC_ID for S1
FC_ID for H2
Path #3
FCoE-C MAC
FCoE-B MAC
FC_ID for S1
FC_ID for H2
Path #4
FCoE-S1 MAC
FCoE-C MAC
FC_ID for S1
FC_ID for H2
48
H2
Ethernet
Switch
Ethernet
Switch
#1
FCoE A
Switch
FCoE
Switch
Ethernet
Switch
Ethernet
Switch
#2
Lossless Ethernet
C
#3
Ethernet
Switch
S2
S1
H1
Ethernet
Destination
& Source
FC
FC
Path #1
Path #2
FCoE-A MAC
FCoE-H2 MAC
FCoE-C MAC
FCoE-A MAC
Path #3
FCoE-S1 MAC
FCoE-C MAC
Encapsulated
FC Frame
D_ID
FC_ID for S1
FC_ID for S1
FC_ID for S1
S_ID
FC_ID for H2
FC_ID for H2
FC_ID for H2
Logical
Transaction Path
Ethernet
FC
49
S2
H2
Ethernet
Switch
Ethernet
Switch
#3
FCoE
Switch
Ethernet
Switch
FCoE
A
Switch
FC
FC
#2
Ethernet
Switch
Lossless Ethernet
S3
Ethernet
Switch
S1
H1
Ethernet
Destination
& Source
#1
Path #3
Path #2
FCoE-S2 MAC
FCoE-A MAC
FCoE-A MAC
FCoE-C MAC
Path #1
Encapsulated
FC Frame
D_ID
FC_ID for S2
FC_ID for S2
FC_ID for S2
S_ID
FC_ID for H5
FC_ID for H5
FC_ID for H5
Logical
Transaction Path
Ethernet
FC
50
Additional Topologies
51
Additional Topologies
FCoE
Switch (A)
(1)
Lossless
Ethernet
Switch
FCoE
Switch (E)
FC
FC
FCoE
Switch (B)
Lossless
Ethernet
Switch
FCoE
Switch (F)
52
Additional Topologies
Lossless
Ethernet
Switch
(2)
FCoE
Switch (A)
FCoE
Switch (E)
FC
FC
Lossless
Ethernet
Switch
FCoE
Switch (B)
FCoE
Switch (F)
53
54
55
56
Equivalent FC topology
57
58
59
60
ENode Actions
When an ENode becomes operational:
The ENode discovers the VF_Port capable FCF-MACs
connected to the same Lossless Ethernet segment
Transmits a multicast Solicitation to All-FCFs (with the FCF bit set
to zero)
Receives Jumbo unicast Advertisements from select compatible
VF_Port capable FCF-MACs
Stores the discovered FCF-MACs in an FCF list
61
FIP Descriptors
62
63
64
65