100G Deployment Considerations

Igor Giangrossi
Consulting Systems Engineer
100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

Agenda
The need for 100GE Standards Work Status 100G and Optical Transport Systems

100GE Router Implementations

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

Global IP Traffic Growth

IP Traffic will increase 4.3x from 2009 to 2014
70
34% CAGR 20092014

60
Exabytes/mo 50 40 30 20 10 0

Business Consumer

13%

87%

2009

2010

2011

2012

2013

2014

Source: Cisco Visual Networking Index (VNI) Global Forecast, 20092014

100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

Why Even Higher Speed Interfaces?

Current Interfaces:
10Gbps: Ethernet 40Gbps: SONET/SDH

Ethernet Link Aggregation

Max 8 links (Std) Unequal Load Balance Limits flow BW Harder to manage Inefficient fiber use

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

Different SDOs for 100Gb/s Interfaces

Defined Ethernet MAC and PHY specifications

Defining 100Gb/s signals over 50GHz optical channels on DWDM systems (modulation, etc)

Defining how to map 100Gb/s signals over OTN/G.709 encapsulation

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

IEEE 802.3ba Standard

Ratified on IEEE July 2010 Meeting 40 and 100 Gb/s Ethernet MAC and PHY Cooper and fiber connectivity

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

Why 40G and 100G Ethernet?

Addressing Needs of Core and Access Network
1 000 000 Core Networking Doubling 18 months Server I/O Doubling 24 months

100 000

Bandwidth (Mb/s)

100G Ethernet 40G Ethernet

10G Ethernet

10 000

1000

Gigabit Ethernet

100 1995

2000

2005

2010

2015

2020

100G forecast to be needed in the core in 2010 Server I/O will not require 100G for several years
- Will require 40G much earlier - Growth follows Moores Law
100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

IEEE 803.2ba Summary

40 GbE 1 m Backplane 7 m Copper 100 m MM Fiber 10 km SM Fiber 40 km SM Fiber 40GBASE-KR4 40GBASE-CR4 40GBASE-SR4 40GBASE-LR4 100GBASE-CR4 4/10 cooper @ 10Gb/s 100GbE Comments

100GBASE-SR10 4/10 fibers @ 10Gb/s 100GBASE-LR4 100GBASE-ER4 CWDM (4) CWDM (4)

Additional Info: - 802.3 Frame format is maintained - 64B/66B Encoding: 40GBASE-R (40Gb/s) and 100GBASE-R (100Gb/s) - Cooper and backplane PHYs have an optional FEC layer

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

10

ITU Work on 40/100G (SG15 OTN)

OTU1 (2.7Gb/s) OTU2 (10.7Gb/s) OTU3 (43Gb/s) OTU4 (100Gb/s) G.709: OTN Network Node Interface (NNI) Client interfaces specified for SDH services (STM-16, STM-64 and STM-256) OTU3 supports STM-256; proposed for 40GbE OTU4 (>100Gb/s) proposed for 100GbE
100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

Wide Area Network

OTU4

11

Optical Characteristics for 100G/40G

100G vs 10G OSNR Requirement CD Tolerance

100G vs 40G

10 dB higher
100 X less 10 X less 100 X less 10 X

4 dB higher
6.25 X less 2.5 X less 6.25 X less 2.5 X

DGD Tolerance PMD Limited Distance

Optical Bandwidth

Implementing 100G and 40G is much harder than 10G in the Optical Domain
100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

12

100G Optical Systems

Transmitter: Increasing speed means
Complex Optics = Complex Electronics = $$$$$$ More Optical Impairments

Receiver: Address impairments

Optical compensation vs. Electrical compensation Coherent vs. Direct detection

Forward Error Correction (FEC)

Hard Decision FEC Standard FEC 6dB Coding Gain Enhanced FEC 8+dB Coding Gain Newer FECs 9+dB Coding Gains Soft Decision FEC

Each of the above FECs can be used for any of the options
100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

100G Work at the OIF

Several Projects for 100G DWDM Systems
100G Long Distance DWDM Transmission Framework 100G Long Distance DWDM Integrated Photonics Receiver 100G Long Distance DWDM Integrated Photonics Transmitter Forward Error Correction (FEC) for 100G DP-QPSK Long Distance Communication 100G Long-Haul DWDM Transmission Module Electromechanical 100G Long-Haul DWDM Transmission Module MDI

Agreed on PM-QPSK* as the modulation scheme, using coherent receivers

* Polarization Multiplexing Quadrature Phase Shift Keying
100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

16

100G The Transmitter

Need to go slower
Optical impairments are directly related to signaling rates

Need to increase modulation efficiency

Signaling speed decreases & Information Rate increases NRZ to ODB to (D)PSK to (D)QPSK

Need to increase optical efficiency

Split signal over two polarizations (PM Mod Scheme)
1 bit/symbol 1 bit/symbol 2 bits/symbol 00 01 11 QPSK 10

0 NRZ
100G LACNOG 2010

-1

PSK
Cisco Confidential

2010 Cisco Systems, Inc. All rights reserved.

OSNR Required at 111 Gb/s

32

Spectral efficiency [bit/s/Hz]

16 8 4

Higher level modulation formats require PM-1024QAM Shannon Bound increasingly higher PM-256QAM for PM formats OSNR even at PM-64QAM constant bit rate PM-16QAM
PM-8QAM

PM-QPSK

Requires extra 8 dB to triple spectral efficiency Requires extra 4 dB to double spectral efficiency

PM-16QAM requires 4 dB more than PMQPSK

2 1

PM-64QAM requires 8 dB more than PMQPSK BER=1E-3 PM-256QAM would require 13 dB more than PM-QPSK!

0.5

12

15

18

21

24

27

30

OSNR over 0.1 nm [dB]

100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

100GE Interfaces on a Router

Linecard 1
Fabric

Linecard 2
M M

Fabric IF
M M

FWD Engine
M M

M A C

CFP

CFP

M A C

FWD Engine
M M

Fabric IF
M M

Fabric IF

FWD Engine
M M

M A C

CFP

Not every router will be able to implement 100GE

100G LACNOG 2010 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

21

Some Challenges for a 100GE Linecard

Substantially larger and power hungry pluggable Many current MAC Chip on FPGAs: more space, power Forwarding Engine: 100GE != 2x50GE
Higher PPS, faster memory lookups, faster interfaces

Packet Buffering @ 100Gbps Circuit Board challenges:

Real Estate difficult to fit everything Circuit routing several layers Xtalk between lanes @ high speed SerDes interfaces Power dissipation

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

22

Pluggable Comparison: CFP vs XFP

XFP Length Width Height Power Consumption 78 cm 18 cm 8.5 cm 3.5W

CFP 145 cm 77 cm 13.6 cm 32W

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

23

Key Takeaways
100Gb/s Ethernet is already a reality Implementing 100Gb/s links poses many challenges to many currently deployed systems Optical costs become much more relevant at 100Gb/s links Careful planning is needed to take full advantage of the bandwidth gains

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

25

100G LACNOG 2010

2010 Cisco Systems, Inc. All rights reserved.

Cisco Confidential

26