High-speed backplane interconnect

Vladimir Stojanovic
(with slides from J. Zerbe, P. Desai, R. Kollipara)

Outline
Inside the router Backplane channel problem
What can backplane designer do about it What can IC designer do about it

Scaling the system to 10-100Tb/s

Inside the Router

Line Cards: 8 to 16 per System
Passive Backplane Switch Cards: 2 to 4 per System

MEM

MEM

MEM

MEM

SerDes

Crossbar

Optics

SerDes

MAC

NPU

TM/ Fabric IF

SerDes

Past
OC-12 622 MHz LVDS parallel GigE 1.25 Gbps serial

Present
OC48 2.5 Gbps serial 10GigE XAUI (3.125 Gbps) serial

Serial Links in Networking Systems

Line Card: Optics
MAC/ Framer NPU/ TM CSIX Proprietary Switch Fabric IF

XAUI SPI4.2 4, 3.125 Gbps Serial Links

Backplane 8 to 16 of 1-3.2Gbps Serial Links

Switch Card:
32 to 64 Backplane Serial Links (1-3.2 Gbps) Switch Crossbar IC

Backplane interconnect path

Package to board transition

Line card trace

Backplane trace

Package Backplane connector

Chip

Line card via

Backplane via

There are many components on the signal path, potential source of problems

RaSer X Link Features

20 bit Parallel Interface Serializer Tx Tx Link 1-10 Gbps TX PLL

Eq

RefClk

PLL

RX

20 bit Parallel Interface

Deserializer and CDR

Rx

Rx Link 1-10 Gbps

Eq

Process Power Area 2-PAM Range 4-PAM Range

0.13m CMOS 40mW / Gb 1mm2 2 6.4 Gb/s 5 10 Gb/s

I/O Driver Scheme (Example)

Vtt
50 W
Zo = 50 W

Vtt
50 W

Tx

Rx
Zo = 50 W

Impedance-controlled (CML) I/Os Integrated terminations Adjustable output-voltage/common mode

System Issues
Goal Increase Router Throughput Limitations
Backplane channel Power Mechanical/Physical density constraints Backplane and linecard routing density Connector pin density Package I/O density

Outline
Inside the router Backplane channel problem
What can backplane designer do about it What can IC designer do about it

Scaling the system to 10-100Tb/s

Backplane Component Effects

PCB only

PCB + Connectors

PCB, Connectors, Via stubs & Devices

Deterministic Noise
Attenuation [dB]

pulse response

0 -10 -20 -30 -40 -50 -60

FEXT NEXT THROUGH

1 0.8 0.6 0.4 0.2 0 Tsymbol=160ps

6 8 10 frequency [GHz]

ns

Inter-symbol interference
Dispersion (skin-effect, dielectric loss) - short latency Reflections (impedance mismatches connectors, via stubs, device parasitics, package) long latency

XTALK and reflections

Far-end XTALK (FEXT) Desired signal

Reflections Near-end XTALK (NEXT)

Primary reflection sources are at the connector/backplane transition

Grouped in time as a function of backplane length

Backplane channel variations

Attenuation [dB]

0 -10 -20 -30 -40 -50 -60 0 2 4 9" FR4, via stub 26" FR4, via stub 6 8 10 frequency [GHz] 26" FR4 9" FR4

Variability in trace length, routing layer and via stub Significantly different transfer functions even within the same backplane

Test Backplane Example

Trace lengths: 1.5, 9, 14, 20 and 32 Effective number of signal layers: 13 Effective number of total layers: 28
Dielectric material FR-4 Nelco 6000 Roger 4350

FR4 Cross Section

Dielectric constant

4.2

4 0.005 0.007

3.6 0.0035 0.0035 0.297"

Loss tangent (1 MHz) 0.016 Loss tangent (1 GHz) 0.017 Thickness

0.295" 0.299"

Backdrilling - A Solution to the Stub Effect

Stub Effect Eye Pattern Analysis

(2.5 Gbits/sec FR-4)
MAX STUB MIN STUB

Stub Effect Eye Pattern Analysis

(5.0 Gbits/sec FR-4)
MAX STUB MIN STUB

Stub Effect Eye Pattern Analysis

(12.0 Gbits/sec FR-4)
MAX STUB MIN STUB

Connector design

GBX Teradyne

Connector Density
Teradynes GbX Connector Differentia l Pairs/inch 5 pair 69 Card Pitch 1.25" min. (30 mm) Bandwidth/linear inch (at 6.25 Gbps) 431 Gb

4 pair
3 pair 2 pair

55
41 27.5

1.00" min. (24.7mm)

.80" min. (20 mm) .575" min. (14 mm)

343 Gb
256 Gb 171 Gb

Reducing Crosstalk within the Connector

D/C Shield Cross talk is reduced in the mating interface by surrounding each pair with a ground shield

Mated pair

B/P Shield

Backplane Connector Considerations

Many connector types:
Teradyne: VHDM, HSD, GbX, Tyco: HS3, HMZd, FCI: Metral 2000, 3000, 4000, 3M/Harting: HSHM, ERNI: ERmetZd, ErmetXT,

Issues
Loss, impedance profile, crosstalk, skew Foot print: routability, pin density, via impedance

Single-ended and differential Press-fit and SMT

10Gbps Test Package Design Example

Ceramic BGA Wire-bonded 4-Layer 1 mm pitch

Source: Designed for Rambus by Kyocera

Example of a really good backplane

Works with simple OC192 xcvr 15 FR4

20 Roger

Outline
Inside the router Backplane channel problem
What can backplane designer do about it What can IC designer do about it

Scaling the system to 10-100Tb/s

Loss : Equalize to Flatten Response

=
Channel is band-limited Equalization : boost high-frequencies relative to lower frequencies

Receiver Linear Equalizer

Amplifies high-frequencies attenuated by the channel Digital or Analog FIR filter Issues
Also amplifies noise! Precision Tuning delays (if analog) Setting coefficients Adaptive algorithms such as LMS

W1

WL-1

WL

H(s)

freq

Transmitter Linear Equalizer

Attenuates low-frequencies
Need to be careful about output amplitude : limited output power If you could make bigger swings you would EQ really attenuates lowfrequencies to match high frequencies Also FIR filter : D/A converter

Can get better precision than Rx Issues

How to set EQ weights? Doesnt help loss at f

H(s)

freq

Transmit Linear Equalizer : Single Bit Operation

0.7

0.5

Unequalized Equalization Pulse End of Line

Voltage

0.3

0.1

-0.1

-0.3 0.0

0.3

0.6

0.9

1.2

time (ns)

Decision Feedback Equalization (DFE)

Dont invert channel just remove ISI
Know ISI because already received symbols Doesnt amplify noise
Feed-forward EQ Decision (slicer)

FIR filter

FIR filter
Feed-back EQ

Requires a feed-forward equalizer for precursor ISI

Reshapes pulse to eliminate precursor

DFE Example

Transmit and Receive Equalization

TX DATA RX DATA

TAP SEL LOGIC

Transmit and receive equalizers are combined to make a range restricted DFE
Tx equalizer functions as the feed-forward filter Rx equalizer restricted in performance of loop

Tx & Rx Equalization Ranges

RX Equalizer 5-17 taps after main Pick any 5 taps

TX Driver/Equalizer : 5 taps 1(pre)+1(main)+3(post)

Pulse Amplitude Modulation

Binary (NRZ) is 2-PAM 2-PAM uses 2-levels to send one bit per symbol Signaling rate = 2 x Nyquist

4-PAM uses 4-levels to send 2 bits per symbol Each level has 2 bit value Signaling rate = 4 x Nyquist

00 0 0 01

11 1 1 10

When Does 4-PAM Make Sense?

Nyquist Frequency (GHz)
0.0 1.0 2.0 3.0 4.0 5.0

-20db

-40db

-60db

|H(f)|

First order : slope of S21

3 eyes : 1 eye = 10db loss > 10db/octave : 4-PAM should be considered

Example : 5Gbps Over 26 FR4 With No Equalization

Example : 5Gbps Over 26 FR4 Correct Tx Equalization

Example : 5Gbps Over 26 FR4 Under Equalized

Example : 5Gbps Over 26 FR4 Over Equalized

26 FR4 Bot 3.125Gbps, 2P noEQ

26 FR4 Bot 3.125Gbps, 2P w/EQ

26 FR4 Bot 6.4Gbps, 2P w/3G EQ

26 FR4 Bot 6.4Gbps, 2P w/EQ

26 FR4 Top 6.4Gbps, 2P w/EQ

26 FR4 Top 6.4Gbps, 4P w/EQ

26 Nelco6k-cb Top 10Gbps, 4P

26 Nelco6k-cb Top 6.4Gbps, 2P

Scaling the router throughput

System (Tot. throughput ~2.5Tb/s)
8-16 Line Cards <40Gbs / LC 3-6Gbs links 40mW/Gbs Link power in 0.13um Speedup ~2x #links at switch card ~ 200

Limitations
#diff pairs at switch card 16*40Gbs/6Gbs*2*2 ~ 400 Switch card power from links ~200*6Gbs*40mW/Gbs ~ 50W Connector density 50diff pairs/inch: (tot length=400/50= 8 ) BP/LC routing pitch 0.050 Num. Layers (BP=13, LC=4) 100diff pairs/layer = 5 LC routing width Package ball pitch (1mm/200um)

Scaling the router throughput

System (Tot. throughput ~100Tb/s)
100 Line Cards 1Tbs / LC 10Gbs links 4mW/Gbs Link power in 0.065um Speedup ~1x #links at switch card ~ 10k

Limitations
#diff pairs at switch card 20k Switch card power from links in 0.13um ~10k*10Gbs*4mW/Gbs ~ 400W Connector density 50diff pairs/inch: (tot length=20k/50= 400 ) BP/LC routing pitch 0.050 Num. Layers (BP=13, LC=4) 5k diff pairs/layer = 250 LC routing width Package ball pitch (1mm/200um)