Architecting Data Center

Networks in the era of Big Data

and Cloud
Brad Hedlund
Spring InteropMay 2012

Global Marketing

Two approaches to DC Networking

THE SAME OLD

Centralized, Scale-up Layer 2 networks

Monstrous chassis switches
TRILL

OpenFlow VEPA

SPB

Or a Different Approach
Distributed, Scale-out Layer 3 fabrics
Efficient fixed switches
Open, industry standard protocols
2

Global
Brad Marketing
Hedlund

Networks that suck for Cloud & Big Data

Core

Network
Topology

Dist
Access

VM
PARTITIONED

Capacity
Topology

CAPACITY

Data center networks are in my way

-James Hamilton, AWS
3

Global
Brad Marketing
Hedlund

Networks that Dont suck for Cloud & Big Data

Spine

Network
Topology

Leaf
VM

UNIFORM CAPACITY

Capacity
Topology

All points equidistant

4

Global
Brad Marketing
Hedlund

Big Data
switch

TCP

TCP

switch

switch

switch

Name Node

Job Tracker

Secondary NN

Node

Node

Node

switch

switch

switch
Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Rack 1

Rack 2

Rack 3

Client

Rack 4

TCP

TCP
World

Client

Rack N

Inverse Virtualization
Workloads orchestrated like cattle
L2 or L3 network. Does it matter?
5

Global
Brad Marketing
Hedlund

Basic requirements of Cloud (IaaS)

switch

switch

switch

switch

Physical
Network

switch

FW

switch

LB

World

VM

VM

VM

VM

Virtual
Network

Secure, Scalable Multi Tenancy

Location independence
On Demand virtual networks
6

Global
Brad Marketing
Hedlund

Blend the Virtual and Physical Networks

VLAN 20
switch

VLAN 10
switch

switch

vSwitch

vSwitch

VM VM

VM VM

Host

Host

VM

VM

VM

VM

Tenant subnet = Network VLAN

7

Global
Brad Marketing
Hedlund

Abstract the Virtual Network from Physical

Segment ID 20
switch
Segment ID 10
switch

switch

vSwitch

vSwitch

VM VM

VM VM

Host

Host

VM

VM

VM

VM

Network Virtualization Overlay

Tenant subnet = Software VLAN
8

Global
Brad Marketing
Hedlund

Scale-up centralized Layer 2

L3

2-post Rooted Architecture

L2

Centralized L2/L3
L2/L3/ARP table scale?

Scale w/ Bigger Boxes

Precious Pets
VLAN Provisioning?
Broadcasts

vSwitch

vSwitch

VM VM

VM VM
Global
Brad Marketing
Hedlund

Scale-out Layer 3 Leaf/Spine Fabric

(16)

(8)

L3

(2)

L2

(128)
(64)

(16)
1980
768 Server
Server
ports
3072
6144
Serverports
ports

Mesh from Leaf to Spine

Non-blocking Spine

OSPF, ISIS, BGP, TRILL

3:1 @ ToR

ToR w/ 16 uplinks (ECMP)

128 port 2RU Spine

10

Global
Brad Marketing
Hedlund

Uniform fabric for Cloud & Big Data

(16)

(8)

(2)

Storage Access

Hadoop
(128)
(64)

L3

(16)

L2

Database
6144 Server ports

Name Node

Job Tracker

Node

Node

Secondary NN

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

11

Client

Client

Rack 1

Rack 2

Node
Node

Block I/O
NAS
Object

Rack 3

vSwitch

vSwitch

VM VM

VM VM

Rack N

Global
Brad Marketing
Hedlund

Attaching Services & North/South

(16)

(8)

(2)

(128)
(64)

(16)

Name Node

Job Tracker

Node

Secondary NN

Node

Node

Node

Node

Node

Node

Node

Node

Client

L3

Client

Firewall
Firewall

LB
LB

vswitch
VM

VM

VM

vswitch

Rack 1

12

Rack N

VM

VM

vswitch

vswitch

VM

VM

VM

VM

VM

VM

VM

VM

x86

Gateways

World

vswitch

vswitch
VM

VM

L2

VM

VM

VM

Global
Brad Marketing
Hedlund

Generic Logical Architecture 1

World

Overlay based L2
Physical/Static FW

FW

Fabric

L3
NAT

DC router

FW

L3
NAT

L2

LB

LB

L2
L3

VM

VM

L2

VM

VM

Green Co.
13

VM

L2

Big Data
Orange Co.
Global
Brad Marketing
Hedlund

Generic Logical Architecture 2

World

Overlay based L2
Virtual/Mobile FW
Overlay Gateway
FW

Fabric

DC router
Pub DMZ

L3
NAT

FW

L3
NAT

L2

LB

LB

L2
L3

VM

VM

L2

VM

VM

Green Co.
14

VM

L2

Big Data
Orange Co.
Global
Brad Marketing
Hedlund

Density: Fixed vs. Chassis

140

10G per RU @ Line Rate (L3)

120
100
80
Chassis
Fixed

60
40
20
0
2008
15

2010

2012

2014
Global
Brad Marketing
Hedlund

Power: Fixed vs. Chassis

18

Max Watts / Line Rate 10G (L3)

16
14
12
10

Chassis
Fixed

8
6
4
2
0
2010
16

2012

2014
Global
Brad Marketing
Hedlund

What are the Challenges?

(16)

(8)

(2)

BOOTH DEMO
Dell Fabric Manager

(128)

Validate deployment
Automate fabric configuration
(16)
Monitoring & Operations

L3
L2

Deployment & Cabling

Layer 2 (TRILL?)

Configuration & Policy

Design Best Practices

Monitor & Troubleshoot

Dell Fabric Manager

17

Global
Brad Marketing
Hedlund

Webinar: CLOS Fabrics Explained

DISCOUNT CODE: BradInterop

http://closfabric.eventbrite.com/
Wednesday, June 20, 2012 from 10:00 AM to 1:00 PM (ET)
HOST
CO-HOST

Yours Truly

Ivan Peplenjak
18

Global
Brad Marketing
Hedlund

The power to do more

Visit us at booth #1727
19

Generic Logical Architecture 3

World

No Overlays
TRILL based L2
Virtual/Mobile FW

Fabric

DC router
Pub DMZ

TRILL

FW

L3
NAT

FW

L3
NAT

L2

LB

LB

L2
L3

VM

VM

L2

VM

VM

Green Co.
20

VM

L2

Big Data
Orange Co.
Global
Brad Marketing
Hedlund

Three Stage Layer 3 Leaf/Spine Fabric

(64)

(8)

(2)
/26

0/0

(128)
/26
(512)

/26

0/0

L3

L2

24,576 Server ports

Leaf+ToR mesh groups

Non-blocking @ top tiers
Default route @ ToR & Leaf ~8usec worst case
21

Global
Brad Marketing
Hedlund

8192 non-blocking Fabric

(64)

(8)

(2)

(128)

L3

L2

8192 non-blocking

22

Global
Brad Marketing
Hedlund

4096 non-blocking Fabric

(32)

(2)

L3

(64)

L2

4096 non-blocking

23

Global
Brad Marketing
Hedlund

2048 non-blocking Fabric

(16)

(2)

L3

(32)

L2

2048 non-blocking

24

Global
Brad Marketing
Hedlund

256 non-blocking Fabric

(2)

(4)

256 non-blocking

25

Global
Brad Marketing
Hedlund

The case for 40G QSFP switch ports

10G 10G 10G 10G

10G

10G

10G

10G

SFP+

SFP+

SFP+

SFP+

$1K

$1K

$1K

$1K

VS
QSFP
$1,800

32 ToR

$230K

$512K

26

Global
Brad Marketing
Hedlund