BRKDCN 16598639898 PDF

#CLUS
Cisco Nexus 3000

Switch Architecture
Faraz Taifehesmatian, Technical Marketing
Engineer
CCIE R&S, DC
BRKDCN-3734
#CLUS
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKDCN-3734

by the speaker until June 18, 2018.
#CLUS BRKDCN-3734 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Abstract
This session presents an in-depth study of the architecture of the latest
generation of Nexus 3000 top-of-rack data center switches. Topics
include merchant silicon architecture and capabilities (Broadcom
Trident3, Tomahawk 2, Jericho+ and Barefoot Tofino) , forwarding
hardware, and other physical design elements, as well as a discussion
of key hardware-enabled features and capabilities that combine to
provide high-performance, low latency data center network services.
What This Session Covers
• Overview of Merchant Silicon with Cisco Nexus Switches
• Latest generation of Nexus 3000 switches
• System and hardware architecture, key forwarding functions,
packet walks
Not covered:
• Nexus 9000 ASIC/platform architectures
• Nexus 9500 merchant-silicon based architectures
• Other Nexus platforms
Agenda
• Merchant Silicon Overview
• Nexus 3000 Portfolio
• ASIC overview
• Forwarding Pipeline
• Platform Specific Details
• Key Takeaways
Merchant Silicon
Overview
Merchant Silicon in Cisco DC switches
N3000 N3200, N3100V,9500-R N3100Z,N3200E,N3400
StrataXGS StrataXGS StrataXGS

Trident +, 480 Gbps Tomahawk, 3.2 Tbps Tomahawk II, 6.4 Tbps
Trident II+, 1.28 Tbps Trident III, 3.2 Tbps
StrataDNX StrataDNX
Jericho, 600 Gbps Jericho II, 2.4 Tbps
Barefoot
Tofino, 1.8, 6.4 Tbps
N3100,N9300 N3600R
StrataXGS StrataDNX
Trident II, 1280 Gbps Jericho+, 900 Gbps
Nexus 3000 Portfolio
Nexus 3000 Series Switch Portfolio
Nexus 3100 Nexus 3200 Nexus 3400

• ToR Leaf • Fixed High Density • Programmable pipeline
• Full-featured DC access • High throughput & performance • Support for P4-INT
• Broad switch portfolio • Flexible connectivity options • Enable custom use cases
• Based on Trident ASIC family • Based on Tomahawk ASIC family • Based on Tofino ASIC
Nexus 3500 Nexus 3600

• Ultra Low Latency • Deep Buffer
• Financial/HFT workloads • High route scale
• Based on Cisco Monticello ASICs • Video & Drop sensitive deployments
• Based on Jericho+ ASIC family
• Nexus 3100 Switch Family
• Trident 3 ASIC Architecture
• ASIC Single Pipeline Block
Nexus 3100 • N3K-C3132C-Z Switch
Architecture
• N3K-C3132C-Z ASIC Port-
map
BRKDCN-3734 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Nexus 3100 Switch Family
• Based on Trident 2
3100 • 1.28Tbps
• 12.2 MB Buffer
• Based on Trident (2+)

3100-V • 1.28Tbps
• 16MB Buffer
• Based on Trident (3)

3100-Z • 3.2Tbps
• 32MB Buffer
Trident 3 ASIC Architecture
• BCM56870 from StrataXGS family (BCM56870)
Dynamic Memory Manager Fully Shared
• 3.2Tbps Single Chip Ethernet Switch (8 CoS, SP/WRR/WDRR) 32MB Buffer
• 2 Pipes @1.6 Tbps Multistage Content Aware Engine

(Classify, Filter, Modify, Parallel Lookup)
L2/L3
Host/ARP
• 32 MB of Buffer L2 MAC
L2/L3 Processing L3
Processing
VLAN, VRF, VSI, ECMP
Flex
Hashing/ Lead Balancing Engine Counters
CPU
128 x 25G SerDes MerlinCore
PCIe
…………
32 x 100GE, 10GE, 4 x Gen3
64 x 40GE/50GE 2.5GE,
128 x 10GE/25GE 1GE
Trident 3 ASIC Single Pipeline Block
FleXGS Ingress Pipeline 0
Lookup Lookup Lookup Special
Port Macro Parser Engine(s) Engine(s) Functions,
Engine(s)
ECMP, Hash
Memory
Management
Unit (MMU)
FleXGS Egress Pipeline 0
Editor Lookup
Port Macro Editor
Control Engine(s)
Trident 3 Cut-Through Vs Store-and-Forward
• Trident 3 MMU supports both • CT mode is used in latency-
store-and-forward (SF) and cut- sensitive applications
through (CT) modes
• In CT mode, the packet is
• In SF mode, an entire incoming scheduled through the cut-
packet is written into the buffer through path and dequeued to the
first. The packet is held in the EP before it has been completely
buffer until the scheduler selects received from the ingress pipeline.
that particular egress port's queue
• In CT mode, After one or more
packet cells are received by the
MMU, the packet becomes eligible
for dequeening
Trident 3 Cut-Through switching Matrix
Destination port
Min Ingress Speed Max Egress Speed
Speed 10G ->10G
25G -> 10G
10G 50G 10G 40G -> 10G
25G 50G 25G 50G -> 10G
40G 100G 40G

50G 100G 50G
100G 100G 100G
N3K-C3132C-Z
Lane-selected LED
Beacon LED
Status LED
Environment LED
32 40/100GE QSFP28 1 RU
2x10GE SPF+
Each QSFP28 port can operate at 10, 25, 40, 50, and 100 G
Up to a maximum of 128 x 25-Gbps ports
N3K-C3132C-Z Switch Architecture
Fan 1 Chimay CPU Board
Management DIMM 650W
Console G1x1PCIe PSU
Fan 2 USB
CPU
Leffe Board
Fan 3 1.8 GHz Intel Xeon
650W
Fan Interface Board PSU
Fan 4
G2x4PCIe G1x1PCIe CTRL/Status
Haggan Board
x4PCIe MI FPGA CPLD

CPLD
CPLD
CPLD
Trident 3
MC FCs
2 x 10 G 32 x QSPF28
SFP
N3K-C3132C-Z ASIC Port-map
Trident 3 FalconCores
M FC FC FC
C FC FC FC FC FC FC FC FC FC FC FC FC FC
1 1 5 9 13 17 21 25 29 2 6 10 14 18 22 26 30
M FC FC FC
C FC FC FC FC FC FC FC FC FC FC FC FC FC
2 3 7 11 15 19 23 27 31 4 8 12 16 20 24 28 32
Port Group 0
Pipe 0
Port Group 1
Port Group 2
Pipe 1
Port Group 3
3
3 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
3
4 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
C3132C-Z Front Panel Ports


• Tomahawk2 ASIC
Architecture
Nexus 3200 • ASIC Forwarding Pipeline

• N3K-C3264C-E Switch
Architecture
• N3K-C3264C-E ASIC Port-
map
• Based on Tomahawk
3200 • 3.2Tbps
• 16MB (4x4MB) Buffer
• Based on Tomahawk 2
3200-E • 6.4Tbps
• 42MB (4x10.5MB) Buffer
Tomahawk2 ASIC Architecture
• BCM56970 from StrataXGS family (BCM56970)
Dynamic Memory Manager 42MB
• 6.4Tbps Single Chip Ethernet Switch (8 CoS, SP/WRR/WDRR) SmartBuffer
• 4 Pipes @1.6 Tbps Multistage Content Aware Engine

(Classify, Filter, Modify, Parallel Lookup)
L2/L3
Host/ARP
• 42MB (4x10.5MB) of Buffer L2 MAC

L2/L3 Processing L3
Processing
VLAN, VRF, VSI, ECMP
• Ingress & Egress Packet Time

Hashing/ Lead Balancing Engine Flex
Stamping Counters
CPU
256 x 25G SerDes MerlinCore
PCIe
…………
64 x 100GE, 1x100GE, Gen3
128 x 40GE/50GE 2x40GE/50GE x4/x2/x1
128 x 10GE/25GE 4x1/2.5/10/25GE
Tomahawk2 ASIC Single Pipeline Block
Ingress Pipeline
Intelligent Tunnel Vlan L2 L3

ICAP
Parser Termination Processing Switching Switching
Memory
Management
Egress Pipeline Unit (MMU)
Packet Egress ACL Egress Vlan Egress
Modification Switching Processing Parsing
Tomahawk2 Cut-Through switching Matrix
Destination port
Min Ingress Speed Max Egress Speed
Speed 10G ->10G
25G -> 10G
10G 50G 10G 40G -> 10G
25G 50G 25G 50G -> 10G
40G 100G 40G

50G 100G 50G
100G 100G 100G
N3K-C3264C-E
16 QSFP28 +MACsec
Lane-selected LED
Beacon LED
Status LED
Environment LED
2 RU
64 40/100GE QSFP28
2x10GE SPF+
Each QSFP28 port can operate at 10, 25, 40, 50, and 100 G
Up to a maximum of 128 x 25-Gbps ports
N3K-C3264C-E Switch Architecture
Tyskie CPU Board
Fan1
Console
G1x1PCIe
PSU
USB Shocktop
Fan 2 CPU PSU
BlondeBock Board
Board 1.8 GHz Intel Xeon
1200W
Fan 3 PSU
G2x4PCIe G1x1PCIe G1x1PCIe
Tetley Board
x4PCIe MIFPGA MIFPGA CPLD0

0 1 CPLD1
Tomahawk2
FC
MC FC
8 X BearValley
2 x SFP 48 x QSFP28 16 x QSFP28

+ MACsec
64 x QSFP28
N3K-C3264C-E ASIC Port-map
Tomahawk2 FalconCores
FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC
9 13 18 21 6 25 36 60 39 44 52 56 32 47 64 4
FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC
10 14 17 22 5 26 35 59 40 43 51 55 31 46 63 3
M
C FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC
1 11 16 20 24 8 27 34 58 38 42 50 54 30 45 62 2
M
C FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC FC
2 12 15 19 23 7 28 33 57 37 41 49 53 29 44 61 1
Pipe 0
Pipe 1
8 X BearValley
Pipe 2
Pipe 3
1 2 3 4 5 6 7 8 9 10 11 12 49 50 51 52
13 14 15 16 17 18 19 20 21 22 23 24 53 54 55 56
6
5 25 26 27 28 29 30 31 32 33 34 35 36 57 58 59 60
6
6 37 38 39 40 41 42 43 44 45 46 47 48 61 62 63 64
C3264C-E Front Panel Ports

N3K-C3264C-E Breakout Port-mode
Following Port-modes are supported:
Default
• 96x50g + 16x100g
• 96x25g + 32x100g
• 128x25g
CLI commands for Port-mode change: System reload is required
Profile CLI : “hardware profile portmode <config>”

<config> is “96x50g+16x100g” or “96x25g+32x100g” or “128x25g”
Dynamic breakout CLI: “interface breakout module 1 port <front_port_num> map
<config>” <config> is “50g-2x” or “25g-4x” or “10g-4x”
N3K-C3264C-E 96x50g+16x100g
• All ports are operational and first 48 front ports will support 2x50G dynamic breakout.
• 64x100G, 64x50G + 32x100G, 96x50G + 16x100G will be met with this h/w profile
• 49-64 MACSEC Ports will support 100G and 40G operational modes
• SLIC adaptor is not supported in this h/w profile
Breakout Capable port

No Breakout
1 2 3 4 5 6 7 8 9 10 11 12 49 50 51 52
13 14 15 16 17 18 19 20 21 22 23 24 53 54 55 56
6
5 25 26 27 28 29 30 31 32 33 34 35 36 57 58 59 60
6
6 37 38 39 40 41 42 43 44 45 46 47 48 61 62 63 64
N3K-C3264C-E 96x25g+32x100g
• Front port 1-24, 29-32, 37-64 will be operational
• 1-24 front ports will support 2x50G, 4x25G, 4x10G dynamic breakout
• 29-32, 37-48, 49-64 MACSEC Ports will support 100G and 40G operational modes
• SLIC adaptor is supported on 1-24 Front port

No Breakout
Disabled
1 2 3 4 5 6 7 8 9 10 11 12 49 50 51 52
13 14 15 16 17 18 19 20 21 22 23 24 53 54 55 56
6
5 25 26 27 28 29 30 31 32 33 34 35 36 57 58 59 60
6
6 37 38 39 40 41 42 43 44 45 46 47 48 61 62 63 64
N3K-C3264C-E 128x25g
• Front port 1-28, 33-36 will be operational
• 1-28, 33-36 front ports will support 2x50G, 4x25G, 4x10G dynamic breakout
• SLIC adaptor is supported on 1-24 Front port

Disabled
1 2 3 4 5 6 7 8 9 10 11 12 49 50 51 52
13 14 15 16 17 18 19 20 21 22 23 24 53 54 55 56
6
5 25 26 27 28 29 30 31 32 33 34 35 36 57 58 59 60
6
6 37 38 39 40 41 42 43 44 45 46 47 48 61 62 63 64


• Barefoot Tofino ASIC Architecture
• Tofino Simplified Block Diagram
• Programmable Switch Approach
• Match-Action Packet Processing
Nexus 3400 • Match-Action Unit

• Hardware Telemetry
• Nexus 34180YC Switch
Architecture
• Nexus 34180YC ASIC Port-map
Barefoot Tofino ASIC Architecture
• BFN-T10-018D from Tofino family (BFN-T10-018D)
SRAM Memory
• 1.8Tbps Single Chip Ethernet Switch (Tables, Counters, Policers)
TCAM Table Memory

• 2 Pipes @0.9 Tbps (ACL, LPM)
• P4-programmable pipeline Match-Action Engine

(Exact Match, TCAM, range)
• Single 20 MB Unified Packet Buffer Packet Buffer

• Inband Network Telemetry (INT) CPU
CPU
72 x 25G SerDes
Eth. PCIe
…………
18 x 100GE, 1x100GE, Gen3
36 x 40GE/50GE 2x40GE/50GE x4/x2/x1
72 x 25GE/10GE 4x1/2.5/10/25GE
Tofino Simplified Block Diagram
Reset and PCIe CPU DMA
Clocks MAC Engines
Control and Configurations
Pipe0
10/25/ Ingress Egress 10/25/
….
….
40/50/100G Match-Action Match-Action 40/50/100G
Rx MACs Pipeline Pipeline Tx MACs
Common
Queuing
Pipe1 and Packet
10/25/ Ingress Data Egress 10/25/
….
….
40/50/100G Match-Action Buffers Match-Action 40/50/100G

Rx MACs Pipeline Pipeline Tx MACs
Tofino Programmable Switch Approach
Bottom-up Network element design Top-down Network element design
Network Demands Switch OS Network Demands NX-OS
Run Time API Auto Generated API
Driver Driver
ASIC
Tofino
Fixed-function Switch Programmable Switch
What Is P4?
• P4 - Programming Protocol-Independent Packet Processors
• Programming language designed to allow the definition of data planes
• Open-source, permissively-licensed language
• Designed to be protocol-independent, implementation-independent
• Protocol independence and the abstract language model allow for re-
configurability, target-independence
Tofino Match-Action Packet Processing
Programmable Ingress Match-Action Pipeline Programmable Egress Match-Action Pipeline
Buffer
Memory ALU Memory ALU Memory ALU Memory ALU Memory ALU Memory ALU
Programmable
… …
Parser
Action Action
Match Match
(PHV in) (PHV out) (PHV in) (PHV out)
Table Table
PHV’
PHV
Parameters
PHV
Parameters
PHV’
ALUs ALUs
(Match Action Unit 0 ) (Match Action Unit N )
PISA: Protocol Independent Switch Architecture
Multiple simultaneous lookups and actions can be supported
Memory ALU
Memory ALU
N lookups M actions
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Match + Action Stage (Unit)
PISA: Match and Action are Separate Phases
Sequential Execution (Match dependency)
Memory ALU Memory ALU Memory ALU
Memory
Total Latency = 3
Memory ALU Memory ALU ALU
Staggered Execution (Action Dependency)
Memory ALU
Memory ALU
Memory
Memory
ALU
ALU
Total Latency = 2
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Parallel Execution (No Dependencies)
Memory ALU
Memory ALU
Memory
Memory
ALU
ALU
Total Latency =
Memory ALU
1.1
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Memory ALU
Tofino Ingress Processing
• All packets processed by the ingress buffer & parser
• Parser splits packet header into separate PHV filed, TPHV files and packet body
• PHV files traverse through Ingress Match-Action Pipeline for table lookup and manipulation
• Deparser reassembles packets based on files in PHV
PHV PHV’ Common

Ingress Queuing and
10/ Ingress Buffer Match-Action Deparser
….
40/100G & Parser Packet Data

Pipeline Buffers
Rx MACs
Full Packet
Tofino Programmable Parser
• Will receive the packet data from the Ethernet MACs
Ethernet MACs and then it would parse 10Gb/s
the packet stream according to the 10Gb/s
10Gb/s
pre-computed parse graph 10Gb/s
SerDes
40Gb/s
100Gb/s Parser
40Gb/s
Next, the fields from the parsed
Packet Header Vector

Ingress Pipeline Arbiter
•
100Gb/s
protocol headers are extracted into the
corresponding PHVs 10Gb/s
10Gb/s
• Once the Parser has assembled a PHV 10Gb/s
10Gb/s
SerDes
Parser
it can then insert that PHV into the
100Gb/s
40Gb/s
40Gb/s
Match-Action Pipeline
100Gb/s
A single Parser unit can process packets at
about 100Gb/s, it connects to either:
4 x10/25Gb/s MACs, or PCIe
Host PCIe
Packet Interface
100Gb/s
Parser
2 x40/50Gb/s MACs, or
1 x100Gb/s MAC
Tofino Egress Pipeline
• Egress parser extracts metadata from ingress and packet header from the packet
• Egress Match-Action Pipeline performs additional processing
• Egress deparser assembles outgoing packet
PHV PHV’ 10/
….
Common
Egress 40/100G
Queuing and Egress Buffer Deparser
Match-Action Tx MACs
Packet Data & Parser Pipeline
Buffers
Full Packet
Tofino Egress Match-Action Pipeline
• Additional lookups for packet header modifications (i.e. tunnel
encap, multicast replicated packets)
• Perform calculations (such as WRED) based on intrinsic
metadata from TM
• Additional stats and policing as specified by P4 program
Tofino Egress Deparser
• PHV data is reassembled with packet
payload
• Unnecessary fields are omitted from
reassembled packet
Eth(L2) Fields
Vlan TCP UDP
• Final outgoing packet length fed back Vlan
IPv4
in
order
Eth(L2) IPv4
to TM for scheduling and shaping Fields

Dictionary TCP
feedback UDP Eth(L2) IPv4 UDP
All fields with holes removed

• Optionally send copy of packet to
mirror buffer for egress mirroring
• Optionally capture PHV data into digest
buffer for coalescing
Tofino Combined Ingress/Egress Pipeline
Combined Ingress/Egress Match-Action Pipeline Egress Packet Body
MAU 0 MAU n
Egress MAC
Parser PHV PHV
0 MAC
Egress Egress
…
Deparser Packet
…
Constructor
Egress
MAC
Parser
m MAC
…
Ingress
MAC Parser
0 Queues
MAC Ingress
And
Packet
…
Ingress Ingress Packet

…
Deparser Constructor
Buffer Buffers
Ingress
MAC Parser
m
MAC
Recirculation
Buffer
Ingress Packet Body

Telemetry Modes
Inband Network Telemetry

Postcard Mode
(INT)
• In the postcard mode, each network • Metadata is embedded in between

device generates its own telemetry the original headers of data packets
reports as they traverse the network
• The collector will receive reports • This is done by INT data plane
from different network devices, each specifications
describing the telemetry metadata
Postcard Mode vs INT Mode
Host Host
Header Payload Header Payload

Switch A Switch A
Watchlist Watchlist
Event detection
Packet Header + Info A Endpoint (Source)
Header Payload Header Switch A info Payload

Switch B Switch B
Watchlist
Event detection
Transit
Packet Header + Info B
Header Payload Header Switch A info Switch B info Payload

Switch C Switch C
Watchlist
Event detection
Endpoint (Sink)
Event detection
Packet Header + Info C Packet Header +

Header Payload Collector
Info A, B, C
Header Payload
Host Host
Inband Network Telemetry (INT)
• First Record (INT instruction
+metadata) will be inserted in data
INT Transit INT Transit
packet at INT Source node
• Second Record (INT metadata) will be
appended to same data packet at INT
Transit node INT Source INT Sink
• Third Record (INT metadata) will be

Host
appended to INT stack at INT Sink. Host
• INT Sink will remove INT record and

forward to INT Collector while original
packet will be forwarded to server
facing port
INT Collector
INT Per-switch information captured
Flow Watch List (zoom-in view per 5- Flow Drop List (Drop due to various
tuple of flow + DSCP bits) – 1K drop reasons) - 256
Switch ID Switch ID
Hop latency Ingress Port ID
Queue ID + Queue occupancy Egress Port ID
Ingress timestamp Queue ID
Egress timestamp Drop Reason
• Node-to-Node: Reserved DSCP bit will be inserted temporarily in data packets to indicate that
packets also carry INT data
• Node-to-Collector: A UDP encapsulation is used to pack collected INT stack at INT Sink and send to
collector. Flow-affinity is maintained to send same flow-record to same collector for easy processing
NX-OS INT Configuration Model
Exporter
Where to send?
(IP-Address, Source)
INT Record
What to capture?
“feature hw_telemetry”
(timestamp, Queue-id)
System Monitor
Watch List
Which flow?
Enable INT
(5-tuple, Permit/Deny) Monitor Session

<n>
Queue Profile (Role Transit/Endpoint,
What queue thresholds? Mode INT/Postcard)
(Depth, Latency)
Flow Profile(DSCP, Age,

How to identify the flow? Latency Quantization)
Which drops to report? Drop List
NX-OS INT Configuration Example
feature hw_telemetry
inband-telemetry exporter E1
destination 10.200.20.2
source Ethernet1/10
inband-telemetry record R1
collect switch-id
collect port-id
collect queue-occupancy
INT Transit INT Transit
collect ingress-timestamp
collect egress-timestamp
inband-telemetry watchlist ip WL1

10 permit ip 1.1.1.1/24 10.10.10.10/24 interface ethernet x/x interface ethernet x/x
20 deny ip 2.2.2.2/24 4.4.4.4/24 port-type fabric port-type fabric
inband-telemetry monitor M1
record R1
exporter E1
watchlist WL1 INT Source INT Sink
inband-telemetry queue-profile QP1
depth 1000 Host Host
latency 1000
inband-telemetry flow-profile FP1

dscp 1
age 5
latency quantization 10
inband-telemetry system monitor M1
N3K-C34180YC
Beacon LED 6 QSFP28 40/100GE
Status LED
Environment LED
1 RU
48 x 10/25GE SFP28
Nexus 34180YC Switch Architecture
Fan 1 Chimay CPU Board
Console G1x1PCIe PSU
Fan 2 USB
CPU
Leffe Board
Fan 3 1.8 GHz Intel Xeon
650W
Fan Interface Board PSU
Fan 4
G2x4PCIe G1x1PCIe CTRL/Status
Boddington Board
x4PCIe MI FPGA CPLD

CPLD
CPLD
CPLD
Barefoot Tofino
ETH
ETH 3 X BearValley
6 x QSFP+
48 x SPF+ /QSFP28
10/25G
40/100G
Nexus 34180YC ASIC Port-map
Tofino ETH ports

ETH ETH ETH
2 30 26
ETH ETH ETH ETH ETH ETH ETH ETH ETH ETH ETH ETH
22 23 24 20 18 16 14 12 10 8 7 6 ETH ETH ETH
4 0 28
Pipe 0
3 X BearValley
Pipe 1
1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4
1 3 5 7 9 1 3 5 7 9 1 3 5 7 9 1 3 5 7 9 1 3 5 7 49 51 53
2 4 6 8 1
0
1
2
1
4
1
6
1
8
2
0
2
2
2
4
2
6
2
8
3
0
3
2
3
4
3
6
3
8
4
0
4
2
4
4
4
6
4
8
50 52 54
48 SPF28 10/25Gig 6 QSPF28 40/100Gig
34180YC Front panel ports

Nexus 34180YC Generic Profile
48x25G, 6x100G (Breakout 100/25, 40/10)
L3
L2 •
• Interfaces (L3, SVI (2k), L3 port-channels)
L3 Routing v4/v6 (v4 Host 32k and v4 LPM 4k, v6 hosts
• Interfaces (access, Trunk, Q-in-Q, Port-Channels (128)) 16k, v6 LPM 4k, Next hops 48k (shared) )
• 4k VLAN, STP, Storm Control • ECMP (32-way, 1k groups)
• Unicast Bridging (32k MAC), Multicast forwarding/ IGMP snooping • BGP, OSPF, BFD
• Peer-link less VPC • HSRP, VRRP
• LACP/UDLD • Multicast Routing, PIM-SM, SSM (SG 8k/2k, *.G 4k/1k.
Data plane Telemetry

ACL, QoS • In-band Telemetry (Flow reports, Queue reports, Drop reports) -1k
Flow watchlist, 256 Drop watchlist
• Ingress ACL (MACL, VACL, RACL) (7k)
• SPAN/E Data plane Telemetry RSPAN/Mirroring ( 64 sessions)
• Egress Policing
• Object Models
• Ingress QOS (Classification, Policing, Marking, Shaping, scheduling)
• PTP (Boundary mode)
• CoPP, Custom CoPP
• DHCP v4/v6 Relay
• PFC, LLFC, ECN
N3K-C34180YC L3-Heavy Profile
48x25G, 6x100G (Breakout 100/25, 40/10)
L3
L2 • Interfaces (L3, SVI (2k), L3 port-channels)
• L3 Routing v4 (Host 64K and LPM 64K, next hops 64K)
• Interfaces (access, Trunk, Port-Channels (128))
• ECMP (32-way, 1k groups)
• 4k VLAN, STP, Storm Control
• BGP, OSPF, BFD
• Unicast Bridging (2k MAC)
• HSRP, VRRP
• Peer-link less VPC (FCS-only)
• LACP/UDLD
Data plane Telemetry

ACL, QoS • In-band Telemetry (Flow reports, Queue reports, Drop reports) -
• Ingress ACL (RACL) (1k), (+ Minimum Ingress ACL 1k Flow watchlist, 256 Drop watchlist
support for System ACL) • SPAN/ERSPAN/Mirroring (64 sessions)
• Egress Shaping • Object Models
• QOS (Classification, Policing, Marking, Shaping, scheduling) • PTP (Boundary mode)
• CoPP, Custom CoPP • DHCP v4 Relay
• ECN


• Monticello Architecture
• ASIC Block Diagram
• ASIC Forwarding Paths
• ASIC Forwarding Pipeline
• Normal Mode
Nexus 3500 • Warp Mode
• Monticello Warp Span
• N3K-C3548P-XL Switch
Architecture
• N3K-C3548P-XL ASIC
Port-map
Monticello ASIC Architecture
• 480 Gbps Single Chip Ethernet Switch (MonticelloCR)
• 720 MPPS @ 64 Bytes Decision Engine
• 18 MB of Buffer (3 x 6MB) Admission Control
• UUL 250ns (~200ns in Warp Mode) Output Output Output

Buffer 1 Buffer 2 Buffer 3
ReWrite ReWrite ReWrite
CPU
48 x 10G XFI
PCIe
…………
Monticello ASIC Block Diagram
Ports: Ports:
Output
9-12 9-12
Buffer
x16
….
x16
….
21-24 Parser ReWrite 21-24
0
33-36 33-36
6MB
45-48 45-48
Admission Control
Decision Engine
Ports: Ports:
5-8 Output
5-8
ReWrite
x16
….
Buffer
x16
….
17-20 Parser 17-20
29-32 1
29-32
41-44 6MB
41-44
Ports: Ports:
1-4 Output 1-4
Buffer
x16
….
….
x16 13-16
25-28
Parser
2
ReWrite 13-16
25-28
37-40 6MB
37-40
Monticello ASIC Forwarding Paths
Normal
L2
L3
Classification
ACL
Egress Port
Incoming
Packet
L2 + L3
Warp
Warp Span
Monticello ASIC Forwarding Pipeline
Normal Mode
Lookup Engine
Unicast LPM
Table 24K
Multicast
Parsed/ Route Table L3 UC
Classified 8K ECMP MAC Table Decision To Buffer
Traffic Table 16K 64K Engine
Unicast Host
Table 64K
ACL Table
4K Egress ACL Result
For Egress ACL lookup
Monticello ASIC Forwarding Pipeline
WARP Mode
Lookup Engine
Unicast LPM 4K
Multicast Table
8K
MAC Table 8K
Host Table 8K
Parsed/ L3 UC
Classified Control 4K ECMP MAC Table Decision To Buffer
Traffic Table 16K 64K Engine
Unicast Host
Table 64K
ACL Table Ingress ACL Result

4K
Normal vs. Warp Mode Forwarding
Feature Normal Warp
Latency 250ns 190ns
NAT Yes Yes
Ingress RACL/VACL Yes Yes
Egress RACL/VACL Yes No
Unicast Route 24K 4K
Multicast Route 8K 8K
L3 ECMP Yes No
Monticello Warp Span
Exchange Feed
• WARP SPAN can be enabled both in
normal and WARP mode 1/36
25-28
1-4
• Latency ~50 ns
WARP SPAN source has to be port
29-32
•
5-8
1/36
Destination ports would be group of
33-35
9-12
•
4 ports Shared Buffer
13-16
37-40
17-20
41-44
45-48
21-24
N3K-C3548P-XL
48 1/10GE SPF+
10/100/1000 management port
RS-232 serial console port
Same Physical device as 3524-XL

1 RU
40 GE by combining
four sequential SFP+
Two USB ports
N3K-C3548P-XL Switch Architecture
16GB Bootflash Fan1
400W DIMM
PSU
Fan Fan2
CPU Board
Fan3
400W 2.50GHz GHz Intel
PSU Core(TM) i3-3227U
Fan4
G1x1PCIe
G2xPCIe G1x1PCIe CTRL/Status
PCIe MI FPGA CPLD

CPLD
CPLD
CPLD
Monticello CR
48 x XFI
XFI
24 x XFI to SFI
Port
LEDs
48 x SFI
Management
Console 48 x SPF+ 10G
USB
N3K-C3548P-XL ASIC Port-map
MonticelloCR OB Ports
1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4
2 4 4 6 8 0 2 7 9 1 3 9 1 3 5 7 9 5 7
0 5 7 1 3
1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4
3 5 5 7 9 1 3 8 0 2 4 0 2 4 6 8 0 2 4 6 8
1 6 8
OB 0
OB 1
OB 2
1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4
1 3 5 7 9 3 5 7 9 1 3 9 1 3 5 7 9 5 7
1 5 7 1 3
1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4
2 4 6 8 4 6 8 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8
0 2
48 SPF+ 10Gig


• Jericho+ ASIC Architecture
• ASIC Packet Forwarding
Nexus 3600 • N3K-C3636C-R Switch
Architecture
• N3K-C3636C-R ASIC Port-
map
• Based on 2 Jericho +
36180YC-R • 1.8Tbps
• 8 GB of Buffer
• Based on 4 Jericho +
3636C-R • 3.6Tbps
• 16 GB of Buffer
Jericho+ ASIC Architecture
• BCM88680 from StrataDNX family (BCM88680)
• 900Gbps, 835Mpps 48 x 25G SerDes
• Integrated Forwarding and Fabric 1588 Time stamping &

Clock Recovery Flexible Counters
interface
Packet Processor (PP)
Two packet processing cores (PP)
L2/L3/ MPLS/ VPLS/ L2VPN/ L3VPN/VXLAN
•
Packet Buffer Memory Traffic Manager (TM) On Chip
• 96K Virtual Output Queues DRAM Controller Deep buffered VOQs, hierarchical
Buffer
(16MB)
Scheduling
48 x 25G +24 x 12.5G SerDes
…………
12 x 100GE,
24 x 50GE
30 x 40GE
48 x 25GE
Jericho+ High-Level Forwarding Architecture
On-chip resources Fabric Interface

• 16MB Internal Buffer &TCAM
Ingress Transmit Egress Transmit
• Forwarding Tables Packet Packet
Processing Processing
Expansion via off-chip resources
Ingress Egress
• Deep GDDR5 external packet buffers Packet Buffer Traffic
On Chip
Buffer
On Chip
Buffer Traffic
DRAM Manager Manager
Ingress/Egress Traffic Managers
Ingress Receive Egress Receive
• 96k Virtual Output Queues Packet Forwarding Packet
Processing Table Processing
• WRED, Distributed Arbitration
Network Interface
Jericho+ Buffering
• Nexus N3600-R switches use traditional VoQ architecture
• Big buffer on Ingress side dedicated to VoQ buffer
• 4GB GDDR5 DRAM-based buffering per port-group used for VoQ buffer
• VOQ buffer has dedicated portion per port and shard buffer among ports
in the same port group
• 16MB of On-chip buffer used for egress buffer
N3K-C3636C-R
RS-232 serial console port
8 QSFP28 + MACsec
RS-FEC for 25GE
Lane-selected LED
10/100/1000-Mbps
management port 1 RU
(copper or fiber)
USB Port 36 QSFP28
Each QSFP28 can breakout to 4 x 10 or 4 x 25GE
N3K-C3636C-R Switch Architecture
Redcastle CPU Board
Fan 1 DIMM
SSD Flash
Drive 128GB 400W
PSU
Fan 2 CPU
NVRAM
Broadwell-DE 8 Core,
2MB 2.0GHz 400W
Fan 3 PSU
G1x1PCIe
Redcastle Main Board
Switch
XBAR1 XBAR2
24 x 25G
Jericho+ 1 Jericho+ 2 Jericho+ 3 Jericho+ 4

9x100 9x100 9x100 8x100
4 X BearValley
Management
Console 28 x QSPF28 8 x QSPF28 +
USB MACsec
32 x QSPF28
N3K-C3636C-R ASIC Port-map
Jericho+ Ports
1 3 5 7 9 2 4 6 8 1 3 5 7 9 2 4 6 8
2 4 6 8 1 3 5 7 9 2 4 6 8 1 3 5 7 9
Jericho+ 1
Jericho+ 2
Jericho+ 3 4 X BearValley
Jericho+ 4
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36
C3636C-R Front Panel Ports


Key Takeaways
Key Takeaways
Full-featured DC access Programmable Pipeline

Broad Switch Portfolio Inband Network Telemetry (INT)
High throughput & performance 3100,3200 3400 In-Network DDoS Detection
Flexible connectivity Layer 4 Load Balancer
Cisco Nexus
3000 Series
Switches
Deep Buffer
Ultra-low latency 3500 3600 16 GB of Buffer
Less than 200ns Ingress VoQ
Normal, Warp, Warp SPAN Multiple ASICs
Complete your online session evaluation
Give us your feedback to be entered

into a Daily Survey Drawing.
Complete your session surveys through
the Cisco Live mobile app or on
www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing
on demand after the event at www.CiscoLive.com/Online.
Continue
your Demos in
the Cisco
Walk-in
self-paced
Meet the
engineer
Related
sessions
education campus labs 1:1
meetings
Thank you
#CLUS
#CLUS

BRKDCN 16598639898 PDF

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

BRKDCN 16598639898 PDF

Caricato da

Copyright:

Formati disponibili

#CLUS

Cisco Nexus 3000

Webex Teams will be moderated cs.co/ciscolivebot#BRKDCN-3734

StrataXGS StrataXGS StrataXGS

Nexus 3100 Nexus 3200 Nexus 3400

Nexus 3500 Nexus 3600

• Based on Trident (2+)

• Based on Trident (3)

• 2 Pipes @1.6 Tbps Multistage Content Aware Engine

40G 100G 40G

x4PCIe MI FPGA CPLD

C3132C-Z Front Panel Ports

Nexus 3100 Nexus 3200 Nexus 3400

Nexus 3500 Nexus 3600

Nexus 3200 • ASIC Forwarding Pipeline

• 4 Pipes @1.6 Tbps Multistage Content Aware Engine

• 42MB (4x10.5MB) of Buffer L2 MAC

• Ingress & Egress Packet Time

Intelligent Tunnel Vlan L2 L3

40G 100G 40G

G2x4PCIe G1x1PCIe G1x1PCIe

x4PCIe MIFPGA MIFPGA CPLD0

2 x SFP 48 x QSFP28 16 x QSFP28

C3264C-E Front Panel Ports

CLI commands for Port-mode change: System reload is required

Profile CLI : “hardware profile portmode <config>”

Breakout Capable port

Breakout Capable port

Breakout Capable port

Nexus 3100 Nexus 3200 Nexus 3400

Nexus 3500 Nexus 3600

Nexus 3400 • Match-Action Unit

TCAM Table Memory

• P4-programmable pipeline Match-Action Engine

• Single 20 MB Unified Packet Buffer Packet Buffer

Control and Configurations

40/50/100G Match-Action Buffers Match-Action 40/50/100G

Network Demands Switch OS Network Demands NX-OS

Run Time API Auto Generated API

Fixed-function Switch Programmable Switch

(Match Action Unit 0 ) (Match Action Unit N )

Match + Action Stage (Unit)

Memory ALU Memory ALU Memory ALU

Memory ALU Memory ALU Memory ALU

Memory ALU Memory ALU Memory ALU

PHV PHV’ Common

40/100G & Parser Packet Data

Packet Header Vector

PHV PHV’ 10/

to TM for scheduling and shaping Fields

All fields with holes removed

Ingress Ingress Packet

Ingress Packet Body

Inband Network Telemetry

• In the postcard mode, each network • Metadata is embedded in between

Header Payload Header Payload

Packet Header + Info A Endpoint (Source)

Header Payload Header Switch A info Payload

Header Payload Header Switch A info Switch B info Payload

Packet Header + Info C Packet Header +

• Third Record (INT metadata) will be

• INT Sink will remove INT record and

(5-tuple, Permit/Deny) Monitor Session