1 Asr9k Arc Base

ASR9K Overview
ASR9K Chassis
Architecture
RSP
Fabric
ASR9K Overview
Linecard
Life of a Packet
Power Management
Media and File Systems

Online Diagnostics
Viking Hardware Overview
2008 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
ASR 9000 At a Glance

Optimized for Aggregation
of Dense 10GE and 100GE
Designed for Longevity:

Scalable up to 400 Gbps of
Bandwidth per Slot
Based on IOS-XR & ANA
for Nonstop Availability and
Manageability
Enables Network
Convergence of Business
and Residential Services
Cisco Confidential
ASR 9000
The (R)evolution Begins!
Minimize Operational
Expense
Simplicity
Carrier Ethernet Focus

High Density GE/10G
Extensive L2VPN Services
Converged
Incorporated SynchE
IPoDWDM Ready
Power Reduction
Industry Leading Watts/Gbps
Pay as You Grow Power
Minimal Carbon Footprint
Carrier Class
Resilient
Purpose
Built
Green
Video
Optimized
* Please refer to Road Map
for time line of feature
support
Complete EMS/NMS Support

Consistent Provisioning
Model Comprehensive OAM
Integrated Video
Cisco Confidential
IOS-XR Modular OS
Absolute HW Redundancy
Broad Network Resiliency
Schemes
Built to Last
6.4Tbps Capable System
Engineered for Nx100G
Superior H-QoS/Scale
Video Quality Monitoring

Local Ad-Insertion
Optimal Intelligent Multicast Forwarding
VoD Streaming/Cache
3
Designed with Operators in Mind
Simplicity
Software Optimized for Automation, Ease of Use

Manageable via Cisco ANA Toolkit
Cross-platform support of ASR9k, 7600, CRS-1
Standards-based northbound interface to carrier
OSS
XR image mgmt, device inventory, fault
correlation
Carrier Ethernet end-to-end service provisioning
Embedded ASR9k mgmt with XML over HTTP,
SNMP
Complete OAM Coverage

802.3ah for link level fault detection
802.1ag draft 8.1 compliance
MPLS OAM/VCCV support

EtherSLA/Y.1731 Performance Monitoring
(future)
Cisco Confidential
IOS-XR Powers the Edge with ASR 9000
Resilient
A Fully Distributed, Microkernel-Based Architecture

Modular Components
Routing
Composite
MPLS
Multicast
RIP
BGP
OSPF
ISIS
Applications Architecture
IS-IS
Distributed Middleware
QoS
Manageability
Host
Composite
Security
Forwarding
Base
Multicast
BGP
Distributed Service Separation

OAM
MAC
OAM
MAC
OAM
MAC
Admin
VLAN Subs
Line Card
Next Generation Architecture

Microkernel-based design
Highly modular, highly extensible
Service-enabled blade architecture
LC1
VLAN Subs
LC2
VLAN Subs
LCn
Designed for Scale, HA, and Performance

For IP NGN Applications
Scale through distribution
Unique address tables per linecard
Process-level, stateful subscriber HA
Cisco Confidential
Purpose
Built
Cisco ASR 9000

Architecturally Superior by Design: Chassis & Fabric
Scales to 2x 200G Fabric / Slot
Half Terabit More Dense than Competition
Maximixed Slot Real Estate

Length x Width x Height
Optimized for 200 Gbps
with Daughtercard
Up to 8x NPUs per Card
Now
200G
Card
Power-Up as You Grow

Modular 1500W Power Bricks
For Green Efficiency
Per-slot power & cooling

ready for 200G today
Backplane signal integrity
validated for 10G SerDes
ASR9k: 8 slots * 20x 10GE = 160x 10GE

Box J: 11 slots * 10x 10GE = 110x 10GE
Box A: 10 slots * 10x 10GE = 100x 10GE
500G 600G Density Advantage for ASR9k
Cisco Confidential
ASR 9000 System Scalability

Outlasting the Future
50,000
PB/mo
Mobility
Business Internet
Business IP WAN
Consumer Internet
Consumer IPTV/CATV
25,000
2005 2006 2007 2008 2009 2010 2011 2012
Source: Cisco Visual Networking Index Forecast,

20072012
Dual Fia
10 slots
6 slots
22 Slots
8 LC + 2 RSP
4 LC + 2 RSP
20 LC + 2 RSP
Linecard Density
200
80
Gbps
40 Gbps
200
80
Gbps
40 Gbps
80 Gbps
400 Gbps
Bandwidth per Slot
400 Gbps
Gbps
180
400
180 Gbps
550 Gbps
6.4
2.8 Terabits
3.2
1.4 Terabits
20 Terabits
Linecards per Chassis
Bandwidth per Chassis

Cisco Confidential
Viking History
Viking was initially conceptualized as an L2 aggregation switch
with the Gladiator chassis (with Gracchus and Marcus line cards)
in the Metro Ethernet Market with Goryeo taking the routing role
fitting into the SP edge.
When Goryeo was found to be no longer an advantageous
investment, Viking adapted to SP edge market.
This posed changes both in the chassis requirement when Red
October (with Ryan, Ramius and Mancuso line cards) was
conceptualized and also in the multitude of feature and scale
requirements.
Cisco Confidential
ASR9K Overview
ASR9K Chassis
Architecture
RSP
Fabric
ASR9K Chassis
Linecard
Life of a Packet
Power Management

Platform Fault Management (PFM)
Online Diagnostics
Service Console
Cisco Confidential
Chassis Overview
ASR-9000 10-slot system
10 slots: 8x linecards + 2x RSP
Half-rack: 17.38w x 36.75h x 28d
Bandwidth (initial)
400Gbps backplane
180Gbps fabric 400G
40G/80G linecards N-100G
Carrier-class hardware redundancy

AC & DC systems
Pay-as-you-grow, modular power
Green emphasis throughout
Cisco Confidential
10
Hardware Subsystem: 10-slot Power

Viking Power Details
750W power & cooling available per LC slot
FCS cards draw significantly less (~500W)
Max system draw 8.5KW, max FCS draw is <= 5KW

AC power supplies are 3kW, DC are 2kW or 1.5kW
Max power configs : 5+1 DC (or) 3+3 AC
FCS power configs: 3+1 DC (or) 2+2 AC
Cisco Confidential
11
Hardware Subsystem: 10-slot power

Power Distribution (DC N:1 protection)
Shelf 0 (Top)
LC
Feed A
Feed A
Feed B
Feed A
Feed B
Feed A
Feed B
Feed A
Feed B
Feed A
Feed B
PS 0
PS 1
PS 2
PS 3
PS 4
PS 5
Power Distribution Bus
Feed B
LC
Single power zone, one

distribution bus
LC
All modules load share
RSP
RSP
LC
LC
Fans
2kW and 1.5kW supplies

Each power supply is
wired to both A and B
feed
Feed failure doubles draw
on remaining feed
supply failure increases
draw on remaining
supplies
Fans
Shelf 1 (bottom)
Cisco Confidential
12

Power Distribution (AC 1:1 protection)
Shelf 0 (Top)
LC
Feed A
Feed A
Feed B
Feed B
Feed B
PS 1
PS 2
PS 3
PS 4
PS 5
Feed A
PS 0
LC
LC

distribution bus
RSP
AC power supplies are

rates @ 3KW ea.
RSP
A feed wired to top power

shelf
LC
LC
B feed wired to bottom

power shelf
Fans
Fans
Shelf 1 (bottom)
Cisco Confidential
13
10-slot chassis: 6kW redundant commons

AC 1:1 protection vs. DC N:1 protection
Shelf 0 (Top)
Feed A
Feed B
AC PS
3kW
AC PS
3kW
AC PS
3kW
Shelf 1 (bottom)
S
Y
S
T
E
M
L
O
A
D
For 6KW AC, you need 4x 3KW power

supplies (feed failure takes down
two of them, supply failure kills one)
Feed A & B
DC PS
2kW
Feed A & B
DC PS
2kW
Feed A & B
DC PS
2kW
Feed A & B
DC PS
2kW
Shelf 1 (bottom)
Feed B
AC PS
3kW
Feed A
Shelf 0 (Top)
S
Y
S
T
E
M
L
O
A
D
For 6KW DC, 4x 2KW power supplies

(2kw each, feed failure has no
impact, so we protect only against a
supply failure)
Cisco Confidential
14
Hardware Subsystem: 10-slot thermals

Viking Cooling Details
Front-to-back airflow
750W cooling per LC slot
FCS cards significantly less
Dual stacked fan trays below

LCs
Variable speed fans
at normal ambient temperatures
lower power draw
lower noise profile
Cisco Confidential
15
Chassis Overview
ASR-9000 6-slot system
6 slots: 4x linecards + 2x RSP
rack: 17.38w x 17.35h x 28d
Bandwidth
400Gbps backplane
180Gbps fabric 400G
40G/80G linecards Nx100G
Carrier-class hardware redundancy

AC & DC systems
Pay-as-you-grow, modular power
Green emphasis throughout
Cisco Confidential
16
Hardware Subsystem: 6-slot

Viking Power & Cooling Details
side-to-back airflow (no stage
heating!)
750W cooling per LC slot
Dual vertical fan trays above card
cage
Tray power draw from 40W-300W
Temperature sensitive green
engineering
FCS draw is significantly less than

750W
Variable speed fans
at normal ambient temperatures
lower power draw, lower noise
profile
Cisco Confidential
17

Power Distribution (AC 1:1 protection)
Feed A
PS 0
Feed B
PS 1
Feed C
PS 2
Power Entry Shelf
LC

distribution bus, all
modules load share
LC
AC power supplies are

rated @ 3KW ea.
LC
Fully populated 40G

system requires <3kW
LC
RSP
Third source required for

redundancy for >3KW
installations.
RSP
Fans
Fans
Cisco Confidential
18

Power Distribution (DC N:1 protection)
Feed A
Feed B
Feed A
Feed B
Feed A
Feed B
PS 0
PS 1
PS 2
Power Entry Shelf
LC
LC

distribution bus
LC
LC
RSP
2kW and 1.5kW supplies

Each power supply is
wired to both A and B
feed
RSP
Fans
Fans
Cisco Confidential
19
ASR9K Power Calculator:

http://wwwin.cisco.com/sptg/crbu/products/asr9000/files/ASR_9000_Power_Model.xls
computed power values
per-component values
9006 FanTray
9010 FanTray
RSP
4xNP linecard
8xNP linecard
TOTALS
Qty
0
2
2
4
4
Typical
(25C)
Max (40C)
Ext. Temp
(55C)
Typical
(25C)
Max
(40C)
Ext. Temp
(55C)
100
275
375
200
300
600
400
600
1200
175
205
235
350
410
470
310
320
350
1240
1280
1400
565
575
630
2260
2300
2520
4250
4590
5590
To use: select 2 fan trays based on the chassis type. Enter the number of cards and RSPs
configured.
Max power values are tested values with full wire-rate traffic running on all ports, and worst-case
feature sets.
Note that it's the NUMBER OF NPs that matters, NOT the number of ports.
Specific example: the 8-port card with 4 NPs uses far far less power than the 8-port card with
8xNPs
Deployed "real world" systems are likely (but not guaranteed) to draw 10-33% less power than these numbers.
Cisco Confidential
20
ASR9K Overview
ASR9K Chassis
Architecture
RSP
ASR9K System
Architecture
Fabric
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
21
Viking Architecture : Summary

Carrier Class Design
High Availability integral to system architecture
Not an afterthought
Modularized System Components HW and SW

Isolate failure and faults to subsystem/component
NPU
NPU
NPU CPU
MAC
MAC
CPU
HW based signaling for fabric failover
MAC
BITS/DTI
FIC
NPUNPU
MAC
FIC
NPU
CPU
BITS/DTI
FIC
NPU
NPU
FIC
~ Zero packet loss on switchover
Redundancy built-in throughout system
NPU
NPU
CPU
MAC
MAC
NPU
RSP, Fabric, Control Plane Ethernet, Chassis Control

Bus, Power supplies, Fan Trays
MAC
FIC
NPU
FIC
NPU
MAC
NPU
Backplane Point-to-Point connectivity oriented

Limit single point of failures
Isolate faults
Highly Scalable System and dense QoS

Synchronous Ethernet Support
BITS/Network Timing Distribution
Cisco Confidential
22
ASR9K SW Architecture Highlights

Scalable Forwarding Plane
Ethernet centric, multilayer switch design
Fully distributed packet processing
Two stage forwarding model
IOS XR Based
Sharing same XR release and L3 feature set with CRS-1
Carrier Class infrastructure
Offering S/W modularity, HA, Restartability, Fault Containment, NSF/SSO,
SMU, ISSU
Focusing on Carrier Ethernet features

VPLS, EoMPLS, Bridge Domain, EFP, MSTP, Link Bundling, E-OAM, QinQ,
IGMP Snooping
High throughput traffic management support
Hierarchical QoS, CBWFQ, WRED, 2R3C Policing
Security, Protection Features
L2,/L3 Security ACLs, CoPP, IP SLA, Mac Limiting, BPDU Filtering
Cisco Confidential
23
ASR9K Platform basic blocks
Switch Fabric
Switch
Fabric
Route
Processor
Route Processor
Distributed
Forwarding Line
Card
Distributed
Forwarding Line
Card
Distributed
Forwarding Line
Card
Line Cards carry forwarding

engines, control plane processing
and integral I/O ports
Switch Fabric and Route Processor

are combined on one RSP module.
Redundant Route-Switch
Processors provide central point of
control for the router.
RSP also provides shared
resources for backplane Ethernet,
timing and chassis control
Cisco Confidential
24
Viking System Block Diagram

RSP0
Octopus
CPU
8641D
Punt Path
RSP1
Punt Path
System
Timing
Octopus
Legend
CPU
8641D
Punt Path
Timing Plane
Data Plane
Punt Path
2 Santa
Cruz
GE
Switch
Bellagio 2
System
Timing
2 Santa
Cruz
Arbitration Plane
GE
Switch
Bellagio 2
Control Plane
Punt Path
Backplane
10Px
SFP
10Px
SFP
10Px
SFP
NP3
NP3
NP3
10G PHY SFP
10Px
SFP
NP3
10G PHY SFP
NP3
10G PHY SFP
NP3
10G PHY SFP
NP3
4x10GE
Mancuso
LC
GE
PHY
Octopus
CPU
8641
BRG
BRG
10G PHY SFP
NP3
CPU
8641
BRG
BRG
GE
PHY
Octopus
10G PHY SFP
BRG
CPU
8641
8x10GE
Ramius
LC
10G PHY SFP
GE
PHY
Octopus
10G PHY SFP
40x1GE
Ryan LC
BRG
NP3
NP3
NP3
NP3
10G
PHY
XFP
10G
PHY
XFP
10G
PHY
XFP
10G
PHY
XFP
Cisco Confidential
20x1GE
Thor LC
GE
PHY
Octopus
CPU
8641
SERP
SERP
CPP
CPP
PLIM ASIC
10Px
SFP
10Px
SFP
25
Viking Logical Architecture Overview

RSP 0
C PU
8 64 1 D
Oct opus
RSP 1
Legend
C PU
8 641 D
Oc topus
Data Plane
P unt Path
Sy s t e m
Timing
2 Santa
C ruz
GE
Switc h
B ella gio 2
Sy s te m
T im ing
2 Sa nta
Cr uz
GE
Switc h
B e llagio 2
Control Plane
Bac kp l an e
40 x 1 GE
Ryan LC
FPGA
O ctopus
GE
PHY
8 x 10 GE
CPU
8641
LC
FPGA
FPGA
Trident Trident Trident Trident
Octopus
Ramius
GE
PHY
4 x 10 GE
CPU
8641
LC
O ctopus
Mancuso
20 x 1 GE
GE
PHY
Octopus
Thor LC
CPU
8641
FPGA
FPGA
Trident Trident Trident Trident
GE
PHY
Trident Trident
FPGA
CPU
8641
FPGA
Trident Trident
FPGA
CPP
CPP
Cisco Confidential
10GE XFP
10GE XFP
10GE XFP
10GE XFP
10GE XFP
10GE XFP
10GE XFP
10GE XFP
10 x
SFP
10GE XFP
10 x
SFP
10GE XFP
10 x
SFP
10GE XFP
10 x
SFP
10GE XFP
PLIM ASIC
SPA
SPA
SPA
SPA
26
ASR9K Overview
ASR9K Chassis
Architecture
RSP
ASR9K RSP
(Route Switch Processor)
Fabric
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
27
ASR 9K RSP (Route/Switch Processors )

Overview
RSP2
RSP440
Processors
2 x 1.5GHz Freescale
8641D CPU
Intel x86 Jasper Forest 4

Core 2.27 GHz
RAM (user
expandable)
4GB @133MHz SDR
6GB (RSP440-TR) and

12GB (RSP440-SE)
version @1066MHz DDR3
Cache
L1: 32KB
L2: 1MB
L1: 32KB per Core

L2: 8MB shared
Primary persistent
storage
4GB
16GB - SDD
Secondary persistent
storage (HD/SSD)
30GB - HDD
16GB - SDD
USB 2.0 port
No
Yes
nV Cluster EOBC
ports
No
Yes, 2 x 1G/10G SFP+
Switch fabric
bandwidth
184G/slot (with dual

RSP)
440G/slot (with dual RSP)
8GB
RSP440
Cisco Confidential
28
Hardware Subsystem: RSP

Route/Switch Processor Details
High performance control plane processor
Redundant RP config
Active/Active fabric model

External interfaces
Console/AUX
2x 10/100/1000 Ethernet
Compact Flash
USB
BITS/DTI
Cisco Confidential
29

Main Board Components
CPU: FreeScale 8641D
Memory: 4GB
Hard Drive: 40G HDD

I/O controller: Long Beach
Fabric Interface: Octopus
Fabric Arbitration: Bellagio

Fabric Data path: Santa Cruz
Cisco Confidential
30

Main Board Components
BITS
Clock
Time
FPGA
Timing Domain
4GB MEM
HDD
CF card
Mgt Eth
CPU
8641D
Mgt Eth
Ether
Switch
Console
Aux
4G CF
Punt
FPGA
Long Beach
I/O FPGA
Front Panel
Arbitration
Bellagio
Santa
Cruz 1
Alarm
NVRAM
Octopus
EOBC/
Control Plane GE
Santa
Cruz 0
Boot Flash
CPU Complex
Fabric Interface
Cisco Confidential
31
ASR9K Overview
ASR9K Chassis
Architecture
RSP
Fabric
ASR9K Fabric
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
32
Hardware Subsystem: Fabric

Fabric Architecture
Fabric is logically separate from LC/RSP
Physically resides on RSP
Similar to 7600, not like CRS/GSR
RSP requests access to fabric just like LCs do
Separate data and arbitration paths
Each LC/RSP has a fabric interface ASIC

1x Octopus 40G usable ethernet connectivity
2x Octopus 80G usable ethernet connectivity
Next-gen fabric interface chips in development
Cisco Confidential
33

Fabric Architecture (single RSP shown)
Octopus
(LC)
Octopus
(LC)
Santa Cruz 0
Octopus
(LC)
Octopus
(LC)
Santa Cruz 1
Octopus
(LC)
Octopus
(LC)
Data
Bellagio
?
Octopus
(RSP)
Octopus
(RSP)
Arbitration
23G fabric channels

arbitration grant/request signals
Cisco Confidential
34

Fabric Architecture
Santa
Cruz 0
Santa
Cruz 1
Bellagio
Octopus
(LC)
RSP0
Octopus
(LC)
Santa
Cruz 0
Santa
Cruz 1
Bellagio
RSP1
23G fabric channels

fabric data is active/active, arbiter is active/standby
Cisco Confidential
35

Scaling Fabric architecture for 80Gbps linecard
Santa
Cruz 0
Santa
Cruz 1
Bellagio
Octopus
RSP0
Santa
Cruz 0
Octopus
Octopus
Santa
Cruz 1
Single-Octopus
40G Linecard
Dual-Octopus
80G Linecard
Bellagio
RSP1
Linecards are forwards/backwards compatible: fabric and backplane designed for 80G today
No changes to existing architecture, arbitration scheme stays the same
FCS fabric has enough switch fabric ports to support 80G/Linecard
Fabric and Backplane designed to support two Fabric Interface Devices seamlessly
Cisco Confidential
36
Hardware Subsystem: Fabric loadsharing

Unicast loadsharing
Santa
Cruz 0
Santa
Cruz 1
Bellagio
Octopus
(unicast fabric plane)
Octopus
Santa
Cruz 0
Santa
Cruz 1
Bellagio
(unicast fabric plane)
Unicast traffic is sent across first available fabric link to destination (maximizes efficiency)
Each frame (or superframe) contains sequencing information
All destination octopus have re-sequencing logic
Additional re-sequencing latency is measured in nanoseconds
Cisco Confidential
37
Hardware Subsystem: Fabric loadsharing

Multicast loadsharing
Santa
Cruz 0
Santa
Cruz 1
Bellagio
Octopus
Octopus
(multicast fabric plane)
C1
B2
A3
B1
A2
A1
Flows exit in-order

Santa
Cruz 0
Santa
Cruz 1
Bellagio
(multicast fabric plane)
Multicast traffic is hashed based on (S,G) info to maintain flow integrity

Very large set of multicast destinations preclude resequencing
Multicast traffic is non arbitrated sent across a different fabric plane
Ratio of unicast credit to multicast credit is TBD
Cisco Confidential
38

Active/Active Fabric Details
Each LC connects to and uses BOTH fabrics
One RSP provides enough bandwidth for 80G/slot
HW-based failure detection

very fast RSP failure notification (better MTTR)
Unicast traffic is loadbalanced across active links
Additional RSP provides more speedup, better hashing

No such thing as a single-RSP system (not sold)
Always always always test, sell, and compete with both
RSPs installed and running!
Cisco Confidential
39

Fabric Wiring & Scaling Details
10-slot chassis has 8 LC slots + 2 RSP slots
2x Santa Cruz fabric chips per RSP
Santa Cruz has 20 switch ports @ 20Gbps each

Each LC wired for two ports per Santa Cruz
40G linecards use one port for each Santa Cruz
80G linecards use two Santa Cruz ports each
6-slot chassis has several unwired ports
Cisco Confidential
40

Fully Populated 40G/slot fabric subsystem: 10x octopus
Octopus
(LC)
Octopus
(LC)
Santa Cruz 0
Octopus
(LC)
Santa Cruz 1
Octopus
(LC)
RSP0
Octopus
(LC)
Octopus
(LC)
Santa Cruz 0
Octopus
(LC)
Octopus
(LC)
Santa Cruz 1
Octopus
(RSP)
Octopus
(RSP)
RSP1
(bellagio & arbitration links are not shown)

Cisco Confidential
41

Fully Populated 80G/slot fabric subsystem: 18x octopus
80G
Octopus
Octopus
Octopus
Octopus
80G
Octopus
Octopus
80G
80G
Santa Cruz 0
80G
Octopus
Octopus
Santa Cruz 1
RSP0
80G
Octopus
Octopus
Octopus
Octopus
80G
Octopus
Octopus
Santa Cruz 0
Octopus
Octopus
80G
Santa Cruz 1
40G
Octopus
(RSP)
RSP1
Octopus
(RSP)
40G
(bellagio & arbitration links are not shown)

Cisco Confidential
42

Arbitration & Superframing Mechanism
20-port 20G crossbar switch w/ external arbitration
Scheduler granularity is per 10G (or 10x1) block
Ingress LC sends request for output LC
Arbiter replies with credit for egress LC
Egress returns credit to LC when packet arrives
Arbitration not strictly required, but ...

Significantly improves switch efficiency
Intelligent traffic cop
Maintains state of available credit for each destination port
When no credit we dont schedule for that port
Cisco Confidential
43

Arbitration & Superframing Mechanism
Superframing significantly improves total throughput
multiple unicast frames from/to same destinations
Packet 3
Packet 1
No superframing (1)
Packet 2
Packet 1
Min reached (2)
Packet 2
Packet 1
Max reached (3)

Jumbo (1)
Packet 1
Max
MTU
Max
Superframe
Min Sufficient
for Superframe
Cisco Confidential
0 (Empty)
44
Hardware Subsystem: Deep Fabric Math

Counting the Bits
each octopus acts and is wired to one 20G fabric

channels on each santa cruz
raw bandwidth per 20G fabric channel in the santa cruz
fabric chip is:
8 parallel serial lines @ 3.125Mhz each, using 24/26 encoding.
8 * 3.125 * (24 / 26) 23.077 Gbps ~~> 23Gbps
there are two santa cruz chips per RSP:

two santa cruz per RSP: 23 * 2 == 46Gbps per RSP
there are two RSPs per chassis:

46Gbps/RSP * 2 92Gbps per octopus (in active/active mode)
Cisco Confidential
45
Superframes
With 32 byte DC3 hdr + FPOE + error control + 8-bye alignment overhead, smaller pkts are
inefficient in the fabric. So unicast packets are grouped into Superframes < 2K bytes. Since there
could be huge combination of multicast destinations, multicast packets are neither put into
Superframes nor are arbitrated through the VOQ mechanism.
8 bytes
1st packet DC3 fabric
header inclusive
Min. 1068
bytes to
guarantee
line rate
perfmnce
3 FPOE bytes
(interpreted
by fabric)
Word0
Word1
Word2
2nd packet DC3 fabric header

inclusive
3rd packet DC3 fabric header

inclusive
Sequencing
information in
32-byte DC3
header
Nth packet
4 byte CRC covers
entire frame.
Pad with zero to align to 8 bytes
Cisco Confidential
46
Load Balancing from Active/Active
0
1
14
15
SCZ
Octo
To-Fab
Octo From- Octo FromFab

Fab
0
1
14
14
15
15
RSP0
0
1
15
First Line Card / Slot 0

Egress Path
0
1
14
14
15
15
RSP1
0
1
14
15
SCZ
Octo
To-Fab
Octo From- Octo FromFab

Fab
Octo
To-Fab
SCZ
0
1
Last Line Card / Slot 9

Ingress Path
SCZ
Octo
To-Fab
First Line Card / Slot 0
Ingress Path
0
1
0
1
14
15
Last Line Card / Slot 9

Egress Path
Parallel Paths in Redundant, Eight Line Card Chassis
Cisco Confidential
48
ASR9K Overview
ASR9K Chassis
Architecture
RSP
Fabric
ASR9K Linecards
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
49
ASR 9000 Series Line Cards

Integrated L2 / L3 Services
Dense Hierarchical QoS
High Density Service Scale
Distributed High Performance Control Plane

Modular Operating System - IOS-XR
Distributed / Flexible Forwarding Plane
Integrated Timing Support
Integrated Media Monitoring
Distributed OA&M Processing
Distributed Netflow
Online Insertion & Removal
40G Line Rate Performance
Pluggable Optical / Electrical Interfaces
Feature
Cisco Confidential
A9K-40GE-B
40 Port GE SFP Line Card
A9K-4T-B
4 Port 10GE XFP Line Card
A9K-4T/8-B
8 Port 10GE XFP Line Card
support may be software release dependent
50
Cisco Confidential
51
L/B/E Line Cards
Whats the Difference?

NP complex
FIB
STATS MEMORY
MAC
LOOKUP
MEMORY
Network Process Unit
FRAME MEMORY
TCAM
Each NPU has Four Main memories:

Lookup/Search Memory (RLDRAM): stores MAC, FIB, and Adjacencies Tables
TCAM: classification (Vlan Tag (EVCs), QoS and Security ACL
Stats QDR memory: interface and forwarding statistics, policers data, etc
Frame memory: buffer memory for Queues
3 LC versions low, base and extended - differ for size of memories
TCAM, QDR and Frame memory sizes depend on LC version
Affects number of QoS queues and L2 sub-interfaces supported
Search Memory is same
System level scale (unicast, multicast, MPLS label) adjacency and MAC
address) not affected by a mix of LCs
Cisco Confidential
52
ASR 9000 Ethernet Linecards
Low Queue
Medium Queue
High Queue
512K
512K
512K
1M/128K
1M/128K
1M/128K
32K
32K
32K
512K/128K
512K/128K
512K/128K
VRFs
4k
4k
4k
L3 Subif/Port
4k
4k
4k
Bridge Domains
8k
8k
8k
128k
128k
128k
4k
16k
32k
Queues
8/port
64k/32k
256k/128k
Policers
8k
128k
256k
MAC Addresses
IPv4 Routes (total/per VRF)
ARP entries
IPv6 Routes (total/per VRF)
MPLS Labels
EFPs (L2 sub-int)
Cisco Confidential
Different
Metric
Common
Capability Comparison
53
How to Choose the Right Linecard ?
Cisco Confidential
54
Typhoon LC
Part Number
Target
Release
A9K-RSP440-SE
4.2.0
A9K-RSP440-TR
4.2.0
A9K-MOD80-SE
4.2.0
A9K-MOD80-TR
4.2.0
A9K-MOD160-SE
4.2.1
Part Number
Target
Release
A9K-24x10GE-SE
4.2.0
A9K-24x10GE-TR
4.2.0
A9K-2x100GE-SE
4.2.0
A9K-2x100GE-TR
4.2.0
A9K-1x100GE-SE
4.2.2
A9K-MOD160-TR
4.2.1
A9K-1x100GE-TR
4.2.2
A9K-MPA-2x10GE
4.2.1
A9K-36x10GE-SE
4.2.2
A9K-MPA-4x10GE
4.2.0
A9K-36x10GE-TR
4.2.2
A9K-MPA-8x10GE
4.3.1
A9K-MPA-20x1GE
4.2.0
A9K-MPA-1x40GE
4.2.3
A9K-MPA-2x40GE
4.2.1
Cisco Confidential
55
Netflow Support
Cisco Confidential
56
Licensable Features
L3 VPN Service Line Card License
Enables Provisioning of L3VPN Service Interfaces on the LC
Infrastructure VRF Line Card License

Enables Provisioning of L3 Interfaces in up to 8 VRFs on the LC
Media Monitoring System License

Enabled Provisioning of Media Monitoring Points on any LC
Feature
Cisco Confidential
57
Do I need a License ?
Cisco Confidential
58
Cisco Confidential
59
Linecards
Codename
BW
Engine
Ryan
40 x GE
40G
4 x Trident NPU
Ramius
8 x 10GE
40G
4 x Trident NPU
Mancuso
4 x 10GE
40G
4 x Trident NPU
Thor
4 x SPA
20G
2 x CPP10 (Popeye)
Son-of-Thor
4 x SPA
40G
2 x CPP10 (Olive)
Odin
8 x 10GE
80G
8 x Trident NPU
Valhalla
16x10GE
80G
8 x Trident NPU
Comments
2:1 oversubscribed
RED OCTOBER
I/O
2:1 oversubscribed
FCS switch fabric provides ~80Gbps per LC slot
Roadmap to 200 Gbps with future switching silicon

Cisco Confidential
60
Hardware Subsystem: Red October Linecard

Linecard Details
40Gbps linecard
Scalable architecture
-4xTen, 40xGE, 2 plus 20
-80G linecard in 2H CY09
Carrier Ethernet/Transport focus

Flexible, microcoded architecture
Base & expanded memory options
additional NP memory -> more scale
Advanced IP software licence

better L3 routing scale, adv. features
Cisco Confidential
61

Red October Board Components
Physical interfaces:
8xTenGE, 4xTenGE, 40xGE, 2+20
Forwarding/Queueing
Trident Network Processor
Header translation and interconnect

Bridge
Fabric Interface
Octopus
CPU: 8641D w/ 2GB memory

Internal GE for chip programming
Cisco Confidential
62
Hardware Subsystem: Red October LC

8x10GE 2:1 oversubscribed Linecard : Ramius
2GB flash
2GB memory
CPU
8641D
Trident Memory
XFP
3
10GE PHY
XFP
7
10GE PHY
Trident 0
Trident Memory
XFP
2
10GE PHY
XFP
6
10GE PHY
Bridge
FPGA 0
Trident 1
Trident Memory
XFP
1
10GE PHY
XFP
5
10GE PHY
gigE
control
network
Octopus
RSP 0
Trident 2
Trident Memory
XFP
0
10GE PHY
XFP
4
10GE PHY
Bridge
FPGA 1
Trident 3
Network
Clocking
daughtercard
Cisco Confidential
RSP 1
via backplane
63

4x10GE linerate Linecard: Mancuso
2GB flash
2GB memory
CPU
8641D
Trident Memory
XFP
10GE PHY
Trident 0
Trident Memory
XFP
10GE PHY
Trident 1
Trident Memory
XFP
10GE PHY
10GE PHY
gigE
control
network
RSP 0
Bridge
FPGA 1
Trident 3
Network
Clocking
daughtercard
Octopus
Trident 2
Trident Memory
XFP
Bridge
FPGA 0
RSP 1
via backplane
Cisco Confidential
64

40xGE linerate Linecard: Ryan
2GB flash
2GB memory
Trident Memory
10x
GE SFP
Trident 0
Trident Memory
10x
GE SFP
gigE
control
network
RSP 0
Bridge
FPGA 1
Trident 3
Network
Clocking
daughtercard
Octopus
Trident 2
Trident Memory
10x
GE SFP
Bridge
FPGA 0
Trident 1
Trident Memory
10x
GE SFP
CPU
8641D
Cisco Confidential
RSP 1
via backplane
65
Hardware Subsystem
8x10GE line rate Linecard : Odin
CPU memory
2GB flash
CPU
8641D
Trident Memory
XFP
10GE PHY
XFP
10GE PHY
Trident 0
Trident Memory
Trident 1
Trident Memory
XFP
10GE PHY
Trident 2
Raven
FPGA 0
Octopus 0
Trident Memory
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
Trident 3
Trident Memory
Trident 4
Trident Memory
10GE PHY
Octopus 1
Trident 5
Trident Memory
XFP
RSP 0
gigE
control
network
Trident 6
Raven
FPGA 1
RSP 1
Trident Memory
XFP
10GE PHY
Trident 7
Network
Clocking
Cisco Confidential
via backplane
66
Hardware Subsystem
16x10GE 2:1 oversubscribed Linecard : Valhalla
CPU memory
2GB flash
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
Trident Memory
Trident 0
Trident Memory
Trident 1
Trident Memory
XFP
10GE PHY
XFP
10GE PHY
Trident Memory
XFP
10GE PHY
Trident 3
Trident 2
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
Trident 5
XFP
10GE PHY
Trident Memory
XFP
10GE PHY
Trident 6
XFP
10GE PHY
Trident Memory
XFP
10GE PHY
Trident 7
CPU
8641D
Trident Memory
Raven
FPGA 0
Octopus 0
RSP 0
gigE
control
network
Trident 4
Trident Memory
Octopus 1
Raven
FPGA 1
RSP 1
Network
Clocking
Cisco Confidential
via backplane
67
Hardware Subsystem
2x10GE + 20xGE linerate : JetFire

2GB flash
CPU memory
CPU
8641D
Trident Memory
XFP
10GE PHY
Trident 0
Trident Memory
XFP
10GE PHY
Trident 1
Trident Memory
10x
GE SFP
gigE
control
network
RSP 0
Bridge
FPGA 1
Trident 3
Network
Clocking
daughtercard
Octopus
Trident 2
Trident Memory
10x
GE SFP
Bridge
FPGA 0
RSP 1
via backplane
Cisco Confidential
68

Synchronous Ethernet support on existing HW
RSP has BITS input, DTI and
centralized Clock Distribution hardware
Full support for L1 Sync-E on linecards
(XR 3.9)
2GB
flash
2GB memory
CPU
8641D
Trident Memory
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
Trident 0
Trident Memory
Bridge
FPGA 0
Trident 1
Trident Memory
gigE
control
network
Octopus
RSP 0
Trident 2
XFP
10GE PHY
XFP
10GE PHY
XFP
10GE PHY
Trident Memory
Bridge
FPGA 1
Trident 3
daughtercard
Network
Clocking
RSP 1
via backplane
Flexible time sourcing: Line cards

capable of recovering clock and
sending to RSP and receiving Transmit
clock from RSP
Frequency synchronization support via
SPAs also (CEoPS, Metronome ..)
Future Hardware capable of
IEEE1588-v2
Cisco Confidential
69
Hardware Subsystem: Trident Fwd Engine

Trident: 10Gbps forwarding/feature engine
10Gbps feature rich ethernet forwarding ASIC
Utilizes multiple cores for packet processing
(Similar to CPP/QuantumFlow Processor)
Multi-stage microcoded architecture

Integrated Traffic managers, Ethernet MACs
Multiple external memories, amount determines scale
Classification TCAMs, MAC/IP lookups,
Frame Buffers, Counters/stats
Cisco Confidential
70
Trident NPU Block Diagram

Frame
Memory
DRAM
333MHz
DDR2/
RLDRAM2
TM Control
Memory
DRAM
333MHz
DDR2/
RLDRAM2
Lookup
Tables
DRAM
333MHz
DDR2/
RLDRAM2
Counters
Statistics
Lookups
SRAM
(36x1)
300MHz
QDR
(72x1)
300MHz
Fabric Side
15 Gbit/s XAUI+
Micro-Engine
Pipeline
TMb
TMc
Cisco Confidential
Fabric Interface
TMa
Six-lane XAUI / 15 Gbit/sec
Ethernet Side
GigE SGMII MACs (or)
XAUI 10G MACs
10 x GigE -or- 2 x 10GigE

Ethernet Interfaces
Trident NPU Memory Interfaces
Host PCIExpress
Optional
TCAM72
Host
SGMII
(punt path)
71
Trident NPU Architecture Highlights

Loopback interface
High-learn I/f
From
Link/
SF/
Host
Frame
Memory
TOP
TOP
Search I
parse
keys
256KB
TOP
resolve
TOP
Search II
search
results
TOP
modify
TM
TM
TM
To
Link/
SF/
Host
search
results
Embedded search
Memory
256KB
Frame
Memory
External Search
Memory( RLDII, TCAM)
Control
Memory
Statistics
Memory
Trident NPU
Single Chip, Programmable, up to 12Gbps w/ 64B packets, full duplex, wire speed network
processor (up to 12 Gbps ingress plus 12 Gbps egress per chip)
Traffic Manager
Support WRED, total 32K queues (8 queues per subscriber x 4K subscribers) per TM
Up to 5 levels of traffic shaping hierarchy
Programming
Model
Pipeline programming model w/o parallel programming or multi-threading. Parallelization done

by HW
Programming
Language
CISC-like assembly instruction set with powerful bit manipulation instructions.
Macro and Assembly programming.
HW Assistance
QoS (Traffic Manager)
Classification / Searching
Counter Updates for statistics maintenance
Cisco Confidential
72
Hardware: Trident Network Processor

10Gbps+ full duplex forwarding/feature engine
High Speed on-chip memory
Ingress
BQS
engine
Interface
MAC/PHY
Egress
BQS
engine
64k
queues
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
PPE
32k
queues
switch
fabric
interface
RSP
External Memory Interfaces
RSP
Multicore/Multistage processing complex
fabric
BQS : buffering / queueing / scheduling

Cisco Confidential
73
Hardware Subsystem: Trident Fwd Engine

Trident: TCAM partitioning
256k entry TCAM on each NP, divided into tables:
96k entries @ 144b key size (QoS, ACL, iFIB)
16k entries @ 576b key size (QoS, ACL, iFIB)
96k for EFP classification

specific application determines which key size used
applications share space WITHIN a key-size table

i.e. no hard-partitioning between QoS/ACL/iFIBetc.
ingress & egress features share same TCAM space
Cisco Confidential
74
show controller np all summary

RP/0/RSP1/CPU0:ASR9010-A#sh controller np summ all
Mon Apr
6 10:31:21.480 UTC
Node: 0/0/CPU0:
---------------------------------------------------------------[total 4 NP] Driver - Version 10.26a Build 1010 ( Jan 28 2009, 12:00:00 )
NP 0 : Hardware Revision: v2 A1
:
NP 1 :
Ucode - Version: 255.255 Build Date: ( Mar 19 2009, 21:56:00 )

Hardware Revision: v2 A1
:
NP 2 :

:
NP 3 :

Node: 0/5/CPU0:
---------------------------------------------------------------[total 4 NP] Driver - Version 10.26a Build 1010 ( Jan 28 2009, 12:00:00 )
NP 0 : Hardware Revision: v2 A1
:
NP 1 :

:
NP 2 :

:
NP 3 :

Cisco Confidential
75
show controller np counter np0 loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller np counters np0 loc 0/0/cpu0
Mon Apr
6 10:34:11.161 UTC
Node: 0/0/CPU0:
---------------------------------------------------------------Show global stats counters for NP0, revision v3

Read 7 non-zero NP counters:
Offset
Counter
FrameValue
------------------------------------------------------------------------------25
PARSE_FABRIC_RECEIVE_CNT
53
XAUI_TRAINING_PKT_DISCARD
57
RESOLVE_INGRESS_L2_PUNT_CNT
85
DIAGS
19601
1093
29294
9786
197
PUNT_STATISTICS
447536
199
PUNT_DIAGS_RSP_ACT
9789
201
PUNT_DIAGS_RSP_STBY
9719
For detail explanation on what these counters mean, please look at: asr9k_ucode-cntrs.doc ( EDCS-778662)
Cisco Confidential
76
show controller np ports all loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller np ports all loc 0/0/cpu0
Mon Apr 6 10:44:41.289 UTC
Node: 0/0/CPU0:
---------------------------------------------------------------NP
-0
1
2
3
Bridge
-----1
1
0
0
Fia
--0
0
0
0
Ports
--------------------------------------------------TenGigE0/0/0/3
TenGigE0/0/0/2
TenGigE0/0/0/1
TenGigE0/0/0/0
Cisco Confidential
77
sh controller fabric fia link-status loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller fabric fia link-status loc 0/0/cpu0
Mon Apr 6 09:40:18.677 UTC
Fia to Arbiter sync status
ARB
0
SYNCED
ARB
Fia to
Fabric
Fabric
Fabric
Crossbar sync status

Link 0
SYNCED
Link 1
SYNCED
Link 2
SYNCED
Fabric Link 3
SYNCED
SYNCED
Cisco Confidential
78
sh controller fabric fia stats loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller fabric fia stats loc 0/0/cpu0
FIA DDR Packet counters:
==========================
From Bridge#[0] 228791
To Bridge #[0] 91984
From Bridge#[1] 955781
To Bridge #[1] 942233
FIA SuperFrame counters:
==========================
To Unicast Xbar[0]
39221
To Unicast Xbar[1]
34181
To Unicast Xbar[2]
694682
To Unicast Xbar[3]
34133
To MultiCast Xbar[0]
0
45666
289374
46303
From
From
From
From
From
From
From
From
Unicast Xbar[0]
Unicast Xbar[1]
Unicast Xbar[2]
Unicast Xbar[3]
MultiCast Xbar[0]
MultiCast Xbar[1]
MultiCast Xbar[2]
MultiCast Xbar[3]
39225
34181
834384
34133
0
86872
0
4456
FIA Total Drop counters:

====================
Ingress drop: 0
Egress drop: 0
Total drop: 0
Cisco Confidential
79
sh controllers fabric fia bridge stats loc <>

UC - Unicast , MC - Multicast
LP - LowPriority , HP - HighPriority
Cast/
Packet
Packet
Error
Threshold
Prio
Direction
Count
Drops
Drops
-------------------------------------------------------------------------------Unicast Egress Stats
********************
UC HP
Fabric to NP-0
0
0
0
UC LP
Fabric to NP-0
25497
0
0
UC HP
Fabric to NP-1
0
0
0
UC LP
Fabric to NP-1
25497
0
0
UC HP
Fabric to NP-2
172099
0
0
UC LP
Fabric to NP-2
25497
0
0
UC HP
Fabric to NP-3
456510
0
0
UC LP
Fabric to NP-3
445408
0
0
---------------------------------------------------------------UC
Total Egress
1150508
0
0
Multicast Egress Stats
*********************
MC HP
Fabric to NP-0
25424
0
0
MC LP
Fabric to NP-0
0
0
0
MC HP
Fabric to NP-1
25424
0
0
MC LP
Fabric to NP-1
0
0
0
MC HP
Fabric to NP-2
25424
0
0
MC LP
Fabric to NP-2
0
0
0
MC HP
Fabric to NP-3
25424
0
0
MC LP
Fabric to NP-3
0
0
0
--------------------------------------------------------------MC
Total Egress
101696
0
0
Cast/
Packet
Packet
Prio
Direction
Count
Viking
Hardware Overview
--------------------------------------------------
Cisco Confidential
80
sh controllers fabric fia bridge ddr-status

location <>
RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge ddr-status loc 0/rsp0/cpu0
Fri Apr 17 08:37:38.032 UTC
Fia DDR#
Status
--------
------
SYNCED
SYNCED
RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge ddr-status loc 0/0/cpu0

Fri Apr 17 08:38:45.050 UTC
Fia DDR#
Status
--------
------
SYNCED
SYNCED
Cisco Confidential
81
sh controllers fabric fia bridge sync-status loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge sync-status loc 0/0/cpu0
Fri Apr 17 08:33:15.429 UTC
Bridge#
NP#
Status
-------
---
------
SYNCED
SYNCED
SYNCED
SYNCED
RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge sync-status loc 0/rsp0/cpu0

Fri Apr 17 08:42:49.929 UTC
Invalid Node. NP link status can be checked only on LCs
Cisco Confidential
82
sh controllers fabric fia bridge flow-control loc

0/0/cpu0
RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge flow-control loc 0/0/cpu0
Fri Apr 17 08:41:53.502 UTC
UC - Unicast , MC - Multicast
LP - LowPriority , HP - HighPriority
Cast/
BackPressure
Backpressure
Prio
Direction
Count
---------------------------------------------------
Ingress BackPressure Counters

******************************
FIA to bridge-0
0
FIA to bridge-1
0
All
All
All
All
Bridge
Bridge
Bridge
Bridge
to
to
to
to
NP-0
NP-1
NP-2
NP-3
0
0
0
0
Egress BackPressure Counters

****************************
Bridge-0 to FIA
0
Bridge-1 to FIA
0
UC
UC
UC
UC
LP
LP
LP
LP
NP-0
NP-1
NP-2
NP-3
to
to
to
to
Bridge0
Bridge0
Bridge1
Bridge1
0
0
0
0
UC
UC
UC
UC
HP
HP
HP
HP
NP-0
NP-1
NP-2
NP-3
to
to
to
to
Bridge0
Bridge0
Bridge1
Bridge1
0
0
0
0
MC LP
NP-0 to Bridge0
MC LP
NP-1 to Bridge0
Viking Hardware
Overview
MC LP
NP-2 to Bridge1
0
0
0
Cisco Confidential
83
Extended (-E) Line Cards

Address the highest service scale requirements
Address the highest buffering capacity requirements
Feature / Performance are virtually identical B Line Cards
A9K-40GE-E
40 Port GE SFP Extended Line Card
A9K-4T-E
4 Port 10GE XCP Extended Line Card
A9K-4T/8-E
8 Port 10GE XCP Extended Line Card
Cisco Confidential
84
Major Differences from B Line Cards

Relative to -B, the Extended (-E) Line Cards provide
~2x increase in Interface Density
~2x increase in Bridge Domain Density
~4x increase in Queue Density
~3x Packet Buffer Capacity
Relative to B, the Extended Line Cards have

additional High-Speed Memories / TCAM Resources
Otherwise the E Line Cards are virtually identical to -B
Cisco Confidential
85
Examples of Non-Differences From B LCs

Layer 3 Route Scale
Layer 2 Route Scale
Support of BGP
Support of L3VPNs
Simultaneous Support of L2 / L3
Support of Dense Hierarchical QoS

Forwarding Performance
Cisco Confidential
86
Line Card Key Metrics

Metric
-B Line Cards
-E Line Cards
MAC Addresses
1M
1M
IPv4 Routes
1M
1M
IPv6 Routes
256k
256k
VRFs
4k
4k
EFPs
16k
32k
Bridge Domains
8k
8k
Queues
96k
384k
Policers
128k
256k
DHCP Clients
128k
128k
Packet Buffer
50ms
150ms
Scale
Cisco Confidential
87
Hardware Subsystem: SIP 20G (Thor)

Next Gen SPA interface Processor
(SIP) for Viking :
Dynamic Subscriber Management
Flexibility to include additional applications
like Mobile Backhaul
20G SIP based on Quantum Flow

Processor (QFP)
Two SPA variants (at FCS) : 10 x 1GE,

1 x 10GE
Target future support for ChOC / ATM /
TDM SPAs ..
8641D Dual Core CPU and memory
Cisco Confidential
88
Hardware Subsystem: Thor

Block Diagram w/ 4x Half-height SPAs
Ingress QFP complex
QFP Memory
HH SPA
SPA
I/F
QFP 0
HH SPA
DRAM
HyTop
FPGA
PEX8509
CPU
Serpent
FPGA
Fabric
Interface
Drakar
FPGA
CPU daughtercard
HH SPA
SPA
I/F
QFP 1
HH SPA
QFP Memory
SPA daughtercard
Egress QFP complex

Cisco Confidential
89
Hardware Subsystem: QFP

QuantumFlow Processor (formerly CPP)
40 Packet processors 4 contexts (threads) each
1.2GHz processors with associated caches + DRAM packet memory
Single TCAM4 I/F (can cascade 1-4 devices)
C-language (supports stack)

HW assist for locks, PLU, hash LU, stats, WRED, policers, range lookup,
crypto, CRC
Buffer/queue subsystem (CPP-BQS)
supports up to 128k HW queues per direction

256MB max packet buffer
HW hierarchical scheduler
Fully configurable # of layers based on HQF
Cisco Confidential
90
Viking Codenames
Ryan 40x1GE Linerate LC for Red October/Samurai
Ramius 8x10GE Over-subscribed LC for Red October/Samurai
Mancuso - 4 x10GE Linerate LC for Red October/Samurai
Jetfire 2x10GE + 20 GE Linerate LC for Red October/Samurai
Odin 8x10GE Linerate LC for Red October/Samurai
Valhalla 16x10GE Over-subscribed LC for Red October/Samurai
Maximus RSP for Red October/Samurai

Red October 10 Slot Chassis
Samurai 6 Slot Chassis
Cisco Confidential
91
ASR9K Overview
ASR9K Chassis
Architecture
RSP
ASR9K Life of a
Packet
Fabric
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
92
Packet Path Snapshot

Normal Path
Moto 8641D
Punt Path
SFP/XFP
Optics
NP3c
15G XAUI
per NP3
Moto 8641D
Octopus
32G DDR
per bridge
2 x 23G
Chico
per LC
Backplane
10G MAC
per PHY
Bridge
FPGA
Santa
Cruz
Santa
Cruz
Octopus
Punt
FPGA
Punted to RSP if
FPOE says so
SFP/XFP
Optics
NP3c
Bridge
FPGA
Octopus
Cisco Confidential
93
Life of a Packet
Ingress Packet Path
2GB flash
2GB memory
1
XFP
10GE PHY
CPU
8641D
Trident Memory
Trident 0
3
Trident Memory
XFP
10GE PHY
Trident 1
Trident Memory
XFP
10GE PHY
10GE PHY
4
gigE
control
network
RSP 0
Bridge
FPGA 1
Trident 3
Network
Clocking
daughtercard
Octopus
Trident 2
Trident Memory
XFP
Bridge
FPGA 0
Cisco Confidential
RSP 1
via backplane
94
Bridge FPGA
The Bridge FPGA is acting as a traffic adaptor
between DDR interface of Octopus ASIC and
XAUI-like interface of G8000 Network
processor
There are two data paths from the NP
Bridge3 FPGA:
Manipulates the header and does interface conversion, to the
Octopus
This path handles all main data and also central punt data that
are routed
Local CPU through a switched GE link
used to process punt data locally or the packets may be sent to
the RSP card via the backplane GE or fabric link
Cisco Confidential
95
Internal Packet Headers
Cisco Confidential
96
Life of a Packet
Fabric Packet Path
Santa
Cruz 0
5: credit return
1: Fabric Request
Santa
Cruz 1
Bellagio
Octopus
(LC)
RSP0
2: Arbitration
Octopus (LC)
Santa
Cruz 0
3: Fabric Grant
Santa
Cruz 1
Bellagio
RSP1
4: load-balanced
transmission across fabric links
Cisco Confidential
97
Life of a Packet
Egress Packet Path
2GB flash
2GB memory
CPU
8641D
Trident Memory
XFP
10GE PHY
Trident 0
Bridge
FPGA 0
Trident Memory
XFP
Trident 1
10GE PHY
gigE
control
network
Trident Memory
XFP
Octopus
RSP 0
Trident 2
10GE PHY
2
5
XFP
10GE PHY
Bridge
FPGA 1
4
Trident Memory
Trident 3
Network
Clocking
daughtercard
Cisco Confidential
RSP 1
via backplane
98
Linecard backplane GE links

The backplane GE links, one to each RSP card, are
used primarily for control plane functions such as
application image download, system configuration data
from IOS-XR, statistics gathering and line card powerup and reset control.
Cisco Confidential
99
sh controller backplane ethernet links loc <>
RP/0/RSP1/CPU0:ASR9010-A#sh controller backplane ethernet links loc 0/rsp0/cpu0

Fri Apr 17 12:43:02.946 UTC
Node
| Link State
---------------------------------0/RSP0/CPU0 |
Up (Local CPU)
0/RSP1/CPU0 |
Up (Peer CPU)
0/0/CPU0 |
Up
0/1/CPU0 | Down
0/2/CPU0 | Down
0/3/CPU0
0/4/CPU0
0/5/CPU0
0/6/CPU0
0/7/CPU0
|
|
|
|
|
Down
Down
Up
Down
Down
RP/0/RSP1/CPU0:ASR9010-A#sh controller backplane ethernet links loc 0/rsp1/cpu0

Fri Apr 17 12:43:41.535 UTC
Node
| Link State
---------------------------------0/RSP1/CPU0 |
Up (Local CPU)
0/RSP0/CPU0 |
Up (Peer CPU)
0/0/CPU0 |
Up
0/1/CPU0 |
0/2/CPU0 |
Down
Down
0/3/CPU0 |
0/4/CPU0 |
Down
Down
0/5/CPU0 |
0/6/CPU0 |
Up
Down
0/7/CPU0 |
Down
Cisco Confidential
100
Life of a Packet
Egress Multicast Packet Path
2GB memory
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Trident Memory
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Trident Memory
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
CPU
8641D
Trident 0
Bridge
FPGA 0
Trident 1
Trident Memory
gigE
control
network
Trident 2
Trident Memory
Octopus
RSP 0
mcast
Bridge
FPGA 1
Trident 3
RSP 1
via backplane
Cisco Confidential
101
Life of a Packet
2GB memory
New
Sub
Non-subscriber
Subscriber
Non-subscriber
Non-subscriber
Trident Memory
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Trident Memory
CPU
8641D
Trident 0
Bridge
FPGA 0
Trident 1
Octopus
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
RSP 0
Trident Memory
Trident 2
Trident Memory
mcast
Bridge
FPGA 1
Trident 3
RSP 1
via backplane
Cisco Confidential
102
Life of a Packet
Non-subscriber
Subscriber
Subscriber
Trident Memory
Add replication only as new subscribers

are added, prune when tree is empty...
Trident 0
Subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Trident Memory
Bridge
FPGA 0
Trident 1
Subscriber
Octopus
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
RSP 0
Trident Memory
Trident 2
Trident Memory
mcast
Bridge
FPGA 1
Trident 3
RSP 1
via backplane
Cisco Confidential
103
Life of a Packet
Trident Memory
Non-subscriber
Subscriber
Trident 0
Subscriber
Subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Non-subscriber
Bridge
FPGA 0
Trident Memory
(Pruned)
Trident 1
(Pruned)
(leave)
Octopus
Subscriber
Subscriber
Subscriber
Subscriber
Non-subscriber
Non-subscriber
Subscriber
Subscriber
RSP 0
Trident Memory
Trident 2
Trident Memory
mcast
Bridge
FPGA 1
Trident 3
RSP 1
via backplane
Replicate intelligently and using need-based tree

saves bandwidth and processing cycles, while allowing for per-replica
services delivery
Cisco Confidential
104
Viking Multicast
Viking Multicast Forwarding Plane
R
[2] Switch Fabric Replicates

the packet to the appropriate
Fabric Interface chips on the
Line Cards using the FGID.
(Ex: 1110). Does not go
through VOQ
Replication Engines
RP
Switch Fabric
LC3
Fabric Interface
B0
[1] Ingress
Trident RPF
check.
LC2
Fabric Interface
B0
B1
LC1
R
Fabric Interface
1
0
R
B0
B1
R
T0
T1
T2
T3
T0
T1
T2
T3
T0
T1
[0] IGMP
joins for
subscriber
registration.
T2
[4] Bridge Replicates the

packet to the appropriate Ts
using the MGID. Has 32K
MGID entries w/ 2 bits for T
identification
T3
[5] T Replicates the packet to

the local Interfaces using the
OIL
Multicast
Source
B1
[3] Fabric Interface Replicates

the packet to the appropriate
Bridges using the MGID.
150KB Buffer for Multicast and
32K MGID entries.
IGMP joins
FGID : Fabric Group ID
Cisco Confidential
MGID : Multicast Group ID

105
ASR9K Overview
ASR9K Chassis
Architecture
RSP
ASR9K Power
Management
Fabric
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
106
Power Management
Power Management is a customer requirement that the feature
available start from FCI (3.7.2) release.
Power Management enabled ASR9K chassis have the capability to
prevent an FRU from being powered on, in the event that there is
not enough system power available to accommodate the FRU.
Power Management uses Power Monitor software feature which
allows user to monitor the power used and how much is available.
Power Management feature also alarms the user if there is not
sufficient power or any other related events.
Power management feature is not available in 3.7.1 and on
Gladiator chassis.
Cisco Confidential
107
System Overview
PwrMon process
Identifies and communicates with power supply modules via I2C
Shelfmgr process
Allows or Denies LC power-up based on (1) estimated power
requirements & (2) available power modules
Publishes Power Supply OIR events to internal consumers.
EnvMon process
Publishes power budget status to CLI
InvMgr process
Publishes detailed power supply description to inventory
Cisco Confidential
108
Feature Description
Inventory Description for Power Supplies
Indicate how many power module in system
Indicate power module source is AC or DC
RP/0/RSP0/CPU0:Samurai(admin)#show inventory

NAME: "power-module 0/PM0/SP", DESCR: "1.5kW DC Power Module"
PID: A9K-1.5KW-DC
, VID: V00, SN: TLD123950YS
NAME: "power-module 0/PM1/SP", DESCR: "2kW DC Power Module"
PID: A9K-2KW-DC
, VID: V01, SN: TLD1214001S
==================
RP/0/RSP0/CPU0:RO(admin)#show inventory

NAME: "power-module 0/PM0/SP", DESCR: "3kW AC Power Module"
PID: A9K-3KW-AC
, VID: V00, SN: TLD123850NH
NAME: "power-module 0/PM2/SP", DESCR: "3kW AC Power Module"
PID: A9K-3KW-AC
, VID: VOO, SN: TLD123650RA
Cisco Confidential
109
Feature Description
Power Management Command Summary
Power Management Command Summary
Command
Purpose
RP/0/RP0/CPU0:Router(admin)# show platform
Displays information and status for each

node in the system includes power supply
RP/0/RP0/CPU0:Router(admin)# show inventory
To retrieve and display information about

all the Cisco products that are installed in
the router, includes power supply
RP/0/RP0/CPU0:Router(admin)# show environment

power-supply
Displays power supply capacity and

current information
RP/0/RP0/CPU0:Router (admin-config)#hw-module
power [disable] location { location}
To power on a specified line card or
RP/0/RP0/CPU0:Router (admin-config)#hw-module
power override location { location}
To allows a card to be forced to power up,

regardless of an unprogrammed EEPROM
value
disable the node power-on feature
Cisco Confidential
110
Feature Description
Checking available power capacity
RP/0/RSP0/CPU0:VKG-3(admin)#show environment power-supply
Mon Oct 12 12:25:05.399 UTC
R/S/I
Modules Sensor
Watts
Status
0/PM0/*
host
PM
3000
Ok
host
PM
3000
Ok
host
PM
3000
Ok
0/PM1/*
0/PM2/*
Power Shelves Type: AC

Total Power Capacity:
Usable Power Capacity:
Supply Failure Protected Capacity:
Feed Failure Protected Capacity:
Worst Case Power Used:
9000W
9000W
6000W
3000W
2615W
Slot
---0/0/CPU0
0/1/CPU0
0/RSP0/CPU0
0/RSP1/CPU0
0/7/CPU0
0/FT0/SP
0/FT1/SP
Worst Case Power Available:
Supply Protected Capacity Available:
Feed Protected Capacity Available:
Max Watts
--------375
375
250
250
375
495
495
(default)
(default)
6385W
3385W
385W
Cisco Confidential
111
ASR9K Overview
ASR9K Chassis
Architecture
RSP
Media and File

System
Fabric
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
112
Devices On RSP
Device
Harddisk
eUSB
C. Flash
Type
FS Type
70GB
QNX4
NAND
2GB/ea
QNX4
Compact
1GB
DOS/QNX4
64MB
QNX FFSv3
512KB
Cisco
Flash
Flash
NOR
Size
SAS
(removable)
NVRAM
Quantity
Cisco Confidential
113
Devices On LC
Device
Type
Quantity
Size
FS Type
eUSB
NAND
2GB
QNX4
Flash
NOR
128MB
QNX FFSv3
Cisco Confidential
114
Partitions On RSP
Device
Harddisk
Partition
/harddisk:
/harddiska:
Size
70GB
(total)
Purpose
dumps/ etc.
/harddiskb:
Raw
eUSB
C. Flash
Primary for kernel dumps
/disk0:
1.6GB
/disk0a:
0.4GB
/disk1:
1.6GB
/disk1a:
0.4GB
/compactflash:
1GB
IOS-XR Packages
Golden Disk or Mirror Disk
General Purpose Storage
(removable)
Cisco Confidential
115
Partitions On RSP (contd)

Device
Flash 1
Partition
Raw
/bootflash:
Flash 2
Purpose
MONLIB
43MB
MBI images
Raw
ROMMON/Firmware
Raw
MONLIB
/configflash:
NVRAM
Size
28MB
OBFL data
Raw
Secondary for kernel dumps
/nvram:
Variables, logs
/nvram-raw:
Reboot history; crashinfo,

PCDS
Cisco Confidential
116
Partitions On LC
Device
eUSB
Flash
Partition
/eusb0:
1.6GB
/eusb0a:
0.4GB
Raw
/bootflash:
Raw
Size
Purpose
Local Kernel Dumps
MONLIB
56MB
ROMMON/Firmware/Kernel
dumps
Cisco Confidential
117
DOS-QNX4 File System Utility Comparison
DOS
QNX4
Types
6, 4, 11, 12
77, 78, 79, 80
File
Organization
Block-based
Extent-based
Patition
fdisk
fdisk
Initialize
mkdosfs
dinit
fs check
chkdosfs
chkfsys
Verify media
dcheck
dcheck
Standard QNX utilities. Refer to QSSL web site.
Cisco Confidential
118
CLI: show filesystem

RP/0/RSP0/CPU0:ios#show filesystem
File Systems:
Size(b)
Free(b)
Type
Flags
Prefixes
network
rw
qsm/dev/fs/tftp:
network
rw
qsm/dev/fs/rcp:
network
rw
qsm/dev/fs/ftp:
1644150784
1404023296
flash-disk
rw
disk1:
1644150784
1404023296
flash-disk
rw
disk0:
35645292544 35628865024
harddisk
rw
harddisk:
1022427136
1022418944
flash-disk
rw
compactflash:
8075067392
8073070080
harddisk
rw
harddiska:
7840202752
7838258688
harddisk
rw
harddiskb:
411041792
410914816
flash-disk
rw
disk1a:
411041792
410903552
flash-disk
rw
disk0a:
224256
165888
nvram
rw
nvram:
290816
290816
nvram
rw
nvram-raw:
44695552
23889200
flash
rw
bootflash:
28966912
28862044
rw
configflash:
Cisco Confidential
119
show media
RP/0/RSP0/CPU0:VKG-3(admin)#sh media
Fri Apr 9 15:05:34.364 EST
Media Information for 0/RSP0/CPU0.
Image
Current Part
Mountpoint
FsType FsType
Size
State
DrvrPid Mirror Flags
================================================================================
/disk0:
QNX4
QNX4
1.5G Mounted
0069666 Enabled
/disk0a:
QNX4
QNX4
0.4G Mounted
0069666
/disk1:
QNX4
QNX4
1.5G Mounted
0069676
/disk1a:
QNX4
QNX4
0.4G Mounted
0069676
/compactflash:
FAT16
FAT16
1.0G Mounted
0081977
/harddiska:
QNX4
QNX4
7.5G Mounted
0049184
/harddiskb:
QNX4
QNX4
7.3G Mounted
0049184
/harddisk:
QNX4
QNX4
33.2G Mounted
0049184
/lcdisk0:
QNX4
(?)
Not Present
/lcdisk0a:
QNX4
(?)
Not Present
RP/0/RSP0/CPU0:VKG-3#sh media loc 0/0/cpu0
Fri Apr 9 15:13:51.680 EST
Media Information for 0/0/CPU0.
Image
Current Part
Mountpoint
FsType FsType
Size
State
DrvrPid Mirror Flags
================================================================================
/lcdisk0:
QNX4
QNX4
1.5G Mounted
0049185
/lcdisk0a:
QNX4
QNX4
0.4G Mounted
0049185
Cisco Confidential
120
ASR9K Overview
ASR9K Chassis
Architecture
RSP
Fabric
Online Diagnostics
Linecard
Life of a Packet
Power Management

Online Diagnostics
Service Console
Cisco Confidential
121
What is online diagnostics

It defines a suite of tests for each card in system, although some
tests test inter-card links
Tests run in background, faults reported to Platform Fault Manager
Tests can be enabled and disabled individually, test frequency can

be changed through configuration commands
Components Covered by FCI:
- Control path and data path links
- access to scratch registers for major ASICs and FPGAs/CPLDs
- integrity of FPD images
Cisco Confidential
122
EOBC paths
Cisco Confidential
123
EOBC heartbeat tests

Bi-directional;
Runs on all Line cards and Standby RSP
every 5 seconds, by default

Interval can be changed by changing configuration
Cisco Confidential
124
User Interface
Generic user interface : GOLD-XR, same as on CRS
Only admin users have privilege to access online diagnostic commands:
commands are in admin mode
Most tests running in the background by default
Show commands to see what tests are available, running frequency of

each test, etc.:
show diagnostic content location <node>
Show commands to see tests result:
show diagnostic result location <node> [test <test-id> detail ]
Change test running frequency or enable/disable a test:
use admin configuration commands:
diagnostic monitor interval location <node> test <test-id> <day>
<hh:mm:ss.mille-sec>
diagnostic monitor location <node> test <test-id> disable
Cisco Confidential
125
Sample screen shots

RP/0/RSP0/CPU0:ios(admin)#show diagnostic result location 0/rsp0/cpu0
Current bootup diagnostic level for RP 0/RSP0/CPU0: minimal
RP 0/RSP0/CPU0:
Overall diagnostic result: PASS
Diagnostic level at card bootup: minimal
Test results: (. = Pass, F = Fail, U = Untested)

1
) PuntFPGAScratchRegister ---------> .
) XbarInterfaceScratchRegister ----> .
) ClkCtrlScratchRegister ----------> .
) IntCtrlScratchRegister ----------> .
) CPUCtrlScratchRegister ----------> .
) XbarSwitchIdRegister ------------> .
) EccSbeTest ----------------------> U
) SrspStandbyEobcHeartbeat --------> U
) SrspActiveEobcHeartbeat ---------> U
10 ) FabricLoopback ------------------> .
11 ) PuntFabricDataPath --------------> U
12 ) BootflashVerify -----------------> U
Cisco Confidential
126
Referencias
ASR9000/XR Understanding and
troubleshooting Fabric issues in the A9K
https://supportforums.cisco.com/document/12135016/asr9
000xr-understanding-and-troubleshooting-fabricissues-a9k
ASR9000/XR: Troubleshooting packet

drops and understanding NP drop
counters
https://supportforums.cisco.com/docs/DOC-15552
Cisco Confidential
127
Cisco Confidential
128

1 Asr9k Arc Base

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

1 Asr9k Arc Base

Caricato da

Copyright:

Formati disponibili

ASR9K Overview

Media and File Systems

Viking Hardware Overview

2008 Cisco Systems, Inc. All rights reserved.

ASR 9000 At a Glance

Designed for Longevity:

Carrier Ethernet Focus

Viking Hardware Overview

Complete EMS/NMS Support

Video Quality Monitoring

Designed with Operators in Mind

Software Optimized for Automation, Ease of Use

Complete OAM Coverage

MPLS OAM/VCCV support

IOS-XR Powers the Edge with ASR 9000

A Fully Distributed, Microkernel-Based Architecture

Distributed Service Separation

Next Generation Architecture

Viking Hardware Overview

Designed for Scale, HA, and Performance

Cisco ASR 9000

Half Terabit More Dense than Competition

Maximixed Slot Real Estate

Power-Up as You Grow

Per-slot power & cooling

ASR9k: 8 slots * 20x 10GE = 160x 10GE

ASR 9000 System Scalability

2005 2006 2007 2008 2009 2010 2011 2012

Source: Cisco Visual Networking Index Forecast,

Bandwidth per Slot

Linecards per Chassis

Bandwidth per Chassis

Viking Hardware Overview

Media and File Systems

Viking Hardware Overview

2008 Cisco Systems, Inc. All rights reserved.

Carrier-class hardware redundancy

Hardware Subsystem: 10-slot Power

Max system draw 8.5KW, max FCS draw is <= 5KW

FCS power configs: 3+1 DC (or) 2+2 AC

Viking Hardware Overview

Hardware Subsystem: 10-slot power

Power Distribution Bus

Single power zone, one

All modules load share

2kW and 1.5kW supplies

Hardware Subsystem: 10-slot power

Power Distribution Bus

Single power zone, one

AC power supplies are

A feed wired to top power

B feed wired to bottom

10-slot chassis: 6kW redundant commons

For 6KW AC, you need 4x 3KW power

Power Distribution Bus

Power Distribution Bus

For 6KW DC, 4x 2KW power supplies

Hardware Subsystem: 10-slot thermals

Dual stacked fan trays below

Viking Hardware Overview

Carrier-class hardware redundancy

Hardware Subsystem: 6-slot

FCS draw is significantly less than

Hardware Subsystem: 6-slot power