Cisco ASR9000 series

Platform Architecture
ASR 9000 “At-a-Glance”
Next-Generation SP Edge & Aggregation

Optimized for
high dense10G &
100G aggregation

IOS-XR
Video DNA Non-Stopping
service
400G/Slot
Longevity
Green

ANA service
management and
Fully converged provisioning
L2 & L3 service

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
ASR 9000 System Architecture “At-a-Glance”

Line Card
Fully Distributed
Architecture for High RSP
Performance and High Multi-
dimensional Control Plane
Scale CPU
CPU BITS/DTI

Data forwarding is fully FIA
distributed on the line cards
FIC

Control plane split among RSP

and LC CPU (same type of CPU as
RSP)

L2 protocols, BFD, CFM, Netflow
runs on the LC CPU for high scale

IOS-XR: True Modular OS for HA

and Operational Simplicity

Micro-kernel based, true modular OS Switch Fabric

High availability and System stability

SW patch granularity for operational simplicity Active-Active Intelligent Switch Fabric
Guarantee “0” packet loss during RSP failover
Advanced internal system QoS, Intelligent
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
multicast replication
ASR 9010 and ASR 9006 Chassis
Identical HW components across two chassis*
Front-to-
Integrated cable back airflow
management
with cover

System fan trays Side-to-back

airflow
RSP (0-1)
Line Card
(0-3) Line Card
(0-3, 4-7)

RSP (0-1)
System fan trays

Air draw

cable Six Modular

management Power Supplies
Three Modular
Power Supplies
* Fan tray is different
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Power and Cooling
Existing Power Supply and Fan are ready for 400G/slot

Fans unique to chassis

Variable speed for
ambient temperature variation

Redundant fan-tray
ASR 9010 Fan Tray ASR 9006 Fan Tray
Low noise, NEBS and OSHA compliant

DC Supplies

A
B 2.1/1.5 kW
6 & 10 slot use same power supplies
A
Single power zone
B 2.1 kW

All power supplies run in active mode

Power draw shared evenly
AC Supplies
Power Supply
50 Amp DC Input or 16 Amp AC
for Easy CO Install
A 3 kW
B 3 kW

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
 Power Check and Rules

Available power is checked when:
–An LC card is inserted
–An LC card is powered up via the CLI
–An LC card is reset via “hw-mod reload”

If the system does not have enough available power to
accommodate the LC, then the LC becomes “UNPOWERED”

Installing new power supplies will not automatically power up any
UNPOWERED line cards. The user can force a recheck using:
“hw-mod reload loc <>”

RSP and Fan Tray cards are given priority allocation of power
budget

LC power budget is checked in numeric order until it is exhausted.
The actual power up of the LC's is in parallel, and asynchronous

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
RSP Engine

Performs control plane and management functions

Dual Core CPU processor with 4GB or 8GB DRAM

2MB NVRAM, 4GB internal bootdisk, 2 external compact flash
slots

Dual Out-of-band 10/100/1000 management interface

Console & auxiliary serial ports

Hard Drive: 70G HDD

Status light
Console Port BITS ALARM
Compact Flash Status LED
Management AUX Port Slots
Ethernet
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
RSP Operations Impact Fabric ?
Guarantee “0” packet loss for RSP failover or OIR

Switch fabric ASIC reside on the RSP blade physically

Switch fabric ASIC is controlled by low level hardware, it operates separately
from RSP function

All fabric ASIC run in active mode regardless of the RSP status

RSP SW switch over, reload, crash including kernel crash have NO impact on
fabric operation

RSP OIR has no traffic impact due to long/short pin backplane design and
instant fabric switch over
–Short pin trig the control signaling for fabric switchover in hardware
–Long pin is used for data packet. It can continue draining the in-flight
packets from the fabric during the extended short period of time

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Ethernet Line Card Family
Common HW ASIC and HW architecture

Identical SW features

Line card Memory options for QoS scale

A9K-40G A9K-4T A9K-8T/4 A9K-2T20G A9K-8T A9K-16T/8

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 Line Card Architecture – Hardware Components
CPU (same as RSP)
Program HW forwarding tables
NP: Network Processor Distributed Control planes
Main forwarding ASIC SW switched packets
L2 & L3 forwarding, Inline Netflow
features (QoS, ACL, etc), CPU
control plane policing, PHY NP0
mcast replication, etc
PHY NP1
10Gbps bi-directional with B0
features applied FIA0 FIA: Fabric Interface ASIC
PHY NP2 Provide non-blocking data
B1 connection to switch fabric
Internal system queues/VoQ
PHY NP3 Intelligent mcast replication

PHY NP4

10G PHY for one 10G

PHY NP5 B0
B: Bridge FPGA
port, or 10x1G port FIA1 Provide non-blocking data
PHY NP6 B1 connection between NP and FIA
Internal System queues
PHY NP7 Intelligent mcast replication

Example: A9K-8T Note, Bridge FPGA provide non-blocking

connection between NP and the FIA. Functionally it
does the HW conversion due to different interface
format on NP and FIA. It’s part of the switch fabric
connection. To make it logically simple, it will be
removed from the remaining slides.
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
4xNPs Line Card Family
A9K-4T-E/B/L A9K-2T20G-E/B/L

PHY 3 NP0 PHY NP0

PHY 2 NP1 PHY NP1

FIA FIA
PHY 1 NP2 NP2

PHY 0 NP3 NP3

A9K-8T/4-E/B/L A9K-40G-E/B/L
PHY 3
PHY 7
NP0 NP0
PHY 2
PHY 6
NP1 NP1
FIA FIA
PHY 1
PHY 5
NP2 NP2
PHY 0
PHY 4 NP3 NP3

Oversubscribed line card

Up to 60Gbps bandwidth
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
8xNPs Line Card Family

A9K-8T-E/B/L A9K-16T/8-B
PHY
PHY NP0 PHY NP0
PHY
PHY NP1 PHY NP1
FIA0 PHY
FIA0
PHY NP2 PHY NP2
PHY
PHY NP3 PHY NP3
PHY
PHY NP4 PHY NP4
PHY
PHY NP5 PHY NP5
FIA1 PHY
FIA1
PHY NP6 PHY NP6
PHY
PHY NP7 PHY NP7

Oversubscribed line card

Up to 120Gbps (~117Gbps)
bandwidth

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Line Card Architecture
A9K-8T/4
60Gbps raw bandwidth
A9K-16T/8-B
PHY CPU 120Gbps raw bandwidth
Crossbar
NP0 Fabric
ASIC
PHY
NP1 Crossbar
B0 Fabric
PHY FIA0 ASIC
NP2
B1
PHY Arbiter
NP3
RSP0
PHY
NP4 Crossbar
Fabric
PHY ASIC
NP5 B0
PHY
FIA1 Crossbar
NP6 B1 Fabric
ASIC
PHY NP7 Arbiter
RSP1

30Gbps and 25M pps 15Gbps bi- 30Gbps bi-

(combined ingress and egress) directional directional Each FIA has one fabric channel
60Gbps bi- which is 23 Gbps bi-directional, to
30Gbps bi-
directional each of the switch fabric ASIC
directional
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Multicast Packet Replication
Switch Fabric and Egress LC Replication
2 FIA Replication  replicate
single copy to Bridge which has
1 Fabric Replication  IGMP join, based on MGID NP Replication  replicate
4
replicate single copy to table in FIA copy per receiver based on
LCs which receive IGMP multicast FIB table
3 Bridge Replication  similar
join, based on FGID table as FIA replication, single copy
in switch fabric to NP MFIB
IGMP joins
CPU 4
NP0 PHY
MGID
MGID
B0
3 NP1 PHY
2
IGMP joins CPU FIA
PHY NP0 B1 NP2 PHY
FGID/
3 FPOE
PHY NP1 2 NP3 PHY
B0 1 LC2
IGMP joins FIA
PHY NP2
B1 Switch
Multicast Fabric CPU
Source PHY NP3 NP0 PHY

B0 NP1 PHY
FIA
FGID – Fabric Group ID B1 NP2 PHY
MGID – Multicast Group ID
MFIB – Multicast Forwarding Information Base NP3 PHY
LC3
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Multicast Packet Replication (2)
Switch Fabric and Egress LC Replication
2 FIA Replication  replicate
single copy to NPs which
1 Fabric Replication  receive IGMP join, based on NP Replication  replicate
4
replicate single copy to MGID table in FIA copy per receiver based on
LCs which receive IGMP multicast FIB table
3 Bridge Replication  similar
join, based on FGID table as FIA replication, single copy
in switch fabric to NP MFIB
1 0000000000 CPU
IGMP joins
4
2 0000000001 NP0 PHY
FGID=3 MGID
MGID
3 0000000011
hit 3 NP1
2 B0 PHY
IGMP joins CPU …
FIA
PHY NP0 N 1111001111 B1 NP2 PHY
1 00 Replicate to the first
3 FGID
PHY NP1 FPOE
B0
Table2 MGID 2 hit 01 NP3
Bridge only PHY
1 LC2
IGMP joins FIA 3 10
PHY NP2
B1 Switch …
Multicast Fabric CPU
Source PHY NP3 N 10 NP0 PHY
MGID Table in FIA NP1
B0 PHY
FIA
FGID – Fabric Group ID B1 NP2 PHY
MGID – Multicast Group ID
MFIB – Multicast Forwarding Information Base NP3 PHY
LC3
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Multicast Packet Replication (3)
Inside NP

To egress
port Second lookup 14Gbps
replication
TM capacity per NP

From Fabric Initial lookup

NP

Initial Lookup
– Original packet is lookup in the MFIB/L2FIB returning the number of copies
to be replicated on the given NPU, each copy per output logical interface
– Packet is sent to a TM (traffic manager). TM replicates the packets and send
the copy to the processing engine

Second Lookup
–Each copy is processed and sent out of interface. Egress feature like QoS,
ACL is applied at this pass

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
L3 Multicast Control Plane Overview

4 MFIB
PHY NP0 PIM IGMP
CPU 3 2
5
PHY NP1
B0 MRIB CPU
FIA 1
PHY NP2
B1 Crossbar Crossbar
Fabric Fabric
PHY NP3 ASIC
Switch Fabric
ASIC

Incoming IGMP and PIM packets are punted to RP directly bypassing LC CPU

Protocols (PIM/IGMP) send their Route/OLIST Information to MRIB process to build multicast
route/olist table

MRIB sends the multicast state information to MFIB process on all LCs

MFIB program HW forwarding tables in NP, Bridge FPGA and Fabric interface ASIC

Software switched multicast packet or data packet for protocol signaling is sent to local line
card CPU

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 L2 Multicast Control Plane
4 L2FIB IGMP snooping
PHY NP0 CPU 3 2
5
PHY NP1
B0 L2FIB CPU
FIA 1
PHY NP2
B1 Crossbar Crossbar
Fabric Fabric
PHY NP3 ASIC
Switch Fabric
ASIC

Incoming IGMP packets are punted to RP directly bypass LC CPU

IGMP snooping process send its Information to L2FIB

L2FIB sends the multicast state to L2FIB on all LCs

L2FIB on LC program HW forwarding tables in NPU, Bridge FPGA and Fabric interface

Software switched multicast packet or data packet for protocol signaling is sent to local line card CPU

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Line Card Memory Options – Queue Scale
High

3 memory options for each line card: Extended (or high queue),
Base (medium queue), Low (low queue)* Medium

Different memory option has different QoS queue scale and L2 Low
sub-interface scale. All other system wide scale is the same
across different type of the line cards, including FIB, MAC
address, Bridge-domain, L3 sub-interface, VRF, etc

All line cards have the same HW  Identical features

Mixed different type of line cards are supported on the same

chassis with same system wide scale and identical features

* A9K-16T/8 only have “B” option. It doesn’t have “E” or “L”

option as of the 3.9.1 release

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
L/B/E Line Cards – What’s the Difference?

STATS MEMORY
FIB MAC
LOOKUP FRAME MEMORY
Forwarding ASIC
MEMORY
-
TCAM
NP complex

Each NPU has Four Main Associated memories TCAM , Search/Lookup memory , Frame/buffer
memory and statistics memory
–TCAM is used for VLAN tag, QoS and ACL classification
–Lookup Memory is used for storing FIB tables, Mac address table and Adjacencies
–Stats memory is used for all interface statistics, forwarding statistics etc
–Frame memory is buffer memory for Queues

E/B/L line card have different TCAM , Stats and Frame Memory size, which give different scale number
of the QoS queues and L2 sub-interfaces per line card

Lookup Memory is the same across line card s why?
–To support mix of the line cards without impacting the system wide scale including routing,
multicast, MAC address, L3 interface, MPLS label space scale

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 Optics
Standard, CWDM, & DWDM XFPs/SFPs/SFP+ available
IPoDWDM G.709 FEC/EFEC

GE - SFP Optics TenGE – XFP Optics

T, S, L, & Z LR, ZR, & ER
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
SIP/SPA – Non-Ethernet Support

Strong QoS Scalability

• 128+K Queues, • Distributed control
• 128+K Policers and data plane; High Availability
• H-QoS 20G; 4 Bays
• IC-Stateful Switch
• Color Policing • L3 i/f, route, session, Over capability,
protocol – scaled up
for MSE needs • MR-APS
• IOS-XR base for
high scale and
reliability
Powerful and
Flexible QFP
Processor
• Flexible ucode Architecture
for Feature Richness
• L2 + L3 Services: FR, PPP,
HDLC, MLPPP, LFI, L3VPN,
MPLS, Netflow, 6PE/VPE
A9K-SIP-700

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public *Not Commtted 22
 Switch Fabric Overview

Active-active load balancing: Unicast: per-packet load balancing, Multicast: per (S,G) load balancing

Arbiter for fabric access control. Arbiter is in active/standby mode, which is controlled by low level
hardware signalling

Frame format over fabric: super-frame, it can aggregate multiple small packet into a big sup-frame to
improve the fabric throughput

Crossbar 23Gbps per fabric channel

Fabric
ASIC

Crossbar FIA0
Fabric
ASIC

Arbiter FIA1
FIA RSP0
Dual-FIA
8xNPs
Crossbar Linecard
Single-FIA Fabric
4xNPs ASIC
Linecard
Crossbar
Fabric
4x23Gbps =92Gbps with dual RSP ASIC 8x23Gbps =184Gbps with dual RSP
2x23Gbps=46Gbps with single RSP 4x23Gbps=92Gbps with single RSP
Arbiter

RSP1
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Fabric Redundancy
Guarantee “0” packet loss during RSP failover and OIR

Separate data and arbitration(control) paths

All fabric data channels run in active mode for extra fabric bandwidth and instant fabric
switch over

Both Arbiters work in parallel – both answer all requests, FIAs follow active Arbiter, FIAs
switch to backup if needed  instant control path switchover

Arbiter switchover is controlled by low level hardware signalling
Crossbar
Fabric
ASIC

Crossbar FIA0
Fabric
ASIC

Arbiter FIA1
FIA RSP0
Dual-FIA
8xNPs
Crossbar Linecard
Single-FIA Fabric
4xNPs ASIC
Linecard
Crossbar
Fabric
ASIC

Arbiter

Presentation_ID RSP1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
L3 Control Plane Overview

BGP OSPF
LDP RSVP-TE
Static
ISIS EIGRP

LSD RIB RP

ARP
FIB Adjacency
SW FIB
AIB LC NPU

AIB: Adjacency Information Base

RIB: Routing Information Base
LC CPU FIB: Forwarding Information Base
LSD: Label Switch Database
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
L3 Forwarding – Hierarchical Data Structure
FIB and ADJ Tables Overview

0 1 2 3 . . . … n
BGP PIC
VRF table
TE-FRR PIC
Link bundle PIC

PIC: Prefix independent convergence

Search Tree LAG: Link aggregation group
LDI: Load information

OutI/F
adj
LAG OutI/F
Recursive
NR Prefix adj
Leaf Table Prefix Leaf
Leaf Table
Table
Protected
LAG OutI/F
TE adj

NR LDI 0
Recursive
LDI/ADJ LDI (4-way) NR LDI 1
… Backup
OutI/F
NR LDI N TE adj

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Two-Stage Packet Forwarding
Fully Distributed Forwarding on Line Cards

Each line card has independent AIB only
for local interfaces
Packet is forwarded to the egress
Each line card has independent Interface
NP based on the information in DB for local interfaces
the NP/fabric header

Both Ingress and Egress FIB – allows
forwarding features to be independently
applied on LCs

CPU CPU
PHY NP0 NP0 PHY
2
PHY NP1 NP1 PHY
FIA FIA
PHY NP2 NP2 PHY
Switch
Fabric
PHY NP3 Ingress LC Egress LC NP3 PHY
1

Ingress NP look up  Egress NP look up 

Get egress NP information, Get egress logical port, VLAN,
FGID, MGID, etc, add those MAC, ADJ information, etc for
information into fabric/NP header packet rewrite
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 28