Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
BRKARC-3004
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
What Makes the CRS Different?
What does Carrier Class architecture mean?
‒ Reliability
‒ Scalability
‒ Predictability
Where do you need it?
How does a router deliver?
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Before We Begin
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
CRS Architecture Review – System
PLIM – Physical Layer Interface Module Switch Fabric
MSC – Modular Service Card - 4 or 8 redundant planes
RP – Route Processor - Multi-Chassis option
RP (active) RP (standby)
PLIM M
I
MSC M
I
F
A
F
A
M
I
MSC M
I
PLIM
D D B B D D
Packets PLIM P MSC P R F R P
MSC P
PLIM
In Packets
L L I I L L
PLIM A MSC A C
A C A MSC A PLIM Out
N N B N N
PLIM E MSC E S1 S3 E MSC E PLIM
R
I
PLIM M
MSC M F F M
MSC M
PLIM
I I A C A I I
D D B B D D
PLIM P MSC P R R P
MSC P
PLIM
PLIM
L
MSC
L I S2 I L
MSC
L
PLIM
A A C C A A
N N N N
PLIM E MSC E S1 S3 E MSC E PLIM
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
CRS Architecture Review – Line Card
F
Ingress A
PSE IngressQ B
R
Queuing and I F
M
L3 Lookup Segmentation M C
& Features A
I I S1
P D D B
L P P R
L CPU L
I I
A A
M N N
C
Egress F
E E
EgressQ PSE 2 FabricQs A
B S2
R
Output Reassembly & I
L3 Lookup
Queuing Queuing into PSE C
& features
S3
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
CRS Architecture – Line Card Details
HW CEF TCAM Shape Queues Fabric Dest Qs F
HP
BE A
HP B
BE Cells
R
HP
I
BE
C
F
M HP
BE
M A
I I S1
P D PSE IngressQ D B
L P P R
CPU
I L L I
A A
M N
Priority
Assured Forwarding
HP
N
C
Assured Forwarding HW CEF TCAM HP
F
E Assured Forwarding
Assured Forwarding
AF
FQ E
Best Effort AF A
Priority
Assured Forwarding
BE
BE
B S2
Best Effort
Priority HP FabricQs R
Best Effort
Priority
HP I
AF
Assured Forwarding
Assured Forwarding
AF FQ C
Best Effort
BE S3
EgressQ PSE BE
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Key Presentation Topics Discard
Fabric Bitmap
Queuing
PLIM
Stats HW CEF TCAM Shape Queues Fabric Dest Qs F
HP
PSE BE A
Operation HP B
BE Cells R
HP
BE
I F
M M C
HP
BE
A
I CEF I S1
P D PSE IngressQ D B
L P P R
L Output CPU Reassembly
L
I Queuing
TX Features Queues I
A A
M N
Priority
Assured Forwarding
HW CEF TCAM
HP
HP N
C
Assured Forwarding
Assured Forwarding
AF
F
E Assured Forwarding
Best Effort AF FQ E A
Priority
Assured Forwarding
Best Effort
BE
BE
B S2
Priority HP FabricQs R
Best Effort
Priority
HP I
AF
Assured Forwarding
Assured Forwarding
AF FQ C
Best Effort
BE
EgressQ PSE BE S3
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Ingress PLIM
1 – 4 PLIM ASICs (PLAs) depending on card type
Nominally 40 Gbps for CRS-1, 140G for CRS-3
Oversubscription allowed when all Ethernet
96 Gbps aggregate bandwidth into PSE
‒ No bandwidth bottleneck even when oversubscribed
CRS-3 adds PLIM QoS
CRS-1 PLIMs
PLIM PLAs BW to PSE (per PLA) M HW CEF TCAM
4xOC192 POS 2 48 Gbps
PLA I
D
16xOC48 POS 4 24 Gbps
P
1xOC768 POS 1 96 Gbps PLA L
4/8xTenGE 2 48 Gbps A
SIP-800 2 48 Gbps PLIM N
E PSE
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Detecting PLIM Drops – Receive Side
show controllers plim asic stat int pos 0/1/0/0 loc 0/1/CPU0
DRP/0/0/CPU0:BUY-CRS# show contr plim asic stat int pos 0/1/0/0 loc 0/1/CPU0
Node: 0/1/CPU0
POS0/1/0/0 Rx Statistics
------------------------------------------------------------------------------
TotalOctets : 135458452621652
TotalPkts : 3069907840506 UnicastPkts : 3069907840532
MulticastPkts : 0 BroadcastPkts : 0
64Octets : 0 65to127Octets : 0
128to255Octets : 0 256to511Octets : 1
512to1023Octets : 0 1024to1518Octets : 0
1519to1548Octets : 0 1549to9216Octets : 3
>9216Octets : 0 BadCRCPkts : 0
BadCodedPkts : 0 Runt : 0
ShortPkts : 3069907840561 802.1QPkts : 0
Drop : 4941487731 PausePkts : 0
ControlPkts : 0 Jabbers : 0
BadPreamble : 0
POS0/1/0/0 Drop
------------------------------------------------------------------------------
RxFiFO Drop : 4294967295 PAR Tail Drop : 4294967295
Audience Question – What is the most likely cause of Ingress PLA drops?
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Ingress PSE Topics
ASIC Architecture
Ingress PPE Operation
IngressQ
PSE TCAMs/ACLs
Features
CPU CEF - IP/MPLS
ACLs
Policing*
EgressQ Egress
FabricQs Netflow*
PSE
uRPF*
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
CRS-1 PSE Forwarding ASIC Architecture
Cluster – 12 PPEs Instruction Memory
Packet
Input Buffer
Stream
Interface IMEM MUX
IMEM MUX
R PLU
IMEM MUX E
IMEM MUX S
IMEM MUX O
IMEM MUX U
TLU
IMEM MUX R
C
Packets IMEM MUX
IMEM MUX
R
IMEM MUX
C
IMEM MUX
E TCAM
IMEM MUX
Header IMEM MUX
F
IMEM MUX A STATS
IMEM MUX B
IMEM MUX R
IMEM MUX I POLICER
IMEM MUX C
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Packet Path Within PSE
Perform Lookup and Features Using Resources
Packet
Buffer
IMEM MUX
R
IMEM MUX
E
PLU
IMEM MUX
IMEM MUX
S
IMEM MUX
O
U TLU
IMEM MUX
IMEM MUX
R
IMEM MUX
C
IMEM MUX
E TCAM
IMEM MUX
IMEM MUX
F
IMEM MUX
A STATS
IMEM MUX
B
IMEM MUX R
I
IMEM MUX
POLICER
IMEM MUX C
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Packet Path Within PSE
Recombine Header and Tail and Send to IngressQ
Packet
Buffer
IMEM MUX
R
IMEM MUX
E
PLU
IMEM MUX
IMEM MUX
S
IMEM MUX
O
U TLU
IMEM MUX
IMEM MUX
R
IMEM MUX
C
IMEM MUX
E TCAM
IMEM MUX
IMEM MUX
F
IMEM MUX
A STATS
IMEM MUX
B
IMEM MUX
R
I
IMEM MUX POLICER
IMEM MUX
C
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Inside the PSE - Cisco Express Forwarding
PSE consists of 188 PPEs (Packet Processing Engines) (CRS-3 - 256)
Each packet is forwarded by a single PPE
‒ No pipeline (no stall risk)
‒ Lookups and features for a packet executed by same PPE
CEF uses Tree Bitmap for forwarding table
‒ Low memory utilization
‒ Fast lookup times
‒ Fast update times (improves convergence)
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Detecting PSE Drops Both Ingress and Egress
RP/0/RP0/CPU0:XR-ROCKS# show controllers pse statistics 0/0/CPU0
Node 0/0/CPU0 Ingress PSE Stats
--------------------------------
Punt Stats Punted Policed & Dropped
---------- ------ -----------------
CDP 2145978 0
ARP 121223275 10
Bundle Control 223294 0
IPv4 TTL expiration 3510 0
IPv4 L2LI punt 1369 0
IFIB lookup miss 51 0
ACL deny gen ICMP 21170743 18935344
IPv6 link local 478990 0
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Looking at PSE Drops
RP/0/RP0/CPU0:XR#show captured packets ingress location 0/1/CPU0
-------------------------------------------------------
packets dropped in hardware in the ingress direction
buffer overflow pkt drops:3941067, current: 200, non wrapping: 0 maximum: 200
-------------------------------------------------------
Wrapping entries
-------------------------------------------------------
[1] Jun 21 18:31:55.700, len: 239, hits: 1, i/p i/f: TenGigE0/1/0/3
[Distribution header: exception: 0 error 0 dummy 0 moose_congestion 0
gpm_allocated 1 gpm_segment 903]
[Moose Header: pkt_length: 74 port_num: 580 mark: 0 vlan_result: 7
mac_da_result: 0 mac_sa_result]0
[punt reason: IPv4 not enabled] [PPE used: cluster=15 ppe=8]
[ether dst: 0100.5e00.000a src: 001b.5374.cab6 type/len: 0x800]
[IPV4: source 172.18.26.6, dest 224.0.0.10 ihl 5, ver 4, tos 192
id 365, len 60, prot 88, ttl 1, sum 111b, offset 0]
0205f769 00000000 00000000 00000000 00000064 0001000c 01000100 0000000f
00040008 03030102 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 000000
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Security Access List Implementation
ACLs are executed on Ingress and Egress in TCAMs
‒ Content Addressable Memory – search returns first matching entry
‒ Ternary – Adds “don‟t care” option to regular binary CAM (3 choices)
Support for large ACLs w/o performance impact
‒ 16K lines per ACL and 64K lines of ACLs per card per direction
Source Source Proto Source Dest Destination Destination
IP Mask L4 Port IP Mask L4 Port
TCAM Example
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Monitoring TCAM Utilization Before adding ACL
RP/0/RP0/CPU0:XR-ROCKS# show controllers pse tcam sum loc 0/5/CPU0
TCAM Device Information for Ingress Metro, CAM channel 0:
Device size: 18M (256K array entries of 72-bits), 262051 available
Current mode of operation: Turbo
Software Initialization:
Memory management state: complete
Range block state: complete
IPv6 prefix compression state: complete 262051 available
Hardware Initialization:
Device registers: complete
CAM/SRAM Memory: complete
Default entries for applications: complete
Range Logic Block registers: complete
IPv6 prefix compression region: complete
Feature specific information:
QoS (id 1):
Owner client id: 1. Limit 262144 cells
Duplication enabled in Turbo mode into CAM channel 1
Fab QoS (id 2):
Owner client id: 14. Limit 262144 cells
Total 1 regions using 11 CAM cells
Duplication enabled in Turbo mode into CAM channel 1
ipv6 prefix compress (id 10):
Owner client id: 13. Limit 262144 cells
Total 1 regions using 2 CAM cells
Entry duplication enabled in Turbo and Feature modes into CAM channel 1
tcam_mgr (id 11):
Owner client id: 13. Limit 262144 cells
Total 2 regions using 80 CAM cells
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Monitoring TCAM Utilization After adding 6 line input ACL
RP/0/RP0/CPU0:XR-ROCKS# show controllers pse tcam sum loc 0/5/CPU0
TCAM Device Information for Ingress Metro, CAM channel 0:
Device size: 18M (256K array entries of 72-bits), 262045 available
Current mode of operation: Turbo
Software Initialization:
Memory management state: complete
Range block state: complete 262045 available
IPv6 prefix compression state: complete
Hardware Initialization: -> 6 used for ACL
Device registers: complete
CAM/SRAM Memory: complete
<snip>
Feature specific information:
packet filtering (id 0):
Owner client id: 3. Limit 262144 cells
Total 1 regions using 6 CAM cells
QoS (id 1):
Owner client id: 1. Limit 262144 cells
Duplication enabled in Turbo mode into CAM channel 1
Fab QoS (id 2):
Owner client id: 14. Limit 262144 cells
Total 1 regions using 11 CAM cells
Duplication enabled in Turbo mode into CAM channel 1
ipv6 prefix compress (id 10):
Owner client id: 13. Limit 262144 cells
Total 1 regions using 2 CAM cells
Entry duplication enabled in Turbo and Feature modes into CAM channel 1
tcam_mgr (id 11):
Owner client id: 13. Limit 262144 cells
Total 2 regions using 80 CAM cells
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
IngressQ Topics
Ingress Shaping
Ingress To-Fabric Queuing
PSE IngressQ
Discard Bitmap*
CPU Fabric QoS*
EgressQ Egress
PSE FabricQs
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
IngressQ
Shape Queues Fabric Dest Qs F
Input Shaping Queues HP
BE
A
P HP
Cells B
Per Interface S
BE
R
HP
I
HP & LP if configured E BE
HP C
Fabric Destination Queues
BE
IngressQ S1
HP & LP for every FabricQ in system
4 queues for every MSC in entire system
Queue determined by Ingress QoS or Fabric QoS
Segmentation of packets into cells
45 Gbps limit between Shape Queues and Fabric Destination Queues
140 Gbps in CRS-3
Discard bitmap (discussed later but occurs between these two queues)
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Monitoring Shaping Parameters
Before and After Adding Input Shaper (on gig 0/5/0/1)
RP/0/RP0/CPU0:CRS# show controllers ingressq queue all loc 0/5/CPU0
iqm queues:
legend: (*) default q, LP low priority,HP high priority, bs burst sizebw (kbps), bs (usec),
quant(10s of 2*Fab MTU).
name owner q-hd port type max_bw min_bw max_bs min_bs quant pktcnt
-----------------------------------------------------------------------------------------
default_queue(*) hfr_pm 32 8 LP 999936 0 639 0 10 0
default_queue(*) hfr_pm 33 9 LP 999936 0 639 0 10 0
default_queue(*) hfr_pm 34 10 LP 999936 0 639 0 10 0
default_queue(*) hfr_pm 35 11 LP 999936 0 639 0 10 0
default_queue(*) hfr_pm 36 12 LP 999936 0 639 0 10 0
<snip>
RP/0/RP0/CPU0:CRS# show controllers ingressq queue all loc 0/5/CPU0
iqm queues:
legend: (*) default q, LP low priority,HP high priority, bs burst sizebw (kbps), bs (usec), quant(10s of
2*Fab MTU).
name owner q-hd port type max_bw min_bw max_bs min_bs quant pktcnt
-----------------------------------------------------------------------------------------
default_queue(*) hfr_pm 32 8 LP 999936 0 639 0 10 0
default_queue(*) QOS-MGR 33 9 LP 49920 0 9512 0 10 923
default_queue(*) hfr_pm 34 10 LP 999936 0 639 0 10 0
default_queue(*) hfr_pm 35 11 LP 999936 0 639 0 10 0
default_queue(*) hfr_pm 36 12 LP 999936 0 639 0 10 0
<snip>
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Fabric Destination Queue Numbering
First Queue for Each MSC = (Rack * 128) + (Slot * 8)
0 0 0 HP 1 HP
Cells
BE
0 0 1 LP 2
HP
0 0 1 HP 3 BE
HP
0 1 0 LP 8 BE
0 1 0 HP 9 IngressQ
0 1 1 LP 10
0 1 1 HP 11
1 0 0 LP 128
1 0 0 HP 129
1 0 1 LP 130
1 0 1 HP 131
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Monitoring Fabric Destination Queues
Show Controllers Ingressq Block dqs Queues
DRP/0/2/CPU0:CARRIER-CLASS#
show controllers ingressq block dqs queues 0 63 loc 0/3/CPU0
Queue Fabric Dest Num QE Q Len(bytes) Packet Seq Num
--------------------------------------------------------------
16 0/2 LP 0 0 4172
18 0/2 LP 0 0 2788
24 0/3 LP 0 0 331
26 0/3 LP 0 0 5547
27 0/3 HP 0 0 6591
62 0/RP1 LP 0 0 518
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
FabricQ Topics
FabricQ queues
Ingress Monitoring queues
PSE IngressQ
Discard Bitmap
Fabric QoS
CPU
EgressQ Egress
PSE FabricQs
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
FabricQ Queues – Before Egress PSE
INT0 HP
INT1 HP
INT0 AF
Cells reassembled into packets
INT1 AF Packets queued prior to PSE
E INT0 BE Unicast queues
INT1 BE
FQ0
G ‒ Per type of service (HP/AF/BE)
F
R MCAST HP
‒ Per output interface
MCAST AF
A Multicast queues
E MCAST BE
To CPU
B ‒ Per type of service
S
R Raw (to CPU) queues
S INT2 HP
INT3 HP I ‒ 8 queues
INT2 AF C
P INT3 AF
S INT2 BE
FQ1 S3
INT3 BE
E MCAST HP
MCAST AF
MCAST BE
To CPU
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Monitoring FabricQs – Queue Length
RP/0/RP0/CPU0:GO-CISCO# show controllers fabricq queues loc 0/5/CPU0
Fabric Queue Manager Queue Information:
=======================================
Location: 0/5/CPU0 Which FabricQ ASIC?
Asic Instance: 0
Fabric Destination Address: 0
CpuCtrl Cast range : 0 - 7
Multicast Range : 8 - 71 3 FabricQ Queues per interface
Unicast Quanta in KBytes : 58, Multicast Quanta : 14
+-------------------------------------------------------------------------------------+
|Type/Ifname |Port|Queue| Q |P-quanta|Q-quanta|HighW |LowW |Q Len |BW |
| | num| num |pri| KBytes | KBytes |KBytes|KBytes|KBytes |(kbps) |
+-------------------------------------------------------------------------------------+
|Cpuctrl Cast | 0|0 - 7| HI| 13| 13| 1021| 919| 0| N/A|
|Multicast | 0| 9| BE| 13| 13| 5118| 4606| 0| N/A|
|Multicast | 0| 10| AF| 13| 13| 5118| 4606| 0| N/A|
|Multicast | 0| 11| HI| 13| 1905| 5118| 4606| 0| N/A|
|GigabitEthernet0/5/0/0| 1| 1025| BE| 13| 187| 3661| 3295| 0|1000000|
|GigabitEthernet0/5/0/0| 2| 2049| AF| 13| 187| 3661| 3295| 0|1000000|
|GigabitEthernet0/5/0/0| 3| 3073| HI| 1905| 187| 3661| 3295| 0|1000000|
|GigabitEthernet0/5/0/1| 1| 1026| BE| 13| 187| 3661| 3295| 0|1000000|
<snip>
Dropped packets
drops
+----------------------------------------------------------+
Ucast pkts : 0 (+ 0 ) Change since last
Mcast pkts : 0 (+ 0 )
Cpuctrlcast pkts : 0 (+ 0 ) time command run
Vital denied pkts : 0 (+ 0 )
NonVital denied pkts : 0 (+ 0 )
Unicast lost pkts : 0 (+ 0 )
Ucast partial pkts : 0 (+ 0 )
PSM OOR Drops : 0 (+ 0 )
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Drop on IngressQ when a FabricQ queue is
Discard Bitmap Concept congested
HW CEF TCAM Shape Queues Fabric Dest Qs F
HP A
BE
HP
B
BE Cells R
?
HP I F
BE
C A
M HP M
BE
I I B
P D PSE IngressQ D
S1
R
L P P
I
L
CPU L
I C
M A HP
A
N
Priority
Assured Forwarding
Assured Forwarding HW CEF TCAM HP N F
E
Assured Forwarding
Assured Forwarding
AF
FQ E A S2
AF
B
Best Effort
Priority BE
Assured Forwarding
Best Effort
BE
R
Priority
Best Effort
HP FabricQs I
HP
Priority
Assured Forwarding AF C
Assured Forwarding
Best Effort
AF FQ
BE
EgressQ PSE BE S3
Audience Question – What is the path for the notification?
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Discard Bitmap Example Sent by every FabricQ to every IngressQ
Rack MSC Fabric Q Priority Interface Congested
0 0 0 HP Pos 0/0/0/0 NO
0 0 0 HP Pos 0/0/0/1 NO
0 0 0 AF Pos 0/0/0/0 NO
0 0 0 AF Pos 0/0/0/1 YES
0 0 0 BE Pos 0/0/0/0 NO
0 0 0 BE Pos 0/0/0/1 NO
0 0 0 HP Multicast NO
0 0 0 AF Multicast NO
0 0 0 BE Multicast NO
0 0 1 HP Pos 0/0/0/2 NO
0 0 1 HP Pos 0/0/0/3 NO
0 0 1 AF Pos 0/0/0/2 NO
0 0 1 AF Pos 0/0/0/3 NO
0 0 1 BE Pos 0/0/0/2 NO
0 0 1 BE Pos 0/0/0/3 NO
0 0 1 HP Multicast NO
0 0 1 AF Multicast NO
0 0 1 BE Multicast NO
Audience Question: How many DB entries for a multi-chassis system with 128 8xTenGE Line Cards?
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Monitoring Discard Bitmap Drops
Show Controllers Ingressq Statistics Location <loc>
RP/0/RP0/CPU0:MULTI-CHASSIS# show controllers ingressq statistics location 0/2/CPU0
Ingressq Rx Statistics.
------------------------------------------------------------------------
rx pkts : 8098091 ( 12286872455 bytes)
rx pkts from cpu : 5350901 ( 4843791985 bytes)
<snip>
Ingressq Tx Statistics.
------------------------------------------------------------------------
tx pkts : 8098090 ( 12405816530 bytes)
tx pkts to cpu : 648410 ( 1013046860 bytes)
<snip>
tx cells to fabric : 96232534
Ingressq Drops.
------------------------------------------------------------------------
length error drops - PSE : 0
length error drops - Cpuctrl : 0
crc error drops - PSE : 0
crc error drops - Cpuctrl : 0
OOR error drops - PSE : 0
OOR error drops - Cpuctrl : 0
discard drops : 0
tail drops : 0
tail drops - no QE : 0
cell drops : 56
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Fabric QoS Introduction
Normal Traffic Pattern – No Internal Congestion
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Does This Ever Happen? Can we protect priority traffic?
Routing problems – bugs or misconfiguration
Link failures during peak traffic periods
Denial of Service attacks 100 Gbps
(CRS-1)
“It’s just like that all the time” Egress PSE
M M F F M M
PLIM I MSC I A A I
MSC I
PLIM
D D B B D D
PLIM P MSC P R F R P MSC P
PLIM
L L I A I L L
PLIM A MSC A C C A MSC A
PLIM
N N B N N
PLIM MSC MSC PLIM
E E S1 R S3 E E
Port Bandwidth
M M F I F M M
PLIM I MSC I A A I
MSC I
PLIM
C
D D B B D D
PLIM P MSC P R R P MSC P
PLIM
PLIM L
MSC L I S2 I L
MSC L
PLIM
A A C C A A
N N N N
PLIM E MSC E S1 S3 E MSC E PLIM
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Switch Fabric QoS
Managing Internal Congestion
Protects priority traffic against potential internal congestion
Potential sources of internal congestion?
Multiple switch fabric failures or accidental fabric removal
Unusual traffic patterns
F
DoS attacks A
Links down Ingress
B
IngressQ R
Routing problems PSE I F
C A
Ingress PSE classifies traffic S1 B
Via Ingress QoS per interface CPU
R
I
Full MQC/ACL matching F C
Set priority A
EgressQ Egress B S2
Set QoS Group FabricQs R
PSE I
Set Discard Class C
Via Fabric QoS S3
BE
S3
EgressQ PSE BE
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Switch Fabric QoS
Setting “Priority” in Fabric QoS Policy
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Switch Fabric QoS
Setting “Bandwidth Remaining Percent” in Fabric QoS Policy
HW CEF TCAM Shape Queues Fabric Dest Qs
HP
F
BE A
HP B
BE Cells
R
HP I F
BE
C
HP
BE
M
S1
A
I B
PSE IngressQ D
P
R
CPU L I
A C
Priority HP
Assured Forwarding
Assured Forwarding HW CEF TCAM HP N F
Assured Forwarding
AF
FQ E A
S2
Assured Forwarding
Best Effort AF
Priority
Assured Forwarding
BE B
BE
Best Effort
R
Priority
Best Effort
HP FabricQs I
HP
Priority
Assured Forwarding AF C
Assured Forwarding
AF FQ
Best Effort
BE
S3
EgressQ PSE BE
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Configuring Switch Fabric QoS
Priority Treatment for VOIP
Prefer Assured Forwarding over Best Effort (by 3:1 ratio)
Preference into Egress PSE MQC class-maps w/ limited match types
DSCP
IP Precedence
MPLS EXP
QoS Group – can be set by ingress QoS
policy-map fabric-qos Discard Class – can be set by ingress QoS
class fab-high
priority
class fabric-af
bandwidth remaining percent 75
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Egress PSE Topics
2 Stage CEF Lookup
Ingress
Multicast replication
PSE IngressQ
Egress Features*
CPU Output ACLs*
Egress Netflow*
EgressQ Egress
FabricQs
WRED*
PSE
Policing*
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Understanding 2-Stage CEF Lookup
Key Benefit – Improves Scaling
Shape Queues Fabric Dest Qs
Ingress PSE selects
HW CEF TCAM
HP • Output Line Card
• Output Interface
BE
HP
Cells
BE
• Fabric Dest Q (HP/LP)
HP
BE
• FabricQ (FQ0 or FQ1)
HP
BE
• FabricQ Queue
IngressQ • HP, AF, or BE
PSE
CPU
Priority
HP
Assured Forwarding
Assured Forwarding HW CEF TCAM AF Egress PSE selects
• Output Interface
Assured Forwarding
BE
Assured Forwarding
Best Effort HP
FQ
Priority
Assured Forwarding
AF
BE • Output Queue
FabricQs
Best Effort
Priority
Best Effort
HP
AF
• Adjacency
Priority
Assured Forwarding
BE • Dest MAC address
Assured Forwarding
Best Effort
HP FQ
AF
EgressQ PSE BE
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
CLI - CEF 2-Stage Lookup
Forwarding Table CLI Command
RP Routing show route [IP]
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
EgressQ Topics
Egress QoS
Ingress Hierarchical Queuing
PSE IngressQ
Monitoring Queues
CPU
EgressQ Egress
PSE FabricQs
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
EgressQ Overview
EgressQ
‒ Queue – Lowest Level Queuing Engine
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Monitoring EgressQ
Baseline - Interface with No QoS
RP/0/RP0/CPU0:CRS# show policy-map interface pos 0/2/0/0 output
No service policy installed
RP/0/RP0/CPU0:CRS# show qos interface pos 0/2/0/0 output Queue 9
No output QoS Policy applied on this interface ----------------------------------
Group :9
RP/0/RP0/CPU0:CRS# show controllers egressq queue from- Priority : Low
interface pos 0/2/0/0 loc 0/2/CPU0 Max LB Tokens : 38880
---------------------- Max LB Limit Index : 43
Interface POS0/2/0/0 Tokens for shaping Min LB Tokens :0
---------------------------------- 38800 = No shaping
Port 8 Min LB Limit Index : 0
---------------------------------- Quantum : 27
Max LB Tokens : 38880 Instantaneous length : 0
Max LB Limit Index : 43 Length high watermark : 823
Quantum : 27
Default Group :9 No BW guarantee Timestamp of Queue Length high watermark :
High Priority Group : N/A Wed JUN 17 11:39:05 2011
Low Priority Group : 9 ----------------------------------
---------------------------------- Queue 10
Group 9 Was it congested? ----------------------------------
---------------------------------- Group :9
Port :8 Priority : High
Priority : Low
Max LB Tokens : 38880 Max LB Tokens :0
Max LB Limit Index : 43 Max LB Limit Index : 33
Min LB Tokens :0 Min LB Tokens : 38880
Min LB Limit Index : 0 Min LB Limit Index : 57
Quantum : 27
Default Queue :9 Quantum : 27
High Priority Queue : 10 Instantaneous length : 0
Low Priority Queue : 9 Length high watermark : 0
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
EgressQ QOS - the Goal - Avoid Drops!
no drops!
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Monitoring EgressQ - HP Queue & BW Guarantee
Show Policy-map Interface (Maps MQC to HW Queue)
RP/0/RP0/CPU0:CRS# show policy-map interface Class demo-af-5
pos 0/2/0/1 output Classification statistics (pkts/bytes) (rate - kbps)
POS0/2/0/1 output: demo-output-1 Matched : 0/0 0
Transmitted : 0/0 0
Class demo-af-6 Total Dropped : 0/0 0
Classification statistics (pkts/bytes) (rate - kbps) Queueing statistics
Matched : 0/0 0 Vital (packets) : 0
Transmitted : 0/0 0 Queue ID : 28
Total Dropped : 0/0 0 High watermark (bytes) : 0
Policing statistics (pkts/bytes) (rate - kbps) Inst-queue-len (bytes) : 0
Policed(conform) : 0/0 0 Avg-queue-len (bytes) : 0
Policed(exceed) : 0/0 0 Taildropped(packets/bytes) : 0/0
Policed(violate) : 0/0 0
Policed and dropped : 0/0 Class class-default
Queueing statistics Classification statistics (pkts/bytes) (rate - kbps)
Vital (packets) : 0 Matched : 0/0 0
Queue ID : 12 Transmitted : 0/0 0
High watermark (bytes) : 0 Total Dropped : 0/0 0
Inst-queue-len (bytes) : 0 Queueing statistics
Avg-queue-len (bytes) : 0 Vital (packets) : 0
Taildropped(packets/bytes) : 0/0 Queue ID : 11
High watermark (bytes) : 0
Class demo-af-4 Inst-queue-len (bytes) : 0
Classification statistics (pkts/bytes) (rate - kbps)
Matched : 0/0 0 policy-map demo-output-1
Transmitted : 0/0 0
Total Dropped 0/0 0
class demo-af6
Queueing statistics priority
Vital (packets) : 0 police rate 2 gbps
Queue ID : 27
High watermark (bytes) : 0 class demo-af5
Inst-queue-len (bytes) : 0 Queue numbers for bandwidth percent 20
Avg-queue-len (bytes) : 0 show controllers
Taildropped(packets/bytes) : 0/0
commands class demo-af4
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. bandwidth percent
Cisco Public 10 50
HP Queue & BW Guarantee class demo-af6
priority
Show Controllers Egressq police rate 2 gbps
show controllers egressq queue from-
RP/0/RP0/CPU0:CRS# Queue 12
interface pos 0/2/0/1 loc 0/2/CPU0 ----------------------------------
Group : 10 class demo-af5
---------------------------------- Priority : High
Interface POS0/2/0/1 Max LB Tokens : 0
bandwidth percent 20
---------------------------------- Max LB Limit Index : 33
Port 12 Min LB Tokens : 38880
---------------------------------- Min LB Limit Index : 57 class demo-af4
Max LB Tokens : 38880 Quantum : 27
Max LB Limit Index : 43 Instantaneous length : 0 bandwidth percent 10
Quantum : 27 Length high watermark : 0
Default Group : 10 Timestamp of Queue Length high watermark : Wed Jun 17 11:39:05 2011
High Priority Group : N/A ----------------------------------
Low Priority Group : 10 Queue 28 (af-5)
---------------------------------- ----------------------------------
Group 10
----------------------------------
Group
Priority
: 10
: Low
7776 + 31104 = 38880
Port : 12 Max LB Tokens : 31104
Priority : Low Max LB Limit Index : 42
Max LB Tokens : 19440 Min LB Tokens : 7776
Max LB Limit Index : 41 Min LB Limit Index : 51
Min LB Tokens : 19440 Quantum : 27
Min LB Limit Index : 54 Instantaneous length : 0
Quantum : 27 Length high watermark : 0
Default Queue : 11 Timestamp of Queue Length high watermark : Wed Jun 17 11:39:05 2011
High Priority Queue : 12 ----------------------------------
Low Priority Queue : 28 Queue 27 (af-4)
Low Priority Queue : 27 ----------------------------------
Low Priority Queue : 11
----------------------------------
Group
Priority
: 10
: Low
Tokens for BW Guarantee
Queue 11 (class-default) Max LB Tokens : 34992
---------------------------------- Max LB Limit Index : 43
Group : 10 Min LB Tokens : 3888
Priority : Low Min LB Limit Index : 49
Max LB Tokens : 38880 Quantum : 27
Max LB Limit Index : 43 Instantaneous length : 0
Min LB Tokens : 0 Length high watermark : 0
Min LB Limit Index : 0 Timestamp of Queue Length high watermark : Wed Jun 17 11:39:05 2011
Quantum : 27
Instantaneous length : 0
Length high watermark : 0
Timestamp of Queue Length high watermark : Wed Jun 17 11:39:05 2011
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
CRS Packet Path Review
HW CEF TCAM Shape Queues Fabric Dest Qs F
HP A
BE
B
HP
BE Cells R
?
HP I
BE
C F
M M
HP
BE A
I I
P D PSE IngressQ D S1 B
L P P R
CPU
I L L I
M A HP
A C
N
Priority
Assured Forwarding
Assured Forwarding HW CEF TCAM HP
N F
E
Assured Forwarding
Assured Forwarding
AF
FQ E A
Best Effort AF
Priority
Assured Forwarding
BE B S2
Best Effort
BE
R
Priority
Best Effort
HP FabricQs I
HP
Priority
Assured Forwarding
Assured Forwarding
AF
AF FQ C
Best Effort
BE
EgressQ PSE BE
S3
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Introducing CRS-3
Extends CRS architecture to 140G / slot
New MSC/FP, fabric, and line cards
‒ MSC140 & FP-140
All existing chassis are upgradeable
‒ 4, 8, & 16 slot single chassis systems
‒ Multi-chassis Nx16 slot systems
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
CRS-3 PLIMs
1x100GE
14xTenGE
20xTenGE (Oversubscribed)
Adds PLIM QoS to classify traffic before Ingress PSE
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
CRS Fabric Bandwidth Comparison
CRS-1 CRS-3
2.5G x 4 = 10Gbps ingress (per plane) 5G x 5 = 25Gbps ingress (per plane)
10G * 8 planes = 80Gbps total ingress 25G * 8 planes = 200Gbps total ingress
MSC40 Fabric MSC140 Fabric
2.5G 5G
2.5G 5G
IngressQ 2.5G S1 IngressQ 5G S1
2.5G 5G
5G
2.5G 5G
2.5G 5G
FabricQ 2.5G S3 FabricQ 5G S3
2.5G 5G
2.5G 5G
2.5G 5G
FabricQ 2.5G S3 FabricQ 5G S3
2.5G 5G
CRS-1
• IngressQ to S1
‒ = 80Gbps(raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 56Gbps
• S3 to FabricQ
‒ = 160Gbps(raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 112Gbps or
56 Gbps per FabricQ
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
CRS Fabric Bandwidths and Links Between
MSC40 and MSC140
MSC40 MSC140
To Fabric IngressQ has 32 links @2.5G for 40G Scaling IngressQ to 140G requires 40 links @5G
= 32 links * 2.5Gbps *8/10 (coding) IngressQ to Fabric ASIC S1
*120/136 (cell tax) = 56Gbps
Ingress = 40 links * 5Gbps *8/10 (coding) *120/136 (cell tax)
(This is for 8 planes) = 141Gbps
(This is for 8 planes)
From 2 FabricQ ASICs per MSC, each with 2 Fabric ASIC S3 to FabricQ
Fabric RX Links per S3 ASIC. 64 links @ 2.5Gb
= 64 links * 5Gbps * 8/10 (coding) *120/136 (cell tax)
= 64 links * 2.5Gbps *8/10 (coding) = 225Gbps or 113Gbps per FabricQ.
*120/136(cell tax) = 100Gb
Egress (for 8 planes)
(this is for 8 planes)
100 Gb is split across 2 FabricQ ASICs
on the LC. Egress forwarding capacity
is only 40Gb, so the LC must
backpressure when overloaded
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
CRS-3 vs. CRS-1 Architecture Comparison
Data Path Increased to at Least 140G in all ASIC
More PPEs 140G Fabric
Faster PPEs HW CEF TCAM Shape Queues Fabric Dest Qs F
HP A
More memory BE
Higher scale HP
B
BE
Cells R
HP I F
BE
C A
M HP M
PLIM
I
BE
I B
P QoS
D PSE IngressQ D
S1 R
L P More Queues Faster CPU P I
I L 8K -> 64K CPU L C
F
M A HP
A A
N
Priority
N S2
B
Assured Forwarding
Assured Forwarding HW CEF TCAM HP
Assured Forwarding
AF
E Assured Forwarding
Best Effort AF FQ E R
Priority
Assured Forwarding
BE
BE
I
Best Effort
Priority HP FabricQs C
140G Best Effort
HP
PLIMs
Priority
AF
Assured Forwarding
Assured Forwarding
Best Effort
AF FQ S3
BE
EgressQ PSE BE
58
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
New on CRS-3 10GE LineCard’s
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
CRS-3 Line Card ASICs
PLIM MSC Fabric
160G
PSE 160G 141G
10G 2x100G IngressQ SEA
10G
PLA CPU
PLIM
PLIM
10G
10G
PLA CPU Sub-system
SEA 5G
FabricQ PODs
120G 100G 113G
100G 100G EgressQ 160G PSE
PHY PLA
Mac 2x80G 100G 113G
FabricQ
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Two Levels Egress Priority Queues
class hp-1
police rate percent 20
!
priority level 1 >>>> Default level if users didn‟t explicit use “level” statement
!
class hp-2
police rate percent 40
!
priority level 2 >>>> Only available on Taiko linecard
Note:
HP-1 queues have higher priority than HP-2 queues. Multiple HP-1 and/or HP-2 classes can be present in same
group/policy-map. In that case, same HP-1 or HP-2 hardware queue will be shared among those classes. Egress PLIM
has two queues (HP & LP) per port. The PLIM HP queues are mapped to egress MSC HP-1 queues; PLIM LP queues
are mapped to egress MSC HP-2 & LP queues.
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
CRS-3 LC Scalability
Ingress
‒ 64K Ingress queues
‒ 16K Groups
‒ 16 dedicate queues to CPU (TH-PDMA)
‒ 16K HI Queues
Egress
‒ 64K egress queues (HP and LP)
‒ 128 Ports
‒ 16K Groups
‒ 16 dedicate queues to CPU (TH-PDMA)
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Complete Your Online
Session Evaluation
Give us your feedback and you
could win fabulous prizes.
Winners announced daily.
Receive 20 Passport points for each
session evaluation you complete.
Complete your session evaluation
online now (open a browser through
our wireless network to access our Don‟t forget to activate your
portal) or visit one of the Internet Cisco Live Virtual account for access to
stations throughout the Convention all session material, communities, and
on-demand and live activities throughout
Center. the year. Activate your account at the
Cisco booth in the World of Solutions or visit
www.ciscolive.com.
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Final Thoughts
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
BRKARC-2004 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public