Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Frederic Detienne
Agenda
Platform and Hardware Architecture
Software Architecture
Day in the the Life of a Normal Packet
Advanced Example: IPsec Control Plane Programming
Debugging strategies
Road to Simplification: Part I, Data Plane Debugging
Understanding and Extracting ESP Logs
Road to Simplification: Part II, Control Plane Unified Show Commands
Road to Simplification: Part III, Deep Data Plane Debugging
Future: Resource Consumption Monitoring
Wrapping up...
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Objectives
Understand the ASR 1K and ISR 445x architecture
– software
– hardware
– relationship between the two
Understand how features process packets through IOS-XE
Understand how to easily debug the platform
– long journey
– presentation of recent serviceability enhancements
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Platforms and Hardware Architecture
Cisco ASR 1000 Series Routers: Overview
Compact, Powerful Router Business-Critical Resiliency Instant-on Services Delivery
Instant On
Service Delivery
Line-rate performance 2.5G to Fully separated control and Integrated firewall, VPN,
200G+ with services enabled forwarding planes encryption, DPI, CUBE
Investment protection with modular Hardware and software Scalable on-chip service
engines, IOS CLI and SPAs for I/O redundancy provisioning through software
licensing
Hardware based QoS engine with In-service software upgrades
up to 472K queues
SPAs
4 x 1GE
ESP
2RU
RP/SI
P
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Chassis Options: ASR 1004
SPAs
SIP
ESP 4RU
RP
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Rack Mount &
Chassis Options: ASR 1006 Cable Management
SPAs
SIP
6RU
ESP
RP
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
ASR1K Building Blocks
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
QFP
Assist. PPE BQS Assist. PPE BQS
Route Processor
interconn. Handles control plane traffic interconn.
Manages system
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
System Architecture Control Plane
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
Assist. PPE BQS Assist. PPE BQS
EOBC switch in RP
interconn. interconn.
Midplane
Ethernet Out of Band Channel
(aka EOBC)
Inter Integrated Circuit (I2C) Bus 1Gbps Ethernet bus
SIP interconn. SIP interconn. SIP interconn. Used by RP to program system
Slow (few kbps)
Used for system monitoring Used by system to notify RP
(temp., OIR, fan speed,…)
SPA SPA SPA
IOCP IOCP IOCP
Aggreg. Aggreg. Aggreg.
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
System Architecture Forwarding Plane
Hypertransport
10 Gbps Ethernet
ESP RP RP ESP
Active
FECP CPU CPU FECP
Stby
Active
Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
Assist. PPE BQS Assist. PPE BQS
Embedded Service Interconnect
aka ESI Bus
interconn. 11.2 – 40 Gbps Forwarding Bus interconn.
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
RP
CPU
Route Processor Architecture
interconn. GE switch
Highly Scalable Control Plane Processor
Route Processor
System Logging
Manages all chassis functions Not a traffic interface!
Core Dumps
Runs IOS Management only
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
ESP
FECP
intercon.
EEPROM
QFP
DDRAM Packet Processor Engine BQS
PCI* E-RP*
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430) SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
ESP
FECP
PCI* E-RP*
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430) SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
ESP200 Block Diagram
Packet Buffer Packet Buffer
TCAM Resource DRAM Resource DRAM
DRAM DRAM
(80Mbit) (2GB) (2GB)
(512MB) (512MB)
Dispatcher Dispatcher
DDRAM Packet Packet
Buffer Buffer
Dispatcher Dispatcher
Packet Packet
Buffer Buffer
Memory Crypto
RPs RPs SIPs
BRKCRS-3147
ESP RPs
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
ESI Capacity by ESP-xxx and SIP-xxx
Enhanced SerDes Interconnect (ESI) links over midplane
carry
– packets between ESP and other cards (SIPs, RP & other ESP)
QFP Complex – network traffic to/from SPA SIP’s
– punt/inject traffic to/from RP
11.2Gbps 11.2Gbps 25.6Gbps 40+G I/L – state synchronization to/from standby
SPI4.2 SPI4.2 eSPI
ESP-10G Interc. Additional full set of ESI links to/from standby ESP (not
ESP-10G Interc. shown)
ESP-40G:
– 2x23G ESI* to all three SIP slots
– could also support a 6-SIP chassis with 1 ESI to each (e.g. voice
application)
– also 23G between two ESP-40G’s
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Embedded Services Processor – The Real Thing
Interconnect ASIC
SPI MUX
TCAM Crypto
Engine
FECP
QFP Subsystem CPU
PPE + BQS
FECP
DRAM
PPE BQS
DRAM Packet
DRAM
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Cisco “Quantum Flow Processor”
Feature Summary
intercon.
EEPROM
QFP Complex
DDRAM Packet Processor Engine BQS
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Hypertransport, 10Gbps
Other
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
SIP intercon.
SPA
IOCP
Aggreg.
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) (SC854x SOC)
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
SIP intercon.
SPA
IOCP
Aggreg.
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) SPA Aggregation
(SC854x SOC) Forwards and queues
packets (FIFO)
JTAG Ctrl
SPA Aggregation Network
IO Control Processor clock
ASIC (Marmot) … distribution
Manages SPA OIR & drivers …
Linux Kernel
Ingress buffers Egress buffers
(per port) (per port)
Network
clocks
Ingress Classifier
Reset / Pwr Ctrl SPA Agg. C2W
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
SPA Interface Processor – SIP-10G
Physical termination of SPA
Supports up to 4 SPA's
– 4 half-height, 2 full-height, 2 HH+1FH
– full OIR support
Does not participate in forwarding
Limited QoS
– Ingress packet classification – high/low
– Ingress over-subscription buffering (low
priority) until ESP can service them. Up to
128MB of ingress oversubscription buffering
Capture stats on dropped packets
Network clock distribution to SPA's, reference
selection from SPA's
IOCP manages Midplane links, SPA OIR, SPA
drivers
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
ISR 4451-X Hardware Diagram
DDR3
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
ISR 4451-X Hardware Diagram (comments)
10 Cores, 1 thread / core Inline Cryptography
5 fwd cores by default No Crypto Assist chip
4 remaining cores license Crypto “locks” core
activated True run-to-completion
3 Services Core
No hardware TCAM
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
ISR 4451 System Layout (2RU Platform)
Dataplane
DIMM Control & Services Dual
DSP Slot
DIMM
External Dataplane Control & Services CPU
Serviceable CF CPU
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Acronyms
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Software Architecture
ASR1K Software Architecture
RP
CPU
RP
Chassis Manager
IOS CPU
interconn.Linux GE switch
Kernel
ESP ESP
FECP
Chassis Manager
EOBC (1 Gbps) FECP
Drivers Forwarding Manager
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Chassis Manager (CM)
RP CM on RP communicates with CM processes
CPU on ESP and SIP
Chassis Manager – Distributed function
IOS
ESI (10-40 Gbps)
Forwarding Manager Initializes hardware and boots other processes
Linux Kernel – CM on SIP queries SPA type and load SPA
drivers
Manages hardware components
ESP FECP Chassis Manager – Manages EOBC on RP
– Manages ESI links on RP/ESP/SIP
EOBC (1 Gbps)
µ µµ
QFP Communicates IOS hardware components
Crypto
µ BQS
Assist. – Static & OIR
µ µ
ESI (10-40 Gbps)
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Forwarding Manager (FMAN)
RP
CPU
FMAN on RP communicates with
FMAN-RP
Chassis Manager FMAN process on ESP
IOS
– Distributed function
ESI (10-40 Gbps)
Forwarding Manager
Drivers
Drivers
Drivers
Forwarding Manager
FMAN-FP communicates
I2C
SIP
IOCP
FMAN on active RP maintains
SPA Driver
SPA Driver
Chassis state for both active & standby
SPA Driver Manager
ESP’s
Linux Kernel
– Facilitates NSF after re-start with bulk
SPA SPA SPA
download of state information
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
PPE Microcode
RP
CPU Written in C
ESI (10-40 Gbps)
IOS
Chassis Manager – proper features, no hack
Forwarding Manager
Runs on each thread of the PPC
Linux Kernel
Processes packets
ESP FECP Chassis Manager – run to completion
– assisted by various memories
EOBC (1 Gbps)
Linux Kernel
µ µµ
QFP
QFP
Packet Processor Engine
PPE PPE PPE PPE PPE
1 2 3 4 5
BQS
BQS
Crypto
Features applied via FIA
µ …
– Feature Invocation Array
PPE PPE PPE PPE
6
µ µ
7 8 N
Assist.
Dispatcher
Packet Buffer
ESI (10-40 Gbps)
SIP
IOCP
SPA Driver Chassis
FIA per interface
– input FIA, output FIA
SPA Driver
SPA Driver Manager
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
show platform hardware qfp active interface if-name GigabitEthernet 0/0/0
SPA
IOCP
Aggreg.
ESPs
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) (SC854x SOC)
…
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
SPA
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
ESP
FECP
intercon.
EEPROM
QFP Complex
DDRAM Packet Processor Engine BQS
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
ESP
FECP
intercon.
EEPROM
PPE2 QFP Complex
DDRAM Packet Processor Engine BQS
Thread 1
Thread 2
Thread 3
Thread 4
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
ESP
FECP
intercon.
EEPROM
PPE2 QFP Complex
DDRAM Packet Processor Engine BQS
Thread 1
Thread 2
Thread 3
Thread 4
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
ESP
FECP
intercon.
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify IP Unicast …
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN
MQC Policing
NAT IP Multicast
MAC Accounting
PBR
Dispatcher WRED
Dialer IDLE Rst
Packet For
Packet Buffer
Us Queuing
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
ESP
FECP
intercon.
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify IP Unicast …
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN
MQC Policing
NAT IP Multicast
MAC Accounting
PBR
Dispatcher WRED
Dialer IDLE Rst
Packet For
Packet Buffer
Us Queuing
PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
ESP
FECP
intercon.
EEPROM
QFP Complex
DDRAM Packet Processor Engine BQS
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
SIP intercon.
SPA
IOCP
Aggreg.
ESPs
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) (SC854x SOC)
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
SPA
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
An Advanced Example:
IPsec control plane programming
IPsec SA – from IOS to FMAN-FP
show crypto ipsec sa interface virtual-access 1002
RP
CPU
interface: Virtual-Access1002
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1 Chassis Manager
IOS
protected vrf: (none)
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0)
Forwarding Manager
remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0)
current_peer 17.0.0.26 port 500 Linux Kernel
… show platform software ipsec fp active flow identifier <flow_id>
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008
… …
conn id: 30214, flow_id: HW:28214, sibling_flags FFFFFFFF80000008, QFP SA handle: 1892
crypto map: Virtual-Access1002-head-0 …
Crypto SA ctx id: 0x000000002e02b9b6
outbound esp sas:
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
…
conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008,
crypto map: Virtual-Access1002-head-0
…
ESP FECP
Chassis Manager
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
…
conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008,
crypto map: Virtual-Access1002-head-0
…
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822)
… show platform software ipsec fp active
conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008, encryption-processor context 2e02b9b6
crypto map: Virtual-Access1002-head-0
…
=======Context id: 0x02b249
…
SA word 0: 0x5ae0460fc201aa5
action bits: 0x001f84
show platform hardware qfp active classification feature-manager ESP FECP direction: outbound
Chassis Manager
class-group tcam ipsec <SPD-id> global detail mode: transport
protocol: esp
Drivers Forwarding Manager authentication: SHA-1
class-group [ipsec-cg:1008] (classes: 1, total number of vmrs: 4) Drivers
Drivers confidentiality: AES-128
key name: 160_03 value size: 160 result size: 16
Linux Kernel …
region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0
mfs: 1454
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 …
QFP sequence number: 306
Result: : 40000000 905a0400 00000000 00000000 µ µµ Crypto
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000 µ µ Assist. byte count: 25704
packet count: 306
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000
Result: : 20000000 8d458860 00000000 00000000
… TCAM DRAM DRAM
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
IPsec SA – from FMAN-FP to QFP DRAM
show crypto ipsec sa interface virtual-access 1002
RP
CPU
interface: Virtual-Access1002
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1 Chassis Manager
IOS
protected vrf: (none)
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0)
Forwarding Manager
remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0)
current_peer 17.0.0.26 port 500 Linux Kernel
… show platform software ipsec fp active flow identifier <flow_id>
inbound esp sas:
spi: 0x956A1B11(2506758929) SPD id: 1008 FMAN-FP knows
… … everything
conn id: 30214, flow_id: HW:28214, sibling_flags FFFFFFFF80000008, QFP SA handle: 1892
crypto map: Virtual-Access1002-head-0 show platform hardware qfp active feature ipsec sa <qfp_sa_handle> …
Crypto SA ctx id: 0x000000002e02b9b6
outbound esp sas: QFP ipsec sa Information
EOBC (1 Gbps)
spi: 0x51E3FC8E(1373895822) Also indexed by
… QFP sa id: 3623 class-group show platform software ipsec fp active
conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008,
pal sa id: 32085 encryption-processor context 2e02b9b6
crypto map: Virtual-Access1002-head-0 QFP spd id: 3398
… QFP sp id: 1066 =======Context id: 0x02b249
QFP spi: 0x51E3FC8E(1373895822) …
crypto ctx: 0x000000002e02b9b6 SA word 0: 0x5ae0460fc201aa5
… action bits: 0x001f84
show platform hardware qfp active classification feature-manager ESP FECP direction: outbound
Chassis Manager
class-group tcam ipsec <SPD-id> global detail mode: transport
protocol: esp
Drivers Forwarding Manager authentication: SHA-1
class-group [ipsec-cg:1008] (classes: 1, total number of vmrs: 4) Drivers
Drivers confidentiality: AES-128
key name: 160_03 value size: 160 result size: 16
Linux Kernel …
region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0
mfs: 1454
Value: : ac120001 2f000000 00000000 1100001a 12d70000
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 …
QFP sequence number: 306
Result: : 40000000 905a0400 00000000 00000000 µ µµ Crypto
…
µ BQS
Value: : ac120001 2f000000 00000000 1100001a 12d70000 µ µ Assist. byte count: 25704
packet count: 306
Mask: : ffffffff ff000000 00000000 ffffffff ffff0000
Result: : 20000000 8d458860 00000000 00000000
… TCAM DRAM DRAM
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
ESP
FECP
EEPROM
QFP Complex
DDRAM Packet Processor Engine BQS
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
ESP
FECP
intercon.
EEPROM
QFP Complex
DDRAM PPE may be different butPacket
packetProcessor Engine BQS
processing continues where it
Boot Flash
stopped (right after crypto)
PPE1 PPE2 PPE3 PPE4 PPE5
(OBFL,…) FECP
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
ESP
FECP
ZB-FW config objects SPI Mux 5, 10, 20, 40, 100, 200 Gbps
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Hypertransport, 10Gbps
Other
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Debugging strategies
Everyday situations
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Using statistics for troubleshooting packet drops
Not easy… not very practical either.
ESP Let’s dig deeper before making it simpler
SPA
show platform hardware slot {f0|f1} serdes statistics
show interfaces <interface-name>
show platform hardware slot {f0|f1} serdes statistics internal
show interfaces <interface-name> accounting
show platform hardware qfp active bqs 0 ipm mapping
show interfaces <interface-name> stats show platform hardware qfp active bqs 0 ipm statistics channel all
show platform hardware qfp active bqs 0 opm statistics channel all
show platform hardware port <slot/card/port> plim statistics
show platform hardware qfp active statistics drop [detail]
show platform hardware subslot {slot/card} plim statistics
show platform hardware qfp active interface if-name <Interface-name> statistics
show platform hardware slot {slot} plim statistics show platform hardware qfp active infrastructure punt statistics type per-cause |
exclude _0_
show platform hardware slot {0|1|2} plim status internal
show platform hardware qfp active infrastructure punt statistics type punt-drop |
exclude _0_
show platform hardware slot {0|1|2} serdes statistics
show platform hardware qfp active infrastructure punt statistics type inject-drop
| exclude _0_
RP
show platform hardware qfp active infrastructure punt statistics type global-drop
| exclude _0_
show platform hardware slot {r0|r1} serdes statistics
show platform hardware qfp active infrastructure bqs queue output default all
show platform software infrastructure lsmpi
show platform hardware qfp active infrastructure bqs queue output recycle all
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Debugging Strategies to Date
IOS Control Plane
Well Known
• show interface
• show ip route, show bgp …
Top Down
Platform Control Plane
Bottom Up
Very Difficult
• ESP “stuff”
• e.g. show platform … Let’s change
that!!
Data Plane
• ESP “stuff”
• e.g. show platform …
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
The Road to Simplification:
Part I, Data Plane Debugging
55
IOS 3.7
The Embedded Packet Capture
One way of capturing packets…
0
0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E.
0010: 00300000 00000111 CFDC091D 0002E000 .0..............
0020: 000207C1 07C1001C 802A0000 10030AFA .........*......
0030: 1D006369 73636F00 0000091D 0001 ..example....... Excellent tool but insufficient in many cases
1
0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F.
0010: 00200000 00000102 44170000 0000E000 . ......D.......
0020: 00019404 00001700 E8FF0000 0000 .............. http://www.cisco.com/en/US/docs/ios-
xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capture-
2 xe.html
0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E.
0010: 00300000 00000111 CFDB091D 0003E000 .0..............
0020: 000207C1 07C1001C 88B50000 08030A6E ...............n
0030: 1D006369 73636F00 0000091D 0001 ..example.......
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
IOS 3.10
The Packet Tracer and FIA Debugger
X-Connect Reset / Pwr Ctrl
L2 Switch IPv4 IPv6 MPLS
TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
Condition determines packets DRAM SRAM
Temp Sensor
to be traced
Input FIA Output FIA Input ACL
EEPROM
Pak Trace ? PPE2 QFP Complex
MQC Classify
DDRAM Packet Processor Engine BQS
Output ACL NAT
Boot Flash Input ACL PPE1 PPE2 PPE3 PPE4 PPE5
(OBFL,…) FECP
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify NAT PBR
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Output ACL
NAT Encaps
IP Unicast
Statistics and final action will be NAT
PBR Dispatcher Crypto
collected (matched packets dropped,
Packet Buffer punted to RP, forwarded to output
PPE2 Encaps
interface …)
Crypto Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
Optionally, FIA actions can logged per packet
System can capture several packets flows
RPs RPs ESP RPs SIPs Packet flows can be reviewed in show commands
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Packet Tracer Demonstration
Packet-Trace: Configuration Commands
The Pactrac (Packet Tracer) shows us what happens to a series of packets
– True inspection of IOS XE packet forwarding flow
debug platform packet-trace enable
– Enables accounting
– Required for all levels of inspection
debug platform packet-trace packet <pkt-num> \
[fia-trace | summary-only] [circular] [data-size <data-size>]
– Required for any per-packet data capture (e.g. necessary for packet copy to function)
– Specifies maximum number of packets maintained at one time (<pkt-num>)
– Always enables capture of summary data or only summary data (summary-only)
– Captures feature path data by default
– Optionally performs FIA trace (fia-trace) in addition to path data capture
– Allows specifying the size of the path data buffers (defaults to 2048)
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Packet-Trace: Configuration Commands
debug platform packet-trace copy packet {in | out | both} [L2 | L3 | L4] [size
<num-bytes>]
– Enables copy of the ingress and/or egress packets
– Optionally allows specifying where to start copy of the packet (L2 is default)
– Optionally allows specifying the maximum number of octets to copy (64 is default)
Available XE3.11 and forward
debug platform packet-trace drop [code <code-num>]
– Enables retention only for dropped packets
– Optionally allows specifying retaining packets for a specific drop code
– Can be used without global/interface conditions to capture drop events*
*Drop event capture means the only the drop itself is traced not the life of the
packet, but, it still allows capture of summary data, tuple data and the packet to
help refine conditions or provide clues to the next debug step.
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Packet-Trace: Configuration Commands
clear platform packet-trace statistics
– Clears any collected statistics and data buffers
– Tracing must be stopped first (debug platform condition stop)
clear platform packet-trace configuration
– Removes all debug platform packet-trace commands
clear platform condition all
– Removes all debug platform condition and debug platform packet-trace commands
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Packet-Trace: Configuration Commands
Packet-trace relies on the conditional infra to determine which packets are
interesting. The condition infra provides the ability to filter by protocol, IP address
and mask, ACL, interface and direction. A complete discussion of conditions is
not made here but some illustrative examples are:
debug platform condition ingress
– Checks all incoming packets on all interfaces for all protocols
debug platform condition interface g0/0/0 ipv4 ingress
– Checks all IPv4 packets arriving on interface g0/0/0
debug platform condition interface g0/0/0 ipv4 access-list FOO ingress
– Checks incoming IPv4 packets on interface g0/0/0 that match access-list FOO
Conditions are activated or de-activated using debug platform condition start or
debug platform condition stop respectively.
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Packet-Trace: Configuration Commands
NOTA BENE!!!!!
Conditions define what the filters are and when the filters are applied to a
packet. For example, debug platform condition interface g0/0/0 egress means
that a packet will be identified as a match when it reaches the output FIA on
interface g0/0/0 so any packet-processing that took place from ingress up to that
point is missed.
Best Practice
It is highly recommended to use ingress conditions for pactrac to get the most
complete and meaningful data. Egress conditions can be used but just be aware
of the limitation above.
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Packet-Trace: Configuration Example
The following shows how one would trace the most recent 128 packets entering
GigabitEthernet0/0/0 including FIA trace and a copy of up to the first 2048 octets
of the input packet.
debug platform condition interface g0/0/0 ingress
debug platform packet-trace enable
debug platform packet-trace packet 128 fia-trace circular
debug platform packet-trace copy packet input size 2048
debug platform condition start
<…wait until you’ve captured the packets you think you want…>
debug platform condition stop
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Packet-Trace: Configuration Highlights
Pactrac buffers consume QFP DRAM
– Be mindful of how much memory a config needs and how much memory is available
Configure as much detail as you want…more detail…more performance impact
for matched packets
Each pactrac “config” change will temporarily disables pactrac and clears
counts/buffers
– “Cheap” way of ‘debug plat cond stop’, ‘clear plat pack stats’ and ‘debug plat cond start’
Some configs require a ‘stop’ in order to display summary or per packet data
– Currently circular and drop tracing
Conditions define where and when filters are applied to a packet
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Packet-Trace: Show Commands
Show commands are used to display pactrac configuration and each level of data:
show platform packet-trace configuration
– Displays packet-trace configuration including any defaults
show platform packet-trace statistics
– Displays accounting data for all pactrac packets
show platform packet-trace summary
– Displays summary data for the number of packets specified by debug platform packet-trace
packet
show platform packet-trace packet { all | <pkt-num>} [decode]*
– Displays all path data for all packets or the packet specified
– Decode attempts to display packets captured by debug platform packet-trace copy in user
friendly way
– * decode was introduced in XE3.11
NOTE: only a few protocol headers are supported initially (ARPA, IP, TCP, UDP,
ICMP)
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Example of Packet-Trace Configuration
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Example of Packet-Trace Accounting
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Example of Packet-Trace Summary
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Example of Packet-Trace Packet Details
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Example of Clearing Packet-Trace Stats
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Understanding and Extracting ESP Logs
72
ESP Tracing aka Logging
TEMP RAM FS
RP RP logs are first written
CPU
Chassis Manager here (efficiency)
IOS
NFS Shared Disk
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Important logs
RP
CPU
Chassis Manager
fman_rp_R[0|1]-0.log
IOS
SIP
IOCP
SPA Driver Chassis
SPA Driver
SPA Driver Manager
Linux Kernel
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
What log files are important?
All these logs get rotated and are copied to /harddisk/tracelogs directory
on active RP.
Look for the relevant log files depending on the time of the failure
By default, all ERR messages are logged, these should be the first things
to look for
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Example log files
The timestamp…
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/
3768365 -rwx 1048934 Jan 6 2014 18:20:16 +00:00 cpp_cp_F0-0.log.7133.20140106182015
3768330 -rwx 551643 Jan 7 2014 09:27:51 +00:00 cpp_cp_F0-0.log.7133.20140107092751
3768335 -rwx 1048901 Jan 7 2014 08:56:44 +00:00 cpp_cp_F0-
0.log.7133.2014010708564339313059840 bytes total (30680653824 bytes free)
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Rotating the log files
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/traceMy-ASR1000-2#test platform software trace slot rp active forwarding-
manager rotate
Rotated file from: /tmp/rp/trace/stage/fman_rp_R0-0.log.13836.20140107094754, Bytes: 0, Messages:
6535
My-ASR1000-2#test platform software trace slot FP active cpp-control-process rotate
Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7133.20140107093650, Bytes: 154027,
Messages: 786
My-ASR1000-2#test platform software trace slot FP active forwarding-manager rotate
Rotated file from: /tmp/fp/trace/stage/fman-fp_F0-0.log.8247.20140107093738, Bytes: 20170,
Messages: 210
OR use
My-ASR1000-2#request platform software trace rotate all Does not show the rotated file names w/
time stamp have to hunt them down
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
The Road to Simplification – Part II
Control Plane Unified Show Commands
78
Simplifying the IPsec show commands
One show command to rule them all
interface: Virtual-Access1002
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1
------------------ show platform software ipsec fp active flow identifier 34130 ------------------
protected vrf: (none) …
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) ------------------ show platform hardware qfp active feature ipsec sa 1427 ------------------
current_peer 17.0.0.26 port 500 …
PERMIT, flags={origin_is_acl,} ------------------ show platform software ipsec fp active encryption-processor context 6502aa4f ------------------
#pkts encaps: 25227, #pkts encrypt: 25227, #pkts digest: 25227
…
#pkts decaps: 25237, #pkts decrypt: 25237, #pkts verify: 25237
#pkts compressed: 0, #pkts decompressed: 0 ------------------ show platform software ipsec fp active flow identifier 34129 ------------------
#pkts not compressed: 0, #pkts compr. failed: 0 …
#pkts not decompressed: 0, #pkts decompress failed: 0 ------------------ show platform hardware qfp active feature ipsec sa 1867 ------------------
#send errors 0, #recv errors 0
…
local crypto endpt.: 172.18.0.1, remote crypto endpt.: 17.0.0.26 ------------------ show platform software ipsec fp active encryption-processor context 2e02aa4e -----------------
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/2 …
current outbound spi: 0xA7B61FE5(2813730789)
PFS (Y/N): N, DH group: none
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
The Road to Simplification
Part III, Deep Data Plane Debugging
81
IOS 3.11
The Packet Tracer and FIA Debugger
X-Connect Reset / Pwr Ctrl
L2 Switch IPv4 IPv6 MPLS
TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
DRAM SRAM
Temp Sensor
Thread 1
Thread 2
Thread 3
Thread 4
MQC Classify NAT PBR
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Output ACL
NAT Encaps
IP Unicast
NAT
PBR Dispatcher Crypto
Packet Buffer Cond Dbg ?
PPE2 Encaps
Crypto If Conditional Debugging is on for Thread 3 Crypto
SPI Mux feature AND if packet needs to be
Reset / Pwr Ctrl
traced… feature will log its action step
SA table by step in cpp_cp_f0-0.log !!
DRAM Interconnect
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Platform Conditional Debugging
BGL.D.16-ASR1000-1# debug platform condition feature ?
atm ATM feature
atom ATOM feature
bridge-domain Layer2 bridging feature
cft CFT feature
cxsc CXSC feature
evc EVC feature
fw FW feature Debugs get populated in cpp_cp_F0-0.log
ipsec IPSEC feature
nbar NBAR feature
otv OTV feature
subscriber Subscriber feature
vpls VPLS feature
Same match statement as
packet tracer…
BGL.D.16-ASR1000-1#debug platform condition ipv4 172.19.2.1/32 ingress
BGL.D.16-ASR1000-1#debug platform condition feature ipsec dataplane submode cce level info
BGL.D.16-ASR1000-1#debug platform condition start
Tells which feature to
debug
Start and stop debugging
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Conditional Debugger Demonstration
Checking Resource Usage
Coming your way in an IOS-XE near you…
85
Unified show CPU platform summary
Core 0: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 1%
Core 1: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 1%
PID Runtime(ms) uSecs 5Sec 1Min 5Min TTY Process
1 1102 1800 0.20% 0.50% 0.30% 0 init
3 100 1000 0.00% 0.00% 0.05% 0 events/0
4 100 200 0.00% 0.00% 0.00% 0 khelper
6 200 200 0.70% 0.10% 0.00% 0 kthread
…
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Unified show memory platform summary
show memory platform summary
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Wrapping up…
88
New Debugging Strategy
Data Plane
• Packet Tracer
Easy!! • Forwarding plane conditional debugging
• Embedded Packet Capture
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Call to Action…
Visit the World of Solutions:-
Cisco Campus
Walk-in Labs
Technical Solutions Clinics
Recommended Reading: For reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2014
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Complete Your Online Session Evaluation
Complete your online session evaluation
Complete four session evaluations
and the overall conference evaluation to receive your Cisco Live T-shirt
BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 91