Cisco Systems 2005

Netflow Technical Update

on the Catalyst 6500
Carl Solder Technical Marketing Engineer, ISBU

Cisco Systems 2005

Netflow
Introduction - Where are we at today?
Tracks statistics for traffic flows through the system IPv4 statistics entries created in NetFlow table when new flows start IPv6 flows are created in Netflow table but cannot be exported For Exported records, flow removal is timer based Full collection by default Also support sampled NetFlow Flow statistics can be exported using NetFlow Data Export (NDE) NetFlow v5 and v7 NetFlow aggregation with NetFlow v8 Netflow v9 [12.2(18)SXF is Sup720 only] Theoretical maximum utilization versus effective utilization Varies based on hardware implementation and hash efficiency

Cisco Systems 2005

Netflow
Displaying flows on the system
C6500#show mls netflow ip

Displaying Netflow entries in Supervisor Earl

DstIP Pkts SrcIP Bytes Age Prot:SrcPort:DstPort LastSeen Attributes Src i/f :AdjPtr -------------------------------------------------------------------------------------------------------------------------------

10.102.130.213
7 25 10.97.36.200 10.230.215.148

10.214.39.79
17 47 10.155.22.221 10.17.64.177

tcp :46528
15:47:37 tcp :51813 15:47:39 tcp :65211

:www
:45912 :www :60425

:0x0
:0x0 :0x0 :0x0

3766 21329

L3 - Dynamic L3 - Dynamic

9
10.90.33.185 10 <>

7664
5734

17
17

15:47:38
tcp :27077 15:47:38

L3 - Dynamic
L3 - Dynamic

10.46.13.211

Cisco Systems 2005

Netflow
Flow Masks
The Catalyst 6500 supports the following flow masks - these are used to identify which pieces of information in the header will be used as input into generating a key for flow lookups

Cisco Systems 2005

Netflow
Record Types
The following record types have been defined as part of the Netflow specification

Cisco Systems 2005

Netflow
Record Types - v5 and v7

Cisco Systems 2005

Netflow
Configuring the Netflow Export Record Version
C6500(config)#mls netflow

Enable Netflow Optionally set the flow mask

destination flow keyword destination-source flow keyword full flow keyword interface-destination-source flow keyword interface full flow keyword source only flow keyword

C6500(config)#mls flow ip ? destination destination-source full interface-destination-source interface-full source

C6500(config)#mls nde sender version ? 5 7 C6500(config)#mls nde interface

Set the Netflow Record Version on PFC

Populate interface field in NDE packet Set the Netflow Export Destination

C6500(config)#ip flow-export destination 10.66.231.10 C6500(config)#interface g1/1 C6500(config-if)#ip route-cache flow

Enable Netflow on the interface

Cisco Systems 2005

Netflow
Record Types - v8
Netflow v8 flow export uses separate aggregation caches to group flow records allowing it to store a subset of the information contained in a version 5 record - this has the added benefit of reducing bandwidth requirements for exporting records and improving export record scalability - eleven aggregation methods are available

Cisco Systems 2005

Netflow
Record Types - v8
Each of the aggregation schemes contains a slightly different representation of the data contained within a full Netflow v5 record NOTE - the green ToS shows the ToS version of that aggregation scheme (i.e. AS aggregation scheme by itself does not contain the ToS information)
Cisco Systems 2005

Netflow
Configuring the Netflow v8 Aggregation Cache
Configuration of the v8 aggregation cache on the Catalyst 6500 is enabled with the following command
C6500(config)#ip flow-aggregation cache ? as AS aggregation as-tos AS-TOS aggregation bgp-nexthop-tos BGP nexthop TOS aggregation destination-prefix Destination Prefix aggregation destination-prefix-tos Destination Prefix TOS aggregation prefix Prefix aggregation prefix-port Prefix-port aggregation prefix-tos Prefix-TOS aggregation protocol-port Protocol and port aggregation protocol-port-tos Protocol, port and TOS aggregation source-prefix Source Prefix aggregation source-prefix-tos Source Prefix TOS aggregation

Cisco Systems 2005

Netflow
Record Types - v9

Support for Netflow v9 was added in IOS 12.2(18)SXF - this version of Netflow provides a more flexible format in that the sequence of data records is defined by a template that is inherently built into the exported record itself

Cisco Systems 2005

Netflow
v9 Template Flow set Field Descriptors
Built within each Flow Set Template are a number of field descriptors which can be used to define the records within the Data Flow set records

Cisco Systems 2005

Netflow
Record Types - v9 with Options Template
Option Templates can be used to provide information about the Netflow process itself - an example could be the sampling rate (i.e. one in x) used on a given interface on the Catalyst 6500

More information at http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/tflow_wp.htm

Cisco Systems 2005

Netflow
Record Types - v9 with Multicast support

http://www.cisco.com/univercd/cc/td/doc/product/software/ios123/123newft/123_1/nfmultic.htm
Cisco Systems 2005

Netflow
Configuring Netflow v9 on the Catalyst 6500
C6500(config)# mls nde sender Enable Netflow on the PFC

C6500(config)# mls flow ip interface-full

Set flow mask

Enable Netflow v9

C6500(config)# ip flow-export version 9

C6500(config)# ip flow-export destination 10.10.10.1 2111

Set Export Destination

If you wanted to enable v9 export of Multicast data, you can enable this as follows
C6500(config)# interface gigabitethernet 3/1 C6500(config-if)# ip multicast netflow ingress Enable Netflow v9 ingress Multicast collection Enable Netflow v9 egress Multicast collection

C6500(config-if)# interface gigabitethernet 3/2 C6500(config-if)# ip multicast netflow egress

Cisco Systems 2005

Netflow
Sampled Netflow
The Catalyst 6500 supports both full and sampled Netflow record collection - both options are configurable on the switch - Sampled Netflow on the Sup720 uses a full interface flow mask Sampling Rate options - one in every 64, 128, 256, 512, 1024, 2048, 4096 or 8192

Cisco Systems 2005

Netflow
Netflow Capacities across the Supervisor family
Each of the Supervisors support for Netflow yields a different number of flows that can be stored in the Netflow tables - the table below provides a summary of the Netflow capacities for each of the Supervisors

Table Size
Sup2 Sup720 Sup720-3B 128K 128K 128K

Hash Efficiency
25% 50% 90%

Effective Size
32K 64K 115K

Hash Key Size

17 bits 36 bits 36 bits

Sup720-3BXL
Sup32-8GE Sup32-10GE Sup720-10GE-3C

256K
128K 128K 128K

90%
90% 90% 90%

230K
115K 115K 115K

36 bits
36 bits 36 bits 36 bits

Sup720-10GE-3CXL

256K

90%

230K

36 bits

Cisco Systems 2005

Netflow
Architecture
Built within the PFC on the Supervisor are multiple sets of specialized memory each dedicated to storing different pieces of information - for the purposes of Netflow there is a TCAM and two sets of SRAM that, in combination, provide the ability to store information about flows in the system

Cisco Systems 2005

Netflow
Architecture

Netflow Key Table Entry Record Netflow Statistics Table Entry Record

Netflow Key Table Entry

IPv4 Key Table Entry Protocol/ Mask 4 VLAN/ VPN 12 Protocol Type 8 IP DA 32 IP SA 32 SRC Port 16 DST PORT 16 Xtag 4 VPN Valid 1 Re-Circ 1 Central Rewrite 1 Primary Input 1

Netflow Stats Table Entry

First Packet Seen 1 FIN/RST Create Time 22 Last Seen timestamp 24 Byte Count 40 Packet Count 32 Threshold Exceeded Count 39 Bucket Count 25 RPF Fail Cache Update 1 Control Bits 10

Cisco Systems 2005

Netflow
Netflow Step by Step on the PFC3

Cisco Systems 2005

Netflow
Netflow Hash Collision

Cisco Systems 2005

Netflow
Utilization of Netflow TCAM and SRAM Resources
If a flow hashes to the same location as an existing flow, while the packet is still switched, the flow record is not created. Netflow tables are a finite resource, and as such need to be managed to avoid the situation where flow records are not kept
C6500#show mls netflow table-contention detailed Earl in Module 6 Detailed Netflow CAM (TCAM and ICAM) Utilization ================================================ TCAM Utilization ICAM Utilization Netflow TCAM count Netflow ICAM count Netflow Creation Failures Netflow CAM aliases : : : : : : 100% 0% 130944 0 270274 0

Cisco Systems 2005

Netflow
Netflow Aging
Tuning of Netflow aging parameters is a solution to managing the Netflow Table resource Aging is used to define when flows are to flushed from the Netflow tables Three aging parameters to consider Normal - fixed idle time for flows Fast - Threshold based aging for flows Long - Maximum lifetime for flows NOTE - Normal and Long Aging enabled by default: Fast aging is disabled by default Timers are by default CONSERVATIVE

Cisco Systems 2005

Netflow
Netflow Aging
C6500#show mls netflow aging enable timeout packet threshold ------ ------- ---------------normal aging true 300 N/A fast aging false 32 100 long aging true 1920 N/A Feature Aging Feature ------NAT_INGRESS NAT_EGRESS NAT_INGRESS NAT_EGRESS C6500#

Pattern ------4 4 3 3

Agetime ------300 300 300 300

Cisco Systems 2005

Netflow
What can you do with the information? Answer = Plenty!!

Cisco Systems 2005

Netflow
Lets look at the Device List

Cisco Systems 2005

Netflow
Lets choose the Catalyst 6500

Cisco Systems 2005

Netflow
Traffic is broken up by interface - Lets inspect VLAN 64

Cisco Systems 2005

Netflow
Traffic now broken up as IN/OUT traffic

Cisco Systems 2005

Netflow
We can zoom in on a specific time interval

Cisco Systems 2005

Netflow
Now I can see individual Source IP Address info

Cisco Systems 2005

Netflow
Another mouse click away and more info

Cisco Systems 2005

Netflow
Getting to the specifics

Cisco Systems 2005

Netflow
We can also zoom in on specific SRC address info

Cisco Systems 2005

Netflow
Here is who 10.66.236.94 has been talking to

Cisco Systems 2005

Netflow Case Study

Tracking the Hacker at a University customer

Cisco Systems 2005

Netflow
Internal Netflow Resources to check out
Netflow on the Catalyst 6500 White Paper (Marco Foschiano) (includes updated section on Netflow v9) http://wwwin-eng.cisco.com/Eng/ISBU/TME/Netflow_6500_7600.pdf Netflow Performance on the Sup720-3BXL http://bockbock/~icox/presentations/Netflow_Performance_May_2005_subset.ppt Netflow on the Catalyst 6500 and Cisco 7600 Presentation http://bock-bock/~icox/presentations/CCIE_Nov_2003-NDE-WAN_white.ppt Netflow on CISCO.COM http://www.cisco.com/go/netflow

Cisco Systems 2005

 Cisco Systems 2005