FTD TF PDF

Dissecting Firepower-
NGFW(FTD)
“Installation & Troubleshooting”
Veronika Klauzova
BRKSEC-3455
Cisco Spark
Questions?
Use Cisco Spark to communicate Haitham Jaradat
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space John Groetzinger
Cisco Spark spaces will be

available until July 3, 2017. cs.co/clus17/#BRKSEC-3455
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
For Your
Reference
Related sessions - You don’t want to miss at #CLUS
BRKSEC-2020
BRKSEC-2050
Firepower NGFW TECSEC-3301
Firepower NGFW
Deployment in the Data Firepower Data-Path
Internet Edge
Center and Enterprise Troubleshooting
Deployment Scenarios
Network Edge using John Groetzinger
Jeff Fanelli
FTD
Steven Chimes
BRKSEC-3455 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
For Your
Reference
Related sessions - You don’t want to miss at #CLUS
TECSEC-2004
BRKSEC-3020
BRKSEC-3035 Troubleshooting FTD
Troubleshooting ASA
Firepower Platform like a TAC Engineer
Firewalls
Deep Dive Ben Ritter
Kevin Klous
Andrew Ossipov Kevin Klous
Your presenter throughout FTD journey
• Firepower TAC engineer
• Originally from
• Working in
• Slavic countries accent Veronika Klauzova
Agenda
• Introduction
• Hardware & Software review
• Installation and Configuration
• Device registration troubles
• FTD Data-Flow: life of a packet
• Troubleshooting & Tools
• Conclusion
For Your
Reference
Abstract-Review
• The session will cover both operational and maintenance aspects of all relevant
Firepower-NGFW functions from “Installation” to “Operation” to
“Troubleshooting” with a focus on interactive demonstration of the detailed
topics.
• Upon successful completion of this session, the attendee will be able to:
• describe the FTD system architecture
• describe packet flow processing
• perform installation and configuration of FirePOWER Threat Defense(FTD)
• verify and troubleshoot traffic flows traversing FTD
All content and demos are based on the following
• Firepower 4100 series system
• FXOS Version 2.1(1.77)
• Firepower Threat Defense 6.2.0.2 version (Released in May 2017)
• Firepower Management Center 6.2.0.2 version (Released in May 2017)
Hardware & Software Review
LTRSEC-1000
FTD Deployment Hands-on-lab
NGFW evolution Dax Mickelson
What platforms can run FTD Software
For Your
Reference
Platform FTD Support
ASA 5500X-Series (5506X-5555X with SSD) Yes
Firepower 4100 series Yes
Virtual options (VMware, KVM, AWS, Azure) Yes
Cisco ISR 4000/ISR-G2 (UCS-E module) Yes
Firepower 4100 – closer look
2 x 2.5” SSD Bays
Power 8 x optic SFP+ ports
Console Front view
MGMT
Rear view 2x optional NetMods
2 x Power Supply Module Bays

6 x Hot-Swap Fans units
Firepower 8350 – do not run FTD software
Firepower Chassis Manager
Firepower Management Center
Firepower Device Manager
Firepower Threat Defense
DETECTION ENGINE / Snort
Packet Data Transport System (PDTS)
DATA-PATH / LINA
FXOS
FTD CLI modes
FTD CLI modes
There are three CLIs while dealing with a ftd deployment:
Firepower-module1>
• FXOS CLI
• CLISH > > expert $ sudo su #
• LINA CLI firepower#
Moving between different CLI‘s:

FXOS -> CLISH connect ftd
CLISH -> LINA system support diagnostic-cli
LINA -> CLISH CTRL + a, d
CLISH -> FXOS exit
Firepower Threat Defense – CLI MODES
> expert $ sudo su #
>
> system support diagnostic-cli
CTRL + a, d
firepower> enable
firepower#
Firepower-module1> connect ftd
Converged FTD CLISH
• Available over SSH on data and management interface/s
• No switching back and forth between FP and ASA sub-modes
> system support diagnostic-cli
firepower> enable
firepower# show cpu BEFORE 6.1
Ctrl + a + d
> show cpu
> show cpu system

Linux 3.10.62-ltsi-WR6.0.0.27_standard (ftd.cisco.com) 02/07/17 _x86_64_
Time CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
14:32:43 all 20.46 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 79.35
> show cpu

CPU utilization for 5 seconds = 0%; 1 minute: 0%; 5 minutes: 0%> 6.1+
Installation and Configuration
Preparing Firepower 4100 for an installation
Setup Management IP address
KSEC-FPR4100-2-A# scope fabric-interconnect a
KSEC-FPR4100-2-A /fabric-interconnect # set out-of-band gw 10.62.148.1 ip 10.62.148.38
netmask 255.255.255.0
Warning: When committed, this change may disconnect the current CLI session
KSEC-FPR4100-2-A /fabric-interconnect* #
KSEC-FPR4100-2-A /fabric-interconnect* # commit
KSEC-FPR4100-2-A /fabric-interconnect # exit
Verify basic connectivity

KSEC-FPR4100-2-A# connect local-mgmt
KSEC-FPR4100-2-A(local-mgmt)# ping cisco.com
ping: unknown host cisco.com
KSEC-FPR4100-2-A(local-mgmt)# ping 72.163.4.161
64 bytes from 72.163.4.161: icmp_seq=1 ttl=231 time=156 ms
64 bytes from 72.163.4.161: icmp_seq=2 ttl=231 time=156 ms
Verify DNS configuration settings in FXOS CLI
KSEC-FPR4100-2-A# scope system KSEC-FPR4100-2-A /system/services # show dns
KSEC-FPR4100-2-A /system # scope services Domain Name Servers:
KSEC-FPR4100-2-A /system/services # show dns IP Address: 173.38.200.100
KSEC-FPR4100-2-A /system/services # IP Address: 8.8.8.8
Verify and configure DNS settings from FCM
Verify and configure Network Time Synchronization (NTP)
KSEC-FPR4100-2-A# show clock

Tue May 16 16:10:42 UTC 2017
KSEC-FPR4100-2-A# show ntp-overall-status
NTP Overall Time-Sync Status: Time Synchronized
Brief installation steps on Firepower 4100 series
Upgrade the supervisor (FXOS) software bundle
Configure FTD Management and Data Interfaces
Install FTD application image
Provision FTD Settings (mode, IP settings, FMC info)
Add FTD to Firepower Management Center
Upload new supervisor (FXOS) software to FCM
Upgrade the supervisor (FXOS) software bundle
Configure FTD Data & Management Interfaces
FTD logical device creation
For Your
Reference
FTD installation on 4100(1)
For Your
Reference
FTD installation on 4100(2)
For Your
Reference
FTD installation on 4100 (working hard)
FTD Installation „Local Console“ monitoring
KSEC-FPR4100-2-A /ssa/slot # connect module 1 console
Telnet escape character is '~'.
Trying 127.5.1.1...
Connected to 127.5.1.1.
Escape character is '~'.
CISCO Serial Over LAN:

Close Network Connection to Exit [ OK ]
Executing S47install_default_sandbox_EO.pl [ OK ]
Executing S50install-remediation-modules [ OK ]
Executing S51install_health_policy.pl [ OK ]
Executing S52install_system_policy.pl [ OK ]
Executing S53change_reconciliation_baseline.pl [ OK ]
Executing S70remove_casuser.pl [ OK ]
Executing S70update_sensor_objects.sh [ OK ]
Executing S85patch_history-init [ OK ]
Executing S90banner-init [ OK ]
Executing S96grow_var.sh [ OK ]
Executing S96install_vmware_tools.pl [ OK ]
(output truncated)
FTD installation on 4100 (finished)
Device registration
Device registration
Having trouble
registering device?
Device Registration
Encrypted
FMC FTD
Tunnel
192.168.0.0/24
Device Registration
Encrypted
FMC FTD
Tunnel
192.168.0.0/24 10.10.10.0/24
Device Registration
• Keep-Alive messages
Control channel Encrypted

FMC FTD
Events channel Tunnel
• Connection Events
• IPS Events
• Malware Events
• File Events
• SSL Events
Device Registration
ftd-4100-2:/# netstat -lnta | grep 8305
tcp 0 0 10.62.148.85:60563 10.62.148.90:8305 ESTABLISHED
tcp 0 0 10.62.148.85:54849 10.62.148.90:8305 ESTABLISHED

FMC FTD
root@fmc-2:/# netstat -lnta | grep 8305

tcp 0 0 10.62.148.90:8305 10.62.148.85:60563 ESTABLISHED
tcp 0 0 10.62.148.90:8305 10.62.148.85:54849 ESTABLISHED
Device Registration
TCP 8305

FMC FTD
> configure manager add <FMC IP address> <shared

key> <NAT ID>
Trouble 1: FTD has DHCP IP address what now?
FMC FMC - Add FTD into FMC WebUI
1. Keep Host entry EMPTY
2. Registration/Shared Key
3. ACP
4. License
eth0 MGMT interface with static IP address
5. NAT ID (required when host entry not used)
FTD – add FMC details

• Add manager/FMC IP address in CLI
• Shared Key (needs to match with FMC side)
MGMT interface with DHCP IP address mgmt0
• NAT ID (needs to match with FMC side)
> configure manager add <FMC static IP address>
FTD
<shared key> <NAT ID>
Important Note:
NGFW will initiate Registration communication!
Trouble 2: FMC has DHCP IP address what now?
FMC FMC - Add FTD into FMC WebUI
1. Keep Host entry (IP address of FTD)
2. Registration/Shared Key
3. ACP
4. License
eth0 MGMT interface with DHCP IP address
5. NAT ID (optional)
FTD
• Add manager/FMC IP address in CLI
• Shared Key (needs to match with FMC side)
MGMT interface with static IP address mgmt0
• NAT ID (needs to match with FMC side)
> configure manager add DONOTRESOLVE <shared

FTD key> <NAT ID optional>
Important Note:
FMC will initiate Registration communication!
Device registration “headache” error message
"Could not establish a connection with sensor.

Make sure the registration keys match, that the
software versions are compatible, and that the
network is not blocking the connection."
Device registration trouble #3
FTD FMC
> configure manager add 10.62.148.92 key cisco123

Manager successfully configured.
Please make note of reg_key as this will be required while adding Device in FMC.
>
FTD FMC
> configure manager add 10.62.148.92 key cisco123

Please make note of reg_key as this will be
required while adding Device in FMC.
>
CORRECT COMMAND SYNTAX

configure manager add <FMC IP> <REG KEY> <NAT ID>
FMC
FTD
> show managers

Host : 10.62.148.90
Registration Key : ****
Registration : pending
RPC Status :
>
FTD
FMC
#tail –f /ngfw/var/log/messages
May 28 18:04:57 fmc-vklauzov SF-IMS[2769]: [3315]
sftunneld:sf_ssl[WARN] Accept: Failed to authenticate peer
'10.62.148.90’
# tail -n 14 /etc/sf/sftunnel.conf
host 10.62.148.90;
ip 10.62.148.90;
reg_key cisco12345;
FTD
FMC
#tail –f /ngfw/var/log/messages
May 28 18:04:57 fmc-vklauzov SF-IMS[2769]: [3315] sftunneld:sf_ssl[WARN]
Accept: Failed to authenticate peer '10.62.148.90’
# tail -n 14 /etc/sf/sftunnel.conf
host 10.62.148.90;
ip 10.62.148.90;
reg_key cisco12345;
For Your
Reference
Device registration trouble 5
FMC
FTD
Internet
> configure manager add 10.62.148.92 cisco123

>
For Your
Reference
FMC
FTD
Internet
is full of NAT devices 
> configure manager add 10.62.148.92 cisco123

>
For Your
Reference
FTD > capture-traffic

Please choose domain to capture traffic from:
0 - management0
Selection? 0
Options: -n port 8305
18:36:47.642198 IP 10.62.148.90.54216 > 10.62.148.85.8305: Flags [S]
18:36:47.642218 IP 10.62.148.85.8305 > 10.62.148.90.54216: Flags [R.]
# tail -f /ngfw/var/log/messages | grep sftunnnel

(no new logs for encrypted communication channel used for registration)
#
For Your
Reference
FTD > capture-traffic

Please choose domain to capture traffic from:
0 - management0
Selection? 0
18:36:47.642198 IP 10.62.148.90.54216 > 10.62.148.85.8305: Flags [S]
18:36:47.642218 IP 10.62.148.85.8305 > 10.62.148.90.54216: Flags [R.]
> pmtool status

sftunnel (system) - User Disabled
# tail -f /ngfw/var/log/messages | grep sftunnnel
Command: /ngfw/usr/local/sf/bin/sftunnel -d -f
(no new logs for encrypted communication channel used for registration)
/etc/sf/sftunnel.conf
#
PID File: /ngfw/var/sf/run/sftunnel.pid
Enable File: /ngfw/etc/sf/sftunnel.conf
For Your
Reference
FTD FMC
# ifconfig eth0 | grep MTU

UP BROADCAST RUNNING MULTICAST MTU:1500
> show network

==================[ management0 ]===================
State : Enabled
Channels : Management & Events
MTU : 9000
----------------------[ IPv4 ]----------------------
Address : 10.62.148.85
For Your
Reference
FTD FMC

> show network

==================[ management0 ]===================
State : Enabled
MTU : 9000
----------------------[ IPv4 ]----------------------
Address : 10.62.148.85
For Your
Reference
FTD FMC

> show network

==================[ management0 ]===================
State : Enabled
MTU : 9000
----------------------[ IPv4 ]----------------------
Address : 10.62.148.85
For Your
Reference
FTD FMC
# tcpdump -i eth0 port 8305 -n

IP 10.62.148.85.38530 > 10.62.148.90.8305: Flags [S],
seq 2011406652, win 17920, options [mss
8960,sackOK,TS val 73165282 ecr 0,nop,wscale 7],
length 0
IP 10.62.148.90.53985 > 10.62.148.85.8305: Flags [S],
length 0
> capture-traffic IP 10.62.148.85.49249 > 10.62.148.90.8305: Flags [S],
Please choose domain to capture traffic from: seq 4287195732, win 17920, options [mss
0 - management0 8960,sackOK,TS val 73166079 ecr 0,nop,wscale 7],
1 - Router length 0
Selection? 0
For Your
Reference
FTD FMC
# tcpdump -i eth0 port 8305 -n

IP 10.62.148.85.38530 > 10.62.148.90.8305: Flags [S],
length 0
IP 10.62.148.90.53985 > 10.62.148.85.8305: Flags [S],
length 0
> capture-traffic IP 10.62.148.85.49249 > 10.62.148.90.8305: Flags [S],
Please choose domain to capture traffic from: seq 4287195732, win 17920, options [mss
0 - management0 8960,sackOK,TS val 73166079 ecr 0,nop,wscale 7],
1 - Router length 0
Selection? 0
For Your
Reference
FTD
# scp 10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-troubleshoot.tar.gz admin@10.62.148.90:/var/tmp/

10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-
troubleshoot.tar.gz 1% 3MB 1KB/s 01:01
For Your
Reference
FTD
# scp 10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-troubleshoot.tar.gz admin@10.62.148.90:/var/tmp/

10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-
troubleshoot.tar.gz 1% 3MB 1KB/s 01:01
Device-Registration For Your
Reference
Common-Fail-Scenarios Summary
1 Invalid Syntax 6 Low bandwidth between FMC and
FTD
2 Mismatch Between Keys 7 Process down
3 NAT ID not configured 8 MTU changes
4 FTD has DHCP IP address 9 Blocked TCP 8305 port on

what now? network
5 FMC has DHCP IP address 10 NAT ID mismatch
what now?
FTD Data-Flow: life of a packet
Firepower 4100 architecture overview
Security Engine (FTD)
Smart NIC
Internal Switch Fabric
Internal NM NM 1 NM 2
Firepower 4100 architecture overview
Security Engine (FTD)
Detection Engine / Snort
PDTS
Data-Path
FXOS
Packet-Flow
RX
PDTS
YES
DAQ
Existing NO Egress
Ingress L3/L4 ALG L3, L2
Pre-Filter NAT
Interface Conn Interface ACL checks hops
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Packet-Flow
RX
PDTS Lina rule-id matched
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Packet-Flow
SI (DNS/URL), Identity
SI (IP) SSL L7 ACL File/AMP IPS
Snort Verdict (trust, fast-forward, deny/blacklist) PDTS
Data-Path / LINA
Packet-Flow
PDTS Snort Verdict
Data-Path / LINA
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Data-Path – Do we receive any packets?
firepower# sh int eth 1/7

Interface Ethernet1/7 "INSIDE", is up, line protocol is up
MAC address 5897.bdb9.73ee, MTU 1500
IP address 172.16.1.1, subnet mask 255.255.255.0
Traffic Statistics for "INSIDE":
180 packets input, 14853 bytes
155 packets output, 12628 bytes Number of packets dropped in
ASP ‘show asp drop‘
25 packets dropped
1 minute input rate 1 pkts/sec, 94 bytes/sec
Data-Path – Do we receive any packets?
DATA-PATH
> show capture in

1: 15:52:55.249834 172.16.1.56 > 20.20.20.33: icmp: echo request
2: 15:52:55.250643 20.20.20.33 > 172.16.1.56: icmp: echo reply
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Data-Path – Existing Connection
• LINA part checks whether the connection belongs to existing flow or not
• If packet is part of already established flow, then appliance skip basics checks
and process the packet in Fast-Path – and continue with checks at DAQ level
firepower# show cap in2 packet-number 46 trace detail
46: 19:28:20.056012 0050.56b6.0b33 5897.bdb9.73ee 0x8100 Length: 58
802.1Q vlan#208 P0 172.16.2.13.49182 > 20.20.20.11.80: . [tcp sum ok] 2790183968:2790183968(0) ack
1176461110 win 231 (DF) (ttl 128, id 16898)
...
Type: FLOW-LOOKUP
Found flow with id 34550, using existing flow
Unique Connection ID
firepower# sh logging | include 34550
%ASA-6-302013: Built inbound TCP connection 34550 for in2:172.16.2.13/49182
(172.16.2.13/49182) to OUTSIDE:20.20.20.11/80 (20.20.20.11/80)
%ASA-6-302014: Teardown TCP connection 34550 for in2:172.16.2.13/49182 to

OUTSIDE:20.20.20.11/80 duration 0:00:28 bytes 1073752075 Flow closed by inspection
firepower#
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Data-Path – Egress Interface
• Determination of Egress Interface
• Routing table / route lookup – ‘in’ entries of the ASP routing table will be
checked to determine the egress interface
• UN-NAT (destination NAT) – egress interface will be choosen based on NAT
rule
Data-Path / LINA CLI:
firepower# show asp table routing

firepower# show capture <name> packet-number 10 trace detail
firepower# packet-tracer
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Data-Path – Pre-Filter Policy
Flow-offload feature
• Help to offload the flows to Smart NIC for faster throughput and low latency
• Decision to offload is made by DATA-PATH (in future release also Snort would
do this)
• Flow state tracking done by DATA-PATH
• Supported in Clustering deployments, but no offload mode compatibility checks
• Supported in HA failover mode – offload flags are replicated to standby
Motivation:
• Data center FTD deployments with FAT a.k.a. Elephant Flows
• Latency issues in current data plane processing due to x86 CPU complex
involvement
Use cases
• High performance computing research sites
• High frequency trading
• GRE tunneled packets
Configuration
• Enabled by default on FTD (no GUI option to enable/disable feature)
• Flows that match pre-filter policy rule with Fast-Path action or Access
Control Policy rule with TRUST action will be selected for flow offload
• Limitations 6.1 release
• Flows processed by Detection-Engine/Snort cannot be offloaded, only
Data-Path flows
• Flow offload not supported for FTD when interfaces are configured as inline-
set
• DATA-PATH
• Handle decisions to offload based on policies setup by user
• Handle connection establishment and tear-down of offloaded flows
Actions
• Analyze: sends traffic for inspection to Snort
• Block: drops the traffic
• Fastpath: allows traffic and bypass further inspection,
process the rule in hardware, offload the traffic
Data-path policy vs. Snort policy
• Distributed evaluation of policy between LINA and SNORT
Access-control policy
Pre-filter policy
Data-path policy vs. Snort policy
• AC rules that are evaluated by Snort are pushed down to LINA as PERMIT
ACL rules
• Pre-filter rules are presented as Global ACL’s to LINA
Inner-headers packet
Outer-headers packet Permit ACL (appID, URL, User)
Global ACL (5-tuple)
Data-path / LINA “backend” ACL’s
• New type of ACL (Advanced ACL) is introduced for Access control
• Permit/Trust/Deny actions (within show access-list cmd)
• Permit means that packet is punted to Snort
• Trust means to skip Snort/Detection engine checks
• Lina can send start and end of flow events and Snort sends them to FMC
• Lina rule-id uniquely identify a rule and sends to Snort to perform NGFW
policy evaluation
This is example of configuration that triggers flow offload!
firepower# show running-config access-l | exclude remark

access-list CSM_FW_ACL_ advanced trust icmp any any rule-id 268434442 event-log both
access-list CSM_FW_ACL_ advanced trust tcp any any eq ftp rule-id 268434444 event-log
both
Identity Advanced Snort / FirePOWER

DETECTION ENGINE
Packet Data Transport System (PDTS) & DAQ

YES
QoS, VPN Encrypt
BRKSEC-3455 92
Ingress NO Egress
Existing L3/L4 ALG L3, L2
RX Pre-Filter TX
Interface Interface ACL NAT
Conn checks hops
VPN Decrypt
DATA-PATH Traffic that matches pre-filter rule with FAST-PATH Action
Will be offloaded to Hardware

SMART NIC firepower# show flow-offload flow
2 in use, 2 most used, 16% offloaded
TCP intfc 106 src 20.20.20.11:80 dest 172.16.2.14:49191, timestamp
2265924877, packets 191614, bytes 264712022
TCP vlan 208 intfc 107 src 172.16.2.14:49191 dest 20.20.20.11:80, timestamp
2265924879, packets 26301, bytes 1788781
For Your
Reference
Verify that flow-offload is enabled
firepower# show flow-offload info
Current running state : Disabled
User configured state : Enabled
Clear connection table in hardware / flow offloaded flows
firepower# clear flow-offload flow all
This command will not remove connection from DATA-PATH, you have to run clear conn command to do so.
For Your
Reference
Syslog message when flow is offloaded and no longer offloaded
%ASA-6-805001: Offloaded TCP Flow for connection 34892 from
in2:172.16.2.14/49193 (172.16.2.14/49193) to OUTSIDE:20.20.20.11/80
(20.20.20.11/80)
%ASA-6-805001: Offloaded TCP Flow for connection 34892 from
OUTSIDE:20.20.20.11/80 (20.20.20.11/80) to in2:172.16.2.14/49193
(172.16.2.14/49193)
%ASA-6-805002: TCP Flow is no longer offloaded for connection 34892 from

in2:172.16.2.14/49193 (172.16.2.14/49193) to OUTSIDE:20.20.20.11/80
(20.20.20.11/80) BRKSEC-3455
%ASA-6-805002: TCP Flow is no longer offloaded for connection 34892 from

OUTSIDE:20.20.20.11/80 (20.20.20.11/80) to in2:172.16.2.14/49193
(172.16.2.14/49193)
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Access Control rule actions
• Allow – allow packet/s to go

through further IPS/File policy
evaluation (if configured)
• Trust – push traffic through
hardware (Fast-Path traffic), no
further Snort checks needed
Data-Path – L3/L4 ACL
5-TUPLE
FMC
Data-Path
firepower# show access-list | i icmp
access-list CSM_FW_ACL_ line 9 remark rule-id 268441864: L7 RULE: icmp traffic
access-list CSM_FW_ACL_ line 10 advanced permit icmp any any rule-id 268441864
(hitcnt=335) 0xa2dc10fa
FirePOWER
cat ngfw.rules | grep 268441864
root@ftd:/var/sf/detection_engines/ae4faffe-d1b2-11e6-8ea4-817d227fa40c#
268441864 fastpath any any any any any any any 1 (log dcforward both)
Data-Path – L3/L4 ACL
5-TUPLE
FMC
Data-Path
firepower# show access-list | i icmp
access-list CSM_FW_ACL_ line 9 remark rule-id 268441864: L7 RULE: icmp traffic
access-list CSM_FW_ACL_ line 10 advanced permit icmp any any rule-id 268441864
(hitcnt=335) 0xa2dc10fa
Why AC rule with 5-tuple information is not marked as TRUST flow?
Packet-Flow
PDTS Snort Verdict
Data-Path / LINA
Detection engine / Snort – Security Intelligence
• Ability to block dangerous / malicious traffic aka “bad guys”
• SI feed is updated by Cisco TALOS team periodically
• SI whitelist is intentionally processed by rest of the ACP rules
• 2 default SI Lists: Global Whitelist and Blacklist
User Story #1 – Security Intelligence (1)
• Problem description: URL website blocked
Analysis -> Connections -> Security Intelligence Events
User Story #1 – Security Intelligence (2)
Why whitelisted traffic has been not allowed/trusted immediately?
User Story #2 – Security Intelligence
• Problem description: Inability to access local web servers from outside network
No sings of drops:
• IPS Events
• Malware Events
• SI events
root@firepower:/Volume/home/admin# cd /var/sf/iprep_download/
# grep "72.4.119.2\|#" *
d8eea83e-6167-11e1-a154-589de99bfdf1:#Global-Blacklist
d8eea83e-6167-11e1-a154-589de99bfdf1:72.4.119.2
# cat d8eea83e-6167-11e1-a154-589de99bfdf1
#Global-Blacklist
72.163.4.161
Lesson learned …
Packet-Flow
PDTS Snort Verdict
Data-Path / LINA
Detection Engine / Snort – L7 ACL
• Order of operation: rules are being processed from top to bottom

• Differentiate ACP rule operations between (AND operand) and within
columns (OR operand)
• Adaptive profiling needs to be enabled (in order to determine App ID) – “on
by default”
Detection Engine / Snort – L7 ACL
• Identification of App ID occurs usually within 3-5 packets or after SSL
handshake
> system support firewall-engine-debug

172.16.1.10-60467 > 20.20.20.10-21 6 AS 1 I 7 no match rule order 3, 'FTP to be
allowed', app s=-1 c=-1 p=-1 m=-1
or 65 535
20.20.20.10-53156 > 172.16.1.10-21 6 AS 1 I 46 Starting with minimum 3, 'FTP to be
allowed', and SrcZone first with zones 2 -> 1, geo 0 -> 0, vlan 0, sgt tag:
untagged, svc 165, payload 4002, client 2000000165, misc 0, user 9999997, icmpType
0, icmpCode 0
20.20.20.10-53156 > 172.16.1.10-21 6 AS 1 I 46 match rule order 3, 'FTP to be
allowed', action Allow
Packet-Flow
PDTS Snort Verdict
Data-Path / LINA
Traffic IN but not OUT
firepower# sh cap
capture i type raw-data trace detail interface INSIDE [Capturing - 114 bytes]
match icmp any any
capture o type raw-data trace detail interface OUTSIDE [Capturing - 0 bytes]
match icmp any any
Snort: IPS policy
• “Troubleshooting thoughts”
• Connection inspected by SNORT?
• “show conn” – Flag ‘N’
• Packet captures (capture and capture-traffic) shows incoming traffic on
ASA/LINA side, diverted traffic flows are send to the SNORT, but NO outgoing
or there are missing packets after SNORT inspection on outside interface?
• Connection events are triggering? -> FMC Connection table view
• Is the right AC rule being evaluated? -> NGFW debugs
• IPS events are not populated? -> Create custom ICMP rule or enable “ICMP echo-
reply” rule
1:408 to confirm IPS events are generally working
Snort: IPS policy
• In IPS policy rule to “Drop and Generate” action
• Interface should be in the “Inline” mode
• IPS policy needs to have “Drop when Inline” option enabled
How FTD is blocking traffic?
firepower# sh cap i packet-number 1 trace
1: 09:09:18.644467 172.16.1.17 > 20.20.20.100: icmp: echo
request
Type: SNORT
Result: DROP
Snort Verdict: (black-list) black list this flow
Action: drop
Drop-reason: (snort-drop) Snort requested to drop the frame
Preprocessor
• Special Attention when packets are blocked, but there are no IPS events.
Change Rule State:
Drop and Generate
Inline-normalization
IPS policy troubleshooting was never easier as in
6.2+
Capture with trace detail / packet tracer:
Type: SNORT
Result: DROP
Packet: TCP, ACK, seq 3806011039, ack 3309256170
Firewall: allow rule, id 268434444, allow
IPS Event: gid 1, sid 1000000, drop
Snort detect_drop: gid 1, sid 408, drop
AppID: service HTTP (676), application unknown (0)
Firewall: allow rule, id 268434444, allow
Snort: processed decoder alerts or actions queue, drop
IPS Event: gid 1, sid 1000000, drop
Snort detect_drop: gid 1, sid 1000000, drop
NAP id 2, IPS id 1, Verdict BLACKLIST, Blocked by IPS
Snort Verdict: (black-list) black list this flow
Action: drop Drop-reason: (ips) Blocked or blacklisted by the IPS preprocessor
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Data-path: Inspection
firepower# show service-policy flow tcp host 20.20.20.11 host 172.16.2.100 eq 21

Global policy:
Service-policy: global_policy
Class-map: inspection_default
Match: default-inspection-traffic
Action:
Input flow: inspect ftp
Class-map: class-default
Match: any
Action:
Output flow: Input flow: set connection random-sequence-number
disable
set connection advanced-options UM_STATIC_TCP_MAP
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
Data-path: NAT, L2 and L3 next hop
Remaining checks are same as on the standalone ASA:
• Determination of NAT IP header – in capture trace phase ‘NAT’ with translated IP

addresses details
• Based on the packet processing step “Egress Interface” determination the ‘out’ entries
will be now checked in ASP routing table
• Using packet capture trace detail option we can see phase “ROUTE-LOOKUP” with the
next-hop IP address IP address details
Data-Path
RX
PDTS
YES
DAQ
Existing NO Egress
Pre-Filter NAT
VPN Decrypt
QoS, VPN Encrypt
TX
Data-Path / LINA
SI (DNS/URL), Identity Advanced Snort / FirePOWER
Packet processing - TX ring

> show capture in
2: 15:52:55.250643 20.20.20.33 > 172.16.1.56: icmp: echo reply
DATA-PATH
> show capture out

2: 15:52:55.250627 20.20.20.33 > 172.16.1.56: icmp: echo reply
FTD Troubleshooting tools
What are main FTD processes and what they do?
snort inspects network traffic (pass, sftunnel secure tunnel between
block and alert) managed device and FMC
ids_event_processor sends intrusion events to diskmanager, managing disk space and
managing device (FMC) Pruner clean up old files
ids_event_alerter sends intrusion events to Lina Responsible for Firewall
Syslog or SNMP server functionality like ACL, NAT,
Routing etc.
wdt-util used for fail-to-wire / Snmpd, SNMP monitoring,
hardware bypass ntpd responsible for time
synchronization
SFDataCorrelator processing events pm (process responsible for launching
manager) and monitoring of all FTD
relevant processes and
restarting them in case of
failure
Process Management - basics
FMC Root CLI:
fmc-vklauzov:/# pmtool status | grep " - " | head

SFDataCorrelator (normal) - Running 15278
mysqld (system,gui,mysql) - Running 15109
httpsd (system,gui) - Waiting
sftunnel (system) - Running 19857
Process name Status

Process ID
Category
Process Management - basics
FMC Root CLI:
root@fmc-2:/# pmtool disablebyid sftunnel

root@fmc-2:/# pmtool status | grep " - " | grep sftunnel
sftunnel (system) - User Disabled
root@fmc-2:/# pmtool enablebyid sftunnel

root@fmc-2:/# pmtool status | grep " - " | grep sftunnel
sftunnel (system) - Running 1720
Data-path and Snort capture points

> capture-traffic
2. snort inbound/outbound
firepower# capture out
3.
firepower# capture in
data-path outbound
DATA-PATH
1.
data-path inbound
Data-path inbound/outbound - The Wires Never Lie!
Data-path/lina (diagnostic cli):
firepower# capture in interface INSIDE match icmp any any trace detail
Capture name
Interface name
Source
Destination
protocol
Snort Capture - The Wires Never Lie! (1)
CLISH:
> capture-traffic
Options: -s 0 -w capture.pcap icmp and host 172.16.1.17
IP 172.16.1.17 > 20.20.20.100: ICMP echo request,id 24538,seq 1,length 64
Berkeley Packet Filter syntax – same as for tcpdump capturing tool
-s 0 means snaplength, in other words no limit for packet size

-w filename.pcap indicates to which file you want to write output of data captured by specified filter
capture is written to /ngfw/var/common/ folder
Copy file out to SCP server:
file secure-copy <IP address of server> <username> <location where to copy the file>
capture.pcap
Snort Capture - The Wires Never Lie! (2)
NON-VLAN TAGGED TRAFFIC VLAN TAGGED TRAFFIC
CLISH:
> capture-traffic
Options: -v -n -e (icmp and host 172.16.2.11) or (vlan and icmp and host 172.16.2.11)
00:50:56:b6:0b:33 > 58:97:bd:b9:73:ee, ethertype 802.1Q (0x8100), length 78: vlan 208,
p 0, ethertype IPv4, (tos 0x0, ttl 128, id 5366, offset 0, flags [none], proto ICMP
(1), length 60)
IN OUT
LINA CLI: LINA CLI:
firepower# sh cap inside firepower# sh cap outside

172.16.2.11 > 20.20.20.11: icmp: echo
802.1Q vlan#208 P0 172.16.2.11 > request
20.20.20.11: icmp: echo request
Which ACP rule is being evaluated?
• Tool that provides the Access Control Rule evaluation status for each flow as we receive
packets in real time.
• NGFW debug needs to have specified at least one filtering condition.
>system support firewall-engine-debug

Please specify an IP protocol: icmp
Please specify a client IP address: 172.16.1.17
Please specify a server IP address: 20.20.20.100
Monitoring firewall engine debug messages172.16.1.17-8 >
20.20.20.100-0 1 AS 1 I 44 New session
172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 using HW or preset
rule order 2, 'allow and inspect', action Allow and prefilter
rule 0
172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 allow action
Access Control Policy Rule Hit Counters
> show access-control-config
===================[ ciscolive ]==================== Policy name
Description :
Default Action : Allow
Default Policy : Balanced Security and Connectivity
Logging Configuration
DC : Disabled
Beginning : Disabled
End : Disabled
Rule Hits : 10
Variable Set : Default-Set
... (output omitted) ...
# watch ´/usr/local/sf/bin/sfcli.pl show firewall | grep "ciscolive\| Rule\:\|Rule Hits "´

===================[ ciscolive ]====================
Rule Hits : 10
------------------[ Rule: allow ]------------------- Rule name

Rule Hits : 14
Description :
DC : Disabled
End : Disabled
Rule Hits : 14

===================[ ciscolive ]====================
Rule Hits : 14

Rule Hits : 14
Description :
DC : Disabled
End : Disabled
Rule Hits : 19

===================[ ciscolive ]====================
Rule Hits : 19

Rule Hits : 14
Description :
DC : Disabled
End : Disabled
Rule Hits : 26

===================[ ciscolive ]====================
Rule Hits : 26

Rule Hits : 14
ACP Rule Hit Counters – FMC WebUI
• Analysis -> Custom -> Custom Workflows -> Create Custom Workflow and use Table
“Connection Events”
• Add page and fill in fields like: “Access Control Policy”, “Access Control Rule”,
“Count”, “Initiator IP”, “Responder IP”
• Add Table view
ACP Rule Hit Counters – FMC WebUI vs CLISH

------[ Rule: DNS and icmp ]------
Action : Allow
Destination Ports : protocol 6, port 53
protocol 17, port 53
protocol 1
protocol 6, port 80
Logging Configuration Why the hit counters do not match?
DC : Enabled
Beginning : Enabled
End : Enabled
Rule Hits : 28
(truncated)
Event Path
Types of Events
• Network Discovery Events
• information about a host based on traffic seen from the host
• when a session matches an AC rule with logging
• Intrusion Events
• when an IPS rules trigger (Drop and Generate Event)
• File Events
• when a file is captured
• Malware Events
• when a file is captured and it is detected to be Malware
FTD Detection Engine Logging
• When an event is generated in detection engine, it is written to :
/ngfw/var/sf/detection_engine/<uuid>/instance-*/
• Intrusion events – snort-unified.log.1497179589

• Connection/File events – unified_events-2.log.1497179650
• Malware events – unified_events-1.log.1497179650
• Network Discovery events – unified_events-2-rna.log.
1497179650 Decode Linux Epoch Time
date -d@1497179589
Sun Jun 11 11:13:09 UTC 2017
FTD Detection Engine Logging
• Determine detection engine UUID:
ftd-4100-2:/# de_info.pl
________________________________________________________________________
DE Name : Primary Detection Engine (1e149ee0-3f8f-11e7-b625-b451664b5209)
DE Type : ids
DE Description : Primary detection engine for device 1e149ee0-3f8f-11e7-b625-
b451664b5209
DE Resources : 12
DE UUID : 4dec8fce-3f8f-11e7-b0f0-d383664b5209
________________________________________________________________________________
# cd /ngfw/var/sf/detection_engines/4dec8fce-3f8f-11e7-b0f0-d383664b5209/instance-1/
Event Path for IPS event
/ngfw/var/sf/detection_engine/<uuid>/instance-*/snort-unified.log
> sftunnel_status
IDS Event Service:
TOTAL TRANSMITTED MESSAGES <4> for IDS Events service
RECEIVED MESSAGES <b> for service IDS Events service FMC
NGFW SEND MESSAGES <2> for IDS Events service
HALT REQUEST SEND COUNTER <0> for IDS Events service
STORED MESSAGES for IDS Events service (service 0/peer 0)
STATE <Process messages> for IDS Events service
REQUESTED FOR REMOTE <Process messages> for IDS Events service
REQUESTED FROM REMOTE <Process messages> for IDS Events service
Event Path for Malware/Connection event
unified_events-1.log.<timstamp> -- malware
unified_events-2.log.<timestamp> – connection
> sftunnel_status
Priority UE Channel 0 service – high priority queue

TOTAL TRANSMITTED MESSAGES <4> for UE Channel service
RECEIVED MESSAGES <2> for UE Channel service FMC
NGFW
SEND MESSAGES <2> for UE Channel service
HALT REQUEST SEND COUNTER <0> for UE Channel service
STORED MESSAGES for UE Channel service (service 0/peer 0)
STATE <Process messages> for UE Channel service
REQUESTED FOR REMOTE <Process messages> for UE Channel service
REQUESTED FROM REMOTE <Process messages> for UE Channel service
Event Path for Network Discovery event
/ngfw/var/sf/detection_engine/<uuid>/instance-*/unified_events-2-rna.log.<timestamp>
> sftunnel_status
Priority UE Channel 1 service – low priority queue

TOTAL TRANSMITTED MESSAGES <4> for UE Channel service
RECEIVED MESSAGES <2> for UE Channel service FMC
NGFW
SEND MESSAGES <2> for UE Channel service
HALT REQUEST SEND COUNTER <0> for UE Channel service
STORED MESSAGES for UE Channel service (service 0/peer 0)
STATE <Process messages> for UE Channel service
REQUESTED FOR REMOTE <Process messages> for UE Channel service
REQUESTED FROM REMOTE <Process messages> for UE Channel service
Mysteries of IPS events logging
IPS-logging
10.10.10.20 20.20.20.10
NGFW
ICMP request ICMP request
ICMP reply
IPS block SID 1:408:8
2.
IPS-logging FMC
Syslog Servers
eth0
Secured channel
TCP 8305
IPS event/s
management0
1.
snort-unified.log.1497179014
# date -d@1497179014
Sun Jun 11 11:03:34 UTC 2017
NGFW BRKSEC-3455 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
IPS-logging FMC
Syslog Servers
eth0
Secured channel
TCP 8305
IPS event/s
management0
10.10.10.20 20.20.20.10
Possible root cause?
IPS alerting configuration review (1)
• IPS Policy -> Advanced Settings
IPS alerting configuration review (2)
System processes review
> pmtool status
d002ce08-55e0-11e7-a28f-534987204de8-alert (de) – Down 31729

Command: /ngfw/usr/local/sf/bin/ids_event_alerter
PID File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_event_alerter.pid
Enable File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_alert.conf
> pmtool enablebyid d002ce08-55e0-11e7-a28f-534987204de8-alert
d002ce08-55e0-11e7-a28f-534987204de8-alert (de) – Running 41324

Command: /ngfw/usr/local/sf/bin/ids_event_alerter
PID File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_event_alerter.pid
Enable File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_alert.conf
IPS-logging FMC
Syslog Servers
eth0
Secured channel
TCP 8305
IPS event/s
management0
snort-unified.log.1497179014
# date -d@1497179014
Sun Jun 11 11:03:34 UTC 2017
Conclusion
Take the chance and drive your FTD installation to a
success
• Plan your desired hardware based on capabilities and performance
• Plan your desired feature-set and functionality
• Plan your desired operations mode (there are choices)
• Plan a pilot-phase with extra timing for all operational tasks
•
We wish you every success operating and
Upgrades/Downgrades
troubleshooting your new NG-Firewall
• Backup/Restore
• Replacement/RMA
• Practice basic troubleshooting steps
• Have a look at new features and functionality inside a testbed
Complete Your Online
Session Evaluation
for BRKSEC-3455
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
For Your
Reference
Reminder - You don’t want to miss at #CLUS
BRKSEC-2020
BRKSEC-2050
Firepower NGFW TECSEC-3301
Firepower NGFW
Deployment in the Data Firepower Data-Path
Internet Edge
Center and Enterprise Troubleshooting
Deployment Scenarios
Network Edge using John Groetzinger
Jeff Fanelli
FTD
Steven Chimes
For Your
Reference
Reminder - You don’t want to miss at #CLUS
TECSEC-2004
BRKSEC-3020
BRKSEC-3035 Troubleshooting FTD
Troubleshooting ASA
Firepower Platform like a TAC Engineer
Firewalls
Deep Dive Ben Ritter
Kevin Klous
Andrew Ossipov Kevin Klous
Veronika Klauzova
Thank you
BRKSEC-3455
Thank you for attenting
BRKSEC-3455

FTD TF PDF

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

FTD TF PDF

Caricato da

Copyright:

Formati disponibili

Dissecting Firepower-

Cisco Spark spaces will be

• Slavic countries accent Veronika Klauzova

ASA 5500X-Series (5506X-5555X with SSD) Yes

Firepower 4100 series Yes

Firepower 9300 series Yes

Firepower 2100 series Yes

Virtual options (VMware, KVM, AWS, Azure) Yes

Cisco ISR 4000/ISR-G2 (UCS-E module) Yes

Rear view 2x optional NetMods

2 x Power Supply Module Bays

DETECTION ENGINE / Snort

Packet Data Transport System (PDTS)

• LINA CLI firepower#

Moving between different CLI‘s:

CLISH -> LINA system support diagnostic-cli

LINA -> CLISH CTRL + a, d

CLISH -> FXOS exit

Firepower-module1> connect ftd

> show cpu system

> show cpu

Verify basic connectivity

Verify and configure DNS settings from FCM

KSEC-FPR4100-2-A# show clock

Upgrade the supervisor (FXOS) software bundle

Configure FTD Management and Data Interfaces

Install FTD application image

Provision FTD Settings (mode, IP settings, FMC info)

Add FTD to Firepower Management Center

CISCO Serial Over LAN:

Control channel Encrypted

Control channel Encrypted

root@fmc-2:/# netstat -lnta | grep 8305

tcp 0 0 10.62.148.90:8305 10.62.148.85:54849 ESTABLISHED

Control channel Encrypted

> configure manager add <FMC IP address> <shared

FTD – add FMC details

> configure manager add DONOTRESOLVE <shared

"Could not establish a connection with sensor.

> configure manager add 10.62.148.92 key cisco123

> configure manager add 10.62.148.92 key cisco123

CORRECT COMMAND SYNTAX

> show managers

> configure manager add 10.62.148.92 cisco123

> configure manager add 10.62.148.92 cisco123

FTD > capture-traffic

# tail -f /ngfw/var/log/messages | grep sftunnnel

FTD > capture-traffic

> pmtool status

# ifconfig eth0 | grep MTU

> show network

# ifconfig eth0 | grep MTU

> show network

# ifconfig eth0 | grep MTU

> show network

# tcpdump -i eth0 port 8305 -n

# tcpdump -i eth0 port 8305 -n

# scp 10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-troubleshoot.tar.gz admin@10.62.148.90:/var/tmp/

# scp 10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-troubleshoot.tar.gz admin@10.62.148.90:/var/tmp/

3 NAT ID not configured 8 MTU changes

4 FTD has DHCP IP address 9 Blocked TCP 8305 port on

Internal Switch Fabric

Security Engine (FTD)