ETTING STARTED: FLOW BASIC

17378
Created On 02/07/19 23:51 PM - Last Updated 02/07/19 23:52 PM
Resolution
LET ME FIX THAT FOR YOU: FLOW BASIC—

In the previous episode, we leveraged debug filters to allow the Palo Alto Networks firewall to collect packet captures we
could use for troubleshooting. But sometimes, you may need to look deeper into what's going on inside the firewall.

Flow basic is the equivalent of a packet capture on every stage inside the firewall process, from receiving the packet to
making security decisions, applying NAT, App-ID and so on, which makes it a very powerful tool.

Wield this power with due care as the process can be CPU intensive if your filters are set up broadly or lots of traffic needs
to be captured.

Before you get started, make sure the dataplane is not overloaded:

> show running resource-monitor minute

Resource monitoring sampling data (per minute):

CPU load (%) during last 60 minutes:

core 0 1 2 3 4 5

avg max avg max avg max avg max avg max avg max

0 0 0 1 0 0 0 0 0 0 0 0

0 0 0 1 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0

The first thing we need to do is set up filters. Last time I showed you how to do this from the GUI—this time, let's take a look
at the CLI:

First we're going to verify that nothing's been configured yet that could interfere with our new settings:

> debug dataplane packet-diag show setting

--------------------------------------------------------------------------------

Packet diagnosis setting:

--------------------------------------------------------------------------------

Packet filter

Enabled: no
 Match pre-parsed packet: no

--------------------------------------------------------------------------------

Logging

Enabled: no

Log-throttle: no

Sync-log-by-ticks: yes

Features:

Counters:

--------------------------------------------------------------------------------

Packet capture

Enabled: no

Snaplen: 0

--------------------------------------------------------------------------------

If anything's still configured, we can clear out all filters and previous flow basic logs using these commands:

> debug dataplane packet-diag clear all

> debug dataplane packet-diag clear log log

We can now go ahead and create and enable the filters, making sure pre-parse is disabled. A second filter from the server
to the NAT IP on the external interface of the firewall will help capture returning packets before they are NAT'ed in the
'ingress stage.' More about that below:

> debug dataplane packet-diag set filter match source 192.168.0.34 destination
198.51.100.97 destination-port 80 protocol 6 non-ip exclude

> debug dataplane packet-diag set filter match source 198.51.100.97 destination
198.51.100.230 source-port 80 protocol 6 non-ip exclude
> debug dataplane packet-diag set filter on

> debug dataplane packet-diag show setting

--------------------------------------------------------------------------------

Packet diagnosis setting:

--------------------------------------------------------------------------------

Packet filter

Enabled: yes

Match pre-parsed packet: no

 Index 1: 192.168.0.34[0]->198.51.100.97[80], proto 6

ingress-interface any, egress-interface any, exclude non-IP

Index 2: 198.51.100.97[80]->198.51.100.230[0], proto 6

ingress-interface any, egress-interface any, exclude non-IP

--------------------------------------------------------------------------------

Logging

Enabled: no

Log-throttle: no

Sync-log-by-ticks: yes

Features:

Counters:

--------------------------------------------------------------------------------

Packet capture

Enabled: no

Snaplen: 0

--------------------------------------------------------------------------------

When you're ready to enable logging, you'll see there are several features you can enable. Each one sets a capturing
process on a specific engine or daemon that can help drill down even further. 'appid' can help troubleshoot why a certain
app may not be getting identified in a flow and 'ctd' can help troubleshoot vulnerability signatures, and so on:

> debug dataplane packet-diag set log feature

> all all

> appid appid

> cfg cfg

> ctd ctd

> flow flow

> misc misc

> module module

> pow pow

> proxy proxy

> ssl ssl

> tcp tcp

 > tunnel tunnel

> url_trie url_trie

> zip zip

For now, we'll start with the 'flow' feature, which relates to all the base-level operations like inspecting TCP handshake,
building sessions on the firewall, and performing NAT. In each feature, you can enable yet more subsections for even
greater detail, but we'll stick to the basic setting for now.

> debug dataplane packet-diag set log feature flow

ager ager

all all

arp arp

basic basic

ha ha

log log

nd nd

np np

receive receive

track track

> debug dataplane packet-diag set log feature flow basic

> debug dataplane packet-diag show setting

--------------------------------------------------------------------------------

Packet diagnosis setting:

--------------------------------------------------------------------------------

Packet filter

Enabled: yes

Match pre-parsed packet: no

Index 1: 192.168.0.34[0]->198.51.100.97[80], proto 6

ingress-interface any, egress-interface any, exclude non-IP

Index 2: 198.51.100.97[80]->198.51.100.230[0], proto 6

ingress-interface any, egress-interface any, exclude non-IP

--------------------------------------------------------------------------------

Logging

Enabled: no

Log-throttle: no

Sync-log-by-ticks: yes

Features:

flow : basic

Counters:

--------------------------------------------------------------------------------

Packet capture

Enabled: no

Snaplen: 0

--------------------------------------------------------------------------------

When you're ready to initiate traffic make sure any existing sessions have been terminated, then disable session offloading
to ensure all packets are captured even if the session would normally be offloaded into hardware and finally go ahead and
enable the logging feature.

> show session all filter source 192.168.0.34 destination 198.51.100.97

No Active Sessions

If there are still active sessions you can clear them by using the clear session command:
> clear session all filter source 192.168.0.34 destination 198.51.100.97

> set session offload no

> debug dataplane packet-diag set log on

You can now go ahead and start the session you want to capture, wait for it to gracefully end, then disable logging:

> show session all filter source 192.168.0.34 destination 198.51.100.97

--------------------------------------------------------------------------------

ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])

Vsys Dst[Dport]/Zone (translated IP[Port])

--------------------------------------------------------------------------------
51187 web-browsing ACTIVE FLOW NS 192.168.0.34[64969]/trust/6
(198.51.100.230[42882])

vsys1 198.51.100.97[80]/ISP1 (198.51.100.97[80])

> show session id 51187

Session 51187

c2s flow:

source: 192.168.0.34 [trust]

dst: 198.51.100.97

proto: 6

sport: 64969 dport: 80

state: INIT type: FLOW

src user: pantac\tpiens

dst user: unknown

s2c flow:

source: 198.51.100.97 [ISP1]

dst: 198.51.100.230

proto: 6

sport: 80 dport: 42882

state: INIT type: FLOW

src user: unknown

dst user: pantac\tpiens

start time : Wed Feb 10 13:43:27 2016

timeout : 15 sec

total byte count(c2s) : 644

total byte count(s2c) : 308

layer7 packet count(c2s) : 5

layer7 packet count(s2c) : 2

 vsys : vsys1

application : web-browsing

rule : web-out

session to be logged at end : False

session in session ager : False

session updated by HA peer : False

address/port translation : source

nat-rule : outbound-nat(vsys1)

layer7 processing : enabled

URL filtering enabled : False

session via syn-cookies : False

session terminated on host : False

session traverses tunnel : False

captive portal session : False

ingress interface : ethernet1/3

egress interface : ethernet1/1

session QoS rule : N/A (class 4)

tracker stage firewall : TCP RST - client

end-reason : tcp-rst-from-client

> show session all filter source 192.168.0.34 destination 198.51.100.97

No Active Sessions

> debug dataplane packet-diag set log off

Packet log is disabled

> set session offload yes

Each dataplane CPU will generate its own flow log, so depending on the amount of traffic, the type and amount of sessions,
there may be several files located on the dataplane. Each CPU that participated in the capture will have a pan_task_X.log
entry:

> less dp-log

bfd.log brdagent.log dp-monitor.log dp-monitor.log.1

dp-monitor.log.2 dp-monitor.log.3 dp-monitor.log.4 masterd.log

masterd_apps.log masterd_detail.log mprelay.log pan_comm_0.log

pan_dha.log pan_task_1.log pan_task_10.log pan_task_11.log

pan_task_2.log pan_task_3.log pan_task_4.log pan_task_5.log

pan_task_6.log pan_task_7.log pan_task_8.log pan_task_9.log

panio.log panio.log.old pdtrc.log supervisor.log

sysdagent.log

A nifty little tool is provided to aggregate these files into a single file:

> debug dataplane packet-diag aggregate-logs

packet-diag.log is aggregated

The final output file is then stored on the management plane as pan_packet_diag.log:

> less dp-log pan_packet_diag.log for platforms with separate/multiple dataplanes

> less mp-log pan_packet_diag.log for vm and platforms with integrated dataplane

Let's take a look at the stages a packet goes through as it is seen in flow basic:

The packet is received on the ingress interface and checked to see if it matches an existing session. If not, it is sent to
'slowpath' for session creation.

== 2016-02-10 14:53:09.977 -0800 ==

Packet received at ingress stage

Packet info: len 66 port 18 interface 18 vsys 1

wqe index 208735 packet 0x0x80000000b4d600c6

Packet decoded dump:

L2: 00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800

IP: 192.168.0.34->198.51.100.97, protocol 6

version 4, ihl 5, tos 0x00, len 52,

id 93, frag_off 0x4000, ttl 128, checksum 3848

TCP: sport 64984, dport 80, seq 2270663711, ack 0,

reserved 0, offset 8, window 8192, checksum 18967,

flags 0x0002 ( SYN), urgent data 0

TCP option:

00000000: 02 04 05 b4 01 03 03 02 01 01 04 02 ........ ....

Flow lookup, key word0 0xfdd8005000010600 word1 0

Session setup: vsys 1

No active flow found, enqueue to create session

Next, slowpath receives the packet. In slowpath, the packet is checked for source and destination zone based on routes or
PBF entries. The packet's also checked to see if security rules exist that allow this session, based on the 5 tuples (source
zone, source IP subnet, destination zone, destination IP subnet, destination port), and if NAT needs to be applied. If
everything checks out, a session is created.

== 2016-02-10 14:53:09.978 -0800 ==

Packet received at slowpath stage

Packet info: len 66 port 18 interface 18 vsys 1

wqe index 208735 packet 0x0x80000000b4d600c6

Packet decoded dump:

L2: 00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800

IP: 192.168.0.34->198.51.100.97, protocol 6

version 4, ihl 5, tos 0x00, len 52,

id 93, frag_off 0x4000, ttl 128, checksum 3848

TCP: sport 64984, dport 80, seq 2270663711, ack 0,

reserved 0, offset 8, window 8192, checksum 18967,

flags 0x0002 ( SYN), urgent data 0

TCP option:

00000000: 02 04 05 b4 01 03 03 02 01 01 04 02 ........ ....

Session setup: vsys 1

PBF lookup (vsys 1) with application web-browsing

Session setup: ingress interface ethernet1/3 egress interface ethernet1/1 (zone 5)

NAT policy lookup, matched rule index 0

Policy lookup, matched rule index 0,

Allocated new session 51187.

Packet matched vsys 1 NAT rule 'outbound-nat' (index 1),

source translation 192.168.0.34/64984 => 198.51.100.230/52924

Created session, enqueue to install

The packet is forwarded to fastpath, NAT translation is applied, and the translated packet is sent out of the egress interface
to the next hop.
== 2016-02-10 14:53:09.978 -0800 ==

Packet received at fastpath stage

Packet info: len 66 port 18 interface 18 vsys 1

wqe index 208735 packet 0x0x80000000b4d600c6

Packet decoded dump:

L2: 00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800

IP: 192.168.0.34->198.51.100.97, protocol 6

version 4, ihl 5, tos 0x00, len 52,

id 93, frag_off 0x4000, ttl 128, checksum 3848

TCP: sport 64984, dport 80, seq 2270663711, ack 0,

reserved 0, offset 8, window 8192, checksum 18967,

flags 0x0002 ( SYN), urgent data 0

TCP option:

00000000: 02 04 05 b4 01 03 03 02 01 01 04 02 ........ ....

Flow fastpath, session 51187

2016-02-10 14:53:09.978 -0800 pan_flow_process_fastpath(src/pan_flow_proc.c:1414):

SESSION-DSCP: set se

ssion DSCP: 0x00

NAT session, run address/port translation

Syn Cookie: pan_reass(Init statete): c2s:0 c2s:nxtseq 2270663712 c2s:startseq 2270663712

c2s:win 0 c2s:s

t 3 c2s:newsyn 0 :: s2c:nxtseq 0 s2c:startseq 0 s2c:win 8192 s2c:st 0 s2c:newsyn 0 ack 0

nosyn 0 plen 0

Forwarding lookup, ingress interface 18

L3 mode, virtual-router 1

Route lookup in virtual-router 1, IP 198.51.100.97

Route found, interface ethernet1/1, zone 5

Resolve ARP for IP 198.51.100.97 on interface ethernet1/1

ARP entry found on interface 16

Transmit packet on port 16

The returning SYN/ACK packet is received at the ingress stage and matched to the existing session, it is then forwarded to
the fastpath stage. Reverse NAT is applied and the packet is sent out of the internal interface back to the client:

== 2016-02-10 14:53:09.978 -0800 ==

Packet received at ingress stage

Packet info: len 66 port 16 interface 16 vsys 1

wqe index 206169 packet 0x0x80000000b72c28c6

Packet decoded dump:

L2: 00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800

IP: 198.51.100.97->198.51.100.230, protocol 6

version 4, ihl 5, tos 0x00, len 52,

id 10919, frag_off 0x4000, ttl 128, checksum 31342

TCP: sport 80, dport 52924, seq 3191786586, ack 2270663712,

reserved 0, offset 8, window 8192, checksum 33843,

flags 0x0012 ( SYN ACK), urgent data 0

TCP option:

00000000: 02 04 05 b4 01 03 03 08 01 01 04 02 ........ ....

Flow lookup, key word0 0x50cebc00050600 word1 0

Flow 102375 found, state 2, HA 0

Active flow, enqueue to fastpath process

== 2016-02-10 14:53:09.979 -0800 ==

Packet received at fastpath stage

Packet info: len 66 port 16 interface 16 vsys 1

wqe index 206169 packet 0x0x80000000b72c28c6

Packet decoded dump:

L2: 00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800

IP: 198.51.100.97->198.51.100.230, protocol 6

version 4, ihl 5, tos 0x00, len 52,

id 10919, frag_off 0x4000, ttl 128, checksum 31342

TCP: sport 80, dport 52924, seq 3191786586, ack 2270663712,

reserved 0, offset 8, window 8192, checksum 33843,

flags 0x0012 ( SYN ACK), urgent data 0

TCP option:

00000000: 02 04 05 b4 01 03 03 08 01 01 04 02 ........ ....

Flow fastpath, session 51187

NAT session, run address/port translation

Syn Cookie: pan_reass(Init statete): c2s:1 c2s:nxtseq 2270663712 c2s:startseq 2270663712

c2s:win 8192 c2

s:st 3 c2s:newsyn 0 :: s2c:nxtseq 3191786587 s2c:startseq 3191786587 s2c:win 8192 s2c:st 3

s2c:newsyn 0

ack 2270663712 nosyn 0 plen 0

Forwarding lookup, ingress interface 16

L3 mode, virtual-router 1

Route lookup in virtual-router 1, IP 192.168.0.34

Route found, interface ethernet1/3, zone 1

Resolve ARP for IP 192.168.0.34 on interface ethernet1/3

ARP entry found on interface 18

Transmit packet on port 18

The final ACK to complete the handshake is received, which triggers the session to be registered and the flow to be created
in the fastpath stage. No further route or NAT lookups will need to be performed by the firewall.
== 2016-02-10 14:53:09.979 -0800 ==

Packet received at ingress stage

Packet info: len 60 port 18 interface 18 vsys 1

 wqe index 193163 packet 0x0x80000000b49c60c6

Packet decoded dump:

L2: 00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800

IP: 192.168.0.34->198.51.100.97, protocol 6

version 4, ihl 5, tos 0x00, len 40,

id 94, frag_off 0x4000, ttl 128, checksum 3859

TCP: sport 64984, dport 80, seq 2270663712, ack 3191786587,

reserved 0, offset 5, window 16425, checksum 57360,

flags 0x0010 ( ACK), urgent data 0

TCP option:

Flow lookup, key word0 0xfdd8005000010600 word1 0

Flow 102374 found, state 2, HA 0

Active flow, enqueue to fastpath process

== 2016-02-10 14:53:09.979 -0800 ==

Packet received at fastpath stage

Packet info: len 60 port 18 interface 18 vsys 1

wqe index 193163 packet 0x0x80000000b49c60c6

Packet decoded dump:

L2: 00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800

IP: 192.168.0.34->198.51.100.97, protocol 6

version 4, ihl 5, tos 0x00, len 40,

id 94, frag_off 0x4000, ttl 128, checksum 3859

TCP: sport 64984, dport 80, seq 2270663712, ack 3191786587,

reserved 0, offset 5, window 16425, checksum 57360,

flags 0x0010 ( ACK), urgent data 0

TCP option:

Flow fastpath, session 51187

NAT session, run address/port translation

Forwarding lookup, ingress interface 18

L3 mode, virtual-router 1

Route lookup in virtual-router 1, IP 198.51.100.97

Route found, interface ethernet1/1, zone 5

Resolve ARP for IP 198.51.100.97 on interface ethernet1/1

ARP entry found on interface 16

Transmit packet on port 16

Client and server are now free to start communicating:

== 2016-02-10 14:53:09.980 -0800 ==

Packet received at ingress stage

Packet info: len 398 port 18 interface 18 vsys 1

wqe index 202239 packet 0x0x80000000b341e8c6

Packet decoded dump:

L2: 00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800

IP: 192.168.0.34->198.51.100.97, protocol 6

version 4, ihl 5, tos 0x00, len 384,

id 95, frag_off 0x4000, ttl 128, checksum 3514

TCP: sport 64984, dport 80, seq 2270663712, ack 3191786587,

reserved 0, offset 5, window 16425, checksum 24524,

flags 0x0018 ( ACK PSH), urgent data 0

TCP option:

Flow lookup, key word0 0xfdd8005000010600 word1 0

Flow 102374 found, state 2, HA 0

Active flow, enqueue to fastpath process

== 2016-02-10 14:53:09.980 -0800 ==

Packet received at fastpath stage

Packet info: len 398 port 18 interface 18 vsys 1

wqe index 202239 packet 0x0x80000000b341e8c6

Packet decoded dump:

L2: 00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800

IP: 192.168.0.34->198.51.100.97, protocol 6

version 4, ihl 5, tos 0x00, len 384,

id 95, frag_off 0x4000, ttl 128, checksum 3514

TCP: sport 64984, dport 80, seq 2270663712, ack 3191786587,

reserved 0, offset 5, window 16425, checksum 24524,

flags 0x0018 ( ACK PSH), urgent data 0

TCP option:

Flow fastpath, session 51187

NAT session, run address/port translation

session 51187 packet sequeunce old 0 new 1

Forwarding lookup, ingress interface 18

L3 mode, virtual-router 1

Route lookup in virtual-router 1, IP 198.51.100.97

Route found, interface ethernet1/1, zone 5

Resolve ARP for IP 198.51.100.97 on interface ethernet1/1

ARP entry found on interface 16

Transmit packet on port 16

== 2016-02-10 14:53:09.981 -0800 ==

Packet received at ingress stage

Packet info: len 242 port 16 interface 16 vsys 1

wqe index 211483 packet 0x0x80000000b58550c6

Packet decoded dump:

L2: 00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800

IP: 198.51.100.97->198.51.100.230, protocol 6

version 4, ihl 5, tos 0x00, len 228,

id 10920, frag_off 0x4000, ttl 128, checksum 31165

TCP: sport 80, dport 52924, seq 3191786587, ack 2270664056,

reserved 0, offset 5, window 256, checksum 13539,

flags 0x0018 ( ACK PSH), urgent data 0

TCP option:

Flow lookup, key word0 0x50cebc00050600 word1 0

Flow 102375 found, state 2, HA 0

Active flow, enqueue to fastpath process

== 2016-02-10 14:53:09.981 -0800 ==

Packet received at fastpath stage

Packet info: len 242 port 16 interface 16 vsys 1

wqe index 211483 packet 0x0x80000000b58550c6

Packet decoded dump:

L2: 00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800

IP: 198.51.100.97->198.51.100.230, protocol 6

version 4, ihl 5, tos 0x00, len 228,

id 10920, frag_off 0x4000, ttl 128, checksum 31165

TCP: sport 80, dport 52924, seq 3191786587, ack 2270664056,

reserved 0, offset 5, window 256, checksum 13539,

flags 0x0018 ( ACK PSH), urgent data 0

TCP option:

Flow fastpath, session 51187

NAT session, run address/port translation

session 51187 packet sequeunce old 1 new 2

Forwarding lookup, ingress interface 16

L3 mode, virtual-router 1

Route lookup in virtual-router 1, IP 192.168.0.34

Route found, interface ethernet1/3, zone 1

Resolve ARP for IP 192.168.0.34 on interface ethernet1/3

ARP entry found on interface 18

Transmit packet on port 18

You can now use flow basic to follow the packets through the Palo Alto Networks firewall, to better understand all the stages
a packet goes through. Also take a look at this article that explains all this in greater detail: Packet Flow Sequence in
PANOS

When you feel comfortable, feel free to add additional features like 'appid' for more detail, but do keep an eye on the
dataplane resource-monitor to make sure the dataplane is not getting taxed.

I hope you found this article interesting. If you haven't already done so, please check out the other articles in the Getting
Started series.

Feel free to leave a comment below.

Regards,
Tom