Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

If you have intermittent issues with internet traffic, management of the device, or slowness with any other traffic passing through the Palo Alto Networks device, below are several commands with a brief description of each command, to check the load on the device .

>show system info this will give you the system information. Please check the uptime if appears to be reset then the device or dataplane has been reset.

hostname: Corp-FCS-vwire ip-address: 10.16.3.222 netmask: 255.255.252.0 default-gateway: 10.16.0.1 mac-address: 00:30:48:61:67:b8

time: Wed Jan 28 21:04:19 2009

uptime: 1 days, 7:35:43 family: 4000 model: PA-4050 serial: 0001a100269 sw-version: 2.0.8-h1 app-version: 106-807

Generated by Jive SBS on 2011-07-12-05:00 1

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

threat-version: 106-807 url-filtering-version: 2191 logdb-version: 2.0.5

> show proxy memory This command will be added in the initial release of 3.0. To monitor the ssl decryption memory usage; the first sz malloc size is the value to track. This value should increment/decrement, it is a concern if this value only increments. The max value is around 16 mb.

In PANOS 3.1 and above this is broken down into two commands: >show system setting ssl-decrypt memory >show system setting ssl-decrypt certificate-cache

proxy allocator alloc size 516387, max 553169 fixed buf allocator, size 16767736 sz malloc size 1119232, max 1283072

ssl cert cache allocator alloc size 269178, max 269178 fixed chunk allocator, size 8376144 chunk size 3072 malloc size 688128, max 688128

Generated by Jive SBS on 2011-07-12-05:00 2

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

> show system resources this is a snap-shot of the current system processor activity.

top - 21:55:51 up 1 day, 8:27, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 77 total, 1 running, 76 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1035916k total, 1017948k used, 17968k free, 151768k buffers

Swap: 2008084k total, 1000820k used, 1007264k free, 496848k cached

PID USER 1 root 2 root 3 root 4 root 5 root 6 root 7 root 8 root 9 root 12 root 13 root 14 root 120 root

PR NI VIRT RES SHR S %CPU %MEM 16 0 1648 472 452 S RT 0 34 19 RT 0 34 19 10 -5 0 0 0 0 0 0 0 0 0 0 0 0 0 0S 0 0.0 0:01.11 init

TIME+ COMMAND

0 0.0 0:00.00 migration/0 0 0.0 0:00.00 ksoftirqd/0 0 0.0 0:00.00 migration/1 0 0.0 0:00.00 ksoftirqd/1 0 0.0 0:00.00 events/0 0 0.0 0:00.00 events/1 0 0.0 0:00.00 khelper 0 0.0 0:00.00 kthread 0 0.0 0:00.01 kblockd/0 0 0.0 0:00.03 kblockd/1 0 0.0 0:00.00 kacpid 0 0.0 0:00.00 khubd

0 0S 0 0 0 0 0 0 0 0 0 0 0S 0S 0S 0S 0S 0S 0S 0S 0S 0S

10 -5 10 -5 10 -5 10 -5 10 -5 13 -5 10 -5

Generated by Jive SBS on 2011-07-12-05:00 3

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

122 root 178 root 179 root

10 -5 15 0 15 0

0 0 0

0 0 0

0S 0S 0S

0 0.0 0:00.00 kseriod 0 0.0 0:00.01 pdflush 0 0.0 0:02.23 kswapd0

> show session info this will show session statistics and session configuration parameters. Perform this command a few times to make sure that the active sessions increment.

------------------------------------------------------------------------------number of sessions supported: number of active sessions: number of active TCP sessions: number of active UDP sessions: number of active ICMP sessions: session table utilization: 0% 2097151 543 378 148 3

number of sessions created since system bootup: 912668 Packet rate: Throughput: 234/s 1067 Kbps

------------------------------------------------------------------------------session timeout TCP default timeout: 3600 seconds 5 seconds

TCP session timeout before 3-way handshaking: TCP session timeout after FIN/RST:

30 seconds

Generated by Jive SBS on 2011-07-12-05:00 4

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

UDP default timeout: ICMP default timeout: other IP default timeout:

30 seconds 6 seconds 30 seconds

------------------------------------------------------------------------------session accelerated aging: accelerated aging threshold: scaling factor: 2X enabled 80% of utilization

------------------------------------------------------------------------------session setup TCP - reject non-SYN first packet: hardware session offloading: yes yes

------------------------------------------------------------------------------application trickling scan parameters: timeout to determine application trickling: 10 seconds resource utilization threhold to start scan: 80% scan scaling factor over regular aging: 8

-------------------------------------------------------------------------------

> show system statistics this will display real time system statistics; there are additional keys to switch the display. The menu is included below.

Device is up Packet rate

: 1 day 9 hours 10 mins 55 sec : 597/s

Generated by Jive SBS on 2011-07-12-05:00 5

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

Throughput

: 3211 Kbps

Total active sessions : 642 Active TCP sessions : 477 Active UDP sessions : 155 Active ICMP sessions : 4

You can type the following key to switch what to display -------------------------------------------------------'a' - Display application statistics 'h' - Display this help page 'l' - Display logging statistics 'q' - Quit this program 's' - Display system statistics

> debug dataplane pool statistics this command will show the current pool usage, the second number represents the buffer size and the first number represents how many buffers are still available.

Hardware Pools [ 0] Packet Buffers [ 1] Work Queue Entries [ 2] Output Buffers [ 3] DFA Result : : : 57240/57344 0x8000000410000000

: 229290/229376 0x8000000417000000 975/1024 4095/4096 0x8000000418c00000 0x8000000419100000

Generated by Jive SBS on 2011-07-12-05:00 6

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

DFA Result [ 4] Timer Buffers Timer Buffers

: : : : 1024/1024 1023/1024 0x8000000419500000 0x8000000419540000 4092/4096 0x8000000418d00000

[ 5] Buffers with 256 bytes

[ 6] Buffers with 2048 bytes :

Software Pools [ 0] software packet buffer 0 : [ 1] software packet buffer 1 : [ 2] software packet buffer 2 : [ 3] software packet buffer 3 : [ 4] software packet buffer 4 : [ 5] Pktlog logs [ 6] Pktlog threats [ 7] Pktlog packet [ 8] Pktlog large [ 9] CTD Flow [10] CTD AV Block [11] SML VM Fields [12] SML VM Vchecks [13] Detector Threats [14] Regex Results : : : : : : 65514/65536 32768/32768 32768/32768 32768/32768 256/256 0x8000000024d00680 0x8000000026d50780 0x8000000028d78880 0x800000002cda0980 0x800000004edc8a80 0x8000000020c68930 0x8000000020ebec70 0x8000000020fe9e90

10000/10000 4999/5000 4999/5000 56/56

0x8000000021871cf0

: 1048302/1048576 0x8000000099365498 : 32/32 0x80000000b9865598

: 130843/131072 0x80000000b986d718 : 65536/65536 64710/65536 512/512 0x80000000b9d0d818 0x80000000b9e5d918

0x8000000021bf9090

Generated by Jive SBS on 2011-07-12-05:00 7

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

[15] TIMER Chunk [16] FPTCP segs [17] Proxy session : :

: 131072/131072 0x80000000bbbf6460 32768/32768 16384/16384 : 0x80000000bdc96588 0x80000000bdd3e688 0x80000000c2892788

[18] SSL Handshake State

32768/32768

> debug dataplane show resource-monitor this will show the cpu load for different time frames, look for values that are 90% and higher.

PANOS 3.1.x > show running resource-monitor

Resource monitoring sampling data (per second): CPU load (%) during last 60 seconds: core 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Generated by Jive SBS on 2011-07-12-05:00 8

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Generated by Jive SBS on 2011-07-12-05:00 9

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0

Generated by Jive SBS on 2011-07-12-05:00 10

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Resource utilization (%) during last 60 seconds: session: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

packet buffer: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

packet descriptor: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Generated by Jive SBS on 2011-07-12-05:00 11

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

packet descriptor (on-chip): 2 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 2 2 2 2 1 2 2 2 2 2 2 2 2 1 1 1 2 1 2 2 1 2 2 1 1 1 1 1 1 1 2 2 1 1 2 1 1 1 1 1 1 2 1 2

Resource monitoring statistics (per minute): CPU load (%) during last 60 minutes: core 0 1 2 3 4 5 6 7

avg max avg max avg max avg max avg max avg max avg max avg max 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 2 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 2 0 1 0 0 0 0 0 2 0 0 0 0 0 0 0 1 0 2 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0

Generated by Jive SBS on 2011-07-12-05:00 12

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1

> show counter global this will display all of the system counters, verify that the pkt recv and pkt sent counters are incrementing.

Global counters: name value description

Generated by Jive SBS on 2011-07-12-05:00 13

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

------------------------------------------------------------------------------pkt_recv pkt_recv_err pkt_recv_multiple_bufs pkt_recv_short_pkt pkt_recv_throttle_cos pkt_sent pkt_sent_err pkt_outstanding pkt_alloc 0 0 82097891 13 0 3236305 128014692 0 Packets received

Packet receive error 0 Packets received with multiple buffers Packet receive short packets Packets throttled by QoS control Packets transmitted Packet transmit error Outstanding packet to be transmitted Packets allocated

> show counter global | match drop this will display all of the system counters with the word drop, perform this command a few times and look for any counter that is incrementing at a high rate.

flow_rcv_err flow_no_interface flow_np_rcv_err flow_np_rcv_ihdr_err flow_np_rcv_tag_err flow_scan_drop flow_tcp_non_syn_drop match

293 0 0 0

Packets dropped: flow stage receive error Packets dropped: invalid interface Packets dropped: receive error from offload processor Packets dropped: invalid packet header Packets dropped: invalid packet header content Session setup: denied by scan detection Packets dropped: non-SYN TCP without session

0 0

10886

Generated by Jive SBS on 2011-07-12-05:00 14

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

> show counter global | match deny - this will display all of the system counters with the word deny, perform this command a few times and look for any counter that is incrementing at a high rate.

flow_policy_deny flow_host_service_deny binahara@Corp-FCS-vwire>

Session setup: denied by policy 0 Device management session denied

> show counter global | match syn - this will display all of the system counters with the word syn, perform this command a few times and look for any counter that is incrementing at a high rate.

flow_tcp_non_syn flow_tcp_non_syn_drop match flow_parse_l4_tcpsynurg flow_parse_l4_tcpsynrst flow_parse_l4_tcpsynfin flow_dos_red_tcp RED flow_dos_syncookie reached

10896

Non-SYN TCP packets without session match Packets dropped: non-SYN TCP without session

10896

0 0 0 0

Packets dropped: invalid TCP flags (SYN+URG+*) Packets dropped: invalid TCP flags (SYN+RST+*) Packets dropped: invalid TCP flags (SYN+FIN+*)

Packet dropped: Zone protection protocol "tcp-syn"

Packet dropped: SYN cookies maximum threshold

flow_dos_syncookie_cookie_sent flow_dos_syncookie_ack_recv flow_dos_syncookie_ack_err flow_dos_syncookie_svr_ack_recv

0 0 0

TCP SYN cookies: cookies sent TCP SYN cookies: ACKs to cookies received TCP SYN cookies: Invalid ACKs received

TCP SYN cookies: Server ACKs received

Generated by Jive SBS on 2011-07-12-05:00 15

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

tcp_syn_missing

10150

miss SYN packet for tcp session

> show counter global | match error - this will display all of the system counters with the word error, perform this command a few times and look for any counter that is incrementing at a high rate.

pkt_recv_err pkt_sent_err pkt_alloc_failure pkt_alloc_failure_cos pkt_swbuf_alloc_failure wqe_alloc_failure session_alloc_failure session_install_error session_state_error session_peer_not_close session_timer_error flow_rcv_err

0 13 0

Packet receive error Packet transmit error Packet allocation error 0 0 0 0 0 0 Packet allocation error due to QoS control Software packet buffer allocation error Packet descriptor allocation error Session allocation error Sessions installation error Session state error 0 0 installation flow close error

Session aging timer error Packets dropped: flow stage receive error

293

> show system state - this will display a snap-shot of the full system, there are several lines (about 1000) for this command.

<response status="success"><result>cfg.agent.buf-size: 0xf00000 cfg.agent.max-buckets: 0x8000

Generated by Jive SBS on 2011-07-12-05:00 16

Troubleshooting Slowness with Traffic, Management, or Intermittent SSL Decryption

cfg.app.capture.disk: 0x1400000 cfg.apptracker.entries: 0x10000 cfg.capability.regex.alt: 0x0 cfg.cdfa.buf-size: 0x500000 cfg.cfg.buf-size: 0xc00000 cfg.cfg.general.max-device: 1 cfg.cfg.if-shm-size: 0x1000000 cfg.cfg.max-pool-entry: 0x200 cfg.cfg.max-ucache-entry: 0x9c40 cfg.cfg.ucache-size: 0xa00000 cfg.cfg.vsys-size-large: 0x200000 cfg.cfg.vsys-size-medium: 0x80000 cfg.cfg.vsys-size-small: 0x10000

Generated by Jive SBS on 2011-07-12-05:00 17