VM-Series Troubleshooting
Tech Note
PAN-OS 5.0

Revision A ©2012, Palo Alto Networks, Inc. www.paloaltonetworks.com

  

C ontents
Overview ................................................................................................................................................................................ 3  
Troubleshooting Focus Areas ................................................................................................................................................. 3  
Installation Issues ................................................................................................................................................................ 3  
The OVF ......................................................................................................................................................................... 3  
vSphere Client ................................................................................................................................................................. 3  
ESXi Configuration Requirements .................................................................................................................................. 3  
Console Access ................................................................................................................................................................ 4  
Licensing Issues ................................................................................................................................................................... 4  
Adding licenses in the wrong order ................................................................................................................................. 4  
License Status .................................................................................................................................................................. 6  
Insufficient sessions before license is applied ................................................................................................................... 6  
Impact of cloning a licensed device.................................................................................................................................. 7  
Connectivity Issues ............................................................................................................................................................. 7  
Interfaces ........................................................................................................................................................................ 7  
No Traffic ....................................................................................................................................................................... 8  
Poor Performance ......................................................................................................................................................... 11  
Network Troubleshooting Tools ................................................................................................................................... 13  
Summary .............................................................................................................................................................................. 13  

©2012, Palo Alto Networks, Inc. [2]

  
Overview
The VM-Series firewall is the first virtualized version of the Next Generation PAN-OS firewall. It allows for safe application
enablement in the virtualized data center – particularly for traffic that may never cross a physical network. The VM-Series
can be installed on each ESXi host and can protect traffic between virtual machines on the same host, between virtual
machine on different hosts and between virtual machines and the external network.

Because the VM-Series runs in a completely virtual environment, troubleshooting issues requires an understanding of not
only PAN-OS but also the underlying infrastructure. This document covers the most common issues encountered and how
to best detect and resolve them.

Troubleshooting Focus Areas

The key areas this document focuses on for troubleshooting are the:
• Installation issues
• Licensing issues
• Connectivity issues

Installation Issues
The OVF
The VM-Series is delivered as a downloadable Open Virtualization Format (OVF) file. The OVF is downloaded as a zip
archive that is expanded into three files. The ovf extension is for the OVF descriptor file that contains all metadata about the
package and its contents. The mf extension is for the OVF manifest file that contains the SHA-1 digests of individual files in
the package and the vmdk extension is for the virtual disk image file. If you are having trouble deploying the OVF, make
sure the three files are unpacked and present and if necessary, download and extract the OVF again.

The virtual disk in the OVF is large for the VM-Series - nearly 900MB and must be present on the computer running the
vSphere client. Make sure the network connection is sufficient between the vSphere client computer and the target ESXi
host. There needs to be sufficient bandwidth and low latency on the connection otherwise the OVF deployment can take
hours or timeout and fail.

vSphere Client
For ESXi 4.1 and 5.0, the vSphere client must be installed on a computer running Windows XP or later. It will not run on
MAC-OS or any Unix/Linux distribution. The vSphere client can be downloaded from VMware’s website or by browsing to
the ESXi host directly. Not all versions of the vSphere Client are compatible with all versions of vSphere and if the client is
older, it may need to be upgraded. You will automatically be prompted to do so if needed.

As mentioned above, the computer running vSphere client will need sufficient bandwidth and low latency to the ESXi host.
Also, any firewalls in the path will need to allow TCP ports 902 and 443 from the vSphere client to the ESXi host(s).

Note: For a complete guide on how to setup and initially configure the VM-Series firewall, please refer to the “VM-Series
Deployment” TechNote.

ESXi Configuration Requirements

The new VM-Series guest virtual machine must be configured with the minimum settings: 4 GB memory, 2 VCPUs, and
40GB storage. In addition, the VM-Series firewall will need at least two virtual interfaces (VMNICs) setup and the must be
VMXNET3 VMNICs. The first one is always for the management interface. The second interface is for the data-plane
(Ethernet 1/1.) Any additional interfaces will be used for the data-plane up to a total of ten VMNICs (one for management
and nine for the data-plane – Ethernet1/1 through Ethernet 1/9.)

It is important to keep the interface mapping correct. For example, in the screen shot below this VM-Series firewall has
three interfaces:

 
©2012, Palo Alto Networks, Inc. [3]
  

As mentioned above, the first adapter is always the management interface. This means that the data-plane adapters map as
follows:

Network adapter 2 à Ethernet 1/1

Network adapter 3 à Ethernet 1/2

This can be confusing when troubleshooting and it is important to keep the offset correct.

C onsole Access
Console access is only available through the vSphere client. Over high latency or low bandwidth connections, auto-repeat
may start prematurely causing unintended repeating characters. The solution is as follows (from the VMware website):

• Log in with the vSphere client

• Power off the virtual machine
• Right click virtual machine select Edit Settings
• Click Options > General > Configuration Parameters
• Click Add Row
• Under Name enter keyboard.typematicMinDelay
• In the Value field 2000000
• Click OK
• Power on the virtual machine

As with a physical firewall, any issues connecting to the management interface may require console access. Also as with
physical firewalls, the console is used for entering and making changes while in maintenance mode.

Licensing Issues
After the VM-Series firewall is installed, it will need to be licensed to pass more than a minimal number of sessions,
download software updates, get subscriptions, etc. Troubleshooting a new deployment should start with verifying licenses.
The most common license issues are:
1. Adding licenses in the wrong order
2. License Status
3. Insufficient sessions before license is applied
4. Impact of cloning a licensed device

Adding licenses in the wrong order

Before any support or feature licenses can be applied, the capacity authcode must be applied first. The capacity authcode
determines whether the VM-Series will be a VM-100, VM-200 or VM-300. This must be specified first before things like
software updates can be downloaded and installed.
 
©2012, Palo Alto Networks, Inc. [4]
  
You can verify the capacity license is valid by looking at the dashboard under general settings:

A VM-Series without a valid license will have no “VM License” and the Serial # will be “unknown”:

After the capacity authcode has been applied, the support license can be applied. You will need to refresh the support
license page after you apply the license to see the updated status:

If you attempt to apply the support authcode before the capacity authcode is applied, you will get a generic error:

 
©2012, Palo Alto Networks, Inc. [5]
  

License Status
In the WEB-UI, the support license is not shown in the Licenses section. You must go to the Support section under Device /
Support:

As mentioned above, the support license will not immediately display after successfully activating a support authorization
code. You will need to refresh the display to see the valid support license.

Insufficient sessions before license is applied

Before a new VM-Series firewall is licensed, it will pass a minimum amount of traffic for testing. The number of concurrent
sessions is limited to 200. Depending on the environment, the session limit can be reached very quickly. Testing an
unlicensed VM-Series firewall can yield unpredictable results, even for a basic test like ping, if there is other traffic on the
port group(s).

 
©2012, Palo Alto Networks, Inc. [6]
  
Impact of cloning a licensed device
Each instance of VM-Series has two identifiers associated with it: the Universally Unique ID (UUID) and CPU ID. The CPU
ID comes from the host CPU and is not unique to the VM-Series firewall. All virtual machines on the same host (physical
server) will have the same CPU ID.

The UUID is unique to each virtual machine including the VM-Series firewall. If a virtual machine is migrated to another
host, the UUID will remain the same.

If virtual machine is cloned, the UUID will change. Because the Serial number and license for a VM-Series instance is tied to
the UUID, cloning a licensed VM-Series firewall will result in a new firewall with an invalid license. This new clone will
need to have the license removed and a new auth code activated. Until the clone receives a new license, it will behave the
same as a new, unlicensed firewall and will have the low session limit, lack of support, inability to download new software,
etc.

Connectivity Issues
Interfaces
Most connectivity issues for the VM-Series are resolved the same as they are for a physical firewall. As with a physical
firewall, be sure to check the interfaces, the interface type (tap, vwire, layer 2, layer 3.) Link speed and duplex are not
relevant for a virtual interface and do not need to be checked. Also, be sure to check for interface errors using the show
interfaces command.

warby@DEMO1> show interface all

total configured hardware interfaces: 7

name id speed/duplex/state mac address

--------------------------------------------------------------------------------
ethernet1/1 16 ukn/ukn/down(disabled) 00:1b:17:ec:7d:10
ethernet1/2 17 ukn/ukn/down(disabled) 00:1b:17:ec:7d:11
ethernet1/3 18 ukn/ukn/down(disabled) 00:1b:17:ec:7d:12
ethernet1/4 19 ukn/ukn/down(disabled) 00:1b:17:ec:7d:13
vlan 1 [n/a]/[n/a]/up 00:1b:17:ec:7d:01
loopback 3 [n/a]/[n/a]/up 00:1b:17:ec:7d:03
tunnel 4 [n/a]/[n/a]/up 00:1b:17:ec:7d:04

aggregation groups: 0

total configured logical interfaces: 7

name id vsys zone forwarding tag address

------------------- ----- ---- ---------------- ------------------------ ------ ------------------

ethernet1/1 16 1 ServerA vlan:101 0 N/A
ethernet1/2 17 1 ServerB vlan:101 0 N/A
ethernet1/3 18 1 Untrust vlan:101 0 N/A
ethernet1/4 19 1 N/A 0 N/A
vlan 1 1 N/A 0 N/A
loopback 3 1 N/A 0 N/A
tunnel 4 1 N/A 0 N/A

 
©2012, Palo Alto Networks, Inc. [7]
  
warby@DEMO1> show interface ethernet1/1

--------------------------------------------------------------------------------
Name: ethernet1/1, ID: 16
Link status:
Runtime link speed/duplex/state: unknown/unknown/down
Configured link speed/duplex/state: auto/auto/auto
MAC address:
Port MAC address 00:1b:17:ec:7d:10
Operation mode: layer2
Untagged sub-interface support: no
--------------------------------------------------------------------------------
Name: ethernet1/1, ID: 16
Operation mode: layer2
Interface management profile: N/A
Service configured:
Zone: ServerA, virtual system: vsys1
Adjust TCP MSS: no
--------------------------------------------------------------------------------

(remaining output ommitted)

N o Traffic
One of the most common problems that is specific to the VM-Series is an interface not receiving any traffic from the
network. For example, a virtual machine is attached to the same port group as a VM-Series firewall and the virtual machine
is sending traffic but not getting any response.

In this specific scenario, the traffic log has no recent entries and the counters are empty:

warby@DEMO1> show counter global filter delta yes

Global counters:
Elapsed time since last sampling: 594.544 seconds

--------------------------------------------------------------------------------
Total counters shown: 0
--------------------------------------------------------------------------------

So we can see this isn’t an issue of dropped packets – they aren’t even showing up on the interface. At this point, we suspect
the underlying infrastructure. The interface is up but it is behaving similarly to a physical firewall port with no cable.

In the vSphere environment, we need to check the following:

1. Check the port groups
a. Giving port groups helpful names (like “VLAN 101” or “Management PG” can aid with troubleshooting.
b. Are the firewall and the virtual machine(s) on the correct port group?
c. For each VM, check the settings to verify the interface is mapped to the correct port group:

 
©2012, Palo Alto Networks, Inc. [8]
  

d. Make sure the interfaces are mapped correctly (Network adapter 1 = management, Network adapter 2 =
Ethernet1/1, Network adapter 3 = Ethernet1/2, etc.)
2. Check for promiscuous mode
a. Since the data-plane PAN-OS MAC addresses are different than the VMNIC MAC addresses assigned by
vSphere, the port group (or the entire vSwitch) must be in promiscuous mode:

3. Check the VLAN settings in vSphere

a. The use of the VLAN setting for the vSphere port group serves two purposes
i. It determines which port groups share a layer 2 domain
ii. And it determines whether the uplink ports are tagged (802.1Q)

 
©2012, Palo Alto Networks, Inc. [9]
  

4. Check the physical switch port settings

a. If a VLAN ID is specified on a port group with uplink ports, then vSphere will use 802.1Q to tag outbound
frames.
b. The tag must match the configuration on the physical switch or the traffic will not pass
5. Check the port statistics if using virtual distributed switches (vDS)
a. Standard switches do not provide any port statistics
b. But distributed switches do:

6. The columns displayed in this view can be turned on and off which is helpful when troubleshooting:

 
©2012, Palo Alto Networks, Inc. [10]
  

If all other non-test traffic can be eliminated during troubleshooting, running a simple ping with one packet per second while
watching the port statistics (Unicast – ingress and Unicast – egress) can help isolate the problem.

Poor Performance
Firewall performance troubleshooting should start as with a physical firewall – look at management and data-plane CPU
utilization, running jobs, traffic load etc. In addition, we need to take into account the environment the VM-Series firewall
is running in. Resources are shared between multiple virtual machines and PAN-OS isn’t necessarily guaranteed any amount
of CPU cycles.

vSphere provides many helpful tools for monitoring the resources allocated and used by virtual machines. To troubleshoot
possible contention on the physical host:
1. Start with the host-wide view of resources and check the allocation

2. Next, check the historic use of the CPU and memory.

a. Use the overview for some highlights:

 
©2012, Palo Alto Networks, Inc. [11]
  

b. Use advanced to get more detail:

 
©2012, Palo Alto Networks, Inc. [12]
  

N etwork Troubleshooting Tools

As in a physical environment, it is sometimes useful to have a separate troubleshooting station to capture traffic or inject test
packets in the virtualized environment. It can be helpful to build a fresh OS from scratch with common troubleshooting
tools installed such as tcpdump, nmap, hping, traceroute, iperf, tcpedit, netcat, etc. This machine can then be powered
down and converted to a template. Each time the tools are needed, the troubleshooting client (virtual machine) can be
quickly deployed to the virtual switch(es) in question and used to isolate networking problems. When the testing is
complete, the instance can simply be discarded and the template used again the next time it is required.

Summary
Many of the troubleshooting steps for the VM-Series firewalls are like any other PAN-OS troubleshooting. Be sure to check
interface counters, check system log files and if necessary, use debug to create captures. For more in depth PAN-OS
troubleshooting, please refer to the Packet Based Troubleshooting Tech Note at
https://live.paloaltonetworks.com/community/documentation/content.

 
©2012, Palo Alto Networks, Inc. [13]