Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
FORTINET
FortiGate III
Student Guide
for FortiGate 5.2.1
DO NOT REPRINT
FORTINET
FortiGate III Student Guide
for FortiGate 5.2.1
Last Updated: 20 July 2015
We would like to acknowledge the following major contributors: Francois Ropert, David Chan, Adrian
Buckley, Ondrej Holecek, Stephane Hamelin, and Mike Lobban
Fortinet , FortiGate , and FortiGuard are registered trademarks of Fortinet, Inc. in the U.S. and other
jurisdictions, and other Fortinet names herein may also be trademarks, registered or otherwise, of
Fortinet. All other product or company names may be trademarks of their respective owners. Copyright
2002 - 2015 Fortinet, Inc. All rights reserved. Contents and terms are subject to change by Fortinet
without prior notice. No part of this publication may be reproduced in any form or by any means or
used to make any derivative such as translation, transformation, or adaptation without permission from
Fortinet, Inc., as stipulated by the United States Copyright Act of 1976.
DO NOT REPRINT
FORTINET
Table of Contents
VIRTUAL LAB BASICS ...................................................................................7
Topology..................................................................................................................................8
Logging In ...............................................................................................................................8
Disconnections/Timeouts .............................................................................................................................13
NETWORK ....................................................................................................21
Objectives ...............................................................................................................................21
Time to Complete ....................................................................................................................21
Exploring the Session Table ...................................................................................................22
Traffic sniffer ...........................................................................................................................25
Break and Fix: Connectivity Issues .........................................................................................28
Tips for Troubleshooting ...............................................................................................................................28
DO NOT REPRINT
FORTINET
FIREWALL POLICIES .....................................................................................30
Objectives ...............................................................................................................................30
Time to Complete ....................................................................................................................30
Traffic Shaping ........................................................................................................................31
Break and Fix: FTP Traffic ......................................................................................................32
Tips for Troubleshooting ...............................................................................................................................33
FSSO ..........................................................................................................37
Objectives ...............................................................................................................................37
Time to Complete ....................................................................................................................37
Installing FSSO .......................................................................................................................38
Break and Fix: FSSO ..............................................................................................................42
Tips for Troubleshooting ...............................................................................................................................42
IPSEC ..........................................................................................................44
Objectives ...............................................................................................................................44
Time to Complete ....................................................................................................................44
Break and Fix: IPsec VPN ......................................................................................................45
Tips for Troubleshooting ...............................................................................................................................45
DO NOT REPRINT
FORTINET
Break and Fix: Protection Profiles Part 1 ................................................................................48
Tips for Troubleshooting ...............................................................................................................................48
OPERATION MODES......................................................................................55
Objectives ...............................................................................................................................55
Time to Complete ....................................................................................................................55
Transparent Mode ...................................................................................................................56
NAT/Route Mode ....................................................................................................................60
Break and Fix: NAT/Route Mode ............................................................................................62
Tips for Troubleshooting ...............................................................................................................................62
OSPF ..........................................................................................................66
Objectives ...............................................................................................................................66
Time to Complete ....................................................................................................................66
Break and Fix: OSPF ..............................................................................................................67
Tips for Troubleshooting ...............................................................................................................................67
DO NOT REPRINT
FORTINET
HIGH AVAILABILITY ......................................................................................69
Objectives ............................................................................................................................ 69
Time to Complete ................................................................................................................. 69
Break and Fix: High Availability ............................................................................................ 70
Tips for Troubleshooting ...............................................................................................................................71
DO NOT REPRINT
FORTINET
DO NOT REPRINT
FORTINET
Topology
FORTIMANAGER
port2
10.200.1.241
port1
10.0.1.241
10.200.1.1/24
port1
10.200.1.254
eth1
LINUX
10.200.3.254
eth3
STUDENT
port3
10.0.1.254/24
FortiGate
port2
10.200.2.1/24
10.200.3.1/24
port4
REMOTE
eth2
10.200.2.254
port6
10.0.2.254/24
FortiGate
eth4
10.200.4.254
port5
10.200.4.1/24
Internet
WIN-STUDENT
10.0.1.10
WIN-REMOTE
10.0.2.10
Logging In
1. Run the System Checker. This will fully verify both:
It can also diagnose problems with your Java Virtual Machine, firewall, or web proxy.
Use the URL for your location.
North America/South America:
https://remotelabs.training.fortinet.com/training/syscheck/?location=NAM-West
Europe/Middle East/Africa:
https://remotelabs.training.fortinet.com/training/syscheck/?location=Europe
Asia/Pacific:
https://remotelabs.training.fortinet.com/training/syscheck/?location=APAC
If a security confirmation dialog appears, click Run.
DO NOT REPRINT
FORTINET
If your computer successfully connects to the virtual lab, the result messages for the browser
and network checks will each display a check mark icon. Continue to the next step.
If a browser test fails, this will affect your ability to access the virtual lab environment. If a
network test fails, this will affect the usability of the virtual lab environment. For solutions,
either click the Support Knowledge Base link or ask your trainer.
2. With the user name and password from your trainer, log into the URL for the virtual lab. Either:
https://remotelabs.training.fortinet.com/
DO NOT REPRINT
FORTINET
https://virtual.mclabs.com/
3. If prompted, select the time zone for your location, and then click Update.
This ensures that your class schedule is accurate.
4. Click Enter Lab.
A list of virtual machines that exist in your virtual lab should appear.
From this page, you can access the console of any of your virtual devices by either:
10
DO NOT REPRINT
FORTINET
11
DO NOT REPRINT
FORTINET
A new window should open within a few seconds. (Depending on your accounts preferences,
the window may be a Java applet. If this fails, you may need change browser settings to allow
Java to run on this web site. You also may need to review and accept an SSL certificate.)
Depending on the virtual machine, the applet provides access to either the GUI or a textbased CLI. Connections to Windows machines will use a Remote Desktop-like GUI. The
applet should automatically log in, then display the Windows desktop. For most lab
exercises, you will connect to this VM.
FortiGate III Student Guide
12
DO NOT REPRINT
FORTINET
Disconnections/Timeouts
If your computers connection with the virtual machine times out or if you are accidentally
disconnected, to regain access, return to the initial window/tab that contains your sessions list of VMs
and open the VM again.
If your session frequently times out or does not connect, ask your instructor.
13
DO NOT REPRINT
FORTINET
When connecting to a VM, your browser should then open a display in a new window or tab.
Screen Resolution
Some Fortinet devices' user interfaces require a minimum screen size.
In the Java client, to configure the screen resolution, click the arrow at the top of the window.
In the HTML 5 client, to configure screen resolution, open the System menu.
International Keyboards
If characters in your language dont display correctly, keyboard mappings may not be correct.
14
DO NOT REPRINT
FORTINET
To solve this in the HTML 5 client, open the Keyboard menu at the top of the window. Choose to either
display an on-screen keyboard, or send text from your computer to the VM's clipboard.
To solve this in the Java client, copy and paste between your computer and the Java applet. This
sends special characters or combinations using the keyboard icon at the top of the applet window.
Troubleshooting Tips
If the HTML 5 client does not work, try the Java client instead. Remembering this preference
requires that your browser allow cookies.
Do not connect to the virtual lab environment through a low-bandwidth or high-latency connection,
including VPN tunnels or wireless such as 3G or Wi-Fi. For best performance, use a stable
broadband connection such as a LAN.
Do not disable or block Java applets. On Mac OS X since early 2014, to improve security, Java
has been disabled by default. In your browser, you must allow Java for this web site. On
Windows, if the Java applet is allowed and successfully downloads, but does not appear to
launch, you can open the Java console while troubleshooting. To do this, open the Control
Panel, click Java, and change the Java console setting to be Show console.
Network firewalls can also block Java executables.
Note: JavaScript is not the same as Java.
15
DO NOT REPRINT
FORTINET
Change the power saving scheme so that your computer is always on, and does not go to
sleep or hibernate
If disconnected unexpectedly from any of the virtual machines (or from the virtual lab portal),
please attempt to reconnect. If unable to reconnect, please notify the instructor.
If during the labs, particularly when reloading configuration files, you see a message similar to the
one shown below, the VM is waiting for a response to the authentication server.
16
DO NOT REPRINT
FORTINET
System Resources
System Resources
During this lab, you will learn to use some system and memory debug commands to describe the
status of the unit. Additional, you will generate and analyze a crashlog entry after intentionally killing
one of the FortiGate processes.
Objectives
Time to Complete
Estimated: 15 minutes
17
DO NOT REPRINT
FORTINET
System Resources
3. Click Browse and select the configuration file for this lab:
Resources\FortiGate III\System\Student\student-system.conf
Click Restore. The Student FortiGate will reboot. Wait a few minutes until it is back up.
4. Using PuTTY, connect SSH to the Student FortiGate CLI (use the account admin with no
password) and execute these commands to check the memory usage:
# get system status
# get system performance status
# diagnose hardware sysinfo memory
# diagnose hardware sysinfo shm
Analyze the outputs from the above commands and answer these questions:
18
DO NOT REPRINT
FORTINET
System Resources
19
DO NOT REPRINT
FORTINET
System Resources
RBX:
RDX:
0000000000000000
R9:
...
120: 2015-03-04 07:47:35 <00065> [0x0043d14f] => /bin/reportd
121: 2015-03-04 07:47:35 <00065> [0x0043abfa] => /bin/reportd
122: 2015-03-04 07:47:35 <00065> [0x2a95c40475] => ../lib/libc.so.6
(__libc_start_main+0x000000f5)
123: 2015-03-04 07:47:35 liboffset 00021475
124: 2015-03-04 07:47:35 <00065> [0x0043aca1] => /bin/reportd
125: 2015-03-04 07:47:35 reportd received a signal - 11
126: 2015-03-04 07:47:36 the killed daemon is /bin/reportd:
status=0x0
Check the first three lines. They contain the FortiOS build number, the name of the process
that failed (or was killed) and the kill signal number.
20
DO NOT REPRINT
FORTINET
Network
Network
The following lab exercises show how to use some debug commands to troubleshoot connectivity
problems. You will analyze the information in the FortiGate session table, run the built-in sniffer and
use the debug flow to understand how the FortiGate is processing each IP packet.
Objectives
Time to Complete
Estimated: 50 minutes
21
DO NOT REPRINT
FORTINET
Network
22
DO NOT REPRINT
FORTINET
Network
You will also notice that the expiration timer (expire) and timeout are unusually high. The
default timeout for ICMP sessions is 60 seconds. For the purpose of giving more time to
analyze the session information, the ICMP session timeout was increased to 64.000 seconds.
You can see this configuration by typing the CLI command:
# show system session-ttl
7. Stop the ping (if it is still running) and access the Student FortiGate GUI. Go to Policy &
Objects > Policy > IPv4 and edit the firewall policy with the sequence number 1 (the first one
from top to bottom). Click Enable this policy and then OK.
After a firewall policy configuration change, the FortiGate adds the dirty flag to all the session
with the may_dirty flag. Next time there is traffic matching any of those sessions, the FortiGate
will re-evaluate the action to take.
8. Execute this command in the Student FortiGate CLI and observe that the session has the dirty
flag now:
# diagnose sys session list
9. Run the ping one more time from the Win-Student server to 10.200.1.254:
> ping 10.200.1.254
It should fail as the firewall policy enabled earlier is blocking ICMP traffic.
Check quickly the session information one more time:
# diagnose sys session list
If you do it fast enough, you will notice that the session is still there but the block flag was
added. All traffic matching a session with that flag is denied. Also, the session expiration
23
DO NOT REPRINT
FORTINET
Network
time is much smaller now. The session will remain in the FortiGate memory until this timer
expires (30 seconds).
10. Before proceeding to the next lab exercise, go to Policy & Objects > Policy > IPv4 and disable
the firewall policy with the sequence number 1 (the one blocking the ICMP traffic.)
24
DO NOT REPRINT
FORTINET
Network
Traffic sniffer
During this exercise, you will use the FortiGate built-in sniffer to capture traffic. After that, you will use
a Perl script to convert the capture to a PCAP file that can be analyzed by a packet analyzer, such as
Wireshark.
1. Open a SSH connection to the Student FortiGate using PuTTY.
2. Click on the upper left icon and select Change Settings:
Go to Session -> Logging and select All session output. Then click Browse and select the
folder c:\Perl64\bin. Click Save, and then Apply. With this change, PuTTY will save all the
sniffer output into a text file name putty.log:
25
DO NOT REPRINT
FORTINET
Network
3. Type the following command in the Student FortiGate CLI to start the sniffer:
# diagnose sniffer packet port1 "host 10.200.1.254 and port 80" 3
4. Open a browser and access this URL:
http://10.200.1.254
You should observe the packets captured in the PuTTY window.
5. Close PuTTY and open a command prompt window. Execute these commands:
> cd \Perl64\bin
> perl fgt2eth.pl in putty.log
The Perl script fgt2eth.pl converts the output captured to a PCAP file with the name
putty.log.pcap.
6. Use Windows File Explorer and double click the created file:
c:\Perl64\bin\putty.log.pcap
This starts Wireshark and opens the file for analysis.
7. Observe the information in the packets captured. Right click any packet and select Follow
TCP Stream:
26
DO NOT REPRINT
FORTINET
Network
Observe the new Window that pops up. It shows the application-layer data between the client
and the server for that specific TCP session:
Note: Follow TCP Stream is a useful tool to troubleshoot problems at the application
layer.
27
DO NOT REPRINT
FORTINET
Network
10.200.1.254/24
STUDENT
FortiGate
10.200.1.1/24
port1
Web server
10.200.3.254
port3
10.0.1.254/24
port2
10.200.2.1/24
10.200.2.10/24
WIN-STUDENT
10.0.1.10
REMOTE HOST
10.200.4.1/24
Can you ping the destination IP address from the Win-Student server?
Use the sniffer tool to verify that the traffic is actually arriving to the FortiGate's port3. Use
verbosity 4 or 6 and a filter that can capture the traffic both ways
If the traffic is not intended to terminate in the FortiGate, use the sniffer to check that it is
actually been forwarded to the next hop IP address (use the network diagram provided.) Again,
use a filter in the sniffer that can capture the traffic both ways
28
DO NOT REPRINT
FORTINET
Network
Check the session table. Is the FortiGate creating the session? Check the session protocol state.
Do you see anything wrong there?
Clear the related session (if any) from the session table, enable the debug flow and generate more
test traffic. Do you see any debug flow error?
Try to ping the next hop IP address from the Student FortiGate. Sniffer the traffic to the next hop
IP address while doing it. You can have two simultaneous SSH connections, one running the
sniffer and another one running the ping
29
DO NOT REPRINT
FORTINET
Firewall Policies
Firewall Policies
During these lab exercises you will configure and monitor traffic shaping. Additionally, you will
troubleshoot and fix a connection problem to a FTP server.
Objectives
Time to Complete
Estimated: 40 minutes
30
DO NOT REPRINT
FORTINET
Firewall Policies
Traffic Shaping
1. Log on to the Student FortiGates GUI using the account admin with no password:
http://10.0.1.254
2. Upload the Student configuration file for this lab:
Resources\FortiGate III\Firewall-Policies\Student\student-policy.conf
The Student FortiGate will reboot. Wait a few minutes until it is back up.
3. Go to Policy & Objects > Objects > Traffic Shapers and click Create New. Configure the
following settings:
Type
Shared
Name
SharedPolicy
Apply shaper
Traffic Priority
High
Max Bandwidth
10 Kb/s
Click OK.
4. Go to Policy & Objects > Policy > IPv4 and edit the first policy on the top. Enable Shared
Shaper and select SharedPolicy. Click OK.
5. From a browser in Win-Student go to http://www.youtube.com and play some videos.
6. While plying the videos, execute these CLI commands:
# diagnose firewall shaper traffic-shaper stats
# diagnose firewall shaper traffic-shaper list
Locate the counters for packet drops. Execute the above commands a few times more and
notice how those counters increase with the traffic.
7. Before proceeding to the next lab exercise, go to Policy & Objects > Policy > IPv4 and edit the
first policy on the top one more time. Disable Shared Shaper and click OK.
31
DO NOT REPRINT
FORTINET
Firewall Policies
However, you cannot connect to the FTP server from Win-Student. FileZilla shows this error after each
connection attempt:
The problem only happens with the workstations connected behind the Student FortiGate.
Workstations in other subnets can connect successfully.
FortiGate III Student Guide
32
DO NOT REPRINT
FORTINET
Firewall Policies
Understand first which TCP ports are used for this connection. The control channel is using port
TCP 222. The data channel is using the standard port TCP 20
Understand also how the traffic flows. Is this a FTP server working in active or passive mode? In
active mode the data channel is initiated by the server. In passive mode the data channel is
initiated by the client. Sniffer the traffic in the FortiGate and Linux server to determine who is
initiating the data channel. To run the sniffer in the Linux server follow these steps:
1. Connect SSH to the Linux server (10.200.1.254). Use the username root with the password
password
2. Execute the following command to sniffer the data channel:
# tcpdump -v -i any -nn port 20
You can also sniffer the control channel traffic with this other command:
# tcpdump -v -i any -nn port 222
Use the FortiGate's built-in sniffer to capture the control channel traffic (port 222) before and after
the FortiGate. Use a verbosity level of either 3 or 6 and save the output to a file. After that, use the
Perl script to convert it to Wireshark (as explained in an earlier lab exercise) and analyze it
Run the debug flow over the FTP control channel and analyze the output. Is there anything
missing there?
33
DO NOT REPRINT
FORTINET
Firewall Authentication
Firewall Authentication
During this lab you will learn to use the authentication and LDAP debug commands to troubleshoot an
authentication issue.
Objectives
Time to Complete
Estimated: 40 minutes
34
DO NOT REPRINT
FORTINET
Firewall Authentication
Use the authentication and LDAP debug commands learned to isolate and fix the problem.
Can you explain why the FortiGate is not challenging users to authenticate?
Can you change the FortiGate configuration to fix the problem?
Can you change the FortiGate configuration to properly restrict the Internet access to the user
student, while leaving unrestricted access to the user administrator?
First, test the LDAP authentication from the CLI after enabling the real time debug command:
diagnose debug application fnbamd -1
diagnose debug enable
35
DO NOT REPRINT
FORTINET
Firewall Authentication
Check the Distinguished Name (DN) for student and administrator, by running these commands in
Win-Student:
dsquery user -name student
dsquery user -name administrator
Once the LDAP CLI test works, check the firewall authentication by browsing the Internet from
Win-Student. Look at the session table or run the debug flow to know which firewall policy is
matching the traffic
The output of the LDAP test command shows the user groups for each user. Compare them with
the groups configured in each firewall policy
After any configuration change, de-authenticate the users from the FortiGate and clear the
browser cache (or refresh the page with the F5 key). It is also recommended to clear the related
entries in the session table:
# diagnose sys session filter dport 80
# diagnose sys session clear
To de-authenticate a user, go to User & Device -> Monitor -> Firewall, select the user and click on
De-authenticate
36
DO NOT REPRINT
FORTINET
FSSO
FSSO
During this lab you will install the FSSO collector agent and troubleshoot a FSSO problem.
Objectives
Time to Complete
Estimated: 40 minutes
37
DO NOT REPRINT
FORTINET
FSSO
Installing FSSO
1. Log on to the Student FortiGates GUI using the account admin with no password:
http://10.0.1.254
2. Upload the Student configuration file for this lab:
Resources\FortiGate III\FSSO\Student\student-FSSO.conf
The Student FortiGate will reboot. Wait a few minutes until it is back up.
3. On Win-Student, right-click the Fortinet Single Sign On (FSSO) installation file located in
Resources\FSSO, then select Run as administrator.
This should launch the Fortinet Single Sign On Agent Installation Wizard. Follow the wizard to
install the agent on Win-Student.
4. When prompted for the Windows server administrator password, enter "password":
Click Next.
5. In the Install Options window, accept the default settings:
Click Next.
6. Click Install to complete the installation.
FortiGate III Student Guide
38
DO NOT REPRINT
FORTINET
FSSO
7. At the end of the Single Sign On Agent installation, the Launch DC Agent Install Wizard option
will be selected.
Click Finish to complete the collector agent Installation. This launches the Domain Controller
Agent Installation Wizard.
8. In the Install DC Agent Wizard, accept the Collector Agent IP Address of 10.0.1.10 and the
Collector Agent Listening Port of 8002.
Click Next.
9. Select the TRAININGAD:trainingAD.training.lab domain to monitor.
Click Next.
10. Only the student account needs to be monitored in this exercise. Expand the TRAININGAD
domain and disable all the users in the TRAININGAD domain EXCEPT for student:
Click Next.
11. Set the Working Mode to Polling Mode and select Check Windows Security Event Logs.
39
DO NOT REPRINT
FORTINET
FSSO
Click Next.
12. After the installation, open the Windows start screen and run the application Configure Fortinet
Single Sign-on.
Perform the following tasks in the Fortinet single sign-on agent configuration window:
Click Apply.
13. Click Show Monitored DCs to verify the communication between the collector agent and
the domain controller agent. The IP address of 10.0.1.10 should show as being logged in.
Click Close.
14. Click Select Domains to Monitor and verify that the TRAININGAD:trainingAD.training.lab
domain is selected. Click OK.
40
DO NOT REPRINT
FORTINET
FSSO
15. Click Set Group Filters. Click Add and enable the Default filter. Click Advanced and expand
the domain name of TRAININGAD. From the expanded list select Users and Domain Admins.
Click Add, then OK.
Click OK.
Click Save & Close to close the Fortinet single sign-on agent configuration window.
41
DO NOT REPRINT
FORTINET
FSSO
Student
Password:
Fort1net
Ignore the error message indicating that the user is not authorized for remote login. The
objective of these steps is just to generate a logon event without rebooting the server.
3. After that, test the Internet access from a browser.
Can you explain why the FortiGate is blocking the traffic?
Can you change the FortiGate or/and collector agent configurations to fix the problem?
Check the active FSSO users in the collector agent by clicking Show logon users
Use the following command to check the active FSSO users in the FortiGate:
# diagnose debug authd fsso list
42
DO NOT REPRINT
FORTINET
FSSO
Use the Windows Remote Desktop Connections application after each configuration change to
generate new login events
43
DO NOT REPRINT
FORTINET
IPsec
IPsec
During this lab you will troubleshoot an IPsec VPN problem.
Objectives
Use the IKE real time debug to isolate problems during the phase 1 and phase 2 negotiations
Use the debug flow tool to isolate IPsec traffic flow issues
Time to Complete
Estimated: 90 minutes
44
DO NOT REPRINT
FORTINET
IPsec
Keeping the Remote Gateway type in the VPN RemoteSite as Dialup User (for the reason
explained before)
No configuration changes in the DialUPUsers VPN in the Student FortiGate, as this VPN is
already operative and working as expected (you do not need to test this VPN, assume that it is
working)
Check first why the tunnel is not coming up, use the IKE real time debug in both sides to
troubleshoot the problem:
# diagnose debug application ike -1
# diagnose debug enable
After the tunnel is established, check that you can ping from Win-Student to Win-Remote. If
45
DO NOT REPRINT
FORTINET
IPsec
there is a problem, sniffer the traffic and use the debug flow
46
DO NOT REPRINT
FORTINET
Security Profiles
Security Profiles
During the following exercises you will use debug commands to fix FortiGuard and web filtering issues.
Objectives
Time to Complete
Estimated: 45 minutes
47
DO NOT REPRINT
FORTINET
Security Profiles
48
DO NOT REPRINT
FORTINET
Security Profiles
Proxy Avoidance
Streaming Media and Download
Hacking
All the restricted sites seem to be properly blocked now, such as:
http://www.youtube.com (Streaming Media and Download)
http://www.elite-hackers.com (Hacking)
http://www.proxyavoidance.net (Proxy Avoidance)
However, the administrator complains that the following two sites should be blocked, and they are not.
According to him, they belong to blocked categories:
http://www.metacafe.com
http://www.eicar.org
Additionally, customers are reporting two more problems:
They receive certificate warnings each time they connect to an HTTPS site
Even though antivirus is enabled, they can still download the virus sample eicar.com
located at the ftp server 10.200.3.254:222. To test it, open FileZilla and connect to the
preconfigured site FTPSite. Select the Desktop as the local site folder and pub as the
remote site folder. Right click the eicar.com file and select Download:
49
DO NOT REPRINT
FORTINET
Security Profiles
Why are those two sites reported by the administrator not being blocked? How can you change the
FortiGate configuration to fix it?
Why are users getting SSL certificate warnings? How can you resolve it?
Why isn't FortiGate detecting the EICAR virus?
Enable the following real time debug and attempt to browse the two websites not being blocked:
# diagnose debug application urlfilter -1
# diagnose debug enable
The output can be verbose, so save it from PuTTY to a local file.
Remember to clear the browse cache and FortiGate session after doing any configuration change
Sniffer the FTP traffic and analyze the output of the debug flow
Check the entry in the FortiGate session table for the FTP session
50
DO NOT REPRINT
FORTINET
Objectives
Time to Complete
Estimated: 40 minutes
51
DO NOT REPRINT
FORTINET
52
DO NOT REPRINT
FORTINET
5. Select Automatic proxy configuration URL and type the following URL:
http://10.0.1.254:8080/proxy.pac
6. Restart the browser.
Test the proxy by accessing any web site. Additionally, access to the Fortinet web site is essential for
users. So, test it using the following URL:
http://www.fortinet.com
Why isn't the web proxy working at all?
Can you change the FortiGate configuration to fix the problem?
After fixing the web proxy, test the access to the Fortinet web site. Why isn't working yet? Can you
also fix it?
Use the following debug commands to check the status of the web proxy connections:
# diagnose wad session list
# diagnose test application wad 2200
# diagnose test application wad 110
# diagnose test application wad 104
Run the web proxy real time debug using the filter below:
# config web-proxy debug-url
edit fortinet
set url-pattern www.fortinet.com
set status enable
set exact enable
next
edit fortiguard
53
DO NOT REPRINT
FORTINET
Remember to restart the browser after any change to the PAC file
54
DO NOT REPRINT
FORTINET
Operation Modes
Operation Modes
This lab has 3 exercises. The first exercise includes a FortiGate in transparent mode. During exercises
2 and 3 you will troubleshoot routing problems with two FortiGate devices in NAT/route mode.
Objectives
Identify the existing sessions that will be routed through a different path after a change in the
routing table
Segment a layer-2 network into different broadcast domains using a FortiGate in transparent
mode
Time to Complete
Estimated: 45 minutes
55
DO NOT REPRINT
FORTINET
Operation Modes
Transparent Mode
Port1 and port3 of a FortiGate in transparent mode are connected to a network. An administrator
wants to create 4 broadcast domains. For that purpose, the administrator segmented the network into
4 VLANs:
VLAN Name
VLAN tag
FortiGate interfaces
Native VLAN
No tag
port1
port3
VLAN 20
20
port1-VLAN20
port3-VLAN20
VLAN 30
30
port1-VLAN30
port3-VLAN30
VLAN 40
40
port1-VLAN40
port3-VLAN40
1. First, check that Firefox is not configured to use an explicit web proxy.
2. Click Open menu. Then click Options:
56
DO NOT REPRINT
FORTINET
Operation Modes
57
DO NOT REPRINT
FORTINET
Operation Modes
So, broadcast traffic is being forwarded to all the VLAN sub-interfaces. Each VLAN is not a
different broadcast domain, as the administrator wants.
Why is this happening?
What configuration change must be done in the FortiGate to actually make each VLAN a
different broadcast domain?
8. From the FortiGate CLI, execute these configuration changes:
# config system interface
edit port1-VLAN20
set forward-domain 20
next
edit port1-VLAN30
set forward-domain 30
next
edit port1-VLAN40
FortiGate III Student Guide
58
DO NOT REPRINT
FORTINET
Operation Modes
set forward-domain 40
next
edit port3-VLAN20
set forward-domain 20
next
edit port3-VLAN30
set forward-domain 30
next
edit port3-VLAN40
set forward-domain 40
next
end
9. Execute the sniffer and ping one more time. Now you will see that the ARP packets are
confined only to the native VLAN.
59
DO NOT REPRINT
FORTINET
Operation Modes
NAT/Route Mode
1. Log on to the Student FortiGates GUI using the account admin with no password:
http://10.0.1.254
2. Upload the Student configuration file for this lab:
Resources\FortiGate III\Operation-Modes\Student\student-operationmodes-NAT.conf
The Student FortiGate will reboot. Wait a few minutes until it is back up.
3. Log on to the Remote FortiGates GUI using the account admin with no password:
http://10.200.3.1
4. Upload the Remote configuration file for this lab:
Resources\FortiGate III\Operation-Modes\Remote\remote-operationmodes-NAT.conf
The Remote FortiGate will reboot. Wait a few minutes until it is back up.
5. Check the IPsec VPN configuration in both FortiGate units. Go to VPN -> IPsec -> Tunnels.
Check also the firewall policy and the routing table in both devices. Go to Policy & Objects ->
Policy -> IPv4, then check Router -> Monitor -> Routing Monitor.
You will notice that there is an IPsec VPN created between both units to encrypt the traffic
between the subnets 10.0.1.0/24 and 10.0.2.0/24. You will also see a route in the Student
FortiGate to the subnet 10.0.2.0/24 using the IPsec tunnel.
6. Execute a continuous ping from the Win-Student command prompt to Win-Remote:
> ping -t 10.0.2.10
You will receive the echo reply from Win-Remote as an indication that the tunnel is operating
normally.
7. Without stopping the ping, access the Remote FortiGate and go to System -> Network ->
Interfaces. Click the plus icon besides port4 to expand it, and edit the interface ToStudent:
60
DO NOT REPRINT
FORTINET
Operation Modes
61
DO NOT REPRINT
FORTINET
Operation Modes
Check the status of the link health monitors (if any) under System -> Monitor -> Link Monitor
62
DO NOT REPRINT
FORTINET
External BGP
External BGP
During this lab you will troubleshoot some BGP issues between two FortiGate devices.
Objectives
Time to Complete
Estimated: 30 minutes
63
DO NOT REPRINT
FORTINET
External BGP
64
DO NOT REPRINT
FORTINET
External BGP
65
DO NOT REPRINT
FORTINET
OSPF
OSPF
During this lab you will troubleshoot some OSPF over IPsec issues between two FortiGate devices.
Objectives
Time to Complete
Estimated: 40 minutes
66
DO NOT REPRINT
FORTINET
OSPF
67
DO NOT REPRINT
FORTINET
OSPF
Once the OSPF issues are resolved, go to the VPN event logs. Is the IPsec VPN stable? Watch
the log messages for a few minutes
Compare the Student routing table when the tunnel is down with the table when it is up.
What is causing the tunnel to bounce?
68
DO NOT REPRINT
FORTINET
High Availability
High Availability
During this lab you will troubleshoot some high availability problems between two FortiGate devices.
Objectives
Monitor a HA cluster
Time to Complete
Estimated: 30 minutes
69
DO NOT REPRINT
FORTINET
High Availability
FortiGate
REMOTE
port3
port1
LINUX
10.200.1.254
eth1
port2
port2
port3
10.0.1.254/24
WIN-STUDENT
10.0.1.10
port1
10.200.1.1/24
STUDENT
FortiGate
LAN3
0.0.0.0
eth0
LAN0
0.0.0.0
1. Log on to the Remote FortiGates GUI using the account admin with no password:
http://10.200.3.1
2. Upload the Remote configuration file for this lab:
Resources\FortiGate III\High-Availability\Remote\remote-ha.conf
The Remote FortiGate will reboot. Wait a few minutes until it is back up.
3. Log on to the Student FortiGates GUI using the account admin with no password:
http://10.0.1.254
4. Upload the Student configuration file for this lab:
Resources\FortiGate III\High-Availability\Student\student-ha.conf
The Student FortiGate will reboot. Wait a few minutes until it is back up.
After loading both configurations, the cluster is not forming. The Remote unit cannot join the HA
cluster.
Use the debug commands learned in this lesson to troubleshoot the problem.
70
DO NOT REPRINT
FORTINET
High Availability
For easy access to each unit while the cluster is down, each FortiGate starts with different IP
addresses in their port3:
Student: 10.0.1.254
Remote: 10.0.1.253
So, while Remote cannot join the cluster, you can connect to its port3 IP address via SSH and run
the debug commands
Stop and Think
After the Remote FortiGate joins the cluster, you will notice that you cannot access the
Remote FortiGate using the IP address 10.0.1.253 anymore. Can you explain why?
71
DO NOT REPRINT
FORTINET
http://training.fortinet.com
Technical Documentation
http://help.fortinet.com
Knowledge Base
http://kb.fortinet.com
Forums
https://forum.fortinet.com/
https://support.fortinet.com
http://www.fortiguard.com
72
DO NOT REPRINT
FORTINET
73
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
74
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
In this lesson, we will review troubleshooting strategies. We will also introduce some of the
troubleshooting tools available in the FortiGate GUI and CLI.
75
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
76
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
Good administrators know their network well before any problem happens. That includes an
understanding of the normal behavior related with traffic volume, network applications, traffic flows
and devices' CPU and memory utilization. So, when a problem happens, good administrators identify
quickly what is behaving abnormally. This information speeds up the troubleshooting process and
helps to isolate the cause of the problem.
Many tools can be used to gather statistics and information while the network is operating normally:
SNMP, logging, sFlow, and the monitors located in the FortiGate GUI.
77
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
It is also important to keep the network documentation up-to-date. Network diagrams should include
physical connections, interface names and subnets. Good network documentation also includes
change control records to track any change in the network: Who did the change? When was done?
What was changed?
78
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
If a problem happens, the first step is to define it well. For example, if the problem definition is web
filtering is not working, the scope of the problem is too imprecise. Too many things could cause this.
This makes troubleshooting slow. So, we must ask questions to understand the details: Is the
problem happening with one web site? Is it happening with all users? Is it happening randomly? How
can you reproduce the problem?
After answering the right questions, we can define the problem with details. For example: the web
filtering is not blocking the web site X for the user Y. This provides a better place to start the
troubleshooting.
79
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
A general approach for troubleshooting network issues is to follow the TCP/IP model and work the
problem either from the highest layer to the bottom or from the lowest layer to the top.
In the first method you check the physical layer first. If a layer operation is ok, you move to the upper
layer, until you find the layer where the problem is happening.
In the second method you check the application layer first, if a layer is not working properly you move
to the layer below to rule out issues in the lower layers.
80
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
During the second part of this lesson, we will review some of the troubleshooting tools available in
the FortiGate GUI.
81
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
The dashboard is the FortiGate GUI welcome screen. Some of its widgets contain information useful
for troubleshooting, such as the system resources and the alert message console widgets.
82
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
Remember that the dashboard is customizable. Widgets can be added, removed and customized.
83
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
In addition to the dashboards widgets, the GUI includes some monitor screens for specific features.
For example, the IPsec monitor displays the status of each IPsec tunnel. The firewall monitor shows
the list of authenticated users. Another example is the routing monitor, which lists the routes that
have been loaded (active) the routing table.
84
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
Another important tool for troubleshooting is the FortiGate logs. The log viewer includes a filter
setting that is used to display only the logs entries related with one specific user name, IP address,
URL or event type.
85
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
This table illustrates the expected log behavior depending on the different logging settings.
The first column shows the possible values for the log setting in the firewall policies: no log, log
security events, or log all sessions.
The second column indicates if the AV, web filtering or antispam profile log setting is enabled or
disabled. Remember, DLP and IPS profiles always generate logs in the security log section.
The last column shows the behavior. If you enable any profiles on your policy and logging is not
enabled, you will not get logs of any kindeven when profile is blocking traffic. So if you apply a
security profile, its important to consider the logging settings.
86
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
From the information stored in the logs, a FortiGate device with hard disk can generate reports, either
manually or on an automatic schedule.
87
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
88
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
89
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
The real time debug commands generate information in real time about what a specific FortiGate
process or feature is doing.
The debug level is a bitmask value that specifies which types of messages are displayed. This
depends on each process. For all the cases, although, a debug level of 0 means no output
(disabled), and a debug level of '-1' means enabling all possible message types.
90
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
For example, this slide shows the two commands for enabling all the IPsec real time debug output.
You can also enable the option to prepend the system time to each debug line.
It is important to disable any real time debug after using it. They consume FortiGate resources and
some could be CPU intensive.
91
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
On the other hand, the application layer test commands do not display information in real time, but
statistics and configuration information about a feature or process. Some of these commands can
also be used to restart a process or execute a change in its operation.
92
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
Some CLI debug commands generate a lot of output. If we know that the required information is
contained in one specific line of the output, and if we know a keyword to find that line, we can use the
GREP utility. This utility displays only the lines from the output that match a text string. For example,
in this slide we are using the GREP utility to find the IP address 10.0.0.7 in the ARP table. We use
the debug command diagnose ip arp list to get the ARP table, and then we append the GREP utility
to display only the information for one IP address. In this way, the output is only one line (with exactly
the information we need), instead of a long list of entries.
93
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
When displaying the FortiGate configuration via CLI, you can use the GREP utility with the option f.
It will display only the configuration sections (or tables) where the text string matches at least one
value. This is useful, for example, to find all the references in the configuration to a specific object. In
this slide, we are using the f option to find all the references to the wan1 interface. The output shows
only the two tables where wan1 is referenced: the definition of the interface itself, and a firewall policy
where wan1 is the destination interface.
94
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
If you need to expand your FortiGate skills, or if you need more information about troubleshooting a
FortiGate issue, these are some of the resources to get more information and help.
95
DO NOT REPRINT
FORTINET
Troubleshooting Concepts
96
DO NOT REPRINT
FORTINET
System Resources
In this lesson, we will show you how to troubleshoot system resources on FortiGate.
97
DO NOT REPRINT
FORTINET
System Resources
You will learn some debug commands for troubleshooting FortiGate system problems, such as high CPU
or memory usage. The lesson also covers new firmware installations from the BIOS, memory optimization
and crashlog diagnostics.
98
DO NOT REPRINT
FORTINET
System Resources
99
DO NOT REPRINT
FORTINET
System Resources
This is usually one of the first debug commands when troubleshooting. The output shows the firmware
version, FortiGuard database versions, license status, operation mode, number of VDOMs and system
time.
100
DO NOT REPRINT
FORTINET
System Resources
The command 'get system performance status' shows the overall memory and CPU utilization. It also
shows session creation rate, number of viruses caught and number of attacks blocked by the IPS. The last
line displays the system uptime. This output gives a quick overlook at how much traffic the unit is handling.
101
DO NOT REPRINT
FORTINET
System Resources
During this part of the lesson you will learn how to check the use of he FortiGate memory and CPU.
102
DO NOT REPRINT
FORTINET
System Resources
To understand how a FortiGate uses its memory, we need to understand the architecture of FortiOS.
The hearth of FortiOS is its kernel. Here, the unit takes some of the most important and basic decisions,
such as how to route a packet, or when to offload a session to a NPU processor.
FortiOS runs over hardware. The device drivers bridge the kernel with the hardware.
Above the kernel, there is the user space. Several application processes or daemons run at this level.
Above all that, there is the configuration layer, which is basically composed of two modules: the command
line interface and the graphical user interface.
103
DO NOT REPRINT
FORTINET
System Resources
How the memory is segmented depends on the FortiGate model. There are two cases: first, models
running 32-bit FortiOS with more than 1 GB of memory; second, models running 64-bit FortiOS and
models running 32-bit FortiOS with less than 1 GB of memory.
Lets see the first case.
When the FortiOS is 32 bits and the system memory is more than 1 GB, the kernel cannot access directly
the whole memory space. So, memory paging is used to reach the portion of the memory that cannot be
accessed directly. The part of the memory that the kernel can access directly is called low memory. The
part of the memory that is accessed using memory paging is called high memory. The command
'diagnose hardware sysinfo memory' displays:
- The total amount of low memory (LowTotal)
- The amount of free low memory (LowFree)
- The total amount of high memory (HighTotal)
- The amount of free high memory (HighFree)
- The total amount of system memory (MemTotal = LowTotal + HighTotal)
- The total amount of free memory (MemFree = LowFree + HighFree)
104
DO NOT REPRINT
FORTINET
System Resources
For models running 32-bit FortiOS with less than 1 GB, and for models running 64-bit FortiOS, the kernel
doesn't need to use memory paging to access the whole memory space. In those cases the command
'diagnose hardware sysinfo memory' shows a size of 0 kB for the total high memory and free high
memory. All the memory space is considered low memory.
105
DO NOT REPRINT
FORTINET
System Resources
106
DO NOT REPRINT
FORTINET
System Resources
The kernel memory slabs are collections of objects with a common purpose. They are used by the kernel
to store information in the low memory.
This slide shows example of some slabs. There are slabs for storing information about the TCP sessions.
The entries in the route cache are also stored in low-memory slabs.
107
DO NOT REPRINT
FORTINET
System Resources
108
DO NOT REPRINT
FORTINET
System Resources
The system I/O cache is used to speed up the access to information stored in the hard and flash disk
memories. Some processes, such as logging, WAN optimization, and explicit proxy, store information in
the hard disk, so they get the performance boost provided by this memory allocation.
An I/O cache page is labeled as active when it has been recently accessed. It goes to inactive state after
not been used for some time. An inactive page can be reclaimed by the kernel if needed.
109
DO NOT REPRINT
FORTINET
System Resources
110
DO NOT REPRINT
FORTINET
System Resources
As we explained, above the kernel layer there are multiple application processes or daemons running. The
operating system allocates separated blocks of memory for each process. One process can access the
memory that was allocated to it, but it cannot access any memory that was allocated to a different process.
So, a process cannot share information with another process by reading or writing data into the memory
allocated to that other process. For that purpose, the operating system dynamically allocates shared
memory (SHM). The SHM can be accessed by multiple processes, allowing the sharing the information
among them.
111
DO NOT REPRINT
FORTINET
System Resources
We can check how much memory space is being used by each process. The command 'diagnose sys
top' shows that information in the last column. The command also displays, for each process: its ID
number, its state, and the amount of CPU using. You can specify the refresh frequency and the number of
lines to display.
While the command is running, you can press <shift-P> to sort the processes by CPU usage, of <shift-M>
to sort them by memory usage. To stop the command, use <ctrl-C>.
112
DO NOT REPRINT
FORTINET
System Resources
Another useful command for displaying information about process memory usage is 'diagnose sys topsummary'. The h option displays the different options available for this command.
113
DO NOT REPRINT
FORTINET
System Resources
114
DO NOT REPRINT
FORTINET
System Resources
115
DO NOT REPRINT
FORTINET
System Resources
This other table shows more about the most common processes.
116
DO NOT REPRINT
FORTINET
System Resources
The command 'diagnose sys top' shows the state of each process. A process can be in one of four
states: Sleeping (S), running (R), do not disturb (D) or zombie (Z).
The S and R states are normal. It is also normal if a process goes briefly to D state.
The Z state is not normal. Also, it is not normal if a process stays in D state for a long time. These usually
indicate that the process is not working properly.
117
DO NOT REPRINT
FORTINET
System Resources
During this part of the lesson you will learn about conserve mode.
118
DO NOT REPRINT
FORTINET
System Resources
119
DO NOT REPRINT
FORTINET
System Resources
Two margins or thresholds determine when the FortiGate enters and exists the kernel conserve mode. The
margins depend on the total amount of low memory.
When a FortiGate is in kernel conserve mode, any proxy inspection is bypassed and administrators cannot
change the unit configuration.
120
DO NOT REPRINT
FORTINET
System Resources
These are the entries generated in the crashlog, event logs and alert message console when a FortiGate
enters to and leaves the kernel conserve mode.
121
DO NOT REPRINT
FORTINET
System Resources
Similarly to kernel conserve mode, two margins or thresholds determine when the FortiGate enters and
exists the proxy conserve mode. The margins depend on the total amount of overall memory.
122
DO NOT REPRINT
FORTINET
System Resources
These are the entries generated in the crashlog and event logs when a FortiGate enters to and leaves the
proxy conserve mode.
123
DO NOT REPRINT
FORTINET
System Resources
av-fail-open is the CLI setting that controls FortiGates behavior while it is in proxy conserve mode.
124
DO NOT REPRINT
FORTINET
System Resources
An option related to fail-open, is av-failopen-session. This is a setting that kicks in not during a high
memory situation, but when a proxy on the FortiGate runs out of available sessions to process the traffic.
If av-failopen-session is enabled, then FortiGate will act according to the av-failopen setting. Otherwise,
by default, it will block new sessions until proxy connections become available.
125
DO NOT REPRINT
FORTINET
System Resources
Additionally, FortiGate has one more mechanism to free memory when there is not much available. If the
kernel cannot allocate more memory pages, it deletes the oldest sessions. The command diagnose sys
session stat shows the numbers of sessions that has been deleted by the kernel due to this mechanism.
126
DO NOT REPRINT
FORTINET
System Resources
During this part of the lesson you will learn how to optimize the memory usage.
127
DO NOT REPRINT
FORTINET
System Resources
Many FortiGate processes, such as DLP or AV scanning, are memory intensive. So, memory optimization
is important, especially in small units, to guarantee that they will stay away from conserve mode. This slide
shows some recommendations for optimizing the memory usage. These tips might increase significantly
the available memory in a device that is frequently going to conserve mode.
The first and most logical step is to disable the features that are not required. For example, if the network
already has a FortiMail doing antispam, an administrator does not need to do antispam in the FortiGate.
Also, usually not all the IPS signatures are required.
Another important recommendation is to reduce the maximum file size to inspect, which is set to 10MB by
default. This value can be reduced to 2 or 3 MB without reducing significantly the virus caching rate.
128
DO NOT REPRINT
FORTINET
System Resources
Additionally, you can reduce the amount of allocated memory to some caches, such as the ones for
FortiGuard and DNS.
129
DO NOT REPRINT
FORTINET
System Resources
The FortiGate session table, especially in networks with high traffic, can consume an important portion of
the memory. By default, a session without traffic remains in the table for up to 1 hour. Although this high
time to live (TTL) might be required by a few applications, in most of the networks, it can be substantially
reduced to, for example, 5 minutes. So, FortiGate will age out idle sessions much quicker, increasing the
amount of available memory. There are four places in the FortiGate configuration where the session TTL
can be changed. Two of them are:
- Globally for all the traffic
- Per IP protocol and port number
130
DO NOT REPRINT
FORTINET
System Resources
The other two places where the session TTL can be changed are:
- Per firewall policy
- Per application control
If there is one specific application that requires a high session TTL, you can, for example, reduce the TTL
globally to 5 minutes, and set it to 1 hour only for the specific application port number, or firewall policy.
131
DO NOT REPRINT
FORTINET
System Resources
In the case of TCP, there are other session timers. In most of the cases, they can also be reduced from
their default values without causing problems with the applications. The slide shows some recommended
values (equal to or below the default ones) to optimize the memory usage.
We will see what each of those timers are in the next slide.
132
DO NOT REPRINT
FORTINET
System Resources
The tcp-halfopen-timer controls for how long, after a SYN packet, a session without SYN/ACK remains in
the table.
The tcp-halfclose-timer controls for how long, after a FIN packet, a session without FIN/ACK remains in
the table.
The tcp-timewait-timer controls for how long, after a FIN/ACK packet, a session remains in the table. A
closed session remains in the session table for a few seconds more to allow any out-of-sequence packet.
133
DO NOT REPRINT
FORTINET
System Resources
To finish this lesson, we will talk about hardware and firmware troubleshooting.
134
DO NOT REPRINT
FORTINET
System Resources
Lets talk about the crashlog. Each time an application crashes, or closes, an entry is generated in the
crashlog. For the case of crashes, the entry contains information about the name of the application, the
time it crashed and the termination signal.
This slide shows a sample of a crash in the crashlog. In this case the application that failed was the
sslvpnd process, which manages SSL VPN connections. The termination signal is 11, which is
segmentation fault.
135
DO NOT REPRINT
FORTINET
System Resources
This table contains the most common termination signal numbers. Any administrator can manually kill a
process by using the command 'diagnose sys kill', followed by the process ID and the termination signal
number. The command 'diagnose sys top' lists the process ID numbers. Manually killing a process is not
usually required under normal circumstances. Be careful although if, for some reason, you have to do it.
Improperly killing a process might make a FortiGate system unstable.
136
DO NOT REPRINT
FORTINET
System Resources
137
DO NOT REPRINT
FORTINET
System Resources
138
DO NOT REPRINT
FORTINET
System Resources
We say that a FortiGate freezes when stops handling traffic and there is no way to connect to it (there is
even not access to the unit's console port). Only power cycling fixes the problem. For those cases, you
could similarly capture any crashdump in the console port. Additionally, and in the case of models with
more than one CPU, you can enable the nmi-watchdog feature, which crashes the system (and forces
the crashdump) when no more new daemons have been scheduled in the last 10 minutes - This is an
indication that the unit is not operating normally and might have frozen.
Some FortiGate models also have an external NMI button. If the unit is frozen and no crashdump has been
generated, you can press it to force a crash and get a crashdump.
139
DO NOT REPRINT
FORTINET
System Resources
The FortiGate BIOS menu allows administrators to do some operations over the flash memory and the
firmware images. To access the BIOS menu you must reboot the unit while connected to the console port.
The console port will show the booting process and, at one point, it will show the message:
Press any key to display configuration menu
If you press any key while this prompt is displayed, the booting process is interrupted and the BIOS menu
is displayed.
140
DO NOT REPRINT
FORTINET
System Resources
The options available in the BIOS menu vary depending of the BIOS version and hardware model.
However, in most of the cases you will see an output similar to the one in this slide.
141
DO NOT REPRINT
FORTINET
System Resources
142
DO NOT REPRINT
FORTINET
System Resources
143
DO NOT REPRINT
FORTINET
System Resources
144
DO NOT REPRINT
FORTINET
System Resources
There is a hardware test images (HQIP) for each model. You can install it by using the BIOS menu and a
TFTP server. You can run it without saving it to the flash. The HQIP runs some hardware diagnostics to
test the NIC interfaces, hard disk, flash disk and other hardware components.
145
DO NOT REPRINT
FORTINET
System Resources
146
DO NOT REPRINT
FORTINET
Network
147
DO NOT REPRINT
FORTINET
Network
You will learn to use some general network troubleshooting tools, the built-in sniffer and the debug flow.
The lesson also shows how to display and analyze the information in the FortiGate session table.
148
DO NOT REPRINT
FORTINET
Network
The first part of this lesson introduces some general network troubleshooting tools.
149
DO NOT REPRINT
FORTINET
Network
This is one of the most useful commands for troubleshooting layer-1 problems. It displays the link status,
negotiated speed and negotiated duplex mode. It also shows counters related with bytes, packets and
errors received and transmitted. The output of this command varies depending on the FortiGate model and
NIC driver version.
150
DO NOT REPRINT
FORTINET
Network
This table contains some of the fields in the output of the 'get hardware nic' command. Not all these are
displayed for the same NIC. As we explained, the output depends on the model and NIC driver version.
151
DO NOT REPRINT
FORTINET
Network
This other table contains more fields that might be present in the output of the 'get hardware nic'
command.
152
DO NOT REPRINT
FORTINET
Network
To check the ARP table, use the command 'get system arp'. It displays the IP addresses, MAC addresses
and interfaces where the devices are connected. It also shows the age, which is for how long there has not
been traffic. After a predefined time, an entry in the ARP table without traffic is aged out.
153
DO NOT REPRINT
FORTINET
Network
Two of the most common troubleshooting tools are the 'ping' and the 'traceroute'. Remember from our
NSE4 training that the options for the ping can be changed with the command 'execute ping-options'. They
include the packet size, amount, timeout and the source IP address for the ping.
154
DO NOT REPRINT
FORTINET
Network
The CLI also includes a telnet client, which cannot only be used to connect to a remote telnet device, but
could also be used to test the connectivity to an application server. For example, in this slide we are testing
the connectivity from the FortiGate to a SMTP server by doing telnet to the server's port 25.
155
DO NOT REPRINT
FORTINET
Network
The command 'get system performance firewall statistics' displays aggregated statistics per network
application.
156
DO NOT REPRINT
FORTINET
Network
The command 'get system performance firewall packet-distribution' displays number of bytes and
packets per packet size range.
157
DO NOT REPRINT
FORTINET
Network
In the last part of this lesson, we will talk about the FortiGate session table, the built-in sniffer and the
debug flow.
158
DO NOT REPRINT
FORTINET
Network
The FortiGate session table, as explained in the NSE4 training, contains detailed information about every
IP connection crossing or terminating at the FortiGate. The command 'get system session status' displays
the total number of sessions in the active VDOM. The command 'get system session list' provides a brief
summary of each session, one session per line, with the most relevant information, such as protocol, and
source and destination IP address and port. We can use the GREP utility with this command, for example,
to list only the sessions for a specific IP address.
159
DO NOT REPRINT
FORTINET
Network
To display detailed information about the sessions, we use the command 'diagnose sys session list'. It is
recommended to set the session filter first, as an unfiltered output displays all the details about all the
existing sessions (which could be in order of thousands, or even millions, for high-end devices). You can
filter the output, for example, by policy ID, or by source or destination IP address and port.
160
DO NOT REPRINT
FORTINET
Network
Some configuration changes, such as security profile or session helper changes, apply only to new
sessions. For those cases, existing sessions keep using the old configuration until they expire or are
terminated. This is important when troubleshooting problems. For example, after a change in a security
profile, you should clear any related existing session and generate new ones. Use the command 'diagnose
sys session clear' to remove all sessions that match the session filter. You must although be careful with
this command as it could potentially clear all the existing sessions if no filter has been set. Double check
first the session filter with the command 'diagnose sys session filter'.
161
DO NOT REPRINT
FORTINET
Network
162
DO NOT REPRINT
FORTINET
Network
The protocol state in the session table is a two-digit number. In the case of TCP, the first number is related
with the client-side state, and is different than 0 when the session is being handled by a FortiGate proxy
(for example, when the FortiGate is doing proxy-based inspection). The second digit is the server-side
state and is used to know the status of the TCP session. This table and the flow graph correlate the
second digit value with the different TCP session states. For example when the FortiGate receives the
SYN packet, the second digit is 2. It goes to 3 once the SYN/ACK is received. After the three-way
handshake, the state value changes to 1.
When a session is closed by both sides, the FortiGate keeps it in the session table for a few seconds
more, to allow any out-of-order packet that could arrive after the FIN/ACK packet. This is the state value 5.
163
DO NOT REPRINT
FORTINET
Network
For case of UDP, the session state can only have two values: 00 when traffic has been only one way, and
01 once there is traffic two ways. For the case of ICMP the protocol state is always 00.
164
DO NOT REPRINT
FORTINET
Network
This table contains the meaning of the most important session flags. For example the log flag indicates
that the session is being logged. The local flag indicates that the session either was originated from the
FortiGate or is terminating in the FortiGate.
165
DO NOT REPRINT
FORTINET
Network
Lets give more details about the dirty and may dirty flags. When the FortiGate receives the first packet for
a new session (for example, a SYN packet), the unit evaluates if the traffic should or should not be allowed
based on the firewall policies. As long as there are not changes in the firewall policies, this evaluation is
done only on the first session packet. If the traffic is allowed by a firewall policy, the unit creates a session
and flags it as may_dirty.
After that, if there is a change in the firewall policy configuration, all the existing sessions with the
may_dirty flag are also flagged as dirty. This indicates to the FortiGate that it needs to reevaluate the next
session packet to know if the session must now be blocked. If the session is still allowed, the dirty flag is
removed (the may_dirty flag is kept). If the session must be blocked, it is flagged as block and remains in
the session table until it expires. Any packet matching a session with the block flag is dropped.
166
DO NOT REPRINT
FORTINET
Network
We talk about the built-in sniffer in NSE4. This slide reviews this tool. There are 6 verbosity levels. The
table shows what information is displayed in each level. We usually use level 4 to check how the traffic is
flowing and if the FortiGate is not dropping packets. We also use level 3 or 6 to convert the output to PCAP
format, which can be later analyzed with a tool such as WireShark.
167
DO NOT REPRINT
FORTINET
Network
168
DO NOT REPRINT
FORTINET
Network
Another useful FortiGate troubleshoot tool is the debug flow. We also talk about these commands in NSE4,
so we will quickly review them. The debug flow is also called internal sniffer as it works similarly to the
built-in sniffer. But, in this case, the output shows step-by-step, and with details, what the kernel is doing
with each packet.
169
DO NOT REPRINT
FORTINET
Network
170
DO NOT REPRINT
FORTINET
Network
The debug flow is also useful when the FortiGate, for some reason, is dropping packets. In those cases it
usually shows an error message explaining why a packet was dropped.
These are three common messages that we might see in a debug flow output.
The first one means that either there is not firewall policy allowing the traffic or that a disclaimer has not
been accepted yet.
The second one indicates that the IP address has been quarantined by the DLP inspection.
The third one means that the packet was dropped due to a traffic shaper that has exceeded one of its
thresholds.
171
DO NOT REPRINT
FORTINET
Network
These are two more common debug flow error messages. The first error indicates that the packet failed
the reverse path forwarding check.
The second error message is usually because one of these reasons:
if the packet is destined to a FortiGate IP address (for example, management traffic), it might be that the
service is not enabled, is using a different port, or the source IP address if not included in the trusted list.
On the other hand, if the packet is not destined to a FortiGate IP address, but to a device on the other side
of the unit, there might be a virtual IP or IP pool that is wrongly using that IP address. So, check your
virtual IP or IP pool configuration.
172
DO NOT REPRINT
FORTINET
Network
173
DO NOT REPRINT
FORTINET
Firewall Policies
174
DO NOT REPRINT
FORTINET
Firewall Policies
During this lesson, you will learn about how the FortiGate translates the source port when doing
source NAT. The lesson also covers NAT exhaustion, SIP and session helper debug commands.
During the last part of the lesson, you will learn some debug commands for troubleshooting traffic
shaping.
175
DO NOT REPRINT
FORTINET
Firewall Policies
This part of this lesson is about NAT and NAT exhaustion problems.
176
DO NOT REPRINT
FORTINET
Firewall Policies
This slide shows the different FortiGate NAT methods. The differences between each of them are
explained in NSE4.
FortiGate can translate the source IP address of an outgoing connection (source NAT or SNAT).
Three different features can be used for that purpose:
- Policy NAT: Overload NAT of many to one
- IP pool: Four types of IP pools Overload, one-to-one, fixed port range and port block allocation
- Central NAT table
FortiGate can also translate the destination IP address of an incoming connection (destination NAT or
DNAT). Two different features can be used for that purpose:
- Virtual IP (VIP)
- Load Balancer
177
DO NOT REPRINT
FORTINET
Firewall Policies
For policy NAT and overload IP pool, the number of available NAT connections depends on four
variables:
- IP protocol
- NAT IP address
- Destination IP address
- Destination port
For each NAT IP address, there are up to 60,416 TCP and 60,416 NAT UDP ports available.
For each TCP or UDP connection, two indexes are created to match the traffic and determine the NAT
IP address and port. We will see examples of these indexes in the next slides.
178
DO NOT REPRINT
FORTINET
Firewall Policies
179
DO NOT REPRINT
FORTINET
Firewall Policies
So, although both sessions are using the same NAT IP and NAT port, the FortiOS can still use the
indexes to uniquely identify each session.
FortiOS only has to ensure that the chosen NAT port combined with the other four variables are
unique to identify the session.
180
DO NOT REPRINT
FORTINET
Firewall Policies
181
DO NOT REPRINT
FORTINET
Firewall Policies
We can calculate the theoretical maximum number (without exhausting the number of NAT ports
available) of simultaneous connections for one NAT IP and one destination IP.
If we multiply the number of NAT IP addresses (1 in our case), by the number of NAT ports per NAT
IP (60,416), by the number of protocols (2, TCP and UDP), by the number of destination IP addresses
(1 in our case) and by the maximum number of destination ports (65,535), we get almost 8 billion
possible connections.
182
DO NOT REPRINT
FORTINET
Firewall Policies
Although that high number is theoretically possible, in practice, real numbers are much smaller. One
reason is that servers never use all 65,535 possible ports. Indeed, most of the connections to a server
go to the same port number and IP protocol number. For example, all the connections to a HTTP
server are TCP and usually go to port 80 only.
183
DO NOT REPRINT
FORTINET
Firewall Policies
So, if we calculate again the maximum number of simultaneous connections, this time considering one
IP protocol number, and one destination port number, we get 60,416. This is a more realistic number.
184
DO NOT REPRINT
FORTINET
Firewall Policies
Some network applications do not support source port address translation. For those cases, enable
Fixed Port so the FortiGate translates only the source IP address and not the source port.
185
DO NOT REPRINT
FORTINET
Firewall Policies
If at one point the FortiGate does not have any available NAT port for a new connection, the traffic is
rejected and a log is generated.
186
DO NOT REPRINT
FORTINET
Firewall Policies
This part of this lesson is about session helpers and application layer gateways.
187
DO NOT REPRINT
FORTINET
Firewall Policies
To understand what a session helper does, lets see an example of a network protocol that might have
problems when there is a network device doing NAT. The example that we will see is the FTP
protocol, specifically, when working in active mode.
Any FTP file transfer is composed of two TCP sessions: one for the control channel and one for data
transfer. The control channel is always initiated from the client to the server and is used to send the
different FTP commands. The FTP commands allow the client to move through the different server
folder, specify the type of file transfer and initiate the data channel for uploading or downloading a file.
FTP has two modes: active and passive. The mode determines who initiate the data channel. In the
case of passive mode, the data channel is initiated by the client.
In the case of active mode, the client sends the port command through the control channel. The
command includes the client IP address and the TCP port for the incoming data channel. Then, it is
the server who initiates the TCP session to the IP address and port number specified by the port
command.
188
DO NOT REPRINT
FORTINET
Firewall Policies
189
DO NOT REPRINT
FORTINET
Firewall Policies
190
DO NOT REPRINT
FORTINET
Firewall Policies
This slide shows a packet capture of the previous FTP traffic before the port command reaches the
FortiGate. We see the original client IP address 10.0.1.10.
191
DO NOT REPRINT
FORTINET
Firewall Policies
This other slide shows another packet capture, this time after the port command crosses the
FortiGate. The session helper has translated the IP address inside the port command to 10.200.1.1.
192
DO NOT REPRINT
FORTINET
Firewall Policies
193
DO NOT REPRINT
FORTINET
Firewall Policies
There is actually a way to list the expected sessions created by the session helpers. In this example,
the command lists an expected session to allow traffic from 10.171.121.38 to 10.0.1.10, port TCP
50365.
194
DO NOT REPRINT
FORTINET
Firewall Policies
The debug flow shows the name of the session helper (if any) that is inspecting the traffic. In this case,
it is the FTP session helper.
Also, for traffic that matches an expected session previously created by a session helper, the debug
flow shows the message: 'find an EXP session'.
195
DO NOT REPRINT
FORTINET
Firewall Policies
There are other protocols that, in some circumstances, also require a session helper, for example
PPTP, H323 and RSH. We can list the active session helpers by typing the command 'show' after
'config system session-helper'. The output lists the TCP or UDP port numbers that each session
helper is listening to. If one of those protocols is using a different port number, you need to change the
FortiGate configuration to match it.
196
DO NOT REPRINT
FORTINET
Firewall Policies
For SIP traffic inspection, the FortiGate also includes a feature smarter and more versatile than the
SIP session helper. It is the SIP application layer gateway (SIP ALG).
The SIP ALG has all the same functions than the SIP helper, but provides more features.
Another difference is that the all session helpers run in the kernel. SIP ALG, on the other hands, runs
as a user space process.
197
DO NOT REPRINT
FORTINET
Firewall Policies
A FortiGate uses either the SIP helper or the SIP ALG depending on the configuration. The system
setting 'default-voip-alg-mode' specifies which one is used when no VoIP profile is applied. If it is set
to 'proxy-based' (default), SIP ALG is used. If it is set to 'kernel-helper-based', SIP helper is used.
If the SIP traffic matches a firewall policy with a VoIP profile, SIP ALG is always used regardless of the
'default-voip-alg-mode' setting.
Fortinet recommends the use the SIP ALG. The SIP helper should be used only as an alternative
when, for some reason, the SIP ALG is not working as expected.
198
DO NOT REPRINT
FORTINET
Firewall Policies
These are the commands to change the ports for the SIP ALG. The SIP ALG supports SIP over UDP,
SIP over TCP and encrypted (SSL) SIP.
199
DO NOT REPRINT
FORTINET
Firewall Policies
The command 'diagnose sys sip-proxy calls list' displays all the active SIP calls.
The command 'diagnose sys sip-proxy calls clear' disconnects all the active SIP calls.
200
DO NOT REPRINT
FORTINET
Firewall Policies
There are two real time debugs that display information about SIP traffic:
diagnose debug application im 31
and
diagnose debug application sip <debug_level>
201
DO NOT REPRINT
FORTINET
Firewall Policies
202
DO NOT REPRINT
FORTINET
Firewall Policies
203
DO NOT REPRINT
FORTINET
Firewall Policies
Traffic policing limits the amount of traffic ingressing or egressing an interface. A packet is allowed
only if the traffic rate is below the threshold for the respective traffic direction.
204
DO NOT REPRINT
FORTINET
Firewall Policies
Traffic queueing assigns packets to one of different queues, depending on the ToS field of the IP
header. Each queue has different priorities: high, medium and low.
205
DO NOT REPRINT
FORTINET
Firewall Policies
206
DO NOT REPRINT
FORTINET
Firewall Policies
207
DO NOT REPRINT
FORTINET
Firewall Policies
There are two CLI commands that display the amount of packets dropped by shared traffic shapers:
diagnose firewall shaper traffic-shaper list
diagnose firewall shaper traffic-shaper stats
208
DO NOT REPRINT
FORTINET
Firewall Policies
209
DO NOT REPRINT
FORTINET
Firewall Policies
210
DO NOT REPRINT
FORTINET
Firewall Authentication
211
DO NOT REPRINT
FORTINET
Firewall Authentication
During this lesson you will learn to monitor the authentication status of the network users. You will also
learn to troubleshoot the most common problems related with local, LDAP and RADIUS
authentication.
212
DO NOT REPRINT
FORTINET
Firewall Authentication
Lets start with some general authentication troubleshooting commands, tools and tips.
213
DO NOT REPRINT
FORTINET
Firewall Authentication
As in the case of any other type of issue, the first step in troubleshooting authentication problems is to
get the specifics: Is the problem happening with more than one user? Is the problem really related with
user authentication and not with user permissions?
The first place to check is usually the logs. What do they show? Is the username correct? Is the traffic
being blocked after the authentication? What is the user profiled assigned to the user?
In the case of remote server authentication (such as RADIUS or LDAP), check also the logs in the
server side. In those cases the result of the FortiGate authentication depends on the server response.
If the RADIUS credentials are invalid, what do the RADIUS servers logs show? Is the user account
still active in the RADIUS server?
214
DO NOT REPRINT
FORTINET
Firewall Authentication
This is sample of the user authentication logs. A log can be generated each time a user authentication
action is successful or fails.
215
DO NOT REPRINT
FORTINET
Firewall Authentication
You can check the list of active users from either the user monitor in the GUI or from the CLI. The CLI
command for that purpose is 'diagnose firewall auth list'. You can previously filter the output with the
command 'diagnose firewall auth filter'.
216
DO NOT REPRINT
FORTINET
Firewall Authentication
Any session for traffic coming from an authenticated user contains the authed flag. Additionally, the
username and user group is added to the session information.
217
DO NOT REPRINT
FORTINET
Firewall Authentication
There is a real time debug for troubleshooting problems related with any user authentication method:
local, remote or even FSSO. It is 'diagnose debug application authd'.
218
DO NOT REPRINT
FORTINET
Firewall Authentication
The second part of this lesson is about debug commands and tools for troubleshooting LDAP
authentication problems.
219
DO NOT REPRINT
FORTINET
Firewall Authentication
Lets start with a quick review of what we learned about the LDAP protocol in NSE4.
The hierarchy of an LDAP schema is not required to hold any resemblance to the organization.
However, generally the name conventions used and the group structure match with the name of the
company and corporate hierarchy very closely.
On the top, we have the root or DC. This is where an LDAP tree always starts, with any schema.
After that, the groups (or branches) are defined using C, OU, and/or O. The exact behavior and
options used depend on the schema and what exactly is being defined. Branches may contain objects
and each object contains attributes. Objects are uniquely identified by their distinguish names (DNs).
The full DN specifies where the object is and the name and value of an attribute that can be used to
find it.
220
DO NOT REPRINT
FORTINET
Firewall Authentication
221
DO NOT REPRINT
FORTINET
Firewall Authentication
Most of the LDAP authentication problems are caused by misconfigurations. So, lets see how to
quickly check if our regular-bind LDAP configuration is ok. We will analyze the case of an LDAP
server based on Windows AD, which is the most common case.
Misconfigurations usually happen in one of these LDAP settings:
- Common name identifier
- Distinguished name
- User DN
- Password
222
DO NOT REPRINT
FORTINET
Firewall Authentication
How do you check if the distinguished name is ok? You can run either of these two commands in the
Windows AD servers command prompt:
dsquery user name <full_user_name>
dsquery user samid <login_username>
The output displays the user DN. The distinguished name setting specifies a parent branch under
which all users are located. The FortiGate searches users in any sub-branch below this parent branch.
For the case in this slide, we can set the distinguished name setting, for example, to:
dc=tac,dc=ottawa,dc=fortinet,dc=com
223
DO NOT REPRINT
FORTINET
Firewall Authentication
The user DN (or bind DN) setting is the full DN of the LDAP admin account. We can use the same
Windows LDAP server command (dsquery) to find that information.
224
DO NOT REPRINT
FORTINET
Firewall Authentication
You can simply copy and paste the full DN output from the server command prompt to the FortiGate
configuration.
225
DO NOT REPRINT
FORTINET
Firewall Authentication
226
DO NOT REPRINT
FORTINET
Firewall Authentication
The CLI includes a LDAP authentication test command. It is 'diagnose test authserver ldap'. If the
credentials are ok, and if the LDAP configuration is ok, you get not only an authentication confirmation,
but also a list of the user groups for that user.
You can run this test command as soon as you have completed you LDAP server configuration, even
before any user group, or authentication firewall policy have been added to the FortiGate. It tests only
the LDAP server configuration and the LDAP communication between the FortiGate and the server.
227
DO NOT REPRINT
FORTINET
Firewall Authentication
228
DO NOT REPRINT
FORTINET
Firewall Authentication
229
DO NOT REPRINT
FORTINET
Firewall Authentication
If there is any problem with either step 1 (admin bind) or step 3 (user bind) you can sniffer the traffic
between the FortiGate and the LDAP server to get the error code. They are very explicit about the
reason why the biding is failing.
230
DO NOT REPRINT
FORTINET
Firewall Authentication
Let's see the most common problems and how to identify them by the output of the real time debug.
If you see the error invalid credentials right after an authentication attempt, this means a problem in
step 1. There is probably an issue with the admin account credentials.
231
DO NOT REPRINT
FORTINET
Firewall Authentication
The message No more DN left indicates a problem in step 2. The LDAP server couldn't find the user
in the LDAP tree.
232
DO NOT REPRINT
FORTINET
Firewall Authentication
The message auth denied after finding the user indicates a problem in step 3. Either the user
credentials are wrong, or the user account is not active.
233
DO NOT REPRINT
FORTINET
Firewall Authentication
234
DO NOT REPRINT
FORTINET
Firewall Authentication
235
DO NOT REPRINT
FORTINET
Firewall Authentication
As FortiGate RADIUS configuration is simpler than LDAP configuration, the troubleshooting is also
usually simpler. Lets see quickly how to troubleshoot RADIUS problems.
236
DO NOT REPRINT
FORTINET
Firewall Authentication
Normal authentication queries with the RADIUS protocol begin with an "Access-Request" being sent
from the FortiGate to the RADIUS server. Valid responses to this are "Access-Accept" and "AccessReject" (yes and no effectively).
If two-factor authentication is enabled on the server, it will come back with an "Access-Challenge"
message, where it is essentially looking for more information.
237
DO NOT REPRINT
FORTINET
Firewall Authentication
238
DO NOT REPRINT
FORTINET
Firewall Authentication
RADIUS is either a one-step or two-step process (depending on the use of two-factor authentication).
It is not as long as the 4-step process that happens with LDAP regular bind. So, the output of the real
time debug is usually shorter.
239
DO NOT REPRINT
FORTINET
Firewall Authentication
240
DO NOT REPRINT
FORTINET
FSSO
This lesson reviews FSSO operation and provides tools and tips for troubleshooting FSSO problems.
241
DO NOT REPRINT
FORTINET
FSSO
You will learn to check the connectivity between a FortiGate and a collector agent (CA), and between
a CA and the domain controllers (DCs).
You will also learn to track user logon events from the DC, to the CA and to the FortiGate.
After that, the lesson explains how to list the active FSSO users both in the CA and the FortiGate.
Additionally, the lesson covers FSSO troubleshooting.
242
DO NOT REPRINT
FORTINET
FSSO
FSSO operation and configuration is covered in NSE4. So, let's do a review of how this authentication
method works.
243
DO NOT REPRINT
FORTINET
FSSO
FSSO enables FortiGate to leverage your networks existing authentication system for firewall
authentication. Once a user logs in, he or she can access other network resources without having to
authenticate again.
These are the two most common FSSO methods: DC agent and polling.
DC agent requires one collector agent. It also requires DC agents installed in all Windows domain
controllers. The DC agents detect the login events and "push" this information to the CA. The CA
forwards the login events to the FortiGate, together with the workstation IP address and user's group
information.
The polling mode does not require any DC agent installed. In this mode, the CA frequently polls each
DC to get the latest login events. There are three types of polling modes:
NetAPI: Polls NetSessionEnum API every 9 seconds.
WinSecLog: Polls all security event logs every 3 seconds.
WMI: Polls specific security event logs every 3 seconds.
These poll intervals are estimated times that depend on the number of servers and network latency.
If a standalone CA is used, the 3 types of polling methods are supported. Alternatively, the polling can
be done directly from the FortiGate (agentless polling mode), so a standalone CA is not required.
However, the FortiGate acting as a CA only supports the WinSecLog polling type.
NSE4 covers the advantages and disadvantages of each FSSO method.
244
DO NOT REPRINT
FORTINET
FSSO
This slide shows how the FSSO traffic flows and which ports are used.
For the case of polling mode, the polling is done from either the FortiGate or the CA, and in both cases
port TCP 445 is used.
The communication between the CA and the FortiGate, by default, uses port TCP 8000. The
communication between the DCs and the CA uses, also by default, port UDP 8002.
FSSO commonly works by identifying each user based on the source IP address of the traffic. Each
active FSSO user is associated with one or more IP addresses and one IP address is associated only
with one user. This creates conflicts in network using terminal or Citrix servers, where more than one
user are sharing the same IP address. For those cases, you can install a terminal server (TS) agent.
The TS agent provides the CA and FortiGate with not only the source IP address of each user, but
also with the source TCP/UDP ports they are using. So, multiple users sharing the same IP address
can be identified by combining the source IP address with the source port. The communication
between the TS agent and the CA also uses port UDP 8002.
245
DO NOT REPRINT
FORTINET
FSSO
246
DO NOT REPRINT
FORTINET
FSSO
247
DO NOT REPRINT
FORTINET
FSSO
248
DO NOT REPRINT
FORTINET
FSSO
There are three important requirements for any FSSO network, and most of the FSSO problems
happen when any of these requirements are not fulfilled.
As explained, the CA frequently polls each active workstation. For this polling to work properly, TCP
ports 139 and 445 must be open between the CA and the workstations. Firewalls must allow this
traffic. Additionally, the remote registry service must be up and running on the workstations.
As explained in NSE4, DNS is used to get the user IP address. So, administrators must ensure that
each workstation name is registered in the DNS server with the right and up-to-date IP address.
249
DO NOT REPRINT
FORTINET
FSSO
250
DO NOT REPRINT
FORTINET
FSSO
What steps should an administrator do when troubleshooting a FSSO problem? This slide offers a
general checklist.
In the next slides, you will learn how to check each of those steps.
251
DO NOT REPRINT
FORTINET
FSSO
If you can reproduce the FSSO problem, there are different tools to track the logon event as it travels
from the DC, to the CA and to the FortiGate. Follow these steps:
- Perform the login.
- Check which DC received the login using the Windows command: echo %logonserver%
- Go to that DC and use the Windows event viewer to find the generated logon event.
- Check the CA logs and the list of active FSSO users.
- Check that at least one user's group is listed as monitored in the group filter.
- Go to the FortiGate and check that the logon event was received. List the active FSSO users.
Generate traffic from the user workstation and verify that it is added to the FortiGate user monitor.
252
DO NOT REPRINT
FORTINET
FSSO
253
DO NOT REPRINT
FORTINET
FSSO
By clicking on show service status in the CA, you can check the connectivity between the FortiGate
and the CA. The window displays the serial numbers of all the active FortiGate units.
254
DO NOT REPRINT
FORTINET
FSSO
When a FortiGate is successfully connecting to a CA, the CA logs show these messages. Additionally
you can confirm a successful connection while running the FSSO real time debug in the FortiGate.
255
DO NOT REPRINT
FORTINET
FSSO
While the TCP session between the FortiGate and the CA is up, the CA sends heartbeat messages to
the FortiGate unit. Both the CA logs and the FortiGate real time debug show this heartbeat
interchange.
256
DO NOT REPRINT
FORTINET
FSSO
257
DO NOT REPRINT
FORTINET
FSSO
By clicking on the show monitored DCs button on the CA, you can check the communication between
the CA and each DC. The table includes the time when the last logon event was received from each
DC.
258
DO NOT REPRINT
FORTINET
FSSO
The Event Viewer is a Windows tool that displays all the server logs. You can use it to search for
logon event logs.
259
DO NOT REPRINT
FORTINET
FSSO
The FortiGate FSSO real time debug generates messages each time a logon event arrives. They
include the name and IP address of the user, together with the workstation name and user's group
information.
260
DO NOT REPRINT
FORTINET
FSSO
You can check the list of logon users by selecting show login users. The status OK indicates that the
user is active. A user goes to not verified status when either she or he has logged out, or when there
is a problem in the polling done by the CA to the workstation.
261
DO NOT REPRINT
FORTINET
FSSO
To get the list of active users from the FortiGate, use the command 'diagnose debug authd fsso
list'. You can set up a filter first with the command 'diagnose debug authd fsso filter'.
262
DO NOT REPRINT
FORTINET
FSSO
The CLI command 'diagnose debug authd fsso refresh-logons' refreshes the active FSSO user list
in the FortiGate by getting this information again from the CA.
The CLI command 'diagnose debug authd fsso clear-logons' flushes the list of active FSSO users
in the FortiGate.
The CLI command 'diagnose debug authd fsso refresh-groups' refreshes the user group
information in the FortiGate by getting this information again from the CA.
To list the monitored user groups, use the command 'get user adgrp'.
263
DO NOT REPRINT
FORTINET
FSSO
Agentless polling mode allows the FortiGate to directly poll each DC. The FortiGate acts as a CA, so
no standalone CA is required. There are some specific FSSO commands for troubleshooting this
mode.
264
DO NOT REPRINT
FORTINET
FSSO
The command 'diagnose debug fsso-polling detail' displays the status and some statistics related
with the polls done by the FortiGate to each DC.
The command 'diagnose debug fsso-polling refresh-user' flushes the information about all active
FSSO users.
In agentless polling mode, the FortiGate frequently polls all workstations (as a standalone CA does) to
check which users are still logged on. You can sniffer this traffic in port 445.
265
DO NOT REPRINT
FORTINET
FSSO
There is a specific FortiGate daemon that handles the polling mode. It is the fssod daemon. To enable
agentless polling mode real time debug use:
diagnose debug application fssod -1
The slide shows the most common real time debug errors and the possible causes.
266
DO NOT REPRINT
FORTINET
FSSO
To finish this lesson, we will discuss the most common FSSO problems.
267
DO NOT REPRINT
FORTINET
FSSO
268
DO NOT REPRINT
FORTINET
FSSO
If a FSSO user is listed as active in the FortiGate but it cannot browse the Internet, you should first
confirm the IP address listed in the FortiGate. Also check the user's group information, firewall policies
and CA logs.
If the FortiGate is randomly blocking some users after some time, you should check that the CA
service, or any FortiGate process, is not crashing. Confirm that the connection between the FortiGate
and the CA is up and stable. Try to reproduce the problem and check the CA logs after that.
269
DO NOT REPRINT
FORTINET
FSSO
DNS resolution is essential for FSSO operation. If a CA cannot resolve a workstation name, the user
will not be listed as active and the CA logs show the errors in this slide.
270
DO NOT REPRINT
FORTINET
FSSO
Another common problem with FSSO authentication happens when there are applications generating
logon events with an AD account different than the users'. In those cases the logon event overrides
the information about the user for a workstation IP address (a different user is listed as active for the
IP address). The CA is coded to ignore any logon event for anonymous accounts, and account whose
name starts with '$' (which are system accounts). If any application is using an account that is
overriding the user information, the solution is to add that account to the ignore user list.
271
DO NOT REPRINT
FORTINET
FSSO
Active users with the status not verified are also a common problem. That status is normal after a user
has logged out. However, it is not normal for a user that is still logged on, meaning that the polling
from the CA to the workstation is failing. In those cases, check the CA logs. They provide more
information about the cause of the problem.
272
DO NOT REPRINT
FORTINET
FSSO
What should you do if a user lost internet access after the IP address changed?
This type of problem might happen when the user moves back and for from a wired to a wireless
network. It might also happens when a workstation gets back from hibernate mode (a gets a new IP
address from the DHCP server).
The first step is to check the DNS resolution. As explained in a previous slide, when a user changes
the IP address, the DNS server must be automatically updated with the new IP information. If that is
not possible, one workaround to this problem is to configure FSSO guest access. So, users whose IP
addresses have changed can still have some (probably limited) access to some network resources.
For the cases where a workstation was assigned more than one IP address (for example one address
for the wired network and another one for the wireless), the DNS server should be able to return all
those IP addresses when resolving the workstation name. The user is then listed multiple times in the
FSSO active user list, one time for each IP address.
273
DO NOT REPRINT
FORTINET
FSSO
274
DO NOT REPRINT
FORTINET
IPsec
This lesson is about IPsec troubleshooting. It is one of the most important lessons, as a significant
percentage of the support tickets received by Fortinet TAC are related with IPsec.
275
DO NOT REPRINT
FORTINET
IPsec
At the end of this lesson you will know how to bring down, bring up and restart an IPsec VPN. The
lesson also teaches how to check the IPsec offloading and use the real time debug to troubleshoot
problems during the negotiations of phase 1 and 2. Additionally, you will learn about IPsec traffic
capture using the sniffer and debug flow.
276
DO NOT REPRINT
FORTINET
IPsec
277
DO NOT REPRINT
FORTINET
IPsec
278
DO NOT REPRINT
FORTINET
IPsec
279
DO NOT REPRINT
FORTINET
IPsec
This slide details the selection criteria for dial-up. A FortiGate uses the first VPN configuration (in
alphabetical order) that matches:
- Local gateway IP
- Mode (aggressive or main)
- Peer ID, if aggressive mode is used. As explained, only aggressive mode include the Peer ID in the
first packet.
- Authentication method (pre-shared or PKI)
- Digital certificate information, if PKI is used as the authentication method
So, if a FortiGate has multiple dial-up VPNs with pre-shared key sharing the same local gateway, you
must use aggressive mode and different peer IDs. In this way, the FortiGate identifies the right VPN
configuration for each incoming IPsec proposal.
280
DO NOT REPRINT
FORTINET
IPsec
If you need to capture the IPsec traffic, keep in mind that the IP protocol and UDP port numbers
depend on NAT-T and the use of NAT.
If there is no device in the middle doing NAT, IKE traffic uses UDP port 500 and ESP traffic uses IP
protocol 50. The slide shows the two sniffer filters to capture each of those traffic protocols.
If, on the other hand, NAT-T is enabled and there is a device in the middle doing NAT, the sniffer
command must use a different filter. In this cases IKE traffic starts using port UDP 500, but it switches
to UDP port 4500 during the tunnel negotiation. Additionally, ESP traffic is encapsulated inside the
UDP-4500 channel.
281
DO NOT REPRINT
FORTINET
IPsec
282
DO NOT REPRINT
FORTINET
IPsec
283
DO NOT REPRINT
FORTINET
IPsec
The same command 'diagnose vpn tunnel' offers options for shutting down, activating or flushing
any phase 2.
284
DO NOT REPRINT
FORTINET
IPsec
'get vpn ipsec stats tunnel' shows the number of IPsec VPNs in the configuration.
Both 'get vpn ipsec tunnel summary' and 'get ipsec tunnel list' can be used to list summarized
information about each IPsec VPN.
285
DO NOT REPRINT
FORTINET
IPsec
286
DO NOT REPRINT
FORTINET
IPsec
In some FortiGate models, the encryption and decryption of IPsec traffic can be offloaded to hardware.
The supported algorithms depend on the model and type of processor unit doing the offloading.
For the cases of NP2, and NP4 with FortiOS 5.2.2 or lower, there is an additional requirement. If IPsec
anti-replay is enabled, you must check that IPsec offloading is enabled under 'config system npu'.
If the hardware is NP6, NP4lite, or NP4 with FortiOS 5.2.3 or newer, enabling those settings is not
required for IPsec offloading.
NP2 has an important limitation regarding anti-replay. We will talk about it later.
287
DO NOT REPRINT
FORTINET
IPsec
For the case of IPsec traffic, the FortiGate session table includes a field that describes when the
encryption and decryption is being offloaded.
First, when the phase 2 goes up, the IPsec SAs are created and loaded to the kernel. As long as there
is not traffic crossing the tunnel, the SAs are not copied to the NPU and the npu_flag shows 00. The
value of that field also remains in 00 when IPsec offloading is disabled or not supported.
Second, when the first IPsec packet arrives, if it is an outbound packet that can be offloaded, the
outbound SA is copied to the NPU and the npu_flag changes to 01. If, on the other hands, the first
IPsec packet is inbound and can be offloaded, the inbound SA is copied to the NPU and the npu_flag
changes to 02.
Finally, once both SAs are copied to the NPU, the npu_flag changes to 03.
288
DO NOT REPRINT
FORTINET
IPsec
This command shows number of packets encrypted and decrypted by software (CPU), and by each
type of processor unit in the FortiGate.
289
DO NOT REPRINT
FORTINET
IPsec
The IPsec or IKE real time debug is the most useful tool for troubleshooting problems during the
tunnel negotiation. Before enabling the debug, you should first set the filter. The recommended filter
option is dst-addr4 (or dst-addr6 for IPv6). With this filter, the debug shows only and all the
information about the IPsec tunnel whose remote IP address matches the filter.
290
DO NOT REPRINT
FORTINET
IPsec
After setting the filter, enable the IKE real time debug. The table shows what type of output is enabled
by each bit in the bit-mask. The most common values for the bit-mask are -1 (all outputs enabled) and
63 (shorter output). They both show the DPD packets and all the information required for
troubleshooting IPsec negotiation problems.
291
DO NOT REPRINT
FORTINET
IPsec
292
DO NOT REPRINT
FORTINET
IPsec
293
DO NOT REPRINT
FORTINET
IPsec
294
DO NOT REPRINT
FORTINET
IPsec
295
DO NOT REPRINT
FORTINET
IPsec
296
DO NOT REPRINT
FORTINET
IPsec
We explained extended authentication (XAuth) in NSE4. Here, you will see what packets are
interchanged during extended authentication.
XAuth happens after the phase 1 is up and before any phase 2 negotiation. That is why XAuth is also
called phase 1.5.
In any XAuth communication there is always one client and one server. The server sends a
CFG_REQUEST packet, which must be replied by the client with a CFG_REPLY packet. The
CFG_REPLY packet includes the user credentials. If the authentication is ok, the server sends
CFG_SET and the client replies with CFG-ACK.
297
DO NOT REPRINT
FORTINET
IPsec
298
DO NOT REPRINT
FORTINET
IPsec
The messages in the real time debug, related with the CFG_REPLY, show the XAuth user and group
names.
299
DO NOT REPRINT
FORTINET
IPsec
If a VPN is using XAuth, the output of the command 'diagnose vpn ike gateway list' shows the
username. The same information is displayed in the IPsec monitor in the FortiGate GUI.
300
DO NOT REPRINT
FORTINET
IPsec
A FortiGate has two different methods for automatically configuring the IP settings of an IPsec client:
DHCP over IPsec and IKE mode configuration.
We analyze the DHCP over IPsec method first.
After the phase 1 is up, and the extended authentication is completed, the client negotiates a
provisional phase 2 of short duration only for the DHCP traffic. Once that provisional phase 2 is up,
standard DHCP traffic is interchanged to set the client IP settings: DHCP_DISCOVER,
DCHP_OFFER, DHCP_REQUEST and DHCP_ACK.
After the client gets the IP settings (IP address, default gateway, DNS, among others) the phase 2
created for DHCP is turned down. After that, a new phase 2 is negotiated for user traffic.
301
DO NOT REPRINT
FORTINET
IPsec
302
DO NOT REPRINT
FORTINET
IPsec
When troubleshooting DHCP over IPsec problem, you can also use the DHCP real time debug. It
shows information about the DHCP traffic, including all the IP settings assigned to the IPsec client.
303
DO NOT REPRINT
FORTINET
IPsec
The other method for assigning IP settings to an IPsec client is IKE mode configuration. Similarly to
DHCP over IPsec, it happens after the phase 1 and extended authentication and before any phase 2
negotiation. The client sends a CFG_REQUEST message listing the required IP settings (or
attributes). The server replies with a CFG_REPLY which contains the assigned values for each of the
attributes requested.
304
DO NOT REPRINT
FORTINET
IPsec
The IKE real time debug shows when the CFG_REQUEST and CFG_REPLY packets are sent.
305
DO NOT REPRINT
FORTINET
IPsec
Let's say now that the phase 1 goes up, the phase 2 also goes up, but, for some reason, the traffic is
not crossing the tunnel.
For those cases, the best troubleshooting tool is the debug flow. If possible, run it in both gateways. It
will let you know if the local gateway is dropping the packets, if the local gateway is not routing the
packets through the tunnel, or if the remote gateway is the one dropping the packets.
This slide shows the normal output. You should see a message indicating that the packet goes to the
tunnel (with the tunnel name), and another one explaining that the packet is being encrypted.
306
DO NOT REPRINT
FORTINET
IPsec
307
DO NOT REPRINT
FORTINET
IPsec
If the tunnel is up, but traffic does not go through, use the debug flow. It could be that packets are
being dropped at either the local or remote gateway. It could be that packets are not being routed
properly. Or it could be that packets do not match the quick mode selectors, so they are dropped.
308
DO NOT REPRINT
FORTINET
IPsec
Most of the IPsec problems happen when creating tunnels between FortiGate and a third-party device.
Differently than FortiGate, some vendors do not support quick mode selectors that are either 0.0.0.0/0,
or use subnets with different sizes. So, they require a different SA (and so a different phase 2) for
each pair of local and remote protected subnets. On those cases, the FortiGate configuration might
get complicated and long as administrators need to create all the different phases 2.
One alternative is to use the 'mesh-selector-type' setting in the phase 1. When it is set to 'subnet'
the FortiGate automatically creates the phases 2 on the flight and per demand. This might simplify the
FortiGate configuration considerably.
309
DO NOT REPRINT
FORTINET
IPsec
Anti-replay is an IPsec mechanism that adds a sequence number to the ESP headers. The number is
incremented each time a packet is sent. It protects the tunnel against replay attacks.
We mentioned earlier that there is a restriction related with NP2 and IPsec anti-replay. NP2 cannot
properly generate this sequence number. So, if encryption offloading is enabled in a NP2 platform, you
might get warning and packets dropped if the remote site has anti-reply enabled. Therefore, you would
need to disable either anti-replay or encryption offloading. This limitation does not apply to NP4 and
NP6 platforms.
310
DO NOT REPRINT
FORTINET
IPsec
311
DO NOT REPRINT
FORTINET
Security Profiles
During this lesson, you will learn to troubleshoot some of the security profiles (or UTM inspection) features.
312
DO NOT REPRINT
FORTINET
Security Profiles
You will learn to troubleshoot FortiGuard, web filtering and antivirus issues. You will also learn to monitor
the IPS and fix certificate warning problems during full SSL inspection.
Additionally, the lesson compares proxy-based and flow-based inspection modes and describes the
FortiGate packet inspection steps.
313
DO NOT REPRINT
FORTINET
Security Profiles
314
DO NOT REPRINT
FORTINET
Security Profiles
The FortiGate GUI can be used to quickly check the status of the communication between FortiGate and
FortiGuard. A green checkmark beside each FortiGuard service indicates that FortiGate can reach
FortiGuard and that the license for that service is valid.
315
DO NOT REPRINT
FORTINET
Security Profiles
316
DO NOT REPRINT
FORTINET
Security Profiles
For web filtering and antispam, the communication between FortiGate and FortiGuard uses either port 53
or 8888. The GUI also offers a test button to check this connectivity.
317
DO NOT REPRINT
FORTINET
Security Profiles
The command 'diagnose debug rating' shows the list of servers for web filtering and antispam queries.
For each IP address, the table shows:
- The round trip delay
- The server's time zone
- The amount of recent and consecutives queries without reply
- The historical total amount of queries without reply
318
DO NOT REPRINT
FORTINET
Security Profiles
This is how FortiGate selects the server from the list to send the rating requests.
FortiGate initially uses the delta between the server's time zone and the FortiGate's system time zone
multiplied by 10. This is the server initial weight.
The weight goes up with each packet lost.
The weight goes down over time if there are no packets lost. But, it never goes below the initial value.
FortiGate uses the server with the lowest weight as the one for the rating queries.
319
DO NOT REPRINT
FORTINET
Security Profiles
The output of the command 'diagnose debug rating' shows some flags beside some of the servers:
I: This was the server initially contacted to validate the license and get the server list. Usually, there is only
one server with this flag
D: FortiGate got this IP address when resolving the name service.fortiguard.net. If the administrator has
not overwritten the FortiGuard FQDN or IP address in the FortiGate configuration, there are usually two or
three servers with this flag
S: FortiGate got this IP address from a FortiManager
T: Server is not replying to FortiGate queries
F: Server is down
320
DO NOT REPRINT
FORTINET
Security Profiles
321
DO NOT REPRINT
FORTINET
Security Profiles
Let's move to FortiGuard antivirus and IPS. For these two services the communication between FortiGate
and FortiGuard happens much less frequently. In the case of web filtering and antispam, FortiGate goes to
FortiGuard each time it needs to rate a web site or email (if the information is not in the FortiGate cache).
In the case of antivirus and IPS, FortiGate goes to FortiGuard usually once a day to check and download
any new version of the AV or IPS databases and engines. This is done using port TCP 443. By default,
FortiGate connects to update.fortiguard.net.
If FortiGate must connect through a web proxy, these are the CLI commands to use. Usually, clients
connecting to a web proxy do not contact the DNS server to resolve names, as it is the web proxy who
does it. But, in the case of FortiGuard, FortiGate always requires DNS access, even when connecting
through a web proxy.
322
DO NOT REPRINT
FORTINET
Security Profiles
Both the AV and IPS databases can be updated either automatically or manually. Automatic updates
download the portions of the databases that have changed since the last update.
Manual updates download the whole database if there is new version available.
In a few cases, FortiGuard problems are fixed by doing a manual update. This forces FortiGate to
download and install the whole database. For example, if the AV database is corrupted, a manual update
might fix the problem.
323
DO NOT REPRINT
FORTINET
Security Profiles
The command 'diagnose test application dnsproxy 7' displays the FQDN and IP addresses of the
FortiGuard servers available for AV and IPS updates.
The command 'diagnose autoupdate status' provides a summary of the FortiGuard configuration in the
FortiGate.
324
DO NOT REPRINT
FORTINET
Security Profiles
'diagnose autoupdate versions' lists all the FortiGuard databases and engines installed. The information
includes the version, contract expiration date, time it was updated and what happened during last update.
The output displays information about the AV engine, AV database and IPS database.
325
DO NOT REPRINT
FORTINET
Security Profiles
It also shows the IPS engine, device identification database and IP geography database.
326
DO NOT REPRINT
FORTINET
Security Profiles
If there are problems updating the AV or the IPS, or if there are problems validating the license, you can
use the FortiGuard real time debug:
diagnose debug application update -1
diagnose debug enable
After enabling the debug, you can force a manual update from the CLI with the command:
execute update-now
For FortiGuard web filtering and antispam, the commands for enabling the real time debugs are different.
We will see the web filter real time debug later.
327
DO NOT REPRINT
FORTINET
Security Profiles
Remember that FortiGuard traffic originates always from the management VDOM. So, the management
VDOM (which is root by default) must have Internet access.
Proper DNS access from the management VDOM is also important. The FortiGate must be able to resolve
the names:
update.fortiguard.net
service.fortiguard.net
Also, keep in mind that updates to FortiGuard contracts are NOT instantaneous. It usually takes around 1
or 2 hours to update a contract in all the servers. However, in some cases, it could take up to 24 hours. So,
if you have just changed or renewed your FortiGuard contract and you do not see the change in the
FortiGate, most probably you just need to wait a bit more, to give FortiGuard time to synchronize the
information in all the servers.
328
DO NOT REPRINT
FORTINET
Security Profiles
Before talking about how to troubleshooting the most common UTM features, let's talk about inspection
modes and life of a packet.
329
DO NOT REPRINT
FORTINET
Security Profiles
Proxy and flow-based inspection modes are covered in NSE4. Let's do a quick review.
Most of the UTM features (AV, web filtering) can work in either proxy or flow-based mode.
In proxy-based mode, the FortiGate transparently proxies any TCP session between a client and a server.
So, actually two TCP sessions are created: one between the client and the FortiGate, and another one
between the FortiGate and the server. This mode is usually slower than flow-based, but it offers all the
functionality.
330
DO NOT REPRINT
FORTINET
Security Profiles
Flow-based mode, on the other hand, does not proxy TCP sessions. The traffic is inspected as it flows
through the unit as a single TCP session. This mode is usually faster that proxy-based, but offers less
functionality.
331
DO NOT REPRINT
FORTINET
Security Profiles
If traffic matches a firewall policy that combines proxy-based with flow-based inspection profiles, FortiGate
does proxy-based inspection for all the profiles that support both modes.
This does not apply to IPS or application control profiles, which are always and only flow based.
332
DO NOT REPRINT
FORTINET
Security Profiles
Application proxies are used during proxy-based inspection. They split the TCP session into two and
decide what actions to take. They also buffer the traffic and files, generates logs and archives information.
333
DO NOT REPRINT
FORTINET
Security Profiles
Two session flags indicate if the traffic is being inspected in either proxy or flow-based mode. The flag redir
means proxy-based mode. The flag ndr means flow-based mode. Also, and for the case of proxy-based
inspection, the debug flow shows the message "send to application layer".
334
DO NOT REPRINT
FORTINET
Security Profiles
During the following slides we are going to analyze how a packet is processed by FortiGate since arrives
to the unit until egresses the unit. We will see what actions the FortiGate takes and the order of those
actions. We have split the whole process into four stages. The first stage happens when the packet is
ingressing:
First, you can set a limit in the incoming bandwidth ('inbandwidth' parameter). If the traffic has exceeded
that limit, the packet is drop.
After that, FortiGate checks the thresholds in the DoS policies.
The next steps are the reverse path forwarding and IP header integrity checks.
If the traffic terminates at the FortiGate (for example management traffic, web proxy, DNS) the packet is
processed by the specific daemon that handles the requested feature. If the traffic does not terminate at
the FortiGate, it goes to the second stage: routing and firewall policy.
335
DO NOT REPRINT
FORTINET
Security Profiles
Before doing the routing lookup, the FortiGate (if required) applies the destination NAT. If there is no route
to the destination, the packet is dropped. In a similar way, if the packet is denied by a firewall policy, it will
not be allowed. If the packet is allowed, FortiGate checks if the policy requires authentication. After those
steps, FortiGate proceeds to identify the traffic, which is then inspected by any session helper that might
be required.
336
DO NOT REPRINT
FORTINET
Security Profiles
The third stage is UTM inspection. If the traffic is encrypted and full SSL inspection is used, the unit
proceeds to decrypt the traffic. After that, the inspection profiles are applied in this order: IPS, application
control, VoIP, DLP, antispam, web filtering and antivirus.
337
DO NOT REPRINT
FORTINET
Security Profiles
After inspection, FortiGate applies traffic shaping and source NAT. Finally, if the packet must be routed
through an IPsec VPN, it is encrypted before egressing the unit.
338
DO NOT REPRINT
FORTINET
Security Profiles
We will talk now about the most common UTM problems. We will start with antivirus.
339
DO NOT REPRINT
FORTINET
Security Profiles
This is the order of how antivirus inspection is executed. First, the unit does the virus scan, then the
grayware scan, and finally the heuristic scan.
340
DO NOT REPRINT
FORTINET
Security Profiles
One of the best tools for troubleshooting antivirus issues is the FortiGate logs. This is a sample of a log
generated when the FortiGate detects a virus. You can see the name of the file, time and name of the
virus.
341
DO NOT REPRINT
FORTINET
Security Profiles
What should an administrator do if a virus is detected in a workstation behind a FortiGate doing antivirus.
Again, the first step is to get specific information: What AV software did detect the virus? What is the name
of the virus? How did it get inside the network? It might have got inside, for example, through a CD or USB
stick, so the infected file never went through the FortiGate.
342
DO NOT REPRINT
FORTINET
Security Profiles
Test if the FortiGate antivirus is properly inspecting the user traffic. Go to eicar.org from a user workstation
and try to download the EICAR sample file.
If you have a sample of the virus, submit it to FortiGuard and check if it is present in any of the FortiGate
virus databases. If it is, check the name of the AV database where it is present. Does your FortiGate have
that database installed?
343
DO NOT REPRINT
FORTINET
Security Profiles
There are three FortiGate AV databases: normal, extended and extreme. Not all the models support the 3
databases, and not all the databases might be installed in your FortiGate device.
344
DO NOT REPRINT
FORTINET
Security Profiles
345
DO NOT REPRINT
FORTINET
Security Profiles
346
DO NOT REPRINT
FORTINET
Security Profiles
During web filtering inspection, FortiGate first checks the static URL filter list, then the FortiGuard
categories and content filtering. Finally, the web filtering can execute some advanced options.
347
DO NOT REPRINT
FORTINET
Security Profiles
Similarly to other UTM features, one of the best troubleshooting tools for web filtering is the FortiGate logs.
The unit can generate a log each time a web site is blocked. The log lists the URL, the category, action
taken and quota info.
348
DO NOT REPRINT
FORTINET
Security Profiles
Error counters related with web filtering can be listed with the command 'diagnose webfilter fortiguard
statistics list'.
349
DO NOT REPRINT
FORTINET
Security Profiles
The output also shows counters for the number of sites allowed and blocked, and statistics about the
cache (including hit and miss rates).
350
DO NOT REPRINT
FORTINET
Security Profiles
Another tool for web filtering troubleshooting is the web filter real time debug. You can set a filter first by
user (source) IP address with the command:
diagnose debug urlfilter src-addr
After that, you can enable the real time debug:
diagnose debug application urlfilter -1
This slide shows a sample output of the real time debug when the URL to categorize is not in the
FortiGuard cache. The two messages display the URL, category, source, destination IP addresses and
service.
351
DO NOT REPRINT
FORTINET
Security Profiles
This is another sample of the real time debug. But in this case, the URL category was found in the
FortiGuard cache.
352
DO NOT REPRINT
FORTINET
Security Profiles
353
DO NOT REPRINT
FORTINET
Security Profiles
The command 'diagnose test application urlfilter 2' clears the web filter
cache.
The command 'diagnose test application urlfilter 99' restarts the web filtering
daemon.
354
DO NOT REPRINT
FORTINET
Security Profiles
355
DO NOT REPRINT
FORTINET
Security Profiles
We are going to briefly talk about monitoring IPS and DoS operation.
356
DO NOT REPRINT
FORTINET
Security Profiles
IPS logs show information about attacks detected by the IPS. We can use them to see the source and
destination IP addresses, and time and name of the attack.
357
DO NOT REPRINT
FORTINET
Security Profiles
'diagnose ips packet status' shows general IPS statistics, including amount of packet inspected (by IP
protocol) and counters for each action taken.
358
DO NOT REPRINT
FORTINET
Security Profiles
The IPS test application command can be used to restart the IPS process. If, for any reason, you need to
restart the IPS daemon, this is the right way to do it (do not use the command 'diagnose sys kill' to restart
it).
There is also an option for bypassing the IPS. To "toggle" the bypass status (switch it back and for from
enabled to disabled) use the option 5. When IPS bypass is enabled, the FortiGate does not do any kind of
IPS inspection over the traffic. All IPS inspection is actually bypassed. This is useful when troubleshooting
high CPU problems caused by the IPS engine. In those cases, if after enabling IPS bypassing, the CPU
usage is still high, it might indicate a problem in the IPS engine that should be reported to Fortinet TAC.
359
DO NOT REPRINT
FORTINET
Security Profiles
The anomalies (or DoS) logs are aggregated. When several incidents occur together, this reduces the
number of log messages.
In large attacks, the number of incidents can easily reach 100,000 in a few seconds. Generating a log
entry for every packet that matches would completely utilize the CPU. So instead, FortiGate collapses
incidents by periodically recording only one message for all of them, and noting the number of incidents.
Here, the detection threshold was 50, and the total count is 75. So, FortiGate doesnt make 24 separate
log entries (1 for each incident above 50). Its just one log message.
360
DO NOT REPRINT
FORTINET
Security Profiles
The command 'diagnose ips anomaly list' lists the statistics for traffic matching any DoS policy. For each
IP address and DoS policy, the output displays the expiration time (when the entry will be removed from
the DoS table) and the packets per seconds.
361
DO NOT REPRINT
FORTINET
Security Profiles
NSE4 introduces the different SSL inspection modes. Here, we are going to review and expand those
concepts.
362
DO NOT REPRINT
FORTINET
Security Profiles
There are two SSL inspection modes: SSL certificate inspection and full SSL inspection. When using SSL
certificate inspection, the FortiGate is not decrypting the traffic. It is only inspecting the server digital
certificates and the name indication (SNI) field, which are interchanged before the encryption.
The FortiGate tries first to get the URL from the SNI field. The SNI field is a TLS extension that contains
the complete URL that the user is connecting to. It is supported by most of the modern web browsers.
If the SNI field is not present (because the web client may not support it), the FortiGate proceeds to inspect
the server digital certificate. It specifically uses the FQDN in the common name (CN) field as the URL to
categorize. Usually, but not always, this FQDN matches the site the user is connecting to. But, it is not
always accurate. That is why it is only use as an alternative when the first method (SNI) fails.
If you see that the FortiGate is wrongly identifying a HTTPS URL, check the CN field in the site's digital
certificate. It might be that the client does not support SNI and the FortiGate is getting the wrong URL from
the certificate. In those cases the solution would be to use either full SSL inspection, or a client that
supports SNI.
363
DO NOT REPRINT
FORTINET
Security Profiles
Full SSL inspection is available only in some FortiGate models. In this case, the device does indeed
decrypt (and re-encrypt) the SSL traffic.
Under normal circumstances (without full SSL Inspection), encrypted traffic cannot be inspected, as the
firewall does not have the key required to decrypt the data.
In order to work, the FortiGate must be located in the middle of the communication between the users
browser and the web site. When the browser connects to the site, the web server sends its certificate,
which contains its public key.
The FortiGate intercepts the web server certificate and generates a new one on the fly. The new certificate
is issued by the CA installed on the FortiGate, which is not and well-known public CA. The FortiGate also
generates on the fly a new pair of public and private keys.
364
DO NOT REPRINT
FORTINET
Security Profiles
When enabling SSL Inspection, browsers start displaying a certificate warning each time a user is
connecting to a HTTPS site. The reason is that the certificates received by the browsers are now being
signed by the FortiGate, which is a CA that the browsers do not trust. There are two ways to fix this
warning:
The first option is to download the default FortiGate certificate for SSL proxy inspection and install it in all
the workstations.
The second option is to generate a new SSL proxy certificate from a private CA. In this case, the private
CA certificate must still be installed in all the workstations.
This is not a limitation in FortiGate, but a consequence of how digital certificates were designed to work.
365
DO NOT REPRINT
FORTINET
Security Profiles
Replacing the certificate on the traffic can cause problems. Some software and servers have specific
limitations on the certificates that are allowed to be used.
HSTS and PKP are security features designed to detect man-in-the-middle SSL attacks by making sure
that any certificate presented when accessing a server resource is signed by a specific CA. If it detects
any other CA it will simply refuse to continue the SSL handshake and prevent access to the website.
The options available for these are limited. One option is to bypass SSL inspection of that traffic. Another
option is to use SSL certificate inspection instead.
366
DO NOT REPRINT
FORTINET
Security Profiles
367
DO NOT REPRINT
FORTINET
NSE4 covers explicit web proxy configuration. This lesson is about explicit web proxy troubleshooting.
368
DO NOT REPRINT
FORTINET
At the end of this lesson you will be able to troubleshoot explicit web proxy issues by checking the
status of the proxy users, user traffic, sessions and DNS traffic. You will also learn to use the proxy
real time debug.
369
DO NOT REPRINT
FORTINET
With explicit web proxy, clients do not send their requests directly to the servers, but to the proxy. The
proxy processes the requests and forwards them to the servers. Clients must be explicitly configured
with the proxy IP address (or FQDN) and the port that the proxy is listening to. A web proxy can
optionally cache the web traffic.
370
DO NOT REPRINT
FORTINET
There are 3 methods for configuring the explicit web proxy settings in a browser: Manual, using a PAC
file, and using the web proxy autodiscovery protocol (WPAD). WPAD is explained in NSE4.
371
DO NOT REPRINT
FORTINET
A PAC file defines when the browser must use a proxy and how to choose it (when more than one
proxy is available). If you are not using WPAD, and want to use a PAC file, you must configure the
browser with the URL of the HTTP server where the file can be downloaded. Usually, the PAC file is
stored in the proxy itself. In the case of FortiGate, by default, the PAC file can be downloaded from the
URL:
http://<FortiGate_IP>:<Port>/proxy.pac
Both the port and the PAC file name are configurable.
372
DO NOT REPRINT
FORTINET
373
DO NOT REPRINT
FORTINET
Given that cache consumes system resources, do you want all users to be able to use the cache?
You can configure the proxy to allow access only to authenticated users that belong to specific user
groups. Authentication can be either based on either source IP address or HTTP session cookies.
How should you decide which to use?
IP-based authentication requires less memory. However, it should only be used when each user has a
different IP address. If your users are sharing the same IP address, use HTTP session-based
authentication instead. In this mode, each browser inserts an HTTP cookie in its requests. The
cookies identify each user. This method requires slightly more memory because FortiGate must
remember all session cookies.
374
DO NOT REPRINT
FORTINET
The GUI user monitor shows the active explicit web proxy users. To get that list from the CLI, use the
command:
diagnose wad user list
To clear (de-authenticate) all the explicit web proxy user, use the command:
diagnose wad user clear
375
DO NOT REPRINT
FORTINET
376
DO NOT REPRINT
FORTINET
377
DO NOT REPRINT
FORTINET
The command 'diagnose wad session list' shows a summary of the all the sessions related with web
proxy. In this case, each pair of TCP sessions (client to proxy and proxy to server) is displayed as one
entry.
378
DO NOT REPRINT
FORTINET
FortiOS limits the number of simultaneous explicit web proxy users. This limit is hard coded, applies
globally (to all VDOMs) and depends on the model. However, there are no limits in the number of
simultaneous sessions per user.
If user authentication is not used, each source IP address is treated as a different user.
379
DO NOT REPRINT
FORTINET
Before running any explicit proxy debug command, you must enable them with the command:
diagnose test application wad 2200
After that, use 'diagnose test application wad 112' to get the maximum number of simultaneous
proxy users.
380
DO NOT REPRINT
FORTINET
The command 'diagnose test application wad 110' shows the amount of concurrent proxy users.
The output also shows the list of users, their IP addresses and the authentication method they are
using (either IP-based or session-based).
As explained in the previous slide, type first the command 'diagnose test application wad 2200' if
you have not done it yet.
381
DO NOT REPRINT
FORTINET
382
DO NOT REPRINT
FORTINET
383
DO NOT REPRINT
FORTINET
In this example we are enabling the real time debug to display information about connections to the
URL www.fortinet.com.
The debug shows good details about what is happening step-by-step with the proxied traffic.
First, you should see the HTTP GET request coming from the client. The output shows part of the
HTTP header.
384
DO NOT REPRINT
FORTINET
385
DO NOT REPRINT
FORTINET
If it is not in the cache and the server replies, the debug shows an output similar to this one.
386
DO NOT REPRINT
FORTINET
387
DO NOT REPRINT
FORTINET
If, on the contrary, the content was found in the cache, the output is a bit different.
388
DO NOT REPRINT
FORTINET
389
DO NOT REPRINT
FORTINET
Operation Modes
During this lesson, we will talk about the two FortiGate operation modes: NAT/route and transparent.
390
DO NOT REPRINT
FORTINET
Operation Modes
At the end of this lesson you will know how to identify the route path for any traffic session, diagnose and
troubleshoot routing problems and segment a layer-2 network into multiple broadcast domains.
391
DO NOT REPRINT
FORTINET
Operation Modes
Traditional firewalls and NAT mode FortiGate units are routers. So, each interface has to be in different
subnets and each forms different broadcast domains. The FortiGate routes IP packets based on the IP
header information, overriding the source MAC address. So, if a client sends a packet to a server
connected to a different FortiGate interface, the packet arrives to the server with a FortiGates MAC
address, instead of the clients.
In the case of transparent mode, FortiGate forwards frames without changing the MAC addresses. When
the client receives a packet from a server connected to a different FortiGate interface, the frame contains
the servers real MAC address. A transparent mode FortiGate is a Layer-2 bridge or switch. So, the
interfaces do not have IP addresses and all belong (by default) to the same broadcast domain.
This means that a transparent mode FortiGate can be installed in a customer network without changing the
customers IP address plan.
A FortiGate can combine VDOMs in NAT mode with VDOMs in transparent mode.
392
DO NOT REPRINT
FORTINET
Operation Modes
393
DO NOT REPRINT
FORTINET
Operation Modes
A FortiGate is a stateful device, so a lot of information is decided right at the beginning of a session, on the
first packets. For any traffic session, usually only two routing lookups are done: one on the first packet sent
by the originator, and another one on the first reply packet coming from the responder. After that, all the
routing information is written in the FortiGate session table. However, after any change to the routing table,
the route information is flushed from the affected entries in the session table. So, additionally routing table
lookups are done after to repopulate the session table with the new routing information.
394
DO NOT REPRINT
FORTINET
Operation Modes
How does FortiGate decide routes? FortiGate has multiple routing modules. This diagram shows their
logic.
First FortiGate searches its policy routes. You can view them with the command 'diagnose firewall
proute list'. If there is a match in a policy route and the action is Forward Traffic, FortiGate routes the
packet accordingly. If the action is Stop Policy Routing, FortiGate goes to the next table, which is the route
cache. You can view that content with the CLI command 'diagnose ip rtcache list'.
Finally, FortiGate searches the forwarding information base (FIB). The FIB is generated by the routing
process, and is the table used for packet forwarding. Think of the routing tables purpose as for
management, while the FIB is for forwarding. This separation becomes clearer in FortiGate active-active
HA. In an HA cluster, both route management and forwarding tables exist on the master FortiGate. But on
the slave FortiGate, only the forwarding table (FIB) exists.
If theres no match in any of those tables, FortiGate drops the packet because it is unroutable.
395
DO NOT REPRINT
FORTINET
Operation Modes
When there is more than one route to a destination, this is process for selecting which route to use.
First, FortiGate uses the most specific route, which is the one with the longest net mask (smallest subnet).
If there are two or more routes with the same longest net mask, the unit selects the ones with the lowest
distance.
After that, the lowest metric is used as tie breaker for dynamic routes. In the case of static routes, the
priority is used instead.
If there multiples routes with the same net mask, distance, metric, and priority, FortiGate shares the traffic
among all of them. This is what is called equal cost multipath (ECMP), which is supported for static, BGP
and OSPF routes.
396
DO NOT REPRINT
FORTINET
Operation Modes
The FortiGate adds a static route to the routing table only if all these requirements are met:
1- The outgoing interface is up
2- There is no other route with the same net mask and lower distance
3- The link health monitor (if configured) is up
4- The next-hop IP address is in the same subnet as one of the IP addresses assigned to the outgoing
interface
397
DO NOT REPRINT
FORTINET
Operation Modes
Let's quickly review an important routing concept that was introduced in NSE4: reverse path forwarding
check.
It protects against IP spoofing attacks and routing loops by checking the route to the source IP address.
This check is done only on packets on the original direction (not on reply packets) and only when the
session was not created yet, or when the routing table has changed.
If the check fails, the packet is dropped and the debug flow shows the error:
reverse path check fail, drop
398
DO NOT REPRINT
FORTINET
Operation Modes
399
DO NOT REPRINT
FORTINET
Operation Modes
400
DO NOT REPRINT
FORTINET
Operation Modes
UTM traffic inspection requires symmetric routing. Traffic both ways must follow the same path. So,
FortiGate routes traffic symmetrically. This means that, under some network topologies, FortiGate might
not route the return traffic through the best path, but through the path that the originating traffic is using.
For that purpose the unit remembers the interface to source and uses that interface to route the return
packets even when a better route using a different interface exists. Lets see an example in the next slides.
401
DO NOT REPRINT
FORTINET
Operation Modes
402
DO NOT REPRINT
FORTINET
Operation Modes
403
DO NOT REPRINT
FORTINET
Operation Modes
Now, lets see how the FortiGate routes the return packet.
When the FortiGate receives the ICMP echo reply, as there is already a session and a route cache
created, it uses the interface to source. So, in this case the unit routes the packet through out port2
towards the local router, even when there is a better route to the destination 10.1.0.1. The FortiGate
routing table shows port1 as the best route to 10.1.0.1 (locally connected), but it still uses port2. The
objective, as explained, is to keep the traffic flows symmetric.
With the first ICMP echo reply, a second entry is added to the route cache, this time for the return traffic.
404
DO NOT REPRINT
FORTINET
Operation Modes
Additionally, and as explained earlier, the unit does a second routing lookup, this time to find the next-hop
(or gateway) to source. That IP address is added to the session (where 0.0.0.0 was before).
405
DO NOT REPRINT
FORTINET
Operation Modes
Now, this raises one question: What happens is the traffic is originated from the server side instead? Lets
say that the ping is sent from the remote server to the local workstation.
In this case, when the ICMP echo request arrives to the FortiGate, there is not session yet. So, the unit
uses the best route to 10.1.0.1, which is through port1.
These examples show how a FortiGate might, on some network topologies, route packets to the same
destination differently depending on who initiated the session.
406
DO NOT REPRINT
FORTINET
Operation Modes
Something interesting in this last example is the reply traffic. As the local workstation default gateway is
10.1.0.254, the ICMP echo reply goes to the local router first. Then the packet arrives to the FortiGate
port2. The result is asymmetric routing as the return traffic is following a different path than the originating
traffic. The return packet is arriving to port2 instead of port1 (where the originating traffic was sent). In
these cases, the FortiGate accepts this asymmetry, no packets are dropped and UTM inspection is not
affected.
407
DO NOT REPRINT
FORTINET
Operation Modes
408
DO NOT REPRINT
FORTINET
Operation Modes
However, there is an important exception to this rule. After a routing change, sessions with source NAT
keep using the same outbound interface, as long as the old route is still valid. Let's see an example. In this
case, the FortiGate is connected to two different ISPs. There is client with the IP address 10.1.0.1/24
connected behind the FortiGate. The FortiGate is doing source NAT of the client traffic to a public IP
address that depend on which ISP is using. The unit routing table contains two default routes with the
same distance, but different priorities, one default route for each ISP. The route with the lowest priority
(port1) is the primary one. When both ISP connections are up, this is route selected by the FortiGate for
Internet traffic. So, all sessions to the Internet are created using port1 as the outbound interface.
409
DO NOT REPRINT
FORTINET
Operation Modes
If the administrator increases the priority for port1 to a value higher than the priority in port2, this is what
happens:
1- All new sessions start using port2 as it is now the one with the lowest priority
2- However, all the existing sessions keep using port1. The default route through port1, although it is not
the best one now, it is still valid and the unit is doing source NAT. Those sessions will keep the old route
until they expire
If the unit were not doing source NAT, all the existing sessions would switch to port2 after the change.
410
DO NOT REPRINT
FORTINET
Operation Modes
Most of the debug commands for routing are under the 'get router info' tree. Let's see some of them in
the next slides.
411
DO NOT REPRINT
FORTINET
Operation Modes
412
DO NOT REPRINT
FORTINET
Operation Modes
If you want to display both the active and the inactive routes, use instead the command:
get router info routing-table database
In this example the command shows one inactive route at the end. It is inactive because it has a higher
distance than the one above.
413
DO NOT REPRINT
FORTINET
Operation Modes
This low-level command shows the actual forward information database (FIB), which is the routing
information that the kernel uses to route traffic. All the active routes in the routing table must be also
present in the FIB. Additionally, the FIB may contain routes that are not in the routing table and were
automatically added by the FortiGate.
414
DO NOT REPRINT
FORTINET
Operation Modes
The route cache contains recently used routing entries in a fast-to-search table. It is consulted before the
routing table to speeds up the routing lookup process.
415
DO NOT REPRINT
FORTINET
Operation Modes
This table contains all the IP addresses used by the FortiGate and assigned to FortiGate interfaces.
416
DO NOT REPRINT
FORTINET
Operation Modes
The command 'get router info protocols' gives a quick look at the configuration of all the dynamic routing
protocols running in the unit.
417
DO NOT REPRINT
FORTINET
Operation Modes
418
DO NOT REPRINT
FORTINET
Operation Modes
In transparent mode, each VDOM forms a separate broadcast domain. Interfaces, by default, dont. Until
you change the initial VDOM configuration, all interfaces, regardless of their VLAN ID, are part of the same
broadcast domain.
The interface command 'forward-domain' is used to segment the VDOM into multiple broadcast domains.
Interfaces with different forward domain IDs belong to different broadcast domains. Interfaces with the
same ID belong to the same broadcast domain.
419
DO NOT REPRINT
FORTINET
Operation Modes
420
DO NOT REPRINT
FORTINET
Operation Modes
421
DO NOT REPRINT
FORTINET
Operation Modes
This debug command lists the MAC address forwarding table in a VDOM operating in transparent mode.
422
DO NOT REPRINT
FORTINET
Operation Modes
423
DO NOT REPRINT
FORTINET
External BGP
424
DO NOT REPRINT
FORTINET
External BGP
The lesson has three main objectives. First, we will do a quick review of BGP concepts and configuration
on FortiGate devices. Second, you will learn to monitor and check the status of a BGP communication.
Finally, the lesson covers BGP troubleshooting.
BGP is a deep topic more extensive than what is covered here. This lesson shows only the basics about
the protocol and the FortiGate to troubleshoot of the most common external BGP issues.
425
DO NOT REPRINT
FORTINET
External BGP
426
DO NOT REPRINT
FORTINET
External BGP
An autonomous system (AS) is a set of routers and networks under the same administration. Each AS is
identified by a unique number and usually runs an interior gateway protocol, such as OSPF or RIP.
427
DO NOT REPRINT
FORTINET
External BGP
An external routing protocol (EGP) exchanges routing information between autonomous systems. BGP4,
which runs in the Internet, is the dominant EGP protocol today. BGP is typically used when strict control is
required over a large number of routes.
Two BGP routers exchange AS path information for destination prefixes or subnets. When two routers
start a BGP communication, the whole BGP routing table is interchanged. After that, only network updates
are sent.
428
DO NOT REPRINT
FORTINET
External BGP
A BGP speaker or peer is a router that sends and receives BGP routing information. The connection
between two BGP peers is called a BGP session.
429
DO NOT REPRINT
FORTINET
External BGP
430
DO NOT REPRINT
FORTINET
External BGP
A BGP router stores the routing information into three logical tables. The RIB-In table contains all the
routing information received from other BGP routers before any filtering. The local RIB table contains that
same information after the filtering. The RIB-Out table contains the BGP routing information selected to
advertise to other BGP routers.
431
DO NOT REPRINT
FORTINET
External BGP
This flow graph summarizes the BGP process. BGP routes received from other routers are stored in the
RIB-In table. A filter is applied and the resulting routes are stored in the local RIB table. Then routes
redistributed from the routing table are added, and another filter (outbound) is applied. The resulting routes
are the ones being advertised by the BGP router.
432
DO NOT REPRINT
FORTINET
External BGP
BGP routes traffic based on AS paths. Each AS patch includes attributes, which some are used to select
the "best" route to each destination. One of the attributes is the AS list, which contains the autonomous
systems through which the traffic must pass to reach the destination.
433
DO NOT REPRINT
FORTINET
External BGP
434
DO NOT REPRINT
FORTINET
External BGP
435
DO NOT REPRINT
FORTINET
External BGP
Some of the attributes are used during the routing selection process. If all those attributes for multiple
routes to the same destination match, and if ECMP is enabled, FortiGate shares the traffic among up to 10
BGP routes. If ECMP is not enabled, the FortiGate uses the route to the router with the lowest BGP router
ID.
436
DO NOT REPRINT
FORTINET
External BGP
There are three important considerations related with the way of how BGP is implemented in FortiGate
devices.
First, there are not hard-code limits. The limitations regarding the number of neighbors, routes and policies
depend exclusively on the system memory available.
Second, FortiGate by default does not announce any prefix. An administrator must enable redistribution or
manually indicate the prefixes to advertise.
Third, all prefixes received, by default, are accepted. Administrators can optionally filter out or modify some
prefixes.
437
DO NOT REPRINT
FORTINET
External BGP
So, FortiGate BGP, by default, does not advertise any prefix. One way to configure FortiGate to do it is by
using the 'redistribution' commands. Connected, static and routes learned from other routing protocols
can be redistributed into BGP. You can optionally add route maps to filter the prefixes or modify some of
their BGP attributes.
438
DO NOT REPRINT
FORTINET
External BGP
The other way to indicate the prefixes to advertise is by using the 'network' command. It is important to
know that, by default, an exact match of the prefix in the 'network' command must be active in the routing
table. If there isn't any active route whose destination subnet matches the prefix, FortiGate does not
advertise it. This default behavior can be changed by disabling the setting 'network-import-check.' Once
disable it, all prefixes in the BGP network table are always advertised, regardless of the actives routes
present in the routing table.
439
DO NOT REPRINT
FORTINET
External BGP
This is a sample of a basic FortiGate BGP configuration. In this case the administrator has configured the
local AS (65075) and one neighbor with the IP address 10.127.0.117 and AS 65117. The BGP ID of the
local router is 0.0.0.75, and the redistribution of static routes into BGP is enabled.
440
DO NOT REPRINT
FORTINET
External BGP
Also by default, FortiGate accepts BGP sessions only from BGP neighbours that are "locally" connected.
In other words, neighbours that are in the same subnet (and broadcast domain) than the local FortiGate.
There are at least two cases where this requirement creates problems:
First, when there are routers between the two BGP peers, in other words, when both peers are not in the
same subnet.
Second, when a loopback interface is used as the common BGP neighbour IP address for all the BGP
peers. For example, a FortiGate running BGP with two or more ISPs, and all the ISPs using the FortiGate
loopback IP address as their remote neighbor.
In those cases, administrators must enable 'ebgp-enforce-multihop'.
441
DO NOT REPRINT
FORTINET
External BGP
442
DO NOT REPRINT
FORTINET
External BGP
This graph shows the BGP neighbor states and how they change:
Idle: Initial state.
Connect: Waiting for a successful 3-way TCP connection.
Active: Unable to establish the TCP session.
OpenSent: Waiting for an OPEN message from the peer.
OpenConfirm: Waiting from the keepalive messages from the peer.
Established: Peers have successfully interchange OPEN and keepalive messages.
443
DO NOT REPRINT
FORTINET
External BGP
Most of the BGP diagnostic commands are under the 'get router info bgp' tree.
444
DO NOT REPRINT
FORTINET
External BGP
445
DO NOT REPRINT
FORTINET
External BGP
The command 'get router info bgp neighbors' provides detailed information about each BGP neighbor. It
includes packet counters, and number of prefixes announced and accepted.
446
DO NOT REPRINT
FORTINET
External BGP
It also shows the number of times that the session has dropped and last time it was reset.
447
DO NOT REPRINT
FORTINET
External BGP
This command provides details about the prefixes being advertised by the local router.
448
DO NOT REPRINT
FORTINET
External BGP
449
DO NOT REPRINT
FORTINET
External BGP
These are the commands for enabling and disabling the BGP real time debug.
450
DO NOT REPRINT
FORTINET
External BGP
The next two slides show a sample of the real time debug output during the successful establishment of a
BGP session. It shows when the session goes to OpenSent state.
451
DO NOT REPRINT
FORTINET
External BGP
452
DO NOT REPRINT
FORTINET
External BGP
The command 'execute router clear bgp' is used to restart a BGP session between two peers. It is also
used to execute a BGP soft reset, which forces both peers to interchange their complete BGP routing
tables.
453
DO NOT REPRINT
FORTINET
External BGP
Follow these steps if you are troubleshooting a BGP problem between two peers:
- Check that the local router can reach the remote peer
- If the remote peer is not on a connected network, enable EBGP multihop
- Check the TCP session
- Check the BGP session
- If the BGP session is established, check the prefixes received and advertised by each peer.
454
DO NOT REPRINT
FORTINET
External BGP
455
DO NOT REPRINT
FORTINET
OSPF
456
DO NOT REPRINT
FORTINET
OSPF
This lesson reviews some OSPF concepts, such as link state advertisements, cost, adjacencies, and
areas. The lesson also covers OSPF troubleshooting and basic OSPF configuration.
Similarly to BGP, OSPF is a deep topic more extensive than what is covered in this lesson. This lesson
shows only the basics about the protocol and the FortiGate to troubleshoot of the most common OSPF
issues.
457
DO NOT REPRINT
FORTINET
OSPF
458
DO NOT REPRINT
FORTINET
OSPF
In a link state protocol, such as OSPF, every router has a complete view of the network topology. Some of
the advantages of OSPF are scalability and fast convergence. Every 30 minutes routers re-advertise their
OSPF information. Between those 30-minute intervals, only updates are sent if a topology change is
detected. So, it is a relatively quiet protocol as long as the network topology is stable.
OSPF in large networks, although, may require good planning and be difficult to troubleshoot.
459
DO NOT REPRINT
FORTINET
OSPF
460
DO NOT REPRINT
FORTINET
OSPF
The topology information interchanged by OSPF peers is contained in link state advertisements (LSAs). A
router's LSBD is populated with the information from the local LSAs and all the LSAs received from other
routers.
A router sends link state update packets, which contain one or more LSAs.
461
DO NOT REPRINT
FORTINET
OSPF
If there are multiple OSPF routes to the same destination subnet, OSPF selects the route with the lowest
cost. Each router interface is associated with an interface cost, which is usually related with how fast or
preferable that interface is. An OSPF route cost is the sum of all interfaces' costs to the final destination.
462
DO NOT REPRINT
FORTINET
OSPF
The following two slides explain briefly how an OSPF router builds its OSPF tree. The initial information for
each router is the locally connected networks, together with the OSPF cost for each interface. For
example, the router R2 has three locally connected subnets: subnet Net 1 with a cost of 2, subnet Net 2
with a cost of 3, and subnet Net 3 with a cost of 1. Router R1 has only one subnet connected: Net 1 with a
cost of 2, and so on.
Each router starts advertising its locally connected subnets by sending LSAs.
463
DO NOT REPRINT
FORTINET
OSPF
Then, OSPF routers use the Dijkstras algorithm to determine the best route to each destination. The best
routes can be represented as a tree with the local router at the root. The Dijkstras algorithm is a recursive
process that is repeated multiple times until the best routes are found. For example, this slide shows the
OSPF tree for router R2. It indicates that the best route to Net 5 and Net 4 is through R3; and that Net 1,
Net 2 and Net 3 are locally connected.
464
DO NOT REPRINT
FORTINET
OSPF
An OSPF network can be segmented into areas. Each area is identified by a unique number, which can be
represented either in decimal or IP address format.
465
DO NOT REPRINT
FORTINET
OSPF
Each area has its own separated LSDB. All routers in the same area maintain an identical copy of the
area's LSDB. As we will see later, a router can belong to more than one area. In those cases, the router
maintains multiple LSDBs, one LSDB for each area connected to it.
Segmenting big OSPF networks into areas reduces the sizes of the LSDB tables. Additionally, a topologies
change does not impact the whole network, but only the area where the change happens.
Using OSPF areas, although, requires good planning and may complicate the troubleshooting process.
466
DO NOT REPRINT
FORTINET
OSPF
Any OSPF network must have at least one area, the backbone area. The backbone is the core of the
network with all the other areas connected to it in a hub-and-spoke topology.
467
DO NOT REPRINT
FORTINET
OSPF
An internal OSPF router has all its interfaces connected to the same area. So, it maintains only one LSDB.
On the other hand, an area border router is connected to multiple areas, so it keeps multiple LSDBs.
A backbone router has at least one interface connected to the backbone area.
An autonomous system boundary router (ASBR) redistributes non-OSPF routes into the OSPF network.
468
DO NOT REPRINT
FORTINET
OSPF
469
DO NOT REPRINT
FORTINET
OSPF
An OSPF session between two OSPF peers is called an adjacency. This slide shows the initial
interchange between two peers that are forming an adjacency. Any new adjacency goes through different
states: Init, 2-way, ExStart, Exchange, Loading and Full. The Full state indicates that the adjacency has
successfully formed, and both routers have identical copies of the LSDB.
470
DO NOT REPRINT
FORTINET
OSPF
There are the requirements for two peers to form an OSPF adjacency. If any of the requirements is not
met, the adjacency fails and will not reach the Full state.
471
DO NOT REPRINT
FORTINET
OSPF
There are two types of OSPF networks. Point-to-point networks contain only two peers, one at each end of
a point-to-point link. Multi-access networks support two or more peers.
There are two sub-types of multi-access networks. Broadcast networks support multicasting. An example
of a broadcast multi-access network is an Ethernet network. Non-broadcast multi-access networks
(NBMA) do not support multicasting. An example of a NBMA network is Frame Relay.
472
DO NOT REPRINT
FORTINET
OSPF
In any multi-access network there are one designated router (DR) and one backup designated router
(BDR). The router with the highest priority is elected as the DR. If two or more routers are tied with the
highest priority, the router with the highest OSPF ID is elected.
The BDR monitors the DR status. If the DR fails, the BDR takes the DR role.
Other routers form adjacencies only with the DR and the BDR. The DR forwards the link state information
from one router to another. This simplifies the amount of adjacencies required in multi-access networks.
473
DO NOT REPRINT
FORTINET
OSPF
These are the multicast addresses used by OSPF in broadcast multi-access, and point-to-point networks.
Keep in mind that OSPF also uses unicast addresses, for LSA retransmissions and database description
packets.
474
DO NOT REPRINT
FORTINET
OSPF
There are 11 LSA types. This lesson covers the five more commonly used:
-Type 1 describes all the links connected to a router
-Type 2 describes all the routers (if more than one) in a multi-access network
-Type 3 describes the networks within an area (only generated by a ABR)
-Type 4 describes the path to reach an ASBR
-Type 5 describes the external destinations originated by an ASBR
We will see examples of each of these five types in the next slides.
475
DO NOT REPRINT
FORTINET
OSPF
Type 1 describes the networks connected to a router. They are advertised by all the routers in an area.
Type-1 LSAs are not advertised outside the area where they originate.
476
DO NOT REPRINT
FORTINET
OSPF
Type-2 LSAs are advertised only by DRs. In this example, the area has two multi-access networks, each
of them with one DR. The two DRs advertise type-2 LSAs, which contain information about the other
routers connected to their multi-access networks.
477
DO NOT REPRINT
FORTINET
OSPF
Type-3 LSAs contain summarized link state information. They are advertised only by area border routers
(ABRs). In this example, the ABR in the left sends type-3 LSAs to the area 1. They contain link state
information for the summarized subnets in areas 0 and 2. This same ABR also sends type-3 LSAs to the
backbone area with a summary of the subnets in area 1.
Something similar happens with the ABR in the right. It sends type-3 LSAs to the area 2. They contain
links state information for the summarized subnets in areas 0 and 1. This same ABR also sends type-3
LSAs to the backbone area with a summary of the subnets in area 2.
478
DO NOT REPRINT
FORTINET
OSPF
An ASBR advertises itself by sending type-1 LSAs. These LSAs have the E-bit on in the OSPF header. As
any other type 1, the LSAs with the E-bit are confined to the area where they originate. However, ABRs in
the same area send a type-4 LSA to the other areas with information about how to reach the ASBR. In this
example, an ASBR that is redistributing RIP routes into OSPF announces itself by sending type-1 LSAs to
the backbone area. The ABR receives that LSA and sends a type-4 LSA to the area 1.
479
DO NOT REPRINT
FORTINET
OSPF
The last type of LSA covered in this lesson is type 5. They are sent only by the ASBRs and are not
confined to one area. Indeed, they reach all the standard areas. They contain link state information for
routes redistributed to OSPF (also called external routes).
Note: All the area examples in this lesson are standard areas. There are also stub and NSSA areas, which
are not covered in this lesson. Type-5 LSAs are not advertised to stub or NSAA areas.
480
DO NOT REPRINT
FORTINET
OSPF
Each external route is assigned a metric. There are two types of external-route metrics. A type-1 metric is
the sum of the external cost plus the internal cost to reach the ASBR. A type-2 metric is only the external
cost (the internal cost is not considered). If there are two external routes to the same destination, one type
1 and one type 2, an OSPF router selects the type 1 over the type 2.
481
DO NOT REPRINT
FORTINET
OSPF
This is a basic FortiGate OSPF configuration. It has the list of areas, the list of OSPF networks, and the
OSPF router ID.
482
DO NOT REPRINT
FORTINET
OSPF
483
DO NOT REPRINT
FORTINET
OSPF
484
DO NOT REPRINT
FORTINET
OSPF
'get router info ospf status' shows general OSPF information and some statistics about OSPF.
485
DO NOT REPRINT
FORTINET
OSPF
The output of 'get router info ospf status' also shows information about each area the router belongs to.
486
DO NOT REPRINT
FORTINET
OSPF
487
DO NOT REPRINT
FORTINET
OSPF
This command shows a summary of the status of all the OSPF neighbors. For each neighbor, it displays
the adjacency state; and if it is a DR, a BDR, or none of them (DROther). A dash is displayed after the
state if the neighbor is in a point-to-point network.
488
DO NOT REPRINT
FORTINET
OSPF
The command 'get router info ospf database brief' provides a summary of all the LSDB entries in the
FortiGate, order by LSA types. It shows first the LSAs type 1 (router link states), then type 2 (net link
states).
489
DO NOT REPRINT
FORTINET
OSPF
After that, the command shows the LSAs type 3 (summary link state), type 4 (ASBR-summary link states)
and type 5 (AS external link states).
490
DO NOT REPRINT
FORTINET
OSPF
The command 'get router info ospf database self-originate ' lists the LSAs originated in the local
FortiGate.
491
DO NOT REPRINT
FORTINET
OSPF
To get the details about each LSA, use the command 'get router info ospf database router lsa'.
492
DO NOT REPRINT
FORTINET
OSPF
This is a sample of more output from the command 'get router info ospf database router lsa'.
493
DO NOT REPRINT
FORTINET
OSPF
The OSPF real time debug displays information about adjacency establishments and OSPF errors. It also
shows information about network topology changes.
494
DO NOT REPRINT
FORTINET
OSPF
495
DO NOT REPRINT
FORTINET
High Availability
496
DO NOT REPRINT
FORTINET
High Availability
The lesson reviews how virtual MAC addresses are assigned in a HA cluster. It also covers HA monitoring
and troubleshooting.
497
DO NOT REPRINT
FORTINET
High Availability
We will start with a quick review of the HA operation material covered in NSE4. There are two HA modes:
active-passive and active-active.
Lets examine first the active-passive mode. In any of the two HA operation modes, the configurations of
the secondary FortiGates are synchronized with the configuration in the primary device.
In the case of the active-passive mode, the primary FortiGate is the only device that actively processes
traffic. The secondary FortiGates remain in passive mode monitoring the status of the primary unit.
If a problem is detected in the primary FortiGate, one of the secondary devices will take over the primary
role. This event is what we call HA failover.
Session synchronization is an optional setting (which is disabled by default) that enables seamless failover
for some traffic. The information of some sessions is synchronized, so when the primary fails, the new
primary can take over those sessions where they were left and keep them open. Traffic might be
interrupted for a few seconds, but the network applications will not need to reconnect the sessions again.
498
DO NOT REPRINT
FORTINET
High Availability
Like with active-passive HA, in active-active, all FortiGates configurations are synchronized. Also, if a
problem is detected with the primary device, one of the secondary units will take over the primary role.
However, one of the main differences with active-passive is that in active-active all of the FortiGates are
processing traffic. Indeed, one of the tasks of a primary FortiGate in active-active is to balance some of the
sessions among all the secondary devices.
499
DO NOT REPRINT
FORTINET
High Availability
500
DO NOT REPRINT
FORTINET
High Availability
501
DO NOT REPRINT
FORTINET
High Availability
After fail over, the new primary also broadcasts gratuitous ARP packets, notifying the network that each
virtual MAC address is now reachable through a different switch port.
In most of the networks that is enough for the switches to update their MAC forwarding tables with the new
information. However, some high-end switches might not clear properly their MAC tables after a fail-over.
So, they keep sending packets to the former primary even after receiving the gratuitous ARPs. In those
cases, enable the command 'link-failed-signal' to force the former primary to shut down all its nonheartbeat interfaces for 1 second when the failover happens. This simulates a link failure that clears the
related entries from the switches' MAC tables.
502
DO NOT REPRINT
FORTINET
High Availability
The HA virtual MAC addresses assigned to each interface are determined by the HA group ID, the virtual
cluster ID and the interface index. So, if you have two or more HA clusters in the same broadcast domain,
and using the same HA group ID, you might get MAC address conflicts. For those cases, it is strongly
recommended to assign different HA group IDs to each cluster.
503
DO NOT REPRINT
FORTINET
High Availability
The same command we used to get interface traffic statistics can be used to display the HA virtual MAC
address assigned to an interface.
504
DO NOT REPRINT
FORTINET
High Availability
505
DO NOT REPRINT
FORTINET
High Availability
506
DO NOT REPRINT
FORTINET
High Availability
507
DO NOT REPRINT
FORTINET
High Availability
If you connect to a secondary's console port while it is joining a HA cluster, these are the messages you
should see. The secondary tries to synchronize first what are called the "external files". They include the
FortiGuard databases and digital certificates. After that, the secondary synchronizes the configuration. The
last message indicates that the secondary has successfully joined the cluster.
508
DO NOT REPRINT
FORTINET
High Availability
If the HA cluster has formed successfully, the GUI displays all the FortiGates, together with their
hostnames and serial numbers.
509
DO NOT REPRINT
FORTINET
High Availability
When troubleshooting any problem in a HA cluster, it is useful to know that you can connect to the CLI of
any secondary from the primary CLI. You have to use the command 'execute ha manage' with the
secondary HA index for that purpose. To get the list of secondary FortiGates with their HA indexes, use the
question mark at the end of that same command.
510
DO NOT REPRINT
FORTINET
High Availability
511
DO NOT REPRINT
FORTINET
High Availability
As explained in NSE4, the HA uptime is one of variables used to elect the primary unit. Depending on
other variables and configuration, at one point the units might compare their system uptimes to elect the
primary. If that happens and if there is one unit whose system uptime is 5 minutes more than the system
uptimes of all the other units, it is elected primary.
This command can be used to compare the system uptimes of all the units in a cluster.
512
DO NOT REPRINT
FORTINET
High Availability
A good indication of the health of a HA cluster is the status of the configuration synchronization. To check
that all the secondary configurations are synchronized with the primary configuration, you can execute the
command 'diagnose sys ha showcsum' in all the HA units. If a secondary FortiGate displays exactly the
same sequence of numbers than the primary, its configuration it's well synchronized. Also, in each of the
devices, the debugzone and the checksum zone must display the same sequence of numbers. We will
suggest later some troubleshooting tips if that is not case.
513
DO NOT REPRINT
FORTINET
High Availability
Instead of running the previous command in each of the cluster units, you can alternatively run this other
command only on the primary. It shows the checksum for all the cluster members. This command is easier
to use. However, if there are communication problems between one of the secondary units and the
primary, you might need to use the previous command instead.
514
DO NOT REPRINT
FORTINET
High Availability
FortiGate HA uses FGCP, the FortiGate clustering protocol, for HA-related communications. FGCP travels
among the clustered FortiGate units over the links that you have designated as the heartbeats.
The FGCP traffic uses a different Ethernet type than the IP protocol. It actually uses three different
Ethernet types depending on the operation mode (transparent or NAT/route).
515
DO NOT REPRINT
FORTINET
High Availability
HA session synchronization, as explained, is disabled by default. If it is enabled, you can check in the
primary's session table which sessions have been synchronized to the secondary units. They are the ones
with the synced flag. Additionally and for the case of all sessions, the ha_id field shows the HA member ID
of the unit that is processing the traffic.
516
DO NOT REPRINT
FORTINET
High Availability
If a failover happens, the best tool to get information about it is the FortiGate logs. In case the failover
happened because the primary unit failed, the secondary unit's logs should show these log entries.
517
DO NOT REPRINT
FORTINET
High Availability
If a new primary was elected because one or more monitored interfaces failed, the former primary displays
logs similar to these ones. In this case the primary unit is reporting a problem with the monitored interface
port1.
518
DO NOT REPRINT
FORTINET
High Availability
519
DO NOT REPRINT
FORTINET
High Availability
Traffic from session synchronization is bandwidth intensive. If the session creation rate is high, session
synchronization traffic might interfere with the heartbeat traffic, creating delays in the heartbeat replies.
There two configuration changes that might help in those cases:
1- Use an interface different than the heartbeat interface for session synchronization
2- Delay the synchronization of new sessions by 30 seconds, so short-lived sessions are not synchronized
High CPU issues could also create HA heartbeat problems. In those cases, troubleshoot and fix the high
CPU problem first before checking the HA status.
520
DO NOT REPRINT
FORTINET
High Availability
521