IPSec

IPSec which works at the network layer is a framework consisting of

protocols and algorithms for protecting data through an un-trusted network
such as the internet. IPSec provides data security in various ways such as
encrypting and authenticating data, protection against masquerading and
manipulation. IPSec is a complex framework consisting of many settings,
which is why it provides a powerful and flexible set of security features that
can be used.
IPSec is a collection of different protocols or algorithms. IPSec traffic can be
configured using over 30 different settings. IPSec is used to secure traffic
from site to site or site to a mobile user. As the world is constantly changing
and growing with technology, IPSec suits this as its a framework, which
allows you add new and better algorithms coming out.
When two IPSec gateways want to make a VPN connection between them,
they negotiate on various settings and parameters and must make an
agreement on the parameters used. For example what type of authentication
and encryption will be used within the VPN tunnel. This is generally called
VPN negotiation.
IPSec does not use RSA for data encryption. It uses DES, 3DES, or AES. IPSec
uses RSA for IKE internet key exchange for during peer authentication phase,
to ensure the other side is authentic and who they say they are.

4 key functions or services of IPSec are as follows;

1 Confidentiality Encrypting data, and scrambling.
2 Data Integrity data has not been changed.
3 Data Authentication authenticating receiver. Sender receiver is who they
say they are.
4 Anti-replay each packet is unique, has not been duplicated or intercepted.
Security Associations (SA)
A collection of parameters required to establish a secure
session

 Uniquely identified by three parameters consisting of

Security Parameter Index (SPI)
IP destination address
Security protocol (AH or ESP) identifier
An SA is unidirectional
Two SAs required for a bidirectional communication
A single SA can be used for AH or ESP, but not both
must create two (or more) SAs for each direction if using both AH and
ESP
ISAKMP
Internet Security Association and Key Management
Protocol
Defined by RFC 2408
Used for establishing Security Associations (SA) and
cryptographic keys
Only provides the framework for authentication and key
exchange, but key exchange independent
Key exchange protocols
Internet Key Exchange (IKE) and Kerberized Internet Negotiation of
Keys (KINK)
Internet Key Exchange (IKE)
An IPsec component used for performing mutual
authentication and establishing and maintaining Security
Associations. (RFC 5996)
Typically used for establishing IPsec sessions

 A key exchange mechanism

Five variations of an IKE negotiation:
Two modes (aggressive and main modes)
Three authentication methods (pre-shared, public key encryption,
and public key signature)
Uses UDP port 500
IKE Modes
Mode Description
Main mode Three exchanges of information between IPsec peers.
Initiator sends one or more proposals to the other peer
(responder)
Responder selects a proposal
Aggressive Mode Achieves same result as main mode using only 3 packets
First packet sent by initiator containing all info to establish
SA
Second packet by responder with all security parameters
selected
Third packet finalizes authentication of the ISAKMP
session
Quick Mode Negotiates the parameters for the IPsec session.
Entire negotiation occurs within the protection of ISAKMP
Session

Internet Key Exchange (IKE)

Phase I

 Establish a secure channel (ISAKMP SA)

Using either main mode or aggressive mode
Authenticate computer identity using certificates or pre-shared secret
Phase II
Establishes a secure channel between computers intended for the
transmission of data (IPsec SA)
Using quick mode

5 phases of IPSec
1 define interesting traffic
2 IKE phase 1 key exchange phase
3 IKE phase 2 IPSec policy and transform sets are processed
4 Transfer data After the tunnels are established you transfer the data.
5 Tear down the tunnel

IPSec uses two different protocols to encapsulate the data over a VPN tunnel:
Encapsulation Security Payload (ESP): IP Protocol 50
Authentication Header (AH): IP Protocol 51
ESP is more secure as it provides data encryption. AH just provides
authentication.
How IPsec works

An IPsec tunnel establishment process can be broken down into five main
steps: tunnel initiation, IKE Phase 1, IKE Phase 2, data transfer and tunnel
termination.
Tunnel initiation: IPsec tunnel initiation can be triggered manually or
automatically when network traffic is flagged for protection according to the

IPsec security policy configured in the IPsec peers. In both cases, the Internet
Key Exchange Protocol (IKE) process starts.
IKE Phase 1, IKE Phase 2: IKE offers a means to automatically negotiate
security parameters and derive suitable keying material. IKE also manages
the process of re-creating, or refreshing, frequently keys to ensure data
confidentiality between peers. The basic operation of IKE can be broken
down into two phases: IKE phase 1 and IKE phase 2.
IKE phase 1: This phase is used to negotiate the parameters and key
material required to establish an ISAKMP Security Association (ISAKMP SA).
The ISAKMP SA is then used to protect future IKE exchanges and to set up a
secure channel for negotiating IPsec SAs in IKE phase 2.
IKE Phase 2: This phase is used to negotiate the parameters and key material
required to establish two unidirectional IPsec SAs for incoming and outgoing
traffic. The IPSEC SAs are then used to protect network traffic during Data
transfer phase.
Data transfer: Incoming and outgoing network traffic is encapsulated
according to the bundle of algorithms and parameters provided by their
respective negotiated IPsec SA to provide confidentiality and authenticity
(ESP protocol) or authenticity only (AH protocol).
Tunnel termination: A tunnel is closed when its IPsec SAs terminate through
deletion or by timing out. An IPsec SA can time out when a specified number
of seconds have elapsed or when a specified number of bytes have passed
through the tunnel.
IKE Phase 1

IKE phase 1s purpose is to establish a secure authenticated communication

channel by using the DiffieHellman key exchange algorithm to generate a
shared secret key to encrypt further IKE communications. This negotiation
results in one single bi-directional ISAKMP SA. The authentication can be
performed using either pre-shared key (shared secret), digital signatures, or
public key encryption as described by RFC 2409.
IKE Phase 1 operates in either Main Mode or Aggressive Mode. Main Mode
protects the identity of the peers but Aggressive Mode does not.
Main Mode has three two-way exchanges between the initiator and the
receiver.

First exchange: The algorithms and hashes used to secure the IKE
communications are agreed upon in matching ISAKMP SAs in each peer.
Second exchange: Uses a Diffie-Hellman exchange to generate shared secret
keying material used to generate shared secret keys and to pass nonces
numbers sent to the other party and then signed and returned to prove their
identity.
Third exchange: Verifies the other sides identity (the identity value is an IP
address, an FQDN, an email address, a DNS or a KEY ID form in encrypted
form). The main outcome of main mode is matching ISAKMP SAs between
peers to provide a protected pipe for subsequent protected ISAKMP
exchanges between the IKE peers. The ISAKMP SA specifies values for the
IKE exchange: the authentication method used, the encryption and hash
algorithms, the Diffie-Hellman group used, the lifetime of the ISAKMP SA in
seconds or kilobytes, and the shared secret key values for the encryption
algorithms. The ISAKMP SA in each peer is bi-directional.
Aggressive Mode has fewer exchanges with fewer packets. On the first
exchange, almost everything is squeezed into the proposed ISAKMP SA
values: the Diffie-Hellman public key (a nonce that the other party signs) and
an identity packet, which can be used to verify identity via a third party. The
receiver sends everything back that is needed to complete the exchange.
The only thing left is for the initiator to confirm the exchange. The weakness
of using the aggressive mode is that both sides have exchanged information
before theres a secure channel. Therefore, its possible to sniff the wire and
discover who formed the new SA. However, it is faster than main mode.
NAT Traversal also known as UDP encapsulation is a general term for
techniques that establish and maintain Internet protocol connections
traversing network address translation (NAT) gateways and devices. RFC
3947 defines the negotiation during the Internet key exchange (IKE) phase
and RFC 3948 defines the UDP encapsulation.
Dead Peer Detection (DPD) is used to monitor the peer and quickly detect
when it gets unreachable. It works by exchanging probe packets, and if the
peer does not answer for some time, the security associations are killed. DPD
is documented by RFC 3706.
XAUTH, Mode config and Hybrid authentication are optional extensions of the
IKE phase 1 described in RFC drafts.

Extended Authentication (XAUTH) provides additional user authentication by

prompting the user for a username and password.
Mode config is used to deliver parameters such as IP address and DNS
address to the client.
Hybrid authentication makes the IKE phase 1 asymmetric: the VPN IPsec
server authenticates to the IPsec clients by using a certificate, and the client
does not authenticate in IKE phase 1. This extension is usually used with
XAUTH to provide a high security level.
IKE Phase 2

The purpose of IKE phase 2 is to negotiate IPsec SAs to set up the IPsec
tunnel. IKE phase 2 performs the following functions:
Negotiates IPsec SA parameters protected by an existing ISAKMP SA.
Establishes IPsec security associations.
Periodically renegotiates IPsec SAs to ensure security.
Optionally performs an additional Diffie-Hellman exchange.
IKE phase 2 has one mode called quick mode. Quick mode occurs after IKE
has established the secure tunnel in IKE phase 1. It negotiates a shared IPsec
policy, derives shared secret keying material used for the IPsec security
algorithms, and establishes IPsec SAs. Quick mode exchanges nonces that
provide replay protection. The nonces are used to generate new shared
secret key material and prevent replay attacks from generating bogus SAs.
Quick mode is also used to renegotiate a new IPsec SA when the IPsec SA
lifetime expires. Base quick mode is used to refresh the keying material used
to create the shared secret key based on the keying material derived from
the Diffie-Hellman exchange in IKE phase 1.
If perfect forward secrecy (PFS) is specified in the IPsec policy, a new DiffieHellman exchange is performed with each quick mode, providing keying
material that has greater entropy (key material life) and thereby greater
resistance to cryptographic attacks. Each Diffie-Hellman exchange requires
large exponentiations, thereby increasing CPU use and exacting a
performance cost.
When the IPsec SAs terminate, the keys are also discarded. When
subsequent IPsec SAs are needed for a flow, IKE performs a new IKE phase 2

and, if necessary, a new IKE phase 1 negotiation. A successful negotiation

results in new IPsec SAs and new keys. New IPsec SAs can be established
before the existing SAs expire, so that a given flow can continue
uninterrupted.
Data transfer

IPsec protocols are AH (Authentication Header) and ESP (Encapsulating

Security Payloads):
AH is a format protocol defined in RFC 2402 that provides data
authentication, integrity, and non repudiation but does not provide data
confidentiality. This protocol has largely been superseded by ESP.
ESP is a format protocol defined in RFC 2406 that provides data
confidentiality, integrity and data origin authentication, replay attack
protection.
ESP supports the use of symmetric encryption algorithms, including DES,
3DES, and AES, for confidentiality and the use of MD5 HMAC and SHA1 HMAC
for data authentication and integrity.
AH and ESP protocols support two modes of use: Transport and Tunnel.
Transport mode encrypts only the data portion of each packet, but leaves the
header untouched. Tunnel mode is more secure and encrypts both the
header and the payload.
The figures below describe the most common ways to encapsulate original IP
packets:

Tunnel/Transport modes using AH protocol