IPsec (Internet Protocol Security)

Introduction

IPsec (Internet Protocol Security) is a suite of protocols that provides security services for IP network traffic at the network layer (Layer 3) of the OSI model. Unlike application-layer security protocols such as TLS that protect individual application connections, IPsec secures all IP traffic between two endpoints transparently -- applications do not need to be modified or even aware that IPsec is in use.

IPsec provides three fundamental security services:

• Confidentiality: Encryption ensures that captured packets cannot be read by unauthorized parties
• Integrity: Cryptographic hash verification ensures that packets have not been modified in transit
• Authentication: The identity of the communicating parties is verified, preventing impersonation and man-in-the-middle attacks

IPsec is the foundation of most site-to-site VPNs and is also used for remote access VPNs, securing routing protocols (BGP, OSPF), and protecting critical infrastructure communications. It is mandatory in IPv6 (though not always enabled) and widely deployed with IPv4.

"IPsec is the most comprehensive security architecture available for IP communications. It operates at the network layer, providing transparent protection to all applications without requiring any changes to the applications themselves." -- Naganand Doraswamy, IPSec: The New Security Standard for the Internet

History and Standards

IPsec development began in 1992 when the Internet Engineering Task Force (IETF) recognized the need for network-layer security as the internet expanded beyond academic and military use. The original IPsec standards were published in 1995 (RFC 1825-1829), with major revisions in 1998 (RFC 2401-2412) and 2005 (RFC 4301-4309).

The development of IPsec involved significant contributions from the U.S. Naval Research Laboratory (NRL), which built one of the first open-source IPsec implementations for BSD Unix. This implementation influenced many subsequent commercial and open-source IPsec stacks, including those in Linux (XFRM/StrongSwan), FreeBSD, and various router and firewall platforms.

IPsec Architecture

The IPsec architecture (defined in RFC 4301) consists of several interrelated components:

• Security Protocols: AH (Authentication Header) and ESP (Encapsulating Security Payload) -- the protocols that actually protect IP packets
• Security Associations (SA): The set of security parameters (algorithms, keys, lifetimes) agreed upon by two communicating parties for a specific traffic flow
• Security Association Database (SAD): Stores all active Security Associations
• Security Policy Database (SPD): Defines which traffic should be protected by IPsec, which should bypass IPsec, and which should be discarded
• Internet Key Exchange (IKE): The protocol that negotiates Security Associations and manages cryptographic keys

When an IP packet is ready to be sent, the system consults the SPD to determine whether the packet should be protected. If protection is required, the system looks up the appropriate SA in the SAD (or triggers IKE to create one if none exists). The SA specifies which protocol (AH or ESP), algorithm, key, and mode to use. The packet is then processed accordingly and sent to the destination.

AH vs. ESP

Authentication Header (AH)

The Authentication Header (AH) protocol (IP protocol number 51, defined in RFC 4302) provides data integrity and data origin authentication for IP packets. AH computes a cryptographic hash (HMAC) over the IP header (excluding mutable fields like TTL and header checksum), the AH header, and the entire payload. The recipient recalculates the hash and verifies that the packet has not been altered in transit.

AH does NOT provide encryption. The packet payload is transmitted in plaintext. AH only guarantees that the packet is authentic (from the claimed sender) and has not been modified.

AH authenticates the entire IP packet, including the outer IP header (in transport mode). This provides stronger authentication than ESP alone, because it protects against IP address spoofing. However, this same property makes AH incompatible with Network Address Translation (NAT), because NAT modifies the IP header, which invalidates the AH integrity check.

Encapsulating Security Payload (ESP)

The Encapsulating Security Payload (ESP) protocol (IP protocol number 50, defined in RFC 4303) provides confidentiality (encryption), data integrity, and data origin authentication. ESP is by far the more commonly used IPsec protocol because it provides encryption -- which AH does not.

ESP encrypts the packet payload and optionally authenticates it with an HMAC. Unlike AH, ESP does not authenticate the outer IP header, which makes it compatible with NAT when combined with NAT Traversal (NAT-T, RFC 3948), which encapsulates ESP packets in UDP port 4500.

In practice, ESP with authentication is used in the vast majority of IPsec deployments. AH is rarely used because ESP with authentication provides both encryption and integrity/authentication, while AH provides only integrity/authentication without encryption. The primary theoretical advantage of AH -- protecting the outer IP header -- is outweighed by its NAT incompatibility in most real-world environments.

Transport Mode vs. Tunnel Mode

IPsec operates in two modes that determine how packets are encapsulated:

Transport Mode protects the payload of the original IP packet while leaving the original IP header intact. The IPsec header (AH or ESP) is inserted between the IP header and the transport-layer header (TCP/UDP). Transport mode is used for host-to-host communication -- for example, securing traffic between two servers on the same network.

Tunnel Mode encapsulates the entire original IP packet (header and payload) inside a new IP packet with a new outer IP header. The IPsec header is placed between the new outer IP header and the encapsulated original packet. Tunnel mode is used for site-to-site VPNs and remote access VPNs where traffic between two networks must be routed through an encrypted tunnel between gateways.

"Tunnel mode is essential for VPNs because it hides the internal network topology. An observer sees only the gateway addresses -- the actual source and destination of the communication are encrypted inside the tunnel." -- Charlie Kaufman, co-author of RFC 4306 (IKEv2)

Internet Key Exchange (IKE)

IKE (Internet Key Exchange) is the protocol used to negotiate and establish IPsec Security Associations. It handles authentication between the two parties, negotiates cryptographic algorithms, and generates the shared secret keys used by AH and ESP. IKE uses UDP port 500 (or UDP 4500 for NAT Traversal).

IKE Phase 1

IKE Phase 1 establishes a secure, authenticated channel between the two IKE peers. This channel (called an IKE SA or ISAKMP SA) is used to protect the Phase 2 negotiations. Phase 1 can operate in two modes:

• Main Mode: Six messages exchanged. Provides identity protection (the peers' identities are encrypted). More secure but slower.
• Aggressive Mode: Three messages exchanged. Faster but exposes peer identities in cleartext. Vulnerable to offline dictionary attacks against pre-shared keys.

Phase 1 negotiations include:

1. Agreement on IKE SA parameters: encryption algorithm (AES-256), hash algorithm (SHA-256/SHA-384), authentication method (pre-shared key or certificates), Diffie-Hellman group
2. Diffie-Hellman key exchange to generate shared secret material
3. Mutual authentication using pre-shared keys (PSK) or digital certificates (X.509)

IKE Phase 2

IKE Phase 2 negotiates the IPsec SAs that will actually protect the data traffic. Phase 2 uses Quick Mode and is protected by the IKE SA established in Phase 1. Phase 2 negotiations include:

1. Agreement on IPsec SA parameters: protocol (ESP or AH), encryption algorithm, integrity algorithm, SA lifetime
2. Agreement on what traffic to protect (selectors: source/destination IP ranges and ports)
3. Optional Perfect Forward Secrecy (PFS): a new Diffie-Hellman exchange to generate fresh key material, ensuring that compromise of Phase 1 keys does not compromise Phase 2 keys

Phase 2 creates two unidirectional SAs -- one for each direction of traffic. Each SA has a unique Security Parameter Index (SPI) that identifies it in the SAD.

IKEv2

IKEv2 (RFC 7296) is the modern replacement for IKEv1. It simplifies the protocol significantly, reducing the number of exchanges from up to nine messages (IKEv1 Main Mode + Quick Mode) to just four messages for initial SA establishment:

1. IKE_SA_INIT: Two messages -- Diffie-Hellman exchange and algorithm negotiation (equivalent to Phase 1)
2. IKE_AUTH: Two messages -- authentication and IPsec SA negotiation (equivalent to Phase 2)

Key improvements in IKEv2 over IKEv1:

• Simpler, more reliable protocol with fewer message exchanges
• Built-in NAT Traversal (no separate RFC needed)
• Built-in dead peer detection (DPD)
• Support for EAP authentication (enabling integration with RADIUS/Active Directory)
• MOBIKE extension (RFC 4555) for seamless IP address changes (mobile devices)
• Better resistance to denial-of-service attacks (cookie mechanism)

Security Associations (SA)

A Security Association (SA) is the fundamental concept in IPsec. It defines the security parameters for a specific one-way traffic flow between two peers. An SA specifies:

• Security Protocol: AH or ESP
• Encryption Algorithm: AES-128, AES-256, ChaCha20-Poly1305, etc.
• Integrity Algorithm: HMAC-SHA-256, HMAC-SHA-384, AES-XCBC, etc.
• Cryptographic Keys: The actual keys used for encryption and integrity verification
• Mode: Transport or Tunnel
• SA Lifetime: Time-based (seconds) and/or data-based (kilobytes) -- after which the SA must be renegotiated
• Security Parameter Index (SPI): A 32-bit identifier that uniquely identifies the SA
• Sequence Number Counter: Used for anti-replay protection

Because SAs are unidirectional, a bidirectional IPsec tunnel requires a pair of SAs -- one for each direction. A typical site-to-site VPN therefore creates at minimum four SAs: one IKE SA pair (for IKE negotiations) and one IPsec SA pair (for data traffic).

SA lifetimes are critical for security. Keys should be rotated regularly to limit the amount of data encrypted with any single key. Typical lifetimes are 8 hours (time-based) or 100 MB to 1 GB (data-based). When an SA approaches its lifetime, IKE automatically renegotiates a new SA before the old one expires, ensuring seamless key rotation without traffic interruption.

Site-to-Site VPN

The most common deployment of IPsec is the site-to-site VPN, which creates an encrypted tunnel between two network gateways (routers or firewalls), transparently connecting the private networks behind each gateway. All traffic between the two networks is automatically encrypted and authenticated without requiring any configuration on individual hosts.

Site-to-site VPN establishment process:

1. An internal host sends a packet destined for the remote network (e.g., 192.168.1.10 to 10.0.2.50)
2. The local gateway's SPD matches the traffic and determines it must be protected by IPsec
3. If no SA exists for this traffic, the gateway initiates IKE negotiation with the remote gateway
4. IKE Phase 1 (or IKE_SA_INIT + IKE_AUTH in IKEv2) establishes the IKE SA
5. IKE Phase 2 (or included in IKE_AUTH) establishes the IPsec SA pair
6. The local gateway encrypts the original packet (including its original IP header) using ESP in tunnel mode
7. A new outer IP header is added with the gateway addresses as source and destination
8. The encapsulated packet traverses the internet to the remote gateway
9. The remote gateway decrypts the packet, verifies integrity, and forwards the original packet to the destination host
10. The destination host receives the packet as if it came directly from the source -- completely unaware of the VPN

Site-to-site VPNs using IPsec are deployed by organizations of all sizes to connect branch offices, data centers, and cloud environments. They can be configured between any combination of hardware firewalls (Cisco, Palo Alto, Fortinet), software routers (VyOS, pfSense), and cloud VPN gateways (AWS VPN, Azure VPN Gateway).

Security Considerations

While IPsec provides robust security, proper implementation requires attention to several key areas:

• Pre-Shared Key (PSK) security: PSKs must be long, random, and unique per tunnel. Short or dictionary-based PSKs are vulnerable to offline brute-force attacks, especially when IKEv1 Aggressive Mode is used. Digital certificates are preferred for production deployments.
• Algorithm selection: Avoid deprecated algorithms. DES, 3DES, MD5, and DH Group 1 (768-bit) and Group 2 (1024-bit) are considered insecure. Use AES-256 for encryption, SHA-256 or SHA-384 for integrity, and DH Group 14 (2048-bit) or higher.
• Perfect Forward Secrecy (PFS): Always enable PFS for IPsec SAs. Without PFS, compromise of the IKE SA keys allows decryption of all IPsec traffic. With PFS, each IPsec SA uses independently generated keys.
• Dead Peer Detection (DPD): Enable DPD to detect when the remote peer becomes unreachable, allowing timely SA cleanup and failover to backup paths.
• Anti-replay protection: IPsec uses sequence numbers to prevent replay attacks. Ensure anti-replay windows are properly sized for your traffic patterns -- too small causes legitimate packet drops in high-latency environments.
• MTU and fragmentation: IPsec encapsulation adds overhead (typically 50-100 bytes), which can cause fragmentation if the original packet is near the MTU. Configure Path MTU Discovery or set appropriate MSS clamping to avoid fragmentation-related performance issues.

For related topics, explore VPN technologies, SSL/TLS, and symmetric encryption algorithms used by IPsec.

References

• Kent, S., & Seo, K. (2005). RFC 4301: Security Architecture for the Internet Protocol. IETF.
• Kent, S. (2005). RFC 4302: IP Authentication Header. IETF.
• Kent, S. (2005). RFC 4303: IP Encapsulating Security Payload (ESP). IETF.
• Kaufman, C., et al. (2014). RFC 7296: Internet Key Exchange Protocol Version 2 (IKEv2). IETF.
• Kivinen, T. (2018). RFC 8247: Algorithm Implementation Requirements and Usage Guidance for the Internet Key Exchange Protocol Version 2. IETF.
• Doraswamy, N., & Harkins, D. (2003). IPSec: The New Security Standard for the Internet, Intranets, and Virtual Private Networks, 2nd Edition. Prentice Hall.
• Frankel, S., et al. (2005). NIST SP 800-77: Guide to IPsec VPNs.
• Huttunen, A., et al. (2005). RFC 3948: UDP Encapsulation of IPsec ESP Packets. IETF.
• Stallings, W. (2022). Network Security Essentials, 7th Edition. Pearson. Chapter 8: IP Security.
• StrongSwan Project (2024). StrongSwan Documentation. strongswan.org.