IPsec (Internet Protocol Security) is a series of protocols that is used to protect IP traffic between two points on a network. It offers confidentiality, data integrity, and a high degree of security through its advanced packet encryption. For these reasons, IPsec is most commonly used for business VPNs.
In this article, you’ll learn about the two primary modes of IPsec—tunnel mode and transport mode—and the use cases for each.
IPsec Tunnel vs. Transport Mode
In order to authenticate data packets and guarantee their integrity, IPsec includes two protocols. These are the AH (Authentication Header) protocol and the ESP (Encapsulating Security Payload) protocol. Both protocols, in turn, support two encapsulation modes—tunnel mode and transport mode. Let’s break down their core differences.
In tunnel mode, the entire original IP packet is encapsulated to become the payload of a new IP packet. Additionally, a new IP header is added on top of the original IP packet. Since a new packet is created using the original information, tunnel mode is useful for protecting traffic between different networks. An additional advantage of this mode is that it makes it very easy to establish a “tunnel” between two secure IPsec gateways.
These IPsec gateways in turn can connect two different networks securely. Using secure IPsec proxies like the ones shown in the diagram below can be very useful for connecting two distant branches using an encrypted connection.
The process used by IPsec to encapsulate the original IP header differs depending on whether AH tunnel mode or ESP tunnel mode is used:
The original packet is encapsulated in a new IP packet (both its IP header and its payload).
In the case of AH tunnel mode, an AH header and a new IP header are added. For ESP tunnel mode, an ESP header, a new IP header, an ESP trailer, and an ESP authentication trailer are added.
When AH tunnel mode is used, the entire packet is signed for integrity and authentication. But when ESP tunnel mode is used, the encapsulated packet between the ESP header and the ESP trailer is signed for integrity and authentication. The new packet can also be encrypted for greater security.
The main difference in transport mode is that it retains the original IP header. In other words, payload data transmitted within the original IP packet is protected, but not the IP header. In transport mode, encrypted traffic is sent directly between two hosts that previously established a secure IPsec tunnel.
Since a new IP header isn’t created, the process used by transport mode is less complex than tunnel mode:
Depending on the protocol used, a new AH or ESP header is created and inserted just after the original IP header.
For the ESP protocol, both an ESP trailer and an ESP authentication trailer are created and added after the original package.
When using AH transport mode, the entire packet is signed for integrity and authentication. For ESP transport mode, the original packet payload is signed by authentication (that is, not including its IP header) and encrypted if required.
When to Use IPsec Tunnel Mode
Tunnel mode is most commonly used for configurations that need a secure connection between two different networks, separated by an intermediate untrusted network (like the Internet).
Typical tunnel mode use cases are gateway-to-gateway, server-to-gateway, and server-to-server. Here’s a list of various reasons why tunnel mode works best for these use cases:
- Tunnel mode protects internal routing information by encrypting the original packet’s IP header by creating a new IP header on top of it. This allows tunnel mode to protect against traffic analysis, since attackers can only determine the tunnel endpoints.
- Tunnel mode is mandatory when one of the peers is a security gateway applying IPsec on behalf of another host. In other words, it’s more compatible with existing gateways than transport mode.
- Tunnel mode makes it easier to traverse NATs.
- Both VPN clients and VPN gateways can use IPsec tunnel mode.
Despite its advantages, tunnel mode has a greater overhead and smaller MTU than transport mode.
When to Use IPsec Transport Mode
Transport mode is commonly used when fast and secure end-to-end communications are required, such as client-server communications (workstation-to-gateway and host-to-host scenarios). Reasons to use transport mode include:
- Transport mode provides end-to-end security (authentication, integrity, and anti-replay protection).
- Transport mode has a larger MTU than tunnel mode.
- Transport mode has a lower overhead than tunnel mode.
Transport mode is not without its flaws. It has poor compatibility with security gateways, as well as greater difficulty in implementing traversal NATs. For this reason, transport mode can’t be used in protected gateway-to-gateway configurations.
Setting Each Mode Up
To successfully set up each mode, it’s essential to know how IPsec negotiates packet security using the IKE (Internet Key Exchange) protocol.
During the IPsec tunnel set up, the peers establish security associations (SA), defining which parameters will be used to secure the traffic between them. The process of negotiating such parameters happens in two phases:
IKE Phase 1: This phase creates a secure tunnel to protect the negotiation messages peers will exchange in the second phase.
IKE Phase 2: During this phase, the SA parameters of a second IPsec tunnel are negotiated. While the first tunnel is used to protect SA negotiations, this tunnel protects the data.
Once the secure tunnel (IKE Phase 2) has been established, IPsec protects the traffic sent between the two tunnel endpoints. It does this by applying the security parameters defined by the SAs during tunnel configuration. The encapsulation mode is part of these parameters.
For clarification, IPsec only uses the IKE protocol to build secure tunnels between the two devices and set up SA parameters. Authentication and encryption are handled by the AH and ESP protocols, respectively.
Regardless of whether you use tunnel mode or transport mode, the encapsulation mode used by the AH and ESP protocols must be set up during IKE Phase 2—before the actual data transmission.
In this article, you’ve learned the main differences between IPsec’s two encapsulation modes: transport mode and tunnel mode. You should also know the pros and cons of both modes, and consequently understand best use cases for each.
The intricacy of IPsec connections represents an opportunity to consider alternative ways to securely access your remote data—without falling victim to hacking due to a bad configuration. Cutting-edge solutions like Twingate enable your business to rapidly implement a modern, zero-trust network that is more secure and maintainable than conventional VPNs.
Request a Twingate demo today and deploy secure network connections in a matter of minutes.