18.3 Secure Protocols: SSL/TLS, IPsec

SSL/TLS Protocol

SSL/TLS and IPsec solve the same fundamental problem: keeping data safe as it crosses untrusted networks. They just do it at different layers of the network stack, which changes what they protect and how they're deployed.

SSL/TLS secures communication at the application layer, most commonly between web browsers and servers. IPsec operates down at the IP layer, where it can protect all network traffic regardless of the application. Understanding where each protocol sits in the stack is the key to understanding their trade-offs.

Purpose of SSL/TLS

SSL/TLS creates an encrypted channel between a client (usually a web browser) and a server. Its three main goals are:

• Confidentiality: Encrypts data so eavesdroppers can't read it
• Integrity: Detects any tampering with data in transit
• Authentication: Verifies the server's identity (and optionally the client's) using digital certificates

You encounter SSL/TLS constantly. Every time you see HTTPS in your browser's address bar, an SSL/TLS session is protecting that connection. Other examples include FTPS (secure file transfer) and SMTPS (secure email submission).

SSL/TLS sits on top of TCP at the application layer. This means it only protects the data that a specific application sends through it. The underlying IP headers and routing information are not encrypted.

Processes in SSL/TLS

The SSL/TLS handshake establishes the secure connection before any application data is exchanged. Here's how it works:

Key Exchange:

1. The client sends a "ClientHello" message proposing supported cryptographic algorithms and parameters.
2. The server responds with a "ServerHello," selecting algorithms, and sends its digital certificate containing its public key.
3. The client verifies the server's certificate (checking the CA signature, expiration, and domain name).
4. The client generates a random pre-master secret and encrypts it with the server's public key.
5. The server decrypts the pre-master secret using its private key. Both sides now derive the same session keys from this shared secret.

Authentication:

• The server proves its identity through a digital certificate issued by a trusted Certificate Authority (CA). The certificate binds the server's public key to its domain name.
• The client checks the certificate against its local trust store (a list of trusted CAs built into the browser or OS). If the CA signature is valid and the certificate hasn't expired, authentication succeeds.
• Optionally, the server can request a client certificate for mutual (two-way) authentication, though this is less common outside enterprise environments.

Encryption:

• Once the handshake completes, both sides share symmetric session keys. Symmetric encryption (e.g., AES-256, ChaCha20) handles the actual data encryption because it's far faster than asymmetric encryption.
• Data integrity is protected using Message Authentication Codes (MACs), typically HMAC-SHA256. The MAC lets the receiver detect if even a single bit was altered in transit.

The handshake uses asymmetric (public-key) cryptography only to exchange keys securely. After that, everything switches to symmetric encryption for performance.

IPsec Protocol

Role of IPsec

IPsec provides security services directly at the IP layer, which means it can protect all traffic flowing between two endpoints, regardless of the application generating it. This is what makes it the backbone of most VPN implementations.

IPsec relies on two core sub-protocols:

• Authentication Header (AH): Provides data integrity and authentication, but no encryption. AH verifies that packets haven't been modified and confirms the sender's identity.
• Encapsulating Security Payload (ESP): Provides confidentiality (encryption) plus data integrity and authentication. In practice, ESP is used far more often than AH because most deployments need encryption.

Before any IPsec communication begins, both sides must agree on security parameters through a Security Association (SA). An SA defines which algorithms to use, what keys to apply, and how long the session lasts. The IKE (Internet Key Exchange) protocol, typically IKEv2, handles this negotiation automatically.

IPsec Modes

IPsec can operate in two distinct modes, and the choice depends on what you're trying to protect:

Transport Mode:

• Encrypts/authenticates only the payload of the IP packet. The original IP header stays intact.
• Used for end-to-end communication between two specific hosts (e.g., one server talking directly to another).
• Since the original IP header is visible, routers can still make forwarding decisions normally.

Tunnel Mode:

• Encapsulates the entire original IP packet (header and payload) inside a brand-new IP packet with a new header.
• Used to build site-to-site VPNs between gateways or routers. The original source and destination addresses are hidden inside the encrypted payload.
• Also used for remote-access VPNs, where a user's device tunnels all traffic through a corporate gateway.

The distinction matters: transport mode protects the conversation between two machines, while tunnel mode protects traffic between two networks by hiding the internal addressing entirely.

SSL/TLS vs IPsec

The short version: use SSL/TLS when you need to secure specific application connections (especially web traffic), and use IPsec when you need to secure all traffic between networks or hosts at the network layer.