Network Security Protocols

Why This Matters

Network security protocols form the backbone of every secure digital interaction, from logging into a remote server to browsing a website to sending encrypted email. You need to understand where these protocols operate in the network stack, how they achieve confidentiality, integrity, and authentication, and why certain protocols are chosen for specific use cases. These concepts connect directly to broader themes in cryptography: symmetric vs. asymmetric encryption, key exchange mechanisms, digital signatures, and the CIA triad (confidentiality, integrity, availability).

Exam questions about network security rarely ask for simple definitions. You'll need to compare protocols operating at different layers, explain why one protocol might be preferred over another, and identify the vulnerabilities each protocol addresses. Don't just memorize what each protocol does. Know what cryptographic principles each one demonstrates and what attack vectors it defends against.

---

Transport and Application Layer Security

These protocols secure data at the higher layers of the network stack, protecting communication between applications and providing end-to-end encryption for user-facing services. They rely on the underlying network infrastructure being functional, then add security on top.

SSL/TLS (Secure Sockets Layer/Transport Layer Security)

SSL is the predecessor; TLS is the modern, actively maintained standard. When people say "SSL" today, they almost always mean TLS.

• Handshake protocol establishes secure sessions by negotiating cipher suites, exchanging keys, and authenticating parties before any application data flows
• Digital certificates provide authentication through a chain of trust, with Certificate Authorities (CAs) vouching for server identity
• Perfect Forward Secrecy (PFS) in modern TLS (1.2 with appropriate cipher suites, and mandatory in TLS 1.3) uses ephemeral Diffie-Hellman key exchange so that compromising long-term keys doesn't expose past session data

The TLS handshake is worth understanding step by step:

1. ClientHello: The client sends supported TLS versions, cipher suites, and a random nonce.
2. ServerHello: The server selects a cipher suite and TLS version, sends its own random nonce.
3. Certificate: The server presents its digital certificate for the client to verify.
4. Key Exchange: Both sides contribute to generating a shared secret (using, e.g., ephemeral Diffie-Hellman for PFS).
5. Finished: Both sides derive session keys from the shared secret and confirm the handshake wasn't tampered with.

After this, all application data is encrypted with the negotiated symmetric cipher.

HTTPS (Hypertext Transfer Protocol Secure)

• HTTP wrapped in TLS encryption, operating on port 443, providing confidentiality for all web traffic between browser and server
• Certificate validation triggers browser warnings when certificates are expired, self-signed, or issued by untrusted CAs
• Ubiquitous deployment makes it the standard for any sensitive web interaction, from banking to social media authentication

SSH (Secure Shell)

• Secure remote access that replaced insecure protocols like Telnet and rlogin, encrypting the entire session including authentication credentials
• Public-key authentication eliminates password transmission. Users prove identity by signing challenges with their private key, so the private key never leaves the client machine.
• Port forwarding and tunneling allow SSH to secure other protocols, creating encrypted channels for otherwise vulnerable traffic

SFTP (Secure File Transfer Protocol)

• An SSH subsystem for file operations, not simply FTP with encryption added. SFTP is a completely different protocol built on SSH's secure channel.
• Integrity verification ensures files arrive unmodified, protecting against man-in-the-middle tampering during transfer
• Single port operation (typically port 22) simplifies firewall configuration compared to FTP's dual-port architecture (a control port and a separate data port)

---

Network Layer Security

These protocols operate at the IP layer, securing packets regardless of the application generating them. This provides transparent security: applications don't need modification to benefit from protection.

IPsec (Internet Protocol Security)

IPsec has two operational modes and two core sub-protocols. You need to keep these distinctions straight.

Modes:

• Transport mode encrypts only the IP payload (the data), leaving the original IP header intact. Used for host-to-host communication where both endpoints speak IPsec directly.
• Tunnel mode encrypts the entire original IP packet (header and payload), then wraps it in a new IP header. This is what VPN gateways use, because it hides the original source and destination addresses.

Sub-protocols:

• Authentication Header (AH) provides integrity and authentication but no encryption. It verifies that packets haven't been tampered with.
• Encapsulating Security Payload (ESP) provides encryption and can also provide integrity/authentication. In practice, ESP is far more commonly used than AH.

Security Associations (SAs) define the parameters for a secure connection (which algorithms, which keys, which mode). SAs are negotiated through IKE (Internet Key Exchange), which itself uses Diffie-Hellman for key agreement.

VPN Protocols (OpenVPN, L2TP/IPsec, WireGuard)

• Encrypted tunnels create secure pathways through untrusted networks. All traffic appears as a single encrypted stream to outside observers.
• Protocol tradeoffs matter: OpenVPN offers flexibility and strong security but can be slower; L2TP/IPsec provides native OS support without extra software; WireGuard prioritizes speed and simplicity with a much smaller codebase (roughly 4,000 lines of code vs. hundreds of thousands for OpenVPN).
• Common use cases include securing public Wi-Fi connections, connecting remote offices to corporate networks, and accessing geographically restricted resources.

---

Wireless Network Security

Wireless protocols face unique challenges because radio signals can be intercepted by anyone within range. These protocols must assume an adversary can capture every transmitted packet.

WPA/WPA2/WPA3 (Wi-Fi Protected Access)

The evolution here matters because each generation fixed specific weaknesses in its predecessor:

• WEP (the original) used RC4 with static keys and was catastrophically broken. Short initialization vectors meant keys could be recovered in minutes.
• WPA was a stopgap fix that introduced TKIP (Temporal Key Integrity Protocol) to rotate keys, but still relied on RC4.
• WPA2 made AES-CCMP encryption mandatory, providing much stronger confidentiality and integrity.
• WPA3 introduces Simultaneous Authentication of Equals (SAE), replacing the PSK-based handshake. SAE is based on a Diffie-Hellman-like exchange that resists offline dictionary attacks even if an attacker captures the handshake.

The four-way handshake in WPA2 establishes per-session keys, but the KRACK attack (2017) showed that forcing nonce reuse during this handshake could break encryption. This highlighted that correct implementation matters as much as correct design.

• Enterprise vs. Personal modes: Enterprise uses a RADIUS server to issue individual credentials per user. Personal uses a shared passphrase (PSK) for everyone on the network.

---

Authentication Protocols

These protocols focus specifically on proving identity, often using clever cryptographic techniques to avoid transmitting passwords directly. Authentication is distinct from encryption: you can have one without the other.

Kerberos

Kerberos uses a trusted third party model with symmetric cryptography. The flow works like this:

1. The client authenticates to the Authentication Server (AS) and receives a Ticket Granting Ticket (TGT), encrypted with the TGS's secret key.
2. When the client needs to access a service, it presents the TGT to the Ticket Granting Server (TGS) and receives a service ticket for that specific resource.
3. The client presents the service ticket to the target service, which can decrypt and verify it without contacting the KDC again.

Key properties to remember:

• Single sign-on: users authenticate once and receive tickets for multiple services without re-entering passwords.
• Time-sensitive tickets expire quickly, limiting the window for replay attacks. This means all machines on the network need synchronized clocks (typically via NTP).
• Symmetric cryptography throughout: the KDC shares secret keys with every client and service, which makes the KDC a critical single point of failure.

---

Infrastructure Security

These protocols protect the fundamental services that make networking function, ensuring that basic operations like name resolution and email remain trustworthy.

DNSSEC (Domain Name System Security Extensions)

• Digital signatures on DNS records allow resolvers to verify that responses came from authoritative sources and weren't modified in transit
• Chain of trust extends from the root zone down through TLDs to individual domains, with each level signing the public keys of the level below using RRSIG records
• Defends against cache poisoning and spoofing, attacks where adversaries redirect users to malicious sites by corrupting DNS responses

Note that DNSSEC provides integrity and authentication but not confidentiality. DNS queries and responses are still visible to anyone on the network. For DNS privacy, you'd look at DNS over HTTPS (DoH) or DNS over TLS (DoT), which are separate mechanisms.

PGP (Pretty Good Privacy)

• Hybrid encryption model: uses fast symmetric encryption (e.g., AES) for message content, then encrypts the symmetric key with the recipient's public key. This combines the speed of symmetric crypto with the key-distribution advantages of asymmetric crypto.
• Web of trust replaces centralized CAs. Users sign each other's public keys to vouch for identity, creating a decentralized authentication network. The tradeoff is that there's no single authority to revoke compromised keys quickly.
• Digital signatures provide non-repudiation. Recipients can verify messages came from the claimed sender and weren't altered.

---

Quick Reference Table

---

Self-Check Questions

1. Which two protocols both use SSH as their underlying transport mechanism, and what distinguishes their primary use cases?

2. Compare IPsec Transport mode and Tunnel mode. When would you choose each, and what specific data does each mode encrypt?

3. If a user connects to a coffee shop Wi-Fi protected by WPA2-Personal and then establishes an OpenVPN connection, which protocol protects against which threat? Explain the layered security model.

4. Both Kerberos and PGP solve authentication problems, but they use fundamentally different trust models. Compare the "trusted third party" approach versus the "web of trust" and identify one advantage of each.

5. A company wants to ensure employees visiting internal websites aren't redirected to phishing sites through DNS attacks. Which protocol addresses this specific threat, and what cryptographic mechanism does it use to provide protection?