Network Security Protocols

Why This Matters

Network security protocols form the backbone of every secure digital interaction you'll encounter—from logging into a remote server to browsing a website to sending encrypted email. You're being tested on your ability to understand where these protocols operate in the network stack, how they achieve confidentiality, integrity, and authentication, and why certain protocols are chosen for specific use cases. These concepts connect directly to broader themes in cryptography: symmetric vs. asymmetric encryption, key exchange mechanisms, digital signatures, and the CIA triad (confidentiality, integrity, availability).

When exam questions ask about network security, they're rarely looking for simple definitions. Instead, you'll need to compare protocols operating at different layers, explain why one protocol might be preferred over another, and identify vulnerabilities that each protocol addresses. Don't just memorize what each protocol does—know what cryptographic principles each one demonstrates and what attack vectors it defends against.

---

Transport and Application Layer Security

These protocols secure data at the higher layers of the network stack, protecting communication between applications and ensuring end-to-end encryption for user-facing services. They rely on the underlying network infrastructure being functional, then add security on top.

SSL/TLS (Secure Sockets Layer/Transport Layer Security)

• Handshake protocol establishes secure sessions—negotiates cipher suites, exchanges keys, and authenticates parties before any application data flows
• Digital certificates provide authentication through a chain of trust, with Certificate Authorities (CAs) vouching for server identity
• Perfect Forward Secrecy (PFS) in modern TLS ensures that compromising long-term keys doesn't expose past session data

HTTPS (Hypertext Transfer Protocol Secure)

• HTTP wrapped in TLS encryption—operates on port 443 and provides confidentiality for all web traffic between browser and server
• Certificate validation triggers browser warnings when certificates are expired, self-signed, or issued by untrusted CAs
• Ubiquitous deployment makes it the standard for any sensitive web interaction, from banking to social media authentication

SSH (Secure Shell)

• Secure remote access replaces insecure protocols like Telnet and rlogin, encrypting the entire session including authentication credentials
• Public-key authentication eliminates password transmission—users prove identity by signing challenges with their private key
• Port forwarding and tunneling allow SSH to secure other protocols, creating encrypted channels for otherwise vulnerable traffic

SFTP (Secure File Transfer Protocol)

• SSH subsystem for file operations—not just encrypted FTP, but a completely different protocol built on SSH's secure channel
• Integrity verification ensures files arrive unmodified, protecting against man-in-the-middle tampering during transfer
• Single port operation (typically port 22) simplifies firewall configuration compared to FTP's dual-port architecture

---

Network Layer Security

These protocols operate at the IP layer, securing packets regardless of the application generating them. This provides transparent security—applications don't need modification to benefit from protection.

IPsec (Internet Protocol Security)

• Two operational modes—Transport mode encrypts only the payload (used for host-to-host), while Tunnel mode encrypts the entire packet including headers (used for VPNs)
• Authentication Header (AH) and Encapsulating Security Payload (ESP) provide integrity/authentication and encryption respectively
• Security Associations (SAs) define the parameters for secure communication, negotiated through IKE (Internet Key Exchange)

VPN Protocols (OpenVPN, L2TP/IPsec, WireGuard)

• Encrypted tunnels create secure pathways through untrusted networks—all traffic appears as a single encrypted stream to observers
• Protocol tradeoffs matter—OpenVPN offers flexibility and strong security, L2TP/IPsec provides native OS support, WireGuard prioritizes speed and simplicity
• Use cases include bypassing geographic restrictions, securing public Wi-Fi connections, and connecting remote offices to corporate networks

---

Wireless Network Security

Wireless protocols face unique challenges because radio signals can be intercepted by anyone within range. These protocols must assume an adversary can capture every transmitted packet.

WPA/WPA2/WPA3 (Wi-Fi Protected Access)

• Evolution of wireless security—WPA replaced broken WEP, WPA2 added mandatory AES encryption, WPA3 introduces Simultaneous Authentication of Equals (SAE) to resist offline dictionary attacks
• Four-way handshake in WPA2 establishes session keys, but vulnerabilities like KRACK demonstrated the importance of proper implementation
• Enterprise vs. Personal modes—Enterprise uses RADIUS authentication for individual credentials, Personal uses a shared passphrase (PSK)

---

Authentication Protocols

These protocols focus specifically on proving identity, often using clever cryptographic techniques to avoid transmitting passwords directly. Authentication is distinct from encryption—you can have one without the other.

Kerberos

• Ticket-based authentication eliminates repeated password entry—users authenticate once to the Key Distribution Center (KDC) and receive tickets for accessing services
• Three-party model involves the client, the Authentication Server (AS), and the Ticket Granting Server (TGS), all sharing secret keys
• Time-sensitive tickets expire quickly, limiting the window for replay attacks—requires synchronized clocks across the network

---

Infrastructure Security

These protocols protect the fundamental services that make networking function, ensuring that basic operations like name resolution and email remain trustworthy.

DNSSEC (Domain Name System Security Extensions)

• Digital signatures on DNS records allow resolvers to verify that responses came from authoritative sources and weren't modified in transit
• Chain of trust extends from the root zone down through TLDs to individual domains, with each level signing the keys of the level below
• Defends against cache poisoning and spoofing—attacks where adversaries redirect users to malicious sites by corrupting DNS responses

PGP (Pretty Good Privacy)

• Hybrid encryption model—uses fast symmetric encryption for message content, then encrypts the symmetric key with the recipient's public key
• Web of trust replaces centralized CAs—users sign each other's keys to vouch for identity, creating a decentralized authentication network
• Digital signatures provide non-repudiation—recipients can verify messages came from the claimed sender and weren't altered

---

Quick Reference Table

---

Self-Check Questions

1. Which two protocols both use SSH as their underlying transport mechanism, and what distinguishes their primary use cases?

2. Compare IPsec Transport mode and Tunnel mode—when would you choose each, and what specific data does each mode encrypt?

3. If a user connects to a coffee shop Wi-Fi protected by WPA2-Personal and then establishes an OpenVPN connection, which protocol protects against which threat? Explain the layered security model.

4. Both Kerberos and PGP solve authentication problems, but they use fundamentally different trust models. Compare the "trusted third party" approach versus the "web of trust" and identify one advantage of each.

5. A company wants to ensure employees visiting internal websites aren't redirected to phishing sites through DNS attacks. Which protocol addresses this specific threat, and what cryptographic mechanism does it use to provide protection?