AI does not create entirely new attack categories. It makes existing attacks, such as phishing, impersonation, and malware development, faster, cheaper, and more convincing. Adversaries use large language models (LLMs) to write polished phishing messages in any language and use voice cloning and deepfake technology to impersonate real people on calls or video.
- Voice cloning and deepfakes: AI tools use existing voice and image samples to create a digital avatar that can impersonate someone on a phone call or video call, enabling financial fraud or credential theft.
- LLM-generated phishing: Generative AI removes the grammatical errors that once helped users spot phishing messages, making AI-crafted messages harder to distinguish from legitimate communication.
- Shared secrets: A pre-agreed word or phrase known only to two trusted parties can verify identity in high-stakes situations where voice or video impersonation is possible.
- Prompt injection: An adversary embeds malicious instructions in content that an AI tool processes, causing the AI to take unintended actions or leak information.
- Training data poisoning: Adversaries corrupt the data used to train an AI model so the model produces incorrect or harmful outputs.
Explain why LLM-generated phishing messages are harder to detect than traditional phishing, and name two defenses against AI-augmented impersonation attacks.