Automated Red Teaming: Can AI Find Its Own Weaknesses?
The [red teaming guide](/blog/red-teaming-guide/) we published earlier assumes humans are doing the work. Prompt engineers try jailbreaks. Security researchers probe failure modes. Domain experts design edge cases. It's thorough. It's slow. And it doesn't scale.
As AI systems become more capable and more complex, manual red teaming is becoming a bottleneck. The solution being explored across the field is automation: using AI to find AI weaknesses. But this raises a paradox. If you build a system smart enough to find vulnerabilities in other AI systems, how do you ensure it doesn't become a vulnerability itself?
The evolution of automated red teaming
The field has evolved rapidly. Early automated attacks used gradient optimization 鈥?methods like GCG that compute adversarial suffixes by backpropagating through the target model [^raheja-survey]. These are effective but require white-box access, which production systems rarely provide.
The next wave used iterative LLM-driven generation. Systems like PAIR and TAP generate attack prompts, test them against the target, and refine based on success signals 鈥?all without needing model internals [^raheja-survey]. This opened red teaming to black-box settings, where only API access is available.
Spiess et al. (2026) pushed this further with adaptive prompt optimization 鈥?a paradigm where the red team system continuously evolves its attack strategy based on real-time feedback [^spiess-adaptive]. Instead of running a fixed set of tests, the system learns what works against a specific target and doubles down.
Agent skills as attack surfaces
The most recent advance targets a new frontier: agentic systems. As LLMs evolve into tool-using agents with MCP (Model Context Protocol) integrations, the attack surface expands beyond model outputs to tool manipulation, skill exploitation, and multi-step action sequences.
SkillAttack (2026) introduced a framework that treats agent skills as independent attack surfaces [^skillattack]. Rather than trying to jailbreak the model's response generation, it probes whether vulnerabilities in unmodified skills can be exploited through iterative adversarial prompting. The framework operates in three stages: vulnerability analysis identifies attack surfaces, surface-parallel generation constructs attack paths, and feedback-driven refinement iteratively improves them based on agent execution results.
This matters because [agentic systems](/blog/agentic-alignment-gap/) are the next deployment wave 鈥?and their attack surface is much larger than chatbots. A model that can browse the web, execute code, and call APIs has more ways to fail, and more ways to be made to fail.
The measurement challenge
Andriushchenko et al. (2024/2025) created AgentHarm, a benchmark for measuring harmfulness of LLM agents in realistic tool-use scenarios [^agentharm]. This provides the evaluation infrastructure that makes automated red teaming measurable 鈥?but it also highlights the gap between finding vulnerabilities and understanding their severity.
Automated systems can generate thousands of potential attacks. But not all vulnerabilities are equal. A prompt that makes the model generate offensive text is different from one that makes it leak user data or execute unauthorized API calls. Automated red teaming excels at discovery. It still needs human judgment for prioritization.
The dual-use problem
The most uncomfortable question about automated red teaming is who gets to use it. A system that can automatically find vulnerabilities in frontier AI models is, by definition, a system that can automatically generate exploits for those models.
This is the dual-use dilemma in its purest form. The same capability that helps safety teams harden their systems can help malicious actors break them. And the economics favor the attacker: an automated red team needs to find one vulnerability to succeed. The defense needs to close them all.
Some researchers have proposed differential access 鈥?sharing automated red teaming tools with vetted researchers while restricting public access. But in practice, the underlying techniques (gradient optimization, iterative LLM generation, semantic perturbation) are already public knowledge. The genie is out of the bottle.
The future of red teaming
Automated red teaming isn't going to replace human judgment. But it's going to change the scale and speed at which vulnerabilities are found. A future where every major AI release is automatically probed by thousands of adversarial agents before deployment is plausible 鈥?and probably necessary.
The question is whether the same automation can be applied to defense. Can we build systems that automatically patch the vulnerabilities they find? Can we create self-healing guardrails that adapt to new attack patterns in real time? So far, the answer is: not yet. Attack automation is ahead of defense automation by a significant margin.
Frequently Asked Questions
What is automated red teaming? Automated red teaming uses AI systems to discover vulnerabilities in other AI systems without human-in-the-loop testing. Methods range from gradient-based adversarial optimization (white-box) to iterative LLM-driven prompt generation (black-box) to agent-specific skill exploitation [^raheja-survey] [^skillattack].
How is automated red teaming different from manual red teaming? Manual red teaming relies on human creativity and domain expertise to design attacks. Automated systems can generate and test thousands of variations faster, find subtle patterns humans miss, and scale to complex agentic systems. But they still need human judgment for severity assessment and prioritization.
Can automated red teaming find all vulnerabilities? No. Automated systems excel at finding known classes of vulnerabilities (jailbreaks, prompt injection, tool manipulation) but may miss novel failure modes. The [jailbreak taxonomy](/blog/jailbreak-economics/) shows that mixed attacks combining multiple strategies are particularly hard to discover and defend against [^pauli-sok].
Is automated red teaming dangerous? Yes, in the sense that dual-use capabilities are dangerous. The same tools that help safety teams harden systems can help attackers break them. The underlying techniques are already public. The balance of power currently favors attack automation over defense automation.
---
[^raheja-survey]: Raheja, T., et al. (2024/2025). "Recent Advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations." https://arxiv.org/abs/2410.09097
[^spiess-adaptive]: Spiess, C., et al. (2026). "Adaptive Red-Teaming of Large Language Models." EACL SRW. https://aclanthology.org/2026.eacl-srw.33.pdf
[^skillattack]: (Anonymous, 2026). "Automated Red Teaming of Agent Skills through Attack Path Refinement." arXiv preprint arXiv:2604.04989. https://arxiv.org/abs/2604.04989
[^agentharm]: Andriushchenko, M., et al. (2024/2025). "AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents." https://arxiv.org/abs/2410.09024
---
DarkMental AI maps the structural shadows where AI learns to break itself.