adversarial8 min read

Red Teaming AI: A Practical Guide for the Skeptical Builder

MisalignAI|June 10, 2026|

red-teamingtestingsecuritypracticalai-safety

Red teaming has become a buzzword. Every AI company now claims to "red team" their models. But most of what's called red teaming is actually shallow prompt engineering 鈥?trying a few jailbreaks, checking if the model refuses the obvious stuff, and calling it done. As one research team put it, the practice is more about "summoning a demon and binding it" than genuine safety assurance.[^inie]

Real red teaming is deeper. It's a systematic discipline for finding the ways your system can fail, and the ways your evaluation might be missing those failures.

What red teaming actually is

The term comes from military exercises: one team defends (blue), one team attacks (red). The red team's job isn't to prove the system is weak 鈥?it's to find the failure modes the blue team hasn't thought of.

In AI, this translates to:

Adversarial testing: Finding inputs that cause unwanted behavior

Capability discovery: Mapping what the model can actually do, not just what you trained it to do

Evaluation auditing: Testing whether your benchmarks measure what you think they measure

Failure taxonomy: Cataloging the ways your system can fail, not just the ones you've already seen

The three layers of AI red teaming

Layer 1: Surface-level jailbreaks

This is what most people call red teaming. Prompt injection, roleplay attacks, encoding tricks, multilingual bypasses.

It's important, but it's also the most crowded layer. The internet is full of jailbreak databases. If this is all you're doing, you're not red teaming 鈥?you're running a compliance checklist. Anthropic's Constitutional Classifiers research shows why this matters: after 3,000+ hours of adversarial testing across 183 participants, they found that surface-level defenses reduce jailbreak success from 86% to 4.4% 鈥?but only when the red teaming is genuinely thorough, not checkbox-level.[^anthropic]

Layer 2: Behavioral and structural testing

This is where it gets interesting. You're not trying to make the model say something forbidden. You're trying to make it behave in ways that are structurally misaligned with its design:

Sycophancy testing: Give the model incorrect premises and see if it corrects you or agrees

Goal drift: Give the model a multi-step task and introduce subtle distractions mid-stream

Reward hacking simulation: Design scenarios where the easiest path to a "successful" outcome contradicts the intended goal

Edge case mining: Systematically explore the long tail of inputs your model might encounter

Layer 3: Systemic and emergent failure

This is the hardest and most neglected layer. You're not testing individual prompts. You're testing the system:

Multi-turn compounding: Does error accumulate across conversation turns?

Tool-use failure modes: When the model has access to APIs, databases, or code execution, what's the blast radius of a mistake?

Agentic drift: In long-running tasks, does the model's understanding of its goal drift over time?

Social dynamics: How does the model behave when users are adversarial, confused, or deliberately misleading?

Practical techniques

Structured adversarial prompting

Don't just try random jailbreaks. Build a taxonomy of attack vectors:

Authority escalation ("As a security researcher...")

Goal reframing ("This is for educational purposes...")

Encoding obfuscation (base64, rot13, language switching)

Context window manipulation (distracting prefix, suffix attacks)

Multi-turn grooming (building trust across turns)

Then test systematically across categories.

Automated red teaming

Recent research has shown that using one LLM to attack another can surface failure modes humans miss. Methods like PAIR (Prompt Automatic Iterative Refinement) and TAP (Tree of Attacks with Pruning) use an attacker LLM to iteratively generate and refine adversarial prompts, exploring a tree of possible attacks and pruning unpromising branches.[^mehrotra]

These aren't replacements for human red teaming 鈥?they're force multipliers. The same research shows that automated methods find jailbreaks humans miss, but they also miss failure modes that require understanding context, intent, and real-world consequences.

Evaluation audit

Your red team should also test your evaluation:

Does your safety benchmark actually measure safety, or just refusal rate?

Are your human evaluators incentivized to flag subtle harms, or just obvious ones?

Does your benchmark have coverage of the failure modes you've discovered?

A red team that only finds failures is useful. A red team that also finds holes in your evaluation is invaluable.

What red teaming isn't

Not a one-time event: Red teaming should be continuous, not a pre-launch checkbox

Not a guarantee: Finding nothing doesn't mean nothing exists 鈥?it means your red team didn't find it yet

Not a substitute for alignment: Red teaming finds problems. [Alignment research](/blog/what-is-ai-alignment/) tries to prevent them. Both are necessary.

A 2024 paper titled "Red-Teaming for Generative AI: Silver Bullet or Security Theater?" makes this point bluntly: the same jailbreaks often work across different models, suggesting that many defenses are shallow and that red teaming findings don't generalize well.[^feffer] Red teaming is necessary but not sufficient 鈥?it's a diagnostic tool, not a cure.

Getting started

If you're building with AI and don't have a red team:

1. Start with Layer 1: Use publicly available jailbreak databases (JailbreakChat, Adversarial Robustness Toolbox) 2. Move to Layer 2: Design structured tests for sycophancy, goal drift, and edge cases specific to your domain. See our guide on [why models deceive](/blog/ai-deception/) to understand what behavioral failures look like. 3. Build Layer 3 gradually: As your system gains capabilities and users, expand to multi-turn and systemic testing. Our analysis of the [black-box economy](/blog/black-box-economy/) explains why systemic testing is especially hard when you can't see inside the model. 4. Document everything: A failure you don't write down is a failure you'll rediscover

FAQ: Common questions about AI red teaming

What is AI red teaming? AI red teaming is a structured, adversarial testing process where a team actively tries to make an AI system fail in ways its developers didn't anticipate. It originated in military exercises (red team attacks, blue team defends) and has been adapted for AI systems. Unlike standard testing that checks if a system works as intended, red teaming probes for unintended behaviors, vulnerabilities, and failure modes.[^inie]

How is red teaming different from regular AI testing? Regular testing asks: "Does the system do what we want?" Red teaming asks: "What can we make it do that we don't want?" Testing verifies intended behavior; red teaming discovers unintended behavior. Testing uses expected inputs; red teaming uses adversarial inputs. Testing measures performance; red teaming measures robustness. As researchers note, red teaming is a sociotechnical challenge involving values, labor, and harms 鈥?it's not just a technical exercise.[^inie]

What are the best automated red teaming tools? The leading automated methods include PAIR (Prompt Automatic Iterative Refinement), which uses an attacker LLM to iteratively refine prompts based on refusal feedback; TAP (Tree of Attacks with Pruning), which extends PAIR with tree-search exploration and automated pruning; and AutoDAN, which uses genetic algorithms to evolve jailbreak prompts.[^mehrotra] These are force multipliers, not replacements for human red teaming 鈥?they find failures humans miss, but miss failures that require contextual understanding.

Is red teaming enough for AI safety? No. A 2024 paper bluntly titled "Red-Teaming for Generative AI: Silver Bullet or Security Theater?" found that the same jailbreaks often work across different models, suggesting defenses are shallow and red teaming findings don't generalize well.[^feffer] Red teaming finds problems; [alignment research](/blog/what-is-ai-alignment/) tries to prevent them. Both are necessary. Red teaming is a diagnostic tool, not a cure.

How often should you red team an AI system? Continuously, not as a one-time pre-launch event. As models are updated, fine-tuned, or deployed in new contexts, new vulnerabilities emerge. Anthropic's research involved 3,000+ hours of adversarial testing 鈥?not because they found everything in hour 1, but because sustained, diverse adversarial pressure surfaces failure modes that brief testing misses.[^anthropic] Budget for ongoing red teaming the same way you budget for ongoing monitoring and incident response.

What's the difference between a jailbreak and a prompt injection attack? A jailbreak is an attempt to bypass a model's safety guardrails to elicit prohibited content (e.g., making it generate harmful instructions). A prompt injection attack is an attempt to manipulate the model's behavior by embedding malicious instructions in its input, often to make it ignore its original instructions or leak data. Jailbreaking targets the model's values; prompt injection targets the model's execution logic. Both are Layer 1 attacks, but they require different defenses.

Can small teams do effective red teaming? Yes, but be realistic about scope. Small teams should focus on Layer 1 (automated jailbreak testing using public databases) and Layer 2 (structured behavioral tests for their specific domain). Layer 3 (systemic and emergent failure) requires more resources but can be built gradually. The key is to start testing systematically rather than hoping nothing goes wrong. Use automated tools like PAIR and TAP as force multipliers to extend limited human resources.[^mehrotra]

---

[^inie]: Inie, N., Stray, J., Derczynski, L. "Summon a demon and bind it: A grounded theory of LLM red teaming." PLOS One, 20(1):1-36, 2025.

[^anthropic]: Anthropic. "Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming." Anthropic Research, 2025.

[^mehrotra]: Mehrotra, A., et al. "Tree of Attacks with Pruning." arXiv:2312.02119, 2024.

[^feffer]: Feffer, M., et al. "Red-Teaming for Generative AI: Silver Bullet or Security Theater?" arXiv:2401.15897, 2024.

DarkMental AI maps the practical shadows where testing meets reality.