EXECUTIVE SUMMARY
Automated LLM red-teaming has moved to multi-turn attacks, yet three gaps still keep results from reflecting real risk:
- No cost/budget realism. Existing multi-turn methods escalate with no regard for the interaction budgets real deployments impose (rate limits, session caps, anti-abuse), so reported success can take far more turns than an attacker actually gets.
- A single success rate. One number counts a vague half-leak the same as complete, operational instructions, hiding how actionable the output really is.
- A single biased judge. LLM-as-judge scoring carries documented positional, verbosity, and self-preference biases.
AMT-X (Adaptive Multi-Turn Exploitation) closes all three. It runs a battle-tested, multi-phase attack, budgets the interaction per phase, and uses a multi-armed-bandit optimizer to select effective techniques, then scores each result with two attack success rates: a lenient one for any leak, and a strict one for output that is realistic and actionable (complete, real, operational), both decided by a multi-model jury rather than a single judge. Across six frontier models, headline success reached 97.6% to 100%, but only 66.7% to 78.6% was fully actionable, a gap of up to 33 percentage points. No model was consistently immune.
What AMT-X Does
AMT-X rebuilds red-teaming around five ideas:
- A battle-tested, multi-phase approach. The attack unfolds as a structured sequence of phases, distilled from real red-team engagements that succeeded against deployed, tool-using AI assistants, rather than improvised, one-off prompts. It’s a proven playbook, not guesswork.
- A budget for every phase. Each phase runs under a defined interaction budget, mirroring the real limits attackers face (rate limits, session caps, anti-abuse heuristics). That keeps the probe realistic and efficient: pressure is applied where it counts, not by brute force.
- Optimization that learns what works. A multi-armed-bandit optimizer continuously picks the most effective techniques from a large library, balancing proven winners against under-tried options, so the framework gets sharper as it goes instead of repeating what fails.
- Two success rates, not one. AMT-X defines and reports two attack success rates side by side: a lenient one for any leak, and a strict one for output that is genuinely realistic and actionable (complete, real, and operational). It’s the difference between “the model said something it shouldn’t” and “the model handed over something someone could actually use.”
- Unbiased grading. Both rates are decided by a jury of different models that deliberate as grader, critic, and defender, replacing the single biased judge most benchmarks rely on.
What We Found
- The honesty gap. Headline success reached 6% to 100%, but only 66.7% to 78.6% (average ~71%) of those were fully actionable, a gap of up to 33 percentage points that the usual single score conceals.
- No safe harbor. Tested across six frontier models from every major lab, none achieved consistent immunity.
- Depth is the real driver. Kept to short exchanges, success stayed low, around 29% to 36%. Allowed to run as longer, adaptive conversations, it jumped to ~98%. The danger isn’t the first prompt; it’s the tenth.
This last finding points straight at a defense: limiting how deep an adaptive conversation can go (through turn ceilings, escalation checks, and anomaly-driven session limits) is a promising lever for cutting real-world risk.
Why It Matters, and How AMT-X Is Different
AMT-X is the only framework that combines a battle-tested multi-phase attack structure, per-phase budgets, bandit-optimized technique selection, adaptive semantic signals, a multi-role jury, and an actionability-gated success test in one place.
Comparison of AMT-X against representative automated jailbreak frameworks
Responsible Research
This work was conducted exclusively for defensive purposes. Findings are reported in aggregate; no attack transcripts, payloads, or harmful instructions are released. Real-world engagements that motivated the framework followed responsible-disclosure practice, and the issues involved have since been remediated. The goal is a more honest, rigorous standard for measuring AI safety, so defenses can be built where they are actually needed.
GitHub: https://github.com/VulcanLab/amt-x
Read the full paper on arXiv: https://arxiv.org/abs/2607.11151
