EXECUTIVE SUMMARY
- Attacks and defenses are benchmarked in isolation. Red-team methods are developed against undefended models, and blue-team methods are developed against fixed corpora of known-bad prompts. Each line of work therefore improves without ever facing the other. No shared framework connects the two, so their scores are not comparable and not trustworthy.
- We designed ACEA as the missing framework that connects red- and blue-teaming. ACEA (Adversarial Co-Evolution Arena) connects a pluggable red adapter and a pluggable blue adapter to a shared target over a minimal HTTP protocol and scores both sides round by round. We open-source ACEA on GitHub (https://github.com/VulcanLab/ACEA) as a research contribution to the AI security community.
Attacks and Defenses Are Benchmarked in Isolation
A jailbreak framework reports success against an undefended model. A guardrail vendor reports block rates against a fixed corpus of known-bad prompts. Each number is internally consistent and externally questionable. The attack was never scored against a live defense, and the defense was never scored against an attacker that could adapt to it.
To fill this gap, we develop ACEA, an open-source framework that connects a red-team project and a blue-team project to the same target and scores both sides round by round.
The ASAP Adapter Protocol
Connecting to ACEA requires no rewrite of an existing tool. We designed the ACEA Standard Adapter Protocol (ASAP) as a minimal HTTP contract that any red- or blue-team project can implement in any language.
Verifiable Attack Success
Without ground truth, attack success and defense effectiveness cannot be verified: when a judge scores free-form harm, no one can tell a real leak from a convincing one.
ACEA seeds the target with synthetic canonical records such as fabricated account numbers, policy codes, and system secrets. If the response reproduces a planted record, we can verify that data is leaked from the target. *The judge still scores dimensions that require judgment, such as policy violation, defense quality, evasion, and stealth.
Scoring the Attack That Was Blocked
ACEA also generates the target’s response to every attack, including attacks that the defense blocked, and scores that response without delivering it. This yields two independent measurements per round: how potent the attack is regardless of interception, and how much harm the defense actually removed.
Each round is attributed to one of four terminating conditions: the defense blocked the attack at the input gate, the defense altered the output, the target refused without help from the defense, or the attack succeeded. This decomposition allows an attributed defense rate, which counts only rounds where the defense actively blocked or filtered, to be reported alongside the raw defense rate. The difference between the two is the credit a guardrail would otherwise receive for the target’s own alignment.
Every round is streamed to a live view and stored in a trace store. The stored traces drive an end-of-battle report that identifies where each failure occurred, with per-round turning points and separate priority lists for the red and blue teams.
A live battle in ACEA: the red team (left), the target (center), and the blue team (right), with the judge and reporter below.
Optional Hints for Adapting Across Rounds
We also provide an optional function that provides hints for adapting across rounds in order to let a red or blue tool learn from what its earlier rounds revealed. Each request therefore carries a hints object computed from prior rounds and enriched from the judge’s verdicts and past traces, so a tool can improve its later rounds without managing any memory of its own. The function is fully optional, and a tool that ignores the hints is scored exactly as it would be without them.
Get the Code
ACEA is open-source, released as a research contribution to the AI security community. The platform is at an early stage, and we are actively adding more functions on top of the functions described here. Contributions are welcome: bug reports, feature ideas, and pull requests alike.
GitHub: https://github.com/VulcanLab/ACEA
Read the full paper on arXiv: https://arxiv.org/abs/2609.08256
