LLM Security Benchmarks Compared: What Each Measures

An LLM security benchmark is not a scanner, and the difference is the source of most confusion in vendor conversations. A scanner such as garak or PyRIT fires attacks at your system and reports what got through. A benchmark is a fixed dataset plus a scoring procedure, run against a model, so that different attacks, defenses, or model versions can be compared on identical inputs. One tells you whether your deployment is exposed. The other tells you whether a model, in isolation, behaves better or worse than another model on a standardised set of prompts.

That distinction matters commercially, because “scores well on HarmBench” is a claim about a base model under a published attack set, not a claim about the application you are about to ship. This comparison covers the benchmarks that show up most often in AI security procurement documents, what each one actually measures, and the questions worth asking before a benchmark number changes a buying decision.

The four parts of any security benchmark

Every benchmark in this article decomposes the same way, and reading a claim means knowing which part is being quoted.

1. The behavior set. The prompts, tasks, or environments used as inputs. Size, sourcing, and category coverage vary enormously.
2. The target. A raw model, a model behind a system prompt, a model wired to tools, or a full agent loop. Results are not transferable between these.
3. The attack budget. Whether the attacker gets one shot, a fixed number of queries, white-box gradient access, or an adaptive multi-turn conversation.
4. The judge. What decides that an output counts as a failure — a keyword list, a fine-tuned classifier, another LLM, or a human. Judges disagree with each other far more than most result tables admit.

A benchmark number without all four is not interpretable. This is the same discipline described in the AI security tool evaluation framework used for product reviews on this site, applied to research artifacts instead of products.

Refusal and jailbreak benchmarks

These measure whether a model complies with harmful requests, and how much attacker effort it takes to make it comply.

HarmBench is the closest thing the field has to a common yardstick for automated red teaming. The HarmBench paper presents a standardized evaluation framework and reports a large-scale comparison of 18 red teaming methods against 33 target LLMs and defenses, alongside an adversarial training method the authors propose off the back of it. Its value is that attack methods and defenses are scored under one harness, so a claim that attack A beats attack B has a controlled basis. Its limitation is scope: HarmBench evaluates refusal behavior on a fixed behavior set, which says nothing about a model’s resistance to injection through retrieved documents or tool output.

JailbreakBench takes the reproducibility problem head-on. Per the JailbreakBench project, it was introduced at the NeurIPS Datasets and Benchmarks track in 2024 and centers on JBB-Behaviors — 100 distinct harmful behaviors paired with 100 benign behaviors, organized into ten categories aligned with OpenAI’s usage policies. Two design choices make it more useful than a bare leaderboard. First, it publishes the adversarial prompt artifacts themselves rather than just scores, so a result can be re-run instead of taken on trust. Second, the benign half of the dataset means over-refusal is measurable in the same run as under-refusal. A defense that blocks every jailbreak by refusing everything should not score well, and JailbreakBench is structured so it does not.

The practical read: HarmBench is the better reference when comparing attack methods; JailbreakBench is the better reference when comparing defenses, because it prices in false positives. That trade-off is the same one that separates a usable runtime guardrail from an unusable one, as covered in how prompt injection detection works.

Agent security benchmarks

Once a model can call tools, the threat model changes: the attacker no longer needs to talk to the model directly, because content the agent retrieves becomes an instruction channel.

AgentDojo targets exactly that. The AgentDojo paper describes a dynamic evaluation environment populated with 97 realistic tasks and 629 security test cases, built to score prompt injection attacks and defenses against tool-using agents rather than against chat completions. It is dynamic rather than static: the agent executes tasks in simulated environments, and the security test cases check whether injected content in tool output diverts it. For any team running retrieval or tool-calling in production, this is the benchmark family whose failure modes most resemble their own, and it pairs directly with the ingestion and retrieval controls in RAG pipeline security best practices.

AgentHarm asks a different question: not whether an agent can be diverted, but whether a jailbroken agent stays competent. The AgentHarm benchmark, published at ICLR 2025, consists of 110 explicitly malicious agent tasks — 440 counting augmentations — across 11 harm categories, and scores both refusal and task completion. Its reported findings are the interesting part: leading models were found surprisingly compliant with malicious agent tasks even without jailbreaking, simple universal templates were often enough to compromise them, and jailbroken agents retained the capability to carry out coherent multi-step malicious work rather than degrading into incoherence. That last result undercuts a common defensive assumption — that a jailbroken model produces harmful text but not harmful action.

Cyber-capability and governance benchmarks

CyberSecEval 3, from Meta’s Purple Llama project, is scoped to cybersecurity risk specifically rather than harm in general. Its technical report states that the suite assesses eight risks across two categories — risk to third parties, and risk to application developers and end users — and adds offensive-capability areas covering automated social engineering, scaling manual offensive cyber operations, and autonomous offensive cyber operations. It is the right reference when the question is “could this model meaningfully uplift an attacker,” and the wrong one when the question is “will this model leak my system prompt.”

AILuminate, from MLCommons, sits at the governance end. The AILuminate benchmark is described as a family of safety and security benchmarks assessing generative AI across 12 hazard categories, with published figures of 59,624 test prompts and 477 test images across 109 evaluated models. It reports graded results rather than raw pass rates, which makes it legible to non-specialist stakeholders and correspondingly coarse for engineering decisions. Treat it as evidence for a risk register, not as a control validation.

Side-by-side

Benchmark	Target	Measures	Best used for
HarmBench	Base and defended models	Refusal robustness under 18 automated red-team methods	Comparing attack methods on a common harness
JailbreakBench	Models and defenses	Jailbreak success plus over-refusal on 100 benign behaviors	Comparing defenses without hiding false positives
AgentDojo	Tool-using agents	Prompt injection via tool output, 629 security test cases	Threat-modelling retrieval and tool-calling systems
AgentHarm	Tool-using agents	Refusal and retained capability on 110 malicious tasks	Testing whether jailbroken agents stay effective
CyberSecEval 3	Models	Eight cyber risks, including offensive uplift	Model-selection risk reviews and policy sign-off
AILuminate	Models	12 hazard categories, graded	Governance reporting and risk registers

What none of them measure

Every benchmark here scores a model or an agent harness. None of them score your application, and the gap between those two is where real incidents live:

• Your system prompt and its leakage surface. Benchmarks generally use a neutral or absent system prompt. Yours contains business logic, tool descriptions, and sometimes credentials-adjacent context.
• Your retrieval corpus. A poisoned internal document is an attack path no public dataset contains.
• Your tool scope. The blast radius of a diverted agent is set by which tools you granted it, not by the model.
• Your guardrail configuration. Thresholds, allowlists, and fallback behavior are deployment choices. Two teams on the same model and the same vendor guardrail can have very different exposure.
• Version drift. A benchmark result is a snapshot of one model version. Provider-side updates invalidate it silently.

This is why benchmark scores belong in the model-selection stage and scanner output belongs in the release gate. The best AI security testing tools of 2026 covers the tooling that closes the second half of that loop, and the OWASP LLM Top 10 mitigation guide maps the control classes a benchmark can never validate for you.

Reading a vendor benchmark claim

When a vendor cites a benchmark, five questions separate a real result from a marketing figure:

1. Which split, and which version? Public test sets get contaminated. A model trained after a benchmark’s release may have seen it.
2. Which judge? LLM-as-judge scoring is common and cheap, and it drifts with the judge model. Ask which judge, at which version.
3. What was the attack budget? A defense that holds for one query and falls at fifty is a different product from one that holds throughout.
4. Was the target the raw model or the full product? Vendors sometimes benchmark the model and market the platform.
5. Is the run reproducible? JailbreakBench-style published artifacts allow verification. A number in a slide deck does not.

The same five questions apply to the tool reviews on this site, which is why every review lists the sources a verdict rests on and why nothing here is presented as a measurement of its own. To weigh individual tools against your own priorities rather than a vendor’s, the interactive AI-Sec tool scorecard exposes the per-dimension scores and the evidence behind each one. For the wider process these numbers feed into, see the AI security review checklist.