AI Security Review: A Checklist for LLM Systems

An AI security review is an architecture review with a different set of trust boundaries, not a penetration test with AI-flavoured payloads. The distinction decides what the review produces. A pentest answers “what can be broken right now.” An architecture review answers “which of these boundaries is load-bearing, and what happens when it fails.” For LLM systems the second question is the urgent one, because the central boundary — between instructions and data — does not exist inside the model at all.

This checklist is structured as six stages, from scoping to retest cadence, with the evidence to collect at each. It is written for a reviewer sitting down in front of someone else’s LLM application, whether that is an internal system, a vendor product, or an acquisition target.

Stage 1: Inventory before analysis

Reviews that skip inventory produce findings about the component the reviewer happened to be shown. Collect, in writing:

• Every model in the path, including embedding models, rerankers, classifiers and any small model used for routing. Note provider, version pin, and whether the version is pinned at all.
• Every prompt template, including tool descriptions, which are prompts even when they live in code.
• Every data source that reaches a context window: retrieval corpora, tool outputs, file uploads, prior conversation turns, other agents.
• Every tool and its scope. Read the actual permission grant, not the description. A tool named lookup_customer with a database credential that can also write is the finding.
• Every output sink. Where model output lands: a rendered page, a shell, a database, an email, another model’s context.

The output of this stage is a single diagram with a box for each component and a labelled arrow for each data flow. Every arrow that crosses from a lower-trust source to a higher-trust consumer is a candidate finding.

Stage 2: Trust boundaries and the five questions

For each arrow on that diagram, ask:

1. Can an attacker influence what travels on this arrow? Directly, or through a document, ticket, webpage, or email that eventually gets retrieved.
2. Does the receiving component distinguish instruction from data? If the receiver is a model, the answer is no. That is not a bug to be fixed in the prompt; it is a property to be contained architecturally.
3. What is the blast radius if this content is fully attacker-controlled? Enumerate the tools, credentials, and data reachable from that point.
4. What survives the request? Poisoned retrieval entries, memory writes, and cached embeddings persist. A one-shot injection that writes to agent memory becomes a standing compromise.
5. What is logged? If the retrieved chunks and tool calls are not logged, no incident involving this arrow will be reconstructable.

Question three is where most reviews find their highest-severity issue, and it is almost never a model problem. It is an over-scoped credential handed to a tool that a diverted model can call.

Stage 3: Map controls to a published taxonomy

Do not invent a risk taxonomy for the report. Use one the organisation will still recognise in a year. The OWASP Top 10 for LLM Applications 2025 is the working technical vocabulary; walk the system against all ten categories rather than the two everyone talks about. Practical anchor points:

• LLM01 Prompt Injection — direct and indirect. The indirect path is the one that needs a diagram, not a paragraph. Detection options and their limits are covered in how prompt injection detection works.
• LLM02 Sensitive Information Disclosure — check output filters, but check retrieval scoping first, because a filter cannot un-retrieve a document.
• LLM03 Supply Chain — model weights, adapters, and serialized artifacts. Model files execute code on load in several common formats; scanning them is a distinct control from scanning dependencies, as covered in the ModelScan and NB Defense review.
• LLM05 Improper Output Handling — the classic web bugs return here. Model output rendered as HTML is XSS; model output passed to a shell or SQL layer is command or query injection with an unusual source.
• LLM06 Excessive Agency — the tool-scope finding from stage 2, formalised.
• LLM08 Vector and Embedding Weaknesses — permissions lost at ingestion time, covered in depth in RAG pipeline security best practices.
• LLM10 Unbounded Consumption — cost and rate limits, including recursive agent loops that spend without a ceiling.

Controls for each category, with the failure modes of each control, are laid out in the OWASP LLM Top 10 mitigation guide. For adversary tactics rather than risk categories, MITRE ATLAS provides the ATT&CK-style technique catalog scoped to AI systems, which is the better reference when the deliverable needs to describe an attack chain rather than a control gap.

Stage 4: Demand evidence, not assurances

A review that records what the team says the system does is a documentation exercise. Ask for artifacts:

• Scanner output against the deployed configuration, not the base model. An open-source scan run in CI is adequate evidence that someone is looking; the best AI security testing tools of 2026 covers what each class of tool can and cannot see.
• Multi-turn red-team results, because single-turn probes miss agent-specific failures. A framework such as PyRIT exists precisely to compose those sequences.
• Guardrail configuration and its threshold, plus what the system does on a guardrail failure — fail-open and fail-closed are opposite security postures and teams frequently do not know which they shipped.
• Benchmark citations with provenance. If a model was selected on benchmark evidence, check which benchmark measured what; LLM security benchmarks compared covers what each one does and does not cover.
• The retrieval and tool-call logs for a sample request. If they cannot be produced on demand, incident response is theoretical.

When the subject of the review is a vendor product rather than an internal system, the vendor questions in the AI security tool evaluation framework apply directly, and the per-dimension scores in the AI-Sec tool scorecard give a starting comparison for the tools already reviewed here.

Stage 5: Governance mapping

Engineering findings need a governance home or they do not get funded. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, organises actions under the framework’s four functions — Govern, Map, Measure, Manage — and is the usual bridge between a technical finding and an accountable owner. The OWASP LLM Applications Cybersecurity and Governance Checklist is the shorter companion document, written for leadership audiences, and is the more practical artifact to hand to a non-engineering stakeholder.

Two mappings are worth making explicit in every report: which finding belongs to the model provider rather than the team under review, and which residual risk is being accepted rather than mitigated. Both get lost otherwise.

Stage 6: Severity and retest cadence

Conventional severity rubrics under-rate LLM findings because exploitability looks probabilistic. Rate on blast radius and persistence instead:

• Critical — attacker-controlled content reaches a tool with write access to production data or a credential store.
• High — attacker-controlled content reaches a tool with read access to data the requesting user is not entitled to.
• Medium — output handling flaws, over-broad retrieval scope, missing retrieval logging.
• Low — cost and rate-limit gaps with no data-integrity consequence.

Retest cadence is where AI reviews differ most from application reviews. A conventional finding stays fixed until someone changes the code. An LLM finding can regress when nothing in the repository changed, because the provider updated the model, the retrieval corpus grew, or a tool description was reworded. Set retest triggers on model version change, corpus expansion, and any new tool grant, in addition to the calendar.

What reviewers routinely miss

• Tool descriptions as an injection surface. They are attacker-visible in many agent frameworks and they are instructions.
• The second model. Reranker and classifier models take the same untrusted input and are almost never in scope.
• Memory and caches. Agent memory writes and embedding caches carry an injection forward past the session that created it.
• Evaluation contamination. A team’s internal eval set, once committed to a repository that feeds a fine-tune, stops measuring anything.
• Fail-open guardrails under load. A timeout on the guardrail API that defaults to allowing the request is a control that disappears exactly when traffic is unusual.

A review that covers those five, maps findings to OWASP and NIST, and sets retest triggers on model drift will be more useful than one that lists twenty prompt-injection payloads. The payloads change; the boundaries do not.