PyRIT Review: Microsoft's AI Red Teaming Framework

PyRIT, the Python Risk Identification Tool for generative AI, is an open-source framework whose stated purpose is to “empower security professionals and engineers to proactively identify risks in generative AI systems.” It came out of Microsoft’s AI Red Team, a group with real production AI security experience, and the design reflects that origin: it is built around the workflow of a security engineer running repeatable assessments, not around one-shot research breadth.

One provenance note that matters for anyone pinning dependencies or following links: PyRIT now lives at microsoft/PyRIT. The older Azure/PyRIT repository has been archived and redirects to the Microsoft org. It is released under the MIT license.

The comparison with garak is the one most teams reach for, and it is the right frame. Garak has more probes and is oriented toward comprehensive research scanning. PyRIT has better workflow integration, better result management, and was designed from the start for “security engineer running repeatable tests on AI applications.” This review covers the architecture, the multi-turn capability that justifies the extra setup, and where it is the wrong tool. All capabilities below are drawn from PyRIT’s repository and documentation.

Architecture

PyRIT composes red-team runs from a small set of well-defined building blocks. Understanding these five is most of understanding the tool.

Prompt targets are the thing being tested: the model or endpoint that receives prompts. PyRIT abstracts targets so the same attack logic can point at different backends, including OpenAI-style chat endpoints, Azure OpenAI, and other API endpoints. Targets can also serve secondary roles, for example a target used by a scorer to perform an LLM-as-judge evaluation.

Orchestrators are the engine that drives an assessment. They take a set of prompts, or a strategy for generating them, send them to a target through any configured converters, and route responses to scorers. The orchestrator owns the interaction loop, which is where single-turn versus multi-turn behaviour is decided. This is the component that turns a loose set of attack prompts into a structured campaign with scored results.

Prompt converters transform prompts before they reach the target, and they are PyRIT’s most distinctive idea. Rather than hand-writing every variant of an attack, you apply converters to mutate a base prompt: encoding it, translating it, rephrasing it, or otherwise transforming it to probe whether the transformation slips past a model’s defenses. Converters chain, and they expand a small seed set into broad coverage automatically.

Scorers decide whether an attack succeeded. PyRIT supports programmatic evaluation including LLM-as-judge scoring against a rubric and simpler matching approaches. Because scoring is a first-class pluggable component, you define what success means for your specific risk and get consistent, machine-readable verdicts rather than eyeballing transcripts.

Memory persists what happens. The memory subsystem stores prompts, responses and scores to a database (SQLite by default, easily swapped for PostgreSQL or Azure SQL) so runs are durable and comparable over time. Run an assessment, change the model, run it again, compare: did a model update change behaviour on a specific attack class? Memory also underpins multi-turn attacks, because conversation context has to be tracked across turns.

A basic red team run:

from pyrit.orchestrator import PromptSendingOrchestrator
from pyrit.prompt_target import AzureOpenAIChatTarget
from pyrit.prompt_converter import TranslationConverter
from pyrit.datasets import fetch_harmbench_examples

target = AzureOpenAIChatTarget()

orchestrator = PromptSendingOrchestrator(
    prompt_target=target,
    prompt_converters=[TranslationConverter(language="Spanish")]
)

harmbench_prompts = fetch_harmbench_examples(harm_category="physical_safety")

result = await orchestrator.send_prompts_async(
    prompt_list=harmbench_prompts
)

Converters are the force multiplier

It is worth dwelling on converters, because they are the component that most distinguishes PyRIT’s philosophy. The naive way to build an attack corpus is to hand-write every variation: the base request, the base64-encoded version, the translated version, the leetspeak version, the politely reframed version. That does not scale, and it goes stale the moment a model’s defenses shift.

PyRIT inverts this. You write or pick a base prompt and apply transformations programmatically. Because converters are composable and chainable, a handful of them multiply a small seed set into broad coverage. Encode it, then translate it, then reframe it: each combination is a distinct test, generated rather than authored. Two things follow.

Coverage per unit of effort. A small, well-chosen seed set plus a converter chain explores far more of the attack space than the same effort spent writing static variants, and it stays maintainable: improve the seed or add a converter, and every downstream combination updates.

Probing the boundary between instruction and obfuscation. Many real bypasses work by obfuscating an otherwise-blocked request until a classifier, or the model itself, no longer recognises it as prohibited. Converters operationalise exactly that class of probe, which is why they fit a tool built by people who run these engagements.

The trade-off to be honest about: converter chains generate a large volume of prompts quickly, so you scope them deliberately and lean on scorers to triage results rather than reading every transcript by hand.

Single-turn versus multi-turn, in practice

“Multi-turn support” gets thrown around loosely, so it is worth being concrete about what it buys. A single-turn test sends one prompt and judges one response, which is fine for “does this jailbreak string work.” But a meaningful fraction of real failures only appear across a conversation: a model that refuses a direct request and then complies after being incrementally reframed over several exchanges, or an agent slowly steered off its task.

Because PyRIT’s orchestrators own the interaction loop and memory tracks conversation state across turns, the framework can represent these conversational attack patterns. In practice you can build an orchestrator that adapts subsequent prompts based on prior responses, and the whole exchange, every turn with its scores, lands in memory for later review. A single-turn scanner structurally cannot model this; it has no notion of state between requests. For teams red-teaming conversational assistants or agents specifically, this is the capability that justifies reaching for PyRIT over a simpler tool.

Why it is a security-team tool, not a research scanner

The recurring theme is that PyRIT is engineered for a process, not a one-off scan:

• Repeatability. The orchestrator, target, converter and scorer composition is reusable. Build an assessment once and run it on a schedule or before releases.
• Trackability. Memory makes results durable and comparable, which is what regression detection requires.
• Programmatic success criteria. Pluggable scorers mean “did it work” is defined explicitly and evaluated consistently, not judged ad hoc.
• CI friendliness. A focused assessment on one attack category is a normal Python run that fits into a pipeline, finishing in minutes rather than the hours a full research sweep takes.

That orientation costs breadth. PyRIT’s curated attack and converter set is narrower than a maximalist scanner’s probe library. The trade-off is deliberate: focused, trackable, repeatable testing over exhaustive one-time coverage.

Coverage comparison with garak

PyRIT’s probe coverage is narrower than garak’s but more curated:

• Jailbreak attacks: comparable coverage of known patterns
• Prompt injection: good coverage, including multi-turn patterns garak cannot represent
• Data leakage: more focused than garak
• Encoding-based attacks: less comprehensive than garak’s encoding probes
• Research-oriented probes (GCG variants, transfer attacks): garak wins here

A mature program uses both: garak for periodic breadth sweeps, PyRIT for the ongoing, integrated assessment loop.

Enterprise and Azure context

PyRIT works against any compatible API endpoint, so it is not Azure-locked. That said, for organizations already running LLM applications on Azure, the Azure OpenAI target support and the fit with Microsoft’s broader security tooling are real conveniences: shared identity, logging to Azure Monitor, and integration with Defender for Cloud AI security findings all reduce setup friction. For non-Azure deployments the Azure-specific integrations are simply irrelevant rather than an obstacle.

Practical adoption notes

Start with one orchestrator and one scorer. The component model is powerful but can feel abstract. Build a single working PromptSendingOrchestrator-style run with a clear scorer before composing converters and multi-turn flows.

Treat memory as an asset. Point it at a durable store and keep your run history; the comparative value compounds over releases.

Define success per risk. The default scorers are a starting point. The payoff comes from scorers that encode what failure actually means for your application.

Verdict

PyRIT is the right choice for security teams, as opposed to ML research teams, running regular repeatable assessments of LLM applications, especially where multi-turn attacks, result tracking and CI integration matter. The architecture is coherent, memory and scoring make it a process tool rather than a one-off, and the converter model is an elegant way to expand coverage from a small seed set.

It is the less natural fit if you want maximal one-time probe breadth with minimal setup, which is garak’s lane. For most teams the answer is not either/or: PyRIT for the assessment loop, garak for breadth sweeps, and a validation layer like Giskard for application and RAG-level testing.

PyRIT’s multi-turn orchestration is also the capability that public benchmarks are weakest at approximating: the agent-focused datasets described in LLM security benchmarks compared score models on fixed task sets, while PyRIT composes sequences against your own system. Where that output belongs in a formal assessment is covered in the AI security review checklist, and the surrounding tool landscape in best AI security testing tools 2026.

Both were assessed against our AI security tool evaluation framework. For the operational reference on orchestrator, converter and scorer configuration rather than the buying decision, bestllmscanners.com’s PyRIT explainer covers the mechanics, and its scanner comparison data puts PyRIT against garak and the commercial options side by side.