PyRIT, the Python Risk Identification Tool for generative AI, is an open-source framework whose stated purpose is to “empower security professionals and engineers to proactively identify risks in generative AI systems.” It came out of Microsoft’s AI Red Team, and the design reflects that origin: it is built around the workflow of a security engineer running repeatable assessments, not around one-shot research breadth.
Two provenance notes matter for anyone pinning dependencies or following links. PyRIT now lives at microsoft/PyRIT; the older Azure/PyRIT repository has been archived and redirects to the Microsoft org, and it is released under the MIT license. And the 1.x line reorganised the public API: as of v1.1.0, released 2026-09-04, there is no pyrit.orchestrator package. Orchestrators are now attack strategies under pyrit.executor.attack (PromptSendingOrchestrator became PromptSendingAttack), converters moved to pyrit.converter, and AzureOpenAIChatTarget is gone in favour of the single OpenAIChatTarget. Code written against a 0.x example will not import on a current install, so pin the version your examples were written for or port them.
The comparison with garak is the one most teams reach for, and it is the right frame. Garak has more probes and is oriented toward comprehensive research scanning. PyRIT has better workflow integration, better result management, and was designed from the start for “security engineer running repeatable tests on AI applications.” This review covers the architecture, the multi-turn capability that justifies the extra setup, and where it is the wrong tool. All capabilities below are drawn from PyRIT’s repository and documentation.
Architecture
PyRIT composes red-team runs from a small set of well-defined building blocks. Understanding these five is most of understanding the tool.
Prompt targets are the thing being tested: the model or endpoint that receives prompts. PyRIT abstracts targets so the same attack logic can point at different backends, including OpenAI-style chat endpoints, Azure OpenAI, and other API endpoints. Targets can also serve secondary roles, for example a target used by a scorer to perform an LLM-as-judge evaluation.
Attack strategies (called orchestrators before the 1.x rename) are the engine that drives an assessment. They take an objective, or a set of them, send prompts to a target through any configured converters, and route responses to scorers. The attack owns the interaction loop, which is where single-turn versus multi-turn behaviour is decided. This is the component that turns a loose set of attack prompts into a structured campaign with scored results, and AttackExecutor runs one strategy across many objectives in parallel.
Prompt converters transform prompts before they reach the target, and they are PyRIT’s most distinctive idea. Rather than hand-writing every variant of an attack, you apply converters to mutate a base prompt: encoding it, translating it, rephrasing it, or otherwise transforming it to probe whether the transformation slips past a model’s defenses. Converters chain, and they expand a small seed set into broad coverage automatically.
Scorers decide whether an attack succeeded. PyRIT supports programmatic evaluation including LLM-as-judge scoring against a rubric and simpler matching approaches. Because scoring is a first-class pluggable component, you define what success means for your specific risk and get consistent, machine-readable verdicts rather than eyeballing transcripts.
Memory persists what happens. The memory subsystem stores prompts, responses and scores to a database (SQLite by default, or Azure SQL with Entra ID authentication) so runs are durable and comparable over time. Run an assessment, change the model, run it again, compare: did a model update change behaviour on a specific attack class? Memory also underpins multi-turn attacks, because conversation context has to be tracked across turns.
A basic red team run:
import asyncio
from pyrit.converter import TranslationConverter
from pyrit.executor.attack import (
AttackConverterConfig,
AttackExecutor,
PromptSendingAttack,
)
from pyrit.prompt_normalizer import ConverterConfiguration
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
async def main() -> None:
await initialize_pyrit_async(IN_MEMORY)
target = OpenAIChatTarget()
attack = PromptSendingAttack(
objective_target=target,
attack_converter_config=AttackConverterConfig(
request_converters=ConverterConfiguration.from_converters(
converters=[TranslationConverter(converter_target=target, language="Spanish")]
)
),
)
results = await AttackExecutor().execute_attack_async(
attack=attack,
objectives=["<your first objective>", "<your second objective>"],
)
for result in results:
print(result.outcome, result.objective)
asyncio.run(main())Two shapes changed with the API: objectives are plain strings you supply rather than rows pulled from a bundled dataset helper, and converters are wrapped in an AttackConverterConfig instead of passed as a bare list.
Converters are the force multiplier
It is worth dwelling on converters, because they are the component that most distinguishes PyRIT’s philosophy. The naive way to build an attack corpus is to hand-write every variation: the base request, the base64-encoded version, the translated version, the leetspeak version, the politely reframed version. That does not scale, and it goes stale the moment a model’s defenses shift.
PyRIT inverts this. You write or pick a base prompt and apply transformations programmatically. Because converters are composable and chainable, a handful of them multiply a small seed set into broad coverage. Encode it, then translate it, then reframe it: each combination is a distinct test, generated rather than authored. Two things follow.
Coverage per unit of effort. A small, well-chosen seed set plus a converter chain explores far more of the attack space than the same effort spent writing static variants, and it stays maintainable: improve the seed or add a converter, and every downstream combination updates.
Probing the boundary between instruction and obfuscation. Many real bypasses work by obfuscating an otherwise-blocked request until a classifier, or the model itself, no longer recognises it as prohibited. Converters operationalise exactly that class of probe, which is why they fit a tool built by people who run these engagements.
The trade-off to be honest about: converter chains generate a large volume of prompts quickly, so you scope them deliberately and lean on scorers to triage results rather than reading every transcript by hand.
Single-turn versus multi-turn, in practice
“Multi-turn support” gets thrown around loosely, so it is worth being concrete about what it buys. A single-turn test sends one prompt and judges one response, which is fine for “does this jailbreak string work.” But a meaningful fraction of real failures only appear across a conversation: a model that refuses a direct request and then complies after being incrementally reframed over several exchanges, or an agent slowly steered off its task.
Because PyRIT’s attack strategies own the interaction loop and memory tracks conversation state across turns, the framework can represent these conversational attack patterns. In practice you can build an attack strategy that adapts subsequent prompts based on prior responses, and the whole exchange, every turn with its scores, lands in memory for later review. A single-turn scanner structurally cannot model this; it has no notion of state between requests. For teams red-teaming conversational assistants or agents specifically, this is the capability that justifies reaching for PyRIT over a simpler tool.
Why it is a security-team tool, not a research scanner
The recurring theme is that PyRIT is engineered for a process, not a one-off scan:
- Repeatability. The attack, target, converter and scorer composition is reusable. Build an assessment once and run it on a schedule or before releases.
- Trackability. Memory makes results durable and comparable, which is what regression detection requires.
- Programmatic success criteria. Pluggable scorers mean “did it work” is defined explicitly and evaluated consistently, not judged ad hoc.
- CI friendliness. A focused assessment on one attack category is a normal Python run that fits into a pipeline, finishing in minutes rather than the hours a full research sweep takes.
That orientation costs breadth. PyRIT’s curated attack and converter set is narrower than a maximalist scanner’s probe library. The trade-off is deliberate: focused, trackable, repeatable testing over exhaustive one-time coverage.
Coverage comparison with garak
PyRIT’s probe coverage is narrower than garak’s but more curated:
- Jailbreak attacks: comparable coverage of known patterns
- Prompt injection: good coverage, including multi-turn patterns garak cannot represent
- Data leakage: more focused than garak
- Encoding-based attacks: less comprehensive than garak’s encoding probes
- Research-oriented probes (GCG variants, transfer attacks): garak wins here
A mature program uses both: garak for periodic breadth sweeps, PyRIT for the ongoing, integrated assessment loop.
Enterprise and Azure context
PyRIT works against any compatible API endpoint, so it is not Azure-locked. For Azure shops, the Azure OpenAI endpoint support and the Azure SQL memory backend with Entra ID authentication are real conveniences. PyRIT itself has no Azure Monitor or Defender for Cloud integration: any alerts an assessment trips come from monitoring that already protects the target endpoint, not from PyRIT. For non-Azure deployments the Azure-specific options are simply irrelevant rather than an obstacle.
Practical adoption notes
Start with one attack and one scorer. The component model is powerful but can feel abstract. Build a single working PromptSendingAttack run with a clear scorer before composing converters and multi-turn flows.
Treat memory as an asset. Point it at a durable store and keep your run history; the comparative value compounds over releases.
Define success per risk. The default scorers are a starting point. The payoff comes from scorers that encode what failure actually means for your application.
Verdict
PyRIT is the right choice for security teams, as opposed to ML research teams, running regular repeatable assessments of LLM applications, especially where multi-turn attacks, result tracking and CI integration matter. The architecture is coherent, memory and scoring make it a process tool rather than a one-off, and the converter model is an elegant way to expand coverage from a small seed set.
It is the less natural fit if you want maximal one-time probe breadth with minimal setup, which is garak’s lane. For most teams the answer is not either/or: PyRIT for the assessment loop, garak for breadth sweeps, and a validation layer like Giskard for application and RAG-level testing.
Teams that would rather buy the whole assessment-and-runtime layer than assemble it should read the Cisco AI Defense review alongside this one: it covers the consolidated commercial alternative to running PyRIT and garak yourself. Published benchmarks of the same tool category are collected at AI Sec Bench.
PyRIT’s multi-turn orchestration is also the capability that public benchmarks are weakest at approximating: the agent-focused datasets described in LLM security benchmarks compared score models on fixed task sets, while PyRIT composes sequences against your own system. Where that output belongs in a formal assessment is covered in the AI security review checklist, and the surrounding tool landscape in best AI security testing tools 2026.
Both were assessed against our AI security tool evaluation framework. For the operational reference on attack, converter and scorer configuration rather than the buying decision, bestllmscanners.com’s PyRIT explainer covers the mechanics, and its scanner comparison data puts PyRIT against garak and the commercial options side by side.
