Arize Phoenix occupies a useful position in the LLM tooling landscape: it’s free to self-host under the Elastic License 2.0, a source-available license that bars offering Phoenix as a managed service; it has production-ready observability features; and it has developed well beyond what “free, self-hosted LLM observability” typically implies.
This review focuses on the security-relevant features (hallucination detection, prompt monitoring and retrieval quality evaluation) and what Arize’s documentation and published evaluator benchmarks say about their accuracy and limits.
The product in brief
Phoenix runs as a local or self-hosted web application. You instrument your LLM application with OpenInference (Arize’s instrumentation library), which captures spans for LLM calls, retrieval steps, tool invocations, and other events. These spans are sent to Phoenix, which provides:
- Trace visualization (full request/response flows)
- LLM evaluation (hallucination, relevance, toxicity scoring via judge LLM)
- Dataset management (golden datasets, golden queries)
- Prompt playground (test prompts against datasets)
- Drift detection (input distribution, output length, score distribution)
The instrumentation is clean:
from phoenix.otel import register
tracer_provider = register(
project_name="my-llm-app",
endpoint="http://localhost:6006/v1/traces",
auto_instrument=True,
)
# auto_instrument=True instruments every library whose
# openinference-instrumentation-<lib> package is installed
# (for example openinference-instrumentation-openai)Auto-instrumentation is opt-in. Called without auto_instrument=True, register() only creates the OpenTelemetry tracer provider, and even with it, OpenAI, LangChain, LlamaIndex and the rest are traced only once their OpenInference instrumentor packages are installed in the application’s environment.
Hallucination detection: how it works and where it falls short
Phoenix’s hallucination evaluation uses a judge LLM to score responses for groundedness — whether claims in the response are supported by the provided context.
The evaluator, in the phoenix-evals 2.x API:
from phoenix.evals import LLM
from phoenix.evals.metrics import HallucinationEvaluator
evaluator = HallucinationEvaluator(llm=LLM(provider="openai", model="gpt-4o"))
scores = evaluator.evaluate({
"input": query,
"output": response,
"context": retrieved_context,
})
# evaluate() returns a list of Score objects
# scores[0].label: "factual" | "hallucinated"Earlier releases ran the same evaluator over a dataframe through run_evals, so match any example you copy to the phoenix-evals version you install.
Performance is bounded by the judge model you point it at and by the quality of the context you pass. LLM-as-a-judge groundedness scoring is a probabilistic classifier, not a deterministic check: it will both miss some unsupported claims and flag some well-grounded responses as hallucinated. Arize documents this limitation and publishes evaluator-benchmark results in its LLM-as-a-judge documentation rather than asking teams to take the accuracy on faith.
Because the false-positive rate is non-trivial for high-volume use, the evaluator is most useful as a sampling tool: run it on a fraction of production traffic and watch aggregate trends, rather than treating any individual score as ground truth.
Retrieval quality evaluation
For RAG applications, Phoenix provides retrieval quality scoring: given a query and retrieved documents, does the retrieval include the information needed to answer the query?
This is distinct from hallucination detection (which asks “is the response grounded in the context?”) — it asks “did the retrieval give the model what it needed?”
This separation is where retrieval-quality scoring earns its place: a class of queries that consistently retrieves tangentially related rather than directly relevant documents will drive a downstream rise in apparent hallucination, and the retrieval-quality score surfaces the root cause before the hallucination metric does. Per Arize’s documentation, that makes it a useful leading indicator for RAG-pipeline regressions that user feedback alone tends to miss.
What the score cannot tell you is whether the querying user was entitled to the documents that came back. That is a permissions question rather than a relevance one, handled at the index: see secure vector database access for the query-time control, and RAG pipeline security best practices for the ingestion-to-generation view.
Prompt monitoring
Phoenix’s trace view shows you the system prompt, user message, and model response for every logged span. For security monitoring, the most useful feature is the ability to search and filter traces:
- Filter by model (catch model version changes)
- Filter by high latency (correlates with context stuffing attacks)
- Filter by high token count (long-context injection attempts)
- Full text search on prompts (find specific injection patterns after the fact)
This is incident response tooling as much as monitoring tooling — when you suspect an attack, you can search historical traces for the pattern.
The self-hosted story
Phoenix runs in Docker with a one-command start:
docker run -p 6006:6006 arizephoenix/phoenixArize’s docs describe adding PostgreSQL persistence and a reverse proxy for production.
The alternative is Arize’s commercial cloud platform, which offers hosted Phoenix with better storage, authentication, and integration with the broader Arize product suite.
Verdict
Phoenix is the best free, self-hostable LLM observability platform (source-available under ELv2 rather than OSI-approved open source) for teams that want to avoid vendor lock-in and have the engineering bandwidth to run their own infrastructure. The evaluation features (hallucination detection, retrieval quality) are more mature than most alternatives at any price point.
The limitations: the evaluator accuracy is good but not sufficient for high-confidence individual decisions; the self-hosted infrastructure requires maintenance; and the security-specific features (attack detection, injection monitoring) are limited compared to dedicated security tools.
For teams that need more than observability and less than a dedicated security platform, combining Phoenix with a dedicated injection detector like Lakera Guard or Rebuff covers the coverage gap. If the requirement is scored evaluation rather than traces, Patronus AI evaluation is the managed alternative to running Phoenix’s evaluators yourself.
For the methodology behind these comparisons, see our AI security tool evaluation framework, which covers the full tooling landscape, and the interactive AI-Sec tool scorecard to rank Phoenix against the security-first tools on weights you set. Observability output is also one of the evidence artifacts an assessor should ask for — see the AI security review checklist.
