What is Arize Phoenix strongest at according to the lesson?

Deep agent-specific evaluation: trace clustering, anomaly detection, RAG retrieval relevancy, OpenInference auto-instrumentation

What does the lesson say about vendor-published platform benchmarks?

Take them as directional; measure your own corpus

They are forbidden by OpenTelemetry

Why is tracing without evaluation considered expensive logging?

You see runs but you do not score them, so regressions are invisible and bisection is impossible

What is the lesson's warning about self-rolled LLM-judges?

CRITIC applies: judges need external grounding for factual verification or they rubber-stamp

They are forbidden by Apache 2.0

Agent Observability: Langfuse, Phoenix, Opik

Three self-hostable agent observability platforms dominate 2026. Langfuse (MIT) — 6M+ installs/month, tracing + prompt management + evals + session replay. Arize Phoenix (Elastic 2.0) — deep agent-specific evals, RAG relevancy, OpenInference auto-instrumentation. Comet Opik (Apache 2.0) — automated prompt optimization, guardrails, LLM-judge hallucination detection.

Lesson 24 of 54Learn~45 minutesPython (stdlib)

Before this: Lesson 23 of Phase 14 (OTel GenAI)

On this page

Learning Objectives

Name the three top self-hostable agent observability platforms and their licenses, and say which license is not OSI open source.
Distinguish what each one is strongest at: Langfuse (prompt mgmt + sessions), Phoenix (RAG + auto-instrumentation), Opik (optimization + guardrails).
Explain why 89% of organizations report having agent observability in place by 2026.
Implement a stdlib trace-to-dashboard pipeline with LLM-judge evaluation.

The Problem

OTel GenAI (Lesson 23) gives you the schema. You still need the platform that ingests spans, runs evaluations, stores prompt versions, and surfaces regressions. The three contenders each emphasize different parts of the lifecycle.

The Concept

Langfuse (MIT)

6M+ SDK installs/month, 19k+ GitHub stars.
Features: tracing, prompt management with versioning + playground, evaluations (LLM-as-judge, user feedback, custom), session replays.
June 2025: formerly commercial modules (LLM-as-a-judge, annotation queues, prompt experiments, Playground) open-sourced under MIT.
Strongest for: end-to-end observability with tight prompt-management loop.

Arize Phoenix (Elastic License 2.0)

Deeper agent-specific evaluation: trace clustering, anomaly detection, retrieval relevancy for RAG.
Native OpenInference auto-instrumentation.
Pairs with managed Arize AX for production.
No prompt versioning — positioned as a drift/behavioral-regression tool alongside broader platforms.
Strongest for: RAG relevancy, behavioral drift, anomaly detection.

Comet Opik (Apache 2.0)

Automated prompt optimization through A/B experiments.
Guardrails (PII redaction, topical constraints).
LLM-judge hallucination detection.
Benchmark from Comet's own measurement: Opik logs + evals in 23.44s vs Langfuse 327.15s (~14x gap) — take vendor benchmarks as directional.
Strongest for: optimization loop, automated experimentation, guardrail enforcement.

Industry data

Per Maxim (2026 field analysis): 89% of organizations have agent observability in place; quality issues are the top production barrier (32% of respondents cite them).

Picking one

Need	Pick
All-in-one with prompt management	Langfuse
Deep RAG evaluation + drift	Phoenix
Automated optimization + guardrails	Opik
Open licensing, no ELv2	Langfuse (MIT) or Opik (Apache 2.0)
Datadog / New Relic integration	Any — they all export OTel

Where this pattern goes wrong

No eval strategy. Tracing without evaluation is just expensive logging.
Self-rolled LLM-judge without grounding. CRITIC pattern (Lesson 05) applies — judges need external tools for factual verification.
Prompt versions not tied to traces. When prod regresses, you cannot bisect to the prompt that caused it.

wb-trace-ingest

Build It

code/main.py implements a stdlib trace collector + LLM-judge evaluator:

Ingest GenAI-shaped spans.
Group by session, tag failed runs (guardrail trips, low-confidence evals).
A scripted LLM-judge that scores agent responses on a rubric.
A dashboard-like summary: failure rate, top failure reasons, eval score distribution.

Run it:

python3 code/main.py

Output: per-session eval scores and failure categorization matching what Langfuse/Phoenix/Opik would show.

Use It

Langfuse self-hosted or cloud; wire via OTel or their SDK.
Arize Phoenix self-hosted; auto-instrument OpenInference.
Comet Opik self-hosted or cloud; automated optimization loop.
Datadog LLM Observability for mixed ops+ML teams that already run Datadog.

Ship It

outputs/skill-obs-platform-wiring.md picks a platform and wires traces + evals + prompt versions into an existing agent.

Exercises

Export a week of OTel traces to Langfuse cloud (free tier). Which sessions failed? Why?
Write an LLM-judge rubric for your domain (factual correctness, tone, scope adherence). Test on 50 traces.
Compare Langfuse prompt versioning against Phoenix's trace clustering. Which tells you what broke faster?
Read Opik's guardrail docs. Wire a PII redaction guardrail to one of your agent runs.
Benchmark the three on your corpus. Ignore vendor-published numbers; measure your own.

Key Terms

Term	What people say	What it actually means
Tracing	"Spans collector"	Ingest OTel / SDK spans; index by session
Prompt management	"Prompt CMS"	Versioned prompts tied to traces
LLM-as-judge	"Automated eval"	Separate LLM scores agent output against a rubric
Session replay	"Trace playback"	Step through past runs for debugging
RAG relevancy	"Retrieval quality"	Does the retrieved context match the query
Trace clustering	"Behavioral grouping"	Cluster similar runs for drift detection
Guardrail enforcement	"Policy at log time"	PII/toxicity/scope checks on logged content

Quick quiz

What license does Langfuse ship under after the June 2025 open-sourcing of formerly commercial modules?
1. Apache 2.0
2. GPLv3
3. Elastic License 2.0
4. MIT
Show the answer
D. MIT Langfuse is MIT, including LLM-as-a-judge, annotation queues, prompt experiments, and Playground after June 2025.
What is Arize Phoenix strongest at according to the lesson?
1. Deep agent-specific evaluation: trace clustering, anomaly detection, RAG retrieval relevancy, OpenInference auto-instrumentation
2. Automated optimization loop
3. Prompt versioning
4. Static analysis of code
Show the answer
A. Deep agent-specific evaluation: trace clustering, anomaly detection, RAG retrieval relevancy, OpenInference auto-instrumentation Phoenix focuses on behavioral drift and RAG evaluation with OpenInference auto-instrumentation.
Which platform pairs automated prompt optimization with guardrails (PII redaction, topical constraints) and LLM-judge hallucination detection?
1. Langfuse
2. Phoenix
3. Jaeger
4. Opik
Show the answer
D. Opik Opik centers on the optimization + guardrail loop.
Which platform does the lesson recommend for an all-in-one with prompt management?
1. Opik
2. Datadog APM
3. Phoenix
4. Langfuse
Show the answer
D. Langfuse Langfuse covers tracing + prompt management + evals + session replay end-to-end.
What does the lesson say about vendor-published platform benchmarks?
1. They are definitive
2. Take them as directional; measure your own corpus
3. Only Opik's are reliable
4. They are forbidden by OpenTelemetry
Show the answer
B. Take them as directional; measure your own corpus Even the cited 14x gap between Opik and Langfuse should be measured on your own data before deciding.
Why is tracing without evaluation considered expensive logging?
1. Providers charge for spans
2. You see runs but you do not score them, so regressions are invisible and bisection is impossible
3. Disk costs more than CPU
4. Spans cannot be replayed
Show the answer
B. You see runs but you do not score them, so regressions are invisible and bisection is impossible Evals are what turn traces into actionable quality signals.
What is the lesson's warning about self-rolled LLM-judges?
1. CRITIC applies: judges need external grounding for factual verification or they rubber-stamp
2. They must run on GPUs
3. They are forbidden by Apache 2.0
4. They only work in TypeScript
Show the answer
A. CRITIC applies: judges need external grounding for factual verification or they rubber-stamp Without external tool-grounded verification, LLM-judges drift toward rubber-stamping (CRITIC, Lesson 5).

Questions people ask

How do you monitor an AI agent in production?

Record a trace for every run: inputs, each model and tool call, outputs, time and cost. Add automatic quality scores and sample runs for human review. Open-source tools from Langfuse, Arize and Comet provide this out of the box.

What is Langfuse?

Langfuse is an open-source platform for LLM apps that records traces, manages prompt versions, runs evaluations, and tracks cost. It can be self-hosted or used as a cloud service.

Run the code Read it on GitHub

Next lessonMulti-Agent Debate and Collaboration Previous: OpenTelemetry GenAI Semantic Conventions

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License