Phase 19 · Lesson 11 of 85

Capstone 11 — LLM Observability & Eval Dashboard

Langfuse went open-core. Arize Phoenix published the 2026 GenAI semconv mappings. Helicone and Braintrust both doubled down on per-user cost attribution. Traceloop's OpenLLMetry became the de-facto SDK instrumentation. The production shape is ClickHouse for traces, Postgres for metadata, Next.js for UI, and a small army of eval jobs (DeepEval, RAGAS, LLM-judge) running over sampled traces. Build one self-hosted, ingest from at least four SDK families, and demonstrate catching an injected regression in under five minutes.

Lesson 11 of 85Capstone25 hoursTypeScript (UI), Python / TypeScript (ingest + evals), SQL (ClickHouse)

Before this: Phase 11 (LLM engineering), Phase 13 (tools), Phase 17 (infrastructure), Phase 18 (safety)

On this page

Problem

Every AI team running production traffic in 2026 keeps an observability plane alongside the model. Cost attribution. Hallucination detection. Drift monitoring. Jailbreak signal. SLO dashboards. PII leak alerts. The open-source references — Langfuse, Phoenix, OpenLLMetry — converged on OpenTelemetry GenAI semantic conventions as the ingest schema. You can now instrument OpenAI, Anthropic, Google, LangChain, LlamaIndex, and vLLM with one SDK and ship compatible spans.

You will build a self-hosted dashboard that ingests from at least four SDK families, runs a small set of eval jobs over sampled traces, detects drift, and alerts. The measurement bar: given a deliberately injected regression (a prompt that starts producing PII), the dashboard catches it and fires an alert in under five minutes.

Concept

Ingest is OTLP HTTP. The SDK produces GenAI-semconv spans: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.response.id, llm.prompts, llm.completions. Spans land in ClickHouse for columnar analytics; metadata (users, sessions, apps) lands in Postgres.

Evals run as batch jobs over sampled traces. DeepEval scores faithfulness, toxicity, and answer relevance. RAGAS scores retrieval metrics when the trace carries retrieval context. Custom LLM-judges run domain-specific checks (PII leak, off-policy response). Eval runs write back to the same ClickHouse as eval spans linked to the parent trace.

Drift detection watches embedding-space distributions over time (PSI or KL divergence on prompt embeddings) plus eval-score trends. Alerts feed Prometheus Alertmanager and then Slack / PagerDuty. The UI is Next.js 15 with Recharts.

Architecture

production apps:
  OpenAI SDK  +  Anthropic SDK  +  Google GenAI SDK
  LangChain + LlamaIndex + vLLM
       |
       v
  OpenTelemetry SDK with GenAI semconv
       |
       v  OTLP HTTP
  collector (ingest, sample, fan-out)
       |
       +-------------+-----------+
       v             v           v
   ClickHouse    Postgres    S3 archive
   (spans)       (metadata)  (raw events)
       |
       +---> eval jobs (DeepEval, RAGAS, LLM-judge)
       |     sampled or all-trace
       |     write eval spans back
       |
       +---> drift detector (PSI / KL on prompt embeddings)
       |
       +---> Prometheus metrics -> Alertmanager -> Slack / PagerDuty
       |
       v
   Next.js 15 dashboard (Recharts)

Stack

  • Ingest: OpenTelemetry SDKs + GenAI semantic conventions; OTLP HTTP transport
  • Collector: OpenTelemetry Collector with tail-sampling processor (for cost control)
  • Storage: ClickHouse for spans, Postgres for metadata, S3 for raw event archive
  • Evals: DeepEval, RAGAS 0.2, Arize Phoenix evaluator pack, custom LLM-judge
  • Drift: PSI / KL on pooled prompt embeddings (sentence-transformers) weekly
  • Alerting: Prometheus Alertmanager -> Slack / PagerDuty
  • UI: Next.js 15 App Router + Recharts + server actions
  • SDKs supported out of the box: OpenAI, Anthropic, Google GenAI, LangChain, LlamaIndex, vLLM
ce-otel-drift

Build It

  1. Collector config. OpenTelemetry Collector with the OTLP HTTP receiver, a tail-sampler keeping 100% of errored traces and 10% of successes, and exporters to ClickHouse and S3.

  2. ClickHouse schema. Table spans with columns mirroring GenAI semconv: gen_ai_system, gen_ai_request_model, input_tokens, output_tokens, latency_ms, prompt_hash, trace_id, parent_span_id, plus JSON bag for long payloads. Add secondary indexes by user_id and app_id.

  3. SDK coverage test. Write a small client app using each SDK (OpenAI, Anthropic, Google, LangChain, LlamaIndex, vLLM) with OpenLLMetry auto-instrument. Verify each produces canonical GenAI spans that land in ClickHouse.

  4. Eval jobs. A scheduled job reads last-15-min sampled traces and runs DeepEval faithfulness, toxicity, and answer relevance. Outputs are eval spans linked to the parent trace.

  5. Custom LLM-judge. A PII-leak judge: given a response, call a guard LLM to score likelihood of PII leak. High-score responses land in a triage queue.

  6. Drift detection. Weekly job computes PSI between this week's pooled prompt embeddings and the trailing 4-week baseline. If PSI above threshold, alert.

  7. Dashboard. Next.js 15 with pages: overview (spans/sec, cost/user, p95 latency), traces (search + waterfall), evals (faithfulness trend, toxicity), drift (PSI over time), alerts.

  8. Alerting chain. Prometheus exporter reads eval score aggregates and latency percentiles; Alertmanager routes to Slack for warnings and PagerDuty for critical breaches.

  9. Regression probe. Inject a bug: the evaluated chatbot starts leaking fake SSNs 1% of the time. Measure time to detect (MTTD): from bug deployed to Slack alert. Detection is the part of MTTR the dashboard owns; the fix and redeploy come after.

Use It

$ curl -X POST https://my-otel-collector/v1/traces -d @trace.json
[collector]  accepted 1 trace, 3 spans
[clickhouse] inserted 3 spans (app=chat, user=u_42)
[eval]       DeepEval faithfulness 0.82, toxicity 0.03
[drift]      weekly PSI 0.08 (below 0.2 threshold)
[ui]         live at https://obs.example.com

Ship It

outputs/skill-llm-observability.md is the deliverable. Given an LLM application, the dashboard ingests its traces, runs evals, alerts on drift, and surfaces cost/user breakdown in Next.js.

Weight Criterion How it is measured
25 Trace-schema coverage Number of SDK families producing canonical GenAI spans (target: 6+)
20 Eval correctness DeepEval / RAGAS scores vs hand-labeled set
20 Dashboard UX MTTD on injected regression (under 5 minutes target)
20 Cost / scale Sustained ingest at 1k spans/sec without backlog
15 Alerting + drift detection Prometheus/Alertmanager chain exercised end to end
100

Exercises

  1. Add custom instrumentation for the Haystack framework. Verify canonical spans land in ClickHouse with faithful gen_ai.* attributes.

  2. Swap DeepEval for Phoenix evaluators on the same traces. Measure score drift between the two eval engines.

  3. Sharpen the drift detector: compute PSI per app-id rather than globally. Show per-app drift trails.

  4. Add a "user impact" page: cost-per-user and failure-rate-per-user with sparklines.

  5. Build a tail-sampling policy that keeps 100% of traces with toxicity > 0.5 plus a 10% stratified sample of the rest. Measure sampling bias introduced.

Key Terms

Term What people say What it actually means
GenAI semconv "OTel LLM attributes" 2025 OpenTelemetry spec for LLM span attributes (system, model, tokens)
Tail sampling "Post-trace sample" Collector decides to keep or drop a trace after it completes (can peek errors)
PSI "Population stability index" Drift metric comparing two distributions; > 0.2 typically signals meaningful drift
LLM-judge "Eval as model" An LLM scoring another LLM's output on a rubric (faithfulness, toxicity, PII)
Tail-sampling policy "Keep-rule" Rule that decides which traces to persist vs drop; errored + sample-rate
Eval span "Linked eval trace" Child span carrying an eval score linked to the original LLM call span
Cost per user "Unit economics" Dollar cost attributed to a user_id over a window; key product metric

Further Reading

Check yourself

Quick quiz

  1. Which ingest schema do Langfuse, Phoenix, and OpenLLMetry converge on?

    1. Prometheus exposition format
    2. OpenTelemetry GenAI semantic conventions over OTLP HTTP
    3. Proprietary JSON per vendor
    4. Plain CSV log files
    Show the answer

    B. OpenTelemetry GenAI semantic conventions over OTLP HTTP

  2. Why separate ClickHouse and Postgres in the storage tier?

    1. They are interchangeable and one is chosen at random
    2. Postgres is faster for span ingest
    3. ClickHouse handles columnar analytics over spans while Postgres holds users, sessions, and app metadata
    4. ClickHouse cannot store strings
    Show the answer

    C. ClickHouse handles columnar analytics over spans while Postgres holds users, sessions, and app metadata

  3. What does the tail-sampling processor in the OpenTelemetry Collector do?

    1. Decides whether to keep a trace after it completes, using rules like keep errors plus sample successes
    2. Replays old traces into Postgres
    3. Streams every byte unconditionally
    4. Truncates spans below 100ms
    Show the answer

    A. Decides whether to keep a trace after it completes, using rules like keep errors plus sample successes

  4. How does the dashboard detect drift across weeks?

    1. Counts unique trace IDs
    2. Computes PSI or KL divergence on pooled prompt embeddings and watches eval-score trends
    3. Manual eyeballing of the dashboard
    4. Reads the latest deploy timestamp
    Show the answer

    B. Computes PSI or KL divergence on pooled prompt embeddings and watches eval-score trends

  5. What is the deliverable's time-to-detect (MTTD) target on an injected PII-leak regression?

    1. Under 1 hour
    2. Within 24 hours
    3. Under 5 minutes from bug deployed to Slack alert
    4. Within the next on-call shift
    Show the answer

    C. Under 5 minutes from bug deployed to Slack alert

  6. Which SDK families must produce canonical GenAI spans to meet the trace-coverage rubric?

    1. OpenAI and Anthropic only
    2. Any one SDK is enough
    3. Only vLLM
    4. At least six: OpenAI, Anthropic, Google GenAI, LangChain, LlamaIndex, and vLLM
    Show the answer

    D. At least six: OpenAI, Anthropic, Google GenAI, LangChain, LlamaIndex, and vLLM

  7. How are evaluation results linked back to the original LLM call?

    1. As a separate Postgres table with no trace ID
    2. As a CSV emailed nightly
    3. As Slack messages only
    4. As eval spans written as children of the parent trace in ClickHouse
    Show the answer

    D. As eval spans written as children of the parent trace in ClickHouse

Questions people ask

How do you track the cost of AI API calls per user or feature?

Log every call with its model, input and output tokens, latency and tags such as user, feature and team. Multiply tokens by the model price to get cost, then group by tag. Observability tools do this automatically and show spend by user, feature or day.

What is LLM observability?

It is the practice of recording what an AI system does on each request: the prompt, the response, tool calls, timings, cost and errors. Teams use these traces to debug bad answers, find slow steps, control spending and run quality checks on real traffic.

Run the code Read it on GitHub

Next lessonCapstone 12 — Video Understanding Pipeline (Scene, QA, Search)Previous: Capstone 10 — Multi-Agent Software Engineering Team

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes