Phase 14 · Lesson 23 of 54

OpenTelemetry GenAI Semantic Conventions

OpenTelemetry's GenAI SIG (launched April 2024) defines the standard schema for agent telemetry. Span names, attributes, and content-capture rules converge across vendors so agent traces mean the same thing in Datadog, Grafana, Jaeger, and Honeycomb.

Lesson 23 of 54Learn + Build~60 minutesPython (stdlib)

Before this: Lesson 13 of Phase 14 (LangGraph), lesson 24 of Phase 14 (Observability Platforms)

On this page

Learning Objectives

  • Name the GenAI span categories: model/client, agent, tool.
  • Distinguish invoke_agent CLIENT vs INTERNAL spans and when each applies.
  • List the top-level GenAI attributes: provider name, request model, data-source ID.
  • Explain the content-capture contract: opt-in, OTEL_SEMCONV_STABILITY_OPT_IN, external-reference recommendation.

The Problem

Every vendor invents their own span names. Ops teams end up building per-framework dashboards. OpenTelemetry's GenAI SIG fixes this by defining one standard the whole ecosystem targets.

The Concept

Span categories

  1. Model / client spans. Cover raw LLM calls. Emitted by provider SDKs (Anthropic, OpenAI, Bedrock) and framework model adapters.
  2. Agent spans. create_agent (when the agent is constructed) and invoke_agent (when it runs).
  3. Tool spans. One per tool invocation; connected to the agent span by parent-child relation.

Agent span naming

  • Span name: invoke_agent {gen_ai.agent.name} if named; fallback to invoke_agent.
  • Span kind:
    • CLIENT — for remote agent services (OpenAI Assistants API, Bedrock Agents).
    • INTERNAL — for in-process agent frameworks (LangChain, CrewAI, local ReAct).

Key attributes

  • gen_ai.provider.name — anthropic, openai, aws.bedrock, google.vertex.
  • gen_ai.request.model — the model ID.
  • gen_ai.response.model — the resolved model (may differ from request due to routing).
  • gen_ai.agent.name — agent identifier.
  • gen_ai.operation.name — chat, completion, invoke_agent, tool_call.
  • gen_ai.data_source.id — for RAG: which corpus or store was consulted.

Technology-specific conventions exist for Anthropic, Azure AI Inference, AWS Bedrock, OpenAI.

Content capture

The default rule: instrumentations SHOULD NOT capture inputs/outputs by default. Capture is opt-in via:

  • gen_ai.system_instructions
  • gen_ai.input.messages
  • gen_ai.output.messages

Recommended production pattern: store content externally (S3, your log store), record references on spans (pointer IDs, not prose). This is the Lesson 27 content-poisoning defense wired into observability.

Stability

Most conventions are experimental as of March 2026. Opt in to the stable preview with:

OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental

Datadog v1.37+ maps GenAI attributes natively into its LLM Observability schema. Other backends (Grafana, Honeycomb, Jaeger) support the raw attributes.

Where this pattern goes wrong

  • Capturing full prompts in spans. PII, secrets, customer data in traces that ops can read. Store externally.
  • No gen_ai.provider.name. Multi-provider dashboards break when attribution is missing.
  • Spans without parent links. Orphaned tool spans. Always propagate context.
  • Not setting stability opt-in. Your attributes may get renamed on backend upgrade.
ae-genai-span-tree

Build It

code/main.py implements a stdlib span emitter matching GenAI conventions:

  • Span with GenAI attribute schema.
  • Tracer with start_span, nested contexts.
  • A scripted agent run that emits: create_agent, invoke_agent (INTERNAL), per-tool spans, chat spans for LLM calls.
  • A content-capture mode that stores prompts externally and records IDs on spans.

Run it:

python3 code/main.py

Output: a span tree with all required GenAI attributes, and an "external store" showing the opt-in content references.

Use It

  • Datadog LLM Observability (v1.37+) maps attributes natively.
  • Langfuse / Phoenix / Opik (Lesson 24) — auto-instrument the ecosystem.
  • Jaeger / Honeycomb / Grafana Tempo — raw OTel traces; build dashboards from GenAI attributes.
  • Self-hosted — run the OTel Collector with a GenAI processor.

Ship It

outputs/skill-otel-genai.md wires OTel GenAI spans into an existing agent with content-capture defaults and external-reference storage.

Exercises

  1. Instrument your Lesson 01 ReAct loop with invoke_agent (INTERNAL) + per-tool spans. Send to a Jaeger instance.
  2. Add content capture in "references only" mode: prompts to SQLite, span attributes carry only row IDs.
  3. Read the spec for gen_ai.data_source.id. Wire it into your Lesson 09 Mem0 search.
  4. Set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental and verify your attributes don't get renamed by the collector.
  5. Build a dashboard: "which tool errors correlate with which models" from GenAI attributes alone.

Key Terms

Term What people say What it actually means
GenAI SIG "OpenTelemetry GenAI group" OTel working group defining the schema
invoke_agent "Agent span" Name of the span representing an agent run
CLIENT span "Remote call" Span for a call to a remote agent service
INTERNAL span "In-process" Span for an in-process agent run
gen_ai.provider.name "Provider" anthropic / openai / aws.bedrock / google.vertex
gen_ai.data_source.id "RAG source" Which corpus/store a retrieval hit
Content capture "Prompt logging" Opt-in capture of messages; store externally in prod
Stability opt-in "Preview mode" Env var to pin experimental conventions

Further Reading

Check yourself

Quick quiz

  1. Which span name represents an agent run in the GenAI conventions?

    1. invoke_agent (optionally suffixed by gen_ai.agent.name)
    2. agent.start
    3. agent_loop
    4. run_chain
    Show the answer

    A. invoke_agent (optionally suffixed by gen_ai.agent.name) invoke_agent is the canonical name; the agent's name attribute appears in the span name when set.

  2. Which three span categories does the GenAI SIG define?

    1. Frontend, backend, db
    2. Read, write, exec
    3. Latency, error, throughput
    4. Model/client, agent, tool
    Show the answer

    D. Model/client, agent, tool Model/client (LLM calls), agent (create/invoke), tool (per invocation) are the categories.

  3. When does an invoke_agent span use kind CLIENT vs INTERNAL?

    1. Always INTERNAL
    2. CLIENT for HTTPS only
    3. Always CLIENT
    4. CLIENT for remote agent services (OpenAI Assistants, Bedrock Agents); INTERNAL for in-process frameworks (LangChain, CrewAI, local ReAct)
    Show the answer

    D. CLIENT for remote agent services (OpenAI Assistants, Bedrock Agents); INTERNAL for in-process frameworks (LangChain, CrewAI, local ReAct) Remote agent calls are CLIENT; in-process agent runs are INTERNAL.

  4. What is the default rule for content capture (inputs and outputs)?

    1. Capture everything by default
    2. Instrumentations SHOULD NOT capture by default; capture is opt-in via gen_ai.* attributes
    3. Capture only on errors
    4. Capture only output, never input
    Show the answer

    B. Instrumentations SHOULD NOT capture by default; capture is opt-in via gen_ai.* attributes Default-off content capture protects PII and secrets; opt-in is explicit.

  5. Which attribute identifies the corpus or store consulted for a retrieval?

    1. gen_ai.data_source.id
    2. gen_ai.provider.name
    3. gen_ai.agent.name
    4. gen_ai.request.model
    Show the answer

    A. gen_ai.data_source.id gen_ai.data_source.id labels the RAG corpus or store hit by retrieval.

  6. What does the lesson recommend for storing prompt content in production?

    1. Store content externally (S3, log store) and record reference IDs on the span instead
    2. Capture full prompts on every span as plain text
    3. Encrypt prompts in span attributes
    4. Never store any content
    Show the answer

    A. Store content externally (S3, log store) and record reference IDs on the span instead External storage with pointer IDs avoids PII leaking into traces that ops can read.

  7. How do you opt into the experimental stable preview of GenAI conventions?

    1. Pay for a license
    2. Edit the OTel collector config
    3. Set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
    4. Rebuild the SDK from source
    Show the answer

    C. Set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental The env var pins the experimental conventions so attributes are not silently renamed.

Questions people ask

What is OpenTelemetry for AI?

It is a set of naming rules within OpenTelemetry for recording AI activity, such as model name, token counts, tool calls and agent steps. Following them lets traces from different tools show up the same way in any compatible monitoring product.

Should AI traces include the full prompts and responses?

Be careful. Full text helps debugging but can store customer data and private details. The conventions treat content capture as opt-in, so teams can log metadata by default and turn on full text only where privacy rules allow.

Run the code Read it on GitHub

Next lessonAgent Observability: Langfuse, Phoenix, OpikPrevious: Voice Agents: Pipecat and LiveKit

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes