agent-otel-trajectory

agent-otel-trajectory

Agent runs are a black box unless they emit telemetry. To debug, optimize cost, and grade the journey (not just the answer), an agent should emit OpenTelemetry trajectory spans as it works: one span per tool-call / iteration, with latency, token usage, and outcome. Backend-only or LLM-API-only metrics miss the agent's actual decision path.

The landscape (survey)

• equinix-labs/otel-cli — the mature, vendor-neutral CLI for emitting OTLP spans from shell: otel-cli exec --name "<step>" -- <command> wraps any command in a span (attributes, events, span kinds, context propagation, background spans). Best fit for EMITTING trajectory telemetry from shell-driven agents/harnesses.
• traceloop/opentelemetry-mcp-server — an MCP server for querying/analyzing existing traces (Jaeger/Tempo/Traceloop). Use it for an agent that debugs via traces, NOT for emitting its own — a different job.
• OTel GenAI/MCP semantic conventions (opentelemetry.io/docs/specs/semconv/gen-ai/) — use the standard attribute names so spans are portable.

Recommendation

Don't make the agent call a tool to emit its own spans (circular, unreliable). Instrument the harness/loop (the thing that runs tool-calls) to wrap each tool-call in an otel-cli span. No MCP-server dependency; works anywhere a shell does.

The pattern

1. One root span per agent run, child spans per iteration/tool-call:

   otel-cli exec --service "<agent>" --name "tool:<name>" \
     --attrs "gen_ai.operation.name=<tool>,gen_ai.request.model=<model>,gen_ai.usage.input_tokens=<n>,gen_ai.usage.output_tokens=<n>" \
     -- <the tool command>
   

2. Attributes that matter (use semconv): gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens, gen_ai.operation.name, the tool name, the task/run id, success/error. Latency is the span duration (automatic).
3. Context propagation: propagate the traceparent down so child tool-calls nest under the run (otel-cli --tp-print/env propagation).
4. Export vendor-neutral OTLP → the environment's backend (e.g. Cloud Trace on GCP) via the OTLP endpoint env (OTEL_EXPORTER_OTLP_ENDPOINT). Never hardcode a vendor.
5. Emit on failure too — error spans with the failure attribute are the most valuable (where the trajectory broke).

DONE means

An agent run produces a trace in the backend with one span per tool-call showing latency + token usage + outcome, nested under a run span, using gen_ai semconv attributes — so a failure or a cost spike is visible in the trace, not inferred. Verified by opening the trace, not by "instrumented".