feat: integrate OpenTelemetry tracing for LLM agent observability

[STRRL]

Summary

Add OpenTelemetry (OTel) tracing support to LAPP so that LLM agent calls (via eino ADK) are observable in standard tracing backends (Jaeger, Grafana Tempo, etc.).

Background

eino provides a callbacks.Handler mechanism that fires on every component invocation (ChatModel, Tool, etc.) within the ADK agent execution graph. Global handlers registered via callbacks.AppendGlobalHandlers() work transparently with ADK — no changes to analyzer.go needed.

Current state of eino-ext tracing

• eino-ext/callbacks/apmplus — OTel-based but vendor-locked to Volcengine APMPlus (requires AppKey)
• eino-ext/callbacks/langfuse — works well but uses Langfuse's own API, not OTLP
• No vendor-neutral OTel callback handler exists in eino-ext

Proposal

Write a custom callbacks.Handler that uses standard go.opentelemetry.io/otel/trace to:

1. OnStart — create a span with GenAI semantic attributes (gen_ai.request.model, prompt messages, etc.)
2. OnEnd — end the span, record token usage (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), response content
3. OnError — set span error status
4. Streaming variants — OnStartWithStreamInput / OnEndWithStreamOutput

Register it globally at process startup. The handler should accept a standard trace.TracerProvider so users can plug in any OTel-compatible backend.

Scope

• ~100-200 lines for the callback handler
• OTel SDK initialization (TracerProvider + OTLP exporter) configurable via env vars (OTEL_EXPORTER_OTLP_ENDPOINT, etc.)
• Opt-in: tracing is enabled only when an OTLP endpoint is configured

References

• eino callbacks interface: github.com/cloudwego/eino/callbacks
• apmplus implementation (reference): https://github.com/cloudwego/eino-ext/tree/main/callbacks/apmplus
• OpenTelemetry GenAI Semantic Conventions

[m13v]

OTel is the right choice for agent tracing since it gives you standard tooling for visualization. the callback handler approach is clean, we do something similar where each tool call creates a child span and the LLM inference is the parent span. one thing to instrument that people often miss: the time between the LLM returning a tool call and the tool actually starting execution. that gap reveals queuing delays and serialization overhead that are invisible in request-level metrics.

[m13v]

our tool execution layer with tracing-friendly structure (parent LLM span -> child tool spans): https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift - each tool call is isolated with timing and result tracking