Phase 3 — Agent Observability (Langfuse Tracing)

[haJ1t]

Phase 3 — Agent Observability (Langfuse Tracing)

End-to-end tracing for every agent execution. Track which agent handled what, how many tokens were used, what it cost, how long it took, and whether it succeeded. Essential for debugging multi-agent interactions and cost optimization.

Dependencies

• Phase 3 — Coordinator Agent and Mastra Agent Network #3728 (Coordinator Agent) — traces start from the Coordinator
• Phase 3 — Specialized Domain Agents (Risk, Compliance, Vendor, Policy, Incident, Model) #3729 (Specialized Agents) — each agent generates spans

Tasks

Langfuse Setup

• Add Langfuse to Docker Compose (self-hosted, runs alongside PostgreSQL and Redis)
• Create Servers/advisor/observability/langfuseConfig.ts — Langfuse client initialization with:
  • API keys from environment variables
  • Flush interval configuration
  • Sampling rate (100% in dev, configurable in prod)
• Add environment variables: LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY

Trace Integration

• Create Servers/advisor/observability/traceManager.ts:
  • startTrace(userId, sessionId, metadata) — begin a new trace when user sends message
  • startSpan(traceName, agentName, operation) — create span for each agent/tool execution
  • endSpan(span, result, tokensUsed, cost) — close span with results
  • endTrace(trace, finalResult) — close trace with overall result
  • logError(span, error) — log errors within a span
• Wire into Coordinator Agent — trace starts when message arrives, spans for routing decision
• Wire into each Specialized Agent — span per agent invocation
• Wire into tool execution — span per tool call with input/output
• Wire into Approval Gateway — span for approval flow (evaluate, wait, approve/reject)
• Wire into Memory operations — span for memory read/write

Metrics to Track

• Per trace: total duration, total tokens (input + output), total cost, agent count, tool count, final status
• Per agent span: agent name, duration, tokens, cost, tools called, result status
• Per tool span: tool name, input params, output, duration, success/failure
• Per LLM call: model used, provider, tokens (input/output/total), cost, latency

Observability Dashboard

• Create Servers/routes/observability.route.ts — proxy endpoints to Langfuse data:
  • GET /api/observability/traces — recent traces with filters
  • GET /api/observability/traces/:id — trace detail with full span tree
  • GET /api/observability/costs — cost breakdown by agent, model, time period
  • GET /api/observability/performance — latency percentiles, error rates
• Register routes in Servers/index.ts
• Frontend page at /ai-observability — embedded Langfuse dashboard or custom UI showing:
  • Trace list with search/filter
  • Trace detail view (waterfall diagram of spans)
  • Cost tracking (daily/weekly/monthly by agent and model)
  • Error rate and latency metrics

File Structure

Servers/advisor/observability/
├── langfuseConfig.ts
├── traceManager.ts
└── costTracker.ts

Docker Addition

langfuse:
  image: langfuse/langfuse:latest
  ports:
    - "3001:3000"
  environment:
    - DATABASE_URL=postgresql://...
    - NEXTAUTH_SECRET=...
  depends_on:
    - postgres

Branch

feat/ai-foundation

---

Part of Phase 3 — Multi-Agent Orchestration