Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
183 lines (135 loc) · 4.69 KB

ARCHITECTURE.md

File metadata and controls

183 lines (135 loc) · 4.69 KB

Agent Observability Platform - Architecture

System Overview

A production-ready observability platform for monitoring, tracking, and optimizing AI agent performance. The system provides comprehensive metrics collection, cost tracking, alerting, and real-time dashboards.

Architecture Components

1. Metrics Collection System

The core of the platform collects and stores metrics for every agent request:

  • Request Tracking: Unique request IDs, timestamps, agent names
  • Performance Metrics: Latency (p50, p95, p99), success/failure rates
  • Cost Metrics: Token usage, cost per request, daily totals
  • Error Tracking: Error messages, failure reasons, stack traces
  • LangSmith Integration: Trace IDs for correlation with LangSmith

2. Database Layer

SQLite database (upgradeable to PostgreSQL) stores:

  • Request Metrics: All request data with timestamps
  • Alerts: Alert history and resolution tracking
  • Agent Configurations: Agent settings and metadata

3. Monitoring Components

Metrics Collector

  • Records all request metrics
  • Aggregates statistics
  • Provides summary views

Cost Tracker

  • Calculates costs based on model pricing
  • Tracks daily/monthly costs
  • Provides optimization suggestions

Alert Manager

  • Monitors thresholds
  • Creates alerts automatically
  • Manages alert lifecycle

4. LangSmith Integration

  • Automatic tracing for all agent requests
  • Trace ID correlation with metrics
  • Direct links to LangSmith dashboard
  • Optional integration (works without LangSmith)

5. Rate Limiting

  • Per-agent rate limiting
  • Configurable limits
  • In-memory implementation (Redis-ready)
  • Request queuing support

6. API Layer

FastAPI-based REST API with:

  • Agent execution endpoints
  • Metrics retrieval endpoints
  • Alert management endpoints
  • Health check endpoints
  • Real-time dashboard data

7. Frontend Dashboard

Modern web interface showing:

  • Real-time statistics
  • Active alerts
  • Recent metrics table
  • Cost breakdowns
  • Performance charts

Data Flow

Agent Request
    ↓
[Rate Limiter] → Check rate limits
    ↓
[Agent Execution] → Execute with LangSmith tracing
    ↓
[Metrics Collector] → Record metrics
    ↓
[Cost Tracker] → Calculate costs
    ↓
[Database] → Store metrics
    ↓
[Alert Manager] → Check thresholds, create alerts
    ↓
[Dashboard] → Display real-time data

Technology Stack

Backend

  • FastAPI: Modern async web framework
  • SQLAlchemy: Database ORM with async support
  • LangSmith: Observability and tracing
  • LangChain: Agent framework integration

Frontend

  • HTML/CSS/JavaScript: Modern dashboard
  • Real-time Updates: Auto-refresh every 30 seconds

Database

  • SQLite: Development database
  • PostgreSQL: Production-ready option

Monitoring

  • LangSmith: External observability platform
  • Custom Metrics: Internal tracking system

Key Design Decisions

1. Async Architecture

All database operations and API endpoints are async for better performance.

2. Modular Design

Separate components for metrics, alerts, and cost tracking allow independent scaling.

3. Optional LangSmith

Platform works with or without LangSmith, making it flexible for different setups.

4. In-Memory Rate Limiting

Simple implementation that can be upgraded to Redis for distributed systems.

5. SQLite First

Easy setup with SQLite, but designed to work with PostgreSQL in production.

Scalability Considerations

Current Limitations

  • In-memory rate limiting (single server)
  • SQLite database (single file)
  • No distributed tracing aggregation

Production Upgrades

  • Redis for distributed rate limiting
  • PostgreSQL for scalable database
  • Message queue for async processing
  • CDN for static assets
  • Load balancer for multiple instances

Security Considerations

  • API key management via environment variables
  • Rate limiting to prevent abuse
  • Input validation on all endpoints
  • SQL injection protection via ORM
  • CORS configuration for frontend

Performance Optimizations

  • Async database operations
  • Indexed database queries
  • Efficient metric aggregation
  • Cached summary calculations
  • Minimal frontend dependencies

Monitoring Strategy

  1. Application Metrics: Tracked internally
  2. External Observability: LangSmith integration
  3. Health Checks: Built-in health endpoints
  4. Alerting: Automatic threshold monitoring
  5. Cost Tracking: Real-time cost analysis

Future Architecture Enhancements

  • Microservices architecture
  • Event-driven alerting
  • Real-time WebSocket updates
  • Advanced analytics engine
  • Machine learning anomaly detection
  • Multi-region deployment
  • GraphQL API option
  • Plugin system for custom metrics