Refactor Statistics Cache System

[aptend]

Refactor Statistics Cache System

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #22336

What this PR does / why we need it:

This PR refactors the statistics cache system to improve query optimization performance and accuracy. The changes introduce a two-tier caching architecture:

1. Session Stats Cache

• Fast path: Returns cached stats within 3 seconds if valid (AccurateObjectNumber > 0)
• Aggressive retry: Immediately recalculates invalid stats to ensure BVT tests get fresh data after table creation/insertion
• Empty table handling: Returns nil for empty tables, allowing callers to use DefaultStats (Outcnt=1000)
• Value-type cache: Uses map[uint64]StatsInfoWrapper to reduce small object allocations
• Optimize BackExec: Reuse BackExec in one session to eliminate the overhead of repeated init.

2. Global Stats Cache

• Event-driven updates: Asynchronously updated by Logtail events
• Two-level filtering:
  • Queue condition: keyExists + event type + large table throttling
  • Execution condition: inProgress + MinUpdateInterval (15s)
• Large table throttling:
  • Small tables (< 500 objects): Update on any change
  • Large tables (≥ 500 objects): Update when change rate ≥ 5% or timeout 30min
• Disk-only stats: Only counts persisted objects, ignoring in-memory dirty blocks
• Concurrent traversal: Uses concurrentExecutor for parallel object metadata loading

3. Sampling Statistics for Large Tables

• Two-phase approach:
  • Phase 1: Exact table-level stats (row count, block count) from ObjectStats (no IO)
  • Phase 2: Sampled column-level stats (ZoneMap, NDV, NullCnt) from ObjectMeta (with IO)
• Sampling strategy:
  • ≤100 objects: Full scan
  • >100 objects: targetCount = clamp(max(sqrt(N), 0.02·N), 100, 2000)
• UUID-based sampling: Leverages UUIDv7's random bytes for zero-overhead deterministic sampling
• Row-based scaling: Scales column stats by row count ratio to reduce object size variance impact

4. New Table Function: table_stats()

• Provides SQL interface to query table statistics
• Returns: table_name, row_count, block_count, object_count, stats_json
• Supports optional refresh mode: 'auto' (default) or 'full' (force refresh)

Key Benefits:

1. Reduced S3 IO: Sampling reduces IO for large tables while maintaining accuracy
2. Better cache hit rate: Session cache provides fast path for repeated queries
3. Scalable updates: Large table throttling prevents excessive update frequency
4. Observability: New table_stats() function for debugging and monitoring

[XuPeng-SH]

Please check which Markdown documents were accidentally added