Retrieval-Focused Data System for AI Applications
Vector Search · Hybrid Retrieval · Ingestion-Aware Architecture
Barq-DB v2 is a retrieval-focused data system built in Rust for modern AI workloads.
It combines:
- Dense vector search
- BM25 text retrieval
- Async ingestion pipelines
- Segment-based storage lifecycle
into a unified architecture designed for:
- RAG systems
- semantic search
- AI-powered recommendations
Barq-DB is designed as a retrieval system rather than a standalone vector store.
Ingestion, indexing, and querying are treated as coordinated stages of a single pipeline, enabling better control over performance, memory usage, and long-running stability.
- Disk-backed vector storage using mmap
- Configurable memory budgeting and eviction
- Reduced RAM pressure for large datasets
- Queue-based ingestion with batching
- Explicit backpressure handling
- Stable under sustained write load
- Explicit lifecycle: Growing → Sealed → Compacted
- Background compaction
- Improved long-running stability
- Combined vector similarity and BM25 keyword search
- Reciprocal Rank Fusion (RRF)
- Deterministic result merging
proto/barq.protois the canonical API contract- SDKs aligned to gRPC
- REST maintained for compatibility
- Hot segments and indexes may reside in memory
- Cold data is accessed through mmap-backed storage
- Memory usage is bounded through configurable limits
- Eviction policies prevent uncontrolled memory growth
- Writes are persisted through WAL before acknowledgment (configurable)
- Recovery replays WAL into segment state
- Snapshots and compaction reduce recovery time
- Single-node deployments acknowledge writes with
NodeLocaldurability - Replicated multi-node deployments now route writes through per-shard Raft quorum commit before acknowledgment
- The runtime consensus path is backed by deterministic Raft leader election, stale-leader rejection, and follower catch-up logic
- Single-replica multi-node deployments remain routed replication without quorum durability
- The current Raft engine is deterministic and in-memory; durable term/log persistence and real inter-node transport are still future work
Barq-DB v2 includes built-in benchmarking tools.
Designed to evaluate:
- Ingestion throughput
- Query latency (p50 / p95 / p99) from live in-process searches
- Memory usage under load
- RSS before and after a benchmark run
Supports dataset simulations at scale (1M, 10M, and higher).
Benchmark smoke coverage is checked in CI through .github/workflows/benchmarks.yml.
Barq-DB v2 introduces a gRPC-first architecture.
- gRPC is the primary API surface
- REST is maintained for compatibility
- SDKs available in:
- Python
- TypeScript
- Go
- Rust
- No breaking changes to existing SDK methods
- New features exposed via optional parameters
- Insert options:
- wait_for_commit
- Search options:
- allow_fallback
- consistency
- Async ingestion support
- Metrics and admin APIs
docker-compose up -dcargo run --bin barq-serverEndpoints:
- HTTP: http://localhost:8080
- gRPC: localhost:50051
from barq import BarqClient
client = BarqClient("http://localhost:8080", api_key="your-key")
client.create_collection(name="products", dimension=384, metric="Cosine")
client.insert_document(
collection="products",
id=1,
vector=[0.1, 0.2, ...],
payload={"name": "Widget"}
)
results = client.search(collection="products", vector=query_vector, top_k=10)| Crate | Description |
|---|---|
| barq-core | Data structures and catalog |
| barq-index | HNSW, IVF, SIMD kernels |
| barq-bm25 | Text search engine |
| barq-storage | WAL, snapshots, persistence |
| barq-cluster | Sharding and routing |
| barq-api | gRPC and REST APIs |
Barq-DB v2 introduces a stronger and more structured architecture.
However, it still requires continued validation under real-world workloads, particularly for large-scale and distributed scenarios.
MIT License