Baradb/docs/en/architecture.md

BaraDB Architecture

Overview

BaraDB is a multimodal database engine written in Nim that combines document (KV), graph, vector, columnar, and full-text search storage in a single engine with a unified query language called BaraQL.

Layer Architecture

┌─────────────────────────────────────────────────────────┐
│ 1. CLIENT LAYER                                          │
│    Binary Protocol │ HTTP/REST │ WebSocket │ Embedded    │
├─────────────────────────────────────────────────────────┤
│ 2. QUERY LAYER (BaraQL)                                  │
│    Lexer → Parser → AST → IR → Optimizer → Codegen      │
├─────────────────────────────────────────────────────────┤
│ 3. EXECUTION ENGINE                                      │
│    Document │ Graph │ Vector │ Columnar │ FTS            │
├─────────────────────────────────────────────────────────┤
│ 4. STORAGE                                               │
│    LSM-Tree │ B-Tree │ WAL │ Bloom │ Compaction │ Cache  │
├─────────────────────────────────────────────────────────┤
│ 5. DISTRIBUTED                                           │
│    Raft Consensus │ Sharding │ Replication │ Gossip      │
└─────────────────────────────────────────────────────────┘

Layer 1: Client Layer

Multiple communication protocols:

• Binary Protocol (protocol/wire.nim): Efficient big-endian binary protocol with 16 message types
• HTTP/REST (core/httpserver.nim): JSON-based REST API with multi-threading
• WebSocket (core/websocket.nim): Full-duplex streaming
• Embedded (storage/lsm.nim): Direct in-process access

Server Architecture

The TCP and HTTP servers share a single LSMTree instance to ensure data consistency:

┌─────────────────────────────────────────────────────────┐
│                    SHARED STORAGE                        │
│                    LSMTree Instance                      │
├────────────────────────┬────────────────────────────────┤
│   TCP Server           │   HTTP Server                  │
│   (Binary Protocol)    │   (REST API)                   │
│   Port: 9472           │   Port: 9912 (TCP + 440)       │
│   TCP_NODELAY: ON      │   Multi-threaded               │
├────────────────────────┴────────────────────────────────┤
│   WebSocket Server                                      │
│   (Streaming/Pub-Sub)                                   │
│   Port: 9913 (TCP + 441)                                │
│   TCP_NODELAY: ON                                       │
└─────────────────────────────────────────────────────────┘

Key optimizations:

• Shared LSMTree — All servers operate on the same database instance, eliminating data inconsistency
• TCP_NODELAY — Enabled on listening and client sockets in TCP/WebSocket servers for lower latency on small messages. The HTTP server (via hunos) also handles TCP_NODELAY internally.
• Safe byte conversion — Proper bytesToString/stringToBytes functions instead of unsafe cast operations in the wire protocol

Connection Management

• Connection Pool (protocol/pool.nim): Min/max connection limits with idle timeout
• Rate Limiting (protocol/ratelimit.nim): Token-bucket global and per-client limits, integrated in both TCP and HTTP handlers
• Authentication (protocol/auth.nim): JWT with HMAC-SHA256 and role-based access
• TLS/SSL (protocol/ssl.nim): TLS 1.3 with auto-generated certificates

Layer 2: Query Layer (BaraQL)

The BaraQL pipeline:

1. Lexer (query/lexer.nim): Tokenizes input into 80+ token types
2. Parser (query/parser.nim): Recursive descent parser producing AST
3. AST (query/ast.nim): 300+ lines covering 25+ node kinds
4. IR (query/ir.nim): Intermediate representation for execution plans
5. Optimizer (query/adaptive.nim): Adaptive cross-modal query optimization
6. Codegen (query/codegen.nim): Translates IR to storage operations
7. Executor (query/executor.nim): Executes plans with parallelization

Cross-Modal Planning

The optimizer (query/adaptive.nim) determines execution order across engines:

1. Estimate selectivity for each predicate
2. Push most selective predicate to its engine first
3. Use bloom filters for KV lookups
4. Parallelize independent branches
5. Stream results to avoid materialization

Layer 3: Execution Engine

Document/KV Engine

• LSM-Tree (storage/lsm.nim): Write-optimized storage with MemTable, WAL, SSTables
• B-Tree Index (storage/btree.nim): Ordered index for range scans with COW

Vector Engine (vector/)

• HNSW Index (vector/engine.nim): Hierarchical Navigable Small World graph
• IVF-PQ Index (vector/engine.nim): Inverted File Index with Product Quantization
• SIMD Operations (vector/simd.nim): AVX2-optimized distance computations
• Quantization (vector/quant.nim): Scalar, product, and binary quantization

Graph Engine (graph/)

• Adjacency List (graph/engine.nim): Edge-weighted directed graph
• Algorithms (graph/engine.nim): BFS, DFS, Dijkstra, PageRank
• Community Detection (graph/community.nim): Louvain algorithm
• Pattern Matching (graph/community.nim): Subgraph isomorphism
• Cypher Parser (graph/cypher.nim): Cypher-like graph queries

Full-Text Search (fts/)

• Inverted Index (fts/engine.nim): Term-document index
• Ranking (fts/engine.nim): BM25 and TF-IDF scoring
• Fuzzy Search (fts/engine.nim): Levenshtein distance matching
• Multi-Language (fts/multilang.nim): Tokenizers for EN, BG, DE, FR, RU

Columnar Engine (core/columnar.nim)

• Per-column storage for analytical queries
• RLE and dictionary encoding
• SIMD-accelerated aggregates

Layer 4: Storage

• LSM-Tree (storage/lsm.nim): MemTable, WAL, SSTable, Bloom Filter, Compaction
• Page Cache (storage/compaction.nim): LRU cache with hit rate tracking
• Memory-mapped I/O (storage/mmap.nim): mmap-based file access
• Recovery (storage/recovery.nim): WAL replay and crash recovery

Write Path

Client → Protocol → Auth → Parser → AST → IR → Codegen
  → StorageOp → MVCC Txn → WAL Write → MemTable → Commit

Read Path

Client → Protocol → Auth → Parser → AST → IR → Codegen
  → StorageOp → MVCC Snapshot → MemTable → SSTable → Result

Layer 5: Distributed

• Raft Consensus (core/raft.nim): Leader election, log replication
• Sharding (core/sharding.nim): Hash, range, and consistent hashing
• Replication (core/replication.nim): Sync, async, semi-sync modes
• Gossip Protocol (core/gossip.nim): SWIM-like membership management with exponential backoff error recovery
• Distributed Transactions (core/disttxn.nim): Two-phase commit

Key Design Decisions

1. Pure Nim: No Cython, Python, or Rust dependencies
2. Unified Storage: One engine handles KV, graph, vector, FTS, and columnar
3. Embedded Mode: Can run as library or server
4. Binary Protocol: Custom efficient wire protocol
5. MVCC: Multi-version concurrency control
6. Schema-First: Strongly typed schema system with inheritance
7. Cross-Modal: Single query language across all data models
8. Formally Verified: Core distributed algorithms specified in TLA+ and model-checked with TLC

Module Statistics

Category	Modules	Lines of Code	Purpose
Core	16	~4,200	Server, protocols, transactions, distributed
Storage	7	~3,100	LSM, B-Tree, WAL, bloom, compaction, mmap
Query	7	~2,800	Lexer, parser, AST, IR, optimizer, codegen, executor
Vector	3	~1,200	HNSW, IVF-PQ, quantization, SIMD
Graph	3	~1,000	Adjacency list, algorithms, community detection
FTS	2	~900	Inverted index, BM25, fuzzy, multi-language
Protocol	7	~2,400	Wire, HTTP, WebSocket, pool, auth, rate limit, SSL
Schema	1	~600	Types, links, inheritance, migrations
Client	2	~800	Nim binary client, file helpers
CLI	1	~400	Interactive BaraQL shell
Total	49	~14,100

Data Flow Diagrams

Simple Query

┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐
│ Client │───→│ Lexer  │───→│ Parser │───→│  IR    │───→│ Codegen│
└────────┘    └────────┘    └────────┘    └────────┘    └───┬────┘
                                                            │
┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐       │
│ Result │←───│ Format │←───│ Execute│←───│ Storage│←──────┘
└────────┘    └────────┘    └────────┘    └────────┘

Cross-Modal Query

                    ┌─────────────┐
                    │   Parser    │
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │   Adaptive  │
                    │   Optimizer │
                    └──────┬──────┘
                           │
           ┌───────────────┼───────────────┐
           │               │               │
    ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
    │    Vector   │ │    Graph    │ │     FTS     │
    │    Engine   │ │    Engine   │ │   Engine    │
    └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
           │               │               │
           └───────────────┼───────────────┘
                           │
                    ┌──────▼──────┐
                    │    Join     │
                    │   & Sort    │
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │    Result   │
                    └─────────────┘