# BaraDB Architecture

## Overview

BaraDB is a **multimodal database engine** written in Nim that combines document (KV), graph, vector, columnar, and full-text search storage in a single engine with a unified query language called **BaraQL**.

The architecture follows a 5-layer design:

```
┌─────────────────────────────────────────────────────────┐
│ 1. CLIENT LAYER                                          │
│    Binary Protocol │ HTTP/REST │ WebSocket │ Embedded    │
├─────────────────────────────────────────────────────────┤
│ 2. QUERY LAYER (BaraQL)                                  │
│    Lexer → Parser → AST → IR → Optimizer → Codegen      │
├─────────────────────────────────────────────────────────┤
│ 3. EXECUTION ENGINE                                      │
│    Document │ Graph │ Vector │ Columnar │ FTS            │
├─────────────────────────────────────────────────────────┤
│ 4. STORAGE                                               │
│    LSM-Tree │ B-Tree │ WAL │ Bloom │ Compaction │ Cache  │
├─────────────────────────────────────────────────────────┤
│ 5. DISTRIBUTED                                           │
│    Raft Consensus │ Sharding │ Replication │ Gossip      │
└─────────────────────────────────────────────────────────┘
```

## Layer 1: Client Layer

The client layer provides multiple ways to communicate with BaraDB:

- **Binary Protocol** (`protocol/wire.nim`): Efficient big-endian binary protocol with 16 message types for high-performance data transfer. Supports query, batch, transaction, and auth messages.
- **HTTP/REST** (`protocol/http.nim`): JSON-based REST API with routing, CORS, and path parameters. Suitable for web applications.
- **WebSocket** (`protocol/websocket.nim`): Full-duplex streaming for real-time data feeds and push notifications.
- **Embedded** (`storage/lsm.nim`): Direct in-process access using the LSM-Tree API, similar to SQLite's embedded mode.

Supporting infrastructure:
- **Connection Pool** (`protocol/pool.nim`): Load-balanced connection management with min/max limits and eviction.
- **Authentication** (`protocol/auth.nim`): JWT-based authentication with token management.
- **Rate Limiting** (`protocol/ratelimit.nim`): Token bucket and sliding window rate limiting.
- **TLS/SSL** (`protocol/ssl.nim`): Encrypted connections with certificate management.

## Layer 2: Query Layer (BaraQL)

The query layer processes BaraQL — a SQL-compatible query language with extensions for graph, vector, and document operations.

### Pipeline

1. **Lexer** (`query/lexer.nim`): Tokenizes input into 80+ token types including keywords, identifiers, operators, and literals. Supports Unicode input and case-insensitive keywords.

2. **Parser** (`query/parser.nim`): Recursive descent parser that produces an Abstract Syntax Tree (AST). Supports:
   - SELECT with WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, OFFSET
   - INSERT, UPDATE, DELETE, SET
   - CREATE TYPE, DROP TYPE, CREATE INDEX
   - JOIN (INNER, LEFT, RIGHT, FULL, CROSS)
   - CTE (WITH ... AS)
   - Subqueries in FROM and WHERE
   - CASE/WHEN expressions
   - Aggregate functions (COUNT, SUM, AVG, MIN, MAX)
   - Dotted path identifiers (table.column)

3. **AST** (`query/ast.nim`): Nodes for all statement types, expressions, clauses, and schema definitions. 300+ lines covering 25+ node kinds.

4. **IR - Intermediate Representation** (`query/ir.nim`): Self-contained representation that abstracts from the query language syntax. Includes:
   - `IRPlan`: Execution plan nodes (Scan, Filter, Project, Join, GroupBy, Sort, Limit, etc.)
   - `IRExpr`: Expression nodes (Literal, Field, Binary, Aggregate, Function, etc.)
   - `TypeChecker`: Expression type inference with context.

5. **Optimizer / Codegen** (`query/codegen.nim`): Translates IR plans to storage operations. Supports:
   - Predicate pushdown: filter conditions moved to storage level
   - Point read optimization: equality filters converted to direct key lookups
   - Cost estimation for plan comparison
   - EXPLAIN output for debugging

6. **Adaptive Query Execution** (`query/adaptive.nim`): Runtime plan adaptation:
   - Cardinality estimation with exponential moving average
   - Automatic reoptimization when estimates are off by >3x
   - Plan caching via hash-based lookup
   - Parallelism hints for execution contexts

## Layer 3: Execution Engine

### Document/KV Engine
- **LSM-Tree** (`storage/lsm.nim`): Write-optimized log-structured merge tree for key-value storage.
- **B-Tree Index** (`storage/btree.nim`): Ordered index for range scans and point queries.

### Vector Engine (`vector/`)
- **HNSW Index** (`engine.nim`): Hierarchical Navigable Small World graph for approximate nearest neighbor search.
- **IVF-PQ Index** (`engine.nim`): Inverted File Index with Product Quantization.
- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression.
- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan).
- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput.
- **SQL Integration** (`query/executor.nim`):
  - `VECTOR(n)` column type with dimension validation
  - `CREATE INDEX ... USING hnsw` / `USING ivfpq`
  - Distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
  - `<->` nearest-neighbor operator
  - Automatic index maintenance on INSERT/UPDATE

### Graph Engine (`graph/`)
- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency.
- **Algorithms**: BFS, DFS, Dijkstra shortest path, PageRank.
- **Community Detection** (`community.nim`): Louvain algorithm with modularity optimization.
- **Pattern Matching** (`community.nim`): Subgraph isomorphism via backtracking search.
- **Cypher Queries** (`cypher.nim`): MATCH/RETURN/WHERE/LIMIT query parser and executor.

### Full-Text Search (`fts/`)
- **Inverted Index** (`engine.nim`): Term-document index with position tracking.
- **Ranking**: BM25 and TF-IDF scoring.
- **Fuzzy Search**: Levenshtein distance up to configurable threshold.
- **Regex Search**: Wildcard pattern matching (*prefix, suffix*, *both*).
- **Multi-Language** (`multilang.nim`): Tokenizers and stemmers for EN, BG, DE, FR, RU with stop word lists and automatic language detection.

### Columnar Engine (`core/columnar.nim`)
- **Columnar Storage**: Per-column data arrays for analytical queries.
- **Encoding**: Run-length encoding (RLE) and dictionary encoding.
- **Aggregates**: sum, avg, min, max, count over columns.
- **GroupBy**: Multi-column grouping with aggregation.

### Cross-Modal Engine (`core/crossmodal.nim`)
- **Unified Query Interface**: Hybrid search across document, vector, graph, and FTS.
- **Weighted Scoring**: Configurable weights for each modality in hybrid queries.
- **2PC Transactions**: Two-phase commit for atomic cross-modal operations.

## Layer 4: Storage

- **LSM-Tree** (`storage/lsm.nim`): Core storage engine with MemTable, immutable table, and SSTable on disk.
- **WAL** (`storage/wal.nim`): Write-Ahead Log for durability. Fixes crash consistency.
- **Bloom Filter** (`storage/bloom.nim`): Probabilistic data structure for fast negative lookups (1% false positive rate).
- **Compaction** (`storage/compaction.nim`): Size-tiered strategy with level management.
- **Page Cache** (`storage/compaction.nim`): LRU cache with hit rate tracking.
- **Memory-mapped I/O** (`storage/mmap.nim`): mmap-based file access with madvise hints.
- **Crash Recovery** (`storage/recovery.nim`): WAL replay for REDO/UNDO on startup.

## Layer 5: Distributed

- **Raft Consensus** (`core/raft.nim`): Leader election, log replication, RequestVote, AppendEntries.
- **Election Timer** (`core/raft.nim`): Configurable timeout-based leader election loop.
- **Sharding** (`core/sharding.nim`): Hash-based, range-based, and consistent hashing.
- **Cluster Membership** (`core/sharding.nim`): Auto-rebalance on node join/leave/fail.
- **Replication** (`core/replication.nim`): Sync, async, and semi-sync replication modes.
- **Gossip Protocol** (`core/gossip.nim`): Membership management with alive/suspect/dead states.
- **Distributed Transactions** (`core/disttxn.nim`): Two-phase commit across nodes with saga pattern.
- **Transaction Manager** (`core/mvcc.nim`): Multi-Version Concurrency Control with snapshot isolation.
- **Deadlock Detection** (`core/deadlock.nim`): Wait-for graph cycle detection.

## Data Flow

### Write Path
```
Client → Protocol → Auth → Parser → AST → IR → Codegen
  → StorageOp → MVCC Txn → WAL Write → MemTable → Commit
```

### Read Path
```
Client → Protocol → Auth → Parser → AST → IR → Codegen
  → StorageOp → MVCC Snapshot → MemTable → SSTable → Result
```

### Vector Search Path
```
Client → Query "SIMILAR vec TO [...]" → Parser → Codegen
  → HNSW Index → Distance Computation → Top-K → Result
```

### Graph Path
```
Client → Query "MATCH (n)-[r]->(m) RETURN n" → Cypher Parser
  → Graph Engine → BFS/DFS/Dijkstra → Result
```

## Key Design Decisions

1. **Pure Nim**: No Cython, Python, or Rust dependencies. Single compiler (nim) builds everything.
2. **Unified Storage**: One engine handles KV, graph, vector, FTS, and columnar — no separate services.
3. **Embedded Mode**: Can run as a library (like SQLite) or as a server (like PostgreSQL).
4. **Binary Protocol**: Custom efficient protocol instead of text-based SQL at wire level.
5. **Copy-on-Write MVCC**: Multi-version concurrency control for non-blocking reads without explicit locks.
6. **Schema-First Design**: Strongly typed schema system with inheritance, computed properties, and automatic migrations.

## Module Count

| Category | Modules | Lines (est.) |
|----------|---------|--------------|
| Core | 10 | ~2500 |
| Storage | 7 | ~1500 |
| Query | 7 | ~2500 |
| Vector | 3 | ~1000 |
| Graph | 3 | ~1000 |
| FTS | 2 | ~800 |
| Protocol | 7 | ~1500 |
| Schema | 1 | ~400 |
| Client | 2 | ~500 |
| CLI | 1 | ~200 |
| Distributed | 5 | ~1500 |
| **Total** | **48** | **~14,000** |