Baradb/README.md

BaraDB

A multimodal database engine written in Nim — 100% native, zero dependencies.

BaraDB combines document, graph, vector, columnar, and full-text search storage in a single engine with a unified query language (BaraQL). It compiles to a single 286KB binary with no runtime dependencies.

Current Status: BaraDB is an active development project and educational proof-of-concept. Many core algorithms are implemented and tested, but several critical production features are still placeholders or incomplete. See Limitations below for details.

Why BaraDB?

Feature	GEL/EdgeDB	BaraDB
Core language	Python + Cython + Rust	100% Nim
Storage backend	PostgreSQL only	Native multi-engine
Vector search	pgvector extension	Built-in HNSW/IVF-PQ
Graph algorithms	None	BFS, DFS, Dijkstra, PageRank, Louvain
Full-text search	PG FTS extension	Built-in BM25 + TF-IDF
Embedded mode	No	Yes (SQLite-like)
Binary size	~50MB+	286KB
Dependencies	PostgreSQL, Python, many libs	Zero

Architecture

┌─────────────────────────────────────────────────────────┐
│                    CLIENT LAYER                          │
│  Binary Protocol │ HTTP/REST │ WebSocket │ Embedded      │
├─────────────────────────────────────────────────────────┤
│                 QUERY LAYER (BaraQL)                     │
│  Lexer → Parser → AST → IR → Optimizer → Codegen        │
├─────────────────────────────────────────────────────────┤
│                EXECUTION ENGINE                          │
│  Document │ Graph │ Vector │ Columnar │ FTS              │
├─────────────────────────────────────────────────────────┤
│                    STORAGE                               │
│  LSM-Tree │ B-Tree │ WAL │ Bloom Filter │ mmap           │
├─────────────────────────────────────────────────────────┤
│                DISTRIBUTED                               │
│  Raft Consensus │ Sharding │ Replication                 │
└─────────────────────────────────────────────────────────┘

Quick Start

# Build
nim c -d:release -o:build/baradadb src/baradadb.nim

# Run tests
nim c --path:src -r tests/test_all.nim

# Run benchmarks
nim c -d:release -r benchmarks/bench_all.nim

# Start server
./build/baradadb

BaraQL — Query Language

BaraQL is SQL-compatible with extensions for graph, vector, and document queries.

Basic Queries

-- SELECT with WHERE, ORDER BY, LIMIT
SELECT name, age FROM users WHERE age > 18 ORDER BY name LIMIT 10;

-- INSERT
INSERT users { name := 'Alice', age := 30 };

-- UPDATE
UPDATE users SET age = 31 WHERE name = 'Alice';

-- DELETE
DELETE FROM users WHERE name = 'Alice';

Aggregates and Grouping

-- GROUP BY with HAVING
SELECT department, count(*), avg(salary)
FROM employees
GROUP BY department
HAVING count(*) > 5;

-- Aggregates: count, sum, avg, min, max
SELECT count(*), sum(amount), avg(price) FROM orders;

JOINs

-- INNER JOIN
SELECT u.name, o.total
FROM users u
INNER JOIN orders o ON u.id = o.user_id;

-- LEFT JOIN
SELECT u.name, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

-- Multiple JOINs
SELECT *
FROM orders o
JOIN users u ON o.user_id = u.id
JOIN products p ON o.product_id = p.id;

CTEs (Common Table Expressions)

-- Single CTE
WITH active_users AS (
  SELECT * FROM users WHERE active = true
)
SELECT * FROM active_users;

-- Multiple CTEs
WITH
  recent AS (SELECT * FROM orders WHERE date > '2025-01-01'),
  totals AS (SELECT user_id, sum(amount) as total FROM recent GROUP BY user_id)
SELECT u.name, t.total FROM users u JOIN totals t ON u.id = t.user_id;

Subqueries

-- Subquery in FROM
SELECT * FROM (SELECT id, name FROM users WHERE active = true) AS active;

-- EXISTS subquery
SELECT name FROM users WHERE EXISTS (SELECT 1 FROM orders WHERE orders.user_id = users.id);

CASE Expressions

SELECT name,
  CASE
    WHEN age < 18 THEN 'minor'
    WHEN age < 65 THEN 'adult'
    ELSE 'senior'
  END AS category
FROM users;

Schema Definition

-- Create type with properties and links
CREATE TYPE Person {
  name: str,
  age: int32
};

CREATE TYPE Movie {
  title: str,
  director: Person
};

Storage Engines

LSM-Tree (Key-Value)

The primary storage engine with write-optimized append-only log structure.

import barabadb/storage/lsm

var db = newLSMTree("./data")
db.put("key1", cast[seq[byte]]("value1"))
let (found, value) = db.get("key1")
db.close()

Components:

• MemTable — in-memory sorted buffer
• WAL — write-ahead log for durability
• SSTable — sorted string tables on disk
• Bloom Filter — probabilistic set membership
• Compaction — size-tiered strategy with level management
• Page Cache — LRU cache with hit rate tracking

B-Tree Index

Ordered index for range scans and point lookups.

import barabadb/storage/btree

var btree = newBTreeIndex[string, string]()
btree.insert("key1", "value1")
let values = btree.get("key1")
let range = btree.scan("key_a", "key_z")

Vector Engine

Native HNSW and IVF-PQ indexes for similarity search.

import barabadb/vector/engine

var idx = newHNSWIndex(dimensions = 128)
idx.insert(1, @[1.0'f32, 0.0'f32, ...], {"category": "A"}.toTable)
let results = idx.search(queryVector, k = 10)

# With metadata filtering
let filtered = idx.searchWithFilter(queryVector, k = 10,
  filter = proc(meta: Table[string, string]): bool =
    return meta.getOrDefault("category") == "A")

Features:

• HNSW — hierarchical navigable small world graph
• IVF-PQ — inverted file index with product quantization
• Distance metrics — cosine, euclidean, dot product, Manhattan
• Quantization — scalar 8-bit/4-bit, product, binary
• Metadata filtering — filter results by key-value pairs

Graph Engine

Adjacency list storage with built-in algorithms.

import barabadb/graph/engine

var g = newGraph()
let alice = g.addNode("Person", {"name": "Alice"}.toTable)
let bob = g.addNode("Person", {"name": "Bob"}.toTable)
discard g.addEdge(alice, bob, "knows")

# Traversal
let bfs = g.bfs(alice)
let dfs = g.dfs(alice)
let path = g.shortestPath(alice, bob)
let ranks = g.pageRank()

Algorithms:

• BFS/DFS — breadth-first and depth-first traversal
• Dijkstra — shortest weighted path
• PageRank — node importance ranking
• Louvain — community detection
• Pattern matching — subgraph isomorphism search

Full-Text Search

Inverted index with BM25 and TF-IDF ranking.

import barabadb/fts/engine

var idx = newInvertedIndex()
idx.addDocument(1, "Nim is a fast programming language")
idx.addDocument(2, "Python is popular for data science")

# BM25 search
let results = idx.search("programming language")

# TF-IDF search
let tfidf = idx.searchTfidf("programming language")

# Fuzzy search (typo tolerance)
let fuzzy = idx.fuzzySearch("programing", maxDistance = 2)

# Wildcard search
let wild = idx.regexSearch("prog*")

Columnar Engine

Column-oriented storage for analytical queries.

import barabadb/core/columnar

var batch = newColumnBatch()
var ageCol = batch.addInt64Col("age")
var nameCol = batch.addStringCol("name")
ageCol.appendInt64(25)
nameCol.appendString("Alice")

# Aggregates
echo ageCol.sumInt64()
echo ageCol.avgInt64()

# Encoding
let rle = rleEncode(@[1'i64, 1, 1, 2, 2, 3])
let dict = dictEncode(@["apple", "banana", "apple"])

Transactions

MVCC with snapshot isolation and deadlock detection.

import barabadb/core/mvcc

var tm = newTxnManager()
let txn = tm.beginTxn()
discard tm.write(txn, "key1", cast[seq[byte]]("value1"))
discard tm.write(txn, "key2", cast[seq[byte]]("value2"))

# Savepoint
tm.savepoint(txn)
discard tm.write(txn, "key3", cast[seq[byte]]("value3"))
discard tm.rollbackToSavepoint(txn)  # undo key3

discard tm.commit(txn)

Protocol

Binary Wire Protocol

16 message types with big-endian serialization.

import barabadb/protocol/wire

let msg = makeQueryMessage(1, "SELECT * FROM users")
let ready = makeReadyMessage(1)
let error = makeErrorMessage(1, 42, "Syntax error")

HTTP/REST API

import barabadb/protocol/http

var router = newHttpRouter(port = 8080)
router.get("/api/users", proc(req: Request): Future[JsonNode] {.async.} =
  return %*[{"id": 1, "name": "Alice"}])

WebSocket Streaming

import barabadb/protocol/websocket

var server = newWsServer(port = 8081)
server.onMessage = proc(ws: WebSocket, data: seq[byte]) {.gcsafe.} =
  echo "Received: ", cast[string](data)
asyncCheck server.run()

Authentication

import barabadb/protocol/auth

var am = newAuthManager("secret-key")
let token = am.createToken(JWTClaims(sub: "user1", role: "admin"))
let result = am.validateCredentials(AuthCredentials(authMethod: amToken, payload: token))

Rate Limiting

import barabadb/protocol/ratelimit

var rl = newRateLimiter(rlaTokenBucket, globalRate = 1000, perClientRate = 100)
if rl.allowRequest("client-123"):
  echo "Request allowed"

Schema System

import barabadb/schema/schema

var s = newSchema()

let person = newType("Person")
person.addProperty("name", "str", required = true)
person.addProperty("age", "int32")
s.addType("default", person)

# Inheritance
let employee = newType("Employee")
employee.setBases(@["Person"])
employee.addProperty("department", "str")
s.addType("default", employee)

# Resolve inheritance — Employee gets name, age, department
let resolved = s.resolveInheritance(employee)

# Diff schemas
let diff = s.diff(oldSchema, newSchema)

Distributed

Raft Consensus

import barabadb/core/raft

var cluster = newRaftCluster()
cluster.addNode("node1")
cluster.addNode("node2")
cluster.addNode("node3")

let n1 = cluster.nodes["n1"]
n1.becomeCandidate()
n1.becomeLeader()
let entry = n1.appendLog("SET key1 value1")

Sharding

import barabadb/core/sharding

var router = newShardRouter(ShardConfig(numShards: 4, replicas: 2, strategy: ssHash))
router.rebalance(@["node1", "node2", "node3"])
let shard = router.getShard("user_123")

Replication

import barabadb/core/replication

var rm = newReplicationManager(rmSync)
rm.addReplica(newReplica("r1", "10.0.0.1", 5432))
rm.connectReplica("r1")
let lsn = rm.writeLsn(@[1'u8, 2, 3])
rm.ackLsn("r1", lsn)  # blocks until acked

User Defined Functions

import barabadb/query/udf

var reg = newUDFRegistry()
reg.registerStdlib()  # abs, sqrt, pow, lower, upper, len, trim, substr, toString, toInt

# Custom function
reg.register("greet", @[UDFParam(name: "name", typeName: "str")],
  "str", proc(args: seq[Value]): Value =
    return Value(kind: vkString, strVal: "Hello, " & args[0].strVal & "!"))

Project Structure

src/barabadb/
├── core/
│   ├── types.nim         # Type system (17 types)
│   ├── config.nim        # Configuration
│   ├── server.nim        # Async TCP server
│   ├── mvcc.nim          # Multi-version concurrency control
│   ├── deadlock.nim      # Deadlock detection
│   ├── raft.nim          # Raft consensus
│   ├── sharding.nim      # Hash/range/consistent sharding
│   ├── replication.nim   # Sync/async/semi-sync replication
│   └── columnar.nim      # Columnar storage + encoding
├── storage/
│   ├── lsm.nim           # LSM-Tree storage engine
│   ├── btree.nim         # B-Tree index
│   ├── wal.nim           # Write-ahead log
│   ├── bloom.nim         # Bloom filter
│   ├── compaction.nim    # SSTable compaction + page cache
│   └── mmap.nim          # Memory-mapped I/O
├── query/
│   ├── lexer.nim         # Tokenizer (80+ tokens)
│   ├── parser.nim        # Recursive descent parser
│   ├── ast.nim           # Abstract syntax tree
│   ├── ir.nim            # Intermediate representation
│   ├── codegen.nim       # IR → storage operations
│   └── udf.nim           # User defined functions
├── vector/
│   ├── engine.nim        # HNSW + IVF-PQ indexes
│   ├── quant.nim         # Scalar/product/binary quantization
│   └── simd.nim          # SIMD-optimized distance ops
├── graph/
│   ├── engine.nim        # Adjacency list + algorithms
│   └── community.nim     # Louvain + pattern matching
├── fts/
│   └── engine.nim        # Inverted index + BM25 + fuzzy
├── protocol/
│   ├── wire.nim          # Binary wire protocol
│   ├── http.nim          # HTTP/REST router
│   ├── websocket.nim     # WebSocket streaming
│   ├── pool.nim          # Connection pool
│   ├── auth.nim          # JWT authentication
│   └── ratelimit.nim     # Rate limiting
├── schema/
│   └── schema.nim        # Types, links, inheritance, migrations
└── cli/
    └── shell.nim         # Interactive query shell

Tests

# Run all tests (162 tests, 35 suites)
nim c --path:src -r tests/test_all.nim

# Run benchmarks
nim c -d:release -r benchmarks/bench_all.nim

Roadmap Progress

Phase	Status	Progress
Core (LSM + B-Tree + compaction + cache + mmap)	✅	95%
BaraQL (GROUP BY + JOIN + CTE + aggregates + codegen + UDF)	✅	100%
Multimodal storage (KV + graph + vector + columnar)	🟡	75%
Transactions (MVCC + deadlock + WAL + savepoints)	✅	85%
Protocol (binary + HTTP + WS + pool + auth + ratelimit)	✅	85%
Schema (inheritance + computed + migrations)	✅	95%
Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata)	✅	95%
Graph engine (all algorithms + pattern matching)	✅	90%
FTS (BM25 + TF-IDF + fuzzy + regex)	✅	85%
CLI shell	🟡	50%
Cluster (Raft + sharding + replication)	✅	60%
Optimizations (SIMD + mmap done)	🟡	40%

Current Limitations

While BaraDB demonstrates a wide range of database concepts with passing tests, several components are simplified or incomplete for production use:

Component	Status	Note
LSM-Tree SSTable reads	🟡 Placeholder	get() finds the key in the SSTable index but returns an empty value. Real disk I/O is pending.
HNSW vector search	🟡 Linear scan	search() scans all vectors (O(N)). True hierarchical graph navigation is not yet implemented.
TCP server execution	🟡 Stub	The async server accepts connections and echoes "OK\n". It does not parse the wire protocol or execute queries.
Raft consensus	🟡 In-memory only	Raft algorithm logic is implemented and tested, but there is no network transport between nodes.
Graph / FTS / Columnar	🟡 In-memory only	These engines store data in RAM. Persistence to disk is not yet implemented.
Query codegen	🟡 Partial	IR plans are generated, but execution against storage engines is limited.

We are actively working to close these gaps. See the Roadmap above for per-phase progress.

License

BSD 3-Clause License

Copyright (c) 2024, BaraDB Authors All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.