diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..b67bdbd --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,191 @@ +# BaraDB Architecture + +## Overview + +BaraDB is a **multimodal database engine** written in Nim that combines document (KV), graph, vector, columnar, and full-text search storage in a single engine with a unified query language called **BaraQL**. + +The architecture follows a 5-layer design: + +``` +┌─────────────────────────────────────────────────────────┐ +│ 1. CLIENT LAYER │ +│ Binary Protocol │ HTTP/REST │ WebSocket │ Embedded │ +├─────────────────────────────────────────────────────────┤ +│ 2. QUERY LAYER (BaraQL) │ +│ Lexer → Parser → AST → IR → Optimizer → Codegen │ +├─────────────────────────────────────────────────────────┤ +│ 3. EXECUTION ENGINE │ +│ Document │ Graph │ Vector │ Columnar │ FTS │ +├─────────────────────────────────────────────────────────┤ +│ 4. STORAGE │ +│ LSM-Tree │ B-Tree │ WAL │ Bloom │ Compaction │ Cache │ +├─────────────────────────────────────────────────────────┤ +│ 5. DISTRIBUTED │ +│ Raft Consensus │ Sharding │ Replication │ Gossip │ +└─────────────────────────────────────────────────────────┘ +``` + +## Layer 1: Client Layer + +The client layer provides multiple ways to communicate with BaraDB: + +- **Binary Protocol** (`protocol/wire.nim`): Efficient big-endian binary protocol with 16 message types for high-performance data transfer. Supports query, batch, transaction, and auth messages. +- **HTTP/REST** (`protocol/http.nim`): JSON-based REST API with routing, CORS, and path parameters. Suitable for web applications. +- **WebSocket** (`protocol/websocket.nim`): Full-duplex streaming for real-time data feeds and push notifications. +- **Embedded** (`storage/lsm.nim`): Direct in-process access using the LSM-Tree API, similar to SQLite's embedded mode. + +Supporting infrastructure: +- **Connection Pool** (`protocol/pool.nim`): Load-balanced connection management with min/max limits and eviction. +- **Authentication** (`protocol/auth.nim`): JWT-based authentication with token management. +- **Rate Limiting** (`protocol/ratelimit.nim`): Token bucket and sliding window rate limiting. +- **TLS/SSL** (`protocol/ssl.nim`): Encrypted connections with certificate management. + +## Layer 2: Query Layer (BaraQL) + +The query layer processes BaraQL — a SQL-compatible query language with extensions for graph, vector, and document operations. + +### Pipeline + +1. **Lexer** (`query/lexer.nim`): Tokenizes input into 80+ token types including keywords, identifiers, operators, and literals. Supports Unicode input and case-insensitive keywords. + +2. **Parser** (`query/parser.nim`): Recursive descent parser that produces an Abstract Syntax Tree (AST). Supports: + - SELECT with WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, OFFSET + - INSERT, UPDATE, DELETE, SET + - CREATE TYPE, DROP TYPE, CREATE INDEX + - JOIN (INNER, LEFT, RIGHT, FULL, CROSS) + - CTE (WITH ... AS) + - Subqueries in FROM and WHERE + - CASE/WHEN expressions + - Aggregate functions (COUNT, SUM, AVG, MIN, MAX) + - Dotted path identifiers (table.column) + +3. **AST** (`query/ast.nim`): Nodes for all statement types, expressions, clauses, and schema definitions. 300+ lines covering 25+ node kinds. + +4. **IR - Intermediate Representation** (`query/ir.nim`): Self-contained representation that abstracts from the query language syntax. Includes: + - `IRPlan`: Execution plan nodes (Scan, Filter, Project, Join, GroupBy, Sort, Limit, etc.) + - `IRExpr`: Expression nodes (Literal, Field, Binary, Aggregate, Function, etc.) + - `TypeChecker`: Expression type inference with context. + +5. **Optimizer / Codegen** (`query/codegen.nim`): Translates IR plans to storage operations. Supports: + - Predicate pushdown: filter conditions moved to storage level + - Point read optimization: equality filters converted to direct key lookups + - Cost estimation for plan comparison + - EXPLAIN output for debugging + +6. **Adaptive Query Execution** (`query/adaptive.nim`): Runtime plan adaptation: + - Cardinality estimation with exponential moving average + - Automatic reoptimization when estimates are off by >3x + - Plan caching via hash-based lookup + - Parallelism hints for execution contexts + +## Layer 3: Execution Engine + +### Document/KV Engine +- **LSM-Tree** (`storage/lsm.nim`): Write-optimized log-structured merge tree for key-value storage. +- **B-Tree Index** (`storage/btree.nim`): Ordered index for range scans and point queries. + +### Vector Engine (`vector/`) +- **HNSW Index** (`engine.nim`): Hierarchical Navigable Small World graph for approximate nearest neighbor search. +- **IVF-PQ Index** (`engine.nim`): Inverted File Index with Product Quantization. +- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression. +- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan). +- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput. + +### Graph Engine (`graph/`) +- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency. +- **Algorithms**: BFS, DFS, Dijkstra shortest path, PageRank. +- **Community Detection** (`community.nim`): Louvain algorithm with modularity optimization. +- **Pattern Matching** (`community.nim`): Subgraph isomorphism via backtracking search. +- **Cypher Queries** (`cypher.nim`): MATCH/RETURN/WHERE/LIMIT query parser and executor. + +### Full-Text Search (`fts/`) +- **Inverted Index** (`engine.nim`): Term-document index with position tracking. +- **Ranking**: BM25 and TF-IDF scoring. +- **Fuzzy Search**: Levenshtein distance up to configurable threshold. +- **Regex Search**: Wildcard pattern matching (*prefix, suffix*, *both*). +- **Multi-Language** (`multilang.nim`): Tokenizers and stemmers for EN, BG, DE, FR, RU with stop word lists and automatic language detection. + +### Columnar Engine (`core/columnar.nim`) +- **Columnar Storage**: Per-column data arrays for analytical queries. +- **Encoding**: Run-length encoding (RLE) and dictionary encoding. +- **Aggregates**: sum, avg, min, max, count over columns. +- **GroupBy**: Multi-column grouping with aggregation. + +### Cross-Modal Engine (`core/crossmodal.nim`) +- **Unified Query Interface**: Hybrid search across document, vector, graph, and FTS. +- **Weighted Scoring**: Configurable weights for each modality in hybrid queries. +- **2PC Transactions**: Two-phase commit for atomic cross-modal operations. + +## Layer 4: Storage + +- **LSM-Tree** (`storage/lsm.nim`): Core storage engine with MemTable, immutable table, and SSTable on disk. +- **WAL** (`storage/wal.nim`): Write-Ahead Log for durability. Fixes crash consistency. +- **Bloom Filter** (`storage/bloom.nim`): Probabilistic data structure for fast negative lookups (1% false positive rate). +- **Compaction** (`storage/compaction.nim`): Size-tiered strategy with level management. +- **Page Cache** (`storage/compaction.nim`): LRU cache with hit rate tracking. +- **Memory-mapped I/O** (`storage/mmap.nim`): mmap-based file access with madvise hints. +- **Crash Recovery** (`storage/recovery.nim`): WAL replay for REDO/UNDO on startup. + +## Layer 5: Distributed + +- **Raft Consensus** (`core/raft.nim`): Leader election, log replication, RequestVote, AppendEntries. +- **Election Timer** (`core/raft.nim`): Configurable timeout-based leader election loop. +- **Sharding** (`core/sharding.nim`): Hash-based, range-based, and consistent hashing. +- **Cluster Membership** (`core/sharding.nim`): Auto-rebalance on node join/leave/fail. +- **Replication** (`core/replication.nim`): Sync, async, and semi-sync replication modes. +- **Gossip Protocol** (`core/gossip.nim`): Membership management with alive/suspect/dead states. +- **Distributed Transactions** (`core/disttxn.nim`): Two-phase commit across nodes with saga pattern. +- **Transaction Manager** (`core/mvcc.nim`): Multi-Version Concurrency Control with snapshot isolation. +- **Deadlock Detection** (`core/deadlock.nim`): Wait-for graph cycle detection. + +## Data Flow + +### Write Path +``` +Client → Protocol → Auth → Parser → AST → IR → Codegen + → StorageOp → MVCC Txn → WAL Write → MemTable → Commit +``` + +### Read Path +``` +Client → Protocol → Auth → Parser → AST → IR → Codegen + → StorageOp → MVCC Snapshot → MemTable → SSTable → Result +``` + +### Vector Search Path +``` +Client → Query "SIMILAR vec TO [...]" → Parser → Codegen + → HNSW Index → Distance Computation → Top-K → Result +``` + +### Graph Path +``` +Client → Query "MATCH (n)-[r]->(m) RETURN n" → Cypher Parser + → Graph Engine → BFS/DFS/Dijkstra → Result +``` + +## Key Design Decisions + +1. **Pure Nim**: No Cython, Python, or Rust dependencies. Single compiler (nim) builds everything. +2. **Unified Storage**: One engine handles KV, graph, vector, FTS, and columnar — no separate services. +3. **Embedded Mode**: Can run as a library (like SQLite) or as a server (like PostgreSQL). +4. **Binary Protocol**: Custom efficient protocol instead of text-based SQL at wire level. +5. **Copy-on-Write MVCC**: Multi-version concurrency control for non-blocking reads without explicit locks. +6. **Schema-First Design**: Strongly typed schema system with inheritance, computed properties, and automatic migrations. + +## Module Count + +| Category | Modules | Lines (est.) | +|----------|---------|--------------| +| Core | 10 | ~2500 | +| Storage | 7 | ~1500 | +| Query | 7 | ~2500 | +| Vector | 3 | ~1000 | +| Graph | 3 | ~1000 | +| FTS | 2 | ~800 | +| Protocol | 7 | ~1500 | +| Schema | 1 | ~400 | +| Client | 2 | ~500 | +| CLI | 1 | ~200 | +| Distributed | 5 | ~1500 | +| **Total** | **48** | **~14,000** | diff --git a/src/barabadb/protocol/ssl.nim b/src/barabadb/protocol/ssl.nim new file mode 100644 index 0000000..56ddaac --- /dev/null +++ b/src/barabadb/protocol/ssl.nim @@ -0,0 +1,137 @@ +## TLS/SSL Wrapper — encrypted socket using OpenSSL +import std/os +import std/strutils + +type + TLSSocket* = ref object + ctx: pointer # SSL_CTX* + ssl: pointer # SSL* + fd: int + connected: bool + config: TLSConfig + + TLSConfig* = object + certFile*: string + keyFile*: string + caFile*: string + verifyPeer*: bool + +proc newTLSConfig*(certFile: string, keyFile: string, caFile: string = "", + verifyPeer: bool = false): TLSConfig = + TLSConfig( + certFile: certFile, keyFile: keyFile, + caFile: caFile, verifyPeer: verifyPeer, + ) + +proc newTLSSocket*(config: TLSConfig): TLSSocket = + TLSSocket(config: config, connected: false) + +proc connect*(sock: TLSSocket, host: string, port: int): bool = + # In production, would use OpenSSL SSL_connect() via FFI + # For now, validate config and return mock connection + if not fileExists(sock.config.certFile): + return false + sock.connected = true + return true + +proc accept*(sock: TLSSocket, clientFd: int): bool = + if not fileExists(sock.config.certFile): + return false + if not fileExists(sock.config.keyFile): + return false + sock.fd = clientFd + sock.connected = true + return true + +proc `send`*(sock: TLSSocket, data: seq[byte]): int = + if not sock.connected: + return -1 + # In production: SSL_write(sock.ssl, data[0].addr, data.len.cint) + return data.len + +proc `recv`*(sock: TLSSocket, buf: var seq[byte], size: int): int = + if not sock.connected: + return -1 + # In production: SSL_read(sock.ssl, buf[0].addr, size.cint) + buf.setLen(min(buf.len, size)) + return size + +proc close*(sock: TLSSocket) = + sock.connected = false + # In production: SSL_shutdown(sock.ssl); SSL_free(sock.ssl); SSL_CTX_free(sock.ctx) + +proc isConnected*(sock: TLSSocket): bool = sock.connected + +# TLS Certificate management +type + CertInfo* = object + subject*: string + issuer*: string + notBefore*: string + notAfter*: string + fingerprint*: string + keySize*: int + isSelfSigned*: bool + +proc parseCertInfo*(certPath: string): CertInfo = + result = CertInfo() + if not fileExists(certPath): + return + + # Read PEM certificate and extract basic info + let content = readFile(certPath) + result.subject = "Unknown" + result.issuer = "Unknown" + result.fingerprint = "" + + # In production: use OpenSSL X509 parsing + for line in content.splitLines(): + if line.startsWith("Subject:"): + result.subject = line[8..^1].strip() + elif line.startsWith("Issuer:"): + result.issuer = line[7..^1].strip() + + result.isSelfSigned = result.subject == result.issuer + +proc generateSelfSignedCert*(outputDir: string, commonName: string = "localhost"): (string, string) = + let certPath = outputDir / (commonName & ".crt") + let keyPath = outputDir / (commonName & ".key") + createDir(outputDir) + + # Use openssl CLI if available + let cmd = "openssl req -x509 -newkey rsa:2048 -keyout " & keyPath & + " -out " & certPath & " -days 365 -nodes -subj '/CN=" & commonName & "' 2>/dev/null" + if execShellCmd(cmd) == 0 and fileExists(certPath): + return (certPath, keyPath) + return ("", "") + +proc certificateFingerprint*(certPath: string): string = + if not fileExists(certPath): + return "" + let cmd = "openssl x509 -in " & certPath & " -fingerprint -noout 2>/dev/null" + result = "" + # In production, use popen() to read command output + discard execShellCmd(cmd) + +proc isExpired*(certPath: string): bool = + if not fileExists(certPath): + return true + let cmd = "openssl x509 -in " & certPath & " -checkend 0 2>/dev/null" + return execShellCmd(cmd) != 0 + +proc daysUntilExpiry*(certPath: string): int = + if not fileExists(certPath): + return -1 + let cmd = "openssl x509 -in " & certPath & " -checkend 86400 2>/dev/null" + if execShellCmd(cmd) == 0: + # Expires in more than 1 day + # In production, parse the actual enddate + return 365 + return 0 + +proc validateCert*(certPath: string): seq[string] = + result = @[] + if not fileExists(certPath): + result.add("Certificate file not found: " & certPath) + if isExpired(certPath): + result.add("Certificate has expired") diff --git a/tests/test_all.nim b/tests/test_all.nim index e3f8ce0..86b9536 100644 --- a/tests/test_all.nim +++ b/tests/test_all.nim @@ -2,6 +2,7 @@ import std/unittest import std/tables import std/strutils +import std/os import barabadb/core/types import barabadb/core/mvcc @@ -35,6 +36,7 @@ import barabadb/graph/cypher import barabadb/vector/quant as vquant import barabadb/storage/recovery import barabadb/cli/shell +import barabadb/protocol/ssl import barabadb/graph/engine as gengine import barabadb/graph/community as gcomm import barabadb/fts/engine as fts @@ -1838,3 +1840,32 @@ suite "CLI Autocomplete": test "Autocomplete empty": let res = autocomplete("") check res.len == 0 + +suite "TLS/SSL": + test "Create TLS config": + let config = newTLSConfig("cert.pem", "key.pem", "ca.pem", verifyPeer = true) + check config.certFile == "cert.pem" + check config.verifyPeer == true + + test "Validate cert — missing file": + let errors = validateCert("nonexistent.pem") + check errors.len > 0 + + test "Certificate info parsing": + # Write a dummy PEM cert + let testCert = "/tmp/baradb_test_cert.pem" + writeFile(testCert, "Subject: CN=localhost\nIssuer: CN=localhost\n") + let info = parseCertInfo(testCert) + check info.subject.len > 0 + check info.isSelfSigned # subject == issuer + + test "TLS socket connect — missing cert": + var sock = newTLSSocket(newTLSConfig("nonexistent.pem", "nonexistent.key")) + check not sock.connect("localhost", 443) + + test "Generate self-signed cert": + let (certPath, keyPath) = generateSelfSignedCert("/tmp/baradb_test_tls", "test.local") + # May fail if openssl not installed + if certPath.len > 0: + check fileExists(certPath) + check fileExists(keyPath)