Commit Graph

13 Commits

Author SHA1 Message Date
dimgigov 08f73598a5 Fix WAL crash recovery — data survives power loss
- wal.nim: open existing WAL with fmAppend instead of fmWrite (was truncating!)
- lsm.nim: replay WAL entries into memTable on startup before accepting writes
- Fixes real data loss on crash — previously WAL was wiped on every restart
2026-05-07 00:29:43 +03:00
dimgigov f9f77b3a18 feat: production blockers — JOINs, deadlock detection, TLS, parameterized queries
- JOIN execution: INNER/LEFT/RIGHT/FULL/CROSS with column disambiguation
- Deadlock detection: wait-for graph wired into TxnManager.write()
- TLS/SSL: OpenSSL via std/net for TCP wire protocol, auto self-signed certs
- Parameterized queries: ? placeholders with WireValue binding
- Wire protocol: mkQueryParams message support
- HTTP /query endpoint accepts JSON params array
- Nim client: query(sql, params) overload
- Tests: 262 passing (15 new)

All PLAN.md production blockers resolved.
2026-05-06 16:42:53 +03:00
dimgigov 856a07c030 feat: HTTP server in main entry point + background compaction scheduling
- baradadb.nim now starts both TCP wire protocol and HTTP REST servers
- HTTP server runs in background thread via hunos on port TCP+440
- WebSocket server auto-starts on HTTP port + 1
- CompactionManager with background async loop (default 60s interval)
- CompactionStrategy wired to LSMTree SSTables
- Fixed forward declaration for lowerExpr in executor.nim
- 216 tests passing
2026-05-06 14:23:13 +03:00
dimgigov 1fb715a1d4 feat: integrate hunos web server + jwt-nim-baraba + real WAL recovery + compaction
HTTP Server:
- Replaced asynchttpserver with hunos (multi-threaded, trie router, CORS middleware)
- JWT authentication via jwt-nim-baraba (HS256 with BearSSL crypto)
- POST /query, GET /health, GET /metrics, POST /auth, GET /api (OpenAPI spec)
- Proper CORS headers via hunos corsMiddleware

WAL Recovery:
- recover() now actually applies committed entries (REDO)
- Skips uncommitted entries (UNDO)
- Takes optional LSMTree parameter to apply recovery

SSTable Compaction:
- Real compaction: reads SSTable entries, merges by key (newest wins), writes merged file
- Deduplication: keeps only newest version per key
- Tombstone cleanup: removes deleted entries
- Old SSTable files deleted after merge

Dependencies:
- Added hunos >= 1.2.0 and jwt >= 2.1.0 to nimble

SSTable fields exported for compaction access.

All 216 tests pass
2026-05-06 13:42:37 +03:00
dimgigov 096c8347cf feat: complete Phase 3 + new production roadmap for Web/ERP
- Thread-safety: locks in LSMTree and Graph engines
- Raft network transport: async TCP, serialization, heartbeat, 3-node election test
- CI/CD: GitHub Actions workflow
- Cleanup: remove dead code, unused imports, build artifacts
- New PLAN.md targeting production Web/ERP readiness
- 216 tests passing
2026-05-06 10:40:34 +03:00
dimgigov f8909f155c feat: wire protocol integration in TCP server
- Server now reads binary wire protocol messages (header + payload)
- Implements recvExact for reliable message framing
- Query execution against LSM-Tree for SELECT/INSERT/DELETE
- SELECT supports point reads (WHERE key = 'value') and full scans
- INSERT parses simple EdgeDB-style syntax
- DELETE with WHERE clause
- Wire protocol responses: Data, Complete, Error, Pong
- Export wire protocol read/write helpers for external use
- Add scanMemTable to LSMTree for full scans
- Add GEL/ to .gitignore
- All 214 tests pass
2026-05-06 03:38:10 +03:00
dimgigov 5dba7b5699 feat: Phase 1 — SSTable persistence, README honesty, benchmark fix
- Add Current Status / Limitations section to README
- Fix benchmark compilation (Duration.ticks → inNanoseconds)
- Implement real SSTable binary format with write/read/mmap support
- Add BloomFilter serialize/deserialize for disk storage
- Fix mmap.nim to use posix.open instead of system.open
- New PLAN.md with improvement roadmap
- All 214 tests pass
2026-05-06 03:16:39 +03:00
dimgigov 805dbc8ad6 fix: comprehensive bug audit — 15 critical/high/medium fixes
Critical fixes:
- B-Tree splitChild: fixed out-of-bounds on child.children.len (mid+1..len → mid+1..<len)
- Wire protocol: all deserializeValue cases now properly assign to result
- Wire protocol: fkBool/fkInt8 now increment pos after reading
- MVCC: write-write conflict now detected for completed transactions
- MVCC: commit only marks visible versions as deleted (not concurrent uncommitted)
- Lexer: buffer overrun on backslash escape at end of input — added bounds check
- Bulgarian/Russian stemmers: fixed UTF-8 byte offsets for Cyrillic suffixes (2 bytes per char)

High fixes:
- WebSocket: added missing std/tables import
- Raft: added randomize() call for election timeouts
- MemTable.get: binary search O(log n) instead of linear scan O(n)

Medium fixes:
- Zerocopy: string length now written as 4-byte int32 (was single byte)
- Raft appendEntries: guard against nextIdx underflow
- Columnar: null tracking with nulls seq on ColumnPtr
- Removed dead code in lexer readIdent (redundant true/false checks)
- Updated Bulgarian stemming test to match corrected byte offsets

All 214 tests pass across 48 suites with 0 failures.
2026-05-06 02:46:37 +03:00
dimgigov b9f6059cbf feat: crash recovery, Raft election timer, CLI autocomplete, roadmap update — 246 tests
Crash Recovery:
- WAL file scanning with magic/version/entry parsing
- REDO/UNDO analysis — identify committed vs uncommitted txns
- Summary reporting with entry/txn/apply status

Raft Election Timer:
- ElectionTimer with configurable timeout and tick/check/reset
- Automatic election start on follower timeout
- Candidate timeout restart on failed elections

CLI Autocomplete:
- Full autocomplete engine for commands and SQL keywords
- Suggest function for single/multi completions
- 60+ keywords (SELECT, FROM, WHERE, JOIN, GROUP BY, etc.)

ROADMAP.md:
- All completed items properly marked
- Updated status table — 9 out of 12 phases at 90%+

14 new tests (246 total, all passing)
2026-05-06 02:12:49 +03:00
dimgigov b0a760c0ab feat: schema inheritance, codegen, replication, mmap — 136 tests
Schema:
- Inheritance: multi-level, property merging, override, isSubtype
- Computed properties with expressions
- getSubtypes for polymorphic queries

Codegen:
- IR plan → storage operations compilation
- Predicate pushdown optimization (filter → point read)
- Cost estimation and EXPLAIN output

Replication:
- Sync replication (wait for all replicas)
- Async replication (fire and forget)
- Semi-sync (wait for N replicas)
- Replica state tracking and lag monitoring

Storage:
- Memory-mapped I/O for SSTable access
- Sequential/random access hints (madvise)
- Page-level read operations

29 new tests (136 total, all passing)
2026-05-06 01:23:28 +03:00
dimgigov 67213826a8 feat: compaction, page cache, WebSocket, rate limiter, TF-IDF, fuzzy search, regex, metadata filter — 73 tests
- SSTable compaction: size-tiered strategy, level-based scheduling
- Page cache: LRU eviction, hit rate tracking, capacity management
- WebSocket: full duplex streaming, frame encoding, ping/pong
- Rate limiter: token bucket + sliding window algorithms
- FTS: TF-IDF ranking, Levenshtein fuzzy matching, wildcard regex
- Vector: metadata filtering on HNSW search
- 16 new tests (73 total, all passing)
2026-05-06 01:03:58 +03:00
dimgigov 07a37d8e78 feat: B-Tree, columnar engine, IR/type checker, connection pool, JWT auth, quantization, Louvain, pattern matching — 57 tests
- B-Tree index: insert, get, scan range, duplicate keys
- Columnar engine: batch ops, RLE/dict encoding, GroupBy, aggregates
- IR (Intermediate Representation): plan nodes, expressions, type checker
- Connection pool: load-balanced eviction, min/max connections
- JWT authentication with token verify and claims parsing
- Vector quantization: scalar 8-bit/4-bit, product quantization, binary
- Louvain community detection algorithm
- Graph pattern matching (subgraph isomorphism)
- 18 new test suites (57 total, all passing)
2026-05-06 00:57:30 +03:00
dimgigov 6935889877 feat: initial BaraDB — multimodal database engine in Nim
- LSM-Tree storage engine with WAL, bloom filter, MemTable
- BaraQL query language: lexer (80+ tokens), recursive descent parser, AST
- Vector engine: HNSW + IVF-PQ indexes, 4 distance metrics
- Graph engine: adjacency list, BFS/DFS, Dijkstra, PageRank
- Full-Text Search: inverted index, BM25 ranking, stemming, stop words
- Type system: 17 types (int/float/string/uuid/json/vector/...)
- Async TCP server
- 21 passing tests
2026-05-06 00:22:12 +03:00