baraba/Baradb
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
180 Commits 2 Branches 0 Tags
cd62f9864181b1f7fa18dea25ace81989f58411f
Commit Graph

180 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dimgigov 39e07b542d FV-3, FV-4: 2PC coordinator crash/recovery + participant timeout 
- twopc.tla: Add coordinatorLog (persistent WAL for decisions)
- twopc.tla: Add CrashCoordinator action
- twopc.tla: Add RecoverCoordinator action (reads from coordinatorLog)
- twopc.tla: Add ParticipantTimeout (only if coordinatorLog = Nil)
- twopc.tla: Add RecoveryConsistency invariant
- models/twopc.cfg: Add RecoveryConsistency to checked invariants
- Update PLAN.md, ANALYSIS.md, README.md with new metrics
 2026-05-07 19:40:09 +03:00
dimgigov 54711d0190 FV-1, FV-13: Raft prevLogIndex/prevLogTerm + CI fix 
- raft.tla: Add HasCompatiblePrefix check (prevLogIndex/prevLogTerm)
- raft.tla: Add RejectAppendEntries action for follower rejection
- raft.tla: Add conflict truncation + commitIndex/matchIndex adjustment
- raft.tla: Restore LogMatching invariant
- raft.tla: Guard AppendEntry to prevent term gaps in leader log
- models/raft.cfg: Add LogMatching to checked invariants
- .github/workflows/ci.yml: Replace container with setup-java + cache
- run_all.sh: Use -workers auto and -XX:+UseParallelGC
- Update PLAN.md, PLAN_DONE.md, ANALYSIS.md, README.md
 2026-05-07 19:21:01 +03:00
dimgigov ce3b078979 docs: unified improvement plan for verification + software (53 tasks) 
Covers both TLA+ specs and Nim code:
- P1: Raft prevLogIndex/LogMatching, replication data transfer,
  sharding migration (14 tasks)
- P2: backup.tla, crossmodal.tla, recovery.tla + Nim tests (14 tasks)
- P3: Liveness properties, write skew, 2PC crash recovery (12 tasks)
- P4: CI fixes, symmetry reduction, missing tests, inter-module
  integration (13 tasks)

Ties verification improvements directly to code improvements.
 2026-05-07 18:39:59 +03:00
dimgigov e715454c4e docs(verify): add detailed improvement plan with 13 tasks across 4 priorities 
Covers Raft prevLogIndex fix, 2PC recovery, MVCC write skew,
liveness properties, backup.tla, crossmodal.tla, CI fixes,
symmetry reduction, and Apalache migration.
 2026-05-07 18:31:20 +03:00
dimgigov 19d0ff7366 feat(verify): expand formal verification to 7 specs, add versioning, CI integration 
Added 3 new TLA+ specs:
- gossip.tla — SWIM gossip membership & failure detection
- deadlock.tla — wait-for graph deadlock detection
- sharding.tla — consistent hashing shard assignment

Improved existing 4 specs:
- raft.tla: added CommittedIndexValid invariant
- twopc.tla: added CoordinatorConsistency, NoDecideWithoutConsensus,
  ParticipantStateValid; increased to MaxTxnId=3
- mvcc.tla: added CommittedMustStart, CommittedVersionsUnique
- replication.tla: added AppliedLteCurrent, SemiSyncQuorum

Infrastructure:
- VERSION file (1.0.0)
- CHANGELOG.md
- ANALYSIS.md (weak spots & improvement plan)
- run_all.sh script for batch TLC verification
- CI verify job in GitHub Actions
- Updated README with new specs and properties

Total: 7 specs, 26 invariants, 11.6M states checked, 0 errors
 2026-05-07 18:26:55 +03:00
dimgigov 215df1cdf3 docs: update README and all docs with formal verification, new BaraQL features, OpenTelemetry tracing 
- Add Formal Verification section to README and architecture docs
- Document TLA+ specs for Raft, 2PC, MVCC, Replication
- Add new BaraQL features: JSON path (->, ->>), FTS @@, CREATE INDEX USING FTS,
  RECOVER TO TIMESTAMP, UNION/INTERSECT/EXCEPT
- Add OpenTelemetry tracing example to README
- Update Quick Start to use nimble test/bench
- Update EN and BG documentation
 2026-05-07 16:06:13 +03:00
dimgigov e73bfb450c feat: formal verification suite — TLA+ specs for Raft, 2PC, MVCC, Replication 
Verified with TLC model checker:
- Raft: 475k states, ElectionSafety + StateMachineSafety
- 2PC: 22k states, Atomicity + NoOrphanBlocks
- MVCC: 177k states, NoDirtyReads + ReadOwnWrites + WriteWriteConflict
- Replication: 3.6M states, MonotonicLsn + AcksRemovePending
 2026-05-07 15:57:33 +03:00
dimgigov c8633b9589 fix: non-blocking replication/disttxn timeouts, OTLP epoch time, test paths, compiler warnings 2026-05-07 15:41:33 +03:00
dimgigov d3b38251c3 update core modules and project config 2026-05-07 15:16:40 +03:00
dimgigov c2dc280ddd fix(raft): state machine callback, start from main, remove asyncCheck, config 
- applyCommand callback: committed entries applied to state machine
- lastApplied incremented on commit via applyCommitted()
- RaftNetwork.run() started from main() when raftEnabled=true
- asyncCheck replaced with try/await in processMessage
- bindAddr without hardcoded 127.0.0.1
- raft config: raftEnabled/Port/Peers/NodeId + env vars
- 283 tests, 0 failures
 2026-05-07 14:22:04 +03:00
dimgigov d259c1a1fc fix(distributed): 2PC atomicity, DISTTXN handler, sharding bugs, gossip timers 
- disttxn: prepare failure rolls back already-prepared participants (C1)
- disttxn: commit failure rolls back committed participants (C2)
- server: DISTTXN handler uses real DistTxnManager (C3)
- server: DistTxnManager initialized in newServer() (H1)
- sharding: getShardRange returns -1 instead of 0 for out-of-range keys (M6)
- sharding: binary search in consistent hashing ring (M7)
- gossip: startHealthCheck() and startGossipRound() async loops
- 283 tests, 0 failures
 2026-05-07 14:13:17 +03:00
dimgigov 32187099dd feat: full FTS BM25 integration — CREATE INDEX USING FTS, BM25 ranking on @@ 
- evalExpr accepts optional ExecutionContext for FTS index lookup
- CREATE INDEX ... USING FTS builds InvertedIndex from existing data
- @@ uses BM25 scoring when FTS index exists, falls back to term match
- INSERT/UPDATE/DELETE auto-maintain FTS indexes
- 283 tests, 0 failures
 2026-05-07 13:58:18 +03:00
dimgigov b1aadf77f9 feat: PITR RECOVER TO TIMESTAMP, OpenTelemetry OTLP export, FTS infrastructure 
- RECOVER TO TIMESTAMP '...' — parser + executor, replays WAL entries
- core/tracing.nim — OTLP/HTTP export via exportOtlp()
- FTS: ftsIndexes table in ExecutionContext, engine import ready
- 281 tests, 0 failures
 2026-05-07 13:44:59 +03:00
dimgigov 651375a156 feat: production hardening — JWT security, WAL reader, tracing, 2PC real RPC, stress test fix 
- JWT: getEffectiveJwtSecret() helper + warning log when not configured
- WAL: readEntries() with timestamp filter for PITR
- Tracing: core/tracing.nim with span recording + executeQuery integration
- 2PC: real TCP RPC via sendDistTxnRpc (host='' = local fallback)
- Stress test: updated comment, 10K ops confirmed with internal locks
- SSTable metadata comments clarified (offsets patched correctly)
- addParticipant has default params (host='', port=0)
 2026-05-07 13:31:29 +03:00
dimgigov 5deb38feb2 feat: production polish — recursive CTE, UNION/INTERSECT/EXCEPT, DROP INDEX, JSON path, FTS SQL, covering index, SCRAM auth 
- Recursive CTE execution (WITH RECURSIVE + UNION ALL, work-table loop)
- UNION / INTERSECT / EXCEPT parser + executor (nkSetOp)
- DROP INDEX parser + executor
- VIEW DDL persistence via AST-to-SQL serializer
- JSON path operators -> and ->> (lexer, parser, IR, executor)
- FTS SQL wiring WHERE col @@ 'query' (case-insensitive term match)
- Multi-column index range scans (prefix equality + range on last col)
- Covering index optimization (skip LSM read when index covers SELECT)
- SCRAM-SHA-256 authentication (password-based validation)
- 279 tests, 0 failures
 2026-05-07 13:00:36 +03:00
dimgigov 7faae09ad3 docs: add reddit showcase post draft 2026-05-07 11:39:52 +03:00
dimgigov f45f6b7127 fix(ci): add hunos git url to nimble deps, install libpcre3-dev for runtime 2026-05-07 11:27:43 +03:00
dimgigov 2a3d50c185 ci: clean up debug steps, keep libssl-dev and depsOnly fixes 2026-05-07 11:17:58 +03:00
dimgigov 9c7dc5579e ci(debug): capture test output to logs for debugging 2026-05-07 11:15:41 +03:00
dimgigov aa63c29c4c fix(ci): install libssl-dev for OpenSSL support 2026-05-07 11:13:20 +03:00
dimgigov a1aad227c1 fix(ci): lower jwt requirement to >= 0.3.0 to match upstream repo version 2026-05-07 11:10:45 +03:00
dimgigov a90d92867b ci(debug): add debug step to inspect nimble state 2026-05-07 10:55:36 +03:00
dimgigov 2bb49a9cb7 fix(ci): use nimble install --depsOnly to avoid building project during dep install 2026-05-07 10:53:39 +03:00
dimgigov a426255036 fix(ci): install nimble dependencies before compiling 2026-05-07 10:47:02 +03:00
dimgigov 2c6582cc37 fix(tests): use getMonoTime().ticks for unique temp dirs to avoid CI collisions 2026-05-07 10:38:28 +03:00
dimgigov 773c142f74 fix(ci): add -d:ssl flag to all nim compile steps 2026-05-07 10:32:28 +03:00
dimgigov eaa5760fd4 feat: JSON/JSONB validation, multi-column indexes, CTE execution, wire protocol column types 
- Add fkJson to wire protocol with serialization/deserialization
- Validate JSON/JSONB on INSERT/UPDATE via std/json
- Send real column type metadata in wire protocol responses
- Update all 4 clients (Nim, Python, JS, Rust) for JSON and columnTypes
- Implement multi-column index parser (CREATE INDEX idx ON t(a, b))
- Add ciColumns to AST nkCreateIndex
- Build composite index keys as table.col1.col2 with val1|val2
- Support multi-column exact match in SELECT for AND chains
- Implement non-recursive CTE execution via ctx.cteTables materialization
- Add tkRecursive token and parse WITH RECURSIVE
- Fix test isolation: use temp dirs for JOIN, migration, and RLS tests
- All tests passing (0 failures)
 2026-05-07 10:27:40 +03:00
dimgigov ca345eb256 feat: B-Tree range scan support in query executor 
- Wire btree.scan() for BETWEEN, >, >=, <, <= conditions
- When a B-Tree index exists and WHERE clause is a range condition,
  the executor now uses index range scan instead of full table scan
- Add 3 unit tests: BETWEEN, >, <=
- Update PLAN.md / PLAN_DONE.md
 2026-05-07 00:48:05 +03:00
dimgigov 29425c45f8 Add WebSocket auth — JWT token validation on WS upgrade handshake 
- websocket.nim: check Authorization Bearer token before WebSocket upgrade
- httpserver.nim: pass config + jwtSecret to WsServer
 2026-05-07 00:38:38 +03:00
dimgigov 18af892232 Rename PLAN.md → PLAN_DONE.md, create new PLAN.md with remaining tasks 
- PLAN_DONE.md: everything completed so far
- PLAN.md: only real remaining work (no Partitioning, no Kubernetes)
- Explicitly marks WebSocket auth, B-Tree range scans, JSON types as critical
 2026-05-07 00:32:51 +03:00
dimgigov 08f73598a5 Fix WAL crash recovery — data survives power loss 
- wal.nim: open existing WAL with fmAppend instead of fmWrite (was truncating!)
- lsm.nim: replay WAL entries into memTable on startup before accepting writes
- Fixes real data loss on crash — previously WAL was wiped on every restart
 2026-05-07 00:29:43 +03:00
dimgigov b65b066c36 Fix auth wiring: HTTP + TCP wire protocol authentication 
- httpserver.nim:
  - Use config.jwtSecret instead of hardcoded default
  - Enforce auth on /query, /tables, /metrics when authEnabled=true
  - /auth endpoint now validates password against jwtSecret
- server.nim:
  - Add TCP wire protocol auth (mkAuth message)
  - Reject queries with 401 until client authenticates with valid JWT
- wire.nim: Add makeAuthOkMessage, makeAuthChallengeMessage, parseAuthMessage helpers
 2026-05-07 00:21:00 +03:00
dimgigov 011bc5bef4 Fix production gaps: config file loading, structured JSON logging, update PLAN 
- config.nim: Add baradb.json parsing + full env var support (BARADB_*)
- logging.nim: Wire structured JSON logger into server.nim and baradadb.nim
- server.nim: Replace echo with info/warn/errorMsg logging calls
- baradadb.nim: Init logger from config (logLevel, logFile)
- PLAN.md: Update honest assessment — deadlock, compaction, JOIN, TLS, slow query all verified working
 2026-05-07 00:09:09 +03:00
dimgigov b6cdf8c88e Add Docker deployment support with compose, entrypoint and docs 
- Add Dockerfile (pre-built binary) and Dockerfile.source
- Add docker-compose.yml, docker-compose.prod.yml, docker-compose.override.yml
- Add docker-entrypoint.sh with non-root user support
- Add .dockerignore and helper scripts (scripts/docker-build.sh, scripts/docker-run.sh)
- Add docs/en/docker.md with full Docker guide
- Update docs/en/deployment.md and README.md
- Fix src/barabadb/core/config.nim to read BARADB_ADDRESS, BARADB_PORT, BARADB_DATA_DIR from env
- Fix src/barabadb/core/httpserver.nim missing wire import
 2026-05-06 23:58:43 +03:00
dimgigov 9b7ed1fca8 chore: change default ports to non-standard ones (9472, 9470, 9471) 2026-05-06 23:23:29 +03:00
dimgigov be370fb76b feat(backup): add dry-run, force, history log, auto-verify, rollback, disk check 2026-05-06 23:20:08 +03:00
dimgigov 7f2f5664fa feat(backup): improve backup manager with detailed CLI, verification, retention 2026-05-06 23:16:00 +03:00
dimgigov c3c14d64df docs: update README logo to new PNG image 2026-05-06 23:09:27 +03:00
dimgigov f7ded185a0 fix: production readiness — duplicate SSTable, missing eval handlers, connection limits, timeouts, slow query log, wire types 
- Remove duplicate SSTable add loop in baradadb.nim
- Fix missing evalExpr handlers for IN/BETWEEN/ILIKE/MOD/POW operators
- Enforce maxConnections limit in TCP server accept loop
- Preserve wire protocol value types (int/float/bool/null) from column metadata
- Add idle timeout (recv with deadline) and query timeout config
- Add slow query log (queries > threshold logged to file with timing)
- Update PLAN.md to reflect actual state (phases A, B, C complete, score 9.5/10)
 2026-05-06 18:41:29 +03:00
dimgigov 51da05b0da Add logo image file 2026-05-06 17:47:02 +03:00
dimgigov db7515b338 Add logo image to README 2026-05-06 17:45:27 +03:00
dimgigov 8993cdc6f3 docs: expand README and documentation for production release 
- Update README.md status from 'educational proof-of-concept' to 'production-ready'
- Fix binary size: 286KB -> 3.3MB
- Update test count: 162/35 -> 262/56
- Add sections: benchmarks, Docker, clients, security, config, monitoring, backup, cross-modal queries, troubleshooting
- Expand project structure with all 49 modules and ~14,100 LOC
- Add 10 new docs: performance, deployment, configuration, clients, security, monitoring, backup, crossmodal, troubleshooting, changelog
- Expand docs/en: architecture, baraql, installation, protocol
- Update docs/bg: architecture, installation
- Update docs/index.md with new links
- Update .gitignore for __pycache__, rust/target, nim binaries
 2026-05-06 17:19:16 +03:00
dimgigov e1bae0c7a0 Add comprehensive documentation with i18n support (EN/BG) 
- Add docs/ folder with English (en/) and Bulgarian (bg/) documentation
- Create index.md with language switching and links
- English docs: installation, quickstart, architecture, baraql, storage,
  schema, lsm, btree, vector, graph, fts, columnar, transactions,
  distributed, protocol, udf, api-binary, api-http, api-websocket
- Bulgarian docs: installation, quickstart, architecture, baraql,
  schema, lsm, btree, vector, graph, fts, transactions, distributed
- Update README license to BSD 3-Clause
- Add LICENSE file with BSD 3-Clause text
 2026-05-06 16:51:14 +03:00
dimgigov f9f77b3a18 feat: production blockers — JOINs, deadlock detection, TLS, parameterized queries 
- JOIN execution: INNER/LEFT/RIGHT/FULL/CROSS with column disambiguation
- Deadlock detection: wait-for graph wired into TxnManager.write()
- TLS/SSL: OpenSSL via std/net for TCP wire protocol, auto self-signed certs
- Parameterized queries: ? placeholders with WireValue binding
- Wire protocol: mkQueryParams message support
- HTTP /query endpoint accepts JSON params array
- Nim client: query(sql, params) overload
- Tests: 262 passing (15 new)

All PLAN.md production blockers resolved.
 2026-05-06 16:42:53 +03:00
dimgigov 856a07c030 feat: HTTP server in main entry point + background compaction scheduling 
- baradadb.nim now starts both TCP wire protocol and HTTP REST servers
- HTTP server runs in background thread via hunos on port TCP+440
- WebSocket server auto-starts on HTTP port + 1
- CompactionManager with background async loop (default 60s interval)
- CompactionStrategy wired to LSMTree SSTables
- Fixed forward declaration for lowerExpr in executor.nim
- 216 tests passing
 2026-05-06 14:23:13 +03:00
dimgigov f135d8c61d feat: wire WebSocket broadcast, CHECK constraints, VIEW persistence, migration tracking 
- WebSocket: WsServer wired to HttpServer via ExecutionContext.onChange
  - INSERT/UPDATE/DELETE now broadcast to subscribed WebSocket clients
- CHECK constraints: evaluate AST expressions via evalExpr(lowerExpr())
  - Works on both INSERT and UPDATE paths
- VIEW persistence: CREATE VIEW stores to LSM-Tree, restoreSchema loads views
- Migration tracking: APPLY MIGRATION marks applied with _schema:migrations:applied:<name>
- Update PLAN.md honesty for WebSocket, CHECK, Views, Migrations
- 216 tests passing
 2026-05-06 14:12:52 +03:00
dimgigov 47eda4c5c3 fix: APPLY MIGRATION parser + add std/tables import to server.nim 
- Fixed parseApplyMigration to use p.expect(tkApply) instead of p.expect(tkIdent)
- Added std/tables import to server.nim for Row table operations
- All 216 tests pass + 15/15 integration tests pass
 2026-05-06 13:58:11 +03:00
dimgigov 675ab26a6e feat: Phase 5-6 — CREATE VIEW, migrations, deadlock detection, JSON logging, admin UI 
Phase 5 — ERP Features:
- CREATE VIEW name AS SELECT ... — parser + executor with view expansion
- DROP VIEW — parser + executor
- CREATE MIGRATION name AS 'sql' — parser + executor, stored in LSM-Tree
- APPLY MIGRATION name — parser + executor, replays stored migration SQL
- Views table in ExecutionContext, expanded on SELECT FROM view

Phase 6 — Production Readiness:
- Deadlock detection: timeout-based auto-abort for stale transactions (30s default)
- TxnManager.txnTimeoutMs configurable
- Structured JSON logging module (logging.nim) with levels: debug/info/warn/error
- Admin dashboard Web UI at GET /admin — SQL playground, schema browser, metrics
- Dark theme HTML/CSS with tabs

Lexer: added tkView, tkMigration, tkApply tokens
Parser: parseCreateView, parseDropView, parseCreateMigration, parseApplyMigration
AST: nkCreateView, nkDropView, nkCreateMigration, nkApplyMigration nodes

All 216 tests pass
 2026-05-06 13:54:16 +03:00
dimgigov 633c5d127c docs: update PLAN.md — reflect hunos, jwt-nim-baraba, WAL recovery, compaction, FK, type enforcement 
Updated status for:
- Phase 0: ALTER TABLE ADD COLUMN, CREATE INDEX, type enforcement all done
- Phase 1: FK enforcement, type validation (INTEGER/FLOAT/BOOLEAN/TIMESTAMP), GROUP BY done
- Phase 2: WAL recovery (REDO/UNDO), SSTable compaction (real merge) done
- Phase 3: hunos web server, jwt-nim-baraba (HS256 BearSSL), CORS, rate limiting done
- Phase 4: WebSocket broadcast wired to executor
- Dependencies section added (hunos, jwt-nim-baraba)
- Score updated to 8.5/10
 2026-05-06 13:45:15 +03:00
dimgigov 1fb715a1d4 feat: integrate hunos web server + jwt-nim-baraba + real WAL recovery + compaction 
HTTP Server:
- Replaced asynchttpserver with hunos (multi-threaded, trie router, CORS middleware)
- JWT authentication via jwt-nim-baraba (HS256 with BearSSL crypto)
- POST /query, GET /health, GET /metrics, POST /auth, GET /api (OpenAPI spec)
- Proper CORS headers via hunos corsMiddleware

WAL Recovery:
- recover() now actually applies committed entries (REDO)
- Skips uncommitted entries (UNDO)
- Takes optional LSMTree parameter to apply recovery

SSTable Compaction:
- Real compaction: reads SSTable entries, merges by key (newest wins), writes merged file
- Deduplication: keeps only newest version per key
- Tombstone cleanup: removes deleted entries
- Old SSTable files deleted after merge

Dependencies:
- Added hunos >= 1.2.0 and jwt >= 2.1.0 to nimble

SSTable fields exported for compaction access.

All 216 tests pass
 2026-05-06 13:42:37 +03:00