baraba/Baradb
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
196 Commits 2 Branches 0 Tags
e23b1d61d2d0be4b58aa703d9ebf8dba093d27f6
Commit Graph

196 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dimgigov fecb52dca8 Remove reddit_post_draft.md 2026-05-08 01:32:47 +03:00
dimgigov c4b374e1db Initial commit 2026-05-08 01:13:54 +03:00
dimgigov 8f4299614d ci(clients): use docker healthcheck instead of nc 2026-05-08 00:38:19 +03:00
dimgigov 33194937db ci(clients): fix rust setup and nim server wait 2026-05-08 00:25:57 +03:00
dimgigov 13c2e94fb1 ci(clients): build Nim binaries in CI before Docker image 2026-05-08 00:21:31 +03:00
dimgigov ae2e07e626 feat(clients): professional clients with tests, CI/CD, and examples 
- Python: pyproject.toml, pytest suite (unit + integration), README, examples
- JavaScript: package.json, TypeScript definitions, node:test suite, README, examples
- Nim: integration tests, examples, README, improved nimble tasks
- Rust: fixed nodelay/localhost IPv6 bug, integration tests, examples, README
- Added GitHub Actions CI for all 4 clients against Docker server
- Added docker-compose.test.yml for local integration testing
- Added .gitignore for each client
 2026-05-08 00:18:34 +03:00
dimgigov e9d3c7e9de docs: add multi-language documentation (ru, fa, zh, tr, ar) 2026-05-07 23:42:55 +03:00
dimgigov abb382c419 feat(clients): sync all client drivers with current wire protocol 
- Python: fix column types read order, add auth/ping/query_params, proper recv
- JavaScript: wire up real TCP socket, add auth/ping/query_params, WireValue class
- Rust: add WireValue enum with structured types, fix type fidelity for ARRAY/OBJECT/VECTOR, add auth/query_params, fix ping/error parsing
- Nim: add missing MsgKind values, makeQueryParamsMessage, auth/ping, fix forward decl

All clients now support AUTH (0x09), QUERY_PARAMS (0x03), PING (0x08)
and align with server.nim + wire.nim protocol definitions.
 2026-05-07 22:38:09 +03:00
dimgigov 09247ffcad docs: add version badge at the top of README 2026-05-07 22:18:26 +03:00
dimgigov 8d39d633cf chore: update version strings to v1.0.0 in Docker and Nim code 
- Dockerfile, Dockerfile.source: LABEL version=1.0.0
- Dockerfile.source: update Nim base image 2.2.2 -> 2.2.10
- src/baradadb.nim, src/barabadb/core/httpserver.nim: v0.1.0 -> v1.0.0
- Rebuild baradadb binary with v1.0.0
 2026-05-07 22:06:04 +03:00
dimgigov 157775794d Release v1.0.0 
- 7 TLA+ specs formally verified (~69M states, 0 errors)
- 283+ Nim tests, all passing
- Raft: leader election + log replication + step-down on partition
- 2PC: coordinator crash/recovery + participant timeout
- MVCC: snapshot isolation + write skew detection
- Gossip: SWIM-like membership with strength-based gossip
- Replication: async/sync/semi-sync LSN tracking
- Sharding: consistent hashing with virtual nodes
- Deadlock detection: cycle detection in wait-for graph
- TLA+ faithfulness tests bridging specs to Nim code
- CI: GitHub Actions with automated TLC verification
 2026-05-07 22:01:43 +03:00
dimgigov 8f87491eca FV-14: TLA+ faithfulness tests for Nim state machines 
Add tests/tla_faithfulness.nim verifying Nim code against TLA+ invariants:
- Raft: ElectionSafety, LogMatching, CommittedIndexValid
- MVCC: NoDirtyReads, CommittedMustStart
- 2PC: Atomicity, RecoveryConsistency

Documented gap: Nim MVCC allows multiple committed versions per key,
while TLA+ spec enforces first-committer-wins (WriteWriteConflict).
This test ensures future changes to Nim code maintain alignment with specs.

Update PLAN.md (FV-14 done), CHANGELOG.md, test_all.nim includes new suite.
 2026-05-07 21:59:06 +03:00
dimgigov 4791434cea FV-2: Raft leader step-down on partition loss 
- Add heartbeatReceived variable tracking last term with heartbeat
- Add Heartbeat action: leader sends keep-alive; recipient steps down on higher term
- Add HeartbeatTimeout: follower starts election if no heartbeat in current term
- Add LeaderLeaseExpired: leader steps down if it loses quorum connectivity
- Add LeaderHasSelfHeartbeat invariant: every leader must have valid heartbeat
- raft.tla now models realistic partition behavior (38M states verified)
- Update CHANGELOG -> v1.2.0, VERSION -> 1.2.0, PLAN.md, README.md
 2026-05-07 21:48:03 +03:00
dimgigov 181e7d7019 docs: update README with FV v1.1.0 stats and all 7 specs 2026-05-07 21:22:06 +03:00
dimgigov f9e5169c3d FV-6: Liveness properties + fairness constraints 
- mvcc.tla: Add CommitProgress liveness (verified with WF_vars)
- raft/twopc/mvcc/gossip: Add Spec definitions with WF_vars(Next)
- gossip.tla: Fix LearnViaGossip strength-based filtering
  (prevents overwriting Dead/Suspect with weaker state)
- twopc.tla: Add DecideCommitAction/DecideAbortAction for SF fairness
- Update all .cfg files to use SPECIFICATION Spec
- Update CHANGELOG.md -> v1.1.0, VERSION -> 1.1.0, PLAN.md
 2026-05-07 21:21:22 +03:00
dimgigov 112ed4764d FV-7: MVCC write skew detection 
- mvcc.tla: Add NoWriteSkew invariant
- mvcc.tla: CommitTxn checks for circular read-write dependencies
- models/mvcc.cfg: Add NoWriteSkew to checked invariants
- Update PLAN.md, ANALYSIS.md, README.md
 2026-05-07 19:43:58 +03:00
dimgigov 39e07b542d FV-3, FV-4: 2PC coordinator crash/recovery + participant timeout 
- twopc.tla: Add coordinatorLog (persistent WAL for decisions)
- twopc.tla: Add CrashCoordinator action
- twopc.tla: Add RecoverCoordinator action (reads from coordinatorLog)
- twopc.tla: Add ParticipantTimeout (only if coordinatorLog = Nil)
- twopc.tla: Add RecoveryConsistency invariant
- models/twopc.cfg: Add RecoveryConsistency to checked invariants
- Update PLAN.md, ANALYSIS.md, README.md with new metrics
 2026-05-07 19:40:09 +03:00
dimgigov 54711d0190 FV-1, FV-13: Raft prevLogIndex/prevLogTerm + CI fix 
- raft.tla: Add HasCompatiblePrefix check (prevLogIndex/prevLogTerm)
- raft.tla: Add RejectAppendEntries action for follower rejection
- raft.tla: Add conflict truncation + commitIndex/matchIndex adjustment
- raft.tla: Restore LogMatching invariant
- raft.tla: Guard AppendEntry to prevent term gaps in leader log
- models/raft.cfg: Add LogMatching to checked invariants
- .github/workflows/ci.yml: Replace container with setup-java + cache
- run_all.sh: Use -workers auto and -XX:+UseParallelGC
- Update PLAN.md, PLAN_DONE.md, ANALYSIS.md, README.md
 2026-05-07 19:21:01 +03:00
dimgigov ce3b078979 docs: unified improvement plan for verification + software (53 tasks) 
Covers both TLA+ specs and Nim code:
- P1: Raft prevLogIndex/LogMatching, replication data transfer,
  sharding migration (14 tasks)
- P2: backup.tla, crossmodal.tla, recovery.tla + Nim tests (14 tasks)
- P3: Liveness properties, write skew, 2PC crash recovery (12 tasks)
- P4: CI fixes, symmetry reduction, missing tests, inter-module
  integration (13 tasks)

Ties verification improvements directly to code improvements.
 2026-05-07 18:39:59 +03:00
dimgigov e715454c4e docs(verify): add detailed improvement plan with 13 tasks across 4 priorities 
Covers Raft prevLogIndex fix, 2PC recovery, MVCC write skew,
liveness properties, backup.tla, crossmodal.tla, CI fixes,
symmetry reduction, and Apalache migration.
 2026-05-07 18:31:20 +03:00
dimgigov 19d0ff7366 feat(verify): expand formal verification to 7 specs, add versioning, CI integration 
Added 3 new TLA+ specs:
- gossip.tla — SWIM gossip membership & failure detection
- deadlock.tla — wait-for graph deadlock detection
- sharding.tla — consistent hashing shard assignment

Improved existing 4 specs:
- raft.tla: added CommittedIndexValid invariant
- twopc.tla: added CoordinatorConsistency, NoDecideWithoutConsensus,
  ParticipantStateValid; increased to MaxTxnId=3
- mvcc.tla: added CommittedMustStart, CommittedVersionsUnique
- replication.tla: added AppliedLteCurrent, SemiSyncQuorum

Infrastructure:
- VERSION file (1.0.0)
- CHANGELOG.md
- ANALYSIS.md (weak spots & improvement plan)
- run_all.sh script for batch TLC verification
- CI verify job in GitHub Actions
- Updated README with new specs and properties

Total: 7 specs, 26 invariants, 11.6M states checked, 0 errors
 2026-05-07 18:26:55 +03:00
dimgigov 215df1cdf3 docs: update README and all docs with formal verification, new BaraQL features, OpenTelemetry tracing 
- Add Formal Verification section to README and architecture docs
- Document TLA+ specs for Raft, 2PC, MVCC, Replication
- Add new BaraQL features: JSON path (->, ->>), FTS @@, CREATE INDEX USING FTS,
  RECOVER TO TIMESTAMP, UNION/INTERSECT/EXCEPT
- Add OpenTelemetry tracing example to README
- Update Quick Start to use nimble test/bench
- Update EN and BG documentation
 2026-05-07 16:06:13 +03:00
dimgigov e73bfb450c feat: formal verification suite — TLA+ specs for Raft, 2PC, MVCC, Replication 
Verified with TLC model checker:
- Raft: 475k states, ElectionSafety + StateMachineSafety
- 2PC: 22k states, Atomicity + NoOrphanBlocks
- MVCC: 177k states, NoDirtyReads + ReadOwnWrites + WriteWriteConflict
- Replication: 3.6M states, MonotonicLsn + AcksRemovePending
 2026-05-07 15:57:33 +03:00
dimgigov c8633b9589 fix: non-blocking replication/disttxn timeouts, OTLP epoch time, test paths, compiler warnings 2026-05-07 15:41:33 +03:00
dimgigov d3b38251c3 update core modules and project config 2026-05-07 15:16:40 +03:00
dimgigov c2dc280ddd fix(raft): state machine callback, start from main, remove asyncCheck, config 
- applyCommand callback: committed entries applied to state machine
- lastApplied incremented on commit via applyCommitted()
- RaftNetwork.run() started from main() when raftEnabled=true
- asyncCheck replaced with try/await in processMessage
- bindAddr without hardcoded 127.0.0.1
- raft config: raftEnabled/Port/Peers/NodeId + env vars
- 283 tests, 0 failures
 2026-05-07 14:22:04 +03:00
dimgigov d259c1a1fc fix(distributed): 2PC atomicity, DISTTXN handler, sharding bugs, gossip timers 
- disttxn: prepare failure rolls back already-prepared participants (C1)
- disttxn: commit failure rolls back committed participants (C2)
- server: DISTTXN handler uses real DistTxnManager (C3)
- server: DistTxnManager initialized in newServer() (H1)
- sharding: getShardRange returns -1 instead of 0 for out-of-range keys (M6)
- sharding: binary search in consistent hashing ring (M7)
- gossip: startHealthCheck() and startGossipRound() async loops
- 283 tests, 0 failures
 2026-05-07 14:13:17 +03:00
dimgigov 32187099dd feat: full FTS BM25 integration — CREATE INDEX USING FTS, BM25 ranking on @@ 
- evalExpr accepts optional ExecutionContext for FTS index lookup
- CREATE INDEX ... USING FTS builds InvertedIndex from existing data
- @@ uses BM25 scoring when FTS index exists, falls back to term match
- INSERT/UPDATE/DELETE auto-maintain FTS indexes
- 283 tests, 0 failures
 2026-05-07 13:58:18 +03:00
dimgigov b1aadf77f9 feat: PITR RECOVER TO TIMESTAMP, OpenTelemetry OTLP export, FTS infrastructure 
- RECOVER TO TIMESTAMP '...' — parser + executor, replays WAL entries
- core/tracing.nim — OTLP/HTTP export via exportOtlp()
- FTS: ftsIndexes table in ExecutionContext, engine import ready
- 281 tests, 0 failures
 2026-05-07 13:44:59 +03:00
dimgigov 651375a156 feat: production hardening — JWT security, WAL reader, tracing, 2PC real RPC, stress test fix 
- JWT: getEffectiveJwtSecret() helper + warning log when not configured
- WAL: readEntries() with timestamp filter for PITR
- Tracing: core/tracing.nim with span recording + executeQuery integration
- 2PC: real TCP RPC via sendDistTxnRpc (host='' = local fallback)
- Stress test: updated comment, 10K ops confirmed with internal locks
- SSTable metadata comments clarified (offsets patched correctly)
- addParticipant has default params (host='', port=0)
 2026-05-07 13:31:29 +03:00
dimgigov 5deb38feb2 feat: production polish — recursive CTE, UNION/INTERSECT/EXCEPT, DROP INDEX, JSON path, FTS SQL, covering index, SCRAM auth 
- Recursive CTE execution (WITH RECURSIVE + UNION ALL, work-table loop)
- UNION / INTERSECT / EXCEPT parser + executor (nkSetOp)
- DROP INDEX parser + executor
- VIEW DDL persistence via AST-to-SQL serializer
- JSON path operators -> and ->> (lexer, parser, IR, executor)
- FTS SQL wiring WHERE col @@ 'query' (case-insensitive term match)
- Multi-column index range scans (prefix equality + range on last col)
- Covering index optimization (skip LSM read when index covers SELECT)
- SCRAM-SHA-256 authentication (password-based validation)
- 279 tests, 0 failures
 2026-05-07 13:00:36 +03:00
dimgigov 7faae09ad3 docs: add reddit showcase post draft 2026-05-07 11:39:52 +03:00
dimgigov f45f6b7127 fix(ci): add hunos git url to nimble deps, install libpcre3-dev for runtime 2026-05-07 11:27:43 +03:00
dimgigov 2a3d50c185 ci: clean up debug steps, keep libssl-dev and depsOnly fixes 2026-05-07 11:17:58 +03:00
dimgigov 9c7dc5579e ci(debug): capture test output to logs for debugging 2026-05-07 11:15:41 +03:00
dimgigov aa63c29c4c fix(ci): install libssl-dev for OpenSSL support 2026-05-07 11:13:20 +03:00
dimgigov a1aad227c1 fix(ci): lower jwt requirement to >= 0.3.0 to match upstream repo version 2026-05-07 11:10:45 +03:00
dimgigov a90d92867b ci(debug): add debug step to inspect nimble state 2026-05-07 10:55:36 +03:00
dimgigov 2bb49a9cb7 fix(ci): use nimble install --depsOnly to avoid building project during dep install 2026-05-07 10:53:39 +03:00
dimgigov a426255036 fix(ci): install nimble dependencies before compiling 2026-05-07 10:47:02 +03:00
dimgigov 2c6582cc37 fix(tests): use getMonoTime().ticks for unique temp dirs to avoid CI collisions 2026-05-07 10:38:28 +03:00
dimgigov 773c142f74 fix(ci): add -d:ssl flag to all nim compile steps 2026-05-07 10:32:28 +03:00
dimgigov eaa5760fd4 feat: JSON/JSONB validation, multi-column indexes, CTE execution, wire protocol column types 
- Add fkJson to wire protocol with serialization/deserialization
- Validate JSON/JSONB on INSERT/UPDATE via std/json
- Send real column type metadata in wire protocol responses
- Update all 4 clients (Nim, Python, JS, Rust) for JSON and columnTypes
- Implement multi-column index parser (CREATE INDEX idx ON t(a, b))
- Add ciColumns to AST nkCreateIndex
- Build composite index keys as table.col1.col2 with val1|val2
- Support multi-column exact match in SELECT for AND chains
- Implement non-recursive CTE execution via ctx.cteTables materialization
- Add tkRecursive token and parse WITH RECURSIVE
- Fix test isolation: use temp dirs for JOIN, migration, and RLS tests
- All tests passing (0 failures)
 2026-05-07 10:27:40 +03:00
dimgigov ca345eb256 feat: B-Tree range scan support in query executor 
- Wire btree.scan() for BETWEEN, >, >=, <, <= conditions
- When a B-Tree index exists and WHERE clause is a range condition,
  the executor now uses index range scan instead of full table scan
- Add 3 unit tests: BETWEEN, >, <=
- Update PLAN.md / PLAN_DONE.md
 2026-05-07 00:48:05 +03:00
dimgigov 29425c45f8 Add WebSocket auth — JWT token validation on WS upgrade handshake 
- websocket.nim: check Authorization Bearer token before WebSocket upgrade
- httpserver.nim: pass config + jwtSecret to WsServer
 2026-05-07 00:38:38 +03:00
dimgigov 18af892232 Rename PLAN.md → PLAN_DONE.md, create new PLAN.md with remaining tasks 
- PLAN_DONE.md: everything completed so far
- PLAN.md: only real remaining work (no Partitioning, no Kubernetes)
- Explicitly marks WebSocket auth, B-Tree range scans, JSON types as critical
 2026-05-07 00:32:51 +03:00
dimgigov 08f73598a5 Fix WAL crash recovery — data survives power loss 
- wal.nim: open existing WAL with fmAppend instead of fmWrite (was truncating!)
- lsm.nim: replay WAL entries into memTable on startup before accepting writes
- Fixes real data loss on crash — previously WAL was wiped on every restart
 2026-05-07 00:29:43 +03:00
dimgigov b65b066c36 Fix auth wiring: HTTP + TCP wire protocol authentication 
- httpserver.nim:
  - Use config.jwtSecret instead of hardcoded default
  - Enforce auth on /query, /tables, /metrics when authEnabled=true
  - /auth endpoint now validates password against jwtSecret
- server.nim:
  - Add TCP wire protocol auth (mkAuth message)
  - Reject queries with 401 until client authenticates with valid JWT
- wire.nim: Add makeAuthOkMessage, makeAuthChallengeMessage, parseAuthMessage helpers
 2026-05-07 00:21:00 +03:00
dimgigov 011bc5bef4 Fix production gaps: config file loading, structured JSON logging, update PLAN 
- config.nim: Add baradb.json parsing + full env var support (BARADB_*)
- logging.nim: Wire structured JSON logger into server.nim and baradadb.nim
- server.nim: Replace echo with info/warn/errorMsg logging calls
- baradadb.nim: Init logger from config (logLevel, logFile)
- PLAN.md: Update honest assessment — deadlock, compaction, JOIN, TLS, slow query all verified working
 2026-05-07 00:09:09 +03:00
dimgigov b6cdf8c88e Add Docker deployment support with compose, entrypoint and docs 
- Add Dockerfile (pre-built binary) and Dockerfile.source
- Add docker-compose.yml, docker-compose.prod.yml, docker-compose.override.yml
- Add docker-entrypoint.sh with non-root user support
- Add .dockerignore and helper scripts (scripts/docker-build.sh, scripts/docker-run.sh)
- Add docs/en/docker.md with full Docker guide
- Update docs/en/deployment.md and README.md
- Fix src/barabadb/core/config.nim to read BARADB_ADDRESS, BARADB_PORT, BARADB_DATA_DIR from env
- Fix src/barabadb/core/httpserver.nim missing wire import
 2026-05-06 23:58:43 +03:00