dimgigov
f2b7ed1ce2
test(raft): 3-node TLS cluster e2e with plaintext rejection
2026-07-31 01:55:43 +03:00
dimgigov
ed89c88afa
feat(raft): optional TLS on raft transport (server + dialer)
2026-07-31 01:31:20 +03:00
dimgigov
8d083f5fdc
feat(raft): TLS config surface with fail-closed startup
2026-07-31 01:14:20 +03:00
dimgigov
efa46b05c6
ci(raft): dedicated mandatory raft e2e job; no silent skips under CI
2026-07-31 01:07:11 +03:00
dimgigov
431334b70a
fix(raft): distinguish put-with-empty-value from delete in write path
2026-07-31 00:58:51 +03:00
dimgigov
63cb05afe2
test(raft): failover under sustained write load e2e
2026-07-31 00:35:55 +03:00
dimgigov
09f462f467
release: v1.2.0 Production GA (single-node)
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- known-limitations, deployment runbook, release checklist
- prod compose requires JWT secret; BARADB_ENV=production fail-closed
- scripts/backup-restore-drill.sh (backup → wipe → restore → verify)
- version bump 1.2.0 (nimble, Dockerfile, health, CHANGELOG dated)
2026-07-30 21:52:21 +03:00
dimgigov
1b3c26123a
feat(raft): expose raft metrics on /metrics and /health
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- RaftMetrics counters: elections, term changes, appends, commit waits/ms,
timeouts, lost leadership, forwards, applies, compactions
- Gauges via prometheusText: is_leader, term, log size, commit/applied,
apply lag, snapshot index
- Wire httpServer.raftNode; extend GET /metrics and GET /health
2026-07-30 21:38:44 +03:00
dimgigov
53704e1036
feat(raft): safe log compaction with snapshot metadata
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- lastSnapshotIndex/Term bound the compacted prefix; lastLogIndex/Term
and AppendEntries prevLog checks respect the snapshot base
- compactLog drops entries only through min(matchIndex, lastApplied) on
the leader so lagging peers can still catch up via AppendEntries
- Persist snapshot fields in raft_state.bin; BARADB_RAFT_LOG_MAX_ENTRIES
- Fix commit-index scan to use findLogEntryByIndex (works after compact)
2026-07-30 21:35:41 +03:00
dimgigov
9df8316305
feat(raft): transparent leader write/DDL forwarding
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- BARADB_RAFT_CLIENT_PEERS maps node id → SQL client host:port
- Followers proxy DML/DDL to the known leader over the wire protocol
(falls back to "not leader" when the map is missing)
- E2E: follower CREATE/INSERT succeed via forward; docs updated
2026-07-30 21:31:05 +03:00
dimgigov
095698ba82
feat(raft): replicate schema DDL through the raft log
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- isRaftDdl + leader-only gate for CREATE/DROP/ALTER (not DATABASE)
- appendDdlToRaft ships original SQL; applyCommand re-executes via
applyReplicatedDdl (idempotent on leader double-apply)
- Mixed DDL+DML batches use the DDL path so order is preserved
- Fix secondary-index point lookup to use entry.lsmKey (not filter col)
- E2E: CREATE only on leader, schema + index SELECT on follower
2026-07-30 21:25:58 +03:00
dimgigov
50f827f8cf
fix(raft): graph apply + reject non-default DB writes
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- applyReplicatedPut/Delete keep in-memory graphs in sync with node/edge
table rows (idempotent edges via addEdgeWithIdIfAbsent).
- When Raft is enabled, DML is refused on any database other than
'default' (the only DB the state machine is wired to).
- Docs updated (en/bg); unit test for graph apply.
2026-07-30 21:19:01 +03:00
dimgigov
0d51497f57
fix(raft): multi-stmt write gate, rich apply, delete kv convention
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Reject follower writes if any statement in the batch is DML/COMMIT
(not only stmts[0]).
- COMMIT always emits empty-valued kvPairs for isDelete entries.
- applyCommand updates LSM plus secondary B-tree/FTS/HNSW indexes
(applyReplicatedPut/Delete) so follower index scans see replicated rows.
- Tests: not-leader append, commit timeout, index apply unit, E2E
index-backed SELECT on follower.
2026-07-30 21:14:46 +03:00
dimgigov
44060701b7
test(raft): E2E replicated writes; wire SQL path to raft node
...
- Fix runTcpServer to run the already-wired Server (raftNode was assigned
on a different instance that never accepted clients).
- Cap raft peer connect at 200ms and fan out heartbeats in parallel so a
dead peer cannot stall AppendEntries to the live majority.
- Add raft_writes_e2e_test: 3-node write replication, follower rejection,
and post-failover writes; wire into nimble test + gitignore.
2026-07-30 21:06:49 +03:00
dimgigov
333941ab65
feat(raft): leader appends writes to raft log and waits for commit
2026-07-30 19:36:16 +03:00
dimgigov
38c1c01841
feat(raft): classify writes, reject them on follower nodes
2026-07-30 19:07:58 +03:00
dimgigov
b6eccf284e
test(raft): end-to-end 3-node cluster election and failover
2026-07-30 18:35:18 +03:00
dimgigov
f01586354a
fix(raft): partial-read-safe frame reassembly
2026-07-30 18:01:53 +03:00
dimgigov
3f8537eaad
feat(raft): run election timer in production, reset on AppendEntries
2026-07-30 17:47:05 +03:00
dimgigov
853ec7dd3b
feat(raft): parse id@host:port peers, enable raft state persistence
2026-07-30 17:34:39 +03:00
dimgigov
e8f9cbc5bb
fix: CREATE UNIQUE INDEX actually enforces uniqueness (and persists)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-07-30 17:13:29 +03:00
dimgigov
8d2d97ad94
feat(persist): standalone B-tree indexes survive restart
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-07-30 16:53:25 +03:00
dimgigov
821b668d87
fix(persist): unnamed FTS/HNSW indexes survive restart (nameless replay DDL)
2026-07-30 16:27:36 +03:00
dimgigov
214b9cf346
fix: DROP INDEX/TABLE clean up FTS/HNSW indexes and their schema keys
2026-07-30 16:20:23 +03:00
dimgigov
ce6e7aa707
feat(persist): graphs survive restart (rebuild from backing tables)
2026-07-30 16:11:52 +03:00
dimgigov
214e44abd7
feat(persist): HNSW vector indexes survive restart
2026-07-30 16:05:25 +03:00
dimgigov
6703ca2b29
feat(persist): FTS indexes survive restart (schema key + restore replay)
2026-07-30 15:56:48 +03:00
dimgigov
ba7cf195c6
chore: stop tracking compiled test binaries (bugfix_test, nimforum_smoke_test)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
2026-07-30 15:36:41 +03:00
dimgigov
2d09edd9f7
fix+refactor: soft keywords as identifiers, full test wiring, ORC crash docs
...
- parser: clause keywords (header, format, status, user, csv, ...) now work
as identifiers everywhere; IMPORT/EXPORT accept FORMAT csv/HEADER true
- nimble test + CI run all 13 test suites (650 checks green)
- ExecutionContext.registry is now {.cursor.} (breaks registry<->ctx cycle)
- ORC crash reproduced and bisected (tests/orc_repro.py); ARC stays the MM
2026-07-30 13:11:28 +03:00
dimgigov
ed5a71913c
fix: harden exception handling, break ARC cycles, sync license/client
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Replace bare except: with CatchableError across storage, query, Raft,
backup, and protocol code so Defects are not swallowed. Break uncollectable
ARC cycles in server shard/gossip callbacks via Server-owned refs and
cursor locals. Align package license with LICENSE (BSD-3-Clause), sync
README version, and point test_all at the canonical clients/nim baradb
client (parseConnectionString + aliases).
2026-07-23 01:07:51 +03:00
dimgigov
8db5cfe7e1
feat: harden storage, schema persistence, fair benches, fix wire crash
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Core storage: hash MemTable, WAL group commit, L0 compaction rebuild,
reader-writer lock, and a global StorageGate so HTTP workers and TCP
share the LSM safely under multi-thread access.
Schema: durable CREATE/ALTER/DROP under _schema:tables:* with full LSM
restore on open. Executor types/values/schema split into query/exec/.
Wire protocol: switch default MM to ARC — ORC cycle collector segfaulted
after ~20 async INSERTs. Fair multi-tier benchmarks (SQLite/HTTP/wire/PG)
and honesty docs for mixed-tier comparisons.
2026-07-18 16:55:50 +03:00
dimgigov
42043f3946
v1.1.7: deep security & reliability audit — 33 bugs fixed
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Critical (5):
- Reject empty JWT secret when authEnabled (server.nim)
- Fix 2PC marking uncontacted participants as prepared/committed (disttxn.nim)
- Fix Raft commit index calculation for even-sized clusters (raft.nim)
- Fix REP/DISTTXN protocol auth bypass (server.nim)
- Fix HTTP backup/restore path traversal (httpserver.nim)
High (11):
- Fix WAL write race with flush (lsm.nim)
- Fix MVCC savepoint/rollback deep-copy writeSet (mvcc.nim)
- Fix table mutation during deadlock iteration (mvcc.nim)
- Fix LIMIT 0 returning all rows (executor.nim)
- Fix COUNT(col) counting NULL values — 3 locations (executor.nim)
- Fix EXISTS subquery lowering missing subqueryPlan (executor.nim)
- Fix Raft appendEntries/applyCommitted array vs logical index (raft.nim)
- Fix timing attacks on constantTimeCompare and SCRAM (auth.nim, scram.nim)
- Fix B-tree leaf merge phantom separator key (btree.nim)
- Fix SSL verifyPeer not applied to newContext (ssl.nim)
- Fix sharding connectWithTimeout missing SO_ERROR check (sharding.nim)
- Fix sync replication returning success on partial ack (replication.nim)
- Fix WebSocket JWT expiration not validated (websocket.nim)
Medium (13):
- Fix writeSSTable partial file → tmp + atomic rename (lsm.nim)
- Fix multi-CTE table loss (executor.nim)
- Fix nl_to_sql DML restricted to superuser (executor.nim)
- Fix unbounded plan cache — max 10000 (adaptive.nim)
- Fix migration lock crash persistence — timestamp + stale detection (executor.nim)
- Fix admin panel auth (httpserver.nim)
- Fix MVCC unbounded txn tracking — prune in compactVersions (mvcc.nim)
- Fix connection pool maxLifetime check (pool.nim)
- Fix JWT JSON parser backslash escapes (auth.nim)
- Fix substr(s, start) returning single char (udf.nim)
- Fix loadSSTable minimum file-size check (lsm.nim)
- Fix compaction mmap leak (compaction.nim)
- Fix JSON injection in hybrid_search_filtered (executor.nim)
Low (4):
- Raft loadState logs error instead of silent discard
- Replication healthCheck double-close fixed
- Lexer readIdent double column counting fixed
- WebSocket frame 32-bit overflow guard
All 448 tests passing, 0 failures. Bump version to 1.1.7.
2026-05-29 14:17:41 +03:00
dimgigov
a37a62c69e
clients/nim-allographer/src/allographer/query_builder/models/baradb/baradb_exec.nim src/barabadb/core/mvcc.nim src/barabadb/query/codegen.nim src/barabadb/query/lexer.nim src/barabadb/query/parser.nim src/barabadb/query/udf.nim tests/test_all.nim
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-25 18:56:02 +03:00
dimgigov
a9ea80a1f8
fix(prop_test): "Nil expr evaluates to NULL" test — compare Value.kind, not Value == ""
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
The test was comparing a Value to a string literal which always failed.
Changed to check s.kind == vkNull on both calls.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-05-23 22:06:37 +03:00
dimgigov
7d19e06805
fix(btree): critical seq.del vs seq.delete bug and B+ tree leaf traversal
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Root cause: seq.del(i) does swap-with-last (destroys sorted order)
while seq.delete(i) preserves order. This caused massive data loss
during B-tree removes.
Changes:
- btree.nim: Replace seq.del with seq.delete (critical data corruption fix)
- btree.nim: search() traverses leaf linked list for duplicate boundary keys
- btree.nim: splitChild() consolidates duplicate boundary key values before split
- btree.nim: len() counts unique keys via leaf traversal
- btree.nim: remove() traverses leaf linked list to remove ALL occurrences
- btree.nim: Fix separator update in internal node (node.keys[i-1] not [i])
- prop_test.nim: Use set equality for interleaved test (rebalancing reorders)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-05-23 22:01:20 +03:00
dimgigov
072e7a5d13
test(join): add test for duplicate columns in SELECT *
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Verify that JOIN with overlapping column names preserves
both left (bare) and right (qualified) values.
2026-05-21 09:44:32 +03:00
dimgigov
372e5cf627
fix(query,types): refactor Row to Value-typed map, fix IN list, nkPath, multi-table joins
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Refactored Row from Table[string, string] to Table[string, Value]
- Added Value operators: ==, !=, $, in for string interop
- Fixed valueToString for vkNull to return '\\N' instead of ''
- Fixed evalExpr irekField to return vkNull when field not found
- Added IN (val1, val2, ...) parser support
- Fixed nkPath column names in multi-table joins
- Fixed LATERAL JOIN null padding when no matching rows
- Added CREATE/DROP/USE/SHOW DATABASE parser support
- Adapted all tests for new Value type
2026-05-20 23:22:14 +03:00
dimgigov
57d2908066
fix(executor,mvcc,tests): dedup aliases, fix txn delete, fix MIN/MAX null, add regression tests
...
- Add isDelete flag to VersionedRecord to fix COMMIT treating empty values
as DELETE. Previously version.xmax != TxnId(0) was never true for active
transactions, so deletes inside transactions never removed rows.
- Fix MIN/MAX aggregates skipping NULL: was checking for "\\N" instead
of correct "\N" sentinel, so NULL values were never skipped.
- Add duplicate alias deduplication in lowerSelect() and getSelectColumns().
Identical aliases now get _1, _2 suffixes (e.g. count(*), count(*)_1).
- Add nil guard for expr.subqueryPlan in irekSubquery evalExpr handler.
- Initialize txnManager in newExecutionContext to prevent segfault on BEGIN.
- Add regression tests: DELETE in transaction, PK-only row survival,
MIN/MAX NULL skipping.
2026-05-20 21:36:15 +03:00
dimgigov
edd7e728f9
fix(query): add correlated subquery support
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
evalExpr was missing irekSubquery case — scalar subqueries in SELECT/WHERE
fell through to else:return "" producing empty strings. Add:
- irekSubquery handler in evalExpr that executes the subquery plan
- outerRow + subqueryPlan fields on ExecutionContext for correlation
- qualified column injection in execScan for correlated filter refs
- collectCorrelatedTablesFromPlan to discover outer table references
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-05-19 23:16:31 +03:00
dimgigov
14596c9584
fix(adapter,executor,tests): harden nimforum adapter and fix transaction/aggregate bugs
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Adapter (baradb_sqlite.nim):
- getRow now returns @[] for missing rows (db_sqlite compat)
- insertID/nextId handle NULL (\N) values correctly
- preprocessQuery replaces SQLite-specific syntax:
TIMESTAMP->DATETIME, DATETIME('now')->current timestamp, INET->STRING
- add dbBegin/dbCommit/dbRollback for transaction support
Executor:
- fix COMMIT treating empty values as DELETE (rows with only PK were lost)
- fix MIN/MAX aggregates including NULL (\N) which skewed results
- add numeric comparison for irNeq
Tests:
- stress_test: shared DB, 16 workers, correctness scan after run
- nimforum_smoke_test: 10 tests including schema, transactions, iterators
2026-05-19 20:43:00 +03:00
dimgigov
cd46edcb67
Bump version to 1.1.6; fix storage bugs, tests, CI pipeline and Docker config
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Storage fixes:
- Fix bloom filter hash2 OverflowDefect (uint64 instead of int Hash)
- Fix MemTable.put size calculation on overwrite (was leaking size)
- Fix fuzz_test duplicate-key issues in LSM delete and SSTable tests
Test fixes:
- Fix stress_test.nim: replace deprecated threadpool with std/threads
- Fix fuzz_test.nim: add missing imports (tables, algorithm, sets)
- Fix fuzz_test.nim: var sst for close() compatibility
- Remove unused imports from test_all.nim and bench_all.nim
Docker / CI fixes:
- Fix Dockerfile.source: invalid nimlang/nim:2.2.10-alpine tag → 2.2.10 + ubuntu:24.04 runtime
- Fix Dockerfile.source healthcheck (--spider → -qO- for 200 OK)
- Fix Dockerfile run comment ports (9470/9471 → 9912/9913)
- Fix scripts/docker-run.sh healthcheck port (9470 → 9912)
- Add ARG BUILD_DATE/VCS_REF to Dockerfile for docker-build.sh
- Update all version strings 1.1.4 → 1.1.6 across nimble/docker/source/docs
2026-05-19 12:13:33 +03:00
dimgigov
7e6a45e6b7
Стабилизация на storage слоя — 6 фази
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Фаза 1: SSTable Integrity
- SSTable v3 формат с CRC32 footer (data/index/bloom CRC)
- Нов модул storage/crc32.nim (zero-dep, IEEE 802.3)
- verifySSTable() — проверка на magic, version, CRC
- loadSSTable strict mode — отхвърля корумпирани файлове
- newLSMTree логва [WARN] при corrupt SSTables
Фаза 2: baradb repair
- Нов модул tools/repair.nim
- Сканира SSTables, проверява CRC, премества битите в corrupt/
- WAL replay (CrashRecovery) за възстановяване на данни
- CLI: ./baradadb repair --data-dir=... [--dry-run]
Фаза 3: MANIFEST File
- Atomic write MANIFEST (tmp + rename)
- newLSMTree зарежда от MANIFEST, fallback към walkDir
- checkStorageConsistency() — orphan/missing detection
- flushUnsafe и compaction записват MANIFEST
Фаза 4: WAL Rotation & Incremental Backup
- WAL segment rotation при 64MB (wal_archive/wal.000NNN.log)
- maybeRotate() на всеки 1000 записа + при flush
- backup incremental — архивира MANIFEST + SSTables + WAL
- CRC verification на SSTables преди архивиране
Фаза 5: Online Consistent Backup
- checkpoint() — freeze memtable + flush + WAL rotate
- ./baradadb checkpoint — offline consistent snapshot
- backup --online — checkpoint + incremental backup
Фаза 6: SSTable Version Migration
- SSTable.fileVersion поле
- listLegacySSTables() — намира v1/v2 файлове
- migrateSSTable() — пренаписва към v3 (tmp + rename)
- ./baradadb migrate [--dry-run] — offline migration
Документация:
- Обновени docs/en/backup.md, docs/bg/backup.md
- Обновени docs/en/storage.md, docs/bg/storage.md
- Добавени тестове в tests/test_all.nim
2026-05-18 15:04:49 +03:00
dimgigov
cf2aba104f
Phase 5 complete: B-tree property tests + benchmark regression suite
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- prop_test.nim: 11 new property-based B-Tree invariants
(size accuracy, get roundtrip, scan ordering, range correctness,
contains, remove, large order, duplicates, empty tree, interleaved ops)
- bench_all.nim: JSON-based benchmark result tracking with regression
comparison against previous runs; each benchmark reports ops/sec
delta vs last run
2026-05-18 12:52:44 +03:00
dimgigov
132e8c7a31
Phase 5: Storage fuzz tests + WAL entryCount fix + SSTable overflow checks + compaction sizeBytes accuracy
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- WAL: entryCount is now accurate after restart by counting existing entries
- LSM writeSSTable: adds uint32 overflow checks for key/value/entry counts
- Compaction: sizeBytes now uses actual file size instead of rough guess
- Fuzz tests: 14 new tests covering WAL truncation, LSM put/get/delete,
recovery, scan, overwrites, SSTable roundtrip, corruption, tombstones
2026-05-18 12:34:26 +03:00
dimgigov
967c0855a5
Bug fixes: composite PK, nl_to_sql sandbox, FK check, SQL injection, storage correctness
...
Critical fixes:
- Composite PK: execInsert + validateConstraints use all PK columns
- nl_to_sql: non-SELECT SQL no longer executed directly during validation
- FK check: removed O(N) scanMemTable fallback, uses db.get() with SSTables
- exprToSql: nkIdent wrapped in quotes to prevent SQL injection
- restoreSchema: try/except around tokenize/parse for crash resilience
- recovery.nim: lastTxnId tracking + putUnsafe/deleteUnsafe without WAL
- SCRAM: verifyClientProof length check + DefaultIterationCount restored
Storage fixes:
- lsm.nim: SSTable sort order fixed (ascending), close() flushes all memtables
- compaction.nim: tombstones preserved during compaction
- wal.nim: header written for empty existing files, readEntries checks magic
- btree.nim: B+ tree leaf split keeps boundary key
- bloom.nim: deserialize raises on short data
- mmap.nim: bounds checks for adviseWillNeed/DontNeed + posix.close()
Protocol fixes:
- zerocopy.nim: readString bounds check
- wire.nim: deserializeValue 32-bit underflow check
- auth.nim: JWT JSON escaping for claims
- server.nim: readUint32BE bounds check + specific exception handling
- raft.nim: readData checks + specific exception handling
Tests:
- Added Composite Primary Key test suite (4 tests)
Build: 0 warnings, 0 errors
2026-05-18 11:33:11 +03:00
dimgigov
67965ffa8b
feat(hybrid): Session 10.1.3 — Metadata pre-filtering in vector search
...
- Populate HNSW metadata with all relational columns during INSERT and CREATE INDEX
- Add doHybridSearchFiltered() using searchWithFilter() for HNSW pre-filtering
- Add SQL function hybrid_search_filtered(table, vec_col, text_col, query, vector, k, filter_col, filter_val)
- Enforce k-limit on both doHybridSearch and doHybridSearchFiltered results
- 2 tests: tenant isolation + empty filter fallback
2026-05-17 13:41:30 +03:00
dimgigov
836d30d84a
feat(hybrid): Session 10.1 — Hybrid RAG Search with RRF reranking
...
- Add searchEx() to vector engine returning metadata
- Add reciprocalRankFusion(), doHybridSearch(), findRealIdByDocId() helpers
- Add SQL functions: hybrid_search(), hybrid_search_ids(), rerank()
- Fix CREATE INDEX HNSW docId to use hash(fullKey) matching INSERT
- 5 tests covering hybrid search, ids, RRF ranking, rerank, missing indexes
2026-05-17 13:30:19 +03:00
dimgigov
2e0969245c
feat(fk): Foreign Key Enforcement — ON DELETE/UPDATE CASCADE, SET NULL, RESTRICT
...
- Add tkRestrict token to lexer
- Parse ON DELETE CASCADE/SET NULL/RESTRICT and ON UPDATE CASCADE/SET NULL/RESTRICT
in both table-level and column-level FK constraints
- Add fkOnDelete/fkOnUpdate to ColumnDef and ForeignKeyDef
- Fix table-level FK constraint application (third pass after columns are created)
- Implement enforceFkOnDelete, enforceFkOnUpdate, enforceFkOnChildUpdate helpers
- Wire FK enforcement into DELETE and UPDATE execution paths
- Add 9 regression tests covering all FK actions
2026-05-17 13:02:39 +03:00
dimgigov
dc4ad86ee1
Сесия 9: Production Hardening — prop tests (58 invariant-а), fuzz tests (35 сценария), thread safety (SharedLock ref)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 12:37:50 +03:00
dimgigov
19fa760604
Week 4: Production Hardening — Thread Safety, Fuzz Tests, Property-Based Tests, NimForum Smoke Test
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Thread Safety:
- Add lock to BTreeIndex (insert/get/scan/remove/contains)
- Add lock to InvertedIndex (addDocument/removeDocument/search/bm25/tfidf/fuzzy/regex)
- Add ctxLock to ExecutionContext for DDL/schema metadata, autoIncCounters, sequences
- Deep-copy sessionVars in cloneForConnection() to isolate per-connection state
- Fix activeConnections check-then-act race with Lock
Wire Protocol Hardening:
- Fix RangeDefect in deserializeValue for invalid FieldKind bytes
- Fix IndexDefect in fkFloat32/fkFloat64 by reusing readUint32/readUint64
- Fix int32/int64 cast from uint32/uint64 to use cast[] instead of constructor
- Fix parseHeader to validate MsgKind instead of holey enum cast
Property-Based Tests (tests/prop_test.nim):
- 8 invariants for evalExprValue: literal types, commutativity, identity,
double negation, NULL propagation
Wire Protocol Fuzz (tests/fuzz_test.nim):
- 7 fuzz tests: random bytes deserialization, truncated buffers,
MsgKind casts, parseHeader logic, message roundtrips
NimForum Adapter Smoke Test (tests/nimforum_smoke_test.nim):
- Full TCP server lifecycle with adapter CRUD and parameterized queries
Test Results: 423 tests pass, 0 failures
2026-05-17 11:57:55 +03:00
dimgigov