Commit Graph

24 Commits

Author SHA1 Message Date
dimgigov c94bac43e5 fix(raft): ignore intermediate InstallSnapshot chunk replies 2026-07-31 03:02:32 +03:00
dimgigov 862d62590e feat(raft): leader InstallSnapshot send on unrecoverable lag 2026-07-31 02:53:30 +03:00
dimgigov efa04e4b36 feat(raft): follower InstallSnapshot receive and restore 2026-07-31 02:33:32 +03:00
dimgigov cb9cd7415d feat(raft): InstallSnapshot wire protocol (backward-compatible) 2026-07-31 02:12:51 +03:00
dimgigov ed89c88afa feat(raft): optional TLS on raft transport (server + dialer) 2026-07-31 01:31:20 +03:00
dimgigov 1b3c26123a feat(raft): expose raft metrics on /metrics and /health
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- RaftMetrics counters: elections, term changes, appends, commit waits/ms,
  timeouts, lost leadership, forwards, applies, compactions
- Gauges via prometheusText: is_leader, term, log size, commit/applied,
  apply lag, snapshot index
- Wire httpServer.raftNode; extend GET /metrics and GET /health
2026-07-30 21:38:44 +03:00
dimgigov 53704e1036 feat(raft): safe log compaction with snapshot metadata
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- lastSnapshotIndex/Term bound the compacted prefix; lastLogIndex/Term
  and AppendEntries prevLog checks respect the snapshot base
- compactLog drops entries only through min(matchIndex, lastApplied) on
  the leader so lagging peers can still catch up via AppendEntries
- Persist snapshot fields in raft_state.bin; BARADB_RAFT_LOG_MAX_ENTRIES
- Fix commit-index scan to use findLogEntryByIndex (works after compact)
2026-07-30 21:35:41 +03:00
dimgigov 44060701b7 test(raft): E2E replicated writes; wire SQL path to raft node
- Fix runTcpServer to run the already-wired Server (raftNode was assigned
  on a different instance that never accepted clients).
- Cap raft peer connect at 200ms and fan out heartbeats in parallel so a
  dead peer cannot stall AppendEntries to the live majority.
- Add raft_writes_e2e_test: 3-node write replication, follower rejection,
  and post-failover writes; wire into nimble test + gitignore.
2026-07-30 21:06:49 +03:00
dimgigov b6eccf284e test(raft): end-to-end 3-node cluster election and failover 2026-07-30 18:35:18 +03:00
dimgigov f01586354a fix(raft): partial-read-safe frame reassembly 2026-07-30 18:01:53 +03:00
dimgigov 3f8537eaad feat(raft): run election timer in production, reset on AppendEntries 2026-07-30 17:47:05 +03:00
dimgigov ed5a71913c fix: harden exception handling, break ARC cycles, sync license/client
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Replace bare except: with CatchableError across storage, query, Raft,
backup, and protocol code so Defects are not swallowed. Break uncollectable
ARC cycles in server shard/gossip callbacks via Server-owned refs and
cursor locals. Align package license with LICENSE (BSD-3-Clause), sync
README version, and point test_all at the canonical clients/nim baradb
client (parseConnectionString + aliases).
2026-07-23 01:07:51 +03:00
dimgigov 42043f3946 v1.1.7: deep security & reliability audit — 33 bugs fixed
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Critical (5):
- Reject empty JWT secret when authEnabled (server.nim)
- Fix 2PC marking uncontacted participants as prepared/committed (disttxn.nim)
- Fix Raft commit index calculation for even-sized clusters (raft.nim)
- Fix REP/DISTTXN protocol auth bypass (server.nim)
- Fix HTTP backup/restore path traversal (httpserver.nim)

High (11):
- Fix WAL write race with flush (lsm.nim)
- Fix MVCC savepoint/rollback deep-copy writeSet (mvcc.nim)
- Fix table mutation during deadlock iteration (mvcc.nim)
- Fix LIMIT 0 returning all rows (executor.nim)
- Fix COUNT(col) counting NULL values — 3 locations (executor.nim)
- Fix EXISTS subquery lowering missing subqueryPlan (executor.nim)
- Fix Raft appendEntries/applyCommitted array vs logical index (raft.nim)
- Fix timing attacks on constantTimeCompare and SCRAM (auth.nim, scram.nim)
- Fix B-tree leaf merge phantom separator key (btree.nim)
- Fix SSL verifyPeer not applied to newContext (ssl.nim)
- Fix sharding connectWithTimeout missing SO_ERROR check (sharding.nim)
- Fix sync replication returning success on partial ack (replication.nim)
- Fix WebSocket JWT expiration not validated (websocket.nim)

Medium (13):
- Fix writeSSTable partial file → tmp + atomic rename (lsm.nim)
- Fix multi-CTE table loss (executor.nim)
- Fix nl_to_sql DML restricted to superuser (executor.nim)
- Fix unbounded plan cache — max 10000 (adaptive.nim)
- Fix migration lock crash persistence — timestamp + stale detection (executor.nim)
- Fix admin panel auth (httpserver.nim)
- Fix MVCC unbounded txn tracking — prune in compactVersions (mvcc.nim)
- Fix connection pool maxLifetime check (pool.nim)
- Fix JWT JSON parser backslash escapes (auth.nim)
- Fix substr(s, start) returning single char (udf.nim)
- Fix loadSSTable minimum file-size check (lsm.nim)
- Fix compaction mmap leak (compaction.nim)
- Fix JSON injection in hybrid_search_filtered (executor.nim)

Low (4):
- Raft loadState logs error instead of silent discard
- Replication healthCheck double-close fixed
- Lexer readIdent double column counting fixed
- WebSocket frame 32-bit overflow guard

All 448 tests passing, 0 failures. Bump version to 1.1.7.
2026-05-29 14:17:41 +03:00
dimgigov 2d310a33a1 fix: resolve all 55 identified bugs across the codebase
Comprehensive bug fix pass across all subsystems:
- Critical: lexer infinite loop, WAL lock discipline, checkpoint safety,
  compaction verification, HMAC key truncation, healthCheck leak
- High: MVCC lock safety, B-Tree delete rebalancing, Raft stale term
  handling, replication socket leaks and ack cleanup, sharding migration
  with old assignments, 2PC recovery, JWT/SCRAM auth fixes, wire protocol
  bounds checks
- Medium: config error handling, MERGE parser completeness, IR/codegen
  correctness, UDF bounds checks, LIMIT cost accuracy, mmap safety
- Low: timeout handling, float parsing edge cases, uint32 truncation,
  cache entry cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-21 10:59:51 +03:00
dimgigov 47c4393a7e Complete plan execution: gossip/raft bounds, deadlock fix, JSON escaping, memtable limits
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Remaining plan items:
- 1.4 hybrid_search: JSON built with std/json instead of string concat
- 2.3 lsm.nim: reject oversized memtable entries
- 3.3 gossip.nim: bounds checking for idLen/nodeCount/hostLen
- 3.4 raft.nim: max length caps for cmdLen/dataLen/logLen
- 4.3 deadlock.nim: per-path seq stack instead of global parent table
2026-05-18 11:42:46 +03:00
dimgigov 967c0855a5 Bug fixes: composite PK, nl_to_sql sandbox, FK check, SQL injection, storage correctness
Critical fixes:
- Composite PK: execInsert + validateConstraints use all PK columns
- nl_to_sql: non-SELECT SQL no longer executed directly during validation
- FK check: removed O(N) scanMemTable fallback, uses db.get() with SSTables
- exprToSql: nkIdent wrapped in quotes to prevent SQL injection
- restoreSchema: try/except around tokenize/parse for crash resilience
- recovery.nim: lastTxnId tracking + putUnsafe/deleteUnsafe without WAL
- SCRAM: verifyClientProof length check + DefaultIterationCount restored

Storage fixes:
- lsm.nim: SSTable sort order fixed (ascending), close() flushes all memtables
- compaction.nim: tombstones preserved during compaction
- wal.nim: header written for empty existing files, readEntries checks magic
- btree.nim: B+ tree leaf split keeps boundary key
- bloom.nim: deserialize raises on short data
- mmap.nim: bounds checks for adviseWillNeed/DontNeed + posix.close()

Protocol fixes:
- zerocopy.nim: readString bounds check
- wire.nim: deserializeValue 32-bit underflow check
- auth.nim: JWT JSON escaping for claims
- server.nim: readUint32BE bounds check + specific exception handling
- raft.nim: readData checks + specific exception handling

Tests:
- Added Composite Primary Key test suite (4 tests)

Build: 0 warnings, 0 errors
2026-05-18 11:33:11 +03:00
dimgigov c0bd3dff86 feat: distributed gaps filled — gossip UDP transport, sharding data migration, inter-module wiring (raft-disttxn, gossip-sharding, replication-disttxn) 2026-05-13 11:11:53 +03:00
dimgigov a72e17f063 fix: последните 4 критични/средни бъга — LSM-Tree, MVCC, DistTxn, Raft
Поправени проблеми:
- LSM-Tree: WAL write вече е извън db.lock (отделен walLock) → по-добра concurrency
- MVCC: добавен compactVersions() който се изпълнява на всеки 100 commits
  Премахва стари overwritten версии, които не са видими за active транзакции
- DistTxn: commit() вече НЕ rollback-ва commit-нали participants
  Ако някой participant е commit-нал, транзакцията се маркира като committed
- Raft: добавен disk persistence за currentTerm, votedFor, log
  saveState() при всяка промяна; loadState() при стартиране

292 теста, 0 failure-а.
2026-05-12 23:30:58 +03:00
dimgigov 131541a0e5 fix: критични бъгове в storage, MVCC, auth, Raft, query executor
Поправени проблеми:
- MVCC: aborted транзакции вече не стават видими (добавено abortedTxns множество)
- LSM-Tree: поправена загуба на данни при immutable memtable overwrite
- LSM-Tree: поправена SSTable search order (сортиране по id вместо minKey)
- WAL: sync()/close() вече извикват posix.fsync()
- Auth: simpleHash (djb2) заменен с HMAC-SHA256
- Recovery: summary() вече не мутира базата данни
- Raft: поправена majority calculation за четен брой нодове
- Query: EXISTS subqueries вече изпълняват подзаявката
- Nimble: поправен build (-d:ssl) и bench task

Добавен BUG_AUDIT.md с пълен доклад и план за подобрения.

292 теста, 0 failure-а.
2026-05-12 22:45:50 +03:00
dimgigov c2dc280ddd fix(raft): state machine callback, start from main, remove asyncCheck, config
- applyCommand callback: committed entries applied to state machine
- lastApplied incremented on commit via applyCommitted()
- RaftNetwork.run() started from main() when raftEnabled=true
- asyncCheck replaced with try/await in processMessage
- bindAddr without hardcoded 127.0.0.1
- raft config: raftEnabled/Port/Peers/NodeId + env vars
- 283 tests, 0 failures
2026-05-07 14:22:04 +03:00
dimgigov 096c8347cf feat: complete Phase 3 + new production roadmap for Web/ERP
- Thread-safety: locks in LSMTree and Graph engines
- Raft network transport: async TCP, serialization, heartbeat, 3-node election test
- CI/CD: GitHub Actions workflow
- Cleanup: remove dead code, unused imports, build artifacts
- New PLAN.md targeting production Web/ERP readiness
- 216 tests passing
2026-05-06 10:40:34 +03:00
dimgigov 805dbc8ad6 fix: comprehensive bug audit — 15 critical/high/medium fixes
Critical fixes:
- B-Tree splitChild: fixed out-of-bounds on child.children.len (mid+1..len → mid+1..<len)
- Wire protocol: all deserializeValue cases now properly assign to result
- Wire protocol: fkBool/fkInt8 now increment pos after reading
- MVCC: write-write conflict now detected for completed transactions
- MVCC: commit only marks visible versions as deleted (not concurrent uncommitted)
- Lexer: buffer overrun on backslash escape at end of input — added bounds check
- Bulgarian/Russian stemmers: fixed UTF-8 byte offsets for Cyrillic suffixes (2 bytes per char)

High fixes:
- WebSocket: added missing std/tables import
- Raft: added randomize() call for election timeouts
- MemTable.get: binary search O(log n) instead of linear scan O(n)

Medium fixes:
- Zerocopy: string length now written as 4-byte int32 (was single byte)
- Raft appendEntries: guard against nextIdx underflow
- Columnar: null tracking with nulls seq on ColumnPtr
- Removed dead code in lexer readIdent (redundant true/false checks)
- Updated Bulgarian stemming test to match corrected byte offsets

All 214 tests pass across 48 suites with 0 failures.
2026-05-06 02:46:37 +03:00
dimgigov b9f6059cbf feat: crash recovery, Raft election timer, CLI autocomplete, roadmap update — 246 tests
Crash Recovery:
- WAL file scanning with magic/version/entry parsing
- REDO/UNDO analysis — identify committed vs uncommitted txns
- Summary reporting with entry/txn/apply status

Raft Election Timer:
- ElectionTimer with configurable timeout and tick/check/reset
- Automatic election start on follower timeout
- Candidate timeout restart on failed elections

CLI Autocomplete:
- Full autocomplete engine for commands and SQL keywords
- Suggest function for single/multi completions
- 60+ keywords (SELECT, FROM, WHERE, JOIN, GROUP BY, etc.)

ROADMAP.md:
- All completed items properly marked
- Updated status table — 9 out of 12 phases at 90%+

14 new tests (246 total, all passing)
2026-05-06 02:12:49 +03:00
dimgigov 3162271328 feat: full BaraQL (GROUP BY/JOIN/CTE/aggregates), Raft consensus, sharding — 107 tests
BaraQL parser extended:
- GROUP BY + HAVING clauses
- JOIN (INNER, LEFT, RIGHT, FULL, CROSS) with ON conditions
- CTE (WITH ... AS) with multiple CTEs
- Aggregate functions (count, sum, avg, min, max)
- CASE/WHEN/THEN/ELSE/END expressions
- BETWEEN, IN, LIKE/ILIKE, IS NULL/IS NOT NULL
- Dotted path identifiers (table.column)
- Subqueries in FROM clause
- CREATE TYPE with colon syntax (name: type)
- UPDATE SET ... WHERE, DELETE FROM ... WHERE

Infrastructure:
- Raft consensus: leader election, log replication, RequestVote/AppendEntries
- Sharding: hash-based, range-based, consistent hashing, rebalancing
- Fixed lexer: = is equality, := is assignment, added : (colon) token
- 34 new parser tests (107 total, all passing)
2026-05-06 01:15:33 +03:00