Commit Graph

150 Commits

Author SHA1 Message Date
dimgigov 18f3c16b2a Fix/nodebara compatibility (#2) 2026-05-14 02:33:46 +03:00
dimgigov 71dcffecce fix(gossip): use async UDP socket to avoid blocking the event loop
The gossip listener used synchronous newSocket + recvFrom,
which blocks the async event loop and prevents other async
operations from running while waiting for UDP packets.

Switch to newAsyncSocket + async recvFrom so the loop can
continue processing other tasks between gossip rounds.
2026-05-14 02:22:07 +03:00
dimgigov 398769ff97 feat(client): add TCP request queue for safe concurrency
The previous implementation allowed multiple concurrent async calls
(query, execute, ping) to interleave writes on the same socket,
which corrupted the binary protocol framing when NodeBB fired
parallel database operations.

Add an internal _requestQueue and _requestLock so that all TCP
requests are serialized: each async operation enqueues a task,
and tasks are drained one at a time via setImmediate().
2026-05-14 02:22:07 +03:00
dimgigov 8fb5dde858 fix(protocol): serialize float32/float64/fkVector in big-endian
The JavaScript client reads floats via readFloatBE/readDoubleBE,
which expect IEEE-754 values in big-endian byte order.
The Nim server was writing them with copyMem in native byte order,
so on little-endian machines (x86_64) the JS side deserialized
FLOAT64 values as garbage (e.g. 1.0 became 3.03865e-319).

Fix serialization by casting to int32/int64 and using bigEndian32/
bigEndian64, mirroring the existing big-endian handling for ints.
Apply the same fix to deserialization and to fkVector elements.
2026-05-14 02:22:07 +03:00
dimgigov a0d2ca7776 Merge release/v1.1.0 into main 2026-05-13 15:07:44 +03:00
dimgigov 4e7e568525 release: v1.1.0 — bug fixes, Docker, clients, SCRAM auth
- Bump version to 1.1.0 across all components
- Remove debug CI artifacts (debug.yml, test_all.nim.bak, test_lock)
- Update CHANGELOG with v1.1.0 release notes
- Bug fixes: irNeg, distributed txn, sharding, Raft majority, MVCC,
  LSM-Tree, auth (JWT + SCRAM-SHA-256), wire protocol, SQL injection,
  ReDoS, graph, vector, FTS, build warnings
- Client SDKs: JS/TS, Python, Nim, Rust with tests and examples
- Docker: production + source + test compose configs
- TLA+: crossmodal, backup, recovery specs with symmetry reduction
2026-05-13 15:05:59 +03:00
dimgigov 6665ee7e0a fix: irNeg (unary minus) evaluation in query executor
- Added missing irNeg case in evalExpr (query/executor.nim)
- Unary minus previously fell through to else: return 'false'
- Added 2 tests: SELECT expression and WHERE condition
2026-05-13 14:23:28 +03:00
dimgigov 42c675224c fix: distributed transaction and sharding bugs
- disttxn: rollback now correctly updates participant state (was modifying a copy)
- disttxn: bare except replaced with CatchableError
- sharding: migrateData condition was always false, now correctly migrates data
- sharding: unsafe cast[string] replaced with manual byte-to-char conversion
- sharding: bare except replaced with CatchableError
- server: parseUInt replaced with parseBiggestUint for uint64 portability
2026-05-13 14:01:13 +03:00
dimgigov 45849cbe5c fix(tests): use less common ports 29001-29003 in Raft network transport test 2026-05-13 13:53:07 +03:00
dimgigov ac84474789 debug(ci): keep only Enhanced Migrations onwards in test_all.nim 2026-05-13 13:50:03 +03:00
dimgigov 92af288d06 debug(ci): keep only Row-Level Security onwards in test_all.nim 2026-05-13 13:46:54 +03:00
dimgigov baf2b1a504 debug(ci): keep only TLS/SSL onwards in test_all.nim 2026-05-13 13:43:46 +03:00
dimgigov 8d600a6f23 debug(ci): keep only last quarter of test_all.nim 2026-05-13 13:40:38 +03:00
dimgigov ca87acb120 debug(ci): comment out first half of test_all.nim 2026-05-13 13:36:58 +03:00
dimgigov d84033984a debug(ci): keep only second half of test_all.nim 2026-05-13 13:34:05 +03:00
dimgigov 1923aa225b debug(ci): truncate test_all.nim to first half of suites 2026-05-13 13:31:15 +03:00
dimgigov 9777b73e7c fix(tests): add missing import std/tables in join_tests.nim 2026-05-13 13:27:34 +03:00
dimgigov ab2522657f debug(ci): add lock/lsm test to isolate failure 2026-05-13 13:22:06 +03:00
dimgigov adf5937198 debug(ci): add timeout to join_tests 2026-05-13 13:10:39 +03:00
dimgigov 113d7faafa debug(ci): run join_tests and tla_faithfulness separately 2026-05-13 13:07:32 +03:00
dimgigov 15a905bdd1 debug(ci): emit notice with 0x0.st URL for test_all.log 2026-05-13 13:02:14 +03:00
dimgigov 14791d8be6 debug(ci): remove || true mask from test_all run 2026-05-13 12:59:26 +03:00
dimgigov dbe92fab9d debug(ci): always upload test_all.log to branch 2026-05-13 12:55:48 +03:00
dimgigov d9835fe696 debug(ci): remove || echo mask from test_all binary run 2026-05-13 12:52:25 +03:00
dimgigov 1111fbf359 debug(ci): upload logs to new branch on failure 2026-05-13 12:48:05 +03:00
dimgigov 1ed9f90852 debug(ci): add nim version check step before minimal test 2026-05-13 12:40:31 +03:00
dimgigov dea4870d87 debug(ci): try iffy/install-nim instead of jiro4989/setup-nim-action 2026-05-13 12:38:17 +03:00
dimgigov 2926701f8c fix(ci): explicitly install openssl and ca-certificates in all workflows 2026-05-13 12:35:43 +03:00
dimgigov fb2bd1769d debug(ci): remove -d:ssl from minimal test to isolate issue 2026-05-13 12:33:15 +03:00
dimgigov cde443679e debug(ci): simplify debug workflow to isolate failure 2026-05-13 12:29:39 +03:00
dimgigov e0037ae817 debug(ci): add debug workflow to isolate test failure 2026-05-13 12:23:50 +03:00
dimgigov aaa4f37c77 debug(ci): post test log as commit comment on failure 2026-05-13 12:16:56 +03:00
dimgigov 25ba9911b7 debug(ci): emit notice annotation with 0x0.st URL 2026-05-13 12:11:52 +03:00
dimgigov 8e56ff721c debug(ci): upload test log to 0x0.st on failure 2026-05-13 12:11:28 +03:00
dimgigov 942910fae5 debug(ci): print nim version, nimble packages, system info 2026-05-13 12:05:49 +03:00
dimgigov f50cbd1555 debug(ci): upload test_output.log as artifact on failure 2026-05-13 12:04:59 +03:00
dimgigov 6978097768 debug(ci): capture test output to log for debugging GitHub Actions failure 2026-05-13 12:02:01 +03:00
dimgigov f6781c107e fix(ci): add --threads:on to all nim c commands
Commit 6aaabb5 added std/locks usage in vector/engine.nim (HNSW thread-safety).
std/locks requires --threads:on. Local builds worked because /etc/nim/nim.cfg
had threads:on, but GitHub Actions runners (setup-nim-action) compile without
threads by default, causing the test job to fail silently at runtime.

Also updated baradadb.nimble tasks for consistency.

Fixes GitHub Actions CI failure on test / clients-ci workflows.
2026-05-13 11:57:09 +03:00
dimgigov c0bd3dff86 feat: distributed gaps filled — gossip UDP transport, sharding data migration, inter-module wiring (raft-disttxn, gossip-sharding, replication-disttxn) 2026-05-13 11:11:53 +03:00
dimgigov 2a8d2a6cb4 docs: update PLAN.md and BUG_AUDIT.md for v1.0.0 release
- Final metrics: 294 tests, 10 TLA+ specs, 32 bugs fixed
- Mark all critical/high/medium/config bugs as resolved
- Document SCRAM-SHA-256, crossmodal.tla, symmetry reduction
- List remaining post-v1.0.0 non-critical tasks
2026-05-13 09:54:00 +03:00
dimgigov ac55237b72 feat: real SCRAM-SHA-256 authentication (Option C)
Security:
- New scram.nim module with full SCRAM-SHA-256 per RFC 7677
  * PBKDF2-HMAC-SHA-256 key derivation
  * HMAC-SHA-256 with multiple overloads for bytes/strings
  * Secure nonce/salt generation via /dev/urandom
  * Client/server message parsing and proof verification

AuthManager updates:
- registerScramUser(): stores salted credentials (salt + iterations + storedKey + serverKey)
- startScram(): initiates challenge-response handshake
- finishScram(): verifies client proof and returns server signature
- Legacy amSCRAMSHA256 path kept for backward compatibility

HTTP endpoints:
- POST /auth/scram/start — accepts client-first-message, returns server-first
- POST /auth/scram/finish — accepts client-final-message, returns server-final

Tests:
- SCRAM-SHA-256 full handshake test (register → start → compute proof → finish)
- SCRAM-SHA-256 invalid proof rejection test

Build: 0 warnings, all tests pass
2026-05-13 09:50:07 +03:00
dimgigov 422df08ab9 feat: clean build + crossmodal.tla + TLA+ symmetry (v1.0.0-prep)
Build warnings cleanup (0 warnings):
- Suppress threadpool deprecation warning (baradadb.nim)
- Remove unused 'os' import (logging.nim)
- Fix ImplicitDefaultValue for newNode params (ast.nim)
- Add explicit cstring cast for posix.open (wal.nim)
- Fix HoleEnumConv for MsgKind parsing (server.nim)

TLA+ Formal Verification:
- Add symmetry reduction (Permutations) to all 9 existing specs
- Add SYMMETRY directive to all .cfg files
- New crossmodal.tla: cross-modal consistency spec
  * MetadataVectorConsistency, HybridResultValid
  * CommittedAtomicity, AbortedAtomicity, TxnStateValid
- New models/crossmodal.cfg

Tests:
- Add Cross-Modal TLA+ Faithfulness tests (4 tests)

Docs:
- Update PLAN.md and BUG_AUDIT.md with completed tasks
2026-05-13 09:24:04 +03:00
dimgigov d08de7a062 Update BUG_AUDIT.md and PLAN.md: mark all bugs fixed, add session 6-7, plan for tomorrow
- BUG_AUDIT.md: 34 bugs documented (32 fixed + 2 auth hardening), all metrics green
- PLAN.md: backup.tla and recovery.tla marked complete, 3 options for tomorrow
  - Option A: Clean build (warnings + symmetry reduction)
  - Option B: crossmodal.tla
  - Option C: True SCRAM-SHA-256
- Cleared 26GB of TLC state files from formal-verification/states/
2026-05-13 00:17:06 +03:00
dimgigov c0596d6ba1 Update PLAN.md: mark backup.tla and recovery.tla as completed 2026-05-13 00:12:15 +03:00
dimgigov 32b38675a9 Add TLA+ formal verification specs for backup and recovery
- backup.tla: models backup/restore/verify/cleanup with properties:
  BackupSnapshotsValid, RestoreIntegrity, VerifyIntegrity,
  RetentionInvariant, HistoryConsistency, BackupIdValid
- recovery.tla: models WAL replay REDO/UNDO with properties:
  RedoCommitted, RecoveryCompleteness, WalIntegrity
- Added models/backup.cfg and models/recovery.cfg with bounded constants
- Updated run_all.sh to include both new specs
- All 9 TLA+ specs pass TLC model checking
2026-05-13 00:06:45 +03:00
dimgigov 5e27a987dd fix: auth token expiration + constant-time JWT signature comparison
- createToken() добавя exp/iat/nbf в JWT payload
- verifyToken() parse-ва exp/iat/nbf и проверява expiration
- verifyToken() използва constantTimeCompare() за timing-attack resistance
- Добавен import std/times

292 теста, 0 failure-а.
2026-05-12 23:36:17 +03:00
dimgigov a6aeb7801b docs: обновен BUG_AUDIT.md — всички критични/високи/средни бъгове поправени 2026-05-12 23:31:51 +03:00
dimgigov a72e17f063 fix: последните 4 критични/средни бъга — LSM-Tree, MVCC, DistTxn, Raft
Поправени проблеми:
- LSM-Tree: WAL write вече е извън db.lock (отделен walLock) → по-добра concurrency
- MVCC: добавен compactVersions() който се изпълнява на всеки 100 commits
  Премахва стари overwritten версии, които не са видими за active транзакции
- DistTxn: commit() вече НЕ rollback-ва commit-нали participants
  Ако някой participant е commit-нал, транзакцията се маркира като committed
- Raft: добавен disk persistence за currentTerm, votedFor, log
  saveState() при всяка промяна; loadState() при стартиране

292 теста, 0 failure-а.
2026-05-12 23:30:58 +03:00
dimgigov 750746eb18 docs: обновен BUG_AUDIT.md със статус на всички поправки
- Отбелязани 28 поправени бъга ()
- Отбелязани 4 оставащи бъга ()
- Актуализиран план с 4 завършени спринта
- Актуализирани метрики
2026-05-12 23:24:01 +03:00
dimgigov 5e585dd347 fix: deadlock detector, graph/query/lexer бъгове
Поправени проблеми:
- Deadlock detector: cycle reconstruction вече проверява пълен цикъл
- Graph cypher: executeCypher acquire-ва lock при четене
- Graph engine: loadFromFile acquire-ва lock при зареждане
- Graph engine: lock поле е exported за външен достъп
- Query: parseRowData поддържа escape-ване на , и = в стойности
- Query: execUpdateRow escape-ва стойности при сериализация
- Lexer: readNumber спира при втора точка (1.2.3 вече не е валиден float)
- Lexer: skipBlockComment хвърля ValueError при незатворен коментар

292 теста, 0 failure-а.
2026-05-12 23:21:29 +03:00