Commit Graph

147 Commits

Author SHA1 Message Date
dimgigov 8fb5dde858 fix(protocol): serialize float32/float64/fkVector in big-endian
The JavaScript client reads floats via readFloatBE/readDoubleBE,
which expect IEEE-754 values in big-endian byte order.
The Nim server was writing them with copyMem in native byte order,
so on little-endian machines (x86_64) the JS side deserialized
FLOAT64 values as garbage (e.g. 1.0 became 3.03865e-319).

Fix serialization by casting to int32/int64 and using bigEndian32/
bigEndian64, mirroring the existing big-endian handling for ints.
Apply the same fix to deserialization and to fkVector elements.
2026-05-14 02:22:07 +03:00
dimgigov a0d2ca7776 Merge release/v1.1.0 into main 2026-05-13 15:07:44 +03:00
dimgigov 4e7e568525 release: v1.1.0 — bug fixes, Docker, clients, SCRAM auth
- Bump version to 1.1.0 across all components
- Remove debug CI artifacts (debug.yml, test_all.nim.bak, test_lock)
- Update CHANGELOG with v1.1.0 release notes
- Bug fixes: irNeg, distributed txn, sharding, Raft majority, MVCC,
  LSM-Tree, auth (JWT + SCRAM-SHA-256), wire protocol, SQL injection,
  ReDoS, graph, vector, FTS, build warnings
- Client SDKs: JS/TS, Python, Nim, Rust with tests and examples
- Docker: production + source + test compose configs
- TLA+: crossmodal, backup, recovery specs with symmetry reduction
2026-05-13 15:05:59 +03:00
dimgigov 6665ee7e0a fix: irNeg (unary minus) evaluation in query executor
- Added missing irNeg case in evalExpr (query/executor.nim)
- Unary minus previously fell through to else: return 'false'
- Added 2 tests: SELECT expression and WHERE condition
2026-05-13 14:23:28 +03:00
dimgigov 42c675224c fix: distributed transaction and sharding bugs
- disttxn: rollback now correctly updates participant state (was modifying a copy)
- disttxn: bare except replaced with CatchableError
- sharding: migrateData condition was always false, now correctly migrates data
- sharding: unsafe cast[string] replaced with manual byte-to-char conversion
- sharding: bare except replaced with CatchableError
- server: parseUInt replaced with parseBiggestUint for uint64 portability
2026-05-13 14:01:13 +03:00
dimgigov 45849cbe5c fix(tests): use less common ports 29001-29003 in Raft network transport test 2026-05-13 13:53:07 +03:00
dimgigov ac84474789 debug(ci): keep only Enhanced Migrations onwards in test_all.nim 2026-05-13 13:50:03 +03:00
dimgigov 92af288d06 debug(ci): keep only Row-Level Security onwards in test_all.nim 2026-05-13 13:46:54 +03:00
dimgigov baf2b1a504 debug(ci): keep only TLS/SSL onwards in test_all.nim 2026-05-13 13:43:46 +03:00
dimgigov 8d600a6f23 debug(ci): keep only last quarter of test_all.nim 2026-05-13 13:40:38 +03:00
dimgigov ca87acb120 debug(ci): comment out first half of test_all.nim 2026-05-13 13:36:58 +03:00
dimgigov d84033984a debug(ci): keep only second half of test_all.nim 2026-05-13 13:34:05 +03:00
dimgigov 1923aa225b debug(ci): truncate test_all.nim to first half of suites 2026-05-13 13:31:15 +03:00
dimgigov 9777b73e7c fix(tests): add missing import std/tables in join_tests.nim 2026-05-13 13:27:34 +03:00
dimgigov ab2522657f debug(ci): add lock/lsm test to isolate failure 2026-05-13 13:22:06 +03:00
dimgigov adf5937198 debug(ci): add timeout to join_tests 2026-05-13 13:10:39 +03:00
dimgigov 113d7faafa debug(ci): run join_tests and tla_faithfulness separately 2026-05-13 13:07:32 +03:00
dimgigov 15a905bdd1 debug(ci): emit notice with 0x0.st URL for test_all.log 2026-05-13 13:02:14 +03:00
dimgigov 14791d8be6 debug(ci): remove || true mask from test_all run 2026-05-13 12:59:26 +03:00
dimgigov dbe92fab9d debug(ci): always upload test_all.log to branch 2026-05-13 12:55:48 +03:00
dimgigov d9835fe696 debug(ci): remove || echo mask from test_all binary run 2026-05-13 12:52:25 +03:00
dimgigov 1111fbf359 debug(ci): upload logs to new branch on failure 2026-05-13 12:48:05 +03:00
dimgigov 1ed9f90852 debug(ci): add nim version check step before minimal test 2026-05-13 12:40:31 +03:00
dimgigov dea4870d87 debug(ci): try iffy/install-nim instead of jiro4989/setup-nim-action 2026-05-13 12:38:17 +03:00
dimgigov 2926701f8c fix(ci): explicitly install openssl and ca-certificates in all workflows 2026-05-13 12:35:43 +03:00
dimgigov fb2bd1769d debug(ci): remove -d:ssl from minimal test to isolate issue 2026-05-13 12:33:15 +03:00
dimgigov cde443679e debug(ci): simplify debug workflow to isolate failure 2026-05-13 12:29:39 +03:00
dimgigov e0037ae817 debug(ci): add debug workflow to isolate test failure 2026-05-13 12:23:50 +03:00
dimgigov aaa4f37c77 debug(ci): post test log as commit comment on failure 2026-05-13 12:16:56 +03:00
dimgigov 25ba9911b7 debug(ci): emit notice annotation with 0x0.st URL 2026-05-13 12:11:52 +03:00
dimgigov 8e56ff721c debug(ci): upload test log to 0x0.st on failure 2026-05-13 12:11:28 +03:00
dimgigov 942910fae5 debug(ci): print nim version, nimble packages, system info 2026-05-13 12:05:49 +03:00
dimgigov f50cbd1555 debug(ci): upload test_output.log as artifact on failure 2026-05-13 12:04:59 +03:00
dimgigov 6978097768 debug(ci): capture test output to log for debugging GitHub Actions failure 2026-05-13 12:02:01 +03:00
dimgigov f6781c107e fix(ci): add --threads:on to all nim c commands
Commit 6aaabb5 added std/locks usage in vector/engine.nim (HNSW thread-safety).
std/locks requires --threads:on. Local builds worked because /etc/nim/nim.cfg
had threads:on, but GitHub Actions runners (setup-nim-action) compile without
threads by default, causing the test job to fail silently at runtime.

Also updated baradadb.nimble tasks for consistency.

Fixes GitHub Actions CI failure on test / clients-ci workflows.
2026-05-13 11:57:09 +03:00
dimgigov c0bd3dff86 feat: distributed gaps filled — gossip UDP transport, sharding data migration, inter-module wiring (raft-disttxn, gossip-sharding, replication-disttxn) 2026-05-13 11:11:53 +03:00
dimgigov 2a8d2a6cb4 docs: update PLAN.md and BUG_AUDIT.md for v1.0.0 release
- Final metrics: 294 tests, 10 TLA+ specs, 32 bugs fixed
- Mark all critical/high/medium/config bugs as resolved
- Document SCRAM-SHA-256, crossmodal.tla, symmetry reduction
- List remaining post-v1.0.0 non-critical tasks
2026-05-13 09:54:00 +03:00
dimgigov ac55237b72 feat: real SCRAM-SHA-256 authentication (Option C)
Security:
- New scram.nim module with full SCRAM-SHA-256 per RFC 7677
  * PBKDF2-HMAC-SHA-256 key derivation
  * HMAC-SHA-256 with multiple overloads for bytes/strings
  * Secure nonce/salt generation via /dev/urandom
  * Client/server message parsing and proof verification

AuthManager updates:
- registerScramUser(): stores salted credentials (salt + iterations + storedKey + serverKey)
- startScram(): initiates challenge-response handshake
- finishScram(): verifies client proof and returns server signature
- Legacy amSCRAMSHA256 path kept for backward compatibility

HTTP endpoints:
- POST /auth/scram/start — accepts client-first-message, returns server-first
- POST /auth/scram/finish — accepts client-final-message, returns server-final

Tests:
- SCRAM-SHA-256 full handshake test (register → start → compute proof → finish)
- SCRAM-SHA-256 invalid proof rejection test

Build: 0 warnings, all tests pass
2026-05-13 09:50:07 +03:00
dimgigov 422df08ab9 feat: clean build + crossmodal.tla + TLA+ symmetry (v1.0.0-prep)
Build warnings cleanup (0 warnings):
- Suppress threadpool deprecation warning (baradadb.nim)
- Remove unused 'os' import (logging.nim)
- Fix ImplicitDefaultValue for newNode params (ast.nim)
- Add explicit cstring cast for posix.open (wal.nim)
- Fix HoleEnumConv for MsgKind parsing (server.nim)

TLA+ Formal Verification:
- Add symmetry reduction (Permutations) to all 9 existing specs
- Add SYMMETRY directive to all .cfg files
- New crossmodal.tla: cross-modal consistency spec
  * MetadataVectorConsistency, HybridResultValid
  * CommittedAtomicity, AbortedAtomicity, TxnStateValid
- New models/crossmodal.cfg

Tests:
- Add Cross-Modal TLA+ Faithfulness tests (4 tests)

Docs:
- Update PLAN.md and BUG_AUDIT.md with completed tasks
2026-05-13 09:24:04 +03:00
dimgigov d08de7a062 Update BUG_AUDIT.md and PLAN.md: mark all bugs fixed, add session 6-7, plan for tomorrow
- BUG_AUDIT.md: 34 bugs documented (32 fixed + 2 auth hardening), all metrics green
- PLAN.md: backup.tla and recovery.tla marked complete, 3 options for tomorrow
  - Option A: Clean build (warnings + symmetry reduction)
  - Option B: crossmodal.tla
  - Option C: True SCRAM-SHA-256
- Cleared 26GB of TLC state files from formal-verification/states/
2026-05-13 00:17:06 +03:00
dimgigov c0596d6ba1 Update PLAN.md: mark backup.tla and recovery.tla as completed 2026-05-13 00:12:15 +03:00
dimgigov 32b38675a9 Add TLA+ formal verification specs for backup and recovery
- backup.tla: models backup/restore/verify/cleanup with properties:
  BackupSnapshotsValid, RestoreIntegrity, VerifyIntegrity,
  RetentionInvariant, HistoryConsistency, BackupIdValid
- recovery.tla: models WAL replay REDO/UNDO with properties:
  RedoCommitted, RecoveryCompleteness, WalIntegrity
- Added models/backup.cfg and models/recovery.cfg with bounded constants
- Updated run_all.sh to include both new specs
- All 9 TLA+ specs pass TLC model checking
2026-05-13 00:06:45 +03:00
dimgigov 5e27a987dd fix: auth token expiration + constant-time JWT signature comparison
- createToken() добавя exp/iat/nbf в JWT payload
- verifyToken() parse-ва exp/iat/nbf и проверява expiration
- verifyToken() използва constantTimeCompare() за timing-attack resistance
- Добавен import std/times

292 теста, 0 failure-а.
2026-05-12 23:36:17 +03:00
dimgigov a6aeb7801b docs: обновен BUG_AUDIT.md — всички критични/високи/средни бъгове поправени 2026-05-12 23:31:51 +03:00
dimgigov a72e17f063 fix: последните 4 критични/средни бъга — LSM-Tree, MVCC, DistTxn, Raft
Поправени проблеми:
- LSM-Tree: WAL write вече е извън db.lock (отделен walLock) → по-добра concurrency
- MVCC: добавен compactVersions() който се изпълнява на всеки 100 commits
  Премахва стари overwritten версии, които не са видими за active транзакции
- DistTxn: commit() вече НЕ rollback-ва commit-нали participants
  Ако някой participant е commit-нал, транзакцията се маркира като committed
- Raft: добавен disk persistence за currentTerm, votedFor, log
  saveState() при всяка промяна; loadState() при стартиране

292 теста, 0 failure-а.
2026-05-12 23:30:58 +03:00
dimgigov 750746eb18 docs: обновен BUG_AUDIT.md със статус на всички поправки
- Отбелязани 28 поправени бъга ()
- Отбелязани 4 оставащи бъга ()
- Актуализиран план с 4 завършени спринта
- Актуализирани метрики
2026-05-12 23:24:01 +03:00
dimgigov 5e585dd347 fix: deadlock detector, graph/query/lexer бъгове
Поправени проблеми:
- Deadlock detector: cycle reconstruction вече проверява пълен цикъл
- Graph cypher: executeCypher acquire-ва lock при четене
- Graph engine: loadFromFile acquire-ва lock при зареждане
- Graph engine: lock поле е exported за външен достъп
- Query: parseRowData поддържа escape-ване на , и = в стойности
- Query: execUpdateRow escape-ва стойности при сериализация
- Lexer: readNumber спира при втора точка (1.2.3 вече не е валиден float)
- Lexer: skipBlockComment хвърля ValueError при незатворен коментар

292 теста, 0 failure-а.
2026-05-12 23:21:29 +03:00
dimgigov 80d57a36bd fix: rate limiter, graph/vector/query/server бъгове
Поправени проблеми:
- Rate limiter: globalRate вече се enforce-ва; cleanupStaleClients() за memory leak
- Graph engine: addEdge проверява дали src/dst nodes съществуват
- Vector engine: distance() хвърля ValueError при dimension mismatch
- Query: aggregate + * на empty result вече не crash-ва
- Query: exprToSql funcCall вече работи с 0 аргумента
- Query: INSERT trigger вече не crash-ва при празни VALUES
- Server: recvWithTimeout() за всички network reads
- Server: защита от negative activeConnections

292 теста, 0 failure-а.
2026-05-12 23:15:09 +03:00
dimgigov 6aaabb518d fix: високо и средно приоритетни бъгове в query, protocol, storage, vector, fts
Поправени проблеми:
- Wire protocol: bounds checking, max length limits, recursion depth limit
- SQL injection: escape-ване на quotes в exprToSql string literals
- ReDoS: escape-ване на regex metachars в LIKE/ILIKE
- Stale BTree indexes: добавен BTree.remove() + изтриване при UPDATE/DELETE
- SSL: quoteShell за всички shell команди с пътища
- Boolean literals: parser вече разпознава tkTrue/tkFalse
- Unary minus: lowerExpr map-ва ukNeg към irNeg (вместо irNot)
- Non-aggregate UDFs: lowerExpr създава irekFuncCall вместо NULL literal
- UTF-8 FTS: tokenize използва runes вместо байтове
- HNSW: добавен Lock за thread-safe insert/search

292 теста, 0 failure-а.
2026-05-12 23:02:54 +03:00
dimgigov 131541a0e5 fix: критични бъгове в storage, MVCC, auth, Raft, query executor
Поправени проблеми:
- MVCC: aborted транзакции вече не стават видими (добавено abortedTxns множество)
- LSM-Tree: поправена загуба на данни при immutable memtable overwrite
- LSM-Tree: поправена SSTable search order (сортиране по id вместо minKey)
- WAL: sync()/close() вече извикват posix.fsync()
- Auth: simpleHash (djb2) заменен с HMAC-SHA256
- Recovery: summary() вече не мутира базата данни
- Raft: поправена majority calculation за четен брой нодове
- Query: EXISTS subqueries вече изпълняват подзаявката
- Nimble: поправен build (-d:ssl) и bench task

Добавен BUG_AUDIT.md с пълен доклад и план за подобрения.

292 теста, 0 failure-а.
2026-05-12 22:45:50 +03:00