dimgigov
aa4ab11210
feat: canonical Nim client refactor with typed rows, pool, and allographer wrapper
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Extract wire protocol into clients/nim/src/baradb/wire.nim
- Add BaraError exception hierarchy
- Refactor BaraClient/SyncClient with typedRows, AsyncLock request queue, timeouts, TLS config
- Add BaraPool and optional HTTP fallback
- Add mock-server wire and pool unit tests
- Bump baradb nimble package to 1.2.0
- Make nim-allographer depend on canonical baradb client
- Use typed rows in allographer toJson
- Deprecate src/barabadb/client/client.nim
- Update docs/en/clients.md and clients/nim/README.md
2026-06-18 21:29:55 +03:00
dimgigov
1c42eff7ef
fix(vector): use float64 accumulators for distance functions
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Vector distance functions (cosine, euclidean, dot, manhattan) returned
float64 but accumulated sums in float32, causing precision loss in SQL
results (e.g. sqrt(2) truncated to float32). Use float64 accumulators
and casts throughout.
docs(bugs): mark BUG-023 and BUG-024 as fixed
The code already cleans up pendingAcks and passes oldAssignments from
rebalance to migrateData; update BUGS.md to match the implementation.
2026-06-12 23:06:19 +03:00
dimgigov
ef264d7d69
feat: add unified search engine — HNSW heap-opt, segment index, boolean/phrase/ngram/facet
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
New src/barabadb/search/ module with 9 components:
- priority_queue.nim: BoundedHeap for O(log n) search
- hnsw_opt.nim: heap-based searchLayer (2.4x faster, 92-99% recall@10)
- inverted.nim: segment-based index with soft-delete and compaction
- phrase.nim: positional phrase + proximity search
- boolean.nim: recursive descent parser (AND/OR/NOT/ranges/wildcards)
- ngram.nim: trigram index for O(1) fuzzy/prefix/wildcard
- stemmer.nim: Porter2 stemmers (EN/BG/DE/FR/RU)
- facet.nim: faceted search with filter pushdown
- engine.nim: UnifiedSearchEngine combining all search types
Performance (dim=128, efConstruction=200):
N=1K: 0.30ms search, 99.6% recall@10
N=10K: 1.09ms search, 92.6% recall@10
N=50K: 2.26ms search, 75.5% recall@10
Includes search benchmarks (benchmarks/search_bench.nim), updated docs
(en/bg fts.md, en/bg search.md), and crossmodal engine integration.
2026-05-30 13:42:08 +03:00
dimgigov
965ed2f675
perf: optimize FTS and HNSW engines + real PostgreSQL benchmarks
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
FTS Engine (src/barabadb/fts/engine.nim):
- Fix bm25Score doing O(n) linear scan per document
- Cache IDF per token instead of recomputing for each doc
- Use entry.termFreq directly instead of searching postings again
- Result: FTS search +438% (249 -> 1360 queries/s)
HNSW Vector Engine (src/barabadb/vector/engine.nim):
- Optimize distance functions with float32 + 4x loop unrolling
- Rewrite searchLayer: swap+pop instead of O(n) del, track worst-nearest
instead of sorting nearest on every iteration
- Result: HNSW insert +117% (245 -> 543 ops/s), search 2.2x faster
Benchmarks:
- Add real PostgreSQL comparison script (benchmarks/pg_bench.py)
- Add report generator (benchmarks/generate_report.py)
- Fix compare.nim cpuTime() bug (was dividing by 1M incorrectly)
- Add nimble tasks: bench_pg, bench_report
Docs:
- Update README.md and docs/en/performance.md with real measured numbers
- Add benchmarks/REAL_COMPARISON.md
Version bump: 1.1.7 -> 1.1.8
2026-05-29 17:11:22 +03:00
dimgigov
42043f3946
v1.1.7: deep security & reliability audit — 33 bugs fixed
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Critical (5):
- Reject empty JWT secret when authEnabled (server.nim)
- Fix 2PC marking uncontacted participants as prepared/committed (disttxn.nim)
- Fix Raft commit index calculation for even-sized clusters (raft.nim)
- Fix REP/DISTTXN protocol auth bypass (server.nim)
- Fix HTTP backup/restore path traversal (httpserver.nim)
High (11):
- Fix WAL write race with flush (lsm.nim)
- Fix MVCC savepoint/rollback deep-copy writeSet (mvcc.nim)
- Fix table mutation during deadlock iteration (mvcc.nim)
- Fix LIMIT 0 returning all rows (executor.nim)
- Fix COUNT(col) counting NULL values — 3 locations (executor.nim)
- Fix EXISTS subquery lowering missing subqueryPlan (executor.nim)
- Fix Raft appendEntries/applyCommitted array vs logical index (raft.nim)
- Fix timing attacks on constantTimeCompare and SCRAM (auth.nim, scram.nim)
- Fix B-tree leaf merge phantom separator key (btree.nim)
- Fix SSL verifyPeer not applied to newContext (ssl.nim)
- Fix sharding connectWithTimeout missing SO_ERROR check (sharding.nim)
- Fix sync replication returning success on partial ack (replication.nim)
- Fix WebSocket JWT expiration not validated (websocket.nim)
Medium (13):
- Fix writeSSTable partial file → tmp + atomic rename (lsm.nim)
- Fix multi-CTE table loss (executor.nim)
- Fix nl_to_sql DML restricted to superuser (executor.nim)
- Fix unbounded plan cache — max 10000 (adaptive.nim)
- Fix migration lock crash persistence — timestamp + stale detection (executor.nim)
- Fix admin panel auth (httpserver.nim)
- Fix MVCC unbounded txn tracking — prune in compactVersions (mvcc.nim)
- Fix connection pool maxLifetime check (pool.nim)
- Fix JWT JSON parser backslash escapes (auth.nim)
- Fix substr(s, start) returning single char (udf.nim)
- Fix loadSSTable minimum file-size check (lsm.nim)
- Fix compaction mmap leak (compaction.nim)
- Fix JSON injection in hybrid_search_filtered (executor.nim)
Low (4):
- Raft loadState logs error instead of silent discard
- Replication healthCheck double-close fixed
- Lexer readIdent double column counting fixed
- WebSocket frame 32-bit overflow guard
All 448 tests passing, 0 failures. Bump version to 1.1.7.
2026-05-29 14:17:41 +03:00
dimgigov
37a8ed52ba
deps: bump jwt-nim-baraba to v2.1.2 (security fixes & Nim 2.2 compat)
2026-05-26 13:23:54 +03:00
dimgigov
a5abb6031b
fix(backup): fix 7 bugs in backup/restore — strip-components, multi-db path, verify+log in HTTP handler, parseBackupFilename for .tar, getArchiveSize gzip -l, dead code removal
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-25 20:44:14 +03:00
dimgigov
a37a62c69e
clients/nim-allographer/src/allographer/query_builder/models/baradb/baradb_exec.nim src/barabadb/core/mvcc.nim src/barabadb/query/codegen.nim src/barabadb/query/lexer.nim src/barabadb/query/parser.nim src/barabadb/query/udf.nim tests/test_all.nim
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-25 18:56:02 +03:00
dimgigov
0c7847abba
docs: update backup.md with multi-database and HTTP API docs
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-25 18:49:26 +03:00
dimgigov
eb62b381a1
feat: multi-database backup, HTTP API endpoints, admin panel UI
...
- backup CLI: add --all-databases, --database, --data-root flags
- backup format: tar.gz with databases/<name>/ + backup.json metadata
- HTTP API: new endpoints /databases, /backup, /backups, /restore
- HTTP API: database switching via X-Database header
- Admin panel: database selector dropdown, Databases tab, Backups tab
- Admin panel: backup/restore UI with fetch() to new endpoints
- Update deps: hunos 1.3.0, jwt-nim-baraba 2.1.0
- Docker: backup cron uses --all-databases --data-root=/data/databases
- Docs: update api-http.md, protocol.md with correct endpoints
- OpenAPI spec updated to v1.1.6
Known limitation: DELETE /databases disabled due to ORC SIGSEGV
in LSMTree.close(). Use SQL DROP DATABASE instead.
2026-05-25 18:46:07 +03:00
dimgigov
a9ea80a1f8
fix(prop_test): "Nil expr evaluates to NULL" test — compare Value.kind, not Value == ""
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
The test was comparing a Value to a string literal which always failed.
Changed to check s.kind == vkNull on both calls.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-05-23 22:06:37 +03:00
dimgigov
7d19e06805
fix(btree): critical seq.del vs seq.delete bug and B+ tree leaf traversal
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Root cause: seq.del(i) does swap-with-last (destroys sorted order)
while seq.delete(i) preserves order. This caused massive data loss
during B-tree removes.
Changes:
- btree.nim: Replace seq.del with seq.delete (critical data corruption fix)
- btree.nim: search() traverses leaf linked list for duplicate boundary keys
- btree.nim: splitChild() consolidates duplicate boundary key values before split
- btree.nim: len() counts unique keys via leaf traversal
- btree.nim: remove() traverses leaf linked list to remove ALL occurrences
- btree.nim: Fix separator update in internal node (node.keys[i-1] not [i])
- prop_test.nim: Use set equality for interleaved test (rebalancing reorders)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-05-23 22:01:20 +03:00
dimgigov
b25fea4d21
fix: all 5 bugs — reserved words, cloneForConnection, SQL injection, SHOW TABLES, getRowPlain
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-21 22:51:04 +03:00
dimgigov
837b4d56fc
fix(nim-allographer/baradb): critical client fixes and bug audit
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Export hidden symbols in baradb_client.nim (socket, nextId, readQueryResponse, toString, toBytes)
- Add missing imports in baradb_exec.nim (asyncnet, enums, baradb_query)
- Rewrite toJson() to use wire protocol columnTypes instead of guessing types from strings
- Fix aggregate functions (count, avg, sum) to handle JInt/JFloat/JString correctly
- Fix paginate/fastPaginate discard issues
- Add placeholder arguments for whereIn/whereNotIn/whereBetween/whereNotBetween
- Add placeholder count validation in formatSql()
- Fix getRowPlain() to avoid IndexDefect on empty results
- Harden transactionStart/transactionEnd against server crashes
- Reset placeHolder in insertSql() to avoid stale values
- Fix test_query.nim to use backtick quoting for BaraDB compatibility
Known server bugs documented in BARADB_CLIENT_BUGS.md
2026-05-21 22:01:55 +03:00
dimgigov
bb843b9a03
feat: migrate system + cross-DB engine + IMPORT/EXPORT syntax -- 22 files, client+server+docs
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-21 19:32:14 +03:00
dimgigov
f427ba5968
fix(baradb): schema builder, insertId RETURNING, whereNull, rename bugs
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- create_column_query.nim: full RdbTypeKind -> SQL mapping (~350 lines)
- create_table.nim, add_column.nim, change_column.nim: use createColumnString
- insertId: use RETURNING clause instead of SELECT MAX race condition
- whereNull: add missing 'symbol': 'is' key to prevent runtime crash
- rename_column/rename_table: fix changeTo -> previousName
- create_migration_table: use backtick quoting consistently
- PLAN_BARADB.md: update status to reflect completed fixes
2026-05-21 16:06:45 +03:00
dimgigov
8a385cba6a
docs(readme): clarify this is a BaraDB fork of nim-allographer
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-21 15:54:56 +03:00
dimgigov
6bfc5b3a3c
feat(clients): add nim-allographer client with BaraDB driver support
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Pure Nim wire protocol client (async + sync, no C FFI)
- Query Builder: fluent API, SQL generator, execution, transactions
- Schema Builder: create/alter/drop tables, column operations
- Connection pool with aging and timeout handling
- ORM integration via compile-time DB_BARADB switch
- Tests: test_open, test_query
- Documentation: README.md, PLAN_BARADB.md
2026-05-21 15:49:11 +03:00
dimgigov
2d310a33a1
fix: resolve all 55 identified bugs across the codebase
...
Comprehensive bug fix pass across all subsystems:
- Critical: lexer infinite loop, WAL lock discipline, checkpoint safety,
compaction verification, HMAC key truncation, healthCheck leak
- High: MVCC lock safety, B-Tree delete rebalancing, Raft stale term
handling, replication socket leaks and ack cleanup, sharding migration
with old assignments, 2PC recovery, JWT/SCRAM auth fixes, wire protocol
bounds checks
- Medium: config error handling, MERGE parser completeness, IR/codegen
correctness, UDF bounds checks, LIMIT cost accuracy, mmap safety
- Low: timeout handling, float parsing edge cases, uint32 truncation,
cache entry cleanup
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-05-21 10:59:51 +03:00
dimgigov
7fc6adbe47
chore: remove old planning and bug report files
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
These files are outdated and only cause confusion.
Keep only the current PLAN.md.
2026-05-21 09:47:40 +03:00
dimgigov
072e7a5d13
test(join): add test for duplicate columns in SELECT *
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Verify that JOIN with overlapping column names preserves
both left (bare) and right (qualified) values.
2026-05-21 09:44:32 +03:00
dimgigov
d4dc433e48
fix(query): preserve both sides of duplicate column names in JOIN
...
When SELECT * joins tables with overlapping column names
(e.g. 'id' in both users and orders), the right-side value
was being dropped because irekStar filtered out dotted keys.
- Add expandStarRow() that keeps bare columns and adds
qualified aliases (u.id, o.id) only for duplicated names
- Fix r.columns expansion to include all joined tables,
qualifying duplicates from the right side
2026-05-21 09:44:31 +03:00
dimgigov
1ff6fabaff
fix(backup): resolve relative SSTable paths in incremental backup
...
MANIFEST stores relative paths like 'sstables/1.sst'.
The old code checked fileExists against CWD instead of dataDir,
silently omitting SSTables when run from another directory.
Now resolves paths relative to dataDir.
2026-05-21 09:44:28 +03:00
dimgigov
4a83eb6783
docs: update distributed/backup/schema/storage for multi-database
...
- Add multi-database limitations to distributed docs
- Fix backup commands and add multi-database backup guidance
- Add SQL migration section to schema docs
- Update storage examples to use per-database paths
2026-05-21 09:44:25 +03:00
dimgigov
a526a3aef9
docs: remove non-English/Bulgarian language docs, fix links
...
Remove ar, de, fa, ru, tr, zh documentation folders.
Update docs/index.md and docs/bg/index.md to only list en/bg.
Update README badge from 7 to 2 languages.
Update changelog entry for Bulgarian docs.
2026-05-21 09:44:17 +03:00
dimgigov
372e5cf627
fix(query,types): refactor Row to Value-typed map, fix IN list, nkPath, multi-table joins
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Refactored Row from Table[string, string] to Table[string, Value]
- Added Value operators: ==, !=, $, in for string interop
- Fixed valueToString for vkNull to return '\\N' instead of ''
- Fixed evalExpr irekField to return vkNull when field not found
- Added IN (val1, val2, ...) parser support
- Fixed nkPath column names in multi-table joins
- Fixed LATERAL JOIN null padding when no matching rows
- Added CREATE/DROP/USE/SHOW DATABASE parser support
- Adapted all tests for new Value type
2026-05-20 23:22:14 +03:00
dimgigov
57d2908066
fix(executor,mvcc,tests): dedup aliases, fix txn delete, fix MIN/MAX null, add regression tests
...
- Add isDelete flag to VersionedRecord to fix COMMIT treating empty values
as DELETE. Previously version.xmax != TxnId(0) was never true for active
transactions, so deletes inside transactions never removed rows.
- Fix MIN/MAX aggregates skipping NULL: was checking for "\\N" instead
of correct "\N" sentinel, so NULL values were never skipped.
- Add duplicate alias deduplication in lowerSelect() and getSelectColumns().
Identical aliases now get _1, _2 suffixes (e.g. count(*), count(*)_1).
- Add nil guard for expr.subqueryPlan in irekSubquery evalExpr handler.
- Initialize txnManager in newExecutionContext to prevent segfault on BEGIN.
- Add regression tests: DELETE in transaction, PK-only row survival,
MIN/MAX NULL skipping.
2026-05-20 21:36:15 +03:00
dimgigov
edd7e728f9
fix(query): add correlated subquery support
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
evalExpr was missing irekSubquery case — scalar subqueries in SELECT/WHERE
fell through to else:return "" producing empty strings. Add:
- irekSubquery handler in evalExpr that executes the subquery plan
- outerRow + subqueryPlan fields on ExecutionContext for correlation
- qualified column injection in execScan for correlated filter refs
- collectCorrelatedTablesFromPlan to discover outer table references
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-05-19 23:16:31 +03:00
dimgigov
14596c9584
fix(adapter,executor,tests): harden nimforum adapter and fix transaction/aggregate bugs
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Adapter (baradb_sqlite.nim):
- getRow now returns @[] for missing rows (db_sqlite compat)
- insertID/nextId handle NULL (\N) values correctly
- preprocessQuery replaces SQLite-specific syntax:
TIMESTAMP->DATETIME, DATETIME('now')->current timestamp, INET->STRING
- add dbBegin/dbCommit/dbRollback for transaction support
Executor:
- fix COMMIT treating empty values as DELETE (rows with only PK were lost)
- fix MIN/MAX aggregates including NULL (\N) which skewed results
- add numeric comparison for irNeq
Tests:
- stress_test: shared DB, 16 workers, correctness scan after run
- nimforum_smoke_test: 10 tests including schema, transactions, iterators
2026-05-19 20:43:00 +03:00
dimgigov
d3b77174da
fix: use numeric comparison for MIN/MAX aggregates
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-19 15:19:47 +03:00
dimgigov
cd46edcb67
Bump version to 1.1.6; fix storage bugs, tests, CI pipeline and Docker config
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Storage fixes:
- Fix bloom filter hash2 OverflowDefect (uint64 instead of int Hash)
- Fix MemTable.put size calculation on overwrite (was leaking size)
- Fix fuzz_test duplicate-key issues in LSM delete and SSTable tests
Test fixes:
- Fix stress_test.nim: replace deprecated threadpool with std/threads
- Fix fuzz_test.nim: add missing imports (tables, algorithm, sets)
- Fix fuzz_test.nim: var sst for close() compatibility
- Remove unused imports from test_all.nim and bench_all.nim
Docker / CI fixes:
- Fix Dockerfile.source: invalid nimlang/nim:2.2.10-alpine tag → 2.2.10 + ubuntu:24.04 runtime
- Fix Dockerfile.source healthcheck (--spider → -qO- for 200 OK)
- Fix Dockerfile run comment ports (9470/9471 → 9912/9913)
- Fix scripts/docker-run.sh healthcheck port (9470 → 9912)
- Add ARG BUILD_DATE/VCS_REF to Dockerfile for docker-build.sh
- Update all version strings 1.1.4 → 1.1.6 across nimble/docker/source/docs
2026-05-19 12:13:33 +03:00
dimgigov
343f479127
docs(bg,en): fix ports and WebSocket example for new server model
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Update architecture diagram to include WebSocket server (port 9913)
- Fix HTTP port from 9470 to 9912 (TCP + 440) in protocol docs
- Fix WebSocket port from 9471 to 9913 (TCP + 441) in protocol docs
- Fix WebSocket Nim example: remove non-existent onMessage API and
unsafe cast[string](data), use correct WsServer API with safe helpers
- Clarify TCP_NODELAY is also handled internally by hunos (HTTP)
- Note that bytesToString/stringToBytes are wire-protocol helpers
2026-05-18 19:41:21 +03:00
dimgigov
8afa998516
feat: rate limiting, shared DB instance, TCP_NODELAY fix
...
- Add rate limiter to TCP and HTTP servers
- Share single LSMTree between TCP and HTTP servers
- Replace unsafe cast[string]/cast[seq[byte]] with safe helpers
- Stabilize gossip UDP socket with exponential backoff
- Fix TCP_NODELAY via low-level setSockOptInt(IPPROTO_TCP)
(asyncnet.setSockOpt(OptNoDelay) uses SOL_SOCKET and fails)
- Apply TCP_NODELAY to server and websocket sockets
2026-05-18 19:33:40 +03:00
dimgigov
7e6a45e6b7
Стабилизация на storage слоя — 6 фази
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Фаза 1: SSTable Integrity
- SSTable v3 формат с CRC32 footer (data/index/bloom CRC)
- Нов модул storage/crc32.nim (zero-dep, IEEE 802.3)
- verifySSTable() — проверка на magic, version, CRC
- loadSSTable strict mode — отхвърля корумпирани файлове
- newLSMTree логва [WARN] при corrupt SSTables
Фаза 2: baradb repair
- Нов модул tools/repair.nim
- Сканира SSTables, проверява CRC, премества битите в corrupt/
- WAL replay (CrashRecovery) за възстановяване на данни
- CLI: ./baradadb repair --data-dir=... [--dry-run]
Фаза 3: MANIFEST File
- Atomic write MANIFEST (tmp + rename)
- newLSMTree зарежда от MANIFEST, fallback към walkDir
- checkStorageConsistency() — orphan/missing detection
- flushUnsafe и compaction записват MANIFEST
Фаза 4: WAL Rotation & Incremental Backup
- WAL segment rotation при 64MB (wal_archive/wal.000NNN.log)
- maybeRotate() на всеки 1000 записа + при flush
- backup incremental — архивира MANIFEST + SSTables + WAL
- CRC verification на SSTables преди архивиране
Фаза 5: Online Consistent Backup
- checkpoint() — freeze memtable + flush + WAL rotate
- ./baradadb checkpoint — offline consistent snapshot
- backup --online — checkpoint + incremental backup
Фаза 6: SSTable Version Migration
- SSTable.fileVersion поле
- listLegacySSTables() — намира v1/v2 файлове
- migrateSSTable() — пренаписва към v3 (tmp + rename)
- ./baradadb migrate [--dry-run] — offline migration
Документация:
- Обновени docs/en/backup.md, docs/bg/backup.md
- Обновени docs/en/storage.md, docs/bg/storage.md
- Добавени тестове в tests/test_all.nim
2026-05-18 15:04:49 +03:00
dimgigov
cf2aba104f
Phase 5 complete: B-tree property tests + benchmark regression suite
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- prop_test.nim: 11 new property-based B-Tree invariants
(size accuracy, get roundtrip, scan ordering, range correctness,
contains, remove, large order, duplicates, empty tree, interleaved ops)
- bench_all.nim: JSON-based benchmark result tracking with regression
comparison against previous runs; each benchmark reports ops/sec
delta vs last run
2026-05-18 12:52:44 +03:00
dimgigov
132e8c7a31
Phase 5: Storage fuzz tests + WAL entryCount fix + SSTable overflow checks + compaction sizeBytes accuracy
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- WAL: entryCount is now accurate after restart by counting existing entries
- LSM writeSSTable: adds uint32 overflow checks for key/value/entry counts
- Compaction: sizeBytes now uses actual file size instead of rough guess
- Fuzz tests: 14 new tests covering WAL truncation, LSM put/get/delete,
recovery, scan, overwrites, SSTable roundtrip, corruption, tombstones
2026-05-18 12:34:26 +03:00
dimgigov
3f29c519d4
Storage hardening: btree size fix, WAL entryCount note, LSM put error handling
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- btree.nim: insertNonFull returns bool; size only increments for new keys
- wal.nim: comment about entryCount reset on append
- lsm.nim: recovery/put/delete/putUnsafe raise IOError if put fails after flush
2026-05-18 12:14:30 +03:00
dimgigov
8596cea548
Stabilization: replication semi-sync, MVCC aborted deleters, SSTable level, SCRAM fix
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
P0-P1 fixes:
- replication.nim: rmSync/rmSemiSync now wait for replica ACKs
- mvcc.nim: aborted deleters no longer hide records (return true for visibility)
- storage/lsm.nim: SSTable level persisted in header (version 2, backward compat)
- protocol/auth.nim: removed insecure SCRAM raw hash fallback
Build: 0 warnings, 0 errors. All tests pass.
2026-05-18 12:04:45 +03:00
dimgigov
47c4393a7e
Complete plan execution: gossip/raft bounds, deadlock fix, JSON escaping, memtable limits
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Remaining plan items:
- 1.4 hybrid_search: JSON built with std/json instead of string concat
- 2.3 lsm.nim: reject oversized memtable entries
- 3.3 gossip.nim: bounds checking for idLen/nodeCount/hostLen
- 3.4 raft.nim: max length caps for cmdLen/dataLen/logLen
- 4.3 deadlock.nim: per-path seq stack instead of global parent table
2026-05-18 11:42:46 +03:00
dimgigov
967c0855a5
Bug fixes: composite PK, nl_to_sql sandbox, FK check, SQL injection, storage correctness
...
Critical fixes:
- Composite PK: execInsert + validateConstraints use all PK columns
- nl_to_sql: non-SELECT SQL no longer executed directly during validation
- FK check: removed O(N) scanMemTable fallback, uses db.get() with SSTables
- exprToSql: nkIdent wrapped in quotes to prevent SQL injection
- restoreSchema: try/except around tokenize/parse for crash resilience
- recovery.nim: lastTxnId tracking + putUnsafe/deleteUnsafe without WAL
- SCRAM: verifyClientProof length check + DefaultIterationCount restored
Storage fixes:
- lsm.nim: SSTable sort order fixed (ascending), close() flushes all memtables
- compaction.nim: tombstones preserved during compaction
- wal.nim: header written for empty existing files, readEntries checks magic
- btree.nim: B+ tree leaf split keeps boundary key
- bloom.nim: deserialize raises on short data
- mmap.nim: bounds checks for adviseWillNeed/DontNeed + posix.close()
Protocol fixes:
- zerocopy.nim: readString bounds check
- wire.nim: deserializeValue 32-bit underflow check
- auth.nim: JWT JSON escaping for claims
- server.nim: readUint32BE bounds check + specific exception handling
- raft.nim: readData checks + specific exception handling
Tests:
- Added Composite Primary Key test suite (4 tests)
Build: 0 warnings, 0 errors
2026-05-18 11:33:11 +03:00
dimgigov
a28c845476
Bump version to 1.1.4 across all components
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 20:57:19 +03:00
dimgigov
3ac53ecda2
docs(readme): add AI-Native Data Platform features (Sessions 10-12)
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- Update Why BaraDB comparison table with Hybrid RAG, AI Agents,
MCP Server, LangChain integration, Graph SQL, Cypher
- Add Hybrid RAG Search section under Vector Engine
- Add Graph SQL Integration section with CREATE GRAPH, GRAPH_TABLE,
Cypher translation, and similarity/node2vec algorithms
- Add new AI-Native Data Platform section covering:
- Natural Language → SQL (nl_to_sql, schema_prompt)
- Text Chunking & Auto-Embedding (chunk, embed_text)
- MCP Server (Model Context Protocol over STDIO)
- LangChain Vector Store (Python + JS)
- Chat Message History with RLS
- Update Project Structure with ai/ and mcp/ directories
- Update Roadmap Progress with v1.1.2 features
2026-05-17 17:05:47 +03:00
dimgigov
1e38e29f25
fix(security): SQL injection in Python/JS clients + bare except + session leak
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Fix SQL injection vulnerabilities in chat_history.py, langchain_store.py,
rag_pipeline.py, baradb_langchain.js by switching to parameterized queries
- Replace dangerous bare except: with except CatchableError:/ValueError:
in llm.nim, embed.nim, cypher.nim, mcp/server.nim, executor.nim
- Fix session variable leak in MCP handleVectorSearch/handleSchemaInspect
- Build: 0 errors, all tests pass
2026-05-17 16:58:05 +03:00
dimgigov
c95bc4cd44
docs: synchronize documentation across all languages
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- Add mcp.md to: bg, fa, ru, tr, zh, ar
- Add index.md to Bulgarian (bg)
- Add 24 missing German (de) documentation files
Translations for all supported languages now complete.
2026-05-17 16:29:28 +03:00
dimgigov
a5d34c001a
docs: add German (DE) documentation + update all docs for Sessions 10-12
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
New German documentation (docs/de/):
- index.md, quickstart.md, installation.md
- baraql.md, graph.md, vector.md, mcp.md
Updated English documentation:
- changelog.md: all Sessions 10-12 features
- graph.md: SQL GRAPH_TABLE, CREATE GRAPH, all 8 algorithms, Cypher, similarity_nodes, node2vec
- vector.md: hybrid RAG, chunk(), embed_text(), auto-embed, nl_to_sql(), schema_prompt()
- baraql.md: new AI & Cross-Modal Functions section, updated keyword tables
- mcp.md: MCP Server documentation (new file)
- index.md: added German (DE) language link
2026-05-17 16:15:45 +03:00
dimgigov
e783215276
docs: mark all sessions 10-11-12 complete — plan finished
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 16:01:20 +03:00
dimgigov
a0c5ce8598
feat: Session 11 finale — similarity_nodes, node2vec, Cypher layer
...
- similarity_nodes(graph, metric): Jaccard / Adamic-Adar similarity between all node pairs
- node2vec_embed(graph, dimensions): Random-walk based graph embeddings
- cypher(query): Translate Cypher MATCH...RETURN → BaraQL GRAPH_TABLE SQL
- Fixed recursive lock deadlock in similarity functions
- Fixed graph start node parsing (pattern variable vs numeric ID)
- Added sequtils import to graph engine
- All 340+ existing tests pass
2026-05-17 16:00:15 +03:00
dimgigov
898f108963
docs: update PLAN.md - all sessions 10, 11, 12 marked complete
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 15:39:20 +03:00
dimgigov
e23b1d61d2
feat: Session 12 AI Agents & NL->SQL
...
- src/barabadb/ai/llm.nim: LLM client for NL->SQL generation
- Supports OpenAI-compatible and Ollama APIs
- Configurable via BARADB_LLM_ENDPOINT, BARADB_LLM_MODEL, BARADB_LLM_API_KEY
- extractSQL() parses SQL from LLM responses (handles markdown blocks)
- Temperature 0.1 for deterministic SQL generation
- nl_to_sql() SQL function: natural language -> SQL
- Schema-aware prompt with table column definitions + indexes + RLS
- Query validation layer: wraps generated SQL in LIMIT 0 subquery
- Self-correction loop: on error, feeds error back to LLM for fix
- Tenant-aware: respects current session variables
- schema_prompt() SQL function: generates DDL + sample data + indexes
- Returns full CREATE TABLE statement with column types and constraints
- Includes up to 5 sample rows for context
- Lists indexes, RLS policies, foreign keys
- Perfect for feeding into LLM context
- All 340+ existing tests pass
2026-05-17 15:38:14 +03:00
dimgigov
80c3fee9de
feat: 10.2.3 ChatMessageHistory + 10.2.4 RAG pipeline example
...
- clients/python/baradb/chat_history.py: BaraDBChatHistory class
- Stores conversation threads in BaraDB with multi-tenant RLS
- session_id + tenant_id + user_id isolation
- Auto-creates table and index
- Compatible with LangChain message format
- examples/rag_pipeline.py: End-to-end RAG pipeline example
- PDF/text ingestion -> chunking -> embedding -> BaraDB storage
- Hybrid search with vector distance
- LLM generation (OpenAI / Ollama)
- Supports --file and --query modes
- Configurable chunk size, overlap, top-k
- PLAN.md: Updated all Session 10 tasks as complete
2026-05-17 15:30:20 +03:00
dimgigov