Root cause: seq.del(i) does swap-with-last (destroys sorted order)
while seq.delete(i) preserves order. This caused massive data loss
during B-tree removes.
Changes:
- btree.nim: Replace seq.del with seq.delete (critical data corruption fix)
- btree.nim: search() traverses leaf linked list for duplicate boundary keys
- btree.nim: splitChild() consolidates duplicate boundary key values before split
- btree.nim: len() counts unique keys via leaf traversal
- btree.nim: remove() traverses leaf linked list to remove ALL occurrences
- btree.nim: Fix separator update in internal node (node.keys[i-1] not [i])
- prop_test.nim: Use set equality for interleaved test (rebalancing reorders)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Export hidden symbols in baradb_client.nim (socket, nextId, readQueryResponse, toString, toBytes)
- Add missing imports in baradb_exec.nim (asyncnet, enums, baradb_query)
- Rewrite toJson() to use wire protocol columnTypes instead of guessing types from strings
- Fix aggregate functions (count, avg, sum) to handle JInt/JFloat/JString correctly
- Fix paginate/fastPaginate discard issues
- Add placeholder arguments for whereIn/whereNotIn/whereBetween/whereNotBetween
- Add placeholder count validation in formatSql()
- Fix getRowPlain() to avoid IndexDefect on empty results
- Harden transactionStart/transactionEnd against server crashes
- Reset placeHolder in insertSql() to avoid stale values
- Fix test_query.nim to use backtick quoting for BaraDB compatibility
Known server bugs documented in BARADB_CLIENT_BUGS.md
When SELECT * joins tables with overlapping column names
(e.g. 'id' in both users and orders), the right-side value
was being dropped because irekStar filtered out dotted keys.
- Add expandStarRow() that keeps bare columns and adds
qualified aliases (u.id, o.id) only for duplicated names
- Fix r.columns expansion to include all joined tables,
qualifying duplicates from the right side
MANIFEST stores relative paths like 'sstables/1.sst'.
The old code checked fileExists against CWD instead of dataDir,
silently omitting SSTables when run from another directory.
Now resolves paths relative to dataDir.
Remove ar, de, fa, ru, tr, zh documentation folders.
Update docs/index.md and docs/bg/index.md to only list en/bg.
Update README badge from 7 to 2 languages.
Update changelog entry for Bulgarian docs.
- Refactored Row from Table[string, string] to Table[string, Value]
- Added Value operators: ==, !=, $, in for string interop
- Fixed valueToString for vkNull to return '\\N' instead of ''
- Fixed evalExpr irekField to return vkNull when field not found
- Added IN (val1, val2, ...) parser support
- Fixed nkPath column names in multi-table joins
- Fixed LATERAL JOIN null padding when no matching rows
- Added CREATE/DROP/USE/SHOW DATABASE parser support
- Adapted all tests for new Value type
- Add isDelete flag to VersionedRecord to fix COMMIT treating empty values
as DELETE. Previously version.xmax != TxnId(0) was never true for active
transactions, so deletes inside transactions never removed rows.
- Fix MIN/MAX aggregates skipping NULL: was checking for "\\N" instead
of correct "\N" sentinel, so NULL values were never skipped.
- Add duplicate alias deduplication in lowerSelect() and getSelectColumns().
Identical aliases now get _1, _2 suffixes (e.g. count(*), count(*)_1).
- Add nil guard for expr.subqueryPlan in irekSubquery evalExpr handler.
- Initialize txnManager in newExecutionContext to prevent segfault on BEGIN.
- Add regression tests: DELETE in transaction, PK-only row survival,
MIN/MAX NULL skipping.
evalExpr was missing irekSubquery case — scalar subqueries in SELECT/WHERE
fell through to else:return "" producing empty strings. Add:
- irekSubquery handler in evalExpr that executes the subquery plan
- outerRow + subqueryPlan fields on ExecutionContext for correlation
- qualified column injection in execScan for correlated filter refs
- collectCorrelatedTablesFromPlan to discover outer table references
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Update architecture diagram to include WebSocket server (port 9913)
- Fix HTTP port from 9470 to 9912 (TCP + 440) in protocol docs
- Fix WebSocket port from 9471 to 9913 (TCP + 441) in protocol docs
- Fix WebSocket Nim example: remove non-existent onMessage API and
unsafe cast[string](data), use correct WsServer API with safe helpers
- Clarify TCP_NODELAY is also handled internally by hunos (HTTP)
- Note that bytesToString/stringToBytes are wire-protocol helpers
- Add rate limiter to TCP and HTTP servers
- Share single LSMTree between TCP and HTTP servers
- Replace unsafe cast[string]/cast[seq[byte]] with safe helpers
- Stabilize gossip UDP socket with exponential backoff
- Fix TCP_NODELAY via low-level setSockOptInt(IPPROTO_TCP)
(asyncnet.setSockOpt(OptNoDelay) uses SOL_SOCKET and fails)
- Apply TCP_NODELAY to server and websocket sockets
- prop_test.nim: 11 new property-based B-Tree invariants
(size accuracy, get roundtrip, scan ordering, range correctness,
contains, remove, large order, duplicates, empty tree, interleaved ops)
- bench_all.nim: JSON-based benchmark result tracking with regression
comparison against previous runs; each benchmark reports ops/sec
delta vs last run
- btree.nim: insertNonFull returns bool; size only increments for new keys
- wal.nim: comment about entryCount reset on append
- lsm.nim: recovery/put/delete/putUnsafe raise IOError if put fails after flush
- Add mcp.md to: bg, fa, ru, tr, zh, ar
- Add index.md to Bulgarian (bg)
- Add 24 missing German (de) documentation files
Translations for all supported languages now complete.
- New modules: src/barabadb/ai/chunk.nim (text chunking) and embed.nim (HTTP embedding client)
- chunk() SQL function: returns JSON array of chunks with configurable size/overlap
- embed_text() SQL function: calls external embedding API (OpenAI/Ollama compatible)
- Auto-embedding on INSERT: when VECTOR column is null but TEXT column is populated,
generates embeddings via configured embedder
- Configurable via env vars: BARADB_EMBED_ENDPOINT, BARADB_EMBED_MODEL, BARADB_EMBED_API_KEY
- All 340+ existing tests pass
MCP Server (10.3):
- STDIO JSON-RPC 2.0 transport with 3 AI tools: query, vector_search, schema_inspect
- Multi-tenant session vars (tenant_id, user_id) with RLS support
- Standalone binary: build/baramcp
- Tested with all 3 tools + parameterized queries + vector search + schema inspect
Graph Engine Deep Integration (Session 11):
- CREATE GRAPH / DROP GRAPH DDL support
- Graph engine wired to SQL executor via GRAPH_TABLE() function
- 7 algorithms: BFS, DFS, PageRank, ShortestPath, Dijkstra, Louvain, Community
- INSERT into _nodes/_edges tables auto-syncs with native Graph adjacency lists
- Optional MATCH pattern, ALGORITHM, START, END, MAXDEPTH in GRAPH_TABLE syntax
- All 340+ existing tests pass
- BaraDBStore for Python: add_texts, similarity_search, max_marginal_relevance_search, delete
- BaraDBStore for JS: addDocuments, addTexts, similaritySearch, maxMarginalRelevanceSearch, delete
- Both use hybrid_search() / hybrid_search_filtered() for vector+FTS+RRF
- Multi-tenant support via tenant_id session variable + metadata filter
- Embedding function is injected by user (OpenAI, sentence-transformers, etc.)
- MMR reranking for result diversity
- Populate HNSW metadata with all relational columns during INSERT and CREATE INDEX
- Add doHybridSearchFiltered() using searchWithFilter() for HNSW pre-filtering
- Add SQL function hybrid_search_filtered(table, vec_col, text_col, query, vector, k, filter_col, filter_val)
- Enforce k-limit on both doHybridSearch and doHybridSearchFiltered results
- 2 tests: tenant isolation + empty filter fallback
- Add tkRestrict token to lexer
- Parse ON DELETE CASCADE/SET NULL/RESTRICT and ON UPDATE CASCADE/SET NULL/RESTRICT
in both table-level and column-level FK constraints
- Add fkOnDelete/fkOnUpdate to ColumnDef and ForeignKeyDef
- Fix table-level FK constraint application (third pass after columns are created)
- Implement enforceFkOnDelete, enforceFkOnUpdate, enforceFkOnChildUpdate helpers
- Wire FK enforcement into DELETE and UPDATE execution paths
- Add 9 regression tests covering all FK actions
- Добавен IRJoinStrategy enum (nestedLoop, hash, indexNestedLoop)
- Hash Join: build hash table върху по-малката страна, probe от другата
- Index Nested Loop Join: използва B-Tree индекс за point lookup
- Query planner chooseJoinStrategy избира стратегия според наличие на индекс
- LEFT/RIGHT/FULL JOIN fallback-ват към Nested Loop
- PK индексите се игнорират за INL (само explicit индекси)
- Benchmark: JOIN 10K ~115ms (Hash), ~90ms (Index NL)
- 5 нови теста за join performance и planner избор
Тестове: 316 — 0 failures
Build: 0 warnings
- Поправени deficiencies #5-#8 (GROUP BY bare columns, aggregate names,
sync client с blocking socket, thread-safe Lock в SyncClient)
- Добавен BlockingClient (net.Socket + Lock) в clients/nim/src/baradb/client.nim
- Обновен src/barabadb/client/client.nim със sync клиент без waitFor
- Regression тестове за всички 10 deficiencies
- Добавен valueKind в IRExpr за типова информация
- evalExprValue връща Value discriminated union
- Премахнати parseFloat евристики от irAdd/Sub/Mul/Div/Mod/Pow/Neg
- INT+INT→INT, INT+FLOAT→FLOAT, FLOAT/INT→FLOAT
- 12 нови теста за type safety
Тестове: 311 — 0 failures
Build: 0 warnings
Issue #6: getSelectColumns() produced count() for count(*)
and max() for max(id). Now uses exprToSql() on funcArgs
to generate consistent names matching the actual SQL expression.
Before: columns: @[count()], rows: @[{count(*): 2}]
After: columns: @[count(*)], rows: @[{count(*): 2}]
- Fix 3× ResultShadowed in cypher.nim, recovery.nim, shell.nim
- Remove 6× UnusedImport in test files
- Build is now clean: 0 warnings, 294 tests pass
- Add 4-week stabilization plan to PLAN.md (Sessions 9.1–9.4)
- Add missing nkSubquery case in lowerExpr so scalar subqueries
(e.g. budget > (SELECT AVG(budget) FROM projects)) are lowered
correctly instead of being treated as NULL literals.
- Move UPDATE SET expression evaluation inside the row loop so
column references (e.g. salary + 5000) are resolved against the
current row, fixing type validation failures.
- Make irAdd/irSub/irMul/irDiv return integer strings when the
result is a whole number, avoiding spurious float output for INT
columns (e.g. MERGE UPDATE qty = 100 + 50 now yields 150).
These fix regressions introduced by switching NULL representation
from empty string to \N, which exposed the pre-existing bugs above.
Refs: issue #9, issue #10
Issue 9: Add backtick-quoted identifier support in lexer.nim
- Users can now write `key` to use reserved keywords as column/table names
Issue 10: Use \N as internal NULL marker instead of empty string
- isNull() now checks for \N instead of value.len == 0
- Empty strings can now be stored in NOT NULL columns
- NULL values properly distinguished from empty strings in storage, wire protocol, and HTTP JSON responses
- Add --path:src and --threads:on to nim.cfg so tests compile without manual flags
- Add tests/config.nims for automatic path resolution in test directory
- Simplify nimble task definitions (remove redundant flags now in nim.cfg)
- Fix Dockerfile.source comment: remove obsolete nimmax reference
This makes 'nim c tests/test_all.nim' and 'nimble test' work out of the box.
ID Generators:
- AUTO_INCREMENT for INTEGER columns
- SERIAL / BIGSERIAL as syntactic sugar
- gen_random_uuid() / uuid() for UUID v4 generation
- nextval() / currval() for sequence support
- snowflake_id(node_id) for distributed 64-bit IDs
- RETURNING clause for INSERT
Deficiency Fixes:
- Comma join: FROM t1, t2 now works as implicit CROSS JOIN
- DEFAULT in restoreSchema: defaults survive server restart
- Duplicate column names: t.id instead of id in joins
- Empty result sets: always send column metadata to client
- GROUP BY non-agg columns: return first row value instead of empty
- Aggregate column names: count(*) instead of count()
Bug fix: PK uniqueness check now uses correct key format
- Top-level BEGIN/COMMIT/ROLLBACK work (server-side)
- SAVEPOINT (nested tx) does not work — BaraDB server limitation
- RETURNING parsed but ignored by BaraDB executor — server limitation
- docker-compose.ormin.yml: BaraDB + Ormin dev environment
- clients/nim/ormin/docker/Dockerfile: dev image with Nim 2.2.10
- clients/nim/ormin/docker/entrypoint.sh: auto-install deps
- docker-compose.test.yml: add Ormin compile test
- Python: fix wire protocol test method names (bool_val -> bool, etc.)
- Python: make integration tests and example fully async with pytest-asyncio
- Rust: add tokio dev-dependency and convert integration tests to async/await
- Rust: update ping_test example to async
- Nim: remove committed ELF build artifact
- Docker: add BARADB_HOST/BARADB_PORT env vars to test containers
- Docker: fix docker-compose.test.yml usage (remove --abort-on-container-exit)
- Add scripts/test-clients.sh for sequential client test runs
- Remove docker-compose.test.yml from .gitignore so it is tracked
- Fix repository URLs in Python and Rust READMEs
- Add VECTOR(n) column type support in CREATE TABLE
- Add CREATE INDEX ... USING hnsw/ivfpq for vector indexes
- Add cosine_distance(), euclidean_distance(), inner_product(), l1/l2_distance()
SQL functions in expression evaluator
- Add <-> nearest-neighbor operator
- Fix ORDER BY with non-projected columns (move irpkSort before irpkProject)
- Fix execInsert to escape comma-containing values (vector literals)
- Fix MERGE tests by using unique temp dirs per test suite
- Add 8 Vector SQL Integration tests (all passing)
- Update PLAN_SQL_ADVANCED.md
- Add Window Functions: ROW_NUMBER, RANK, DENSE_RANK, LEAD, LAG,
FIRST_VALUE, LAST_VALUE, NTILE with PARTITION BY, ORDER BY,
ROWS/RANGE frame specifications
- Add MERGE/UPSERT statement with WHEN MATCHED UPDATE and
WHEN NOT MATCHED INSERT
- Add SQL/PGQ Property Graph long-term plan in PLAN_SQL_ADVANCED.md
- Add 7 new tests for Window Functions and MERGE
- Update baraql.md documentation
The gossip listener used synchronous newSocket + recvFrom,
which blocks the async event loop and prevents other async
operations from running while waiting for UDP packets.
Switch to newAsyncSocket + async recvFrom so the loop can
continue processing other tasks between gossip rounds.
The previous implementation allowed multiple concurrent async calls
(query, execute, ping) to interleave writes on the same socket,
which corrupted the binary protocol framing when NodeBB fired
parallel database operations.
Add an internal _requestQueue and _requestLock so that all TCP
requests are serialized: each async operation enqueues a task,
and tasks are drained one at a time via setImmediate().
The JavaScript client reads floats via readFloatBE/readDoubleBE,
which expect IEEE-754 values in big-endian byte order.
The Nim server was writing them with copyMem in native byte order,
so on little-endian machines (x86_64) the JS side deserialized
FLOAT64 values as garbage (e.g. 1.0 became 3.03865e-319).
Fix serialization by casting to int32/int64 and using bigEndian32/
bigEndian64, mirroring the existing big-endian handling for ints.
Apply the same fix to deserialization and to fkVector elements.
- Added missing irNeg case in evalExpr (query/executor.nim)
- Unary minus previously fell through to else: return 'false'
- Added 2 tests: SELECT expression and WHERE condition
- disttxn: rollback now correctly updates participant state (was modifying a copy)
- disttxn: bare except replaced with CatchableError
- sharding: migrateData condition was always false, now correctly migrates data
- sharding: unsafe cast[string] replaced with manual byte-to-char conversion
- sharding: bare except replaced with CatchableError
- server: parseUInt replaced with parseBiggestUint for uint64 portability
dimgigovandClaude Opus 4.7