98 Commits

Author	SHA1	Message	Date
dimgigov	50f827f8cf	fix(raft): graph apply + reject non-default DB writes ... - applyReplicatedPut/Delete keep in-memory graphs in sync with node/edge table rows (idempotent edges via addEdgeWithIdIfAbsent). - When Raft is enabled, DML is refused on any database other than 'default' (the only DB the state machine is wired to). - Docs updated (en/bg); unit test for graph apply.	2026-07-30 21:19:01 +03:00
dimgigov	0d51497f57	fix(raft): multi-stmt write gate, rich apply, delete kv convention ... - Reject follower writes if any statement in the batch is DML/COMMIT (not only stmts[0]). - COMMIT always emits empty-valued kvPairs for isDelete entries. - applyCommand updates LSM plus secondary B-tree/FTS/HNSW indexes (applyReplicatedPut/Delete) so follower index scans see replicated rows. - Tests: not-leader append, commit timeout, index apply unit, E2E index-backed SELECT on follower.	2026-07-30 21:14:46 +03:00
dimgigov	38c1c01841	feat(raft): classify writes, reject them on follower nodes	2026-07-30 19:07:58 +03:00
dimgigov	e8f9cbc5bb	fix: CREATE UNIQUE INDEX actually enforces uniqueness (and persists)	2026-07-30 17:13:29 +03:00
dimgigov	8d2d97ad94	feat(persist): standalone B-tree indexes survive restart	2026-07-30 16:53:25 +03:00
dimgigov	821b668d87	fix(persist): unnamed FTS/HNSW indexes survive restart (nameless replay DDL)	2026-07-30 16:27:36 +03:00
dimgigov	214b9cf346	fix: DROP INDEX/TABLE clean up FTS/HNSW indexes and their schema keys	2026-07-30 16:20:23 +03:00
dimgigov	ce6e7aa707	feat(persist): graphs survive restart (rebuild from backing tables)	2026-07-30 16:11:52 +03:00
dimgigov	214e44abd7	feat(persist): HNSW vector indexes survive restart	2026-07-30 16:05:25 +03:00
dimgigov	6703ca2b29	feat(persist): FTS indexes survive restart (schema key + restore replay)	2026-07-30 15:56:48 +03:00
dimgigov	62504aa348	docs(exec): document module layering after executor split	2026-07-30 15:17:31 +03:00
dimgigov	b282a59fb5	refactor(exec): slim executor.nim to dispatcher + hook wiring	2026-07-30 15:12:41 +03:00
dimgigov	a5976f648a	refactor(exec): extract IR plan execution into exec/plan_exec.nim	2026-07-30 15:02:34 +03:00
dimgigov	a147d4b620	refactor(exec): extract window functions into exec/window.nim	2026-07-30 14:52:17 +03:00
dimgigov	e618266325	refactor(exec): extract triggers/constraints into exec/triggers.nim	2026-07-30 14:43:49 +03:00
dimgigov	70c7297e33	refactor(exec): extract FK enforcement into exec/fk.nim	2026-07-30 14:35:15 +03:00
dimgigov	70b7ec7f08	refactor(exec): extract DML row operations into exec/dml.nim	2026-07-30 14:28:33 +03:00
dimgigov	e28c1f4896	refactor(exec): extract table scans into exec/scan.nim	2026-07-30 14:19:11 +03:00
dimgigov	f97e72314f	refactor(exec): extract RLS/privileges into exec/rls.nim	2026-07-30 14:11:08 +03:00
dimgigov	26475058bf	refactor(exec): extract AST->IR lowering into exec/lower.nim	2026-07-30 14:05:16 +03:00
dimgigov	b5e9636fd2	refactor(exec): extract expression evaluation + hybrid search into exec/eval.nim	2026-07-30 13:52:56 +03:00
dimgigov	2b8cc98348	refactor(exec): extract migration storage into exec/migrations.nim	2026-07-30 13:41:52 +03:00
dimgigov	2efcddba19	refactor(exec): extract param binding into exec/params.nim	2026-07-30 13:36:39 +03:00
dimgigov	46e3d7f51e	refactor(exec): extract join/vector helpers into exec/helpers.nim	2026-07-30 13:27:17 +03:00
dimgigov	08fb391ac1	refactor(exec): extract context management into exec/context.nim	2026-07-30 13:18:10 +03:00
dimgigov	2d09edd9f7	fix+refactor: soft keywords as identifiers, full test wiring, ORC crash docs ... - parser: clause keywords (header, format, status, user, csv, ...) now work as identifiers everywhere; IMPORT/EXPORT accept FORMAT csv/HEADER true - nimble test + CI run all 13 test suites (650 checks green) - ExecutionContext.registry is now {.cursor.} (breaks registry<->ctx cycle) - ORC crash reproduced and bisected (tests/orc_repro.py); ARC stays the MM	2026-07-30 13:11:28 +03:00
dimgigov	ed5a71913c	fix: harden exception handling, break ARC cycles, sync license/client ... Replace bare except: with CatchableError across storage, query, Raft, backup, and protocol code so Defects are not swallowed. Break uncollectable ARC cycles in server shard/gossip callbacks via Server-owned refs and cursor locals. Align package license with LICENSE (BSD-3-Clause), sync README version, and point test_all at the canonical clients/nim baradb client (parseConnectionString + aliases).	2026-07-23 01:07:51 +03:00
dimgigov	8db5cfe7e1	feat: harden storage, schema persistence, fair benches, fix wire crash ... Core storage: hash MemTable, WAL group commit, L0 compaction rebuild, reader-writer lock, and a global StorageGate so HTTP workers and TCP share the LSM safely under multi-thread access. Schema: durable CREATE/ALTER/DROP under _schema:tables:* with full LSM restore on open. Executor types/values/schema split into query/exec/. Wire protocol: switch default MM to ARC — ORC cycle collector segfaulted after ~20 async INSERTs. Fair multi-tier benchmarks (SQLite/HTTP/wire/PG) and honesty docs for mixed-tier comparisons.	2026-07-18 16:55:50 +03:00
dimgigov	42043f3946	v1.1.7: deep security & reliability audit — 33 bugs fixed ... Critical (5): - Reject empty JWT secret when authEnabled (server.nim) - Fix 2PC marking uncontacted participants as prepared/committed (disttxn.nim) - Fix Raft commit index calculation for even-sized clusters (raft.nim) - Fix REP/DISTTXN protocol auth bypass (server.nim) - Fix HTTP backup/restore path traversal (httpserver.nim) High (11): - Fix WAL write race with flush (lsm.nim) - Fix MVCC savepoint/rollback deep-copy writeSet (mvcc.nim) - Fix table mutation during deadlock iteration (mvcc.nim) - Fix LIMIT 0 returning all rows (executor.nim) - Fix COUNT(col) counting NULL values — 3 locations (executor.nim) - Fix EXISTS subquery lowering missing subqueryPlan (executor.nim) - Fix Raft appendEntries/applyCommitted array vs logical index (raft.nim) - Fix timing attacks on constantTimeCompare and SCRAM (auth.nim, scram.nim) - Fix B-tree leaf merge phantom separator key (btree.nim) - Fix SSL verifyPeer not applied to newContext (ssl.nim) - Fix sharding connectWithTimeout missing SO_ERROR check (sharding.nim) - Fix sync replication returning success on partial ack (replication.nim) - Fix WebSocket JWT expiration not validated (websocket.nim) Medium (13): - Fix writeSSTable partial file → tmp + atomic rename (lsm.nim) - Fix multi-CTE table loss (executor.nim) - Fix nl_to_sql DML restricted to superuser (executor.nim) - Fix unbounded plan cache — max 10000 (adaptive.nim) - Fix migration lock crash persistence — timestamp + stale detection (executor.nim) - Fix admin panel auth (httpserver.nim) - Fix MVCC unbounded txn tracking — prune in compactVersions (mvcc.nim) - Fix connection pool maxLifetime check (pool.nim) - Fix JWT JSON parser backslash escapes (auth.nim) - Fix substr(s, start) returning single char (udf.nim) - Fix loadSSTable minimum file-size check (lsm.nim) - Fix compaction mmap leak (compaction.nim) - Fix JSON injection in hybrid_search_filtered (executor.nim) Low (4): - Raft loadState logs error instead of silent discard - Replication healthCheck double-close fixed - Lexer readIdent double column counting fixed - WebSocket frame 32-bit overflow guard All 448 tests passing, 0 failures. Bump version to 1.1.7.	2026-05-29 14:17:41 +03:00
dimgigov	a37a62c69e	clients/nim-allographer/src/allographer/query_builder/models/baradb/baradb_exec.nim src/barabadb/core/mvcc.nim src/barabadb/query/codegen.nim src/barabadb/query/lexer.nim src/barabadb/query/parser.nim src/barabadb/query/udf.nim tests/test_all.nim	2026-05-25 18:56:02 +03:00
dimgigov	b25fea4d21	fix: all 5 bugs — reserved words, cloneForConnection, SQL injection, SHOW TABLES, getRowPlain	2026-05-21 22:51:04 +03:00
dimgigov	bb843b9a03	feat: migrate system + cross-DB engine + IMPORT/EXPORT syntax -- 22 files, client+server+docs	2026-05-21 19:32:14 +03:00
dimgigov	2d310a33a1	fix: resolve all 55 identified bugs across the codebase ... Comprehensive bug fix pass across all subsystems: - Critical: lexer infinite loop, WAL lock discipline, checkpoint safety, compaction verification, HMAC key truncation, healthCheck leak - High: MVCC lock safety, B-Tree delete rebalancing, Raft stale term handling, replication socket leaks and ack cleanup, sharding migration with old assignments, 2PC recovery, JWT/SCRAM auth fixes, wire protocol bounds checks - Medium: config error handling, MERGE parser completeness, IR/codegen correctness, UDF bounds checks, LIMIT cost accuracy, mmap safety - Low: timeout handling, float parsing edge cases, uint32 truncation, cache entry cleanup Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-21 10:59:51 +03:00
dimgigov	d4dc433e48	fix(query): preserve both sides of duplicate column names in JOIN ... When SELECT * joins tables with overlapping column names (e.g. 'id' in both users and orders), the right-side value was being dropped because irekStar filtered out dotted keys. - Add expandStarRow() that keeps bare columns and adds qualified aliases (u.id, o.id) only for duplicated names - Fix r.columns expansion to include all joined tables, qualifying duplicates from the right side	2026-05-21 09:44:31 +03:00
dimgigov	372e5cf627	fix(query,types): refactor Row to Value-typed map, fix IN list, nkPath, multi-table joins ... - Refactored Row from Table[string, string] to Table[string, Value] - Added Value operators: ==, !=, $, in for string interop - Fixed valueToString for vkNull to return '\\N' instead of '' - Fixed evalExpr irekField to return vkNull when field not found - Added IN (val1, val2, ...) parser support - Fixed nkPath column names in multi-table joins - Fixed LATERAL JOIN null padding when no matching rows - Added CREATE/DROP/USE/SHOW DATABASE parser support - Adapted all tests for new Value type	2026-05-20 23:22:14 +03:00
dimgigov	57d2908066	fix(executor,mvcc,tests): dedup aliases, fix txn delete, fix MIN/MAX null, add regression tests ... - Add isDelete flag to VersionedRecord to fix COMMIT treating empty values as DELETE. Previously version.xmax != TxnId(0) was never true for active transactions, so deletes inside transactions never removed rows. - Fix MIN/MAX aggregates skipping NULL: was checking for "\\N" instead of correct "\N" sentinel, so NULL values were never skipped. - Add duplicate alias deduplication in lowerSelect() and getSelectColumns(). Identical aliases now get _1, _2 suffixes (e.g. count(*), count(*)_1). - Add nil guard for expr.subqueryPlan in irekSubquery evalExpr handler. - Initialize txnManager in newExecutionContext to prevent segfault on BEGIN. - Add regression tests: DELETE in transaction, PK-only row survival, MIN/MAX NULL skipping.	2026-05-20 21:36:15 +03:00
dimgigov	edd7e728f9	fix(query): add correlated subquery support ... evalExpr was missing irekSubquery case — scalar subqueries in SELECT/WHERE fell through to else:return "" producing empty strings. Add: - irekSubquery handler in evalExpr that executes the subquery plan - outerRow + subqueryPlan fields on ExecutionContext for correlation - qualified column injection in execScan for correlated filter refs - collectCorrelatedTablesFromPlan to discover outer table references Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-19 23:16:31 +03:00
dimgigov	14596c9584	fix(adapter,executor,tests): harden nimforum adapter and fix transaction/aggregate bugs ... Adapter (baradb_sqlite.nim): - getRow now returns @[] for missing rows (db_sqlite compat) - insertID/nextId handle NULL (\N) values correctly - preprocessQuery replaces SQLite-specific syntax: TIMESTAMP->DATETIME, DATETIME('now')->current timestamp, INET->STRING - add dbBegin/dbCommit/dbRollback for transaction support Executor: - fix COMMIT treating empty values as DELETE (rows with only PK were lost) - fix MIN/MAX aggregates including NULL (\N) which skewed results - add numeric comparison for irNeq Tests: - stress_test: shared DB, 16 workers, correctness scan after run - nimforum_smoke_test: 10 tests including schema, transactions, iterators	2026-05-19 20:43:00 +03:00
dimgigov	d3b77174da	fix: use numeric comparison for MIN/MAX aggregates	2026-05-19 15:19:47 +03:00
dimgigov	47c4393a7e	Complete plan execution: gossip/raft bounds, deadlock fix, JSON escaping, memtable limits ... Remaining plan items: - 1.4 hybrid_search: JSON built with std/json instead of string concat - 2.3 lsm.nim: reject oversized memtable entries - 3.3 gossip.nim: bounds checking for idLen/nodeCount/hostLen - 3.4 raft.nim: max length caps for cmdLen/dataLen/logLen - 4.3 deadlock.nim: per-path seq stack instead of global parent table	2026-05-18 11:42:46 +03:00
dimgigov	967c0855a5	Bug fixes: composite PK, nl_to_sql sandbox, FK check, SQL injection, storage correctness ... Critical fixes: - Composite PK: execInsert + validateConstraints use all PK columns - nl_to_sql: non-SELECT SQL no longer executed directly during validation - FK check: removed O(N) scanMemTable fallback, uses db.get() with SSTables - exprToSql: nkIdent wrapped in quotes to prevent SQL injection - restoreSchema: try/except around tokenize/parse for crash resilience - recovery.nim: lastTxnId tracking + putUnsafe/deleteUnsafe without WAL - SCRAM: verifyClientProof length check + DefaultIterationCount restored Storage fixes: - lsm.nim: SSTable sort order fixed (ascending), close() flushes all memtables - compaction.nim: tombstones preserved during compaction - wal.nim: header written for empty existing files, readEntries checks magic - btree.nim: B+ tree leaf split keeps boundary key - bloom.nim: deserialize raises on short data - mmap.nim: bounds checks for adviseWillNeed/DontNeed + posix.close() Protocol fixes: - zerocopy.nim: readString bounds check - wire.nim: deserializeValue 32-bit underflow check - auth.nim: JWT JSON escaping for claims - server.nim: readUint32BE bounds check + specific exception handling - raft.nim: readData checks + specific exception handling Tests: - Added Composite Primary Key test suite (4 tests) Build: 0 warnings, 0 errors	2026-05-18 11:33:11 +03:00
dimgigov	1e38e29f25	fix(security): SQL injection in Python/JS clients + bare except + session leak ... - Fix SQL injection vulnerabilities in chat_history.py, langchain_store.py, rag_pipeline.py, baradb_langchain.js by switching to parameterized queries - Replace dangerous bare except: with except CatchableError:/ValueError: in llm.nim, embed.nim, cypher.nim, mcp/server.nim, executor.nim - Fix session variable leak in MCP handleVectorSearch/handleSchemaInspect - Build: 0 errors, all tests pass	2026-05-17 16:58:05 +03:00
dimgigov	a0c5ce8598	feat: Session 11 finale — similarity_nodes, node2vec, Cypher layer ... - similarity_nodes(graph, metric): Jaccard / Adamic-Adar similarity between all node pairs - node2vec_embed(graph, dimensions): Random-walk based graph embeddings - cypher(query): Translate Cypher MATCH...RETURN → BaraQL GRAPH_TABLE SQL - Fixed recursive lock deadlock in similarity functions - Fixed graph start node parsing (pattern variable vs numeric ID) - Added sequtils import to graph engine - All 340+ existing tests pass	2026-05-17 16:00:15 +03:00
dimgigov	e23b1d61d2	feat: Session 12 AI Agents & NL->SQL ... - src/barabadb/ai/llm.nim: LLM client for NL->SQL generation - Supports OpenAI-compatible and Ollama APIs - Configurable via BARADB_LLM_ENDPOINT, BARADB_LLM_MODEL, BARADB_LLM_API_KEY - extractSQL() parses SQL from LLM responses (handles markdown blocks) - Temperature 0.1 for deterministic SQL generation - nl_to_sql() SQL function: natural language -> SQL - Schema-aware prompt with table column definitions + indexes + RLS - Query validation layer: wraps generated SQL in LIMIT 0 subquery - Self-correction loop: on error, feeds error back to LLM for fix - Tenant-aware: respects current session variables - schema_prompt() SQL function: generates DDL + sample data + indexes - Returns full CREATE TABLE statement with column types and constraints - Includes up to 5 sample rows for context - Lists indexes, RLS policies, foreign keys - Perfect for feeding into LLM context - All 340+ existing tests pass	2026-05-17 15:38:14 +03:00
dimgigov	13bc17cfa8	feat: 10.1.4 Chunking + embedding pipeline ... - New modules: src/barabadb/ai/chunk.nim (text chunking) and embed.nim (HTTP embedding client) - chunk() SQL function: returns JSON array of chunks with configurable size/overlap - embed_text() SQL function: calls external embedding API (OpenAI/Ollama compatible) - Auto-embedding on INSERT: when VECTOR column is null but TEXT column is populated, generates embeddings via configured embedder - Configurable via env vars: BARADB_EMBED_ENDPOINT, BARADB_EMBED_MODEL, BARADB_EMBED_API_KEY - All 340+ existing tests pass	2026-05-17 15:26:24 +03:00
dimgigov	8a395225c0	feat: Session 10.3 MCP Server + Session 11 Graph Engine Deep Integration ... MCP Server (10.3): - STDIO JSON-RPC 2.0 transport with 3 AI tools: query, vector_search, schema_inspect - Multi-tenant session vars (tenant_id, user_id) with RLS support - Standalone binary: build/baramcp - Tested with all 3 tools + parameterized queries + vector search + schema inspect Graph Engine Deep Integration (Session 11): - CREATE GRAPH / DROP GRAPH DDL support - Graph engine wired to SQL executor via GRAPH_TABLE() function - 7 algorithms: BFS, DFS, PageRank, ShortestPath, Dijkstra, Louvain, Community - INSERT into _nodes/_edges tables auto-syncs with native Graph adjacency lists - Optional MATCH pattern, ALGORITHM, START, END, MAXDEPTH in GRAPH_TABLE syntax - All 340+ existing tests pass	2026-05-17 15:18:31 +03:00
dimgigov	67965ffa8b	feat(hybrid): Session 10.1.3 — Metadata pre-filtering in vector search ... - Populate HNSW metadata with all relational columns during INSERT and CREATE INDEX - Add doHybridSearchFiltered() using searchWithFilter() for HNSW pre-filtering - Add SQL function hybrid_search_filtered(table, vec_col, text_col, query, vector, k, filter_col, filter_val) - Enforce k-limit on both doHybridSearch and doHybridSearchFiltered results - 2 tests: tenant isolation + empty filter fallback	2026-05-17 13:41:30 +03:00
dimgigov	836d30d84a	feat(hybrid): Session 10.1 — Hybrid RAG Search with RRF reranking ... - Add searchEx() to vector engine returning metadata - Add reciprocalRankFusion(), doHybridSearch(), findRealIdByDocId() helpers - Add SQL functions: hybrid_search(), hybrid_search_ids(), rerank() - Fix CREATE INDEX HNSW docId to use hash(fullKey) matching INSERT - 5 tests covering hybrid search, ids, RRF ranking, rerank, missing indexes	2026-05-17 13:30:19 +03:00
dimgigov	2e0969245c	feat(fk): Foreign Key Enforcement — ON DELETE/UPDATE CASCADE, SET NULL, RESTRICT ... - Add tkRestrict token to lexer - Parse ON DELETE CASCADE/SET NULL/RESTRICT and ON UPDATE CASCADE/SET NULL/RESTRICT in both table-level and column-level FK constraints - Add fkOnDelete/fkOnUpdate to ColumnDef and ForeignKeyDef - Fix table-level FK constraint application (third pass after columns are created) - Implement enforceFkOnDelete, enforceFkOnUpdate, enforceFkOnChildUpdate helpers - Wire FK enforcement into DELETE and UPDATE execution paths - Add 9 regression tests covering all FK actions	2026-05-17 13:02:39 +03:00
dimgigov	dc4ad86ee1	Сесия 9: Production Hardening — prop tests (58 invariant-а), fuzz tests (35 сценария), thread safety (SharedLock ref)	2026-05-17 12:37:50 +03:00