209 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
dimgigov	132e8c7a31	Phase 5: Storage fuzz tests + WAL entryCount fix + SSTable overflow checks + compaction sizeBytes accuracy ... - WAL: entryCount is now accurate after restart by counting existing entries - LSM writeSSTable: adds uint32 overflow checks for key/value/entry counts - Compaction: sizeBytes now uses actual file size instead of rough guess - Fuzz tests: 14 new tests covering WAL truncation, LSM put/get/delete, recovery, scan, overwrites, SSTable roundtrip, corruption, tombstones	2026-05-18 12:34:26 +03:00
dimgigov	3f29c519d4	Storage hardening: btree size fix, WAL entryCount note, LSM put error handling ... - btree.nim: insertNonFull returns bool; size only increments for new keys - wal.nim: comment about entryCount reset on append - lsm.nim: recovery/put/delete/putUnsafe raise IOError if put fails after flush	2026-05-18 12:14:30 +03:00
dimgigov	8596cea548	Stabilization: replication semi-sync, MVCC aborted deleters, SSTable level, SCRAM fix ... P0-P1 fixes: - replication.nim: rmSync/rmSemiSync now wait for replica ACKs - mvcc.nim: aborted deleters no longer hide records (return true for visibility) - storage/lsm.nim: SSTable level persisted in header (version 2, backward compat) - protocol/auth.nim: removed insecure SCRAM raw hash fallback Build: 0 warnings, 0 errors. All tests pass.	2026-05-18 12:04:45 +03:00
dimgigov	47c4393a7e	Complete plan execution: gossip/raft bounds, deadlock fix, JSON escaping, memtable limits ... Remaining plan items: - 1.4 hybrid_search: JSON built with std/json instead of string concat - 2.3 lsm.nim: reject oversized memtable entries - 3.3 gossip.nim: bounds checking for idLen/nodeCount/hostLen - 3.4 raft.nim: max length caps for cmdLen/dataLen/logLen - 4.3 deadlock.nim: per-path seq stack instead of global parent table	2026-05-18 11:42:46 +03:00
dimgigov	967c0855a5	Bug fixes: composite PK, nl_to_sql sandbox, FK check, SQL injection, storage correctness ... Critical fixes: - Composite PK: execInsert + validateConstraints use all PK columns - nl_to_sql: non-SELECT SQL no longer executed directly during validation - FK check: removed O(N) scanMemTable fallback, uses db.get() with SSTables - exprToSql: nkIdent wrapped in quotes to prevent SQL injection - restoreSchema: try/except around tokenize/parse for crash resilience - recovery.nim: lastTxnId tracking + putUnsafe/deleteUnsafe without WAL - SCRAM: verifyClientProof length check + DefaultIterationCount restored Storage fixes: - lsm.nim: SSTable sort order fixed (ascending), close() flushes all memtables - compaction.nim: tombstones preserved during compaction - wal.nim: header written for empty existing files, readEntries checks magic - btree.nim: B+ tree leaf split keeps boundary key - bloom.nim: deserialize raises on short data - mmap.nim: bounds checks for adviseWillNeed/DontNeed + posix.close() Protocol fixes: - zerocopy.nim: readString bounds check - wire.nim: deserializeValue 32-bit underflow check - auth.nim: JWT JSON escaping for claims - server.nim: readUint32BE bounds check + specific exception handling - raft.nim: readData checks + specific exception handling Tests: - Added Composite Primary Key test suite (4 tests) Build: 0 warnings, 0 errors	2026-05-18 11:33:11 +03:00
dimgigov	a28c845476	Bump version to 1.1.4 across all components	2026-05-17 20:57:19 +03:00
dimgigov	3ac53ecda2	docs(readme): add AI-Native Data Platform features (Sessions 10-12) ... - Update Why BaraDB comparison table with Hybrid RAG, AI Agents, MCP Server, LangChain integration, Graph SQL, Cypher - Add Hybrid RAG Search section under Vector Engine - Add Graph SQL Integration section with CREATE GRAPH, GRAPH_TABLE, Cypher translation, and similarity/node2vec algorithms - Add new AI-Native Data Platform section covering: - Natural Language → SQL (nl_to_sql, schema_prompt) - Text Chunking & Auto-Embedding (chunk, embed_text) - MCP Server (Model Context Protocol over STDIO) - LangChain Vector Store (Python + JS) - Chat Message History with RLS - Update Project Structure with ai/ and mcp/ directories - Update Roadmap Progress with v1.1.2 features	2026-05-17 17:05:47 +03:00
dimgigov	1e38e29f25	fix(security): SQL injection in Python/JS clients + bare except + session leak ... - Fix SQL injection vulnerabilities in chat_history.py, langchain_store.py, rag_pipeline.py, baradb_langchain.js by switching to parameterized queries - Replace dangerous bare except: with except CatchableError:/ValueError: in llm.nim, embed.nim, cypher.nim, mcp/server.nim, executor.nim - Fix session variable leak in MCP handleVectorSearch/handleSchemaInspect - Build: 0 errors, all tests pass	2026-05-17 16:58:05 +03:00
dimgigov	c95bc4cd44	docs: synchronize documentation across all languages ... - Add mcp.md to: bg, fa, ru, tr, zh, ar - Add index.md to Bulgarian (bg) - Add 24 missing German (de) documentation files Translations for all supported languages now complete.	2026-05-17 16:29:28 +03:00
dimgigov	a5d34c001a	docs: add German (DE) documentation + update all docs for Sessions 10-12 ... New German documentation (docs/de/): - index.md, quickstart.md, installation.md - baraql.md, graph.md, vector.md, mcp.md Updated English documentation: - changelog.md: all Sessions 10-12 features - graph.md: SQL GRAPH_TABLE, CREATE GRAPH, all 8 algorithms, Cypher, similarity_nodes, node2vec - vector.md: hybrid RAG, chunk(), embed_text(), auto-embed, nl_to_sql(), schema_prompt() - baraql.md: new AI & Cross-Modal Functions section, updated keyword tables - mcp.md: MCP Server documentation (new file) - index.md: added German (DE) language link	2026-05-17 16:15:45 +03:00
dimgigov	e783215276	docs: mark all sessions 10-11-12 complete — plan finished	2026-05-17 16:01:20 +03:00
dimgigov	a0c5ce8598	feat: Session 11 finale — similarity_nodes, node2vec, Cypher layer ... - similarity_nodes(graph, metric): Jaccard / Adamic-Adar similarity between all node pairs - node2vec_embed(graph, dimensions): Random-walk based graph embeddings - cypher(query): Translate Cypher MATCH...RETURN → BaraQL GRAPH_TABLE SQL - Fixed recursive lock deadlock in similarity functions - Fixed graph start node parsing (pattern variable vs numeric ID) - Added sequtils import to graph engine - All 340+ existing tests pass	2026-05-17 16:00:15 +03:00
dimgigov	898f108963	docs: update PLAN.md - all sessions 10, 11, 12 marked complete	2026-05-17 15:39:20 +03:00
dimgigov	e23b1d61d2	feat: Session 12 AI Agents & NL->SQL ... - src/barabadb/ai/llm.nim: LLM client for NL->SQL generation - Supports OpenAI-compatible and Ollama APIs - Configurable via BARADB_LLM_ENDPOINT, BARADB_LLM_MODEL, BARADB_LLM_API_KEY - extractSQL() parses SQL from LLM responses (handles markdown blocks) - Temperature 0.1 for deterministic SQL generation - nl_to_sql() SQL function: natural language -> SQL - Schema-aware prompt with table column definitions + indexes + RLS - Query validation layer: wraps generated SQL in LIMIT 0 subquery - Self-correction loop: on error, feeds error back to LLM for fix - Tenant-aware: respects current session variables - schema_prompt() SQL function: generates DDL + sample data + indexes - Returns full CREATE TABLE statement with column types and constraints - Includes up to 5 sample rows for context - Lists indexes, RLS policies, foreign keys - Perfect for feeding into LLM context - All 340+ existing tests pass	2026-05-17 15:38:14 +03:00
dimgigov	80c3fee9de	feat: 10.2.3 ChatMessageHistory + 10.2.4 RAG pipeline example ... - clients/python/baradb/chat_history.py: BaraDBChatHistory class - Stores conversation threads in BaraDB with multi-tenant RLS - session_id + tenant_id + user_id isolation - Auto-creates table and index - Compatible with LangChain message format - examples/rag_pipeline.py: End-to-end RAG pipeline example - PDF/text ingestion -> chunking -> embedding -> BaraDB storage - Hybrid search with vector distance - LLM generation (OpenAI / Ollama) - Supports --file and --query modes - Configurable chunk size, overlap, top-k - PLAN.md: Updated all Session 10 tasks as complete	2026-05-17 15:30:20 +03:00
dimgigov	13bc17cfa8	feat: 10.1.4 Chunking + embedding pipeline ... - New modules: src/barabadb/ai/chunk.nim (text chunking) and embed.nim (HTTP embedding client) - chunk() SQL function: returns JSON array of chunks with configurable size/overlap - embed_text() SQL function: calls external embedding API (OpenAI/Ollama compatible) - Auto-embedding on INSERT: when VECTOR column is null but TEXT column is populated, generates embeddings via configured embedder - Configurable via env vars: BARADB_EMBED_ENDPOINT, BARADB_EMBED_MODEL, BARADB_EMBED_API_KEY - All 340+ existing tests pass	2026-05-17 15:26:24 +03:00
dimgigov	8a395225c0	feat: Session 10.3 MCP Server + Session 11 Graph Engine Deep Integration ... MCP Server (10.3): - STDIO JSON-RPC 2.0 transport with 3 AI tools: query, vector_search, schema_inspect - Multi-tenant session vars (tenant_id, user_id) with RLS support - Standalone binary: build/baramcp - Tested with all 3 tools + parameterized queries + vector search + schema inspect Graph Engine Deep Integration (Session 11): - CREATE GRAPH / DROP GRAPH DDL support - Graph engine wired to SQL executor via GRAPH_TABLE() function - 7 algorithms: BFS, DFS, PageRank, ShortestPath, Dijkstra, Louvain, Community - INSERT into _nodes/_edges tables auto-syncs with native Graph adjacency lists - Optional MATCH pattern, ALGORITHM, START, END, MAXDEPTH in GRAPH_TABLE syntax - All 340+ existing tests pass	2026-05-17 15:18:31 +03:00
dimgigov	55bc3e862a	feat(langchain): Session 10.2 — LangChain Vector Store (Python + JS) ... - BaraDBStore for Python: add_texts, similarity_search, max_marginal_relevance_search, delete - BaraDBStore for JS: addDocuments, addTexts, similaritySearch, maxMarginalRelevanceSearch, delete - Both use hybrid_search() / hybrid_search_filtered() for vector+FTS+RRF - Multi-tenant support via tenant_id session variable + metadata filter - Embedding function is injected by user (OpenAI, sentence-transformers, etc.) - MMR reranking for result diversity	2026-05-17 13:46:42 +03:00
dimgigov	67965ffa8b	feat(hybrid): Session 10.1.3 — Metadata pre-filtering in vector search ... - Populate HNSW metadata with all relational columns during INSERT and CREATE INDEX - Add doHybridSearchFiltered() using searchWithFilter() for HNSW pre-filtering - Add SQL function hybrid_search_filtered(table, vec_col, text_col, query, vector, k, filter_col, filter_val) - Enforce k-limit on both doHybridSearch and doHybridSearchFiltered results - 2 tests: tenant isolation + empty filter fallback	2026-05-17 13:41:30 +03:00
dimgigov	836d30d84a	feat(hybrid): Session 10.1 — Hybrid RAG Search with RRF reranking ... - Add searchEx() to vector engine returning metadata - Add reciprocalRankFusion(), doHybridSearch(), findRealIdByDocId() helpers - Add SQL functions: hybrid_search(), hybrid_search_ids(), rerank() - Fix CREATE INDEX HNSW docId to use hash(fullKey) matching INSERT - 5 tests covering hybrid search, ids, RRF ranking, rerank, missing indexes	2026-05-17 13:30:19 +03:00
dimgigov	f622c8f82c	docs(plan): archive old plan, create AI-Native Data Platform roadmap ... - PLAN.md → PLAN_old_3.md (sessions 1-9 completed) - New PLAN.md: Sessions 10-12 focused on: - Vector AI Native Integration (Hybrid RAG, LangChain, MCP Server) - Graph Engine Deep Integration (Native storage, Cypher, Algorithms) - AI Agents & NL → SQL - All features preserve multi-tenant RLS isolation and MVCC safety	2026-05-17 13:08:24 +03:00
dimgigov	d2ac485b2e	docs(plan): mark Phase 2 (JOIN) and Phase 3 (FK Enforcement) as complete	2026-05-17 13:03:15 +03:00
dimgigov	2e0969245c	feat(fk): Foreign Key Enforcement — ON DELETE/UPDATE CASCADE, SET NULL, RESTRICT ... - Add tkRestrict token to lexer - Parse ON DELETE CASCADE/SET NULL/RESTRICT and ON UPDATE CASCADE/SET NULL/RESTRICT in both table-level and column-level FK constraints - Add fkOnDelete/fkOnUpdate to ColumnDef and ForeignKeyDef - Fix table-level FK constraint application (third pass after columns are created) - Implement enforceFkOnDelete, enforceFkOnUpdate, enforceFkOnChildUpdate helpers - Wire FK enforcement into DELETE and UPDATE execution paths - Add 9 regression tests covering all FK actions	2026-05-17 13:02:39 +03:00
dimgigov	dc4ad86ee1	Сесия 9: Production Hardening — prop tests (58 invariant-а), fuzz tests (35 сценария), thread safety (SharedLock ref)	2026-05-17 12:37:50 +03:00
dimgigov	19fa760604	Week 4: Production Hardening — Thread Safety, Fuzz Tests, Property-Based Tests, NimForum Smoke Test ... Thread Safety: - Add lock to BTreeIndex (insert/get/scan/remove/contains) - Add lock to InvertedIndex (addDocument/removeDocument/search/bm25/tfidf/fuzzy/regex) - Add ctxLock to ExecutionContext for DDL/schema metadata, autoIncCounters, sequences - Deep-copy sessionVars in cloneForConnection() to isolate per-connection state - Fix activeConnections check-then-act race with Lock Wire Protocol Hardening: - Fix RangeDefect in deserializeValue for invalid FieldKind bytes - Fix IndexDefect in fkFloat32/fkFloat64 by reusing readUint32/readUint64 - Fix int32/int64 cast from uint32/uint64 to use cast[] instead of constructor - Fix parseHeader to validate MsgKind instead of holey enum cast Property-Based Tests (tests/prop_test.nim): - 8 invariants for evalExprValue: literal types, commutativity, identity, double negation, NULL propagation Wire Protocol Fuzz (tests/fuzz_test.nim): - 7 fuzz tests: random bytes deserialization, truncated buffers, MsgKind casts, parseHeader logic, message roundtrips NimForum Adapter Smoke Test (tests/nimforum_smoke_test.nim): - Full TCP server lifecycle with adapter CRUD and parameterized queries Test Results: 423 tests pass, 0 failures	2026-05-17 11:57:55 +03:00
dimgigov	6021bfcb10	Сесия 9: Седмица 3 — JOIN Performance ... - Добавен IRJoinStrategy enum (nestedLoop, hash, indexNestedLoop) - Hash Join: build hash table върху по-малката страна, probe от другата - Index Nested Loop Join: използва B-Tree индекс за point lookup - Query planner chooseJoinStrategy избира стратегия според наличие на индекс - LEFT/RIGHT/FULL JOIN fallback-ват към Nested Loop - PK индексите се игнорират за INL (само explicit индекси) - Benchmark: JOIN 10K ~115ms (Hash), ~90ms (Index NL) - 5 нови теста за join performance и planner избор Тестове: 316 — 0 failures Build: 0 warnings	2026-05-17 10:50:12 +03:00
dimgigov	9d71edafd4	Сесия 9: Седмици 1+2 — Stabilization + Type Safety ... - Поправени deficiencies #5-#8 (GROUP BY bare columns, aggregate names, sync client с blocking socket, thread-safe Lock в SyncClient) - Добавен BlockingClient (net.Socket + Lock) в clients/nim/src/baradb/client.nim - Обновен src/barabadb/client/client.nim със sync клиент без waitFor - Regression тестове за всички 10 deficiencies - Добавен valueKind в IRExpr за типова информация - evalExprValue връща Value discriminated union - Премахнати parseFloat евристики от irAdd/Sub/Mul/Div/Mod/Pow/Neg - INT+INT→INT, INT+FLOAT→FLOAT, FLOAT/INT→FLOAT - 12 нови теста за type safety Тестове: 311 — 0 failures Build: 0 warnings	2026-05-17 10:21:15 +03:00
dimgigov	4c6c72dc5b	docs(plan): mark 9.1.2 as complete	2026-05-17 03:50:57 +03:00
dimgigov	2e1d982b9f	fix(executor): include arguments in aggregate column names ... Issue #6: getSelectColumns() produced count() for count(*) and max() for max(id). Now uses exprToSql() on funcArgs to generate consistent names matching the actual SQL expression. Before: columns: @[count()], rows: @[{count(*): 2}] After: columns: @[count(*)], rows: @[{count(*): 2}]	2026-05-17 03:50:43 +03:00
dimgigov	cd62f98641	docs(plan): mark 9.1.1 as complete	2026-05-17 03:47:30 +03:00
dimgigov	07a22d305d	chore: fix all build warnings and add stabilization roadmap ... - Fix 3× ResultShadowed in cypher.nim, recovery.nim, shell.nim - Remove 6× UnusedImport in test files - Build is now clean: 0 warnings, 294 tests pass - Add 4-week stabilization plan to PLAN.md (Sessions 9.1–9.4)	2026-05-17 03:47:17 +03:00
dimgigov	5231724e77	fix(executor): handle subqueries in comparisons and per-row UPDATE expressions ... - Add missing nkSubquery case in lowerExpr so scalar subqueries (e.g. budget > (SELECT AVG(budget) FROM projects)) are lowered correctly instead of being treated as NULL literals. - Move UPDATE SET expression evaluation inside the row loop so column references (e.g. salary + 5000) are resolved against the current row, fixing type validation failures. - Make irAdd/irSub/irMul/irDiv return integer strings when the result is a whole number, avoiding spurious float output for INT columns (e.g. MERGE UPDATE qty = 100 + 50 now yields 150). These fix regressions introduced by switching NULL representation from empty string to \N, which exposed the pre-existing bugs above. Refs: issue #9, issue #10	2026-05-17 03:29:25 +03:00
dimgigov	073ec41652	fix: issue #9 reserved keyword 'key' + issue #10 empty string treated as NULL ... Issue 9: Add backtick-quoted identifier support in lexer.nim - Users can now write `key` to use reserved keywords as column/table names Issue 10: Use \N as internal NULL marker instead of empty string - isNull() now checks for \N instead of value.len == 0 - Empty strings can now be stored in NOT NULL columns - NULL values properly distinguished from empty strings in storage, wire protocol, and HTTP JSON responses	2026-05-17 01:59:25 +03:00
dimgigov	58b7598155	fix(dev): streamline build config for new contributors ... - Add --path:src and --threads:on to nim.cfg so tests compile without manual flags - Add tests/config.nims for automatic path resolution in test directory - Simplify nimble task definitions (remove redundant flags now in nim.cfg) - Fix Dockerfile.source comment: remove obsolete nimmax reference This makes 'nim c tests/test_all.nim' and 'nimble test' work out of the box.	2026-05-16 03:20:43 +03:00
dimgigov	3fce374aba	fix(tests): correct vkInt -> vkInt64 in test_minimal, remove tracked binary from git	2026-05-16 03:13:19 +03:00
dimgigov	9756d93cb8	docs: add deficiency audit and ID generators plan	2026-05-16 03:07:11 +03:00
dimgigov	652ed1b477	feat: add ID generators, fix 6 forum-discovered deficiencies ... ID Generators: - AUTO_INCREMENT for INTEGER columns - SERIAL / BIGSERIAL as syntactic sugar - gen_random_uuid() / uuid() for UUID v4 generation - nextval() / currval() for sequence support - snowflake_id(node_id) for distributed 64-bit IDs - RETURNING clause for INSERT Deficiency Fixes: - Comma join: FROM t1, t2 now works as implicit CROSS JOIN - DEFAULT in restoreSchema: defaults survive server restart - Duplicate column names: t.id instead of id in joins - Empty result sets: always send column metadata to client - GROUP BY non-agg columns: return first row value instead of empty - Aggregate column names: count(*) instead of count() Bug fix: PK uniqueness check now uses correct key format	2026-05-16 02:57:30 +03:00
dimgigov	406dcf4f4e	docs(ormin): clarify transaction and returning support ... - Top-level BEGIN/COMMIT/ROLLBACK work (server-side) - SAVEPOINT (nested tx) does not work — BaraDB server limitation - RETURNING parsed but ignored by BaraDB executor — server limitation	2026-05-15 22:56:45 +03:00
dimgigov	4ac147dc6c	Add Docker support for Ormin + BaraDB dev stack ... - docker-compose.ormin.yml: BaraDB + Ormin dev environment - clients/nim/ormin/docker/Dockerfile: dev image with Nim 2.2.10 - clients/nim/ormin/docker/entrypoint.sh: auto-install deps - docker-compose.test.yml: add Ormin compile test	2026-05-15 22:23:45 +03:00
dimgigov	4e54075078	Add Ormin ORM support for BaraDB (Nim client)	2026-05-15 21:49:08 +03:00
dimgigov	2e945c1dcb	release: bump version to 1.1.2 (forum-tested, nested SQL fixes)	2026-05-15 18:34:47 +03:00
dimgigov	2fbafb11bd	docs: add GitHub stars badge and thank you note	2026-05-15 18:30:10 +03:00
dimgigov	3453d3cae7	feat: add forum Nim client improvements and SQLite adaptor	2026-05-15 18:12:01 +03:00
dimgigov	00e9ea7484	Merge branch 'main' of ssh://git.invoicing.top:222/baraba/Baradb	2026-05-15 18:01:25 +03:00
dimgigov	c5751b4a0c	fix: clean up compiler warnings and unused variables	2026-05-15 18:01:11 +03:00
dimgigov	bf374b28f7	chore: ignore docker export tarballs	2026-05-15 17:46:53 +03:00
dimgigov	92e6b99df2	Merge pull request 'feat: subquery IN/NOT IN, datetime functions, scanAll refactor, INSERT/UPDATE eval improvements' (#1) from master into main ... Reviewed-on: #1	2026-05-15 14:35:06 +00:00
dimgigov	f8d466d8f3	feat: subquery IN/NOT IN, datetime functions, scanAll refactor, INSERT/UPDATE eval improvements	2026-05-15 17:33:54 +03:00
dimgigov	40616ad3c0	Update repository URLs from GitHub to Codeberg	2026-05-15 12:11:00 +03:00
dimgigov	8c29e700c7	Revert "Update repository URLs from GitHub to Codeberg" ... This reverts commit 956066631e.	2026-05-15 12:10:36 +03:00