Commit Graph

228 Commits

Author SHA1 Message Date
dimgigov 8a385cba6a docs(readme): clarify this is a BaraDB fork of nim-allographer
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-21 15:54:56 +03:00
dimgigov 6bfc5b3a3c feat(clients): add nim-allographer client with BaraDB driver support
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Pure Nim wire protocol client (async + sync, no C FFI)
- Query Builder: fluent API, SQL generator, execution, transactions
- Schema Builder: create/alter/drop tables, column operations
- Connection pool with aging and timeout handling
- ORM integration via compile-time DB_BARADB switch
- Tests: test_open, test_query
- Documentation: README.md, PLAN_BARADB.md
2026-05-21 15:49:11 +03:00
dimgigov 2d310a33a1 fix: resolve all 55 identified bugs across the codebase
Comprehensive bug fix pass across all subsystems:
- Critical: lexer infinite loop, WAL lock discipline, checkpoint safety,
  compaction verification, HMAC key truncation, healthCheck leak
- High: MVCC lock safety, B-Tree delete rebalancing, Raft stale term
  handling, replication socket leaks and ack cleanup, sharding migration
  with old assignments, 2PC recovery, JWT/SCRAM auth fixes, wire protocol
  bounds checks
- Medium: config error handling, MERGE parser completeness, IR/codegen
  correctness, UDF bounds checks, LIMIT cost accuracy, mmap safety
- Low: timeout handling, float parsing edge cases, uint32 truncation,
  cache entry cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-21 10:59:51 +03:00
dimgigov 7fc6adbe47 chore: remove old planning and bug report files
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
These files are outdated and only cause confusion.
Keep only the current PLAN.md.
2026-05-21 09:47:40 +03:00
dimgigov 072e7a5d13 test(join): add test for duplicate columns in SELECT *
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Verify that JOIN with overlapping column names preserves
both left (bare) and right (qualified) values.
2026-05-21 09:44:32 +03:00
dimgigov d4dc433e48 fix(query): preserve both sides of duplicate column names in JOIN
When SELECT * joins tables with overlapping column names
(e.g. 'id' in both users and orders), the right-side value
was being dropped because irekStar filtered out dotted keys.

- Add expandStarRow() that keeps bare columns and adds
  qualified aliases (u.id, o.id) only for duplicated names
- Fix r.columns expansion to include all joined tables,
  qualifying duplicates from the right side
2026-05-21 09:44:31 +03:00
dimgigov 1ff6fabaff fix(backup): resolve relative SSTable paths in incremental backup
MANIFEST stores relative paths like 'sstables/1.sst'.
The old code checked fileExists against CWD instead of dataDir,
silently omitting SSTables when run from another directory.
Now resolves paths relative to dataDir.
2026-05-21 09:44:28 +03:00
dimgigov 4a83eb6783 docs: update distributed/backup/schema/storage for multi-database
- Add multi-database limitations to distributed docs
- Fix backup commands and add multi-database backup guidance
- Add SQL migration section to schema docs
- Update storage examples to use per-database paths
2026-05-21 09:44:25 +03:00
dimgigov a526a3aef9 docs: remove non-English/Bulgarian language docs, fix links
Remove ar, de, fa, ru, tr, zh documentation folders.
Update docs/index.md and docs/bg/index.md to only list en/bg.
Update README badge from 7 to 2 languages.
Update changelog entry for Bulgarian docs.
2026-05-21 09:44:17 +03:00
dimgigov 372e5cf627 fix(query,types): refactor Row to Value-typed map, fix IN list, nkPath, multi-table joins
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Refactored Row from Table[string, string] to Table[string, Value]
- Added Value operators: ==, !=, $, in for string interop
- Fixed valueToString for vkNull to return '\\N' instead of ''
- Fixed evalExpr irekField to return vkNull when field not found
- Added IN (val1, val2, ...) parser support
- Fixed nkPath column names in multi-table joins
- Fixed LATERAL JOIN null padding when no matching rows
- Added CREATE/DROP/USE/SHOW DATABASE parser support
- Adapted all tests for new Value type
2026-05-20 23:22:14 +03:00
dimgigov 57d2908066 fix(executor,mvcc,tests): dedup aliases, fix txn delete, fix MIN/MAX null, add regression tests
- Add isDelete flag to VersionedRecord to fix COMMIT treating empty values
  as DELETE. Previously version.xmax != TxnId(0) was never true for active
  transactions, so deletes inside transactions never removed rows.
- Fix MIN/MAX aggregates skipping NULL: was checking for "\\N" instead
  of correct "\N" sentinel, so NULL values were never skipped.
- Add duplicate alias deduplication in lowerSelect() and getSelectColumns().
  Identical aliases now get _1, _2 suffixes (e.g. count(*), count(*)_1).
- Add nil guard for expr.subqueryPlan in irekSubquery evalExpr handler.
- Initialize txnManager in newExecutionContext to prevent segfault on BEGIN.
- Add regression tests: DELETE in transaction, PK-only row survival,
  MIN/MAX NULL skipping.
2026-05-20 21:36:15 +03:00
dimgigov edd7e728f9 fix(query): add correlated subquery support
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
evalExpr was missing irekSubquery case — scalar subqueries in SELECT/WHERE
fell through to else:return "" producing empty strings. Add:
- irekSubquery handler in evalExpr that executes the subquery plan
- outerRow + subqueryPlan fields on ExecutionContext for correlation
- qualified column injection in execScan for correlated filter refs
- collectCorrelatedTablesFromPlan to discover outer table references

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-19 23:16:31 +03:00
dimgigov 14596c9584 fix(adapter,executor,tests): harden nimforum adapter and fix transaction/aggregate bugs
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Adapter (baradb_sqlite.nim):
- getRow now returns @[] for missing rows (db_sqlite compat)
- insertID/nextId handle NULL (\N) values correctly
- preprocessQuery replaces SQLite-specific syntax:
  TIMESTAMP->DATETIME, DATETIME('now')->current timestamp, INET->STRING
- add dbBegin/dbCommit/dbRollback for transaction support

Executor:
- fix COMMIT treating empty values as DELETE (rows with only PK were lost)
- fix MIN/MAX aggregates including NULL (\N) which skewed results
- add numeric comparison for irNeq

Tests:
- stress_test: shared DB, 16 workers, correctness scan after run
- nimforum_smoke_test: 10 tests including schema, transactions, iterators
2026-05-19 20:43:00 +03:00
dimgigov d3b77174da fix: use numeric comparison for MIN/MAX aggregates
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-19 15:19:47 +03:00
dimgigov cd46edcb67 Bump version to 1.1.6; fix storage bugs, tests, CI pipeline and Docker config
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Storage fixes:
- Fix bloom filter hash2 OverflowDefect (uint64 instead of int Hash)
- Fix MemTable.put size calculation on overwrite (was leaking size)
- Fix fuzz_test duplicate-key issues in LSM delete and SSTable tests

Test fixes:
- Fix stress_test.nim: replace deprecated threadpool with std/threads
- Fix fuzz_test.nim: add missing imports (tables, algorithm, sets)
- Fix fuzz_test.nim: var sst for close() compatibility
- Remove unused imports from test_all.nim and bench_all.nim

Docker / CI fixes:
- Fix Dockerfile.source: invalid nimlang/nim:2.2.10-alpine tag → 2.2.10 + ubuntu:24.04 runtime
- Fix Dockerfile.source healthcheck (--spider → -qO- for 200 OK)
- Fix Dockerfile run comment ports (9470/9471 → 9912/9913)
- Fix scripts/docker-run.sh healthcheck port (9470 → 9912)
- Add ARG BUILD_DATE/VCS_REF to Dockerfile for docker-build.sh
- Update all version strings 1.1.4 → 1.1.6 across nimble/docker/source/docs
2026-05-19 12:13:33 +03:00
dimgigov 343f479127 docs(bg,en): fix ports and WebSocket example for new server model
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Update architecture diagram to include WebSocket server (port 9913)
- Fix HTTP port from 9470 to 9912 (TCP + 440) in protocol docs
- Fix WebSocket port from 9471 to 9913 (TCP + 441) in protocol docs
- Fix WebSocket Nim example: remove non-existent onMessage API and
  unsafe cast[string](data), use correct WsServer API with safe helpers
- Clarify TCP_NODELAY is also handled internally by hunos (HTTP)
- Note that bytesToString/stringToBytes are wire-protocol helpers
2026-05-18 19:41:21 +03:00
dimgigov 8afa998516 feat: rate limiting, shared DB instance, TCP_NODELAY fix
- Add rate limiter to TCP and HTTP servers
- Share single LSMTree between TCP and HTTP servers
- Replace unsafe cast[string]/cast[seq[byte]] with safe helpers
- Stabilize gossip UDP socket with exponential backoff
- Fix TCP_NODELAY via low-level setSockOptInt(IPPROTO_TCP)
  (asyncnet.setSockOpt(OptNoDelay) uses SOL_SOCKET and fails)
- Apply TCP_NODELAY to server and websocket sockets
2026-05-18 19:33:40 +03:00
dimgigov 7e6a45e6b7 Стабилизация на storage слоя — 6 фази
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Фаза 1: SSTable Integrity
- SSTable v3 формат с CRC32 footer (data/index/bloom CRC)
- Нов модул storage/crc32.nim (zero-dep, IEEE 802.3)
- verifySSTable() — проверка на magic, version, CRC
- loadSSTable strict mode — отхвърля корумпирани файлове
- newLSMTree логва [WARN] при corrupt SSTables

Фаза 2: baradb repair
- Нов модул tools/repair.nim
- Сканира SSTables, проверява CRC, премества битите в corrupt/
- WAL replay (CrashRecovery) за възстановяване на данни
- CLI: ./baradadb repair --data-dir=... [--dry-run]

Фаза 3: MANIFEST File
- Atomic write MANIFEST (tmp + rename)
- newLSMTree зарежда от MANIFEST, fallback към walkDir
- checkStorageConsistency() — orphan/missing detection
- flushUnsafe и compaction записват MANIFEST

Фаза 4: WAL Rotation & Incremental Backup
- WAL segment rotation при 64MB (wal_archive/wal.000NNN.log)
- maybeRotate() на всеки 1000 записа + при flush
- backup incremental — архивира MANIFEST + SSTables + WAL
- CRC verification на SSTables преди архивиране

Фаза 5: Online Consistent Backup
- checkpoint() — freeze memtable + flush + WAL rotate
- ./baradadb checkpoint — offline consistent snapshot
- backup --online — checkpoint + incremental backup

Фаза 6: SSTable Version Migration
- SSTable.fileVersion поле
- listLegacySSTables() — намира v1/v2 файлове
- migrateSSTable() — пренаписва към v3 (tmp + rename)
- ./baradadb migrate [--dry-run] — offline migration

Документация:
- Обновени docs/en/backup.md, docs/bg/backup.md
- Обновени docs/en/storage.md, docs/bg/storage.md
- Добавени тестове в tests/test_all.nim
2026-05-18 15:04:49 +03:00
dimgigov cf2aba104f Phase 5 complete: B-tree property tests + benchmark regression suite
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- prop_test.nim: 11 new property-based B-Tree invariants
  (size accuracy, get roundtrip, scan ordering, range correctness,
   contains, remove, large order, duplicates, empty tree, interleaved ops)
- bench_all.nim: JSON-based benchmark result tracking with regression
  comparison against previous runs; each benchmark reports ops/sec
  delta vs last run
2026-05-18 12:52:44 +03:00
dimgigov 132e8c7a31 Phase 5: Storage fuzz tests + WAL entryCount fix + SSTable overflow checks + compaction sizeBytes accuracy
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- WAL: entryCount is now accurate after restart by counting existing entries
- LSM writeSSTable: adds uint32 overflow checks for key/value/entry counts
- Compaction: sizeBytes now uses actual file size instead of rough guess
- Fuzz tests: 14 new tests covering WAL truncation, LSM put/get/delete,
  recovery, scan, overwrites, SSTable roundtrip, corruption, tombstones
2026-05-18 12:34:26 +03:00
dimgigov 3f29c519d4 Storage hardening: btree size fix, WAL entryCount note, LSM put error handling
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- btree.nim: insertNonFull returns bool; size only increments for new keys
- wal.nim: comment about entryCount reset on append
- lsm.nim: recovery/put/delete/putUnsafe raise IOError if put fails after flush
2026-05-18 12:14:30 +03:00
dimgigov 8596cea548 Stabilization: replication semi-sync, MVCC aborted deleters, SSTable level, SCRAM fix
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
P0-P1 fixes:
- replication.nim: rmSync/rmSemiSync now wait for replica ACKs
- mvcc.nim: aborted deleters no longer hide records (return true for visibility)
- storage/lsm.nim: SSTable level persisted in header (version 2, backward compat)
- protocol/auth.nim: removed insecure SCRAM raw hash fallback

Build: 0 warnings, 0 errors. All tests pass.
2026-05-18 12:04:45 +03:00
dimgigov 47c4393a7e Complete plan execution: gossip/raft bounds, deadlock fix, JSON escaping, memtable limits
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Remaining plan items:
- 1.4 hybrid_search: JSON built with std/json instead of string concat
- 2.3 lsm.nim: reject oversized memtable entries
- 3.3 gossip.nim: bounds checking for idLen/nodeCount/hostLen
- 3.4 raft.nim: max length caps for cmdLen/dataLen/logLen
- 4.3 deadlock.nim: per-path seq stack instead of global parent table
2026-05-18 11:42:46 +03:00
dimgigov 967c0855a5 Bug fixes: composite PK, nl_to_sql sandbox, FK check, SQL injection, storage correctness
Critical fixes:
- Composite PK: execInsert + validateConstraints use all PK columns
- nl_to_sql: non-SELECT SQL no longer executed directly during validation
- FK check: removed O(N) scanMemTable fallback, uses db.get() with SSTables
- exprToSql: nkIdent wrapped in quotes to prevent SQL injection
- restoreSchema: try/except around tokenize/parse for crash resilience
- recovery.nim: lastTxnId tracking + putUnsafe/deleteUnsafe without WAL
- SCRAM: verifyClientProof length check + DefaultIterationCount restored

Storage fixes:
- lsm.nim: SSTable sort order fixed (ascending), close() flushes all memtables
- compaction.nim: tombstones preserved during compaction
- wal.nim: header written for empty existing files, readEntries checks magic
- btree.nim: B+ tree leaf split keeps boundary key
- bloom.nim: deserialize raises on short data
- mmap.nim: bounds checks for adviseWillNeed/DontNeed + posix.close()

Protocol fixes:
- zerocopy.nim: readString bounds check
- wire.nim: deserializeValue 32-bit underflow check
- auth.nim: JWT JSON escaping for claims
- server.nim: readUint32BE bounds check + specific exception handling
- raft.nim: readData checks + specific exception handling

Tests:
- Added Composite Primary Key test suite (4 tests)

Build: 0 warnings, 0 errors
2026-05-18 11:33:11 +03:00
dimgigov a28c845476 Bump version to 1.1.4 across all components
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 20:57:19 +03:00
dimgigov 3ac53ecda2 docs(readme): add AI-Native Data Platform features (Sessions 10-12)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- Update Why BaraDB comparison table with Hybrid RAG, AI Agents,
  MCP Server, LangChain integration, Graph SQL, Cypher
- Add Hybrid RAG Search section under Vector Engine
- Add Graph SQL Integration section with CREATE GRAPH, GRAPH_TABLE,
  Cypher translation, and similarity/node2vec algorithms
- Add new AI-Native Data Platform section covering:
  - Natural Language → SQL (nl_to_sql, schema_prompt)
  - Text Chunking & Auto-Embedding (chunk, embed_text)
  - MCP Server (Model Context Protocol over STDIO)
  - LangChain Vector Store (Python + JS)
  - Chat Message History with RLS
- Update Project Structure with ai/ and mcp/ directories
- Update Roadmap Progress with v1.1.2 features
2026-05-17 17:05:47 +03:00
dimgigov 1e38e29f25 fix(security): SQL injection in Python/JS clients + bare except + session leak
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Fix SQL injection vulnerabilities in chat_history.py, langchain_store.py,
  rag_pipeline.py, baradb_langchain.js by switching to parameterized queries
- Replace dangerous bare except: with except CatchableError:/ValueError:
  in llm.nim, embed.nim, cypher.nim, mcp/server.nim, executor.nim
- Fix session variable leak in MCP handleVectorSearch/handleSchemaInspect
- Build: 0 errors, all tests pass
2026-05-17 16:58:05 +03:00
dimgigov c95bc4cd44 docs: synchronize documentation across all languages
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- Add mcp.md to: bg, fa, ru, tr, zh, ar
- Add index.md to Bulgarian (bg)
- Add 24 missing German (de) documentation files

Translations for all supported languages now complete.
2026-05-17 16:29:28 +03:00
dimgigov a5d34c001a docs: add German (DE) documentation + update all docs for Sessions 10-12
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
New German documentation (docs/de/):
- index.md, quickstart.md, installation.md
- baraql.md, graph.md, vector.md, mcp.md

Updated English documentation:
- changelog.md: all Sessions 10-12 features
- graph.md: SQL GRAPH_TABLE, CREATE GRAPH, all 8 algorithms, Cypher, similarity_nodes, node2vec
- vector.md: hybrid RAG, chunk(), embed_text(), auto-embed, nl_to_sql(), schema_prompt()
- baraql.md: new AI & Cross-Modal Functions section, updated keyword tables
- mcp.md: MCP Server documentation (new file)
- index.md: added German (DE) language link
2026-05-17 16:15:45 +03:00
dimgigov e783215276 docs: mark all sessions 10-11-12 complete — plan finished
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 16:01:20 +03:00
dimgigov a0c5ce8598 feat: Session 11 finale — similarity_nodes, node2vec, Cypher layer
- similarity_nodes(graph, metric): Jaccard / Adamic-Adar similarity between all node pairs
- node2vec_embed(graph, dimensions): Random-walk based graph embeddings
- cypher(query): Translate Cypher MATCH...RETURN → BaraQL GRAPH_TABLE SQL
- Fixed recursive lock deadlock in similarity functions
- Fixed graph start node parsing (pattern variable vs numeric ID)
- Added sequtils import to graph engine
- All 340+ existing tests pass
2026-05-17 16:00:15 +03:00
dimgigov 898f108963 docs: update PLAN.md - all sessions 10, 11, 12 marked complete
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 15:39:20 +03:00
dimgigov e23b1d61d2 feat: Session 12 AI Agents & NL->SQL
- src/barabadb/ai/llm.nim: LLM client for NL->SQL generation
  - Supports OpenAI-compatible and Ollama APIs
  - Configurable via BARADB_LLM_ENDPOINT, BARADB_LLM_MODEL, BARADB_LLM_API_KEY
  - extractSQL() parses SQL from LLM responses (handles markdown blocks)
  - Temperature 0.1 for deterministic SQL generation

- nl_to_sql() SQL function: natural language -> SQL
  - Schema-aware prompt with table column definitions + indexes + RLS
  - Query validation layer: wraps generated SQL in LIMIT 0 subquery
  - Self-correction loop: on error, feeds error back to LLM for fix
  - Tenant-aware: respects current session variables

- schema_prompt() SQL function: generates DDL + sample data + indexes
  - Returns full CREATE TABLE statement with column types and constraints
  - Includes up to 5 sample rows for context
  - Lists indexes, RLS policies, foreign keys
  - Perfect for feeding into LLM context

- All 340+ existing tests pass
2026-05-17 15:38:14 +03:00
dimgigov 80c3fee9de feat: 10.2.3 ChatMessageHistory + 10.2.4 RAG pipeline example
- clients/python/baradb/chat_history.py: BaraDBChatHistory class
  - Stores conversation threads in BaraDB with multi-tenant RLS
  - session_id + tenant_id + user_id isolation
  - Auto-creates table and index
  - Compatible with LangChain message format

- examples/rag_pipeline.py: End-to-end RAG pipeline example
  - PDF/text ingestion -> chunking -> embedding -> BaraDB storage
  - Hybrid search with vector distance
  - LLM generation (OpenAI / Ollama)
  - Supports --file and --query modes
  - Configurable chunk size, overlap, top-k

- PLAN.md: Updated all Session 10 tasks as complete
2026-05-17 15:30:20 +03:00
dimgigov 13bc17cfa8 feat: 10.1.4 Chunking + embedding pipeline
- New modules: src/barabadb/ai/chunk.nim (text chunking) and embed.nim (HTTP embedding client)
- chunk() SQL function: returns JSON array of chunks with configurable size/overlap
- embed_text() SQL function: calls external embedding API (OpenAI/Ollama compatible)
- Auto-embedding on INSERT: when VECTOR column is null but TEXT column is populated,
  generates embeddings via configured embedder
- Configurable via env vars: BARADB_EMBED_ENDPOINT, BARADB_EMBED_MODEL, BARADB_EMBED_API_KEY
- All 340+ existing tests pass
2026-05-17 15:26:24 +03:00
dimgigov 8a395225c0 feat: Session 10.3 MCP Server + Session 11 Graph Engine Deep Integration
MCP Server (10.3):
- STDIO JSON-RPC 2.0 transport with 3 AI tools: query, vector_search, schema_inspect
- Multi-tenant session vars (tenant_id, user_id) with RLS support
- Standalone binary: build/baramcp
- Tested with all 3 tools + parameterized queries + vector search + schema inspect

Graph Engine Deep Integration (Session 11):
- CREATE GRAPH / DROP GRAPH DDL support
- Graph engine wired to SQL executor via GRAPH_TABLE() function
- 7 algorithms: BFS, DFS, PageRank, ShortestPath, Dijkstra, Louvain, Community
- INSERT into _nodes/_edges tables auto-syncs with native Graph adjacency lists
- Optional MATCH pattern, ALGORITHM, START, END, MAXDEPTH in GRAPH_TABLE syntax
- All 340+ existing tests pass
2026-05-17 15:18:31 +03:00
dimgigov 55bc3e862a feat(langchain): Session 10.2 — LangChain Vector Store (Python + JS)
- BaraDBStore for Python: add_texts, similarity_search, max_marginal_relevance_search, delete
- BaraDBStore for JS: addDocuments, addTexts, similaritySearch, maxMarginalRelevanceSearch, delete
- Both use hybrid_search() / hybrid_search_filtered() for vector+FTS+RRF
- Multi-tenant support via tenant_id session variable + metadata filter
- Embedding function is injected by user (OpenAI, sentence-transformers, etc.)
- MMR reranking for result diversity
2026-05-17 13:46:42 +03:00
dimgigov 67965ffa8b feat(hybrid): Session 10.1.3 — Metadata pre-filtering in vector search
- Populate HNSW metadata with all relational columns during INSERT and CREATE INDEX
- Add doHybridSearchFiltered() using searchWithFilter() for HNSW pre-filtering
- Add SQL function hybrid_search_filtered(table, vec_col, text_col, query, vector, k, filter_col, filter_val)
- Enforce k-limit on both doHybridSearch and doHybridSearchFiltered results
- 2 tests: tenant isolation + empty filter fallback
2026-05-17 13:41:30 +03:00
dimgigov 836d30d84a feat(hybrid): Session 10.1 — Hybrid RAG Search with RRF reranking
- Add searchEx() to vector engine returning metadata
- Add reciprocalRankFusion(), doHybridSearch(), findRealIdByDocId() helpers
- Add SQL functions: hybrid_search(), hybrid_search_ids(), rerank()
- Fix CREATE INDEX HNSW docId to use hash(fullKey) matching INSERT
- 5 tests covering hybrid search, ids, RRF ranking, rerank, missing indexes
2026-05-17 13:30:19 +03:00
dimgigov f622c8f82c docs(plan): archive old plan, create AI-Native Data Platform roadmap
- PLAN.md → PLAN_old_3.md (sessions 1-9 completed)
- New PLAN.md: Sessions 10-12 focused on:
  - Vector AI Native Integration (Hybrid RAG, LangChain, MCP Server)
  - Graph Engine Deep Integration (Native storage, Cypher, Algorithms)
  - AI Agents & NL → SQL
- All features preserve multi-tenant RLS isolation and MVCC safety
2026-05-17 13:08:24 +03:00
dimgigov d2ac485b2e docs(plan): mark Phase 2 (JOIN) and Phase 3 (FK Enforcement) as complete 2026-05-17 13:03:15 +03:00
dimgigov 2e0969245c feat(fk): Foreign Key Enforcement — ON DELETE/UPDATE CASCADE, SET NULL, RESTRICT
- Add tkRestrict token to lexer
- Parse ON DELETE CASCADE/SET NULL/RESTRICT and ON UPDATE CASCADE/SET NULL/RESTRICT
  in both table-level and column-level FK constraints
- Add fkOnDelete/fkOnUpdate to ColumnDef and ForeignKeyDef
- Fix table-level FK constraint application (third pass after columns are created)
- Implement enforceFkOnDelete, enforceFkOnUpdate, enforceFkOnChildUpdate helpers
- Wire FK enforcement into DELETE and UPDATE execution paths
- Add 9 regression tests covering all FK actions
2026-05-17 13:02:39 +03:00
dimgigov dc4ad86ee1 Сесия 9: Production Hardening — prop tests (58 invariant-а), fuzz tests (35 сценария), thread safety (SharedLock ref)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 12:37:50 +03:00
dimgigov 19fa760604 Week 4: Production Hardening — Thread Safety, Fuzz Tests, Property-Based Tests, NimForum Smoke Test
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Thread Safety:
- Add lock to BTreeIndex (insert/get/scan/remove/contains)
- Add lock to InvertedIndex (addDocument/removeDocument/search/bm25/tfidf/fuzzy/regex)
- Add ctxLock to ExecutionContext for DDL/schema metadata, autoIncCounters, sequences
- Deep-copy sessionVars in cloneForConnection() to isolate per-connection state
- Fix activeConnections check-then-act race with Lock

Wire Protocol Hardening:
- Fix RangeDefect in deserializeValue for invalid FieldKind bytes
- Fix IndexDefect in fkFloat32/fkFloat64 by reusing readUint32/readUint64
- Fix int32/int64 cast from uint32/uint64 to use cast[] instead of constructor
- Fix parseHeader to validate MsgKind instead of holey enum cast

Property-Based Tests (tests/prop_test.nim):
- 8 invariants for evalExprValue: literal types, commutativity, identity,
  double negation, NULL propagation

Wire Protocol Fuzz (tests/fuzz_test.nim):
- 7 fuzz tests: random bytes deserialization, truncated buffers,
  MsgKind casts, parseHeader logic, message roundtrips

NimForum Adapter Smoke Test (tests/nimforum_smoke_test.nim):
- Full TCP server lifecycle with adapter CRUD and parameterized queries

Test Results: 423 tests pass, 0 failures
2026-05-17 11:57:55 +03:00
dimgigov 6021bfcb10 Сесия 9: Седмица 3 — JOIN Performance
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Добавен IRJoinStrategy enum (nestedLoop, hash, indexNestedLoop)
- Hash Join: build hash table върху по-малката страна, probe от другата
- Index Nested Loop Join: използва B-Tree индекс за point lookup
- Query planner chooseJoinStrategy избира стратегия според наличие на индекс
- LEFT/RIGHT/FULL JOIN fallback-ват към Nested Loop
- PK индексите се игнорират за INL (само explicit индекси)

- Benchmark: JOIN 10K ~115ms (Hash), ~90ms (Index NL)
- 5 нови теста за join performance и planner избор

Тестове: 316 — 0 failures
Build: 0 warnings
2026-05-17 10:50:12 +03:00
dimgigov 9d71edafd4 Сесия 9: Седмици 1+2 — Stabilization + Type Safety
- Поправени deficiencies #5-#8 (GROUP BY bare columns, aggregate names,
  sync client с blocking socket, thread-safe Lock в SyncClient)
- Добавен BlockingClient (net.Socket + Lock) в clients/nim/src/baradb/client.nim
- Обновен src/barabadb/client/client.nim със sync клиент без waitFor
- Regression тестове за всички 10 deficiencies

- Добавен valueKind в IRExpr за типова информация
- evalExprValue връща Value discriminated union
- Премахнати parseFloat евристики от irAdd/Sub/Mul/Div/Mod/Pow/Neg
- INT+INT→INT, INT+FLOAT→FLOAT, FLOAT/INT→FLOAT
- 12 нови теста за type safety

Тестове: 311 — 0 failures
Build: 0 warnings
2026-05-17 10:21:15 +03:00
dimgigov 4c6c72dc5b docs(plan): mark 9.1.2 as complete
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 03:50:57 +03:00
dimgigov 2e1d982b9f fix(executor): include arguments in aggregate column names
Issue #6: getSelectColumns() produced count() for count(*)
and max() for max(id). Now uses exprToSql() on funcArgs
to generate consistent names matching the actual SQL expression.

Before: columns: @[count()], rows: @[{count(*): 2}]
After:  columns: @[count(*)], rows: @[{count(*): 2}]
2026-05-17 03:50:43 +03:00
dimgigov cd62f98641 docs(plan): mark 9.1.1 as complete 2026-05-17 03:47:30 +03:00
dimgigov 07a22d305d chore: fix all build warnings and add stabilization roadmap
- Fix 3× ResultShadowed in cypher.nim, recovery.nim, shell.nim
- Remove 6× UnusedImport in test files
- Build is now clean: 0 warnings, 294 tests pass
- Add 4-week stabilization plan to PLAN.md (Sessions 9.1–9.4)
2026-05-17 03:47:17 +03:00