Baradb/docs/en/changelog.md

Changelog

All notable changes to BaraDB are documented in this file.

[Unreleased] — Server Architecture Improvements

Fixed

• Shared Storage Instance — TCP and HTTP servers now share a single LSMTree instance instead of creating separate ones, eliminating data inconsistency between protocols
• Unsafe Cast Operations — Replaced all cast[string] and cast[seq[byte]] with safe bytesToString/stringToBytes helper functions that properly copy data for ARC/ORC memory management compatibility

Changed

• TCP_NODELAY Enabled — Both listening and client sockets now have TCP_NODELAY set, reducing latency for small messages (queries, acks) by disabling Nagle's algorithm
• Rate Limiter Integrated — Token-bucket rate limiter is now active in both TCP and HTTP server handlers:
  • TCP: Per-client rate limiting on query messages (error code 429)
  • HTTP: Rate limiting on /query and /batch endpoints with X-Forwarded-For support
• Gossip Error Handling — Improved UDP socket error recovery with exponential backoff (100ms-5s) and socket recreation only after 10 consecutive failures

[Unreleased] — AI-Native Platform

Added

• MCP Server (Model Context Protocol) — STDIO JSON-RPC 2.0 server with 3 AI tools:
  • query — SQL execution with parameterized queries + multi-tenant session vars
  • vector_search — Semantic HNSW vector search with tenant isolation
  • schema_inspect — Table/column/index/RLS policy exploration
  • Standalone binary: build/baramcp
• Graph Engine Deep Integration — CREATE GRAPH / DROP GRAPH DDL with native adjacency list storage
  • GRAPH_TABLE() SQL function with 7 algorithms: BFS, DFS, PageRank, ShortestPath, Dijkstra, Louvain, Community
  • INSERT into _nodes/_edges tables auto-syncs with native Graph objects
  • Optional MATCH, ALGORITHM, START, END, MAXDEPTH in GRAPH_TABLE syntax
• Chunking + Embedding Pipeline — Server-side AI data processing:
  • chunk() SQL function — text splitting with configurable size/overlap
  • embed_text() SQL function — calls external embedding API (OpenAI/Ollama compatible)
  • Auto-embedding on INSERT — when VECTOR column is null, generates from TEXT column
  • Configurable via env vars: BARADB_EMBED_ENDPOINT, BARADB_EMBED_MODEL, BARADB_EMBED_API_KEY
• LangChain ChatMessageHistory — Python BaraDBChatHistory class:
  • Stores conversation threads in relational table with RLS
  • Multi-tenant isolation via tenant_id + user_id
• RAG Pipeline Example — End-to-end Python script (examples/rag_pipeline.py):
  • PDF/text ingestion → chunking → embedding → BaraDB storage → hybrid search → LLM generation
  • Supports OpenAI and Ollama APIs
• AI Agents & NL→SQL — Server-side LLM integration:
  • nl_to_sql() SQL function — natural language → SQL generation
  • schema_prompt() — generates DDL + sample data for LLM context
  • Query validation layer — sandbox execution with LIMIT 0 + EXPLAIN
  • Self-correction loop — error feedback to LLM for fix
  • Configurable via env vars: BARADB_LLM_ENDPOINT, BARADB_LLM_MODEL, BARADB_LLM_API_KEY
• Graph Similarity & Embeddings:
  • similarity_nodes() — Jaccard/Adamic-Adar similarity between node pairs
  • node2vec_embed() — Random-walk based graph embeddings
• Cypher Compatibility Layer:
  • cypher() SQL function — translates MATCH (a)-[r]->(b) RETURN ... to GRAPH_TABLE
  • Automatic Cypher → BaraQL conversion
• German Documentation — Full documentation set in German (docs/de/)

Changed

• Graph executor upgraded from stub to real BFS/DFS/PageRank/Dijkstra/Louvain execution
• ExecutionContext extended with graphs, embedder, llmClient fields
• Graph engine extended with addNodeWithId, addEdgeWithId, Jaccard, Adamic-Adar, node2vec

[1.1.0] — 2026-05-13

Added

• Client SDKs v1.1.4 — Full-featured clients for all languages:
  • JavaScript: TypeScript definitions, package.json, examples, unit & integration tests
  • Python: Restructured as proper package (baradb/ with __init__.py and core.py), pyproject.toml, examples, tests (query builder, wire protocol, integration)
  • Nim: Examples, integration tests, README
  • Rust: Examples, integration tests, improved Cargo.toml
• SCRAM-SHA-256 Authentication — RFC 7677 compliant authentication with PBKDF2 + HMAC + SHA-256 + nonce/salt generation
• HTTP SCRAM Endpoints — /auth/scram/start + /auth/scram/finish in HTTP server
• Docker Compose Test Configuration — docker-compose.test.yml for test environments
• CI/CD Clients Pipeline — .github/workflows/clients-ci.yml for automated client testing

Fixed

• Query Executor — Unary minus (irNeg) evaluation now works correctly in SELECT and WHERE clauses
• Distributed Transactions — Rollback after commit attempt no longer violates atomicity
• Sharding — Data migration protocol with TCP + scanAll on LSM
• Raft — Majority calculation for even number of nodes fixed
• MVCC — Aborted transactions no longer become visible
• LSM-Tree — Data loss on immutable memtable overwrite fixed; SSTable lookup sorting fixed
• Auth — JWT signature changed to HMAC-SHA256 (no longer trivially forgeable); token expiration (exp/nbf/iat) now validated; signature comparison is now constant-time
• Recovery — summary() no longer mutates the database
• Wire Protocol — 64MB limit + bounds checking + max depth to prevent OOM/DoS
• SQL Injection — exprToSql now escapes single quotes
• ReDoS — irLike/irILike now escape regex metacharacters
• Graph — addEdge now checks node existence
• Vector — Dimension mismatch validation + HNSW locking
• FTS — UTF-8 tokenization now uses runes instead of bytes
• Build — nim.cfg adds -d:ssl so nimble build works without flags; --threads:on added to all CI commands

Changed

• Version bumped to 1.1.0 across all components (server, Docker images, clients, CLI)
• README — Version badge updated; all feature tables now reference v1.1.4
• TLA+ Formal Verification — Added crossmodal.tla, backup.tla, recovery.tla; symmetry reduction in all 9 specs
• Clean build — 0 compiler warnings on Nim 2.2.10

[0.1.0] — 2025-01-15

Added

• Core Storage Engines

  • LSM-Tree with MemTable, WAL, SSTables, and size-tiered compaction
  • B-Tree ordered index with range scans and MVCC copy-on-write
  • Bloom filters for efficient SSTable skip
  • Memory-mapped I/O for SSTable reads
  • LRU page cache with hit rate tracking

• Query Engine (BaraQL)

  • SQL-compatible lexer with 80+ token types
  • Recursive descent parser producing AST with 25+ node kinds
  • Intermediate representation (IR) for execution plans
  • Code generator translating IR to storage operations
  • Adaptive query optimizer with cross-modal planning
  • Query executor with parallelization

• BaraQL Language Features

  • SELECT, INSERT, UPDATE, DELETE
  • WHERE, ORDER BY, LIMIT, OFFSET
  • GROUP BY, HAVING, aggregate functions (count, sum, avg, min, max)
  • INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN, CROSS JOIN
  • CTEs (Common Table Expressions) with WITH
  • Subqueries (EXISTS, IN, correlated)
  • CASE expressions
  • UNION, INTERSECT, EXCEPT
  • Schema definition: CREATE TYPE, DROP TYPE

• Vector Engine

  • HNSW index for approximate nearest neighbor search
  • IVF-PQ index for large-scale vector search
  • SIMD-optimized distance functions (cosine, L2, dot product, Manhattan)
  • Quantization: scalar 8-bit/4-bit, product quantization, binary
  • Metadata filtering during vector search

• Graph Engine

  • Adjacency list storage for directed, edge-weighted graphs
  • BFS and DFS traversal
  • Dijkstra shortest path
  • PageRank node importance
  • Louvain community detection
  • Subgraph pattern matching
  • Cypher-like graph query parser

• Full-Text Search

  • Inverted index with term-document mapping
  • BM25 ranking algorithm
  • TF-IDF scoring
  • Fuzzy search with Levenshtein distance
  • Wildcard/regex search
  • Multi-language tokenizers (English, Bulgarian, German, French, Russian)

• Columnar Storage

  • Per-column storage for analytical queries
  • RLE (Run-Length Encoding) compression
  • Dictionary encoding for low-cardinality columns
  • SIMD-accelerated aggregates

• Transactions

  • MVCC (Multi-Version Concurrency Control) with snapshot isolation
  • Deadlock detection via wait-for graph
  • Write-ahead log for durability
  • Savepoints and partial rollback

• Protocol Layer

  • Binary wire protocol with 16 message types
  • HTTP/REST JSON API
  • WebSocket streaming
  • Connection pooling
  • JWT-based authentication
  • Token-bucket rate limiting
  • TLS/SSL with auto-generated certificates

• Schema System

  • Strong type system with 17 native types
  • Type inheritance with multi-base support
  • Property links between types
  • Schema diffing and migrations
  • Computed properties

• Distributed Systems

  • Raft consensus (leader election, log replication)
  • Hash, range, and consistent-hash sharding
  • Sync/async/semi-sync replication
  • Gossip protocol for membership management
  • Two-phase commit for distributed transactions

• Cross-Modal Queries

  • Unified query language across all storage engines
  • Cross-engine predicate pushdown
  • Optimized execution plans for multi-modal queries

• Backup & Recovery

  • Online snapshots without downtime
  • Point-in-time recovery via WAL replay
  • Incremental backups

• Client SDKs

  • JavaScript/TypeScript client with binary protocol
  • Python client with sync and async APIs
  • Nim embedded mode and client library
  • Rust client (async)

• Operations

  • Interactive CLI shell (BaraQL REPL)
  • Structured logging (JSON and text formats)
  • Prometheus-compatible metrics endpoint
  • Health and readiness probes
  • CPU/memory profiling endpoints

• Docker Support

  • Multi-stage Dockerfile (Alpine Linux)
  • Docker Compose configuration
  • Health checks

Performance

• LSM-Tree: 580K writes/s, 720K reads/s
• B-Tree: 1.2M inserts/s, 1.5M lookups/s
• Vector SIMD: 850K cosine distances/s (dim=768)
• FTS: 320K docs/s indexing, 28K queries/s BM25
• Graph: 2.5M nodes/s insertion, 12K BFS traversals/s
• Binary protocol: 380K queries/s (100 concurrent connections)

Tests

• 262 tests across 56 test suites
• 100% pass rate

[Unreleased]

Added

• Vector SQL Integration — Full SQL-level vector search support:
  • VECTOR(n) column type in CREATE TABLE with dimension validation
  • CREATE INDEX ... USING hnsw / USING ivfpq for approximate nearest neighbor indexes
  • SQL distance functions: cosine_distance(), euclidean_distance(), inner_product(), l1_distance(), l2_distance()
  • <-> nearest-neighbor operator (Euclidean distance)
  • ORDER BY support for vector distance expressions, including columns not in SELECT
  • Automatic HNSW index maintenance on INSERT and UPDATE
• Advanced SQL Engine — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE)
• JavaScript Client — TCP Request Queue — Internal _requestQueue + _requestLock for safe concurrent queries. Multiple parallel query() / execute() / ping() calls no longer interleave binary frames on the wire.

Fixed

• Query Executor — Row Value Escaping — execInsert now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like [1.0, 2.0, 3.0]
• Query Planner — ORDER BY Projection — irpkSort is now placed before irpkProject in the IR plan, allowing ORDER BY to reference columns that are not selected
• Wire Protocol — Big-Endian Float Serialization — FLOAT32/FLOAT64 and vector float values are now serialized in big-endian byte order, matching the client's readFloatBE() / readDoubleBE() and ensuring cross-platform numeric accuracy.
• Gossip Protocol — Async UDP Socket — Replaced synchronous newSocket + blocking recvFrom with newAsyncSocket + await recvFrom, preventing the async event loop from freezing until a UDP packet arrives.

Planned

• Query plan caching
• Materialized views
• Geospatial index
• Time-series optimizations
• CDC (Change Data Capture) streaming
• Federated queries across BaraDB instances