Baradb/docs/en/changelog.md

# Changelog

All notable changes to BaraDB are documented in this file.

## [Unreleased] — Server Architecture Improvements

### Fixed

- **Shared Storage Instance** — TCP and HTTP servers now share a single LSMTree instance instead of creating separate ones, eliminating data inconsistency between protocols
- **Unsafe Cast Operations** — Replaced all `cast[string]` and `cast[seq[byte]]` with safe `bytesToString`/`stringToBytes` helper functions that properly copy data for ARC/ORC memory management compatibility

### Changed

- **TCP_NODELAY Enabled** — Both listening and client sockets now have TCP_NODELAY set, reducing latency for small messages (queries, acks) by disabling Nagle's algorithm
- **Rate Limiter Integrated** — Token-bucket rate limiter is now active in both TCP and HTTP server handlers:
  - TCP: Per-client rate limiting on query messages (error code 429)
  - HTTP: Rate limiting on `/query` and `/batch` endpoints with `X-Forwarded-For` support
- **Gossip Error Handling** — Improved UDP socket error recovery with exponential backoff (100ms-5s) and socket recreation only after 10 consecutive failures

## [Unreleased] — AI-Native Platform

### Added

- **MCP Server (Model Context Protocol)** — STDIO JSON-RPC 2.0 server with 3 AI tools:
  - `query` — SQL execution with parameterized queries + multi-tenant session vars
  - `vector_search` — Semantic HNSW vector search with tenant isolation
  - `schema_inspect` — Table/column/index/RLS policy exploration
  - Standalone binary: `build/baramcp`
- **Graph Engine Deep Integration** — `CREATE GRAPH` / `DROP GRAPH` DDL with native adjacency list storage
  - `GRAPH_TABLE()` SQL function with 7 algorithms: BFS, DFS, PageRank, ShortestPath, Dijkstra, Louvain, Community
  - INSERT into `_nodes`/`_edges` tables auto-syncs with native Graph objects
  - Optional `MATCH`, `ALGORITHM`, `START`, `END`, `MAXDEPTH` in GRAPH_TABLE syntax
- **Chunking + Embedding Pipeline** — Server-side AI data processing:
  - `chunk()` SQL function — text splitting with configurable size/overlap
  - `embed_text()` SQL function — calls external embedding API (OpenAI/Ollama compatible)
  - Auto-embedding on INSERT — when VECTOR column is null, generates from TEXT column
  - Configurable via env vars: `BARADB_EMBED_ENDPOINT`, `BARADB_EMBED_MODEL`, `BARADB_EMBED_API_KEY`
- **LangChain ChatMessageHistory** — Python `BaraDBChatHistory` class:
  - Stores conversation threads in relational table with RLS
  - Multi-tenant isolation via `tenant_id` + `user_id`
- **RAG Pipeline Example** — End-to-end Python script (`examples/rag_pipeline.py`):
  - PDF/text ingestion → chunking → embedding → BaraDB storage → hybrid search → LLM generation
  - Supports OpenAI and Ollama APIs
- **AI Agents & NL→SQL** — Server-side LLM integration:
  - `nl_to_sql()` SQL function — natural language → SQL generation
  - `schema_prompt()` — generates DDL + sample data for LLM context
  - Query validation layer — sandbox execution with LIMIT 0 + EXPLAIN
  - Self-correction loop — error feedback to LLM for fix
  - Configurable via env vars: `BARADB_LLM_ENDPOINT`, `BARADB_LLM_MODEL`, `BARADB_LLM_API_KEY`
- **Graph Similarity & Embeddings**:
  - `similarity_nodes()` — Jaccard/Adamic-Adar similarity between node pairs
  - `node2vec_embed()` — Random-walk based graph embeddings
- **Cypher Compatibility Layer**:
  - `cypher()` SQL function — translates `MATCH (a)-[r]->(b) RETURN ...` to GRAPH_TABLE
  - Automatic Cypher → BaraQL conversion
- **German Documentation** — Full documentation set in German (`docs/de/`)

### Changed

- Graph executor upgraded from stub to real BFS/DFS/PageRank/Dijkstra/Louvain execution
- ExecutionContext extended with `graphs`, `embedder`, `llmClient` fields
- Graph engine extended with `addNodeWithId`, `addEdgeWithId`, Jaccard, Adamic-Adar, node2vec

## [1.1.0] — 2026-05-13

### Added

- **Client SDKs v1.1.6** — Full-featured clients for all languages:
  - JavaScript: TypeScript definitions, package.json, examples, unit & integration tests
  - Python: Restructured as proper package (`baradb/` with `__init__.py` and `core.py`), pyproject.toml, examples, tests (query builder, wire protocol, integration)
  - Nim: Examples, integration tests, README
  - Rust: Examples, integration tests, improved Cargo.toml
- **SCRAM-SHA-256 Authentication** — RFC 7677 compliant authentication with PBKDF2 + HMAC + SHA-256 + nonce/salt generation
- **HTTP SCRAM Endpoints** — `/auth/scram/start` + `/auth/scram/finish` in HTTP server
- **Docker Compose Test Configuration** — `docker-compose.test.yml` for test environments
- **CI/CD Clients Pipeline** — `.github/workflows/clients-ci.yml` for automated client testing

### Fixed

- **Query Executor** — Unary minus (`irNeg`) evaluation now works correctly in SELECT and WHERE clauses
- **Distributed Transactions** — Rollback after commit attempt no longer violates atomicity
- **Sharding** — Data migration protocol with TCP + `scanAll` on LSM
- **Raft** — Majority calculation for even number of nodes fixed
- **MVCC** — Aborted transactions no longer become visible
- **LSM-Tree** — Data loss on immutable memtable overwrite fixed; SSTable lookup sorting fixed
- **Auth** — JWT signature changed to HMAC-SHA256 (no longer trivially forgeable); token expiration (`exp`/`nbf`/`iat`) now validated; signature comparison is now constant-time
- **Recovery** — `summary()` no longer mutates the database
- **Wire Protocol** — 64MB limit + bounds checking + max depth to prevent OOM/DoS
- **SQL Injection** — `exprToSql` now escapes single quotes
- **ReDoS** — `irLike`/`irILike` now escape regex metacharacters
- **Graph** — `addEdge` now checks node existence
- **Vector** — Dimension mismatch validation + HNSW locking
- **FTS** — UTF-8 tokenization now uses runes instead of bytes
- **Build** — `nim.cfg` adds `-d:ssl` so `nimble build` works without flags; `--threads:on` added to all CI commands

### Changed

- **Version bumped to 1.1.0** across all components (server, Docker images, clients, CLI)
- **README** — Version badge updated; all feature tables now reference v1.1.6
- **TLA+ Formal Verification** — Added `crossmodal.tla`, `backup.tla`, `recovery.tla`; symmetry reduction in all 9 specs
- **Clean build** — 0 compiler warnings on Nim 2.2.10

## [0.1.0] — 2025-01-15

### Added

- **Core Storage Engines**
  - LSM-Tree with MemTable, WAL, SSTables, and size-tiered compaction
  - B-Tree ordered index with range scans and MVCC copy-on-write
  - Bloom filters for efficient SSTable skip
  - Memory-mapped I/O for SSTable reads
  - LRU page cache with hit rate tracking

- **Query Engine (BaraQL)**
  - SQL-compatible lexer with 80+ token types
  - Recursive descent parser producing AST with 25+ node kinds
  - Intermediate representation (IR) for execution plans
  - Code generator translating IR to storage operations
  - Adaptive query optimizer with cross-modal planning
  - Query executor with parallelization

- **BaraQL Language Features**
  - SELECT, INSERT, UPDATE, DELETE
  - WHERE, ORDER BY, LIMIT, OFFSET
  - GROUP BY, HAVING, aggregate functions (count, sum, avg, min, max)
  - INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN, CROSS JOIN
  - CTEs (Common Table Expressions) with WITH
  - Subqueries (EXISTS, IN, correlated)
  - CASE expressions
  - UNION, INTERSECT, EXCEPT
  - Schema definition: CREATE TYPE, DROP TYPE

- **Vector Engine**
  - HNSW index for approximate nearest neighbor search
  - IVF-PQ index for large-scale vector search
  - SIMD-optimized distance functions (cosine, L2, dot product, Manhattan)
  - Quantization: scalar 8-bit/4-bit, product quantization, binary
  - Metadata filtering during vector search

- **Graph Engine**
  - Adjacency list storage for directed, edge-weighted graphs
  - BFS and DFS traversal
  - Dijkstra shortest path
  - PageRank node importance
  - Louvain community detection
  - Subgraph pattern matching
  - Cypher-like graph query parser

- **Full-Text Search**
  - Inverted index with term-document mapping
  - BM25 ranking algorithm
  - TF-IDF scoring
  - Fuzzy search with Levenshtein distance
  - Wildcard/regex search
  - Multi-language tokenizers (English, Bulgarian, German, French, Russian)

- **Columnar Storage**
  - Per-column storage for analytical queries
  - RLE (Run-Length Encoding) compression
  - Dictionary encoding for low-cardinality columns
  - SIMD-accelerated aggregates

- **Transactions**
  - MVCC (Multi-Version Concurrency Control) with snapshot isolation
  - Deadlock detection via wait-for graph
  - Write-ahead log for durability
  - Savepoints and partial rollback

- **Protocol Layer**
  - Binary wire protocol with 16 message types
  - HTTP/REST JSON API
  - WebSocket streaming
  - Connection pooling
  - JWT-based authentication
  - Token-bucket rate limiting
  - TLS/SSL with auto-generated certificates

- **Schema System**
  - Strong type system with 17 native types
  - Type inheritance with multi-base support
  - Property links between types
  - Schema diffing and migrations
  - Computed properties

- **Distributed Systems**
  - Raft consensus (leader election, log replication)
  - Hash, range, and consistent-hash sharding
  - Sync/async/semi-sync replication
  - Gossip protocol for membership management
  - Two-phase commit for distributed transactions

- **Cross-Modal Queries**
  - Unified query language across all storage engines
  - Cross-engine predicate pushdown
  - Optimized execution plans for multi-modal queries

- **Backup & Recovery**
  - Online snapshots without downtime
  - Point-in-time recovery via WAL replay
  - Incremental backups

- **Client SDKs**
  - JavaScript/TypeScript client with binary protocol
  - Python client with sync and async APIs
  - Nim embedded mode and client library
  - Rust client (async)

- **Operations**
  - Interactive CLI shell (BaraQL REPL)
  - Structured logging (JSON and text formats)
  - Prometheus-compatible metrics endpoint
  - Health and readiness probes
  - CPU/memory profiling endpoints

- **Docker Support**
  - Multi-stage Dockerfile (Alpine Linux)
  - Docker Compose configuration
  - Health checks

### Performance

- LSM-Tree: 580K writes/s, 720K reads/s
- B-Tree: 1.2M inserts/s, 1.5M lookups/s
- Vector SIMD: 850K cosine distances/s (dim=768)
- FTS: 320K docs/s indexing, 28K queries/s BM25
- Graph: 2.5M nodes/s insertion, 12K BFS traversals/s
- Binary protocol: 380K queries/s (100 concurrent connections)

### Tests

- 262 tests across 56 test suites
- 100% pass rate

## [Unreleased]

### Added

- **Vector SQL Integration** — Full SQL-level vector search support:
  - `VECTOR(n)` column type in `CREATE TABLE` with dimension validation
  - `CREATE INDEX ... USING hnsw` / `USING ivfpq` for approximate nearest neighbor indexes
  - SQL distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
  - `<->` nearest-neighbor operator (Euclidean distance)
  - `ORDER BY` support for vector distance expressions, including columns not in `SELECT`
  - Automatic HNSW index maintenance on `INSERT` and `UPDATE`
- **Advanced SQL Engine** — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE)
- **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire.

### Fixed

- **Query Executor — Row Value Escaping** — `execInsert` now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like `[1.0, 2.0, 3.0]`
- **Query Planner — ORDER BY Projection** — `irpkSort` is now placed before `irpkProject` in the IR plan, allowing `ORDER BY` to reference columns that are not selected
- **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy.
- **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives.

### Planned

- Query plan caching
- Materialized views
- Geospatial index
- Time-series optimizations
- CDC (Change Data Capture) streaming
- Federated queries across BaraDB instances