Baradb/docs/en/changelog.md

# Changelog

All notable changes to BaraDB are documented in this file.

## [Unreleased] — SQL:2023 Stabilization

### Fixed

- **GROUPING SETS execution** — `lowerSelect` now creates `irpkGroupBy` when `selGroupingSetsKind != gskNone`, even if `selGroupBy` is empty. Previously, queries like `GROUP BY GROUPING SETS ((dept), ())` bypassed the grouping executor entirely.
- **FTS CREATE INDEX docId mismatch** — `CREATE INDEX ... USING FTS` now computes `docId` as a hash of `tableName.$key`, consistent with DML operations (`INSERT`/`UPDATE`/`DELETE`). Previously, index creation used sequential IDs (0, 1, 2...), causing `@@` queries to never match indexed documents.
- **Test isolation (all suites)** — All `newLSMTree("")` calls replaced with unique temporary directories per suite. Eliminates WAL accumulation issues and flaky tests caused by shared database state between test runs.
- **Window frame parser** — `parseFrameBoundary` no longer consumes `tkRow` after `tkCurrent` incorrectly (was using `tkRows`). Also fixed `tkRow` keyword conflict with `ENABLE ROW LEVEL SECURITY` parsing.
- **ORDER BY + SELECT projection** — `lowerSelect` now places `irpkSort` before `irpkProject`, enabling `ORDER BY` on columns not present in the `SELECT` list.
- **UNPIVOT execution** — Verified and fixed missing test coverage for UNPIVOT transformation.

### Added

- **JSON operators** — `@>` (contains), `<@` (contained by), `?` (has key), `?|` (has any), `?&` (has all) now supported in lexer, parser, and executor.
- **Window frame execution** — `ROWS BETWEEN X PRECEDING AND Y FOLLOWING` / `CURRENT ROW` frame boundaries now respected by `FIRST_VALUE` and `LAST_VALUE`.
- **Session variables** — `SET var_name = value` and `current_setting('var_name')` for connection-scoped key/value storage.
- **Current user/role** — `current_user` and `current_role` SQL keywords evaluate to the authenticated session's user and role.
- **Auth-executor bridge** — Wire server and HTTP server now populate `ExecutionContext.currentUser` and `ExecutionContext.currentRole` after JWT/SCRAM authentication.
- **Multi-tenant RLS** — Row-Level Security policies can now reference `current_user`, `current_role`, and `current_setting('app.tenant_id')` for per-tenant data isolation.

## [1.1.0] — 2026-05-13

### Added

- **Client SDKs v1.1.0** — Full-featured clients for all languages:
  - JavaScript: TypeScript definitions, package.json, examples, unit & integration tests
  - Python: Restructured as proper package (`baradb/` with `__init__.py` and `core.py`), pyproject.toml, examples, tests (query builder, wire protocol, integration)
  - Nim: Examples, integration tests, README
  - Rust: Examples, integration tests, improved Cargo.toml
- **SCRAM-SHA-256 Authentication** — RFC 7677 compliant authentication with PBKDF2 + HMAC + SHA-256 + nonce/salt generation
- **HTTP SCRAM Endpoints** — `/auth/scram/start` + `/auth/scram/finish` in HTTP server
- **Docker Compose Test Configuration** — `docker-compose.test.yml` for test environments
- **CI/CD Clients Pipeline** — `.github/workflows/clients-ci.yml` for automated client testing

### Fixed

- **Query Executor** — Unary minus (`irNeg`) evaluation now works correctly in SELECT and WHERE clauses
- **Distributed Transactions** — Rollback after commit attempt no longer violates atomicity
- **Sharding** — Data migration protocol with TCP + `scanAll` on LSM
- **Raft** — Majority calculation for even number of nodes fixed
- **MVCC** — Aborted transactions no longer become visible
- **LSM-Tree** — Data loss on immutable memtable overwrite fixed; SSTable lookup sorting fixed
- **Auth** — JWT signature changed to HMAC-SHA256 (no longer trivially forgeable); token expiration (`exp`/`nbf`/`iat`) now validated; signature comparison is now constant-time
- **Recovery** — `summary()` no longer mutates the database
- **Wire Protocol** — 64MB limit + bounds checking + max depth to prevent OOM/DoS
- **SQL Injection** — `exprToSql` now escapes single quotes
- **ReDoS** — `irLike`/`irILike` now escape regex metacharacters
- **Graph** — `addEdge` now checks node existence
- **Vector** — Dimension mismatch validation + HNSW locking
- **FTS** — UTF-8 tokenization now uses runes instead of bytes
- **Build** — `nim.cfg` adds `-d:ssl` so `nimble build` works without flags; `--threads:on` added to all CI commands

### Changed

- **Version bumped to 1.1.0** across all components (server, Docker images, clients, CLI)
- **README** — Version badge updated; all feature tables now reference v1.1.0
- **TLA+ Formal Verification** — Added `crossmodal.tla`, `backup.tla`, `recovery.tla`; symmetry reduction in all 9 specs
- **Clean build** — 0 compiler warnings on Nim 2.2.10

## [0.1.0] — 2025-01-15

### Added

- **Core Storage Engines**
  - LSM-Tree with MemTable, WAL, SSTables, and size-tiered compaction
  - B-Tree ordered index with range scans and MVCC copy-on-write
  - Bloom filters for efficient SSTable skip
  - Memory-mapped I/O for SSTable reads
  - LRU page cache with hit rate tracking

- **Query Engine (BaraQL)**
  - SQL-compatible lexer with 80+ token types
  - Recursive descent parser producing AST with 25+ node kinds
  - Intermediate representation (IR) for execution plans
  - Code generator translating IR to storage operations
  - Adaptive query optimizer with cross-modal planning
  - Query executor with parallelization

- **BaraQL Language Features**
  - SELECT, INSERT, UPDATE, DELETE
  - WHERE, ORDER BY, LIMIT, OFFSET
  - GROUP BY, HAVING, aggregate functions (count, sum, avg, min, max)
  - INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN, CROSS JOIN
  - CTEs (Common Table Expressions) with WITH
  - Subqueries (EXISTS, IN, correlated)
  - CASE expressions
  - UNION, INTERSECT, EXCEPT
  - Schema definition: CREATE TYPE, DROP TYPE

- **Vector Engine**
  - HNSW index for approximate nearest neighbor search
  - IVF-PQ index for large-scale vector search
  - SIMD-optimized distance functions (cosine, L2, dot product, Manhattan)
  - Quantization: scalar 8-bit/4-bit, product quantization, binary
  - Metadata filtering during vector search

- **Graph Engine**
  - Adjacency list storage for directed, edge-weighted graphs
  - BFS and DFS traversal
  - Dijkstra shortest path
  - PageRank node importance
  - Louvain community detection
  - Subgraph pattern matching
  - Cypher-like graph query parser

- **Full-Text Search**
  - Inverted index with term-document mapping
  - BM25 ranking algorithm
  - TF-IDF scoring
  - Fuzzy search with Levenshtein distance
  - Wildcard/regex search
  - Multi-language tokenizers (English, Bulgarian, German, French, Russian)

- **Columnar Storage**
  - Per-column storage for analytical queries
  - RLE (Run-Length Encoding) compression
  - Dictionary encoding for low-cardinality columns
  - SIMD-accelerated aggregates

- **Transactions**
  - MVCC (Multi-Version Concurrency Control) with snapshot isolation
  - Deadlock detection via wait-for graph
  - Write-ahead log for durability
  - Savepoints and partial rollback

- **Protocol Layer**
  - Binary wire protocol with 16 message types
  - HTTP/REST JSON API
  - WebSocket streaming
  - Connection pooling
  - JWT-based authentication
  - Token-bucket rate limiting
  - TLS/SSL with auto-generated certificates

- **Schema System**
  - Strong type system with 17 native types
  - Type inheritance with multi-base support
  - Property links between types
  - Schema diffing and migrations
  - Computed properties

- **Distributed Systems**
  - Raft consensus (leader election, log replication)
  - Hash, range, and consistent-hash sharding
  - Sync/async/semi-sync replication
  - Gossip protocol for membership management
  - Two-phase commit for distributed transactions

- **Cross-Modal Queries**
  - Unified query language across all storage engines
  - Cross-engine predicate pushdown
  - Optimized execution plans for multi-modal queries

- **Backup & Recovery**
  - Online snapshots without downtime
  - Point-in-time recovery via WAL replay
  - Incremental backups

- **Client SDKs**
  - JavaScript/TypeScript client with binary protocol
  - Python client with sync and async APIs
  - Nim embedded mode and client library
  - Rust client (async)

- **Operations**
  - Interactive CLI shell (BaraQL REPL)
  - Structured logging (JSON and text formats)
  - Prometheus-compatible metrics endpoint
  - Health and readiness probes
  - CPU/memory profiling endpoints

- **Docker Support**
  - Multi-stage Dockerfile (Alpine Linux)
  - Docker Compose configuration
  - Health checks

### Performance

- LSM-Tree: 580K writes/s, 720K reads/s
- B-Tree: 1.2M inserts/s, 1.5M lookups/s
- Vector SIMD: 850K cosine distances/s (dim=768)
- FTS: 320K docs/s indexing, 28K queries/s BM25
- Graph: 2.5M nodes/s insertion, 12K BFS traversals/s
- Binary protocol: 380K queries/s (100 concurrent connections)

### Tests

- 262 tests across 56 test suites
- 100% pass rate

## [Unreleased]

### Added

- **Vector SQL Integration** — Full SQL-level vector search support:
  - `VECTOR(n)` column type in `CREATE TABLE` with dimension validation
  - `CREATE INDEX ... USING hnsw` / `USING ivfpq` for approximate nearest neighbor indexes
  - SQL distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
  - `<->` nearest-neighbor operator (Euclidean distance)
  - `ORDER BY` support for vector distance expressions, including columns not in `SELECT`
  - Automatic HNSW index maintenance on `INSERT` and `UPDATE`
- **Advanced SQL Engine** — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE)
- **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire.

### Fixed

- **Query Executor — Row Value Escaping** — `execInsert` now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like `[1.0, 2.0, 3.0]`
- **Query Planner — ORDER BY Projection** — `irpkSort` is now placed before `irpkProject` in the IR plan, allowing `ORDER BY` to reference columns that are not selected
- **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy.
- **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives.

### Planned

- Query plan caching
- Materialized views
- Geospatial index
- Time-series optimizations
- CDC (Change Data Capture) streaming
- Federated queries across BaraDB instances