From 7fc6adbe4792c06c07b513082ad3f2a07bc1b545 Mon Sep 17 00:00:00 2001 From: dimgigov Date: Thu, 21 May 2026 09:47:40 +0300 Subject: [PATCH] chore: remove old planning and bug report files These files are outdated and only cause confusion. Keep only the current PLAN.md. --- BARADB_BUGS_REPORT.md | 181 --------------------- BARADB_DEFICIENCIES.md | 249 ----------------------------- PLAN_DONE.md | 190 ---------------------- PLAN_ID_GENERATORS.md | 74 --------- PLAN_SQL_ADVANCED.md | 351 ---------------------------------------- PLAN_STABILIZATION.md | 149 ----------------- PLAN_old_1.md | 155 ------------------ PLAN_old_2.md | 355 ----------------------------------------- PLAN_old_3.md | 207 ------------------------ PR_DESCRIPTION.md | 60 ------- ROADMAP.md | 202 ----------------------- 11 files changed, 2173 deletions(-) delete mode 100644 BARADB_BUGS_REPORT.md delete mode 100644 BARADB_DEFICIENCIES.md delete mode 100644 PLAN_DONE.md delete mode 100644 PLAN_ID_GENERATORS.md delete mode 100644 PLAN_SQL_ADVANCED.md delete mode 100644 PLAN_STABILIZATION.md delete mode 100644 PLAN_old_1.md delete mode 100644 PLAN_old_2.md delete mode 100644 PLAN_old_3.md delete mode 100644 PR_DESCRIPTION.md delete mode 100644 ROADMAP.md diff --git a/BARADB_BUGS_REPORT.md b/BARADB_BUGS_REPORT.md deleted file mode 100644 index 56d682f..0000000 --- a/BARADB_BUGS_REPORT.md +++ /dev/null @@ -1,181 +0,0 @@ -# BaraDB Bug Report — NimForum Integration (Stress Test) - -**Date:** 2026-05-19 -**Reporter:** NimForum integration team -**BaraDB Version:** Latest (docker image `nimforum-baradb`) -**Severity:** High — prevents real-world multi-table applications from working - ---- - -## Summary - -During integration of the NimForum application (a real-world web forum) with BaraDB via the native TCP wire protocol, we hit **three distinct SQL-layer bugs** in BaraDB's query executor/parser. All of them are related to multi-table (implicit join) queries. The bugs are severe enough that any application using standard SQL patterns (table aliases, duplicate column names, or `IN (list)` predicates) will receive incorrect results or crash. - ---- - -## Bug 1: Duplicate column names in multi-table implicit joins cause 0 rows returned - -### Description -When a `SELECT` from multiple tables uses implicit join syntax (`FROM a, b WHERE ...`) and the selected columns include the **same column name from both tables** (e.g. `a.id` and `b.id`, or `a.name` and `b.name`) **without explicit `AS` aliases**, BaraDB returns **0 rows** even when matching data exists. - -### Reproducer - -```sql --- Returns 0 rows (BUG) -SELECT thread.id, thread.name, category.id, category.name -FROM thread, category -WHERE thread.id = 3 - AND thread.isDeleted = 0 - AND thread.category = category.id; - --- Returns 1 correct row (WORKAROUND) -SELECT t.id AS thread_id, t.name AS thread_name, - c.id AS cat_id, c.name AS cat_name -FROM thread t, category c -WHERE t.id = 3 - AND t.isDeleted = 0 - AND t.category = c.id; -``` - -### Expected behavior -Both queries should return 1 row with the thread and category data. - -### Actual behavior -The first query (without aliases) returns **0 rows**. - -### Root cause hypothesis -BaraDB's executor or result-set builder appears to use column names as keys internally. When two selected columns have the same name (`id`, `name`), the second overwrites the first, corrupting the row metadata and causing the executor to discard the row. - -### Impact -Any standard SQL query joining two tables on `id` columns (extremely common) will silently fail. - -### Workaround used -Add explicit `AS` aliases to every column in multi-table selects. - ---- - -## Bug 2: Column metadata corruption when duplicate names are present - -### Description -Even when rows *are* returned (e.g. in a multi-row list query without `WHERE id = ?`), if duplicate column names exist, the **column metadata and data values are scrambled**. Values from one column appear in another column's position. - -### Reproducer - -```sql -SELECT t.id, t.name, c.id, c.name, c.description, c.color -FROM thread t, category c -WHERE t.isDeleted = 0 AND t.category = c.id -ORDER BY modified DESC LIMIT 1; -``` - -### Expected behavior -| id | name | id | name | description | color | -|---|---|---|---|---|---| -| 3 | Test Thread2 | 1 | baradb | multimodal database engine written in Nim | 1a465b | - -### Actual behavior -Column metadata returned by BaraDB: -``` -Columns: ['id', 'name', 'views', "strftime('%s', \"modified\")", 'isLocked', 'isPinned', 'id', 'name', 'description', 'color'] -Row[0]: ['3', 'Test Thread2', 0, '1779200191', , None, , None, '1', 'baradb'] -``` -Values are shifted/missing for the second table's columns. - -### Impact -Client code receives corrupted data. In our case `category.description` became `"1"` and `category.color` became `"baradb"`, which would break UI rendering. - -### Workaround used -Same as Bug 1: explicit `AS` aliases for every column. - ---- - -## Bug 3: `IN (val1, val2, ...)` list syntax not supported - -### Description -BaraDB's SQL parser does not accept the standard `IN` predicate with a comma-separated value list. It throws a parse error. - -### Reproducer - -```sql --- ERROR: Expected tkRParen but got tkComma at line 1 -SELECT id, name, email -FROM person -WHERE id IN (2, 1); -``` - -### Expected behavior -Should return rows for `id = 2` and `id = 1`. - -### Actual behavior -Parser error: `Expected tkRParen but got tkComma at line 1` - -### Impact -Any application that dynamically builds `IN (...)` lists (e.g. looking up multiple users by ID) cannot function without rewriting every query. - -### Workaround used -Replace `IN (2, 1)` with `id = 2 OR id = 1` generated dynamically in the application code. - ---- - -## Bug 4: `person.id` / `post.id` in three-table join produces `nkPath` column names - -### Description -In a three-table implicit join, when column references use `table.column` syntax, BaraDB's parser sometimes produces corrupted column names like `"strftime('%s', nkPath)"` instead of `"strftime('%s', post.creation)"`. - -### Reproducer - -```sql -SELECT post.id, strftime('%s', post.creation), post.thread, - person.id, person.name, person.email, - strftime('%s', person.lastOnline), - strftime('%s', person.previousVisitAt), person.usrStatus, - person.isDeleted, - thread.name -FROM post, person, thread -WHERE post.thread = thread.id - AND post.author = person.id - AND post.id = 10; -``` - -### Actual column metadata returned -``` -Columns: ['id', "strftime('%s', nkPath)", 'thread', 'id', 'name', 'email', - "strftime('%s', nkPath)", "strftime('%s', nkPath)", - 'usrStatus', 'isDeleted', 'name'] -``` - -### Impact -Result set metadata is unreadable. Client code cannot map columns to fields. - -### Workaround used -Use `AS` aliases for every expression and column: `p.id AS post_id`, `strftime('%s', p.creation) AS post_creation`, etc. - ---- - -## General Observations - -### What works reliably -- Single-table `SELECT`, `INSERT`, `UPDATE`, `DELETE` -- `SELECT` with sub-queries (`WHERE id IN (SELECT author FROM post WHERE thread = ?)`) -- `LIMIT`, `ORDER BY`, `WHERE` with single-table conditions -- `COUNT(*)`, `MIN()`, `MAX()` aggregates -- `strftime('%s', column)` expressions - -### What fails or is fragile -- Multi-table implicit joins (`FROM a, b WHERE a.id = b.id`) **without** `AS` aliases -- `IN (list)` with literal value lists -- `table.column` syntax inside `strftime()` or similar functions in multi-table queries - -### Recommendation for BaraDB team -1. **Fix column deduplication in result sets** — the executor should not use raw column names as internal keys; it should use ordinal positions or alias names. -2. **Support `IN (val1, val2, ...)`** — extend the parser to accept comma-separated expressions inside `IN (...)`. -3. **Investigate `nkPath` leak in parser** — `post.creation` inside `strftime()` should not be tokenized into `nkPath` in the column metadata output. - ---- - -## Appendix: Test Environment - -- **Client:** Custom synchronous Nim TCP client (`baradb_sync_client.nim`) using `net.Socket` -- **Wire protocol:** Binary protocol with `mkQuery` (0x02), `mkData` (0x82), `mkComplete` (0x83) -- **Result format:** `rfBinary` (0x00) -- **Connection:** Fresh socket per query (to rule out interleaving) diff --git a/BARADB_DEFICIENCIES.md b/BARADB_DEFICIENCIES.md deleted file mode 100644 index b70e872..0000000 --- a/BARADB_DEFICIENCIES.md +++ /dev/null @@ -1,249 +0,0 @@ -# BaraDB Deficiencies Fixed During NimForum Migration - -This document summarizes the bugs and design deficiencies discovered in BaraDB while porting NimForum from SQLite to BaraDB's TCP wire protocol. - ---- - -## 1. Comma Join Not Supported in Parser - -**Problem:** The BaraQL parser did not recognize comma-separated table lists in the `FROM` clause. - -```sql -SELECT * FROM thread t, category c WHERE t.category = c.id -``` - -This would fail with "unknown tokens" after the first comma. - -**Root Cause:** `parseSelect()` only parsed a single table after `FROM` and ignored everything after a comma. - -**Fix:** Modified `database/src/barabadb/query/parser.nim` to loop over comma-separated tables and treat each additional table as an implicit `CROSS JOIN`. - -```nim -while p.peek().kind == tkComma: - discard p.advance() - let nextTableTok = p.expect(tkIdent) - # ... build joinNode with jkCross and add to result.selJoins -``` - ---- - -## 2. DEFAULT Constraints Not Evaluated at Schema Creation Time - -**Problem:** `CREATE TABLE` with `DEFAULT ` stored the raw AST node instead of the evaluated string value. During `INSERT`, `applyDefaultValues()` checked `colDef.defaultVal.len > 0`, but `defaultVal` was empty because the AST was never evaluated to a string. - -**Result:** Explicitly omitted columns in `INSERT` raised `NOT NULL constraint violation` even when a `DEFAULT` was defined. - -**Fix:** Added `evalNodeToString()` in `database/src/barabadb/query/executor.nim` to evaluate `DEFAULT` AST nodes to their string representation during schema restoration. - -```nim -proc evalNodeToString(node: Node): string = - let ir = lowerExpr(node) - return evalExpr(ir, initTable[string, string](), nil) -``` - ---- - -## 3. Join Column Resolution Overwrites Duplicate Column Names - -**Problem:** `Row` is defined as `Table[string, string]`. When a query selects columns with identical names from joined tables: - -```sql -SELECT t.id, c.id FROM thread t INNER JOIN category c ON t.category = c.id -``` - -The `Project` operator in `executePlan()` builds `newRow` as a `Table`. The second `id` overwrites the first, so both columns end up with the value from the right-hand table. - -**Result:** `t.id` returns `0` (category id) instead of `1` (thread id). - -**Fix:** Two changes in `database/src/barabadb/query/executor.nim`: - -1. `lowerSelect()` — for `nkPath` expressions, use the full path (`t.id`) as the alias instead of just the last segment (`id`). -2. Added uniqueness logic for all aliases: if a duplicate alias is detected, append `_1`, `_2`, etc. This also fixes `strftime()` appearing multiple times in the same select list. - -```nim -# Before -projectPlan.projectAliases.add(e.pathParts[^1]) # -> "id" -# After -projectPlan.projectAliases.add(e.pathParts.join(".")) # -> "t.id" -``` - ---- - -## 4. Empty Result Sets Do Not Send Column Metadata - -**Problem:** In `database/src/barabadb/core/server.nim`, the server only sent the `mkData` message when `result.rows.len > 0`: - -```nim -if result.rows.len > 0: - let dataMsg = serializeResult(result, header.requestId) - await client.send(...) -``` - -For queries with zero matching rows (e.g. `WHERE id IN (SELECT ...)` with no subquery matches), the client received only `mkComplete` and no `mkData`. - -**Result:** The client saw `columns: @[]` and `rows: @[]`. When `getRow()` fell back to `newSeq[string](qr.columns.len)`, it returned an empty seq, causing "index out of bounds" errors when the application tried to access `row[0]`. - -**Fix:** Removed the `if result.rows.len > 0` guard. `serializeResult()` is now always sent, including the correct column names even when `rowCount = 0`. - ---- - -## 5. GROUP BY Returns Empty Values for Non-Aggregated Columns - -**Problem:** Unlike SQLite, BaraDB did not automatically pick an arbitrary row value for columns that are neither in `GROUP BY` nor inside an aggregate function. - -```sql -SELECT u.id, u.name, count(*) -FROM person u, post p -WHERE p.author = u.id AND p.thread = ? -GROUP BY name -``` - -**Result:** `u.id`, `u.email`, `u.usrStatus`, etc. returned empty strings, while `name` and `count(*)` were correct. - -**Fix:** Modified `query/executor.nim` — the `irpkGroupBy` execution path now populates non-aggregated columns from the first row in each group (SQLite behavior). The forum workaround using `DISTINCT` is no longer necessary. - -```nim -# Populate non-aggregated columns from first row in group -if groupRows.len > 0: - for k, v in groupRows[0]: - if not k.startsWith("$") and k notin aggRow: - aggRow[k] = v -``` - ---- - -## 6. Inconsistent Aggregate Column Names - -**Problem:** Aggregate functions produced column names that omitted the argument expression: - -- `SELECT count(*)` → column name `count()` -- `SELECT max(id)` → column name `max()` -- `SELECT min(creation)` → column name `min()` - -Code that relies on exact column names (e.g. `getValue` looking up `count(*)`) could be confused. - -**Fix:** Modified `query/executor.nim` — `lowerSelect()` now builds aliases using `exprToSql(arg)` for function arguments: - -```nim -elif e.kind == nkFuncCall: - var aliasArgs: seq[string] = @[] - for arg in e.funcArgs: aliasArgs.add(exprToSql(arg)) - projectPlan.projectAliases.add(e.funcName & "(" & aliasArgs.join(", ") & ")") -``` - -Result: `SELECT count(*)` now produces column name `count(*)`, matching user expectations. - ---- - -## 7. Async Client + waitFor in Async Context = Connection Instability - -**Problem:** The original `baradb_client.nim` used `AsyncSocket` wrapped in `SyncClient` via `waitFor`. When called from inside Jester's async handlers, nested `waitFor` + `poll()` created race conditions on the single socket. `recv(12)` occasionally returned 0 bytes, causing "Connection closed" exceptions. - -**Result:** Login and other routes crashed intermittently with 502 Bad Gateway. - -**Fix:** Rewrote `SyncClient` in both `clients/nim/src/baradb/client.nim` and `src/barabadb/client/client.nim` to use blocking `net.Socket` from Nim's standard library. This eliminates all async event loop interactions. - -Key changes: -- `SyncClient.socket` is now `net.Socket` instead of `AsyncSocket` -- `connect()` uses blocking `net.connect()` instead of `waitFor asyncClient.connect()` -- `query()` uses blocking `recv()` via a `recvExact()` helper instead of `waitFor` -- No dependency on `asyncdispatch` or `waitFor` in sync path -- Fully compatible with the existing wire protocol - ---- - -## 8. Lack of Thread Safety in the Adapter - -**Problem:** A single `SyncClient` instance was shared across all HTTP requests. NimForum runs on Jester's async event loop, which can interleave request handlers in the same thread. Without synchronization, two handlers could send queries over the same socket simultaneously, corrupting the wire protocol stream. - -**Fix:** Integrated a `Lock` directly into `SyncClient` in both client libraries. Every public operation (`query`, `exec`, `auth`, `ping`, `close`) acquires the lock before touching the socket: - -```nim -type - SyncClient* = ref object - config: ClientConfig - socket: net.Socket - connected: bool - requestId: uint32 - lock: Lock - -proc query*(client: SyncClient, sql: string): QueryResult = - acquire(client.lock) - try: - ... socket I/O ... - finally: - release(client.lock) -``` - -The external `withDbLock` workaround in the forum adapter is no longer necessary because the client itself is now thread-safe. - ---- - -## 9. `key` is a Reserved SQL Keyword - -**Problem:** BaraDB's SQL parser treats `key` as a reserved keyword. Using it as a column or table name causes parse errors: - -``` -Expected tkIdent but got tkKey -``` - -**Result:** The `settings` table could not use `key varchar(100) primary key`. - -**Fix:** Added backtick-quoted identifier support in `query/lexer.nim`. Users can now escape reserved keywords using backticks: - -```sql -CREATE TABLE settings( - `key` varchar(100) primary key, - value varchar(500) default '' -); -``` - -Changes made: -- Added `readBacktickIdent()` proc in `lexer.nim` -- Added backtick case in `nextToken()` to recognize `` `identifier` `` as `tkIdent` - ---- - -## 10. Empty String (`""`) Treated as NULL - -**Problem:** BaraDB normalized empty strings to `NULL` internally. A column defined as `NOT NULL` rejected empty string inserts, even when the application explicitly passed `''`. - -**Result:** SMTP settings saved via the admin panel failed with constraint violations when fields were left blank: - -```sql -INSERT INTO settings (skey, value) VALUES ('smtpUser', ''); --- ERROR: NOT NULL constraint violation -``` - -**Fix:** Changed internal NULL representation from empty string `""` to `\N` in `query/executor.nim`: - -1. `isNull()` now checks for `\N` instead of `value.len == 0` -2. NULL literals in INSERT/UPDATE store `\N` instead of `""` -3. Missing columns return `\N` instead of `""` -4. `getValue()` returns `\N` for missing fields -5. JOIN padding uses `\N` for unmatched rows -6. Server wire protocol uses `\N` sentinel to send `fkNull` -7. HTTP server sends JSON `null` for NULL values - -This allows empty strings to be stored in NOT NULL columns while still properly supporting NULL values. - ---- - -## Summary Table - -| # | Issue | Location | Type | -|---|-------|----------|------| -| 1 | Comma join parsing | `query/parser.nim` | Parser bug | -| 2 | DEFAULT not evaluated | `query/executor.nim` | Schema bug | -| 3 | Duplicate column overwrite | `query/executor.nim` | Data structure bug | -| 4 | Empty result = no columns | `core/server.nim` | Protocol bug | -| 5 | GROUP BY empty values | `query/executor.nim` | Semantic difference | -| 6 | Inconsistent agg names | `query/executor.nim` | Naming convention | -| 7 | Async client unstable | `clients/nim/src/baradb/client.nim` | Client design | -| 8 | No thread safety | `clients/nim/src/baradb/client.nim` | Adapter design | -| 9 | `key` reserved keyword | `query/lexer.nim` | Parser limitation | -| 10 | Empty string = NULL | `query/executor.nim` | Data handling bug | - ---- - -*Document version: 2026-05-17* diff --git a/PLAN_DONE.md b/PLAN_DONE.md deleted file mode 100644 index 4948fba..0000000 --- a/PLAN_DONE.md +++ /dev/null @@ -1,190 +0,0 @@ -# BaraDB — Production Roadmap (Minimalist) - -> **Goal:** Get BaraDB to production-ready state without feature creep. Only fix what blocks real usage. - ---- - -## What Works Now (v0.2.0) - -**Core:** -- CREATE TABLE / INDEX / VIEW / TRIGGER / USER / POLICY -- SELECT / INSERT / UPDATE / DELETE with WHERE -- JOIN (inner, left, right, full, cross) — fully tested and executed -- GROUP BY / HAVING / ORDER BY / LIMIT / OFFSET -- Aggregate functions (COUNT, SUM, AVG, MIN, MAX) -- CTE (WITH clause) — parsed; non-recursive execution via subqueries -- Constraints (PK, FK, UNIQUE, NOT NULL, CHECK, DEFAULT) -- B-Tree indexes + query planner + index point-read optimization + range scans (BETWEEN, >, <, >=, <=) -- MVCC transactions (BEGIN / COMMIT / ROLLBACK / savepoints) -- Deadlock detection (wait-for graph, auto-abort victim) — wired into TxnManager -- WAL crash recovery (REDO + UNDO) -- SSTable compaction (manual + background loop) — started on server boot - -**Connectivity:** -- TCP wire protocol with typed binary values (int/float/bool/string/null) -- HTTP REST API (query, health, metrics, auth) -- WebSocket real-time (SUBSCRIBE / broadcasts) -- JWT authentication (HTTP + TCP + WebSocket handshake) -- Admin Dashboard (SQL playground, table browser, live events, metrics) -- TLS/SSL for TCP (OpenSSL-backed, self-signed cert generation) -- Connection limits (max connections enforced + idle timeout) -- Slow query log (configurable threshold, file-based) - -**Operations:** -- Config file loading (`baradb.json`) + environment variables -- Structured JSON logging with configurable level/file/format -- Docker + Docker Compose (dev + production) -- Backup/restore via tar.gz - -**Advanced:** -- Row-Level Security (policies, GRANT/REVOKE) -- Schema migrations (UP/DOWN, checksums, locking, dry-run) -- UTF-8 identifiers + data -- Nim/Python/Rust/JS client SDKs with full DATA decoding -- Prepared statements / parameterized queries (placeholder + wire protocol params) - ---- - -## Phase A: Critical SQL Execution ✅ - -### A.1 JOIN Execution ✅ -- JOIN chain built in `lowerSelect`, executed as nested-loop in `executePlan` -- Inner / left / right / full / cross all supported and tested -- Aliased column projection + JOIN with aggregates tested - -### A.2 CTE Execution 🟡 -- WITH clause is parsed and stored in AST -- Non-recursive CTE works via subquery execution path -- Recursive CTE execution not yet implemented - ---- - -## Phase B: Production Safety ✅ - -### B.1 TLS/SSL ✅ -- Real OpenSSL-backed TLS (not mock) -- `protocol/ssl.nim` uses Nim's `SslContext` (`newContext`, `wrapConnectedSocket`) -- Self-signed cert generation via openssl CLI -- Certificate validation (fingerprint, expiry, info parsing) - -### B.2 Prepared Statements / Parameterized Queries ✅ -- `nkPlaceholder` in parser/lexer -- `bindParams` in executor replaces placeholders with typed WireValues -- Wire protocol `mkQueryParams (0x03)` sends query + typed params -- Tested: SELECT with placeholders, INSERT with placeholders, multiple placeholders - -### B.3 Deadlock Detection Wiring ✅ -- `deadlock.nim` imported into `mvcc.nim` -- Wait-for graph built on write-write conflict -- `hasDeadlock()` / `findDeadlockVictim()` called; victim auto-aborted -- Cleanup on commit / abort - ---- - -## Phase C: Operational Stability ✅ - -### C.1 Background Compaction Scheduling ✅ -- `CompactionManager` wired into `main()` via `asyncCheck cm.startCompactionLoop()` -- Size-tiered compaction strategy with periodic ticks - -### C.2 Connection Limits + Timeouts ✅ -- `maxConnections` enforced in `server.run()` accept loop -- `activeConnections` tracked (increment on connect, decrement on disconnect) -- Idle timeout: `recvExactWithTimeout` wraps header/payload reads -- Config: `idleTimeoutMs` (default 5 min), `queryTimeoutMs` (reserved for async queries) - -### C.3 Slow Query Log ✅ -- `slowQueryThresholdMs` config (default 1000ms) -- `slowQueryLogPath` config (empty = disabled) -- Queries exceeding threshold logged to file with timestamp, client ID, duration, query text - -### C.4 Config File Loading ✅ -- `baradb.json` support with nested sections (server, storage, tls, auth, logging, performance) -- Environment variable overrides (`BARADB_ADDRESS`, `BARADB_PORT`, `BARADB_DATA_DIR`, etc.) -- Priority: defaults → JSON file → env vars - -### C.5 Structured JSON Logging ✅ -- `logging.nim` module with `debug/info/warn/error` levels -- Configurable via `logLevel`, `logFile`, `logFormat` -- Integrated into `server.nim` and `baradadb.nim` - ---- - -## Phase D: Nice-to-Have (Post-Production) - -| Feature | Why Skip for Now | -|---------|-----------------| -| Partitioning | Complex, small DBs don't need it | -| Full-text search SQL | Engine exists; can use `LIKE` for MVP | -| Point-in-time recovery | Backup/restore covers 90% of cases | -| Kubernetes Helm | Docker Compose is enough for solo-dev target | -| OpenTelemetry tracing | Logs + metrics are enough for v1 | -| Multi-column indexes | Point reads cover most web queries | -| Covering index optimization | Premature optimization | -| Recursive CTE | Rarely used in web apps | - ---- - -## Honest Assessment - -**Current score: 9.7/10** — all production blockers resolved. - -**Remaining polish items (not blockers):** -1. Recursive CTE execution (WITH RECURSIVE) -2. Column type metadata in wire protocol serialization (currently inferred heuristically) - -**Total estimated work: ~1 focused session.** - -BaraDB is production-ready for blogs, e-commerce, and small ERP systems. -262 tests across 56 suites — all passing. Stress test: 10000 ops, 0 errors, 555K ops/sec. - ---- - -## 2026-05-07: Formal Verification v1.0.0 - -### TLA+ Спецификации (7 спека, 11.6M състояния, 0 грешки) -- `raft.tla` — Raft consensus (election safety, leader append-only, state-machine safety) -- `twopc.tla` — Two-Phase Commit (atomicity, coordinator consistency, no orphan blocks) -- `mvcc.tla` — MVCC/Snapshot Isolation (no dirty reads, read-own-writes, write-write conflict) -- `replication.tla` — Replication modes (monotonic LSN, sync durability, semi-sync quorum) -- `gossip.tla` — SWIM Gossip (alive-not-falsely-dead, incarnation monotonicity) -- `deadlock.tla` — Deadlock Detection (graph integrity, no self-loops) -- `sharding.tla` — Consistent Hash Sharding (virtual node mapping, assignment consistency) - -### Оставащи FV задачи (виж `PLAN.md`) -- Raft: prevLogIndex/prevLogTerm + LogMatching (критичен gap в модела) -- 2PC: Coordinator crash/recovery + participant timeout -- MVCC: Write skew detection -- Нови спекове: backup.tla, recovery.tla, crossmodal.tla -- CI: Поправка на verify job (container → setup-java) -- Symmetry reduction + Liveness properties - ---- - -## 2026-05-07: Medium Priority Batch - -### JSON/JSONB Types ✅ -- `validateType` валидира JSON при INSERT/UPDATE -- `fkJson = 0x0D` добавен в `FieldKind` -- `valueToWire` връща `fkJson` за JSON/JSONB колони -- Всички 4 клиента обновени за JSON сериализация - -### Column Type Metadata в Wire Protocol ✅ -- `typeToFieldKind` helper в `server.nim` -- `serializeResult` изпраща `colType` bytes след имената на колоните -- Клиентите четат и пазят `columnTypes` - -### Multi-Column Indexes ✅ -- Parser: `CREATE INDEX idx ON t(a, b)` чете списък от колони -- AST: `nkCreateIndex.ciColumns: seq[string]` -- Executor: ключ `table.col1.col2`, стойност `val1|val2` -- INSERT/UPDATE поддържат composite keys -- SELECT: multi-column exact match за AND chain - -### CTE Execution (non-recursive) ✅ -- `tkRecursive` токен в lexer -- `parseWith` поддържа `WITH RECURSIVE` -- AST: `selWith: seq[(string, Node, bool)]` -- Executor: materialization в `ctx.cteTables` -- `execScan` проверява CTE store преди LSM -- `defer: ctx.cteTables.clear()` за cleanup diff --git a/PLAN_ID_GENERATORS.md b/PLAN_ID_GENERATORS.md deleted file mode 100644 index 6e8c752..0000000 --- a/PLAN_ID_GENERATORS.md +++ /dev/null @@ -1,74 +0,0 @@ -# Plan: ID Generators & Sequence System - -## Goal -Add auto-generated ID support to BaraDB so users don't need to manually supply IDs on INSERT. - -## Phase 1: ID Generators - -### 1.1 AUTO_INCREMENT on INTEGER columns -- Add `AUTO_INCREMENT` keyword to lexer -- Parse in CREATE TABLE: `id INTEGER PRIMARY KEY AUTO_INCREMENT` -- Store auto-increment state per table in ExecutionContext (counter) -- On INSERT without explicit ID → auto-populate with next value -- Thread-safe counter (atomic increment) - -### 1.2 SERIAL / BIGSERIAL as syntactic sugar -- `SERIAL` = `INTEGER AUTO_INCREMENT` -- `BIGSERIAL` = `BIGINT AUTO_INCREMENT` -- Already partially parsed — wire to auto-increment logic - -### 1.3 UUID generation -- Add `gen_random_uuid()` or `uuid()` as built-in function -- Can be used in INSERT: `INSERT INTO t (id) VALUES (uuid())` -- Also usable as DEFAULT: `id UUID DEFAULT uuid()` -- Use Nim's std/oids or crypto random - -### 1.4 RETURNING clause -- After INSERT, return generated values -- `INSERT INTO t (name) VALUES ('x') RETURNING id` -- Already partially parsed — wire to execution - -### 1.5 CREATE SEQUENCE / nextval / currval -- `CREATE SEQUENCE seq_name START 1 INCREMENT 1` -- `nextval('seq_name')` → returns next value -- `currval('seq_name')` → returns current value -- Store sequences in ExecutionContext - -### 1.6 Snowflake ID (distributed) -- 64-bit ID = timestamp(41) + node_id(10) + sequence(12) -- `snowflake_id(node_id)` function -- For future distributed use - -## Phase 2: JOIN Optimizations ✅ - -### 2.1 Hash Join ✅ -- For equi-join ON a.col = b.col -- Build hash table on smaller side, probe with larger -- O(N+M) instead of O(N*M) - -### 2.2 Index Nested Loop Join ✅ -- If index exists on join column → probe index per left row -- O(N * log M) instead of O(N*M) - -### 2.3 Merge Join -- For sorted inputs -- Two-pointer sweep O(N+M) -- **Status:** Not yet implemented — can be added if needed - -## Phase 3: Foreign Key Enforcement ✅ - -### 3.1 CASCADE DELETE ✅ -### 3.2 SET NULL on delete ✅ -### 3.3 RESTRICT on delete ✅ -### 3.4 ON UPDATE CASCADE ✅ -### 3.5 ON UPDATE SET NULL ✅ -### 3.6 ON UPDATE RESTRICT ✅ -### 3.7 FK check on UPDATE (not just INSERT) ✅ - -## Implementation Order -1. AUTO_INCREMENT (lexer + parser + executor) -2. SERIAL/BIGSERIAL sugar -3. UUID function -4. RETURNING clause -5. Sequences (CREATE SEQUENCE / nextval / currval) -6. Snowflake ID function diff --git a/PLAN_SQL_ADVANCED.md b/PLAN_SQL_ADVANCED.md deleted file mode 100644 index 0e2fcbe..0000000 --- a/PLAN_SQL_ADVANCED.md +++ /dev/null @@ -1,351 +0,0 @@ -# BaraDB — Универсален план за Advanced SQL Engine - -> **Визия**: BaraDB е самостоятелен, универсален SQL engine с Nim ядро, поддържащ модерни SQL:2023 разширения — Property Graph, Vector Search, JSON документи и прозоречни функции, в една вградена или клиент/сървър конфигурация. -> -> **Принцип**: Само основи. Не се добавят нови светове — само стабилизираме и документираме съществуващите. -> -> **Multi-Tenant фокус**: BaraDB е проектирана да поддържа ERP сценарии с много фирми (tenants) в една база данни. Всеки tenant се изолира чрез Row-Level Security (RLS) + session variables (`SET app.tenant_id = 'X'`), а не чрез отделни бази. - ---- - -## История на разработката - -- **Фаза 1 (Base SQL + MVCC + Raft)**: BaraDB core engine -- **Фаза 2 (Advanced SQL)**: Разработена с **Xiaomi Mimo** (`mimo-v2.5-pro`) — Window Functions, MERGE, LATERAL JOIN, Advanced Aggregates, PIVOT/UNPIVOT, SQL/PGQ Property Graph -- **Фаза 3 (Stabilization + Multi-Tenant)**: Текуща — Vector SQL Integration, Session Variables, `current_user`/`current_role`, RLS tenant isolation, тестове, документация - ---- - ---- - -## Част 1: BaraDB Advanced SQL Engine - -### 1.1 Window Functions ✅ ГОТОВО - -Нови AST nodes: `nkWindowExpr`, `nkOverClause`, `nkFrameSpec`. Нов IR plan: `irpkWindow`. - -| Функция | Описание | Статус | -|---------|----------|--------| -| `ROW_NUMBER()` | Пореден номер в партишъна | ✅ | -| `RANK()` / `DENSE_RANK()` | Класиране с/без gaps | ✅ | -| `LEAD(col, n, default)` / `LAG(col, n, default)` | Достъп до съседни редове | ✅ | -| `FIRST_VALUE(col)` / `LAST_VALUE(col)` | Краен елемент във frame | ✅ | -| `NTILE(n)` | Bucket-ване в n части | ✅ | - -Frame поддръжка: `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` ✅ - -Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`, `codegen.nim` -Тестове: 5 теста в `tests/test_all.nim`, всички зелени. - -### 1.2 MERGE / UPSERT ✅ ГОТОВО - -```sql -MERGE INTO inventory AS target -USING updates AS source -ON target.sku = source.sku -WHEN MATCHED THEN UPDATE SET qty = target.qty + source.delta -WHEN NOT MATCHED THEN INSERT (sku, qty) VALUES (source.sku, source.delta); -``` - -- Поддържа таблица или subquery като source -- WHEN MATCHED UPDATE с eval на изрази (target.col + source.col) -- WHEN NOT MATCHED INSERT с eval на value изрази -- Trigger support (BEFORE/AFTER UPDATE/INSERT) - -Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`, `codegen.nim` -Тестове: 2 теста в `tests/test_all.nim`, всички зелени. - -### 1.3 LATERAL JOIN / CROSS APPLY ✅ ГОТОВО - -Позволява correlated subquery във FROM clause с достъп до лявата таблица. - -```sql -SELECT u.name, recent_orders.* -FROM users u, -LATERAL ( - SELECT order_id, total FROM orders o - WHERE o.user_id = u.id ORDER BY created_at DESC LIMIT 3 -) recent_orders; -``` - -- Поддържа `JOIN LATERAL`, `LEFT JOIN LATERAL`, `CROSS JOIN LATERAL` -- Correlated references (e.g. `u.id`) чрез scan + merge + filter стратегия -- Sort и Limit от subquery се прилагат след merge -- LEFT LATERAL запазва unmatched редове с NULL padding - -Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim` -Тестове: 4 execution теста + 3 parser теста, всички зелени. - -### 1.4 Advanced Aggregates ✅ ГОТОВО - -- `ARRAY_AGG(col ORDER BY ...)` -- `STRING_AGG(col, delimiter)` -- `COUNT(*) FILTER (WHERE ...)` -- `GROUPING SETS`, `CUBE`, `ROLLUP` - -#### GROUP BY + HAVING ✅ ГОТОВО - -- SUM/AVG/MIN/MAX оценяват се в групите -- HAVING филтрира групите по aggregate условия -- Pre-computed aggregates се съхраняват в group rows -- evalExpr поддържа irekAggregate lookup - -Тестове: 6 теста в `tests/test_all.nim`, всички зелени. - -#### FILTER (WHERE ...) ✅ ГОТОВО - -```sql -SELECT COUNT(*) FILTER (WHERE active = true) FROM users; -SELECT dept, SUM(amount) FILTER (WHERE amount > 100) FROM sales GROUP BY dept; -``` - -- Parser: `FILTER (WHERE ...)` след aggregate function call -- AST: `funcFilter*: Node` на `nkFuncCall` -- IR: `aggFilter*: IRExpr` на `irekAggregate` -- Executor: филтрира редове преди aggregate computation - -Тестове: 2 execution теста + 1 parser тест, всички зелени. - -#### ARRAY_AGG / STRING_AGG ✅ ГОТОВО - -```sql -SELECT dept, ARRAY_AGG(amount) AS amounts FROM sales GROUP BY dept; -SELECT dept, STRING_AGG(name, ', ') AS names FROM employees GROUP BY dept; -``` - -- Нови IR aggregate ops: `irArrayAgg`, `irStringAgg` -- Multi-argument aggregate parsing (delimiter за STRING_AGG) -- FILTER support за двете функции - -Тестове: 2 теста, всички зелени. - -#### GROUPING SETS / ROLLUP / CUBE ✅ ГОТОВО - -```sql -SELECT dept, SUM(amount) FROM sales GROUP BY ROLLUP (dept); -SELECT dept, job, SUM(amount) FROM sales GROUP BY CUBE (dept, job); -SELECT dept, job, SUM(amount) FROM sales GROUP BY GROUPING SETS ((dept), (job), ()); -``` - -- ROLLUP(a, b) → GROUPING SETS ((a,b), (a), ()) -- CUBE(a, b) → GROUPING SETS ((a,b), (a), (b), ()) -- Генериране на subsets за CUBE чрез powerset алгоритъм - -Тестове: 4 parser теста + 1 execution тест, всички зелени. - -### 1.5 PIVOT / UNPIVOT ✅ ГОТОВО - -```sql -SELECT * FROM (SELECT name, dept, salary FROM emp) -PIVOT (SUM(salary) FOR dept IN ('Eng', 'Sales')); - -SELECT * FROM emp -UNPIVOT (salary FOR dept IN (eng_salary, sales_salary)); -``` - -- Parser: PIVOT/UNPIVOT в FROM clause -- IR: `irpkPivot`, `irpkUnpivot` -- Executor: group by identity cols → aggregate per pivot value → create columns -- Subquery storage в `nkFrom.fromSubquery` - -Тестове: 1 parser + 1 execution тест, всички зелени. - -### 1.6 SQL:2023 Property Graph (SQL/PGQ) ✅ ГОТОВО (Parser) - -```sql -SELECT * FROM GRAPH_TABLE(org_chart - MATCH (e)-[r]->(d) - COLUMNS (e.name, d.name) -); -``` - -- Lexer: `tkVertex`, `tkEdge`, `tkLabels`, `tkGraphTable`, `tkMatch`, `tkColumns`, `tkSrc`, `tkDst` -- AST: `nkGraphTraversal` с `gtGraphName`, `gtReturnCols` -- IR: `irpkGraphTraversal` с `graphName`, `graphAlgo`, `graphReturnCols` -- Executor: table-based graph storage (`graph_nodes`, `graph_edges`) -- Parser: `GRAPH_TABLE(name MATCH (pattern) COLUMNS (cols))` - -Тестове: 1 parser тест, всички зелени. - ---- - -## Част 1.5: Multi-Tenant ERP Support ✅ ГОТОВО - -BaraDB поддържа multi-tenant архитектура, при която множество фирми (tenants) работят в една физическа база данни. Това е критично за ERP сценарии, където поддръжката на "сто бази" не е опция. - -### Механизъм - -| Компонент | Описание | -|-----------|----------| -| **Session Variables** | `SET app.tenant_id = 'company-123'` — задава tenant за текущата сесия | -| **current_setting()** | `current_setting('app.tenant_id')` — чете session променлива в SQL израз | -| **current_user** | `current_user` — връща автентикирания потребител от JWT/SCRAM | -| **current_role** | `current_role` — връща ролята на автентикирания потребител | -| **RLS Policies** | `CREATE POLICY tenant_isolation ON invoices FOR SELECT USING (tenant_id = current_setting('app.tenant_id'))` | -| **Auth Bridge** | `server.nim` и `httpserver.nim` попълват `ExecutionContext.currentUser`/`currentRole` след верификация | - -### Пример - -```sql --- Една таблица за всички фирми -CREATE TABLE invoices ( - id SERIAL PRIMARY KEY, - tenant_id TEXT NOT NULL, - data JSONB -); - --- Изолация чрез RLS -CREATE POLICY tenant_isolation ON invoices - FOR SELECT USING (tenant_id = current_setting('app.tenant_id')); - --- Всяка сесия вижда само своя tenant -SET app.tenant_id = 'company-a'; -SELECT * FROM invoices; -- → само фактури на company-a -``` - -### Архитектурни предимства - -- **JSONB документи** — schema-flexible, лесно се добавят нови полета без миграции (като ArangoDB) -- **RLS изолация** — базата данни гарантира, че всеки tenant вижда само своите данни -- **Един instance** — един BaraDB сървър обслужва всички tenants, вместо сто отделни бази -- **Auth integration** — JWT/SCRAM токените носят `sub` (user) и `role`, които се пропагират до executor-а - ---- - -## Част 2: Мултимодални Възможности (Core Only) - -### 2.1 JSON / JSONB Документи ✅ ГОТОВО - -```sql -SELECT data->>'name' FROM users WHERE data->'tags' @> '["admin"]'; -``` - -- Типове: `JSON`, `JSONB` колони в таблици -- Оператори: `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?&`, `?|` -- Функции: `jsonb_array_elements`, `jsonb_object_keys`, `jsonb_extract_path` -- Съхранение: двоично parsed tree (не plain text) - -### 2.2 Vector Search ⚠️ ЧАСТИЧНО (Engine ✅, SQL Integration 🔄) - -**Вектор Engine (готов):** -- `src/barabadb/vector/engine.nim` — HNSW index с cosine/euclidean distance -- `src/barabadb/vector/quant.nim` — IVF-PQ quantization -- `src/barabadb/vector/simd.nim` — SIMD оптимизации -- `src/barabadb/core/crossmodal.nim` — CrossModalEngine за хибридно търсене (vector + text) - -**Липсваща SQL интеграция (базова — за стабилизация):** -```sql --- Тип и колона -CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768)); - --- Index -CREATE VECTOR INDEX idx_items_vec ON items(embedding) - USING hnsw WITH (m = 16, ef_construction = 200, metric = 'cosine'); - --- Query functions -SELECT id, cosine_distance(embedding, '[0.1, 0.2, ...]') AS dist -FROM items -ORDER BY dist ASC -LIMIT 10; -``` - -**Задачи за стабилизация (всички изпълнени):** -- [x] `VECTOR(n)` тип в CREATE TABLE (parser + storage) -- [x] `CREATE VECTOR INDEX ... USING hnsw` (DDL) -- [x] `cosine_distance()`, `euclidean_distance()`, `inner_product()` в SQL expression evaluator -- [x] `<->` nearest-neighbor оператор в ORDER BY / WHERE -- [x] Executor integration: HNSW index population при CREATE INDEX и DML - -**Статус:** ✅ ГОТОВО. 8 SQL-level vector теста зелени. - -### 2.3 Full-Text Search ✅ ГОТОВО - -- Inverted Index в `src/barabadb/fts/` -- `MATCH(column, query)` функция -- BM25 scoring -- Интеграция с CrossModalEngine за hybrid search - ---- - -## Част 3: Транзакции и Протоколи ✅ ГОТОВО - -- MVCC с snapshot isolation -- WAL + checkpoint -- Distributed transactions (2PC) — `txn.addParticipant("vector")` -- Wire protocol: binary за vectors, JSON за queries - ---- - -## Имплементационен ред (финален статус) - -1. ✅ **Window Functions** (AST → Parser → IR → Executor → Tests) -2. ✅ **MERGE statement** (Parser → Executor → Tests) -3. ✅ **LATERAL JOIN** (Parser → Executor, correlated subquery strategy) -4. ✅ **GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter) -5. ✅ **FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...)) -6. ✅ **ARRAY_AGG / STRING_AGG** (multi-arg aggregates) -7. ✅ **GROUPING SETS / ROLLUP / CUBE** (powerset generation) -8. ✅ **PIVOT / UNPIVOT** (row-to-column transformation) -9. ✅ **SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser) -10. ✅ **JSON/JSONB** (operators + functions) -11. ✅ **Full-Text Search** (inverted index + BM25) -12. ✅ **Vector Engine** (HNSW + IVF-PQ + SIMD) -13. ✅ **Vector SQL Integration** (тип, index, distance functions, <-> operator, ORDER BY) - ---- - -## Крайно състояние - -**340+ теста зелени.** Всички фундаментални SQL:2023 features имплементирани. - -**Четирите свята:** - -| Свят | Features | Статус | -|------|----------|--------| -| **SQL** | Window, MERGE, LATERAL, GROUP BY/HAVING, FILTER, ARRAY_AGG, STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT | ✅ | -| **JSON** | JSON/JSONB колони, `->` / `->>` оператори | ✅ | -| **Graph** | BFS/DFS/PageRank/Dijkstra engine + SQL/PGQ GRAPH_TABLE | ✅ | -| **Vector** | HNSW index, cosine/euclidean distance, IVF-PQ, SIMD | ✅ Engine
🔄 SQL glue | -| **FTS** | Inverted index, BM25, hybrid search | ✅ | - -**Файлове модифицирани:** -- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube, tkVector -- `ast.nim` — joinLateral, funcFilter, nkPivot, nkUnpivot, GroupingSetsKind, nkGraphTraversal fields -- `ir.nim` — joinLateral, aggFilter, irArrayAgg, irStringAgg, IRGroupingSetsKind, irpkGroupBy grouping sets, irpkPivot, irpkUnpivot, irpkGraphTraversal -- `parser.nim` — LATERAL, FILTER, multi-arg aggregates, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE -- `executor.nim` — LATERAL correlated strategy, GROUP BY aggregates + HAVING, FILTER in aggregates, ARRAY_AGG/STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE, fromTable kind checks -- `codegen.nim` — irpkPivot, irpkUnpivot, irpkGraphTraversal -- `tests/test_all.nim` — 25+ нови теста -- `tests/join_tests.nim` — 4 LATERAL теста - ---- - -## Тестова стратегия - -- **Unit**: Всеки нов AST/IR/Parser тест — property-based (генериране на случайни partition/order) -- **Integration**: HTTP server + клиент тестове -- **TLA+**: `windowfunctions.tla` — deterministic partitioning semantics -- **Benchmark**: Window function performance vs PostgreSQL (опционално) - ---- - -## Поправени грешки при тази сесия - -- **Vector SQL Integration** — имплементиран пълен SQL glue за вектори (тип, индекс, функции, оператор) -- **MERGE тестове** — поправени чрез изолиране на тестовата директория (unique temp dir per suite) -- **Row storage escape** — `escapeRowVal()` в `execInsert` за стойности със запетай (vector literals) -- **ORDER BY + projection** — `irpkSort` сега е преди `irpkProject` в `lowerSelect`, което позволява `ORDER BY` по колони извън `SELECT` -- **GROUPING SETS execution** — `lowerSelect` сега проверява `selGroupingSetsKind != gskNone` освен `selGroupBy.len > 0`, което позволява изпълнение на GROUPING SETS без традиционен GROUP BY -- **FTS CREATE INDEX docId** — поправено несъответствие в изчислението на `docId` при `CREATE INDEX ... USING FTS` (сега използва хеш на `tableName.$key`, съвместим с DML операциите) -- **Тестова изолация (всички suite-ове)** — всички `newLSMTree("")` заменени с уникални temp директории; setup/teardown за suite-ове с изолирана state -- **Multi-tenant ERP support** — имплементирани критични градивни елементи: - - `SET var = value` — session variables за tenant isolation - - `current_setting('var')` — четене на session променливи в SQL изрази - - `current_user` / `current_role` — SQL keywords, които се оценяват от `ExecutionContext` - - Auth bridge — `server.nim` и `httpserver.nim` попълват `currentUser`/`currentRole` след JWT/SCRAM верификация - - RLS tenant isolation тест — `CREATE POLICY` + `current_setting('app.tenant_id')` работи за multi-tenant филтрация - - `evalExpr` вече предава `ctx` рекурсивно — поправен бъг, при който `current_user`/`current_setting` връщаха празни стойности в под-изрази - ---- - -> **Бележка**: Този план е *замразен* за нови светове. Следващата работа е само стабилизация на съществуващото и документация. diff --git a/PLAN_STABILIZATION.md b/PLAN_STABILIZATION.md deleted file mode 100644 index 1a6ae9b..0000000 --- a/PLAN_STABILIZATION.md +++ /dev/null @@ -1,149 +0,0 @@ -# BaraDB — Storage Stabilization Roadmap - -> **Визия**: Да превърнем BaraDB от "feature-rich но storage-fragile" в "production-hardened" база данни, като заемем Btrieve/MicroKernel философията за управление на persistent файлове, адаптирана към LSM-tree архитектурата. -> -> **Принцип**: Всеки файл на диска трябва да може да се самопровери, самопоправи и мигрира версия — без да губим данни. - ---- - -## Текущи проблеми (май 2026) - -| Проблем | Описание | Риск | -|---------|----------|------| -| SSTable без checksum | Няма CRC/xxhash — битова грешка в 2GB файл се открива едва при грешни данни | **Висок** | -| Backup = `tar.gz` | Не е consistent, не е online, няма incremental | **Висок** | -| WAL е един файл | `wal.log` расте безкрайно — няма ротация, няма archive | **Среден** | -| Няма repair утилитка | При повреда единствената опция е restore от backup | **Висок** | -| Няма MANIFEST | LSM-tree няма каталог на SSTable-овете — риск от orphan файлове | **Среден** | -| SSTable v1/v2 без migration path | Поддържаме четене на стари версии, но не мигрираме към нови | **Нисък** | - ---- - -## Фаза 1: SSTable Integrity (CRC Footer + Strict Validation) - -> **Цел**: Всеки SSTable файл да може да се верифицира независимо. - -| # | Задача | Описание | Оценка | Статус | -|---|--------|----------|--------|--------| -| 1.1 | CRC32 модул | `storage/crc32.nim` — zero-dep CRC32 (IEEE 802.3), използван от SSTable и WAL. | 2ч | ✅ | -| 1.2 | SSTable v3 формат | Footer с `dataCrc32`, `indexCrc32`, `bloomCrc32`. Header получава `footerOffset`. | 3ч | ✅ | -| 1.3 | `verifySSTable()` | `proc verifySSTable(path): (bool, string)` — проверява magic, version, CRC. | 2ч | ✅ | -| 1.4 | `loadSSTable` strict mode | При load на v3 проверява CRC; при несъвпадение raise с ясно съобщение. Поддържа v1/v2/v3. | 2ч | ✅ | -| 1.5 | `newLSMTree` corruption logging | При load на SSTables от диск — логва кой файл е корумпиран вместо `except: discard`. | 1ч | ✅ | - -**Метрика**: `verifySSTable()` открива единична битова грешка в 4GB SSTable за под 100ms. ✅ Проверено — unit тестове в `test_all.nim`. - ---- - -## Фаза 2: Storage Repair Tool (`baradb repair`) - -> **Цел**: Btrieve-style `BUTIL -RECOVER` — сканира, проверява, поправя на място. - -| # | Задача | Описание | Оценка | Статус | -|---|--------|----------|--------|--------| -| 2.1 | `baradb repair` CLI модул | `src/barabadb/tools/repair.nim` + CLI entry point в `baradadb.nim`. | 3ч | ✅ | -| 2.2 | Scan phase | Рекурсивно обхожда `data/server/sstables/*.sst`, изпълнява `verifySSTable()` на всеки. | 2ч | ✅ | -| 2.3 | Index rebuild | Ако data block е валиден, но index map-ът е бит — регенерира `index` от data block-а. | 4ч | ⏳ Отложено за Фаза 6 — rare edge case | -| 2.4 | WAL replay integration | Пуска `CrashRecovery` с REDO/UNDO след SSTable repair. | 2ч | ✅ | -| 2.5 | Orphan cleanup | Премества корумпирани SSTables в `/corrupt/`. | 2ч | ✅ | -| 2.6 | Repair report | Текстов отчет: кои файлове са OK, кои са изтрити, колко записи са спасени. | 2ч | ✅ | - -**Метрика**: `baradb repair --data-dir ./data` завършва за под 30 секунди на 100GB data dir. ✅ Проверено — възстановява данни след изтрит корумпиран SSTable. - ---- - -## Фаза 3: MANIFEST File (Consistent SSTable Catalog) - -> **Цел**: LSM-tree да има един източник на истина за това кои SSTable-ове са активни. - -| # | Задача | Описание | Оценка | Статус | -|---|--------|----------|--------|--------| -| 3.1 | MANIFEST формат | JSON файл `data/server/MANIFEST` с: `sequence`, `sstables[]` (id, path, level, minKey, maxKey). | 3ч | ✅ | -| 3.2 | Write MANIFEST | При flush/compaction — atomic write до `MANIFEST.tmp` + `moveFile` към `MANIFEST`. | 2ч | ✅ | -| 3.3 | Read MANIFEST | `newLSMTree` зарежда SSTables от MANIFEST вместо `walkDir`; fallback към scan при липсващ MANIFEST. | 2ч | ✅ | -| 3.4 | Consistency check | `checkStorageConsistency()` сравнява MANIFEST с файловете на диска; докладва orphans и missing. | 2ч | ✅ | - -**Метрика**: При crash по време на compaction — възстановяването е детерминистично, без orphan SSTables. ✅ Проверено — unit тестове в `test_all.nim`. - ---- - -## Фаза 4: WAL Rotation & Incremental Backup - -> **Цел**: Point-in-time recovery чрез WAL archiving, както при PostgreSQL/Btrieve. - -| # | Задача | Описание | Оценка | Статус | -|---|--------|----------|--------|--------| -| 4.1 | WAL segment rotation | `wal.log` → `wal..log` при достигане на 64MB. Проверява на всеки 1000 записа + при flush. | 3ч | ✅ | -| 4.2 | WAL archive директория | `data/server/wal/wal_archive/` — пази затворени сегменти. | 2ч | ✅ | -| 4.3 | Archive cleanup | Пази всички сегменти до изрично cleanup (отложено за административна команда). | 2ч | ⏳ | -| 4.4 | Incremental backup | `backup incremental` — архивира MANIFEST + активни SSTables + WAL (текущ + archive). | 4ч | ✅ | -| 4.5 | Point-in-time recovery | `RECOVER TO TIMESTAMP '...'` — replay-ва WAL до посочения момент. | 4ч | ⏳ | - -**Метрика**: Incremental backup на 500GB база след първоначален full backup е под 5GB (само WAL + delta SSTables). ✅ Проверено — архивира само необходимите файлове. - ---- - -## Фаза 5: Online Consistent Backup - -> **Цел**: Backup без спиране на сървъра, с гарантирана consistency. - -| # | Задача | Описание | Оценка | Статус | -|---|--------|----------|--------|--------| -| 5.1 | Memtable freeze | `checkpoint()` — freeze memtable, flush до SSTable, ротация на WAL. | 3ч | ✅ | -| 5.2 | Snapshot backup | `incrementalBackupDataDir` копира MANIFEST + SSTables + WAL сегменти. | 3ч | ✅ | -| 5.3 | `baradb backup --online` | CLI флаг — checkpoint + incremental backup. | 2ч | ✅ | -| 5.4 | Backup verification | `incrementalBackupDataDir` проверява CRC на всички SSTables преди архивиране. | 2ч | ✅ | - -**Метрика**: Online backup не блокира writes за повече от 100ms (времето за freeze + WAL ротация). ✅ Проверено — checkpoint freeze-ва memtable под lock, flush-ът е извън lock. - ---- - -## Фаза 6: SSTable Version Migration (Background) - -> **Цел**: Стари SSTable версии автоматично да се мигрират към нови при compaction. - -| # | Задача | Описание | Оценка | Статус | -|---|--------|----------|--------|--------| -| 6.1 | Compaction migration | `compact()` вече пише v3 на изхода независимо от входната версия. | 2ч | ✅ | -| 6.2 | Offline migration job | `baradb migrate` — сканира и пренаписва всички v1/v2 SSTables към v3. | 3ч | ✅ | -| 6.3 | Version tracking | `SSTable.fileVersion` поле — load/write го попълват; `listLegacySSTables` ги намира. | 1ч | ✅ | - -**Метрика**: След 6 месеца uptime, 100% от SSTable-овете са v3 (ако compaction работи редовно). ✅ Проверено — `migrateSSTable` пренаписва v2 → v3 с валидиране на данните. - ---- - -## Приоритети - -``` -Фаза 1 (CRC) ──→ Фаза 2 (Repair) ──→ Фаза 3 (MANIFEST) - │ │ - ↓ ↓ -Фаза 6 (Migration) Фаза 4 (WAL Archive) - │ │ - └──────────────┬─────────────────────┘ - ↓ - Фаза 5 (Online Backup) -``` - -**Препоръчителен ред:** -1. **Фаза 1** — Без CRC няма смисъл от repair; това е фундаментът. -2. **Фаза 2** — Repair утилитката дава веднага production стойност. -3. **Фаза 3** — MANIFEST прави backup/repair детерминистични. -4. **Фаза 4** — WAL ротация дава incremental backup и PITR. -5. **Фаза 5** — Online backup е връхът на стабилизацията. -6. **Фаза 6** — Background migration е "nice to have" за дългосрочна поддръжка. - ---- - -## Философия - -> **Btrieve философия в LSM-tree свят:** -> - Всеки файл носи версия и може да се провери сам. -> - Отделен repair инструмент, не вграден в ядрото. -> - Transaction log е свещен — ротира се, архивира се, replay-ва се. -> - MANIFEST е източникът на истина — не файловата система. -> - Backup не е `tar`, а consistent snapshot на логично време. - ---- - -*План версия: 2026-05-18* diff --git a/PLAN_old_1.md b/PLAN_old_1.md deleted file mode 100644 index b1b73b3..0000000 --- a/PLAN_old_1.md +++ /dev/null @@ -1,155 +0,0 @@ -# План за подобряване на BaraDB - -## Цел -Превърне BaraDB от добър proof-of-concept в солиден, изпълним проект с реална дълбочина на критичните компоненти. - ---- - -## Фаза 1: Честност и стабилна основа (1–2 седмици) ✅ ЗАВЪРШЕНА - -### 1.1 Поправи `README.md` да отразява реалното състояние -- ✅ Добавена секция **"Current Status / Limitations"** с конкретни бележки: - - LSM-Tree SSTable четене е placeholder - - HNSW search е линейно сканиране (O(N)) - - TCP сървърът връща само "OK", без execution - - Raft няма мрежов транспорт - - Graph/FTS/Columnar са in-memory само -- ✅ Променена сравнителната таблица с EdgeDB — маркиран като "в разработка / експериментален" - -### 1.2 Поправи компилацията на benchmark-ите -- ✅ В `benchmarks/bench_all.nim`: заменено `(getMonoTime() - start).ticks` с `(getMonoTime() - start).inNanoseconds` -- ✅ Добавен `import std/times` -- ✅ Benchmark-ът се компилира и изпълнява успешно - -### 1.3 Имплементирай реално SSTable четене в `storage/lsm.nim` -**Беше:** `db.get()` намираше ключа в `sst.index`, но връщаше `(true, @[])` — празен масив. - -**Сега:** -- ✅ Дефиниран бинарен SSTable формат (Header → Data Block → Index Block → Bloom Filter Block) -- ✅ Имплементиран `writeSSTable()` — сериализира MemTable към `.sst` файл -- ✅ Имплементиран `loadSSTable()` — зарежда съществуващ `.sst` файл чрез `mmap` -- ✅ Имплементиран `readSSTableEntry()` — чете конкретен ключ от mmap-нат файл -- ✅ `flush()` вече наистина пише SSTable файл -- ✅ `newLSMTree()` вече зарежда съществуващи SSTables при стартиране -- ✅ Добавени `serialize`/`deserialize` на `BloomFilter` за персистентност -- ✅ Поправен `mmap.nim` да използва `posix.open` вместо грешния `system.open` -- ✅ Всички 214 теста минават -- ✅ Persistence тест: write → flush → close → reopen → read работи коректно - ---- - -## Фаза 2: Дълбочина на core engine-ите (2–4 седмици) - -### 2.1 Реализирай истински HNSW search в `vector/engine.nim` ✅ ЗАВЪРШЕНА -**Беше:** `search()` правеше линейно сканиране на всички нодове. - -**Резултат:** -- ✅ Имплементиран `randomLevel()` с геометрично разпределение -- ✅ Имплементиран `searchLayer()` — жадно разширяване на кандидати с `ef` лъч -- ✅ Имплементиран `selectNeighbors()` + `addBidirectionalLink()` с degree pruning -- ✅ `insert()` изгражда йерархичен граф ниво по ниво -- ✅ `search()` слиза от `maxLevel` до 0, рефинирайки entry point -- ✅ `searchWithFilter()` с пост-филтриране на метаданни -- ✅ Тестовете за HNSW (insert/search/filter) минават; benchmark-ът с 10K вектора работи - -### 2.2 Интегрирай wire protocol в TCP сървъра ✅ ЗАВЪРШЕНА -**Беше:** `core/server.nim` връщаше `"OK\n"` за всяка заявка. - -**Резултат:** -- ✅ Сървърът чете 12-byte бинарен header (`kind`, `length`, `requestId`) и payload -- ✅ Имплементиран `recvExact` за надеждно message framing -- ✅ SELECT: point read (WHERE key = '...') и full memTable scan -- ✅ INSERT: парсва EdgeDB-style синтаксис и записва в LSM-Tree -- ✅ DELETE: извлича ключ от WHERE и извиква `db.delete()` -- ✅ Отговори чрез `mkData`, `mkComplete`, `mkError`, `mkPong` -- ✅ Добавен `scanMemTable()` в LSM-Tree за пълни сканирания -- ✅ Всички тестове минават (214+) - -### 2.3 Добави персистентност на поне един от Graph/FTS/Columnar ✅ ЗАВЪРШЕНА -**Изпълнено за Graph engine.** - -**Резултат:** -- ✅ Дефиниран бинарен формат с magic bytes (`BGRF`) и version -- ✅ `saveToFile(path)` сериализира nodes, edges, properties, weights, next IDs -- ✅ `loadFromFile(path)` реконструира графа и adjacency lists от edges -- ✅ Тест "Save and load graph": 3 nodes, 2 edges, properties, shortest path — round-trip успешен -- ✅ Всички тестове минават (215+) - ---- - -## Фаза 3: Production hardening (2–3 седмици) - -### 3.1 Thread-safety и concurrency ✅ ЗАВЪРШЕНА -- ✅ LSM-Tree: конвертиран от `object` на `ref object`, добавен `std/locks.Lock` - - Всички публични операции (`put`, `delete`, `get`, `contains`, `flush`, `close`, `scanMemTable`) са опаковани с `acquire`/`release` - - Премахнато неизползваното `readLocks: int` поле - - Поправени сигнатури в `core/server.nim` от `var LSMTree` на `LSMTree` -- ✅ Graph: добавен `std/locks.Lock` в `Graph` (вече беше `ref object`) - - Всички публични операции (`addNode`, `addEdge`, `removeNode`, `getNode`, `getEdge`, `neighbors`, `inNeighbors`, `bfs`, `dfs`, `shortestPath`, `dijkstra`, `pageRank`, `nodeCount`, `edgeCount`, `saveToFile`) са опаковани с `acquire`/`release` - - Поправени вътрешни deadlock-ове (напр. `bfs`/`dfs`/`shortestPath` вече не извикват `neighbors`, а директен достъп до `adjacency`) -- ✅ Stress тестът (`tests/stress_test.nim`) вече използва `std/threadpool` с `spawn` за паралелни worker-и - - 10 worker-а × 1000 ops, 0 грешки, ~833K ops/sec - -### 3.2 Raft мрежов транспорт ✅ ЗАВЪРШЕНА -- ✅ Добавен `RaftNetwork` тип в `core/raft.nim` с `asyncdispatch` + `asyncnet`: - - `run()` — слуша за входящи Raft RPC връзки на `raftPort` - - `send(peerId, msg)` — изпраща сериализирано съобщение към пиър с persistent TCP socket - - `broadcast(msgs)` — изпраща на всички пиъри - - `receiveLoop(client)` — framed read (4-byte big-endian length + payload), диспатчва към `handleRequestVote`/`handleAppendEntries` - - `heartbeatLoop()` — лидер изпраща `AppendEntries` heartbeat на всеки `heartbeatTimeout` - - `stop()` — graceful shutdown -- ✅ Бинарна сериализация на `RaftMessage` с magic bytes (`RAFT`) + version - - `serialize()` / `deserializeRaftMessage()` чрез `std/streams` - - Поддържа `LogEntry`, `seq[LogEntry]`, всички reply полета -- ✅ Peer addressing: `RaftNode.peerAddrs: Table[string, (host, port)]` + `raftPort` -- ✅ Интеграция с `ElectionTimer`: - - `tick(timer, net)` приема optional `RaftNetwork` - - При follower/candidate timeout се изпращат реални `RequestVote` съобщения по TCP -- ✅ Тест "3-node election over TCP" — 3 нода на портове 19001/19002/19003, проверява че точно 1 става лидер - -### 3.3 CI/CD и качество ✅ ЗАВЪРШЕНА -- ✅ Създаден `.github/workflows/ci.yml`: - - Пуска `nim c --path:src -r tests/test_all.nim` на всеки push/PR - - Компилира `benchmarks/bench_all.nim` в release режим - - Компилира и пуска `tests/stress_test.nim` - - Проверява за `XDeclaredButNotUsed` и `UnusedImport` като GitHub Actions annotations -- ✅ Създаден `tests/stress_test.nim`: - - 10 worker-а, всеки прави 1000 произволни put/get/delete операции - - Всеки worker използва собствена LSMTree инстанция (до Phase 3.1 за thread-safety) - - Проверка за data corruption — 0 грешки при 10 000 ops (~143K ops/sec) - -### 3.4 Изчисти проекта ✅ ЗАВЪРШЕНА -- ✅ Изтрити broken/unused файлове: - - `src/barabadb/protocol/http.nim` (unused, 5 compile errors) - - `src/barabadb/protocol/tls.nim` (unused, 25 errors, overlaps with `ssl.nim`) - - `src/barabadb/protocol/websocket.nim` (unused, 14 errors) - - `src/barabadb.nim` (6-line dead re-export stub) -- ✅ Премахнати ~20 unused imports от `src/` модулите -- ✅ Добавени build artifacts в `.gitignore` (`*.out`, тест/бенчмарк бинарни файлове) - ---- - -## Приоритетна матрица - -| Задача | Влияние | Трудност | Приоритет | -|--------|---------|----------|-----------| -| SSTable реално четене | Критично | Средна | P0 ✅ | -| README честност | Високо | Ниска | P0 ✅ | -| HNSW истински search | Високо | Висока | P1 ✅ | -| Wire protocol в сървъра | Високо | Средна | P1 ✅ | -| Benchmark fix | Ниско | Ниска | P2 ✅ | -| Graph персистентност | Средно | Ниска | P2 ✅ | -| Raft мрежа | Средно | Висока | P2 ✅ | -| Thread-safety | Средно | Средна | P2 ✅ | -| CI/CD | Средно | Ниска | P3 ✅ | -| Изчистване на проекта | Ниско | Ниска | P3 ✅ | - ---- - -## Очакван резултат след изпълнение - -- **Фаза 1:** Проектът е честен, стабилен и benchmark-ите работят. LSM-Tree е валиден key-value store. ✅ -- **Фаза 2:** HNSW работи с реален approximate search. Сървърът изпълнява заявки. Има персистентност. ✅ -- **Фаза 3:** Многонишкова безопасност, CI, по-чист код. ✅ - -**Крайна оценка след плана:** от 6.5/10 към 8.5/10. diff --git a/PLAN_old_2.md b/PLAN_old_2.md deleted file mode 100644 index c4258f0..0000000 --- a/PLAN_old_2.md +++ /dev/null @@ -1,355 +0,0 @@ -# BaraDB — Production Roadmap (Web & ERP) - -## Vision -BaraDB as a production-ready database for: -- **Web applications** (blogs, e-commerce, SaaS) -- **Small ERP systems** (CRM, warehouse, accounting, invoicing) - -> Target user: solo-dev / small team wanting a fast local DB without PostgreSQL/MySQL dependency. - ---- - -## Current State (Baseline) - -| Component | Status | -|-----------|--------| -| LSM-Tree KV store | Stable, thread-safe, persistent | -| HNSW vector search | Working, recall > 0.9 | -| TCP wire protocol | Binary, SELECT/INSERT/DELETE | -| Raft consensus | TCP transport, leader election | -| Graph engine | In-memory + persistence | -| CI/CD | GitHub Actions | -| Test suite | 60 suites, ~250 tests | - -**Critical gaps for production:** -- Server bypasses IR/codegen/MVCC/schema — `executeQuery()` does lex→parse→raw LSMTree calls -- INSERT parser incomplete (no VALUES, column list, RETURNING) -- No CREATE TABLE/ALTER TABLE/DROP TABLE in parser -- MVCC not wired to query path (no BEGIN/COMMIT/ROLLBACK in server) -- B-Tree indexes not integrated with LSM-Tree -- SQL schema system not connected (EdgeQL types only) -- No HTTP REST API -- Auth not wired to server -- No UTF-8 support for identifiers - ---- - -## Phase 0: Pipeline Integration & Parser Completion ✅ DONE - -### 0.1 Complete DML parser (INSERT/UPDATE/DELETE) -- INSERT with column list: `INSERT INTO t (c1, c2) VALUES (v1, v2)` ✅ -- INSERT with RETURNING clause ✅ (parsed, executor returns data) -- UPDATE with RETURNING clause ✅ (parsed) -- DELETE with RETURNING clause ✅ (parsed) -- Multiple VALUES rows: `VALUES (v1), (v2), ...` ✅ - -### 0.2 Add SQL DDL to parser -- `CREATE TABLE` with column definitions, constraints (PK, FK, UNIQUE, NOT NULL, CHECK, DEFAULT) ✅ -- `ALTER TABLE ADD COLUMN` ✅ -- `DROP TABLE` ✅ -- `CREATE INDEX` / `CREATE UNIQUE INDEX` ✅ -- Tokens: tkCreate, tkTable, tkAlter, tkColumn, tkPrimary, tkKey, tkForeign, tkReferences, tkCascade, tkUnique, tkNotNull, tkCheck, tkDefault, tkRename, tkAdd, tkDrop ✅ - -### 0.3 SQL-compatible schema system -- SQL table catalog (separate from EdgeQL type system) ✅ -- Store schema in LSM-Tree (`_schema:migrations:*`) ✅ -- Column type enforcement during INSERT ✅ (INTEGER, FLOAT, BOOLEAN, TIMESTAMP) -- Schema validation on CREATE TABLE ✅ - -### 0.4 AST → IR lowering pass -- Convert Select AST nodes to IR plans (scan → filter → project → sort → limit) ✅ -- Convert Insert AST nodes to IR plans ✅ -- Convert Update/Delete AST nodes to IR plans ✅ (direct execution with WHERE filter) -- CTE AST nodes ❌ **Parsed but not executed** -- Lower JOINs to IR join nodes ❌ **Parsed but not lowered** - -### 0.5 Codegen → Storage execution -- Execute StorageOp tree against LSM-Tree ✅ (via executePlan) -- sokScan: full table scan via `scanMemTable()` ✅ -- sokPointRead: key-based lookup ✅ -- sokFilter: evaluate IR expressions against rows ✅ -- sokProject: column selection ✅ -- sokSort: in-memory sort ✅ -- sokLimit: slice results ✅ -- sokGroupBy: row grouping with count(*) aggregation ✅ - -### 0.6 Wire server to use pipeline -- Replace `execSelect/execInsert/execDelete` with pipeline-based execution ✅ -- Server flow: lex → parse → AST→IR lower → executePlan → LSM ✅ -- Keep backward-compatible wire protocol ✅ -- All 56 existing tests still pass ✅ - ---- - -## Phase 1: Schema & Indexes ✅ DONE - -### 1.1 SQL type system -- `INTEGER`, `BIGINT`, `SMALLINT`, `SERIAL` ✅ (validated on INSERT) -- `FLOAT`, `REAL`, `DOUBLE PRECISION`, `NUMERIC` ✅ (validated on INSERT) -- `BOOLEAN` ✅ (validated: true/false/1/0/t/f/yes/no) -- `TIMESTAMP`, `DATE` ✅ (minimal format validation) -- `VARCHAR(n)`, `TEXT` ⚠️ (stored, no length enforcement) -- `JSON`, `JSONB` ❌ **Stored as string** -- `UUID` (v4 generation) ❌ - -### 1.2 Constraints enforcement -- PRIMARY KEY: unique index + NOT NULL ✅ -- FOREIGN KEY: checks referenced row exists on INSERT ✅ -- UNIQUE: unique index via B-Tree ✅ -- NOT NULL: check on INSERT ✅ -- CHECK: parsed ✅ **Expression evaluated on INSERT and UPDATE** -- DEFAULT: fill missing values on INSERT ✅ (works for all literal types) - -### 1.3 B-Tree index integration -- `CREATE INDEX idx_name ON table(column)` ✅ -- `CREATE UNIQUE INDEX` ✅ -- B-Tree indexes created per PK/UNIQUE column ✅ -- Query planner uses B-Tree for WHERE clauses ✅ (point reads) -- Range scans via B-Tree leaf linked list ❌ **Not implemented** - -### 1.4 Query planner -- Choose index scan vs full scan based on WHERE clause ✅ -- Multi-column index support ❌ -- Covering index optimization ❌ -- `EXPLAIN` output ✅ (returns plan description with index info) -- Adaptive query reoptimization ❌ **Module exists, not wired** - ---- - -## Phase 2: Transactions ✅ DONE - -### 2.1 Wire MVCC into server pipeline -- `BEGIN`, `COMMIT`, `ROLLBACK` commands ✅ -- Server tracks per-connection Transaction state ✅ -- All reads/writes through TxnManager ✅ (INSERT/DELETE) -- Isolation: Read Committed ✅ - -### 2.2 WAL crash recovery -- Implement REDO: replay committed WAL entries ✅ -- Implement UNDO: skip uncommitted entries ✅ -- Checkpoint markers in WAL ⚠️ (WAL commit markers on flush) -- Point-in-time recovery ❌ - -### 2.3 Compaction -- Implement actual SSTable merge ✅ (reads entries, merges by key, deduplicates, removes tombstones) -- Level-based compaction strategy ⚠️ (structure exists, manual trigger) -- Background compaction scheduling ❌ - -### 2.4 Deadlock detection wiring -- Wire deadlock detection into TxnManager ❌ **Module exists, never imported** - ---- - -## Phase 3: HTTP REST API & Authentication ✅ DONE - -### 3.1 HTTP server (hunos) -- Multi-threaded HTTP/1.1 + HTTP/2 server via hunos ✅ -- `POST /query` — execute SQL, return JSON ✅ -- `GET /health` — readiness/liveness ✅ -- `GET /metrics` — Prometheus format ✅ -- `POST /auth` — JWT login endpoint ✅ -- `GET /api` — OpenAPI 3.0 spec ✅ -- CORS via hunos corsMiddleware ✅ -- Rate limiting via hunos ratelimit ✅ - -### 3.2 Authentication (jwt-nim-baraba) -- JWT token creation with HMAC-SHA256 (BearSSL) ✅ -- JWT token verification with time claims ✅ -- `Authorization: Bearer ` in HTTP headers ✅ -- `CREATE USER` / `DROP USER` SQL ✅ -- Password hashing with argon2 ❌ -- Per-user namespace isolation ❌ - -### 3.3 Authorization -- `GRANT` / `REVOKE` for table-level privileges ✅ (parsed, persisted) -- Row-Level Security (RLS) ✅ (CREATE/DROP POLICY, ENABLE/DISABLE RLS, policy evaluation in executor) -- Wire auth into both HTTP and TCP protocol paths ⚠️ **HTTP only** - -### 3.4 TLS -- Wire TLS/SSL ❌ **Mock only, no OpenSSL FFI** -- Self-signed cert generation ✅ (shells to openssl CLI) - ---- - -## Phase 4: WebSocket & Real-time ✅ DONE - -### 4.1 WebSocket server -- `ws://host:port/live` — subscribe to table changes ✅ -- `SUBSCRIBE table_name` / `UNSUBSCRIBE table_name` ✅ -- Push notifications on INSERT/UPDATE/DELETE ✅ **(onChange callback wired to WsServer.broadcastToTable)** -- `NOTIFY` / `LISTEN` analogue ❌ - -### 4.2 CORS & HTTP hardening -- CORS headers for browser access ✅ (via hunos middleware) -- Request size limits ✅ (via hunos maxBodyLen) -- HTTP/2 readiness ✅ (via hunos h2c support) - ---- - -## Phase 4.5: UTF-8 Support ✅ DONE - -### 4.5.1 Storage -- String values stored as raw bytes ✅ (UTF-8 data transparently stored/retrieved) -- Wire protocol uses byte-count lengths ✅ (UTF-8 safe) - -### 4.5.2 Lexer & Parser -- UTF-8 identifiers (table names, column names) ✅ -- `std/unicode` with `Rune` based tokenization ✅ -- Multi-byte characters in string literals ✅ (preserved as raw bytes) - -### 4.5.3 HTTP -- `charset=utf-8` in all Content-Type headers ✅ -- `/metrics` → `text/plain; charset=utf-8` -- `/admin` → `text/html; charset=utf-8` -- `/query` → `application/json; charset=utf-8` (via hunos) - ---- - -## Phase 5: ERP Features ❌ MOSTLY NOT DONE - -### 5.1 Schema migrations -- `CREATE MIGRATION` → `APPLY MIGRATION` ✅ **SQL syntax exists** -- Versioned schema in `_schema_version` table ⚠️ **Uses _schema:migrations:applied: prefix for tracking** -- Up/down migration scripts ❌ -- Dry-run mode ❌ -- CLI: `baradadb migrate status|up|down` ❌ - -### 5.2 Views -- `CREATE VIEW` — virtual table ✅ **Parsed, executable, persisted to LSM-Tree** -- `CREATE MATERIALIZED VIEW` ❌ - -### 5.3 Triggers & stored functions -- `CREATE TRIGGER` ✅ (BEFORE/AFTER/INSTEAD OF, INSERT/UPDATE/DELETE, persisted to LSM-Tree) -- `DROP TRIGGER` ✅ -- Trigger firing in executor ✅ (fires in execInsert/execUpdate/execDelete) -- Stored functions ❌ -- ERP helper functions ❌ - -### 5.4 Full-text search for ERP documents -- `CREATE FULLTEXT INDEX ON table(column)` ❌ **FTS engine exists, not wired to SQL** -- `WHERE content @@ 'search query'` ❌ - -### 5.5 Partitioning -- `CREATE TABLE (...) PARTITION BY RANGE (col)` ❌ - ---- - -## Phase 6: Production Readiness ⚠️ PARTIALLY DONE - -### 6.1 Backup & Restore -- `baradadb backup --output backup.tar.gz` ✅ -- `baradadb restore --input backup.tar.gz` ✅ -- Incremental backup via WAL archiving ❌ -- Point-in-time recovery (PITR) ❌ - -### 6.2 Docker & deployment -- `Dockerfile` — multi-stage build with Nim ✅ -- `docker-compose.yml` — single node ✅ (healthcheck via wget) -- `docker-compose.raft.yml` — 3-node cluster ❌ -- Environment-based config ✅ - -### 6.3 Monitoring -- Structured JSON logging ❌ **Uses echo** -- Prometheus `/metrics` ✅ (basic counters) -- Slow query log ❌ -- OpenTelemetry tracing ❌ - -### 6.4 Admin dashboard -- Inline HTML dashboard ✅ (SQL Playground, Tables browser, Schema, Live WebSocket, Metrics, Cluster) -- JWT login gate ✅ (localStorage token, auto-login) -- WebSocket real-time events ✅ (auto-connect, reconnect) -- Metrics charts ✅ (6 stat boxes + raw Prometheus, auto-refresh) -- Static file serving ❌ - -### 6.5 Client SDK improvements -- Nim client: DATA row decoding ✅, all WireValue types ✅, sync + async ✅ -- Python client: DATA row decoding ✅, all WireValue types ✅ -- Rust client: DATA row decoding ✅, all WireValue types ✅ -- JavaScript client: DATA row decoding ✅, all WireValue types ✅ (ready for socket integration) -- Go client: ❌ **Removed** - ---- - -## Priority Matrix - -| Task | Impact | Difficulty | Priority | -|------|--------|-----------|----------| -| Pipeline integration (Phase 0) | Critical | High | **P0 ✅** | -| SQL DDL parser (Phase 0) | Critical | Medium | **P0 ✅** | -| AST→IR lowering (Phase 0) | Critical | High | **P0 ✅** | -| Codegen execution (Phase 0) | Critical | High | **P0 ✅** | -| SQL schema system (Phase 1) | Critical | High | **P0 ✅** | -| B-Tree index integration (Phase 1) | High | Medium | **P1 ✅** | -| Constraint enforcement (Phase 1) | High | Medium | **P1 ✅** | -| MVCC wiring (Phase 2) | Critical | High | **P0 ✅** | -| WAL recovery (Phase 2) | High | Medium | **P1 ✅** | -| SSTable compaction (Phase 2) | High | Medium | **P1 ✅** | -| HTTP REST API (Phase 3) | Critical | Medium | **P0 ✅** | -| JWT Auth (Phase 3) | High | Medium | **P1 ✅** | -| WebSocket real-time (Phase 4) | Medium | Medium | **P2 ✅** | -| Schema migrations (Phase 5) | High | Medium | P1 ✅ | -| Backup/Restore (Phase 6) | Medium | Medium | **P2 ✅** | -| Docker + Compose (Phase 6) | Medium | Low | **P2 ✅** | -| Admin Dashboard (Phase 6) | Medium | High | P2 ✅ | -| Views + Triggers (Phase 5) | Low | Medium | P3 ✅ | -| UTF-8 Support | Medium | Low | P2 ✅ | -| RLS (Phase 3) | High | High | P1 ✅ | -| Client SDK (Phase 6) | Medium | High | P2 ✅ | -| Kubernetes Helm (Phase 6) | Low | Medium | P3 ❌ | - ---- - -## What Actually Works (Honest) - -**Production-ready NOW:** -- CREATE TABLE with PK, FK, UNIQUE, NOT NULL, DEFAULT, type enforcement -- CREATE INDEX / CREATE UNIQUE INDEX -- INSERT INTO ... VALUES with column list, validation, type checking -- SELECT with WHERE filter (real evaluation), ORDER BY (real sorting), GROUP BY, LIMIT/OFFSET -- UPDATE with WHERE clause (real row modification) -- DELETE with WHERE clause (real row deletion with filter) -- BEGIN / COMMIT / ROLLBACK transactions (MVCC) -- B-Tree index creation, population, and point reads -- FOREIGN KEY enforcement (checks referenced row exists) -- Type enforcement (INTEGER, FLOAT, BOOLEAN, TIMESTAMP validated) -- LIKE pattern matching (regex) -- EXPLAIN output with index usage info -- HTTP REST API via hunos (multi-threaded, CORS, rate limiting) -- JWT authentication via jwt-nim-baraba (HS256 BearSSL) -- WebSocket SUBSCRIBE/UNSUBSCRIBE with broadcast on data changes -- Schema persistence + auto-restore on restart -- WAL crash recovery (REDO committed, UNDO uncommitted) -- SSTable compaction (real merge, dedup, tombstone cleanup) -- Docker + docker-compose deployment -- Backup/restore via tar.gz -- OpenAPI 3.0 spec at GET /api - -**Partially working:** -- ALTER TABLE ADD COLUMN (basic, no DROP/RENAME) -- CTE (WITH clause) — parsed but not executed -- JOINs — parsed but not executed -- CHECK constraints — parsed and evaluated on INSERT/UPDATE -- GRANT/REVOKE — parsed and persisted, but no runtime privilege enforcement (RLS policies work) - -**Not yet working:** -- Schema migrations via SQL (parsed, but no up/down scripts or CLI) -- WAL point-in-time recovery -- Background compaction scheduling -- Deadlock detection wiring -- TLS/SSL (mock only) -- Full-text search via SQL -- Partitioning -- Prepared statements / parameterized queries in clients - ---- - -## Dependencies - -| Package | Version | Purpose | -|---------|---------|---------| -| [hunos](https://github.com/katehonz/hunos) | >= 1.2.0 | Multi-threaded HTTP/WebSocket server | -| [jwt-nim-baraba](https://github.com/katehonz/jwt-nim-baraba) | >= 2.1.0 | JWT authentication (HS256 BearSSL) | - ---- - -**Honest score: 9.2/10 — production-ready foundation with HTTP server, JWT auth, WAL recovery, compaction, WebSocket real-time, admin dashboard, RLS, triggers, views, UTF-8 support, and working client SDKs. Remaining gaps: CTE execution, JOIN lowering, TLS, full-text search SQL integration, partitioning.** diff --git a/PLAN_old_3.md b/PLAN_old_3.md deleted file mode 100644 index 9e1c522..0000000 --- a/PLAN_old_3.md +++ /dev/null @@ -1,207 +0,0 @@ -# BaraDB — PLAN - -> **v1.0.0 READY** — Всички критични/високи/средни/конфигурационни бъгове поправени. Всички 10 TLA+ спецификации са завършени. Build е чист (0 warnings). - ---- - -## Разпределени модули — финален status (след сесия 8) - -### ✅ Поправено - -| Модул | Промяна | -|--------|---------| -| `disttxn` | 2PC atomicity: prepare failure → rollback готови; commit failure → rollback | -| `disttxn` | DISTTXN handler ползва реален `DistTxnManager` | -| `disttxn` | `DistTxnManager` инициализиран в `newServer()` | -| `sharding` | `getShardRange` връща `-1` за out-of-range keys | -| `sharding` | Binary search в consistent hashing ring | -| `gossip` | `startHealthCheck()` + `startGossipRound()` async loops | -| `raft` | `applyCommand` callback — state machine прилага committed entries | -| `raft` | `RaftNetwork.run()` стартира от `main()` ако `raftEnabled=true` | -| `raft` | `asyncCheck` заменен с `try/await` в critical paths | -| `raft` | `bindAddr` без hardcoded IP (приема на 0.0.0.0) | -| `raft` | Disk persistence: `saveState()`/`loadState()` за term/votedFor/log | -| `config` | Raft config: `raftEnabled`, `raftPort`, `raftPeers`, `raftNodeId` + env vars | -| `auth` | JWT `exp`/`nbf`/`iat` validation + constant-time signature comparison | -| `auth` | **SCRAM-SHA-256**: истински challenge-response със salt + iteration count | -| `backup` | TLA+ спек: `BackupSnapshotsValid`, `RestoreIntegrity`, `RetentionInvariant` | -| `recovery` | TLA+ спек: `RedoCommitted`, `RecoveryCompleteness`, `WalIntegrity` | -| `crossmodal` | TLA+ спек: `MetadataVectorConsistency`, `HybridResultValid`, `TxnAtomicity` | - -### ⚠️ Оставащи distributed gaps (non-critical за single-node) - -| Модул | Gap | Статус | -|--------|-----|--------| -| `replication` | `writeLsn` не изпраща данни към replicas | ✅ Добавен UDP transport + binary serialization | -| `gossip` | Няма UDP/TCP transport — in-memory само | ✅ Добавен UDP listener + broadcast + binary serialization | -| `sharding` | `rebalance` не мигрира данни | ✅ Добавен `migrateData` протокол + `scanAll` на LSM | -| `inter-module` | Няма raft→disttxn, gossip→sharding, replication→disttxn връзки | ✅ Всички връзки реализирани | -| `server` | Няма shard-aware routing | ✅ ClusterMembership + ShardRouter в Server | - ---- - -## Formal Verification — финален status - -### 🔴 Критични (всички поправени ✅) - -| # | Задача | Статус | -|---|--------|--------| -| FV-1 | Raft: prevLogIndex/prevLogTerm в Replicate | ✅ | -| FV-2 | Raft: Leader step-down при partition | ✅ | -| FV-3 | 2PC: Coordinator crash/recovery | ✅ | -| FV-4 | 2PC: Participant timeout | ✅ | - -### 🟡 Важни (всички поправени ✅) - -| # | Задача | Статус | -|---|--------|--------| -| FV-5 | Symmetry reduction във всички .cfg | ✅ 10 спеки | -| FV-6 | Liveness свойства | ✅ | -| FV-7 | MVCC: Write skew detection | ✅ | -| FV-8 | Replication: Data consistency | 🟡 Остава — non-critical | -| FV-9 | Sharding: Data migration при rebalance | 🟡 Остава — non-critical | - -### 🟢 Нови спекове (всички завършени ✅) - -| # | Задача | Покрива | Приоритет | -|---|--------|---------|-----------| -| FV-10 | `backup.tla` | `backup.nim` | ✅ | -| FV-11 | `recovery.tla` | `recovery.nim` | ✅ | -| FV-12 | `crossmodal.tla` | `crossmodal.nim` | ✅ | - -### 🔧 Инфраструктурни (всички поправени ✅) - -| # | Задача | Статус | -|---|--------|--------| -| FV-13 | CI: Поправка на verify job | ✅ | -| FV-14 | Property-based testing мост | ✅ | - ---- - -## ✅ Сесия 8 — v1.0.0 финален спринт - -### Опция A: "Clean build" ✅ -- Почистване на 5-те build warnings -- TLA+ symmetry reduction в `.cfg` файловете -- Резултат: чист build без warnings + 3-10x по-бърз TLC - -### Опция B: `crossmodal.tla` ✅ -- TLA+ спек за cross-modal consistency -- Моделира sync между document/vector/graph/FTS индекси -- Резултат: 10-ти TLA+ спек, пълно покритие на core модулите - -### Опция C: Auth hardening + SCRAM ✅ -- Истински SCRAM-SHA-256 със salt (4096 iterations), challenge-response -- Нов `scram.nim` модул per RFC 7677 -- HTTP endpoints: `/auth/scram/start` + `/auth/scram/finish` -- Резултат: production-grade auth - ---- - -## Финални метрики - -| Метрика | Стойност | -|---------|----------| -| **Тестове** | 294 — 0 failures ✅ | -| **Критични бъгове** | 0 ✅ | -| **Високи бъгове** | 0 ✅ | -| **Средни бъгове** | 0 ✅ | -| **TLA+ спецификации** | 10 — всички с symmetry reduction ✅ | -| **Build warnings** | 0 ✅ | -| **Security audit** | Всички 🔴 и 🟠 поправени ✅ | -| **Общ брой поправени бъгове** | 32 (9 критични + 7 високи + 12 средни + 4 конфигурационни) | -| **Общ брой сесии** | 9 | - ---- - -## Оставащи задачи (post-v1.0.0, non-critical) - -| # | Задача | Оценка | -|---|--------|--------| -| — | Няма — всички планирани задачи са завършени | — | - -**BaraDB v1.0.0 е production-ready за blogs, e-commerce и small ERP системи.** -**Всички distributed gaps са запълнени: replication, gossip transport, sharding migration, inter-module wiring.** -**Thread safety: SharedLock ref споделен между всички connection-и — конкурентни DDL/DML защитени.** - ---- - -## 🆕 Сесия 9 — Stabilization Sprint (май 2026) - -> **Цел:** Да махнем всички workaround-и от `BARADB_DEFICIENCIES.md`, да почистим build-а и да подготвим почвата за типова система. -> **Принцип:** Без нови светове — само stabilizaция на съществуващото. - -### Седмица 1: Deficiency Hunt + Build Cleanup - -| # | Задача | Оценка | Статус | -|---|--------|--------|--------| -| 9.1.1 | Почистване на 9-те build warnings (ResultShadowed + UnusedImport) | 1ч | ✅ | -| 9.1.2 | Issue #6: Aggregate column names (`count(*)` → `count(*)`, `max(id)` → `max(id)`) | 2ч | ✅ | -| 9.1.3 | Issue #5: GROUP BY bare columns — първи ред от групата за non-aggregated колони | 4-6ч | ✅ | -| 9.1.4 | Issue #7+8: Решение за async vs sync client + thread safety | 2ч | ✅ | -| 9.1.5 | Regression тестове за всички 10 deficiencies | 2ч | ✅ | - -**Метрика:** NimForum миграционният код маха всички `DISTINCT` workaround-и за GROUP BY. - ---- - -### Седмица 2: Type Safety in Execution Layer - -| # | Задача | Оценка | Статус | -|---|--------|--------|--------| -| 9.2.1 | `IRExpr` носи `valueKind` — всеки AST node знае дали е INT, FLOAT, TEXT, NULL | 4-6ч | ✅ | -| 9.2.2 | `evalExprValue` връща discriminated union (`Value(kind: vkInt64/Float64/String/Null)`) вместо само `string` | 6-8ч | ✅ | -| 9.2.3 | `irAdd`/`irSub`/`irMul`/`irDiv` използват типовата информация (INT+INT → INT, INT+FLOAT → FLOAT) | 3ч | ✅ | -| 9.2.4 | `validateType` използва `Value.kind` вместо `parseInt`/`parseFloat` на string | 2ч | ✅ | - -**Метрика:** Премахваме всички `try: parseFloat catch: return fallback` евристики от `evalExpr`. - ---- - -### Седмица 3: JOIN Performance - -| # | Задача | Оценка | Статус | -|---|--------|--------|--------| -| 9.3.1 | Hash Join: `ON a.col = b.col` с hash table върху по-малката страна | 6ч | ✅ | -| 9.3.2 | Index Nested Loop Join: ако има B-Tree индекс на join колоната | 4ч | ✅ | -| 9.3.3 | Benchmark: `thread JOIN category` с 10K/100K редове | 2ч | ✅ | -| 9.3.4 | Query planner избира между Nested Loop / Hash / Index въз основа на наличие на индекс | 4ч | ✅ | - -**Метрика:** JOIN с 100K редове е под 100ms. - ---- - -### Седмица 4: Production Hardening - -| # | Задача | Оценка | Статус | -|---|--------|--------|--------| -| 9.4.1 | Property-based tests за `evalExpr` — случайни AST-та, проверка на invariant-и | 4ч | ✅ | -| 9.4.2 | Fuzz test за wire protocol — случайни байтове, mutation fuzzing, roundtrip за всички FieldKind | 3ч | ✅ | -| 9.4.3 | Thread safety audit + fix: `execInsert`/`execUpdate`/`execDelete` с shared `ExecutionContext` | 3ч | ✅ | -| 9.4.4 | ~~NimForum integration test~~ — отпада, запазваме универсалност | — | ❌ | - -**Метрика:** 58 property-based invariant-а + 35 fuzz сценария. `ctxLock` → `SharedLock` ref споделен между всички connection-и. - -**Thread safety fix:** `ctxLock` беше per-connection `Lock` — всеки клониран контекст имаше собствен mutex, което не пази shared state (tables, btrees, ftsIndexes, users, policies, etc.) при конкурентни DDL/DML. Преместен в `SharedLock = ref object` споделен между всички клонинги на `ExecutionContext`. - ---- - -### Финални метрики (след сесия 9 — завършена) - -| Метрика | Стойност | -|---------|----------| -| **Тестове** | 316 — 0 failures | -| **Prop тестове** | 58 (commutativity, associativity, distributivity, identity, NULL propagation, type coercion, comparisons) | -| **Fuzz тестове** | 35 (deserializeValue, roundtrip всички FieldKind, mutation, stress) | -| **Build warnings** | 0 | -| **BARADB_DEFICIENCIES** | 0 непоправени (всички 10 поправени) | -| **Workaround-и в NimForum** | 0 | -| **evalExprValue** | Връща `Value(kind: vkInt64/Float64/String/Null)` | -| **Аритметични ops** | INT+INT→INT, INT+FLOAT→FLOAT, FLOAT/INT→FLOAT | -| **Join стратегии** | Hash Join + Index Nested Loop + Nested Loop | -| **JOIN 10K (Hash)** | ~115ms | -| **JOIN 10K (Index NL)** | ~90ms | -| **Shared lock** | `SharedLock` ref — един mutex за всички connection-и | -| **Общ брой сесии** | 9 | - -**BaraDB v1.0.0 — production-ready. Сесия 9 завършена: build чист, типова система в execution layer, JOIN performance, production hardening.** diff --git a/PR_DESCRIPTION.md b/PR_DESCRIPTION.md deleted file mode 100644 index ef084f9..0000000 --- a/PR_DESCRIPTION.md +++ /dev/null @@ -1,60 +0,0 @@ -# Nodebara Compatibility Fixes - -This PR bundles three independent fixes discovered while integrating BaraDB with [NodeBB](https://nodebb.org/) (a large Node.js forum application). Each commit is self-contained and can be reviewed separately. - ---- - -## 1. fix(protocol): serialize float32/float64 in big-endian - -**Problem:** -The JavaScript client reads `FLOAT32`/`FLOAT64` wire values with `readFloatBE()` / `readDoubleBE()` (big-endian), but the Nim server was writing them with `copyMem(..., unsafeAddr fl, N)` — i.e. **native byte order**. On little-endian machines (virtually all x86_64 servers) the client deserializes garbage. Example: a zset `score = 1.0` becomes `3.03865e-319`, which breaks any application relying on numeric scores (user IDs, timestamps, rankings, etc.). - -**Fix:** -Cast floats to `int32`/`int64` and route them through the existing `bigEndian32`/`bigEndian64` helpers, exactly like the integer paths already do. Same change applied to deserialization. - -**Impact:** -Breaking fix for cross-platform wire compatibility. No API changes. - ---- - -## 2. feat(client): add TCP request queue for safe concurrency - -**Problem:** -`Client.query()`, `execute()`, and `ping()` were async methods that wrote directly to the TCP socket. When NodeBB fired multiple parallel DB operations (common on startup), their binary frames interleaved on the wire, causing parse errors, wrong request/response pairing, and random crashes. - -**Fix:** -Introduce an internal `_requestQueue` + `_requestLock`. Every public async method enqueues a closure; a tiny drain loop processes them one at a time via `setImmediate()`. - -**Impact:** -No breaking API change. Existing single-request usage is unchanged; concurrent usage now works safely. - ---- - -## 3. fix(gossip): use async UDP socket to avoid blocking the event loop - -**Problem:** -`startGossipListener` created a **synchronous** UDP socket (`newSocket`) and called blocking `recvFrom` inside an `async` proc. This freezes the entire async event loop until a UDP packet arrives, stalling all other async I/O. - -**Fix:** -Replace `newSocket` with `newAsyncSocket` and `recvFrom` with `await recvFrom`. - -**Impact:** -Non-breaking. Gossip remains optional; when enabled it no longer blocks the main loop. - ---- - -## Testing - -- [x] NodeBB v4.11.2 setup completes end-to-end -- [x] Admin login works (relies on correct FLOAT64 score deserialization) -- [x] Concurrent DB queries during startup no longer corrupt frames -- [x] Gossip listener no longer blocks other async tasks - ---- - -## Checklist - -- [x] Each commit is atomic and compiles on its own -- [x] No Nim compiler warnings introduced -- [x] JS client backward-compatible for single-request callers -- [x] Existing tests (`nimble test`) still pass diff --git a/ROADMAP.md b/ROADMAP.md deleted file mode 100644 index d80a74f..0000000 --- a/ROADMAP.md +++ /dev/null @@ -1,202 +0,0 @@ -# BaraDB — Мултимодална база данни на Nim - -> По-добра от GEL (EdgeDB) — нативна мултимодалност, без Python, без PostgreSQL зависимост - -## Защо BaraDB > GEL? - -| Критерий | GEL (EdgeDB) | BaraDB | -|---|---|---| -| Език на ядрото | Python + Cython + Rust | **100% Nim** | -| Storage backend | Само PostgreSQL | **Нативен multi-engine** | -| Векторно търсене | pgvector (разширение) | **Нативен HNSW/IVF-PQ** | -| Graph алгоритми | Няма | **Нативни (BFS, PageRank, ...)** | -| Full-Text Search | pg FTS (разширение) | **Нативен инвертиран индекс** | -| Embedded режим | Не | **Да (като SQLite)** | -| Кompилация към WASM | Не | **Да** | -| Транзакции cross-modal | Чрез PG | **Нативен 2PC** | -| Протокол | Binary + PG wire + HTTP | **Binary + HTTP/WS + gRPC** | -| Schema миграции | Декларативни (тежки) | **Автоматични + версия** | - ---- - -## Архитектура - -``` -┌─────────────────────────────────────────────────────────────┐ -│ КЛИЕНТСКИ СЛОЙ │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────┐ │ -│ │ Binary │ │ HTTP/REST│ │WebSocket │ │ Embedded │ │ -│ │ Protocol │ │ JSON API │ │ Stream │ │ (in-proc) │ │ -│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────┬──────┘ │ -├───────┼──────────────┼────────────┼────────────────┼────────┤ -│ ЗАПИТЕН СЛОЙ (BaraQL) │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────┐ │ -│ │ Лексер │──│ Парсер │──│ Анализ │──│ Оптимизатор │ │ -│ │ │ │ (AST) │ │ (IR) │ │ (Plan) │ │ -│ └──────────┘ └──────────┘ └──────────┘ └─────────────┘ │ -├─────────────────────────────────────────────────────────────┤ -│ ИЗПЪЛНИТЕЛЕН ДВИГАТЕЛ │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────┐ │ -│ │ Document │ │ Graph │ │ Vector │ │ Columnar │ │ -│ │ Engine │ │ Engine │ │ Engine │ │ Engine │ │ -│ │ (JSON/B) │ │ (AdjList)│ │ (HNSW) │ │ (Arrow) │ │ -│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────┬──────┘ │ -├───────┼──────────────┼────────────┼────────────────┼────────┤ -│ STORAGE СЛОЙ │ -│ ┌─────────────────────────────────────────────────────────┐│ -│ │ LSM-Tree Storage Engine ││ -│ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌──────────────┐ ││ -│ │ │MemTable│ │WAL │ │SSTable │ │Bloom Filter │ ││ -│ │ └────────┘ └────────┘ └────────┘ └──────────────┘ ││ -│ └─────────────────────────────────────────────────────────┘│ -│ ┌─────────────────────────────────────────────────────────┐│ -│ │ Транзакции: MVCC + 2PC + Deadlock Detection ││ -│ └─────────────────────────────────────────────────────────┘│ -└─────────────────────────────────────────────────────────────┘ -``` - ---- - -## Пътна карта (Roadmap) - -### Фаза 1: Ядро на базата данни ✅ -- [x] LSM-Tree storage engine (MemTable, WAL, SSTable) -- [x] Bloom filter за бързо отхвърляне -- [x] Типова система (int, float, string, bool, bytes, uuid, datetime, json, vector) -- [x] Сериялизация на записите -- [x] B-Tree индекс за точкови заявки -- [x] Компактиране на SSTable (compaction strategies) -- [x] Page cache и buffer pool (LRU) - -### Фаза 2: Език за заявки — BaraQL 🟡 -- [x] Лексер с Unicode поддръжка -- [x] Рекурсивен парсер → AST -- [x] SELECT, INSERT, UPDATE, DELETE -- [x] WHERE, ORDER BY, LIMIT, OFFSET -- [x] Бинарни оператори (+, -, *, /, =, !=, <, >, AND, OR, NOT) -- [x] Подзаявки и EXISTS -- [x] Array литерали -- [x] Типов анализатор (type checker) -- [x] IR (Intermediate Representation) -- [x] Оптимизатор на заявки (predicate pushdown, projection pushdown) -- [x] GROUP BY, HAVING -- [x] JOIN (inner, left, right, full, cross) -- [x] CTE (WITH) -- [x] Агрегатни функции (count, sum, avg, min, max) -- [x] Codegen — IR → storage операции (predicate pushdown, cost estimation) -- [x] Потребителски функции (UDF) — stdlib + custom - -### Фаза 3: Мултимодален storage 🟡 -- [x] Документен engine — вложени JSON документи, масиви, вложени обекти -- [x] Граф engine — adjacency list, edge properties, incident index -- [x] Векторен engine — float32 arrays, distance metrics -- [x] Колонен engine — column-oriented storage за analytics (RLE, dict encoding, GroupBy) -- [x] Унифициран query interface през BaraQL (CrossModalEngine) -- [x] Cross-modal заявки (document + vector + graph в една заявка) — hybridSearch - -### Фаза 4: Транзакции и ACID ✅ -- [x] WAL (Write-Ahead Log) за durability -- [x] MVCC (Multi-Version Concurrency Control) -- [x] Snapshot isolation -- [x] Deadlock detection (wait-for graph) -- [x] Savepoints и вложени транзакции -- [x] 2PC за cross-modal транзакции (TPCTransaction) -- [x] Recovery при crash (REDO/UNDO) — WAL replay + analysis - -### Фаза 5: Мрежов протокол ✅ -- [x] TCP сървър с async I/O -- [x] Binary протокол (BaraDB Wire Protocol) -- [x] HTTP/REST API (JSON) -- [x] Connection pooling -- [x] Authentication (JWT, SCRAM-SHA-256) -- [x] WebSocket за streaming -- [x] Rate limiting (token bucket, sliding window) -- [x] TLS/SSL (certificate management, self-signed generation) - -### Фаза 6: Schema система ✅ -- [x] Декларативна schema (SDL) -- [x] Object types с properties -- [x] Links между типове (1:1, 1:N, N:M) -- [x] Наследоване и mixins -- [x] Constraints (unique, check, required) -- [x] Computed properties -- [x] Автоматични миграции (schema diff) -- [x] Версиониране на schema - -### Фаза 7: Векторен engine ✅ -- [x] HNSW индекс (Hierarchical Navigable Small World) -- [x] IVF-PQ индекс (Inverted File + Product Quantization) -- [x] Дистанционни метрики (cosine, euclidean, dot product, Manhattan) -- [x] Квантизация (scalar 8-bit/4-bit, product, binary) -- [x] Metadata filtering при vector search -- [x] Batch insert/update (batchInsert, batchSearch) -- [x] Автоматичен index rebuild при threshold (IndexWatcher) - -### Фаза 8: Graph engine ✅ -- [x] Adjacency list storage -- [x] Edge properties и weights -- [x] BFS (Breadth-First Search) -- [x] DFS (Depth-First Search) -- [x] Най-къс път (Dijkstra) -- [x] PageRank -- [x] Community detection (Louvain) -- [x] Pattern matching (subgraph isomorphism) -- [x] Cypher-подобен query syntax (parseCypher, executeCypher) - -### Фаза 9: Full-Text Search ✅ -- [x] Инвертиран индекс -- [x] Токенизация (Unicode, stemming, stop words) -- [x] BM25 ранкиране -- [x] Highlight на резултати -- [x] TF-IDF ранкиране -- [x] Fuzzy matching (Levenshtein) -- [x] Regex търсене (wildcard patterns) -- [x] Многоезикова поддръжка (EN, BG, DE, FR, RU) - -### Фаза 10: Клиентски библиотеки и CLI ✅ -- [x] CLI tool (bara shell) — интерактивен shell + autocomplete -- [x] Nim client library (standalone nimble package — async/sync, query builder) -- [x] Python client library (clients/python/baradb.py — binary protocol) -- [x] JavaScript/TypeScript client library (clients/javascript/baradb.js) -- [x] Rust client library (clients/rust/) -- [x] Import/Export (JSON, CSV, NDJSON) - -### Фаза 11: Кластеризация и разпределение ✅ -- [x] Raft консенсус протокол (leader election + log replication) -- [x] Sharding (hash-based, range-based, consistent hashing) -- [x] Replication (sync, async, semi-sync) -- [x] Gossip protocol за membership (GossipProtocol) -- [x] Distributed transactions (DistTxnManager + Saga pattern) -- [x] Auto-rebalancing (ClusterMembership — onNodeJoin/Leave/Fail) -- [x] Leader election timer loop (ElectionTimer — tick/check/reset) - -### Фаза 12: Оптимизации, бенчмаркове, документация ✅ -- [x] SIMD оптимизации за vector operations (unrolled loops, batch distance) -- [x] Memory-mapped I/O (mmap + madvise hints) -- [x] Zero-copy serialization (ZeroBuf + ZcSchema) -- [x] Adaptive query execution (AdaptivePlanner + ExecutionContext) -- [x] Бенчмаркове vs PostgreSQL, Redis, MongoDB (benchmarks/compare.nim) -- [x] API документация (README.md — full API reference with examples) -- [x] Архитектурна документация (docs/ARCHITECTURE.md) -- [x] Tutorial и примери (examples/tutorial.nim — 8 tutorials) - ---- - -## Статус - -| Фаза | Статус | Напредък | -|------|--------|----------| -| 1. Ядро | ✅ | 100% | -| 2. BaraQL | ✅ | 100% | -| 3. Мултимодален storage | ✅ | 100% | -| 4. Транзакции | ✅ | 100% | -| 5. Протокол | ✅ | 100% | -| 6. Schema | ✅ | 100% | -| 7. Векторен engine | ✅ | 100% | -| 8. Graph engine | ✅ | 100% | -| 9. FTS | ✅ | 100% | -| 10. Клиенти и CLI | ✅ | 100% | -| 11. Кластер | ✅ | 100% | -| 12. Оптимизации | ✅ | 100% | - -**Легенда:** ⬜ Не стартирана | 🟡 В процес | ✅ Завършена