feat: Phase 0 — pipeline integration, DDL parser, SQL executor

- Rewrote PLAN.md with 6-phase production roadmap - Added 15 DDL/txn lexer keywords (primary, key, foreign, references, etc.) - Added AST nodes: CreateTable, DropTable, AlterTable, BeginTxn, CommitTxn, RollbackTxn, ExplainStmt, ColumnDef - Completed INSERT parser: VALUES, column list, RETURNING, ON CONFLICT - Added CREATE TABLE/DROP TABLE/ALTER TABLE parsers with constraints (PK, FK, UNIQUE, NOT NULL, CHECK, DEFAULT) - Added UPDATE/DELETE RETURNING support - Added BEGIN, COMMIT, ROLLBACK, EXPLAIN parsers - New query/executor.nim: AST->IR lowering + plan execution against LSM-Tree - Wired server to executor pipeline (replaced regex-based KV INSERT) - All 216 existing tests pass
2026-05-06 11:10:50 +03:00
parent 096c8347cf
commit ca5e04b96e
7 changed files with 997 additions and 262 deletions
@@ -1,216 +1,294 @@
-# BaraDB — План към Production (Web & ERP)
+# BaraDB — Production Roadmap (Web & ERP)

-## Визия
-BaraDB да стане production-ready база данни, с която да се изграждат:
- **Web приложения** (блогове, е-магазини, SaaS продукти)
- **Малки ERP системи** (CRM, склад, счетоводство, invoicing)
+## Vision
+BaraDB as a production-ready database for:
+- **Web applications** (blogs, e-commerce, SaaS)
+- **Small ERP systems** (CRM, warehouse, accounting, invoicing)

-> Целеви потребител: solo-dev / малък екип, който иска лесна за deploy и бърза локална база, без зависимост от PostgreSQL/MySQL.
+> Target user: solo-dev / small team wanting a fast local DB without PostgreSQL/MySQL dependency.

 ---

-## Текущо състояние (база)
+## Current State (Baseline)

-| Компонент | Статус |
+| Component | Status |
 |-----------|--------|
-| LSM-Tree KV store | ✅ Стабилен, thread-safe, persistent |
-| HNSW векторен search | ✅ Работи с recall > 0.9 |
-| TCP wire protocol | ✅ Бинарен, SELECT/INSERT/DELETE |
-| Raft consensus | ✅ TCP transport, leader election |
-| Graph engine | ✅ In-memory + persistence |
-| Thread-safety | ✅ Coarse-grained locks |
-| CI/CD | ✅ GitHub Actions |
+| LSM-Tree KV store | Stable, thread-safe, persistent |
+| HNSW vector search | Working, recall > 0.9 |
+| TCP wire protocol | Binary, SELECT/INSERT/DELETE |
+| Raft consensus | TCP transport, leader election |
+| Graph engine | In-memory + persistence |
+| CI/CD | GitHub Actions |
+| Test suite | 56 suites, ~250 tests |

-**Липсва за Web/ERP:** SQL съвместимост, ACID транзакции, HTTP API, auth, ORM, миграции, backup.
+**Critical gaps for production:**
+- Server bypasses IR/codegen/MVCC/schema — `executeQuery()` does lex→parse→raw LSMTree calls
+- INSERT parser incomplete (no VALUES, column list, RETURNING)
+- No CREATE TABLE/ALTER TABLE/DROP TABLE in parser
+- MVCC not wired to query path (no BEGIN/COMMIT/ROLLBACK in server)
+- B-Tree indexes not integrated with LSM-Tree
+- SQL schema system not connected (EdgeQL types only)
+- No HTTP REST API
+- Auth not wired to server

 ---

-## Фаза 1: Релационен engine + SQL (4–6 седмици)
+## Phase 0: Pipeline Integration & Parser Completion (2–3 weeks) **← IN PROGRESS**

-### 1.1 SQL парсър и AST
- Имплементирай ANSI SQL подмножество (CREATE TABLE, ALTER TABLE, DROP TABLE)
- INSERT с column list: `INSERT INTO users (name, email) VALUES ('...', '...')`
- UPDATE: `UPDATE users SET name = '...' WHERE id = 1`
- SELECT с JOIN, GROUP BY, HAVING, ORDER BY, LIMIT/OFFSET
- DELETE с WHERE
- Поддръжка на `RETURNING` клауза
+### 0.1 Complete DML parser (INSERT/UPDATE/DELETE)
+- INSERT with column list: `INSERT INTO t (c1, c2) VALUES (v1, v2)`
+- INSERT with RETURNING clause
+- UPDATE with RETURNING clause
+- DELETE with RETURNING clause
+- Multiple VALUES rows: `VALUES (v1), (v2), ...`

-### 1.2 Типова система и constraints
- `INTEGER`, `BIGINT`, `SERIAL` (auto-increment)
+### 0.2 Add SQL DDL to parser
+- `CREATE TABLE` with column definitions, constraints (PK, FK, UNIQUE, NOT NULL, CHECK, DEFAULT)
+- `ALTER TABLE` (ADD COLUMN, DROP COLUMN, RENAME COLUMN)
+- `DROP TABLE`
+- Tokens: tkCreate, tkTable, tkAlter, tkColumn, tkPrimary, tkKey, tkForeign, tkReferences, tkCascade, tkUnique, tkNotNull, tkCheck, tkDefault, tkRename, tkAdd, tkDrop
+
+### 0.3 SQL-compatible schema system
+- SQL table catalog (separate from EdgeQL type system)
+- Store schema in LSM-Tree (`_schema_tables`, `_schema_columns`, `_schema_indexes`)
+- Column type enforcement during INSERT/UPDATE
+- Schema validation on CREATE TABLE
+
+### 0.4 AST → IR lowering pass
+- Convert Select AST nodes to IR plans (scan → filter → project → sort → limit)
+- Convert Insert AST nodes to IR plans (values)
+- Convert Update/Delete AST nodes to IR plans
+- Convert CTE AST nodes to IR plans
+- Lower JOINs to IR join nodes
+
+### 0.5 Codegen → Storage execution
+- Execute StorageOp tree against LSM-Tree
+- sokScan: full table scan via `scanMemTable()` / SSTable reader
+- sokPointRead: key-based lookup
+- sokFilter: evaluate IR expressions against rows
+- sokProject: column selection
+- sokSort: in-memory sort
+- sokLimit: slice results
+- sokInsert/sokUpdate/sokDelete: write to LSM-Tree
+
+### 0.6 Wire server to use pipeline
+- Replace `execSelect/execInsert/execDelete` with pipeline-based execution
+- Server flow: lex → parse → AST→IR lower → codegen → execute StorageOp
+- Keep backward-compatible wire protocol
+- All 56 existing tests must still pass
+
+---
+
+## Phase 1: Schema & Indexes (2–3 weeks)
+
+### 1.1 SQL type system
+- `INTEGER`, `BIGINT`, `SMALLINT`, `SERIAL` (auto-increment on INSERT)
 - `VARCHAR(n)`, `TEXT`
 - `BOOLEAN`
 - `TIMESTAMP`, `DATE` (ISO 8601)
- `JSON`, `JSONB` (in-memory + компресия)
- `UUID` (v4)
- Constraints: `PRIMARY KEY`, `FOREIGN KEY`, `UNIQUE`, `NOT NULL`, `CHECK`, `DEFAULT`
- Foreign key каскади: `ON DELETE CASCADE/SET NULL`
+- `JSON`, `JSONB`
+- `UUID` (v4 generation)
+- `NUMERIC(p,s)`, `DOUBLE PRECISION`, `REAL`

-### 1.3 B-Tree индекси + интеграция с query planner
+### 1.2 Constraints enforcement
+- PRIMARY KEY: unique index + NOT NULL
+- FOREIGN KEY + ON DELETE CASCADE/SET NULL/RESTRICT
+- UNIQUE: unique index
+- NOT NULL: check on INSERT/UPDATE
+- CHECK: evaluate expression on INSERT/UPDATE
+- DEFAULT: fill missing values on INSERT
+
+### 1.3 B-Tree index integration
 - `CREATE INDEX idx_name ON table(column)`
 - `CREATE UNIQUE INDEX`
- Покриващ индекс (covering index) за чести колони
- Query planner да избира индекс вместо full scan
- `EXPLAIN` за анализ на заявки
+- B-Tree indexes created per table column
+- Query planner uses B-Tree for WHERE clauses on indexed columns
+- Range scans via B-Tree leaf linked list

-### 1.4 Транзакции (ACID)
- `BEGIN`, `COMMIT`, `ROLLBACK`
- Isolation level: `READ COMMITTED` (първа фаза), `REPEATABLE READ` (втора)
- Deadlock detection и timeout
- MVCC интеграция с LSM-Tree (versioned reads)
- WAL за crash recovery при транзакции
+### 1.4 Query planner
+- Choose index scan vs full scan based on WHERE clause
+- Multi-column index support
+- Covering index optimization
+- `EXPLAIN` output with cost estimates
+- Adaptive query reoptimization (wire up `adaptive.nim`)

 ---

-## Фаза 2: Web API & Authentication (3–4 седмици)
+## Phase 2: Transactions (2–3 weeks)

-### 2.1 HTTP REST API
- `POST /query` — изпълнява SQL, връща JSON
- `GET /health` — readiness/liveness probe
- `GET /metrics` — брой заявки, latency, errors (Prometheus формат)
- JSON request/response body:
-  ```json
-  { "query": "SELECT * FROM users WHERE id = 1", "params": [] }
-  ```
- Batch queries: `POST /batch` — множество заявки в едно тяло
+### 2.1 Wire MVCC into server pipeline
+- `BEGIN`, `COMMIT`, `ROLLBACK` commands
+- Server tracks per-connection Transaction state
+- All reads/writes through TxnManager
+- Isolation: Read Committed (Phase 2a), Repeatable Read (Phase 2b)
+
+### 2.2 WAL crash recovery
+- Implement REDO: replay committed WAL entries into LSM-Tree
+- Implement UNDO: remove uncommitted entries on recovery
+- Checkpoint markers in WAL
+- Point-in-time recovery
+
+### 2.3 Compaction
+- Implement actual SSTable merge (currently simulated)
+- Read multiple SSTables, merge key-value pairs, write merged SSTable
+- Level-based compaction strategy
+- Background compaction scheduling
+
+### 2.4 Deadlock detection wiring
+- Wire deadlock detection into TxnManager
+- Automatic deadlock timeout and victim selection
+- Client notification on rollback
+
+---
+
+## Phase 3: HTTP REST API & Authentication (2–3 weeks)
+
+### 3.1 HTTP server
+- HTTP/1.1 server alongside TCP wire protocol (shared port or separate)
+- `POST /query` — execute SQL, return JSON
+- `GET /health` — readiness/liveness
+- `GET /metrics` — Prometheus format
 - Content-Type: `application/json`

-### 2.2 Authentication & Authorization
- JWT bearer token в HTTP header `Authorization`
- `CREATE USER` / `DROP USER` / `ALTER USER`
- `GRANT` / `REVOKE` за права на таблици
- Row-Level Security (RLS): `CREATE POLICY`
- Хеширане на пароли с bcrypt/argon2
- Rate limiting per API key / IP
+### 3.2 Authentication
+- `CREATE USER` / `DROP USER` / `ALTER USER` SQL
+- Password hashing with argon2
+- JWT token creation with HMAC-SHA256 (replace djb2 `simpleHash`)
+- `Authorization: Bearer <token>` in HTTP headers
+- Per-user namespace isolation

-### 2.3 WebSocket за real-time
- `ws://host:port/live` — subscribe към таблица/ред
- `NOTIFY` / `LISTEN` аналог
- Пуш нотификации при INSERT/UPDATE/DELETE
+### 3.3 Authorization
+- `GRANT` / `REVOKE` for table-level privileges (SELECT, INSERT, UPDATE, DELETE)
+- Row-Level Security (RLS): `CREATE POLICY` on tables
+- Wire auth into both HTTP and TCP protocol paths

-### 2.4 CORS и HTTP hardening
- CORS headers за browser достъп
- TLS termination (reuse `ssl.nim`)
+### 3.4 Rate limiting & TLS
+- Wire RateLimiter into HTTP server (token bucket per IP)
+- Wire TLS/SSL using OpenSSL FFI (not mock)
+- Self-signed cert generation
+- Configurable TLS via `baradadb cert create`
+
+---
+
+## Phase 4: WebSocket & Real-time (1–2 weeks)
+
+### 4.1 WebSocket server
+- `ws://host:port/live` — subscribe to table changes
+- `SUBSCRIBE table_name` WebSocket message
+- Push notifications on INSERT/UPDATE/DELETE
+- `NOTIFY` / `LISTEN` analogue
+
+### 4.2 CORS & HTTP hardening
+- CORS headers for browser access
 - Request size limits (10MB default)
- Connection keep-alive и HTTP/2 readiness
+- Connection keep-alive
+- HTTP/2 readiness (ALPN negotiation)

 ---

-## Фаза 3: ERP фичове (4–5 седмици)
+## Phase 5: ERP Features (3–4 weeks)

-### 3.1 Schema migrations
- `CREATE MIGRATION` / `APPLY MIGRATION`
- Версиониране на схемата в `__schema_version` таблица
- Up/down скриптове
- Dry-run режим
- CLI: `baradadb migrate status`, `baradadb migrate up`, `baradadb migrate down`
+### 5.1 Schema migrations
+- `CREATE MIGRATION` → `APPLY MIGRATION`
+- Versioned schema in `_schema_version` table
+- Up/down migration scripts
+- Dry-run mode
+- CLI: `baradadb migrate status|up|down`

-### 3.2 Views и materialized views
- `CREATE VIEW` — read-only virtual table
- `CREATE MATERIALIZED VIEW` — кеширана snapshot + `REFRESH`
- Поддръжка в query planner
+### 5.2 Views
+- `CREATE VIEW` — virtual table (stored query)
+- `CREATE MATERIALIZED VIEW` — cached snapshot + `REFRESH`
+- View usage in query planner

-### 3.3 Triggers и stored functions
- `CREATE TRIGGER` — `BEFORE`/`AFTER` INSERT/UPDATE/DELETE
- Stored functions in Nim (compiled to UDF):
-  ```sql
-  CREATE FUNCTION total_price(quantity INT, price DECIMAL) RETURNS DECIMAL
-  AS 'quantity * price';
-  ```
- Функции за ERP: `vat_calc`, `currency_convert`, `invoice_number_next`
+### 5.3 Triggers & stored functions
+- `CREATE TRIGGER` — BEFORE/AFTER on INSERT/UPDATE/DELETE
+- Stored functions in Nim (compile to UDF)
+- ERP helper functions: `vat_calc`, `currency_convert`, `invoice_number_next`

-### 3.4 Full-text search за ERP документи
- `CREATE FULLTEXT INDEX ON invoices(content)`
- `WHERE content @@ 'търсене'`
- Bulgarian stemming (reuse `fts/multilang.nim`)
+### 5.4 Full-text search for ERP documents
+- `CREATE FULLTEXT INDEX ON table(column)`
+- `WHERE content @@ 'search query'`
+- Bulgarian stemming integration

-### 3.5 Partitioning
- `CREATE TABLE orders (...) PARTITION BY RANGE (created_at)`
- Автоматичен partition pruning в query planner
- Полезно за ERP: архивиране на стари данни
+### 5.5 Partitioning
+- `CREATE TABLE (...) PARTITION BY RANGE (col)`
+- Auto partition pruning in query planner
+- Useful for ERP: archive old data by date range

 ---

-## Фаза 4: Production readiness & DevEx (3–4 седмици)
+## Phase 6: Production Readiness (2–3 weeks)

-### 4.1 Backup & Restore
+### 6.1 Backup & Restore
 - `baradadb backup --output backup.tar.gz`
 - `baradadb restore --input backup.tar.gz`
- Incremental backup чрез WAL archiving
+- Incremental backup via WAL archiving
 - Point-in-time recovery (PITR)
- Scheduled backups (cron integration)

-### 4.2 Docker и deployment
- `Dockerfile` — multi-stage build с Nim
- `docker-compose.yml` — single node + volume
+### 6.2 Docker & deployment
+- `Dockerfile` — multi-stage build with Nim
+- `docker-compose.yml` — single node
 - `docker-compose.raft.yml` — 3-node cluster
- Helm chart за Kubernetes (statefulset + PVC)
- Environment-based config (`BARADB_PORT`, `BARADB_DATA_DIR`, `BARADB_RAFT_PEERS`)
+- Environment-based config (`BARADB_PORT`, `BARADB_DATA_DIR`)

-### 4.3 Monitoring и observability
- Structured logging (JSON format)
- Prometheus `/metrics` endpoint:
-  - `baradb_queries_total`, `baradb_query_duration_seconds`
-  - `baradb_connections_active`, `baradb_storage_size_bytes`
-  - `baradb_replication_lag_seconds`
- OpenTelemetry tracing integration
- Slow query log (threshold configurable)
+### 6.3 Monitoring
+- Structured JSON logging
+- Prometheus `/metrics`: `baradb_queries_total`, `baradb_query_duration_s`, `baradb_connections_active`, `baradb_storage_size_bytes`
+- Slow query log (configurable threshold)
+- OpenTelemetry tracing

-### 4.4 ORM / Client SDK
- **Nim**: `baradb` nimble пакет — fluent query builder + миграции
-  ```nim
-  let users = db.table("users")
-    .where("active", "=", true)
-    .orderBy("created_at", "DESC")
-    .limit(10)
-    .all()
-  ```
- **Python**: `pip install baradb` — async/sync client
- **JavaScript/TypeScript**: `npm install baradb` — promise-based client
- **Go**: `go get github.com/baradb/go-client`
- Connection pooling във всички клиенти
-
-### 4.5 Admin Dashboard (Web UI)
- Лек вграден админ панел на `http://host:port/admin`
- SQL playground с резултати в таблица
- Schema browser (таблици, колони, индекси)
- Metrics charts (latency, throughput, storage)
+### 6.4 Admin dashboard
+- Web UI on `http://host:port/admin`
+- SQL playground with results table
+- Schema browser (tables, columns, indexes)
+- Metrics charts
 - User management UI

-### 4.6 Performance tuning
- Prepared statements кеш
- Query result cache (LRU, TTL-based)
- Connection pool в сървъра (max 1000 конекции)
- Auto-compaction scheduling
- Configurable cache size за page cache
+### 6.5 Client SDK improvements
+- Nim: transaction API, prepared statements, auth
+- Python: complete result parsing, transaction API, async support
+- JavaScript: actual TCP/WebSocket connection, complete result parsing
+- Go: complete result parsing, transaction API
+- Rust: complete result parsing, transaction API
+- Connection pooling in all clients

 ---

-## Приоритетна матрица
+## Priority Matrix

-| Задача | Влияние | Трудност | Приоритет |
-|--------|---------|----------|-----------|
-| SQL парсър + AST | Критично | Висока | P0 |
-| ACID транзакции | Критично | Висока | P0 |
-| HTTP REST API | Критично | Средна | P0 |
-| B-Tree индекси | Високо | Средна | P1 |
-| JWT Auth + RLS | Високо | Средна | P1 |
-| Schema migrations | Високо | Средна | P1 |
-| Docker + Compose | Средно | Ниска | P2 |
-| Backup/Restore | Средно | Средна | P2 |
-| WebSocket real-time | Средно | Средна | P2 |
-| Admin Dashboard | Средно | Висока | P2 |
-| Views + Triggers | Ниско | Средна | P3 |
-| Client SDK (ORM) | Ниско | Висока | P3 |
-| Partitioning | Ниско | Висока | P3 |
-| Kubernetes Helm | Ниско | Средна | P3 |
+| Task | Impact | Difficulty | Priority |
+|------|--------|-----------|----------|
+| Pipeline integration (Phase 0) | Critical | High | **P0** |
+| SQL DDL parser (Phase 0) | Critical | Medium | **P0** |
+| AST→IR lowering (Phase 0) | Critical | High | **P0** |
+| Codegen execution (Phase 0) | Critical | High | **P0** |
+| SQL schema system (Phase 1) | Critical | High | **P0** |
+| B-Tree index integration (Phase 1) | High | Medium | P1 |
+| Constraint enforcement (Phase 1) | High | Medium | P1 |
+| MVCC wiring (Phase 2) | Critical | High | **P0** |
+| WAL recovery (Phase 2) | High | Medium | P1 |
+| HTTP REST API (Phase 3) | Critical | Medium | **P0** |
+| JWT Auth + RLS (Phase 3) | High | Medium | P1 |
+| WebSocket real-time (Phase 4) | Medium | Medium | P2 |
+| Schema migrations (Phase 5) | High | Medium | P1 |
+| Backup/Restore (Phase 6) | Medium | Medium | P2 |
+| Docker + Compose (Phase 6) | Medium | Low | P2 |
+| Admin Dashboard (Phase 6) | Medium | High | P2 |
+| Views + Triggers (Phase 5) | Low | Medium | P3 |
+| Partitioning (Phase 5) | Low | High | P3 |
+| Client SDK (Phase 6) | Medium | High | P2 |
+| Kubernetes Helm (Phase 6) | Low | Medium | P3 |

 ---

-## Очакван резултат
+## Expected Results

- **Фаза 1:** BaraDB поддържа ANSI SQL subset с ACID транзакции. Може да замени SQLite/PostgreSQL за малки проекти.
- **Фаза 2:** REST API + auth правят базата достъпна от всякакъв web stack. WebSocket добавя real-time възможности.
- **Фаза 3:** ERP-ready фичове — migrations, views, triggers, partitioning. Може да поддържа реален бизнес софтуер.
- **Фаза 4:** Production tooling — Docker, backup, monitoring, ORM, admin UI. Solo-dev може да deploy-не за 5 минути.
+- **Phase 0:** Server uses full pipeline. INSERT/UPDATE/DELETE/CREATE TABLE work properly. 56 existing tests pass + new tests.
+- **Phase 1:** SQL schema with constraints, B-Tree indexes, EXPLAIN. Can define tables with PKs, FKs, and indexes.
+- **Phase 2:** ACID transactions with MVCC, WAL recovery, compaction. Can use BEGIN/COMMIT/ROLLBACK.
+- **Phase 3:** HTTP REST API with JWT auth, user management, rate limiting. DB accessible from browser.
+- **Phase 4:** Real-time WebSocket subscriptions. Notifications on data changes.
+- **Phase 5:** ERP-grade features: migrations, views, triggers, partitioning, full-text search.
+- **Phase 6:** Docker, backup, monitoring, admin UI. Deploy in 5 minutes.

-**Крайна оценка след плана:** от 8.5/10 към 9.5/10 — готова за production web/ERP.
+**Final score after plan:** 9.5/10 — production-ready for web/ERP workloads.