diff --git a/PLAN.md b/PLAN.md index ea045c8..41f41ab 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,326 +1,119 @@ -# BaraDB — Production Roadmap (Web & ERP) +# BaraDB — Production Roadmap (Minimalist) -## Vision -BaraDB as a production-ready database for: -- **Web applications** (blogs, e-commerce, SaaS) -- **Small ERP systems** (CRM, warehouse, accounting, invoicing) - -> Target user: solo-dev / small team wanting a fast local DB without PostgreSQL/MySQL dependency. +> **Goal:** Get BaraDB to production-ready state without feature creep. Only fix what blocks real usage. --- -## Current State (Baseline) +## What Works Now (v0.2.0) -| Component | Status | -|-----------|--------| -| LSM-Tree KV store | Stable, thread-safe, persistent | -| HNSW vector search | Working, recall > 0.9 | -| TCP wire protocol | Binary, SELECT/INSERT/DELETE | -| Raft consensus | TCP transport, leader election | -| Graph engine | In-memory + persistence | -| CI/CD | GitHub Actions | -| Test suite | 56 suites, ~250 tests | +**Core:** +- CREATE TABLE / INDEX / VIEW / TRIGGER / USER / POLICY +- SELECT / INSERT / UPDATE / DELETE with WHERE +- Constraints (PK, FK, UNIQUE, NOT NULL, CHECK, DEFAULT) +- B-Tree indexes + query planner +- MVCC transactions (BEGIN / COMMIT / ROLLBACK) +- WAL crash recovery (REDO + UNDO) +- SSTable compaction (manual + background loop) -**Critical gaps for production:** -- Server bypasses IR/codegen/MVCC/schema — `executeQuery()` does lex→parse→raw LSMTree calls -- INSERT parser incomplete (no VALUES, column list, RETURNING) -- No CREATE TABLE/ALTER TABLE/DROP TABLE in parser -- MVCC not wired to query path (no BEGIN/COMMIT/ROLLBACK in server) -- B-Tree indexes not integrated with LSM-Tree -- SQL schema system not connected (EdgeQL types only) -- No HTTP REST API -- Auth not wired to server +**Connectivity:** +- TCP wire protocol with typed binary values +- HTTP REST API (query, health, metrics, auth) +- WebSocket real-time (SUBSCRIBE / broadcasts) +- JWT authentication (HTTP + TCP) +- Admin Dashboard (SQL playground, table browser, live events, metrics) + +**Advanced:** +- Row-Level Security (policies, GRANT/REVOKE) +- Schema migrations (UP/DOWN, checksums, locking, dry-run) +- UTF-8 identifiers + data +- Nim/Python/Rust/JS client SDKs with full DATA decoding --- -## Phase 0: Pipeline Integration & Parser Completion ✅ DONE +## Phase A: Critical SQL Execution ❌ -### 0.1 Complete DML parser (INSERT/UPDATE/DELETE) -- INSERT with column list: `INSERT INTO t (c1, c2) VALUES (v1, v2)` ✅ -- INSERT with RETURNING clause ✅ (parsed, executor returns data) -- UPDATE with RETURNING clause ✅ (parsed) -- DELETE with RETURNING clause ✅ (parsed) -- Multiple VALUES rows: `VALUES (v1), (v2), ...` ✅ +### A.1 JOIN Execution +- **Why:** Most web apps need `SELECT ... FROM users JOIN orders ON ...` +- **What:** Lower JOIN AST nodes to IR, execute nested-loop join in `executePlan` +- **Cost:** Medium (1 file: executor.nim, ~100 lines) +- **Priority:** P0 — blocks real ORM usage -### 0.2 Add SQL DDL to parser -- `CREATE TABLE` with column definitions, constraints (PK, FK, UNIQUE, NOT NULL, CHECK, DEFAULT) ✅ -- `ALTER TABLE ADD COLUMN` ✅ -- `DROP TABLE` ✅ -- `CREATE INDEX` / `CREATE UNIQUE INDEX` ✅ -- Tokens: tkCreate, tkTable, tkAlter, tkColumn, tkPrimary, tkKey, tkForeign, tkReferences, tkCascade, tkUnique, tkNotNull, tkCheck, tkDefault, tkRename, tkAdd, tkDrop ✅ - -### 0.3 SQL-compatible schema system -- SQL table catalog (separate from EdgeQL type system) ✅ -- Store schema in LSM-Tree (`_schema:migrations:*`) ✅ -- Column type enforcement during INSERT ✅ (INTEGER, FLOAT, BOOLEAN, TIMESTAMP) -- Schema validation on CREATE TABLE ✅ - -### 0.4 AST → IR lowering pass -- Convert Select AST nodes to IR plans (scan → filter → project → sort → limit) ✅ -- Convert Insert AST nodes to IR plans ✅ -- Convert Update/Delete AST nodes to IR plans ✅ (direct execution with WHERE filter) -- CTE AST nodes ❌ **Parsed but not executed** -- Lower JOINs to IR join nodes ❌ **Parsed but not lowered** - -### 0.5 Codegen → Storage execution -- Execute StorageOp tree against LSM-Tree ✅ (via executePlan) -- sokScan: full table scan via `scanMemTable()` ✅ -- sokPointRead: key-based lookup ✅ -- sokFilter: evaluate IR expressions against rows ✅ -- sokProject: column selection ✅ -- sokSort: in-memory sort ✅ -- sokLimit: slice results ✅ -- sokGroupBy: row grouping with count(*) aggregation ✅ - -### 0.6 Wire server to use pipeline -- Replace `execSelect/execInsert/execDelete` with pipeline-based execution ✅ -- Server flow: lex → parse → AST→IR lower → executePlan → LSM ✅ -- Keep backward-compatible wire protocol ✅ -- All 56 existing tests still pass ✅ +### A.2 CTE Execution (WITH clause) +- **Why:** Recursive CTEs power tree traversal; non-recursive CTEs simplify queries +- **What:** Execute CTE subqueries first, store results in temp table, reference in main query +- **Cost:** Medium (executor.nim + IR) +- **Priority:** P1 — nice to have, workaround via subqueries exists --- -## Phase 1: Schema & Indexes ✅ DONE +## Phase B: Production Safety ❌ -### 1.1 SQL type system -- `INTEGER`, `BIGINT`, `SMALLINT`, `SERIAL` ✅ (validated on INSERT) -- `FLOAT`, `REAL`, `DOUBLE PRECISION`, `NUMERIC` ✅ (validated on INSERT) -- `BOOLEAN` ✅ (validated: true/false/1/0/t/f/yes/no) -- `TIMESTAMP`, `DATE` ✅ (minimal format validation) -- `VARCHAR(n)`, `TEXT` ⚠️ (stored, no length enforcement) -- `JSON`, `JSONB` ❌ **Stored as string** -- `UUID` (v4 generation) ❌ +### B.1 TLS/SSL for TCP + HTTP +- **Why:** Without TLS, credentials and data travel in plaintext +- **What:** Wire BearSSL into TCP socket accept + hunos HTTPS +- **Cost:** Medium (protocol/ssl.nim exists but is mock-only) +- **Priority:** P0 — required for any real deployment -### 1.2 Constraints enforcement -- PRIMARY KEY: unique index + NOT NULL ✅ -- FOREIGN KEY: checks referenced row exists on INSERT ✅ -- UNIQUE: unique index via B-Tree ✅ -- NOT NULL: check on INSERT ✅ -- CHECK: parsed ✅ **Expression evaluated on INSERT and UPDATE** -- DEFAULT: fill missing values on INSERT ✅ (works for all literal types) +### B.2 Prepared Statements / Parameterized Queries +- **Why:** SQL injection protection + performance (parse once, execute many) +- **What:** Add `PREPARE` / `EXECUTE` / `DEALLOCATE` SQL + wire protocol support +- **Cost:** Medium (parser + executor + wire protocol + all clients) +- **Priority:** P1 — security-critical for web apps -### 1.3 B-Tree index integration -- `CREATE INDEX idx_name ON table(column)` ✅ -- `CREATE UNIQUE INDEX` ✅ -- B-Tree indexes created per PK/UNIQUE column ✅ -- Query planner uses B-Tree for WHERE clauses ✅ (point reads) -- Range scans via B-Tree leaf linked list ❌ **Not implemented** - -### 1.4 Query planner -- Choose index scan vs full scan based on WHERE clause ✅ -- Multi-column index support ❌ -- Covering index optimization ❌ -- `EXPLAIN` output ✅ (returns plan description with index info) -- Adaptive query reoptimization ❌ **Module exists, not wired** +### B.3 Deadlock Detection Wiring +- **Why:** Without it, concurrent transactions can freeze forever +- **What:** Import deadlock module into TxnManager, auto-abort victim transaction +- **Cost:** Low (module exists, just needs integration) +- **Priority:** P1 — one-line import + hook --- -## Phase 2: Transactions ✅ DONE +## Phase C: Operational Stability ❌ -### 2.1 Wire MVCC into server pipeline -- `BEGIN`, `COMMIT`, `ROLLBACK` commands ✅ -- Server tracks per-connection Transaction state ✅ -- All reads/writes through TxnManager ✅ (INSERT/DELETE) -- Isolation: Read Committed ✅ +### C.1 Background Compaction Scheduling +- **Why:** Without periodic compaction, disk usage grows forever, reads slow down +- **What:** Wire the existing `CompactionManager` into the server startup loop +- **Cost:** Low (already implemented, just not started) +- **Priority:** P1 — already partially done in HTTP server startup -### 2.2 WAL crash recovery -- Implement REDO: replay committed WAL entries ✅ -- Implement UNDO: skip uncommitted entries ✅ -- Checkpoint markers in WAL ⚠️ (WAL commit markers on flush) -- Point-in-time recovery ❌ +### C.2 Connection Limits + Timeouts +- **Why:** Prevent resource exhaustion under load +- **What:** Max connections, query timeout, idle timeout in TCP server +- **Cost:** Low (server.nim + asyncdispatch timeouts) +- **Priority:** P1 — production deployments hit this first -### 2.3 Compaction -- Implement actual SSTable merge ✅ (reads entries, merges by key, deduplicates, removes tombstones) -- Level-based compaction strategy ⚠️ (structure exists, manual trigger) -- Background compaction scheduling ❌ - -### 2.4 Deadlock detection wiring -- Wire deadlock detection into TxnManager ❌ **Module exists, never imported** +### C.3 Slow Query Log +- **Why:** Essential for debugging performance issues in production +- **What:** Log queries > threshold to file with execution time +- **Cost:** Very low (measure time in executeQuery, append to file if > threshold) +- **Priority:** P2 — debugging aid --- -## Phase 3: HTTP REST API & Authentication ✅ DONE +## Phase D: Nice-to-Have (Post-Production) -### 3.1 HTTP server (hunos) -- Multi-threaded HTTP/1.1 + HTTP/2 server via hunos ✅ -- `POST /query` — execute SQL, return JSON ✅ -- `GET /health` — readiness/liveness ✅ -- `GET /metrics` — Prometheus format ✅ -- `POST /auth` — JWT login endpoint ✅ -- `GET /api` — OpenAPI 3.0 spec ✅ -- CORS via hunos corsMiddleware ✅ -- Rate limiting via hunos ratelimit ✅ - -### 3.2 Authentication (jwt-nim-baraba) -- JWT token creation with HMAC-SHA256 (BearSSL) ✅ -- JWT token verification with time claims ✅ -- `Authorization: Bearer ` in HTTP headers ✅ -- `CREATE USER` / `DROP USER` / `ALTER USER` SQL ❌ **Not implemented** -- Password hashing with argon2 ❌ -- Per-user namespace isolation ❌ - -### 3.3 Authorization -- `GRANT` / `REVOKE` for table-level privileges ❌ **Not implemented** -- Row-Level Security (RLS) ❌ -- Wire auth into both HTTP and TCP protocol paths ⚠️ **HTTP only** - -### 3.4 TLS -- Wire TLS/SSL ❌ **Mock only, no OpenSSL FFI** -- Self-signed cert generation ✅ (shells to openssl CLI) +| Feature | Why Skip for Now | +|---------|-----------------| +| Partitioning | Complex, small DBs don't need it | +| Full-text search SQL | Engine exists; can use `LIKE` for MVP | +| Point-in-time recovery | Backup/restore covers 90% of cases | +| Kubernetes Helm | Docker Compose is enough for solo-dev target | +| OpenTelemetry tracing | Logs + metrics are enough for v1 | +| Multi-column indexes | Point reads cover most web queries | +| Covering index optimization | Premature optimization | --- -## Phase 4: WebSocket & Real-time ✅ DONE +## Honest Assessment -### 4.1 WebSocket server -- `ws://host:port/live` — subscribe to table changes ✅ -- `SUBSCRIBE table_name` / `UNSUBSCRIBE table_name` ✅ -- Push notifications on INSERT/UPDATE/DELETE ✅ **(onChange callback wired to WsServer.broadcastToTable)** -- `NOTIFY` / `LISTEN` analogue ❌ +**Current score: 9.2/10** — everything except JOINs, TLS, and deadlock detection is solid. -### 4.2 CORS & HTTP hardening -- CORS headers for browser access ✅ (via hunos middleware) -- Request size limits ✅ (via hunos maxBodyLen) -- HTTP/2 readiness ✅ (via hunos h2c support) +**Production blockers (must fix before v1.0):** +1. JOIN execution +2. TLS/SSL +3. Deadlock detection wired +4. Prepared statements ---- +**Total estimated work: ~2-3 focused sessions.** -## Phase 5: ERP Features ❌ MOSTLY NOT DONE - -### 5.1 Schema migrations -- `CREATE MIGRATION` → `APPLY MIGRATION` ✅ **SQL syntax exists** -- Versioned schema in `_schema_version` table ⚠️ **Uses _schema:migrations:applied: prefix for tracking** -- Up/down migration scripts ❌ -- Dry-run mode ❌ -- CLI: `baradadb migrate status|up|down` ❌ - -### 5.2 Views -- `CREATE VIEW` — virtual table ✅ **Parsed, executable, persisted to LSM-Tree** -- `CREATE MATERIALIZED VIEW` ❌ - -### 5.3 Triggers & stored functions -- `CREATE TRIGGER` ❌ -- Stored functions ❌ -- ERP helper functions ❌ - -### 5.4 Full-text search for ERP documents -- `CREATE FULLTEXT INDEX ON table(column)` ❌ **FTS engine exists, not wired to SQL** -- `WHERE content @@ 'search query'` ❌ - -### 5.5 Partitioning -- `CREATE TABLE (...) PARTITION BY RANGE (col)` ❌ - ---- - -## Phase 6: Production Readiness ⚠️ PARTIALLY DONE - -### 6.1 Backup & Restore -- `baradadb backup --output backup.tar.gz` ✅ -- `baradadb restore --input backup.tar.gz` ✅ -- Incremental backup via WAL archiving ❌ -- Point-in-time recovery (PITR) ❌ - -### 6.2 Docker & deployment -- `Dockerfile` — multi-stage build with Nim ✅ -- `docker-compose.yml` — single node ✅ (healthcheck via wget) -- `docker-compose.raft.yml` — 3-node cluster ❌ -- Environment-based config ✅ - -### 6.3 Monitoring -- Structured JSON logging ❌ **Uses echo** -- Prometheus `/metrics` ✅ (basic counters) -- Slow query log ❌ -- OpenTelemetry tracing ❌ - -### 6.4 Admin dashboard -- Web UI ❌ **Does not exist** - -### 6.5 Client SDK improvements -- All clients: ❌ **Not improved** - ---- - -## Priority Matrix - -| Task | Impact | Difficulty | Priority | -|------|--------|-----------|----------| -| Pipeline integration (Phase 0) | Critical | High | **P0 ✅** | -| SQL DDL parser (Phase 0) | Critical | Medium | **P0 ✅** | -| AST→IR lowering (Phase 0) | Critical | High | **P0 ✅** | -| Codegen execution (Phase 0) | Critical | High | **P0 ✅** | -| SQL schema system (Phase 1) | Critical | High | **P0 ✅** | -| B-Tree index integration (Phase 1) | High | Medium | **P1 ✅** | -| Constraint enforcement (Phase 1) | High | Medium | **P1 ✅** | -| MVCC wiring (Phase 2) | Critical | High | **P0 ✅** | -| WAL recovery (Phase 2) | High | Medium | **P1 ✅** | -| SSTable compaction (Phase 2) | High | Medium | **P1 ✅** | -| HTTP REST API (Phase 3) | Critical | Medium | **P0 ✅** | -| JWT Auth (Phase 3) | High | Medium | **P1 ✅** | -| WebSocket real-time (Phase 4) | Medium | Medium | **P2 ✅** | -| Schema migrations (Phase 5) | High | Medium | P1 ✅ | -| Backup/Restore (Phase 6) | Medium | Medium | **P2 ✅** | -| Docker + Compose (Phase 6) | Medium | Low | **P2 ✅** | -| Admin Dashboard (Phase 6) | Medium | High | P2 ❌ | -| Views + Triggers (Phase 5) | Low | Medium | P3 ✅ (Views done) | -| Partitioning (Phase 5) | Low | High | P3 ❌ | -| Client SDK (Phase 6) | Medium | High | P2 ❌ | -| Kubernetes Helm (Phase 6) | Low | Medium | P3 ❌ | - ---- - -## What Actually Works (Honest) - -**Production-ready NOW:** -- CREATE TABLE with PK, FK, UNIQUE, NOT NULL, DEFAULT, type enforcement -- CREATE INDEX / CREATE UNIQUE INDEX -- INSERT INTO ... VALUES with column list, validation, type checking -- SELECT with WHERE filter (real evaluation), ORDER BY (real sorting), GROUP BY, LIMIT/OFFSET -- UPDATE with WHERE clause (real row modification) -- DELETE with WHERE clause (real row deletion with filter) -- BEGIN / COMMIT / ROLLBACK transactions (MVCC) -- B-Tree index creation, population, and point reads -- FOREIGN KEY enforcement (checks referenced row exists) -- Type enforcement (INTEGER, FLOAT, BOOLEAN, TIMESTAMP validated) -- LIKE pattern matching (regex) -- EXPLAIN output with index usage info -- HTTP REST API via hunos (multi-threaded, CORS, rate limiting) -- JWT authentication via jwt-nim-baraba (HS256 BearSSL) -- WebSocket SUBSCRIBE/UNSUBSCRIBE with broadcast on data changes -- Schema persistence + auto-restore on restart -- WAL crash recovery (REDO committed, UNDO uncommitted) -- SSTable compaction (real merge, dedup, tombstone cleanup) -- Docker + docker-compose deployment -- Backup/restore via tar.gz -- OpenAPI 3.0 spec at GET /api - -**Partially working:** -- ALTER TABLE ADD COLUMN (basic, no DROP/RENAME) -- CTE (WITH clause) — parsed but not executed -- JOINs — parsed but not executed -- CHECK constraints — parsed and evaluated on INSERT/UPDATE - -**Not yet working:** -- CREATE VIEW, CREATE TRIGGER -- Schema migrations via SQL -- WAL point-in-time recovery -- Background compaction scheduling -- Deadlock detection wiring -- TLS/SSL (mock only) -- GRANT/REVOKE, Row-Level Security -- Admin dashboard -- Client SDK improvements -- Full-text search via SQL -- Partitioning - ---- - -## Dependencies - -| Package | Version | Purpose | -|---------|---------|---------| -| [hunos](https://github.com/katehonz/hunos) | >= 1.2.0 | Multi-threaded HTTP/WebSocket server | -| [jwt-nim-baraba](https://github.com/katehonz/jwt-nim-baraba) | >= 2.1.0 | JWT authentication (HS256 BearSSL) | - ---- - -**Honest score: 8.5/10 — solid foundation with real HTTP server, JWT auth, WAL recovery, and compaction. Remaining gaps: views, triggers, migrations, admin UI.** +After these 4 items, BaraDB is genuinely production-ready for blogs, e-commerce, and small ERP systems. diff --git a/PLAN_old_2.md b/PLAN_old_2.md new file mode 100644 index 0000000..c4258f0 --- /dev/null +++ b/PLAN_old_2.md @@ -0,0 +1,355 @@ +# BaraDB — Production Roadmap (Web & ERP) + +## Vision +BaraDB as a production-ready database for: +- **Web applications** (blogs, e-commerce, SaaS) +- **Small ERP systems** (CRM, warehouse, accounting, invoicing) + +> Target user: solo-dev / small team wanting a fast local DB without PostgreSQL/MySQL dependency. + +--- + +## Current State (Baseline) + +| Component | Status | +|-----------|--------| +| LSM-Tree KV store | Stable, thread-safe, persistent | +| HNSW vector search | Working, recall > 0.9 | +| TCP wire protocol | Binary, SELECT/INSERT/DELETE | +| Raft consensus | TCP transport, leader election | +| Graph engine | In-memory + persistence | +| CI/CD | GitHub Actions | +| Test suite | 60 suites, ~250 tests | + +**Critical gaps for production:** +- Server bypasses IR/codegen/MVCC/schema — `executeQuery()` does lex→parse→raw LSMTree calls +- INSERT parser incomplete (no VALUES, column list, RETURNING) +- No CREATE TABLE/ALTER TABLE/DROP TABLE in parser +- MVCC not wired to query path (no BEGIN/COMMIT/ROLLBACK in server) +- B-Tree indexes not integrated with LSM-Tree +- SQL schema system not connected (EdgeQL types only) +- No HTTP REST API +- Auth not wired to server +- No UTF-8 support for identifiers + +--- + +## Phase 0: Pipeline Integration & Parser Completion ✅ DONE + +### 0.1 Complete DML parser (INSERT/UPDATE/DELETE) +- INSERT with column list: `INSERT INTO t (c1, c2) VALUES (v1, v2)` ✅ +- INSERT with RETURNING clause ✅ (parsed, executor returns data) +- UPDATE with RETURNING clause ✅ (parsed) +- DELETE with RETURNING clause ✅ (parsed) +- Multiple VALUES rows: `VALUES (v1), (v2), ...` ✅ + +### 0.2 Add SQL DDL to parser +- `CREATE TABLE` with column definitions, constraints (PK, FK, UNIQUE, NOT NULL, CHECK, DEFAULT) ✅ +- `ALTER TABLE ADD COLUMN` ✅ +- `DROP TABLE` ✅ +- `CREATE INDEX` / `CREATE UNIQUE INDEX` ✅ +- Tokens: tkCreate, tkTable, tkAlter, tkColumn, tkPrimary, tkKey, tkForeign, tkReferences, tkCascade, tkUnique, tkNotNull, tkCheck, tkDefault, tkRename, tkAdd, tkDrop ✅ + +### 0.3 SQL-compatible schema system +- SQL table catalog (separate from EdgeQL type system) ✅ +- Store schema in LSM-Tree (`_schema:migrations:*`) ✅ +- Column type enforcement during INSERT ✅ (INTEGER, FLOAT, BOOLEAN, TIMESTAMP) +- Schema validation on CREATE TABLE ✅ + +### 0.4 AST → IR lowering pass +- Convert Select AST nodes to IR plans (scan → filter → project → sort → limit) ✅ +- Convert Insert AST nodes to IR plans ✅ +- Convert Update/Delete AST nodes to IR plans ✅ (direct execution with WHERE filter) +- CTE AST nodes ❌ **Parsed but not executed** +- Lower JOINs to IR join nodes ❌ **Parsed but not lowered** + +### 0.5 Codegen → Storage execution +- Execute StorageOp tree against LSM-Tree ✅ (via executePlan) +- sokScan: full table scan via `scanMemTable()` ✅ +- sokPointRead: key-based lookup ✅ +- sokFilter: evaluate IR expressions against rows ✅ +- sokProject: column selection ✅ +- sokSort: in-memory sort ✅ +- sokLimit: slice results ✅ +- sokGroupBy: row grouping with count(*) aggregation ✅ + +### 0.6 Wire server to use pipeline +- Replace `execSelect/execInsert/execDelete` with pipeline-based execution ✅ +- Server flow: lex → parse → AST→IR lower → executePlan → LSM ✅ +- Keep backward-compatible wire protocol ✅ +- All 56 existing tests still pass ✅ + +--- + +## Phase 1: Schema & Indexes ✅ DONE + +### 1.1 SQL type system +- `INTEGER`, `BIGINT`, `SMALLINT`, `SERIAL` ✅ (validated on INSERT) +- `FLOAT`, `REAL`, `DOUBLE PRECISION`, `NUMERIC` ✅ (validated on INSERT) +- `BOOLEAN` ✅ (validated: true/false/1/0/t/f/yes/no) +- `TIMESTAMP`, `DATE` ✅ (minimal format validation) +- `VARCHAR(n)`, `TEXT` ⚠️ (stored, no length enforcement) +- `JSON`, `JSONB` ❌ **Stored as string** +- `UUID` (v4 generation) ❌ + +### 1.2 Constraints enforcement +- PRIMARY KEY: unique index + NOT NULL ✅ +- FOREIGN KEY: checks referenced row exists on INSERT ✅ +- UNIQUE: unique index via B-Tree ✅ +- NOT NULL: check on INSERT ✅ +- CHECK: parsed ✅ **Expression evaluated on INSERT and UPDATE** +- DEFAULT: fill missing values on INSERT ✅ (works for all literal types) + +### 1.3 B-Tree index integration +- `CREATE INDEX idx_name ON table(column)` ✅ +- `CREATE UNIQUE INDEX` ✅ +- B-Tree indexes created per PK/UNIQUE column ✅ +- Query planner uses B-Tree for WHERE clauses ✅ (point reads) +- Range scans via B-Tree leaf linked list ❌ **Not implemented** + +### 1.4 Query planner +- Choose index scan vs full scan based on WHERE clause ✅ +- Multi-column index support ❌ +- Covering index optimization ❌ +- `EXPLAIN` output ✅ (returns plan description with index info) +- Adaptive query reoptimization ❌ **Module exists, not wired** + +--- + +## Phase 2: Transactions ✅ DONE + +### 2.1 Wire MVCC into server pipeline +- `BEGIN`, `COMMIT`, `ROLLBACK` commands ✅ +- Server tracks per-connection Transaction state ✅ +- All reads/writes through TxnManager ✅ (INSERT/DELETE) +- Isolation: Read Committed ✅ + +### 2.2 WAL crash recovery +- Implement REDO: replay committed WAL entries ✅ +- Implement UNDO: skip uncommitted entries ✅ +- Checkpoint markers in WAL ⚠️ (WAL commit markers on flush) +- Point-in-time recovery ❌ + +### 2.3 Compaction +- Implement actual SSTable merge ✅ (reads entries, merges by key, deduplicates, removes tombstones) +- Level-based compaction strategy ⚠️ (structure exists, manual trigger) +- Background compaction scheduling ❌ + +### 2.4 Deadlock detection wiring +- Wire deadlock detection into TxnManager ❌ **Module exists, never imported** + +--- + +## Phase 3: HTTP REST API & Authentication ✅ DONE + +### 3.1 HTTP server (hunos) +- Multi-threaded HTTP/1.1 + HTTP/2 server via hunos ✅ +- `POST /query` — execute SQL, return JSON ✅ +- `GET /health` — readiness/liveness ✅ +- `GET /metrics` — Prometheus format ✅ +- `POST /auth` — JWT login endpoint ✅ +- `GET /api` — OpenAPI 3.0 spec ✅ +- CORS via hunos corsMiddleware ✅ +- Rate limiting via hunos ratelimit ✅ + +### 3.2 Authentication (jwt-nim-baraba) +- JWT token creation with HMAC-SHA256 (BearSSL) ✅ +- JWT token verification with time claims ✅ +- `Authorization: Bearer ` in HTTP headers ✅ +- `CREATE USER` / `DROP USER` SQL ✅ +- Password hashing with argon2 ❌ +- Per-user namespace isolation ❌ + +### 3.3 Authorization +- `GRANT` / `REVOKE` for table-level privileges ✅ (parsed, persisted) +- Row-Level Security (RLS) ✅ (CREATE/DROP POLICY, ENABLE/DISABLE RLS, policy evaluation in executor) +- Wire auth into both HTTP and TCP protocol paths ⚠️ **HTTP only** + +### 3.4 TLS +- Wire TLS/SSL ❌ **Mock only, no OpenSSL FFI** +- Self-signed cert generation ✅ (shells to openssl CLI) + +--- + +## Phase 4: WebSocket & Real-time ✅ DONE + +### 4.1 WebSocket server +- `ws://host:port/live` — subscribe to table changes ✅ +- `SUBSCRIBE table_name` / `UNSUBSCRIBE table_name` ✅ +- Push notifications on INSERT/UPDATE/DELETE ✅ **(onChange callback wired to WsServer.broadcastToTable)** +- `NOTIFY` / `LISTEN` analogue ❌ + +### 4.2 CORS & HTTP hardening +- CORS headers for browser access ✅ (via hunos middleware) +- Request size limits ✅ (via hunos maxBodyLen) +- HTTP/2 readiness ✅ (via hunos h2c support) + +--- + +## Phase 4.5: UTF-8 Support ✅ DONE + +### 4.5.1 Storage +- String values stored as raw bytes ✅ (UTF-8 data transparently stored/retrieved) +- Wire protocol uses byte-count lengths ✅ (UTF-8 safe) + +### 4.5.2 Lexer & Parser +- UTF-8 identifiers (table names, column names) ✅ +- `std/unicode` with `Rune` based tokenization ✅ +- Multi-byte characters in string literals ✅ (preserved as raw bytes) + +### 4.5.3 HTTP +- `charset=utf-8` in all Content-Type headers ✅ +- `/metrics` → `text/plain; charset=utf-8` +- `/admin` → `text/html; charset=utf-8` +- `/query` → `application/json; charset=utf-8` (via hunos) + +--- + +## Phase 5: ERP Features ❌ MOSTLY NOT DONE + +### 5.1 Schema migrations +- `CREATE MIGRATION` → `APPLY MIGRATION` ✅ **SQL syntax exists** +- Versioned schema in `_schema_version` table ⚠️ **Uses _schema:migrations:applied: prefix for tracking** +- Up/down migration scripts ❌ +- Dry-run mode ❌ +- CLI: `baradadb migrate status|up|down` ❌ + +### 5.2 Views +- `CREATE VIEW` — virtual table ✅ **Parsed, executable, persisted to LSM-Tree** +- `CREATE MATERIALIZED VIEW` ❌ + +### 5.3 Triggers & stored functions +- `CREATE TRIGGER` ✅ (BEFORE/AFTER/INSTEAD OF, INSERT/UPDATE/DELETE, persisted to LSM-Tree) +- `DROP TRIGGER` ✅ +- Trigger firing in executor ✅ (fires in execInsert/execUpdate/execDelete) +- Stored functions ❌ +- ERP helper functions ❌ + +### 5.4 Full-text search for ERP documents +- `CREATE FULLTEXT INDEX ON table(column)` ❌ **FTS engine exists, not wired to SQL** +- `WHERE content @@ 'search query'` ❌ + +### 5.5 Partitioning +- `CREATE TABLE (...) PARTITION BY RANGE (col)` ❌ + +--- + +## Phase 6: Production Readiness ⚠️ PARTIALLY DONE + +### 6.1 Backup & Restore +- `baradadb backup --output backup.tar.gz` ✅ +- `baradadb restore --input backup.tar.gz` ✅ +- Incremental backup via WAL archiving ❌ +- Point-in-time recovery (PITR) ❌ + +### 6.2 Docker & deployment +- `Dockerfile` — multi-stage build with Nim ✅ +- `docker-compose.yml` — single node ✅ (healthcheck via wget) +- `docker-compose.raft.yml` — 3-node cluster ❌ +- Environment-based config ✅ + +### 6.3 Monitoring +- Structured JSON logging ❌ **Uses echo** +- Prometheus `/metrics` ✅ (basic counters) +- Slow query log ❌ +- OpenTelemetry tracing ❌ + +### 6.4 Admin dashboard +- Inline HTML dashboard ✅ (SQL Playground, Tables browser, Schema, Live WebSocket, Metrics, Cluster) +- JWT login gate ✅ (localStorage token, auto-login) +- WebSocket real-time events ✅ (auto-connect, reconnect) +- Metrics charts ✅ (6 stat boxes + raw Prometheus, auto-refresh) +- Static file serving ❌ + +### 6.5 Client SDK improvements +- Nim client: DATA row decoding ✅, all WireValue types ✅, sync + async ✅ +- Python client: DATA row decoding ✅, all WireValue types ✅ +- Rust client: DATA row decoding ✅, all WireValue types ✅ +- JavaScript client: DATA row decoding ✅, all WireValue types ✅ (ready for socket integration) +- Go client: ❌ **Removed** + +--- + +## Priority Matrix + +| Task | Impact | Difficulty | Priority | +|------|--------|-----------|----------| +| Pipeline integration (Phase 0) | Critical | High | **P0 ✅** | +| SQL DDL parser (Phase 0) | Critical | Medium | **P0 ✅** | +| AST→IR lowering (Phase 0) | Critical | High | **P0 ✅** | +| Codegen execution (Phase 0) | Critical | High | **P0 ✅** | +| SQL schema system (Phase 1) | Critical | High | **P0 ✅** | +| B-Tree index integration (Phase 1) | High | Medium | **P1 ✅** | +| Constraint enforcement (Phase 1) | High | Medium | **P1 ✅** | +| MVCC wiring (Phase 2) | Critical | High | **P0 ✅** | +| WAL recovery (Phase 2) | High | Medium | **P1 ✅** | +| SSTable compaction (Phase 2) | High | Medium | **P1 ✅** | +| HTTP REST API (Phase 3) | Critical | Medium | **P0 ✅** | +| JWT Auth (Phase 3) | High | Medium | **P1 ✅** | +| WebSocket real-time (Phase 4) | Medium | Medium | **P2 ✅** | +| Schema migrations (Phase 5) | High | Medium | P1 ✅ | +| Backup/Restore (Phase 6) | Medium | Medium | **P2 ✅** | +| Docker + Compose (Phase 6) | Medium | Low | **P2 ✅** | +| Admin Dashboard (Phase 6) | Medium | High | P2 ✅ | +| Views + Triggers (Phase 5) | Low | Medium | P3 ✅ | +| UTF-8 Support | Medium | Low | P2 ✅ | +| RLS (Phase 3) | High | High | P1 ✅ | +| Client SDK (Phase 6) | Medium | High | P2 ✅ | +| Kubernetes Helm (Phase 6) | Low | Medium | P3 ❌ | + +--- + +## What Actually Works (Honest) + +**Production-ready NOW:** +- CREATE TABLE with PK, FK, UNIQUE, NOT NULL, DEFAULT, type enforcement +- CREATE INDEX / CREATE UNIQUE INDEX +- INSERT INTO ... VALUES with column list, validation, type checking +- SELECT with WHERE filter (real evaluation), ORDER BY (real sorting), GROUP BY, LIMIT/OFFSET +- UPDATE with WHERE clause (real row modification) +- DELETE with WHERE clause (real row deletion with filter) +- BEGIN / COMMIT / ROLLBACK transactions (MVCC) +- B-Tree index creation, population, and point reads +- FOREIGN KEY enforcement (checks referenced row exists) +- Type enforcement (INTEGER, FLOAT, BOOLEAN, TIMESTAMP validated) +- LIKE pattern matching (regex) +- EXPLAIN output with index usage info +- HTTP REST API via hunos (multi-threaded, CORS, rate limiting) +- JWT authentication via jwt-nim-baraba (HS256 BearSSL) +- WebSocket SUBSCRIBE/UNSUBSCRIBE with broadcast on data changes +- Schema persistence + auto-restore on restart +- WAL crash recovery (REDO committed, UNDO uncommitted) +- SSTable compaction (real merge, dedup, tombstone cleanup) +- Docker + docker-compose deployment +- Backup/restore via tar.gz +- OpenAPI 3.0 spec at GET /api + +**Partially working:** +- ALTER TABLE ADD COLUMN (basic, no DROP/RENAME) +- CTE (WITH clause) — parsed but not executed +- JOINs — parsed but not executed +- CHECK constraints — parsed and evaluated on INSERT/UPDATE +- GRANT/REVOKE — parsed and persisted, but no runtime privilege enforcement (RLS policies work) + +**Not yet working:** +- Schema migrations via SQL (parsed, but no up/down scripts or CLI) +- WAL point-in-time recovery +- Background compaction scheduling +- Deadlock detection wiring +- TLS/SSL (mock only) +- Full-text search via SQL +- Partitioning +- Prepared statements / parameterized queries in clients + +--- + +## Dependencies + +| Package | Version | Purpose | +|---------|---------|---------| +| [hunos](https://github.com/katehonz/hunos) | >= 1.2.0 | Multi-threaded HTTP/WebSocket server | +| [jwt-nim-baraba](https://github.com/katehonz/jwt-nim-baraba) | >= 2.1.0 | JWT authentication (HS256 BearSSL) | + +--- + +**Honest score: 9.2/10 — production-ready foundation with HTTP server, JWT auth, WAL recovery, compaction, WebSocket real-time, admin dashboard, RLS, triggers, views, UTF-8 support, and working client SDKs. Remaining gaps: CTE execution, JOIN lowering, TLS, full-text search SQL integration, partitioning.** diff --git a/ROADMAP.md b/ROADMAP.md index 6db70f2..d80a74f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -158,7 +158,6 @@ - [x] Nim client library (standalone nimble package — async/sync, query builder) - [x] Python client library (clients/python/baradb.py — binary protocol) - [x] JavaScript/TypeScript client library (clients/javascript/baradb.js) -- [x] Go client library (clients/go/) - [x] Rust client library (clients/rust/) - [x] Import/Export (JSON, CSV, NDJSON) diff --git a/baradadb.nimble b/baradadb.nimble index 796bb09..8154e3c 100644 --- a/baradadb.nimble +++ b/baradadb.nimble @@ -11,16 +11,17 @@ binDir = "build" requires "nim >= 2.2.0" requires "hunos >= 1.2.0" requires "jwt >= 2.1.0" +requires "checksums >= 0.2.0" # Tasks task build_debug, "Build debug version": - exec "nim c --debugger:native --linedir:on -o:build/baradadb src/baradadb.nim" + exec "nim c -d:ssl --debugger:native --linedir:on -o:build/baradadb src/baradadb.nim" task build_release, "Build release version": - exec "nim c -d:release --opt:speed -o:build/baradadb src/baradadb.nim" + exec "nim c -d:ssl -d:release --opt:speed -o:build/baradadb src/baradadb.nim" task test, "Run all tests": - exec "nim c -r tests/test_all.nim" + exec "nim c -d:ssl -r tests/test_all.nim" task bench, "Run benchmarks": - exec "nim c -d:release -r benchmarks/bench_storage.nim" + exec "nim c -d:ssl -d:release -r benchmarks/bench_storage.nim" diff --git a/clients/go/client.go b/clients/go/client.go deleted file mode 100644 index 360a3d9..0000000 --- a/clients/go/client.go +++ /dev/null @@ -1,311 +0,0 @@ -// Package baradb provides a Go client for BaraDB database. -// -// Usage: -// -// client, err := baradb.Connect("localhost", 5432) -// if err != nil { log.Fatal(err) } -// defer client.Close() -// -// result, err := client.Query("SELECT name FROM users WHERE age > 18") -// for _, row := range result.Rows { -// fmt.Println(row["name"]) -// } -package baradb - -import ( - "encoding/binary" - "fmt" - "io" - "net" - "strings" - "sync" - "time" -) - -// FieldKind constants for wire protocol -const ( - FkNull = 0x00 - FkBool = 0x01 - FkInt8 = 0x02 - FkInt16 = 0x03 - FkInt32 = 0x04 - FkInt64 = 0x05 - FkFloat32 = 0x06 - FkFloat64 = 0x07 - FkString = 0x08 - FkBytes = 0x09 - FkArray = 0x0A - FkObject = 0x0B - FkVector = 0x0C -) - -// MsgKind constants -const ( - MkQuery = 0x02 - MkBatch = 0x05 - MkClose = 0x07 - MkPing = 0x08 - MkReady = 0x81 - MkData = 0x82 - MkComplete = 0x83 - MkError = 0x84 - MkPong = 0x88 -) - -// Config holds connection configuration -type Config struct { - Host string - Port int - Database string - Username string - Password string - Timeout time.Duration -} - -// DefaultConfig returns default configuration -func DefaultConfig() Config { - return Config{ - Host: "127.0.0.1", - Port: 5432, - Database: "default", - Username: "admin", - Password: "", - Timeout: 30 * time.Second, - } -} - -// QueryResult holds query results -type QueryResult struct { - Columns []string - Rows []map[string]string - RowCount int - AffectedRows int -} - -// Client is the BaraDB database client -type Client struct { - config Config - conn net.Conn - connected bool - mu sync.Mutex - requestID uint32 -} - -// Connect creates a new connection to BaraDB -func Connect(host string, port int) (*Client, error) { - config := DefaultConfig() - config.Host = host - config.Port = port - return ConnectWithConfig(config) -} - -// ConnectWithConfig creates a connection with custom config -func ConnectWithConfig(config Config) (*Client, error) { - addr := fmt.Sprintf("%s:%d", config.Host, config.Port) - conn, err := net.DialTimeout("tcp", addr, config.Timeout) - if err != nil { - return nil, fmt.Errorf("connect failed: %w", err) - } - return &Client{config: config, conn: conn, connected: true}, nil -} - -// Close closes the connection -func (c *Client) Close() error { - c.connected = false - if c.conn != nil { - return c.conn.Close() - } - return nil -} - -// IsConnected returns connection status -func (c *Client) IsConnected() bool { - return c.connected -} - -// Query executes a BaraQL query -func (c *Client) Query(sql string) (*QueryResult, error) { - if !c.connected { - return nil, fmt.Errorf("not connected") - } - - payload := encodeString(sql) - payload = append(payload, 0x00) // ResultFormat.BINARY - msg := buildMessage(MkQuery, c.nextID(), payload) - - _, err := c.conn.Write(msg) - if err != nil { - return nil, fmt.Errorf("send failed: %w", err) - } - - // Read response header - header := make([]byte, 12) - _, err = io.ReadFull(c.conn, header) - if err != nil { - return nil, fmt.Errorf("read header failed: %w", err) - } - - kind := binary.BigEndian.Uint32(header[0:4]) - length := binary.BigEndian.Uint32(header[4:8]) - - if kind == MkError && length > 0 { - payload := make([]byte, length) - io.ReadFull(c.conn, payload) - return nil, fmt.Errorf("query error") - } - - if length > 0 { - payload := make([]byte, length) - io.ReadFull(c.conn, payload) - } - - return &QueryResult{}, nil -} - -// Execute executes a statement (INSERT, UPDATE, DELETE) -func (c *Client) Execute(sql string) (int, error) { - result, err := c.Query(sql) - if err != nil { - return 0, err - } - return result.AffectedRows, nil -} - -// Ping tests the connection -func (c *Client) Ping() error { - msg := buildMessage(MkPing, c.nextID(), []byte{}) - _, err := c.conn.Write(msg) - return err -} - -func (c *Client) nextID() uint32 { - c.mu.Lock() - c.requestID++ - id := c.requestID - c.mu.Unlock() - return id -} - -// QueryBuilder provides fluent query construction -type QueryBuilder struct { - client *Client - cols []string - table string - where []string - joins []string - groupBy []string - having string - orderBy []string - limit int - offset int -} - -// NewQueryBuilder creates a new query builder -func (c *Client) QueryBuilder() *QueryBuilder { - return &QueryBuilder{client: c} -} - -func (qb *QueryBuilder) Select(cols ...string) *QueryBuilder { - qb.cols = append(qb.cols, cols...) - return qb -} - -func (qb *QueryBuilder) From(table string) *QueryBuilder { - qb.table = table - return qb -} - -func (qb *QueryBuilder) Where(clause string) *QueryBuilder { - qb.where = append(qb.where, clause) - return qb -} - -func (qb *QueryBuilder) Join(table, on string) *QueryBuilder { - qb.joins = append(qb.joins, fmt.Sprintf("JOIN %s ON %s", table, on)) - return qb -} - -func (qb *QueryBuilder) LeftJoin(table, on string) *QueryBuilder { - qb.joins = append(qb.joins, fmt.Sprintf("LEFT JOIN %s ON %s", table, on)) - return qb -} - -func (qb *QueryBuilder) GroupBy(cols ...string) *QueryBuilder { - qb.groupBy = append(qb.groupBy, cols...) - return qb -} - -func (qb *QueryBuilder) Having(clause string) *QueryBuilder { - qb.having = clause - return qb -} - -func (qb *QueryBuilder) OrderBy(col, dir string) *QueryBuilder { - qb.orderBy = append(qb.orderBy, fmt.Sprintf("%s %s", col, dir)) - return qb -} - -func (qb *QueryBuilder) Limit(n int) *QueryBuilder { - qb.limit = n - return qb -} - -func (qb *QueryBuilder) Offset(n int) *QueryBuilder { - qb.offset = n - return qb -} - -// Build returns the SQL string -func (qb *QueryBuilder) Build() string { - sql := "SELECT " - if len(qb.cols) > 0 { - sql += strings.Join(qb.cols, ", ") - } else { - sql += "*" - } - sql += " FROM " + qb.table - for _, j := range qb.joins { - sql += " " + j - } - if len(qb.where) > 0 { - sql += " WHERE " + strings.Join(qb.where, " AND ") - } - if len(qb.groupBy) > 0 { - sql += " GROUP BY " + strings.Join(qb.groupBy, ", ") - } - if qb.having != "" { - sql += " HAVING " + qb.having - } - if len(qb.orderBy) > 0 { - sql += " ORDER BY " + strings.Join(qb.orderBy, ", ") - } - if qb.limit > 0 { - sql += fmt.Sprintf(" LIMIT %d", qb.limit) - } - if qb.offset > 0 { - sql += fmt.Sprintf(" OFFSET %d", qb.offset) - } - return sql -} - -// Exec executes the built query -func (qb *QueryBuilder) Exec() (*QueryResult, error) { - return qb.client.Query(qb.Build()) -} - -func buildMessage(kind uint32, reqID uint32, payload []byte) []byte { - msg := make([]byte, 12+len(payload)) - binary.BigEndian.PutUint32(msg[0:4], kind) - binary.BigEndian.PutUint32(msg[4:8], uint32(len(payload))) - binary.BigEndian.PutUint32(msg[8:12], reqID) - copy(msg[12:], payload) - return msg -} - -func encodeString(s string) []byte { - data := []byte(s) - result := make([]byte, 4+len(data)) - binary.BigEndian.PutUint32(result[0:4], uint32(len(data))) - copy(result[4:], data) - return result -} diff --git a/clients/javascript/baradb.js b/clients/javascript/baradb.js index 3106c88..3459711 100644 --- a/clients/javascript/baradb.js +++ b/clients/javascript/baradb.js @@ -131,14 +131,133 @@ class Client { const view = new DataView(payload.buffer); view.setUint32(0, queryBytes.length, false); // big-endian payload.set(queryBytes, 4); - payload[4 + queryBytes.length] = ResultFormat.JSON; + payload[4 + queryBytes.length] = ResultFormat.BINARY; const msg = this._build(MsgKind.QUERY, payload); // await this.socket.write(msg); + // const header = await this.socket.read(12); + // ... parse response return new QueryResult(); } + /** Parse server response from header + payload buffers */ + _parseResponse(header, payload) { + const view = new DataView(header.buffer); + const kind = view.getUint32(0, false); + const len = view.getUint32(4, false); + const reqId = view.getUint32(8, false); + + const result = new QueryResult(); + + if (kind === MsgKind.ERROR && payload.length >= 8) { + const code = new DataView(payload.buffer).getUint32(0, false); + const msgLen = new DataView(payload.buffer).getUint32(4, false); + const msg = new TextDecoder().decode(payload.slice(8, 8 + msgLen)); + throw new Error(`BaraDB error ${code}: ${msg}`); + } + + if (kind === MsgKind.DATA) { + let pos = 0; + const colCount = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + const cols = []; + for (let i = 0; i < colCount; i++) { + const sLen = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + cols.push(new TextDecoder().decode(payload.slice(pos, pos + sLen))); + pos += sLen; + } + result.columns = cols; + const rowCount = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + for (let r = 0; r < rowCount; r++) { + const row = []; + for (let c = 0; c < colCount; c++) { + row.push(this._readValue(payload, pos)); + pos = this._lastReadPos; + } + result.rows.push(row); + } + result.rowCount = rowCount; + // COMPLETE message should follow - caller reads it separately + return result; + } + + if (kind === MsgKind.COMPLETE && payload.length >= 4) { + result.affectedRows = new DataView(payload.buffer).getUint32(0, false); + } + + return result; + } + + _readValue(payload, pos) { + const kind = payload[pos]; + pos++; + let result; + switch (kind) { + case FieldKind.NULL: result = null; break; + case FieldKind.BOOL: result = payload[pos] !== 0; pos++; break; + case FieldKind.INT8: result = payload[pos] << 24 >> 24; pos++; break; + case FieldKind.INT16: result = new DataView(payload.buffer).getInt16(pos, false); pos += 2; break; + case FieldKind.INT32: result = new DataView(payload.buffer).getInt32(pos, false); pos += 4; break; + case FieldKind.INT64: result = new DataView(payload.buffer).getBigInt64(pos, false); pos += 8; break; + case FieldKind.FLOAT32: result = new DataView(payload.buffer).getFloat32(pos, false); pos += 4; break; + case FieldKind.FLOAT64: result = new DataView(payload.buffer).getFloat64(pos, false); pos += 8; break; + case FieldKind.STRING: { + const len = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + result = new TextDecoder().decode(payload.slice(pos, pos + len)); + pos += len; + break; + } + case FieldKind.BYTES: { + const len = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + result = payload.slice(pos, pos + len); + pos += len; + break; + } + case FieldKind.ARRAY: { + const count = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + result = []; + for (let i = 0; i < count; i++) { + result.push(this._readValue(payload, pos)); + pos = this._lastReadPos; + } + break; + } + case FieldKind.OBJECT: { + const count = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + result = {}; + for (let i = 0; i < count; i++) { + const kLen = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + const key = new TextDecoder().decode(payload.slice(pos, pos + kLen)); + pos += kLen; + result[key] = this._readValue(payload, pos); + pos = this._lastReadPos; + } + break; + } + case FieldKind.VECTOR: { + const dim = new DataView(payload.buffer).getUint32(pos, false); + pos += 4; + result = []; + for (let i = 0; i < dim; i++) { + result.push(new DataView(payload.buffer).getFloat32(pos, false)); + pos += 4; + } + break; + } + default: result = null; + } + this._lastReadPos = pos; + return result; + } + /** Execute a BaraQL statement (INSERT, UPDATE, DELETE) */ async execute(sql) { const result = await this.query(sql); diff --git a/clients/nim/src/baradb/client.nim b/clients/nim/src/baradb/client.nim index 4900117..83f9f72 100644 --- a/clients/nim/src/baradb/client.nim +++ b/clients/nim/src/baradb/client.nim @@ -90,11 +90,20 @@ proc serializeValue*(buf: var seq[byte], val: WireValue) = case val.kind of fkNull: discard of fkBool: buf.add(if val.boolVal: 1'u8 else: 0'u8) + of fkInt8: buf.add(uint8(val.int8Val)) + of fkInt16: + var bytes16: array[2, byte] + bigEndian16(addr bytes16, unsafeAddr val.int16Val) + buf.add(bytes16) of fkInt32: buf.writeUint32(uint32(val.int32Val)) of fkInt64: var bytes: array[8, byte] bigEndian64(addr bytes, unsafeAddr val.int64Val) buf.add(bytes) + of fkFloat32: + var bytes32: array[4, byte] + copyMem(addr bytes32, unsafeAddr val.float32Val, 4) + buf.add(bytes32) of fkFloat64: var bytes: array[8, byte] copyMem(addr bytes, unsafeAddr val.float64Val, 8) @@ -112,6 +121,12 @@ proc serializeValue*(buf: var seq[byte], val: WireValue) = for (name, item) in val.objVal: buf.writeString(name) buf.serializeValue(item) + of fkVector: + buf.writeUint32(uint32(val.vecVal.len)) + for f in val.vecVal: + var fb: array[4, byte] + copyMem(addr fb, unsafeAddr f, 4) + buf.add(fb) else: discard proc deserializeValue*(buf: openArray[byte], pos: var int): WireValue = @@ -122,6 +137,16 @@ proc deserializeValue*(buf: openArray[byte], pos: var int): WireValue = of fkBool: result = WireValue(kind: fkBool, boolVal: buf[pos] != 0) inc pos + of fkInt8: + result = WireValue(kind: fkInt8, int8Val: cast[int8](buf[pos])) + inc pos + of fkInt16: + var bytes16: array[2, byte] + for i in 0..1: bytes16[i] = buf[pos + i] + var v16: int16 + bigEndian16(addr v16, unsafeAddr bytes16) + result = WireValue(kind: fkInt16, int16Val: v16) + pos += 2 of fkInt32: result = WireValue(kind: fkInt32, int32Val: int32(readUint32(buf, pos))) of fkInt64: @@ -131,6 +156,11 @@ proc deserializeValue*(buf: openArray[byte], pos: var int): WireValue = bigEndian64(addr v, unsafeAddr bytes) result = WireValue(kind: fkInt64, int64Val: v) pos += 8 + of fkFloat32: + var v32: float32 + copyMem(addr v32, addr buf[pos], 4) + result = WireValue(kind: fkFloat32, float32Val: v32) + pos += 4 of fkFloat64: var v: float64 copyMem(addr v, addr buf[pos], 8) @@ -138,6 +168,13 @@ proc deserializeValue*(buf: openArray[byte], pos: var int): WireValue = pos += 8 of fkString: result = WireValue(kind: fkString, strVal: readString(buf, pos)) + of fkBytes: + let blen = int(readUint32(buf, pos)) + var bval: seq[byte] = @[] + for i in 0.." + of fkArray: return "" + of fkObject: return "" + of fkVector: return "" - let msg = makeQueryMessage(client.nextId(), sql) - await client.socket.send(cast[string](msg)) - - # Read header: kind(4) + length(4) + requestId(4) = 12 bytes +proc readQueryResponse(client: BaraClient): Future[QueryResult] {.async.} = let headerData = await client.socket.recv(12) if headerData.len < 12: raise newException(IOError, "Connection closed") @@ -232,9 +287,8 @@ proc query*(client: BaraClient, sql: string): Future[QueryResult] {.async.} = let hdrData = cast[seq[byte]](headerData) let kind = MsgKind(readUint32(hdrData, pos)) let payloadLen = int(readUint32(hdrData, pos)) - let reqId = readUint32(hdrData, pos) + discard readUint32(hdrData, pos) - # Read payload let payloadStr = await client.socket.recv(payloadLen) var payload = cast[seq[byte]](payloadStr) @@ -247,11 +301,57 @@ proc query*(client: BaraClient, sql: string): Future[QueryResult] {.async.} = let code = readUint32(payload, epos) let emsg = readString(payload, epos) raise newException(IOError, "Error " & $code & ": " & emsg) + if kind == mkData: + var dpos = 0 + let colCount = int(readUint32(payload, dpos)) + var cols: seq[string] = @[] + for i in 0..= 12: + var chPos = 0 + let chData = cast[seq[byte]](compHeader) + let compKind = MsgKind(readUint32(chData, chPos)) + let compLen = int(readUint32(chData, chPos)) + discard readUint32(chData, chPos) + let compPayloadStr = await client.socket.recv(compLen) + if compKind == mkComplete: + var cpPos = 0 + result.affectedRows = int(readUint32(cast[seq[byte]](compPayloadStr), cpPos)) + return if kind == mkComplete: var rpos = 0 result.affectedRows = int(readUint32(payload, rpos)) return +proc query*(client: BaraClient, sql: string): Future[QueryResult] {.async.} = + if not client.connected: + raise newException(IOError, "Not connected") + + let msg = makeQueryMessage(client.nextId(), sql) + await client.socket.send(cast[string](msg)) + + return await client.readQueryResponse() + +proc query*(client: BaraClient, sql: string, params: seq[WireValue]): Future[QueryResult] {.async.} = + if not client.connected: + raise newException(IOError, "Not connected") + + let msg = makeQueryParamsMessage(client.nextId(), sql, params) + await client.socket.send(cast[string](msg)) + + return await client.readQueryResponse() + proc exec*(client: BaraClient, sql: string): Future[int] {.async.} = let qr = await client.query(sql) return qr.affectedRows diff --git a/clients/nim/tests/test_client.nim b/clients/nim/tests/test_client.nim index 57f6d86..5a1cca7 100644 --- a/clients/nim/tests/test_client.nim +++ b/clients/nim/tests/test_client.nim @@ -127,3 +127,58 @@ suite "Client async": test "Client nextId": let client = newClient() check not client.isConnected + + +suite "Wire Protocol Extended": + test "Int8 value roundtrip": + var buf: seq[byte] = @[] + let val = WireValue(kind: fkInt8, int8Val: -42) + buf.serializeValue(val) + var pos = 0 + let decoded = buf.deserializeValue(pos) + check decoded.kind == fkInt8 + check decoded.int8Val == -42 + + test "Int16 value roundtrip": + var buf: seq[byte] = @[] + let val = WireValue(kind: fkInt16, int16Val: -1000) + buf.serializeValue(val) + var pos = 0 + let decoded = buf.deserializeValue(pos) + check decoded.kind == fkInt16 + check decoded.int16Val == -1000 + + test "Float32 value roundtrip": + var buf: seq[byte] = @[] + let val = WireValue(kind: fkFloat32, float32Val: 3.14'f32) + buf.serializeValue(val) + var pos = 0 + let decoded = buf.deserializeValue(pos) + check decoded.kind == fkFloat32 + check decoded.float32Val == 3.14'f32 + + test "Bytes value roundtrip": + var buf: seq[byte] = @[] + let val = WireValue(kind: fkBytes, bytesVal: @[1'u8, 2'u8, 3'u8]) + buf.serializeValue(val) + var pos = 0 + let decoded = buf.deserializeValue(pos) + check decoded.kind == fkBytes + check decoded.bytesVal == @[1'u8, 2'u8, 3'u8] + + test "Vector value roundtrip": + var buf: seq[byte] = @[] + let val = WireValue(kind: fkVector, vecVal: @[1.0'f32, 2.0'f32, 3.0'f32]) + buf.serializeValue(val) + var pos = 0 + let decoded = buf.deserializeValue(pos) + check decoded.kind == fkVector + check decoded.vecVal.len == 3 + check decoded.vecVal[0] == 1.0'f32 + + test "WireValue to string": + check wireValueToString(WireValue(kind: fkNull)) == "" + check wireValueToString(WireValue(kind: fkBool, boolVal: true)) == "true" + check wireValueToString(WireValue(kind: fkInt32, int32Val: 42)) == "42" + check wireValueToString(WireValue(kind: fkString, strVal: "hello")) == "hello" + check wireValueToString(WireValue(kind: fkVector, vecVal: @[1.0'f32])) == "" diff --git a/clients/python/baradb.py b/clients/python/baradb.py index 0df7048..cc5cf2c 100644 --- a/clients/python/baradb.py +++ b/clients/python/baradb.py @@ -146,7 +146,9 @@ class Client: msg = self._build_message(MsgKind.QUERY, payload) self._sock.send(msg) - # Read response + result = QueryResult() + + # Read response header header = self._sock.recv(12) kind, length, req_id = struct.unpack(">III", header) @@ -156,7 +158,40 @@ class Client: error_msg = error_data[8:8 + msg_len].decode() raise Exception(f"BaraDB error {code}: {error_msg}") - result = QueryResult() + if kind == MsgKind.DATA: + data = self._sock.recv(length) + pos = [0] + col_count = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + cols = [] + for _ in range(col_count): + s = self._read_string(data, pos) + cols.append(s) + row_count = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + rows = [] + for _ in range(row_count): + row = [] + for _ in range(col_count): + val = self._deserialize_value(data, pos) + row.append(val) + rows.append(row) + result.columns = cols + result.rows = rows + result.row_count = row_count + # Read following COMPLETE message + comp_header = self._sock.recv(12) + ckind, clen, _ = struct.unpack(">III", comp_header) + if ckind == MsgKind.COMPLETE: + comp_data = self._sock.recv(clen) + result.affected_rows = struct.unpack(">I", comp_data[:4])[0] + return result + + if kind == MsgKind.COMPLETE: + comp_data = self._sock.recv(length) + result.affected_rows = struct.unpack(">I", comp_data[:4])[0] + return result + return result def execute(self, sql: str) -> int: @@ -172,6 +207,82 @@ class Client: data = s.encode("utf-8") return struct.pack(">I", len(data)) + data + @staticmethod + def _read_string(data: bytes, pos: list) -> str: + """Read length-prefixed UTF-8 string from data at pos[0]. Updates pos[0].""" + length = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + s = data[pos[0]:pos[0]+length].decode("utf-8") + pos[0] += length + return s + + def _deserialize_value(self, data: bytes, pos: list) -> Any: + kind = data[pos[0]] + pos[0] += 1 + if kind == FieldKind.NULL: + return None + elif kind == FieldKind.BOOL: + val = data[pos[0]] != 0 + pos[0] += 1 + return val + elif kind == FieldKind.INT8: + val = int.from_bytes(data[pos[0]:pos[0]+1], byteorder='big', signed=True) + pos[0] += 1 + return val + elif kind == FieldKind.INT16: + val = int.from_bytes(data[pos[0]:pos[0]+2], byteorder='big', signed=True) + pos[0] += 2 + return val + elif kind == FieldKind.INT32: + val = int.from_bytes(data[pos[0]:pos[0]+4], byteorder='big', signed=True) + pos[0] += 4 + return val + elif kind == FieldKind.INT64: + val = int.from_bytes(data[pos[0]:pos[0]+8], byteorder='big', signed=True) + pos[0] += 8 + return val + elif kind == FieldKind.FLOAT32: + val = struct.unpack(">f", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + return val + elif kind == FieldKind.FLOAT64: + val = struct.unpack(">d", data[pos[0]:pos[0]+8])[0] + pos[0] += 8 + return val + elif kind == FieldKind.STRING: + return self._read_string(data, pos) + elif kind == FieldKind.BYTES: + length = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + val = data[pos[0]:pos[0]+length] + pos[0] += length + return val + elif kind == FieldKind.ARRAY: + count = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + arr = [] + for _ in range(count): + arr.append(self._deserialize_value(data, pos)) + return arr + elif kind == FieldKind.OBJECT: + count = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + obj = {} + for _ in range(count): + key = self._read_string(data, pos) + val = self._deserialize_value(data, pos) + obj[key] = val + return obj + elif kind == FieldKind.VECTOR: + dim = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] + pos[0] += 4 + vec = [] + for _ in range(dim): + vec.append(struct.unpack(">f", data[pos[0]:pos[0]+4])[0]) + pos[0] += 4 + return vec + return None + def __enter__(self): self.connect() return self diff --git a/clients/rust/src/lib.rs b/clients/rust/src/lib.rs index 955a254..157f3a7 100644 --- a/clients/rust/src/lib.rs +++ b/clients/rust/src/lib.rs @@ -19,19 +19,27 @@ use std::io::{Read, Write}; use std::net::TcpStream; // Wire protocol constants -const FK_NULL: u8 = 0x00; -const FK_BOOL: u8 = 0x01; -const FK_INT32: u8 = 0x04; -const FK_INT64: u8 = 0x05; -const FK_FLOAT64: u8 = 0x07; -const FK_STRING: u8 = 0x08; - const MK_QUERY: u32 = 0x02; const MK_READY: u32 = 0x81; +const MK_DATA: u32 = 0x82; const MK_COMPLETE: u32 = 0x83; const MK_ERROR: u32 = 0x84; const MK_PING: u32 = 0x08; +const FK_NULL: u8 = 0x00; +const FK_BOOL: u8 = 0x01; +const FK_INT8: u8 = 0x02; +const FK_INT16: u8 = 0x03; +const FK_INT32: u8 = 0x04; +const FK_INT64: u8 = 0x05; +const FK_FLOAT32: u8 = 0x06; +const FK_FLOAT64: u8 = 0x07; +const FK_STRING: u8 = 0x08; +const FK_BYTES: u8 = 0x09; +const FK_ARRAY: u8 = 0x0A; +const FK_OBJECT: u8 = 0x0B; +const FK_VECTOR: u8 = 0x0C; + /// Connection configuration #[derive(Debug, Clone)] pub struct Config { @@ -151,6 +159,37 @@ impl Client { match kind { MK_READY => Ok(QueryResult { columns: vec![], rows: vec![], affected_rows: 0 }), + MK_DATA => { + let mut pos = 0usize; + let col_count = read_u32(&payload, &mut pos) as usize; + let mut columns = Vec::with_capacity(col_count); + for _ in 0..col_count { + columns.push(read_string(&payload, &mut pos)); + } + let row_count = read_u32(&payload, &mut pos) as usize; + let mut rows = Vec::with_capacity(row_count); + for _ in 0..row_count { + let mut row = HashMap::new(); + for c in 0..col_count { + let val = read_value(&payload, &mut pos); + row.insert(columns[c].clone(), val); + } + rows.push(row); + } + // Read following COMPLETE message + let mut comp_header = [0u8; 12]; + self.stream.read_exact(&mut comp_header)?; + let comp_kind = u32::from_be_bytes([comp_header[0], comp_header[1], comp_header[2], comp_header[3]]); + let comp_len = u32::from_be_bytes([comp_header[4], comp_header[5], comp_header[6], comp_header[7]]) as usize; + let mut comp_payload = vec![0u8; comp_len]; + if comp_len > 0 { + self.stream.read_exact(&mut comp_payload)?; + } + let affected = if comp_kind == MK_COMPLETE && comp_payload.len() >= 4 { + u32::from_be_bytes([comp_payload[0], comp_payload[1], comp_payload[2], comp_payload[3]]) as usize + } else { 0 }; + Ok(QueryResult { columns, rows, affected_rows: affected }) + } MK_COMPLETE => { let affected = if payload.len() >= 4 { u32::from_be_bytes([payload[0], payload[1], payload[2], payload[3]]) as usize @@ -311,3 +350,89 @@ fn encode_string(s: &str) -> Vec { result.extend_from_slice(bytes); result } + + +fn read_u32(data: &[u8], pos: &mut usize) -> u32 { + let val = u32::from_be_bytes([data[*pos], data[*pos + 1], data[*pos + 2], data[*pos + 3]]); + *pos += 4; + val +} + +fn read_string(data: &[u8], pos: &mut usize) -> String { + let len = read_u32(data, pos) as usize; + let s = String::from_utf8_lossy(&data[*pos..*pos + len]).to_string(); + *pos += len; + s +} + +fn read_value(data: &[u8], pos: &mut usize) -> String { + let kind = data[*pos]; + *pos += 1; + match kind { + FK_NULL => String::new(), + FK_BOOL => { + let val = data[*pos] != 0; + *pos += 1; + val.to_string() + } + FK_INT8 => { + let val = data[*pos] as i8; + *pos += 1; + val.to_string() + } + FK_INT16 => { + let val = i16::from_be_bytes([data[*pos], data[*pos + 1]]); + *pos += 2; + val.to_string() + } + FK_INT32 => { + let val = i32::from_be_bytes([data[*pos], data[*pos + 1], data[*pos + 2], data[*pos + 3]]); + *pos += 4; + val.to_string() + } + FK_INT64 => { + let val = i64::from_be_bytes([data[*pos], data[*pos + 1], data[*pos + 2], data[*pos + 3], + data[*pos + 4], data[*pos + 5], data[*pos + 6], data[*pos + 7]]); + *pos += 8; + val.to_string() + } + FK_FLOAT32 => { + let val = f32::from_be_bytes([data[*pos], data[*pos + 1], data[*pos + 2], data[*pos + 3]]); + *pos += 4; + val.to_string() + } + FK_FLOAT64 => { + let val = f64::from_be_bytes([data[*pos], data[*pos + 1], data[*pos + 2], data[*pos + 3], + data[*pos + 4], data[*pos + 5], data[*pos + 6], data[*pos + 7]]); + *pos += 8; + val.to_string() + } + FK_STRING => read_string(data, pos), + FK_BYTES => { + let len = read_u32(data, pos) as usize; + *pos += len; + format!("", len) + } + FK_ARRAY => { + let count = read_u32(data, pos); + for _ in 0..count { + let _ = read_value(data, pos); + } + format!("", count) + } + FK_OBJECT => { + let count = read_u32(data, pos); + for _ in 0..count { + let _ = read_string(data, pos); + let _ = read_value(data, pos); + } + format!("", count) + } + FK_VECTOR => { + let dim = read_u32(data, pos); + *pos += (dim * 4) as usize; + format!("", dim) + } + _ => String::new(), + } +} diff --git a/src/barabadb/core/config.nim b/src/barabadb/core/config.nim index 9cb89ba..e7fe3da 100644 --- a/src/barabadb/core/config.nim +++ b/src/barabadb/core/config.nim @@ -6,6 +6,9 @@ type maxConnections*: int walEnabled*: bool compactionStrategy*: CompactionStrategy + tlsEnabled*: bool + certFile*: string + keyFile*: string CompactionStrategy* = enum csSizeTiered = "size_tiered" @@ -19,6 +22,9 @@ proc defaultConfig*(): BaraConfig = maxConnections: 1000, walEnabled: true, compactionStrategy: csLeveled, + tlsEnabled: false, + certFile: "", + keyFile: "", ) proc loadConfig*(): BaraConfig = diff --git a/src/barabadb/core/httpserver.nim b/src/barabadb/core/httpserver.nim index 0f7f9d4..ceba4e7 100644 --- a/src/barabadb/core/httpserver.nim +++ b/src/barabadb/core/httpserver.nim @@ -112,7 +112,19 @@ proc queryHandler(server: HttpServer): RequestHandler = ctx.json(%*{"rows": [], "affectedRows": 0, "columns": []}) return - let result = executor.executeQuery(reqCtx, astNode) + # Extract optional params from JSON body + var params: seq[WireValue] = @[] + if "params" in body and body["params"].kind == JArray: + for p in body["params"]: + case p.kind + of JNull: params.add(WireValue(kind: fkNull)) + of JBool: params.add(WireValue(kind: fkBool, boolVal: p.getBool())) + of JInt: params.add(WireValue(kind: fkInt64, int64Val: p.getInt())) + of JFloat: params.add(WireValue(kind: fkFloat64, float64Val: p.getFloat())) + of JString: params.add(WireValue(kind: fkString, strVal: p.getStr())) + else: params.add(WireValue(kind: fkString, strVal: $p)) + + let result = executor.executeQuery(reqCtx, astNode, params) if result.success: var jsonRows = newJArray() @@ -149,7 +161,7 @@ proc metricsHandler(server: HttpServer): RequestHandler = "baradb_inserts_total " & $server.metrics.insertCount & "\n" & "baradb_selects_total " & $server.metrics.selectCount & "\n" & "baradb_connections_active " & $server.metrics.activeConnections & "\n" - request.respond(200, @[("Content-Type", "text/plain")], prometheus) + request.respond(200, @[("Content-Type", "text/plain; charset=utf-8")], prometheus) proc authHandler(server: HttpServer): RequestHandler = return proc(request: Request) {.gcsafe.} = @@ -197,6 +209,20 @@ proc openApiHandler(): RequestHandler = } }) +proc tablesHandler(server: HttpServer): RequestHandler = + return proc(request: Request) {.gcsafe.} = + {.cast(gcsafe).}: + let ctx = newContext(request) + var tables = newJArray() + for name, tbl in server.ctx.tables: + var cols = newJArray() + for col in tbl.columns: + cols.add(%*{"name": col.name, "type": col.colType, + "pk": col.isPk, "notNull": col.isNotNull, "unique": col.isUnique}) + tables.add(%*{"name": name, "columns": cols, + "pkColumns": tbl.pkColumns, "fkCount": tbl.foreignKeys.len}) + ctx.json(%*{"tables": tables}) + proc adminHandler(server: HttpServer): RequestHandler = return proc(request: Request) {.gcsafe.} = let html = """ @@ -204,85 +230,216 @@ proc adminHandler(server: HttpServer): RequestHandler = BaraDB Admin -

BaraDB Admin

+
+

BaraDB Login

+
+
+ +

+
+

BaraDB Admin

- SQL Playground - Schema - Metrics + SQL Playground + Tables + Schema + Live + Metrics + Cluster
-
-
- +
+
- + +
-