diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a0494d..aa35676 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,20 @@ All notable changes to BaraDB are documented in this file. ## [1.2.0] — Unreleased +### Raft cluster (C3a / C3b / ops) + +Production-ready path from config → election → SQL/DDL replication → ops. + +- **C3a — Networked bootstrap** — `BARADB_RAFT_PEERS=id@host:port`, election timer in production, heartbeat timer reset on AppendEntries, `raft_state.bin` persistence, partial-read-safe frames; E2E `tests/raft_e2e_test.nim` (3-node election + failover) +- **C3b — SQL writes through Raft** — leader appends DML KV pairs and waits for majority commit; followers reject or **forward** via `BARADB_RAFT_CLIENT_PEERS`; apply path updates LSM + B-tree/FTS/HNSW + graphs; multi-statement write gate; writes only on `default` database +- **DDL replication** — schema DDL (`CREATE`/`DROP`/`ALTER` table, index, view, graph, …) as `ddl` log entries; re-executed on apply; `CREATE`/`DROP DATABASE` excluded +- **Leader write forwarding** — followers proxy DML/DDL to the leader SQL port when client peers are configured +- **Safe log compaction** — soft cap `BARADB_RAFT_LOG_MAX_ENTRIES` (default 256); leader never discards past peer `matchIndex`; snapshot base (`lastSnapshotIndex`/`Term`) persisted +- **Secondary-index point lookup fix** — index scans use `entry.lsmKey` (not the filter column as PK) +- **Metrics** — Prometheus raft series on `GET /metrics` (HTTP = TCP port + 440); `GET /health` includes `raft` role/term/leader/lag/log size +- **E2E writes** — `tests/raft_writes_e2e_test.nim` (schema, forward, index SELECT, failover) +- **Docs** — `docs/en|bg/distributed.md`, `docs/superpowers/specs/2026-07-30-raft-cluster-status.md` + ### Core Storage Hardening Foundational LSM improvements for write performance, durability, and compaction correctness. diff --git a/PLAN.md b/PLAN.md index ad3e227..64cb35f 100644 --- a/PLAN.md +++ b/PLAN.md @@ -155,6 +155,7 @@ | `PLAN_SQL_ADVANCED.md` — Window Functions, MERGE, etc. | ✅ Завършен | | `PLAN_ID_GENERATORS.md` — AUTO_INCREMENT, Sequences, FK | ✅ Завършен | | **Този план** — Сесии 10, 11, 12 | ✅ Завършен | +| Raft C3a/C3b + DDL/forward/compact/metrics (2026-07-30) | ✅ Завършен на `main` — `docs/superpowers/specs/2026-07-30-raft-cluster-status.md` | --- diff --git a/README.md b/README.md index 6b052a4..74e973e 100644 --- a/README.md +++ b/README.md @@ -733,6 +733,23 @@ let diff = s.diff(oldSchema, newSchema) ### Raft Consensus +3-node cluster over TCP (env-driven). SQL DML and schema DDL go through the +raft log on the **default** database. See **[docs/en/distributed.md](docs/en/distributed.md)** +for full env vars, forwarding, compaction, and metrics. + +```bash +# Node n1 example +export BARADB_PORT=46010 +export BARADB_RAFT_ENABLED=true +export BARADB_RAFT_NODE_ID=n1 +export BARADB_RAFT_PORT=46101 +export BARADB_RAFT_PEERS=n1@127.0.0.1:46101,n2@127.0.0.1:46102,n3@127.0.0.1:46103 +export BARADB_RAFT_CLIENT_PEERS=n1@127.0.0.1:46010,n2@127.0.0.1:46020,n3@127.0.0.1:46030 +export BARADB_DATA_DIR=./data/n1 +./build/baradadb +# Health / metrics: HTTP on BARADB_PORT+440 → curl localhost:46450/health +``` + ```nim import barabadb/core/raft @@ -1559,7 +1576,7 @@ features are still being refined: | LSM-Tree SSTable reads | ✅ Implemented | Full disk I/O with compaction, WAL, and bloom filters. | | HNSW vector search | ✅ Implemented | Hierarchical graph navigation with SIMD-optimized distance metrics. | | TCP server execution | ✅ Implemented | Full binary wire protocol parsing and BaraQL query execution. | -| Raft consensus | ✅ Core logic | Leader election + log replication over TCP; SQL DML commits through the raft log when `BARADB_RAFT_ENABLED=true` (followers reject writes). | +| Raft consensus | ✅ Cluster path | TCP election + failover; SQL DML/DDL via raft log; leader forwarding; safe log compact; `/metrics` + `/health` raft gauges. See `docs/en/distributed.md`. | | Graph / FTS / Columnar | ✅ Implemented | In-memory engines with serialization; FTS/vector/graph indexes persist across restarts. | | Query codegen | ✅ Implemented | IR plans compile to storage engine operations with optimization passes. | @@ -1568,7 +1585,7 @@ reflects 100% completion across all major phases. ## Changelog -See [CHANGELOG.md](CHANGELOG.md) for full release history. Package version is **v1.1.8**. The **v1.2.0** line (Unreleased) adds core storage hardening (hash MemTable, WAL group commit, schema persistence, ARC/wire stability) and the Unified Search Engine (heap-optimized HNSW, segment inverted index, boolean/phrase/n-gram/facets, multi-language stemmers). +See [CHANGELOG.md](CHANGELOG.md) for full release history. Package version is **v1.1.8**. The **v1.2.0** line (Unreleased) adds core storage hardening, Unified Search Engine, **engine persistence** (FTS/HNSW/graphs/B-tree indexes across restart), **executor split**, and a full **Raft cluster path** (election, SQL/DDL replication, forwarding, log compact, metrics — see [docs/en/distributed.md](docs/en/distributed.md)). ## License diff --git a/docs/bg/distributed.md b/docs/bg/distributed.md index 0d025b1..6164c2f 100644 --- a/docs/bg/distributed.md +++ b/docs/bg/distributed.md @@ -5,9 +5,11 @@ BaraDB поддържа разпределено внедряване с Raft к > ⚠️ **Ограничение при множество бази данни** > Разпределените модули (Raft, шардиране и репликация) в момента работят само с **`default`** базата данни. Ако използвате множество бази (`CREATE DATABASE`, `USE DATABASE`), разпределените функции още не ги обхващат. Всяка база данни се нуждае от отделна кластър конфигурация. +> **Статус (2026-07-30):** Raft C3a (мрежова election), C3b (SQL записи), DDL репликация, leader forwarding, log compaction и metrics са **на `main`**. Преглед: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. + ## Raft Консенсус -Leader election и log репликация през TCP. Включване: +Leader election и log репликация през TCP; SQL DML/DDL за **default** минават през raft log. Включване: | Env | Значение | |-----|----------| diff --git a/docs/bg/monitoring.md b/docs/bg/monitoring.md index c88140a..dce6f7b 100644 --- a/docs/bg/monitoring.md +++ b/docs/bg/monitoring.md @@ -4,29 +4,28 @@ ### HTTP Health Endpoint +HTTP слуша на **TCP порт + 440** (напр. `BARADB_PORT=9472` → health на `9912`). + ```bash -curl http://localhost:9470/health +curl http://localhost:9912/health ``` -Отговор: +Без raft: ```json { - "status": "healthy", + "status": "ok", "version": "1.1.6", - "uptime_seconds": 86400, - "checks": { - "storage": "ok", - "memory": "ok", - "connections": "ok" - } + "raft": { "enabled": false } } ``` +С `BARADB_RAFT_ENABLED=true` — обект `raft` (`role`, `term`, `leader_id`, `commit_index`, `apply_lag`, `log_entries`, `snapshot_index`). + ### Readiness Probe ```bash -curl http://localhost:9470/ready +curl http://localhost:9912/ready ``` Връща `200 OK` когато сървърът е готов да приема трафик, `503` по време на стартиране. @@ -35,10 +34,16 @@ curl http://localhost:9470/ready ### Prometheus-Съвместими Метрики +Същият HTTP порт като health (`BARADB_PORT + 440`). + ```bash -curl http://localhost:9470/metrics +curl http://localhost:9912/metrics ``` +Базови: `baradb_queries_total`, `baradb_query_errors_total`, `baradb_inserts_total`, `baradb_selects_total`, `baradb_connections_active`. + +С raft: `baradb_raft_is_leader`, `baradb_raft_term`, `baradb_raft_log_entries`, `baradb_raft_apply_lag`, `baradb_raft_commit_wait_ms_total`, `baradb_raft_elections_total`, `baradb_raft_forwards_total`, `baradb_raft_compactions_total` и др. Пълен списък: [distributed.md](distributed.md) / [en/monitoring.md](../en/monitoring.md). + Примерен изход: ``` diff --git a/docs/en/distributed.md b/docs/en/distributed.md index 5530467..a217114 100644 --- a/docs/en/distributed.md +++ b/docs/en/distributed.md @@ -5,9 +5,11 @@ BaraDB supports distributed deployment with Raft consensus, sharding, and replic > ⚠️ **Multi-Database Limitation** > The distributed modules (Raft, sharding, and replication) are currently wired to the **`default`** database only. If you use multiple databases (`CREATE DATABASE`, `USE DATABASE`), distributed features do not yet span across them. Each database would need its own cluster setup. +> **Status (2026-07-30):** Raft C3a (network election), C3b (SQL writes), DDL replication, leader forwarding, log compaction, and metrics are **shipped on `main`**. Design/history: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. + ## Raft Consensus -Leader election and log replication over TCP. Enable with: +Leader election and log replication over TCP; SQL DML/DDL on the default DB go through the raft log. Enable with: | Env | Meaning | |-----|---------| @@ -25,6 +27,26 @@ When Raft is enabled, SQL DML (`INSERT`/`UPDATE`/`DELETE`/`MERGE` and transactio **Metrics:** with raft enabled, `GET /metrics` (HTTP port = `BARADB_PORT + 440`) includes Prometheus lines such as `baradb_raft_is_leader`, `baradb_raft_term`, `baradb_raft_log_entries`, `baradb_raft_apply_lag`, `baradb_raft_commit_wait_ms_total`, `baradb_raft_elections_total`, `baradb_raft_forwards_total`, and `baradb_raft_compactions_total`. `GET /health` embeds a `raft` object (`role`, `term`, `leader_id`, `commit_index`, `apply_lag`, …). +### Minimal 3-node example + +```bash +# Shared peers (raft ports) and client peers (SQL ports) +export BARADB_RAFT_ENABLED=true +export BARADB_RAFT_PEERS=n1@127.0.0.1:46101,n2@127.0.0.1:46102,n3@127.0.0.1:46103 +export BARADB_RAFT_CLIENT_PEERS=n1@127.0.0.1:46010,n2@127.0.0.1:46020,n3@127.0.0.1:46030 + +# Terminal 1 +BARADB_PORT=46010 BARADB_RAFT_PORT=46101 BARADB_RAFT_NODE_ID=n1 \ + BARADB_DATA_DIR=./data/n1 ./build/baradadb + +# Terminal 2 / 3 — n2@46020/46102, n3@46030/46103 similarly + +# After a leader appears (check logs for "became leader"): +# curl http://127.0.0.1:46450/health # n1 HTTP = 46010+440 +``` + +### In-process API (tests / embedding) + ```nim import barabadb/core/raft @@ -39,6 +61,13 @@ n1.becomeLeader() let entry = n1.appendLog("SET key1 value1") ``` +### E2E tests + +| Test | What it proves | +|------|----------------| +| `tests/raft_e2e_test.nim` | 3 real processes; election + kill-leader failover | +| `tests/raft_writes_e2e_test.nim` | DDL/DML via raft, follower forward, index SELECT, failover writes | + ## Sharding Distribute data across nodes: diff --git a/docs/en/monitoring.md b/docs/en/monitoring.md index 5a49ac2..e4841a7 100644 --- a/docs/en/monitoring.md +++ b/docs/en/monitoring.md @@ -4,21 +4,39 @@ ### HTTP Health Endpoint +HTTP listens on **TCP port + 440** (e.g. `BARADB_PORT=9472` → health on `9912`). + ```bash -curl http://localhost:9470/health +curl http://localhost:9912/health ``` -Response: +Response (raft disabled): ```json { - "status": "healthy", - "version": "0.1.0", - "uptime_seconds": 86400, - "checks": { - "storage": "ok", - "memory": "ok", - "connections": "ok" + "status": "ok", + "version": "1.1.6", + "raft": { "enabled": false } +} +``` + +With `BARADB_RAFT_ENABLED=true`, a `raft` object is included: + +```json +{ + "status": "ok", + "version": "1.1.6", + "raft": { + "enabled": true, + "node_id": "n1", + "role": "leader", + "term": 2, + "leader_id": "n1", + "commit_index": 42, + "last_applied": 42, + "apply_lag": 0, + "log_entries": 12, + "snapshot_index": 30 } } ``` @@ -35,10 +53,38 @@ Returns `200 OK` when the server is ready to accept traffic, `503` during startu ### Prometheus-Compatible Metrics +Same HTTP base port as health (`BARADB_PORT + 440`). When auth is enabled, send a Bearer token. + ```bash -curl http://localhost:9470/metrics +curl http://localhost:9912/metrics ``` +Always present: + +| Metric | Meaning | +|--------|---------| +| `baradb_queries_total` | HTTP queries handled | +| `baradb_query_errors_total` | Failed HTTP queries | +| `baradb_inserts_total` / `baradb_selects_total` | Statement class counts | +| `baradb_connections_active` | Active connections | + +With raft enabled, additional series (labels include `node="…"`): + +| Metric | Meaning | +|--------|---------| +| `baradb_raft_is_leader` | 1 if this process is leader | +| `baradb_raft_term` | Current term | +| `baradb_raft_log_entries` | In-memory log length | +| `baradb_raft_commit_index` / `baradb_raft_last_applied` | Raft indices | +| `baradb_raft_apply_lag` | commit − applied | +| `baradb_raft_snapshot_index` | Compacted log base | +| `baradb_raft_elections_total` | Times this node became leader | +| `baradb_raft_commit_wait_ms_total` / `_avg` | Wait-for-commit latency | +| `baradb_raft_forwards_total` | Follower→leader SQL forwards | +| `baradb_raft_compactions_total` | Log prefix compactions | + +See also [distributed.md](distributed.md) for cluster env vars and ops notes. + Example output: ``` diff --git a/docs/superpowers/plans/2026-07-30-raft-network-bootstrap.md b/docs/superpowers/plans/2026-07-30-raft-network-bootstrap.md index de61438..e533d9b 100644 --- a/docs/superpowers/plans/2026-07-30-raft-network-bootstrap.md +++ b/docs/superpowers/plans/2026-07-30-raft-network-bootstrap.md @@ -1,6 +1,7 @@ # Networked Raft Bootstrap (C3a) Implementation Plan -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> **Status: DONE** — shipped on `main`. Historical plan; do not re-run tasks. +> Overview: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. **Goal:** A 3-node BaraDB cluster started with ordinary config elects a leader over TCP, maintains it with heartbeats, and re-elects after the leader is killed. diff --git a/docs/superpowers/plans/2026-07-30-raft-sql-writes.md b/docs/superpowers/plans/2026-07-30-raft-sql-writes.md index ffe8cff..6d8656b 100644 --- a/docs/superpowers/plans/2026-07-30-raft-sql-writes.md +++ b/docs/superpowers/plans/2026-07-30-raft-sql-writes.md @@ -1,6 +1,8 @@ # SQL Writes Through Raft (C3b) Implementation Plan -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> **Status: DONE** — shipped on `main` (plus post-C3b DDL/forward/compact/metrics). +> Historical plan; do not re-run tasks. +> Overview: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. **Goal:** When Raft is enabled, SQL writes commit through the Raft log before the client sees success; followers reject writes naming the leader and apply committed entries via the existing applyCommand loop. diff --git a/docs/superpowers/specs/2026-07-30-raft-cluster-status.md b/docs/superpowers/specs/2026-07-30-raft-cluster-status.md new file mode 100644 index 0000000..4d320d6 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-raft-cluster-status.md @@ -0,0 +1,62 @@ +# Raft Cluster Status — C3a / C3b / post-C3b + +Date: 2026-07-30 +Status: **Shipped on `main`** (tip includes metrics). +Branch: all work merged to `main` only (feature branch removed). + +## Phase map + +| Phase | Spec / plan | Status | What landed | +|-------|-------------|--------|-------------| +| **C3a** Network bootstrap | `raft-network-bootstrap-design.md` + plan | **Done** | `id@host:port` peers, election timer in production, AppendEntries resets timer, `dataDir` persistence, partial-read frames, `tests/raft_e2e_test.nim` | +| **C3b** SQL writes | `raft-sql-writes-design.md` + plan | **Done** | `isWrite`, follower reject / forward, leader append+wait-commit, rich apply (LSM+index+graph), multi-stmt gate, default-DB only, `tests/raft_writes_e2e_test.nim` | +| **C3c-lite** DDL + ops | (this status doc) | **Done** | DDL via `ddl` log entries; leader forwarding (`BARADB_RAFT_CLIENT_PEERS`); safe log compact + snapshot metadata; Prometheus + `/health` raft | + +## Key commits (main) + +| Commit (short) | Summary | +|----------------|---------| +| C3a series | peers, timer, frames, e2e election | +| `38c1c01`…`a462d21` | C3b classification → append → e2e → docs | +| `0d51497` | multi-stmt gate, rich apply, delete kv | +| `50f827f` | graph apply, non-default DB reject | +| `095698b` | DDL through raft | +| `9df8316` | leader write/DDL forwarding | +| `53704e1` | safe log compaction + snapshot base | +| `1b3c261` | raft metrics on `/metrics` + `/health` | + +## Production behavior (summary) + +1. Enable with `BARADB_RAFT_*` env (see `docs/en/distributed.md`). +2. Cluster elects a leader over TCP; state in `dataDir/raft/raft_state.bin`. +3. DML/DDL on **default** DB only: leader appends, waits for majority, returns. +4. Followers forward to leader if `BARADB_RAFT_CLIENT_PEERS` is set; else `not leader`. +5. Apply updates LSM + secondary engines; DDL re-executes SQL on each node. +6. Log soft-cap via safe prefix compact; metrics on HTTP port `BARADB_PORT+440`. + +## Explicit non-goals still open + +- Multi-database raft (only `default`) +- `CREATE`/`DROP DATABASE` replication +- Membership change (join/leave) protocol +- InstallSnapshot with full SM / LSM payload (v1 uses safe-prefix compact only) +- Raft port TLS / mutual auth +- Read consistency levels (read-your-writes, linearizable reads on followers) +- Automatic client redirect without `CLIENT_PEERS` + +## Tests + +| Suite | Covers | +|-------|--------| +| `tests/raft_e2e_test.nim` | 3-process election + failover | +| `tests/raft_writes_e2e_test.nim` | DDL + DML replicate, forward, index SELECT, failover writes | +| `tests/test_all.nim` | in-process raft, append/wait, compact, metrics | +| `tests/tla_faithfulness.nim` | ElectionSafety, LogMatching, … | +| `tests/bugfix_test.nim` | peers / client peers / isWrite / isRaftDdl | + +## Docs to keep in sync + +- `docs/en/distributed.md` / `docs/bg/distributed.md` — operator guide +- `docs/en/monitoring.md` — health/metrics (raft section) +- `CHANGELOG.md` — `[1.2.0] Unreleased` Raft section +- README raft status line diff --git a/docs/superpowers/specs/2026-07-30-raft-network-bootstrap-design.md b/docs/superpowers/specs/2026-07-30-raft-network-bootstrap-design.md index f742c3f..397038c 100644 --- a/docs/superpowers/specs/2026-07-30-raft-network-bootstrap-design.md +++ b/docs/superpowers/specs/2026-07-30-raft-network-bootstrap-design.md @@ -1,116 +1,34 @@ # Networked Raft Bootstrap (C3a) — Design -Date: 2026-07-30 -Status: Approved direction (user: "продължи"); implementation follows. +Date: 2026-07-30 +Status: **Done** (merged to `main`). See also `2026-07-30-raft-cluster-status.md`. ## Problem -Raft in BaraDB is half-wired for real networking. `core/raft.nim` already has -a TCP transport (`RaftNetwork`, binary framing, serialization, election-over- -TCP proven by a test), but in production: - -1. `node.peerAddrs` is **never populated** (`baradadb.nim` creates the node - from `config.raftPeers` but never parses addresses) — all sends silently - no-op (`raft.nim:560-561`). -2. **No election timer runs** — `tick` is only called from tests; a deployed - node never starts an election. -3. `handleAppendEntries` on the wire path does **not reset the election - timer** — even if timers ran, followers would start elections despite a - healthy leader. -4. Raft **state persistence is disabled** in server startup (`dataDir` not - passed to `newRaftNode`, `baradadb.nim:333`). -5. Framing reads assume full TCP reads (`recv(4)`/`recv(payloadLen)`, - `raft.nim:604-611`) — partial reads corrupt the stream. - -Result: `BARADB_RAFT_ENABLED=true` today starts a listening socket that can -never elect anyone. This phase wires what exists; it does NOT change the SQL -write path (that is C3b) or add membership/snapshots (C3c). +Raft in BaraDB was half-wired for real networking. `core/raft.nim` already had +a TCP transport, but production never populated `peerAddrs`, never ran an +election timer, did not reset timers on AppendEntries, skipped state +persistence, and used unsafe partial frame reads. ## Goal -A 3-node BaraDB cluster started with ordinary config elects a leader over -TCP, maintains it with heartbeats, and re-elects after the leader is killed — -verified end-to-end with real server processes. +A 3-node BaraDB cluster elects a leader over TCP, maintains it with heartbeats, +and re-elects after the leader is killed — verified with real processes +(`tests/raft_e2e_test.nim`). -Non-goals: SQL writes through Raft (C3b), membership changes, snapshots, -reconnect/backoff hardening, TLS/auth on the raft port (C3c). +## Delivered -## Design +| Item | Implementation | +|------|----------------| +| Peer addresses | `BARADB_RAFT_PEERS=id@host:port` → `raftPeerAddrs` | +| Election timer | `timerLoop` inside `RaftNetwork.run` | +| Heartbeat reset | `processMessage` resets timer on AppendEntries with valid term | +| State persistence | `dataDir` → `raft_state.bin` | +| Partial reads | `recvExact` frame reassembly | +| E2E | `tests/raft_e2e_test.nim` (election + failover) | -### 1. Peer address configuration +## Non-goals (handled later) -- Env (existing mechanism): `BARADB_RAFT_PEERS` entries extended from bare - `nodeId` to `nodeId@host:port`. Comma-separated, e.g. - `BARADB_RAFT_PEERS="n1@127.0.0.1:9473,n2@127.0.0.1:9474,n3@127.0.0.1:9475"`. - Bare entries (no `@`) keep current meaning (peer id, no address). -- Parsing lives in `core/config.nim` next to the existing `BARADB_RAFT_*` - env handling (config.nim:174-179), producing `raftPeerAddrs: Table[string, - (string, int)]` on BaraConfig. Malformed entries fail startup with a clear - error (config-time, not runtime). -- `baradadb.nim` startup copies `config.raftPeerAddrs` into - `node.peerAddrs` and passes the data dir (see §3). - -### 2. Election timer in production - -- A `timerLoop` async proc in `core/raft.nim` (next to `heartbeatLoop`, - raft.nim:625): every 50ms calls `tick(node)`; on election timeout calls - the existing `startElection` path (raft.nim:659-685), which sends - RequestVote over `RaftNetwork.send`. -- `RaftNetwork.run` (raft.nim:633) starts `timerLoop` alongside - `heartbeatLoop` via `asyncCheck` — single place, no baradadb.nim changes - beyond startup wiring. -- Wire-path timer reset: in the message-receive handling of `RaftNetwork`, - after a valid AppendEntries (or heartbeat) is processed, reset the - follower's election timer (`ElectionTimer.lastHeartbeat = now`) — wherever - the existing handler processes inbound AppendEntries, matching what the - in-process tests do manually. - -### 3. State persistence on - -- `baradadb.nim:333`: pass `config.dataDir` (or a raft subdirectory of it — - check what `newRaftNode` expects; `saveState`/`loadState` write - `raft_state.bin`) so currentTerm/votedFor/log survive restarts, per the - Raft spec (and the TLA models). - -### 4. Framing robustness - -- Replace the `recv(4)` / `recv(payloadLen)` assumptions with the existing - `recvExact`-style helper pattern used by the main server - (`core/server.nim:312-329` has `recvExact`/`recvExactWithTimeout` — mirror - the approach inside raft.nim; do NOT import server.nim into raft.nim). - -### 5. What does NOT change - -- SQL write path (ReplicationManager, server.nim) — untouched. -- Raft state machine, message format, serialization — untouched (the TLA- - faithfulness tests pin them). -- `applyCommand` hook in baradadb.nim — stays as-is; committed entries - (from tests / future C3b) still apply to the default DB. - -## Testing - -TDD where feasible; the headline test is end-to-end: - -1. **Config parsing** (unit, `tests/test_all.nim` or bugfix_test): peers - with and without `@host:port`, malformed entries error clearly. -2. **Framing** (unit): partial-read feed of a serialized message through the - new read path (socketpair or chunked strings) — message reassembles. -3. **E2E cluster** (new file `tests/raft_e2e_test.nim`): start 3 real - `build/baradadb` processes with raft enabled on distinct ports - (client/raft ports offset per node), wait for a leader (queryable how? - — simplest: each node logs its role; or a `RAFT STATUS` text command on - the client port if one exists cheaply — decide in the plan), kill the - leader, assert a new leader is elected within ~5s. Clean teardown. -4. Existing suites stay green (`nimble test`, 659+ `[OK]`) — especially the - in-process raft suites and the 3-node election TCP test. - -## Risks - -- Election-timer/async interactions with the main server's async loop — - `RaftNetwork.run` already runs under `asyncCheck`; timerLoop follows the - same pattern. Watch for CPU spin (50ms sleep, not busy loop). -- Port allocation in the E2E test (parallel CI) — use the same - time-derived port offset pattern as `tests/nimforum_smoke_test.nim`. -- Timer reset wiring point: the inbound message path must reach the node - the timer ticks — same `RaftNode` instance, verified by the E2E test - (no spurious elections under a healthy leader). +SQL writes (C3b), DDL/forward/compact/metrics (post-C3b / C3c-lite) — all +shipped; see cluster status doc. Still open: membership, InstallSnapshot SM +payload, multi-DB raft. diff --git a/docs/superpowers/specs/2026-07-30-raft-sql-writes-design.md b/docs/superpowers/specs/2026-07-30-raft-sql-writes-design.md index b2dedb1..a904040 100644 --- a/docs/superpowers/specs/2026-07-30-raft-sql-writes-design.md +++ b/docs/superpowers/specs/2026-07-30-raft-sql-writes-design.md @@ -1,109 +1,55 @@ # SQL Writes Through Raft (C3b) — Design -Date: 2026-07-30 -Status: Done (implemented on `feat/raft-sql-writes`). +Date: 2026-07-30 +Status: **Done** (merged to `main`). Extended by post-C3b work (DDL, forward, +compact, metrics) — see `2026-07-30-raft-cluster-status.md`. ## Problem With C3a a real Raft cluster elects a leader over TCP, but SQL writes still -bypass Raft entirely: they go straight to the local LSM and (optionally) to -the legacy `ReplicationManager` (unconnected in production). Committed Raft -entries can be applied (`applyCommand` is wired and invoked on commit), yet -nothing ever calls `appendLog` from the write path. +bypassed Raft: they went straight to the local LSM. Committed Raft entries +could be applied (`applyCommand`), yet nothing called `appendLog` from the +write path. -## Goal +## Goal (delivered) -When Raft is enabled, SQL writes are committed through the Raft log before -the client sees success: +When Raft is enabled, SQL writes commit through the Raft log before the client +sees success: -- Leader: execute locally (constraints validated as today), append the - resulting KV pairs to the Raft log, wait for majority commit, then return. -- Followers: reject write statements with a clear "not leader" error naming - the known leader; apply committed entries via the existing `applyCommand` - loop (raft.nim:195-215). -- Non-raft deployments: byte-identical behavior to today. +- **Leader:** execute locally, append KV pairs (`put` / `delete`), wait for + majority `commitIndex`, return. +- **Followers:** originally reject with `not leader; leader is '…'`; with + `BARADB_RAFT_CLIENT_PEERS` they **forward** to the leader (post-C3b). +- **Apply:** LSM + secondary B-tree/FTS/HNSW + in-memory graphs + (`applyReplicatedPut` / `Delete`). +- **Non-raft:** behavior gated on `raftNode == nil`. -Non-goals (C3c+): leader forwarding/proxy, read consistency levels, -schema/DDL replication (DDL produces no keyValuePairs today — out of scope), -membership, snapshots. +## Design (as shipped) -## Design +### Write interception -### Write interception point +`core/server.nim` `executeQuery`: -`core/server.nim:206-227` — the server-level `executeQuery` parses the AST -(server.nim:213) and, after successful execution, already ships -`res.keyValuePairs` to the ReplicationManager (server.nim:219-227). The Raft -path slots into the same place with the same data source: - -1. Classify the statement right after parse: write = `stmt.kind in {nkInsert, - nkUpdate, nkDelete, nkMerge}` (DDL is out of scope; SELECT unaffected). - New helper `isWrite(stmt)` in `exec/params.nim` next to `isDDL`. -2. If Raft is active for this server (`server.raftNode != nil`): - - Not leader (`node.state != rsLeader`): return error - `not leader; leader is ''` (or `no leader elected`). - The statement is NOT executed. - - Leader: execute as today. If `res.success` and - `res.keyValuePairs.len > 0`: for each `(key, value)` append one log - entry — `cmd = "put"` with data `key \x00 value`, or `cmd = "delete"` - with data `key` when value is empty (this is EXACTLY the format the - existing `applyCommand` in baradadb.nim:336-343 already consumes — no - format changes, frozen). Then wait until `node.commitIndex` reaches the - last appended index (see below) before returning `res`. - - The legacy `replication.writeLsn` hook is skipped when the Raft write - path handled the statement (no double shipping). -3. Single-node raft (enabled, no peers): majority is self — appends commit - immediately; behavior is correct with negligible overhead. +1. Classify every statement: `isWrite` / `isRaftDdl` (not only `stmts[0]`). +2. If raft active and write/DDL: require **default** database + leader (or + forward). +3. After success: pure DML → `appendWriteToRaft`; any DDL in batch → + `appendDdlToRaft` (full SQL re-exec on apply). ### Wait-for-commit -No raft.nim changes (the state machine stays TLA-frozen). Poll from -server.nim: after appending, `while node.commitIndex < lastIdx`: sleep 10ms, -up to `config.raftWriteTimeoutMs` (new env `BARADB_RAFT_WRITE_TIMEOUT_MS`, -default 5000). Commit advances on the 50ms heartbeat cadence via the -existing `handleAppendReply`/`applyCommitted` path, so typical latency is -one heartbeat. On timeout: return an error (`raft commit timeout`) — the -entry may still commit later; documented limitation. Local execution already -happened (it validates constraints and produces kvPairs) — leader double- -applies via applyCommand on commit, which is idempotent for put/delete KV. +Poll `commitIndex` outside the storage gate (`appendWriteToRaft` / +`waitRaftCommit`); timeout → `raft commit timeout`. -### Server wiring +### Known v1 limitations (still true) -`Server` gains `raftNode*: RaftNode` (nil by default) in core/server.nim; -baradadb.nim assigns it when raft is enabled (the node already exists there). -server.nim imports core/raft (no cycle: raft.nim does not import server). +- Multi-DB raft not supported (`raft writes only supported on the 'default' database`). +- `CREATE`/`DROP DATABASE` not raft-replicated. +- Leader local execute before majority (timeout leaves local write; documented). +- No InstallSnapshot full SM dump (safe log compact only). +- No membership changes. -### Known v1 limitations (documented in code) +## Tests -- Leader losing leadership between local execution and appendLog (narrow - race): appendLog returns the empty entry (index 0) — surfaced as a - `lost leadership` error; the local write already applied (uncommitted). -- Transactional writes: kvPairs are emitted at COMMIT (executor.nim:970-979) - — the COMMIT statement is the raft-replicated unit (classify nkCommitTxn? - — resolve in planning: COMMIT's kvPairs flow through the same hook since - the existing replication hook already catches them). -- DDL is not replicated (no kvPairs today). - -## Testing - -1. Unit (`tests/bugfix_test.nim`): `isWrite` classification for - INSERT/UPDATE/DELETE/MERGE vs SELECT/DDL. -2. Follower rejection (in-process or E2E): write on a follower returns the - "not leader" error naming the leader; the statement is not executed. -3. E2E (extend `tests/raft_e2e_test.nim` or sibling): 3 real nodes; write a - row on the leader via the Nim client (adaptors/nim/baradb_sqlite pattern - from nimforum_smoke_test); poll a SELECT on a follower until the row - appears (deadline ~5s — follower applies on commit via applyCommand); - write on a follower → rejected; kill leader → re-election → writes work - on the new leader. -4. Full `nimble test` green (673+ `[OK]`). - -## Risks - -- Double application on the leader (executor + applyCommand): idempotent - KV semantics — verify put/delete idempotency holds for the value encoding - (same key → same value; delete of existing key). -- `node.log[idx-1]` positional indexing in the leader commit loop - (raft.nim:407) — pre-existing, out of scope unless the E2E trips it. -- Apply lag on followers (heartbeat cadence): tests poll with deadlines, - never fixed sleeps. +- Unit: classification, append/timeout, apply indexes/graphs, compact, metrics +- E2E: `tests/raft_writes_e2e_test.nim` (schema, forward, index SELECT, failover)