From d303cc56588a155745a9eaa326af9113d07e769e Mon Sep 17 00:00:00 2001 From: dimgigov Date: Fri, 31 Jul 2026 04:16:11 +0300 Subject: [PATCH] docs: v1.3.0 raft-supported docs, limitations, runbook updates --- docs/bg/distributed.md | 15 +- docs/bg/known-limitations.md | 20 +- docs/en/distributed.md | 20 +- docs/en/known-limitations.md | 32 +- docs/en/release-checklist.md | 5 +- .../plans/2026-07-30-v1.3.0-raft-supported.md | 598 ++++++++++++++++++ .../specs/2026-07-30-raft-cluster-status.md | 33 +- .../specs/2026-07-30-raft-supported-design.md | 212 +++++++ 8 files changed, 910 insertions(+), 25 deletions(-) create mode 100644 docs/superpowers/plans/2026-07-30-v1.3.0-raft-supported.md create mode 100644 docs/superpowers/specs/2026-07-30-raft-supported-design.md diff --git a/docs/bg/distributed.md b/docs/bg/distributed.md index 6164c2f..84420d9 100644 --- a/docs/bg/distributed.md +++ b/docs/bg/distributed.md @@ -5,7 +5,7 @@ BaraDB поддържа разпределено внедряване с Raft к > ⚠️ **Ограничение при множество бази данни** > Разпределените модули (Raft, шардиране и репликация) в момента работят само с **`default`** базата данни. Ако използвате множество бази (`CREATE DATABASE`, `USE DATABASE`), разпределените функции още не ги обхващат. Всяка база данни се нуждае от отделна кластър конфигурация. -> **Статус (2026-07-30):** Raft C3a (мрежова election), C3b (SQL записи), DDL репликация, leader forwarding, log compaction и metrics са **на `main`**. Преглед: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. +> **Статус (2026-07-30, v1.3.0):** Raft е **supported** за обхвата single-`default`-DB: failover под товар, raft TLS и cold-node recovery чрез InstallSnapshot са e2e-доказани. Преглед: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. ## Raft Консенсус @@ -20,10 +20,21 @@ Leader election и log репликация през TCP; SQL DML/DDL за **def | `BARADB_RAFT_WRITE_TIMEOUT_MS` | Макс. изчакване за majority commit при SQL записи (по подразбиране 5000) | | `BARADB_RAFT_CLIENT_PEERS` | Опционален `id@host:clientPort` map за leader write forwarding | | `BARADB_RAFT_LOG_MAX_ENTRIES` | Лимит на in-memory raft log (по подразбиране 256); safe prefix compact | +| `BARADB_RAFT_SNAP_CHUNK_KB` | Размер на InstallSnapshot chunk в KiB (по подразбиране 256) | +| `BARADB_RAFT_PEER_STALE_MS` | Peer е „stale“ след толкова ms без ack (по подразбиране 30000); stale peers не блокират compaction | +| `BARADB_RAFT_TLS_ENABLED` | TLS на raft порта (по подразбиране false; стартът спира при липсващ cert/key) | +| `BARADB_RAFT_TLS_CERT_FILE` / `BARADB_RAFT_TLS_KEY_FILE` | Сертификат и ключ за raft listener-а | +| `BARADB_RAFT_TLS_CA_FILE` / `BARADB_RAFT_TLS_VERIFY_PEER` | Опционален CA bundle и mutual auth (по подразбиране false) | Когато Raft е активен, SQL DML и schema DDL се приемат само от лидера на **`default`**. DML отива като put/delete; DDL — като `ddl` запис. Followers **препращат** write/DDL към лидера, ако е зададен `BARADB_RAFT_CLIENT_PEERS`; иначе връщат `not leader; leader is '…'`. Записи към друга database name се отказват. `CREATE`/`DROP DATABASE` не се репликират. Приложен DML обновява и secondary индекси/графи. -**Log compaction (v1):** след apply node-ът може да изреже safe prefix, когато log-ът надхвърли `BARADB_RAFT_LOG_MAX_ENTRIES`. Leader не реже след matchIndex на peer (catch-up с AppendEntries). Snapshot metadata се пази в `raft_state.bin`. +**Log compaction:** след apply node-ът може да изреже safe prefix, когато log-ът надхвърли `BARADB_RAFT_LOG_MAX_ENTRIES`. На лидера safe prefix се смята само по peers с ack в рамките на `BARADB_RAFT_PEER_STALE_MS`; stale peers се възстановяват със snapshot при завръщане. Snapshot metadata се пази в `raft_state.bin`. + +**Snapshot recovery (InstallSnapshot, v1.3.0):** когато изоставането на follower е невъзстановимо, лидерът изпраща `tar.gz` snapshot на default DB на chunk-ове от `BARADB_RAFT_SNAP_CHUNK_KB` KiB. Follower-ът го възстановява през backup/restore пътя и продължава catch-up. Върнат след дълъг прекъсване или **изтрит** (wiped data dir, същото node id) възел конвергира автоматично. E2E: `tests/raft_coldnode_e2e_test.nim`. + +**Клиентски договор при failover:** запис, който е in-flight при смяна на лидера, **гърми бързо с грешка** — клиентът трябва да го повтори (retry). Всеки **потвърден** (acknowledged) запис оцелява failover-а и е наличен на новия лидер и на наваксалите followers. E2E: `tests/raft_failover_load_e2e_test.nim`. + +**Raft TLS (v1.3.0):** `BARADB_RAFT_TLS_ENABLED=true` + cert/key на всеки възел; стартът спира при липсващ cert/key. Целият клъстер трябва да е в един и същ режим — plaintext възел не може да говори с TLS порт и е изключен от клъстера (`tests/raft_tls_e2e_test.nim`). Forwarding-ът follower→leader се обвива в TLS, когато клиентският wire порт е с TLS. **Metrics:** при включен raft `GET /metrics` (HTTP = `BARADB_PORT + 440`) дава Prometheus редове: `baradb_raft_is_leader`, `baradb_raft_term`, `baradb_raft_log_entries`, `baradb_raft_apply_lag`, `baradb_raft_commit_wait_ms_total`, `baradb_raft_elections_total`, `baradb_raft_forwards_total`, `baradb_raft_compactions_total`. `GET /health` включва обект `raft` (`role`, `term`, `leader_id`, …). diff --git a/docs/bg/known-limitations.md b/docs/bg/known-limitations.md index cb17ccf..3464f81 100644 --- a/docs/bg/known-limitations.md +++ b/docs/bg/known-limitations.md @@ -1,4 +1,4 @@ -# Известни ограничения — v1.2.0 Production GA +# Известни ограничения — v1.3.0 | Ниво | Значение | |------|----------| @@ -8,25 +8,33 @@ ## Матрица -| Област | GA (v1.2.0) | Experimental / по-късно | -|--------|-------------|-------------------------| +| Област | v1.3.0 | Experimental / по-късно | +|--------|--------|-------------------------| | Single-node SQL + LSM | **Supported** | — | | Schema / FTS / HNSW / graphs persist | **Supported** | — | | Auth + JWT (когато е конфигуриран) | **Supported** | — | | Backup / restore | **Supported** | — | | Multi-DB (без Raft) | **Supported** | — | -| Raft 3-node + SQL/DDL | **Experimental** | InstallSnapshot, membership | +| Raft 3-node (само `default` DB) | **Supported** | failover под товар, TLS, InstallSnapshot — e2e | | Raft multi-DB | **Not supported** | само `default` | +| `CREATE`/`DROP DATABASE` репликация | **Not supported** | per node | +| Raft membership промени (join/leave) | **Not supported** | фиксиран `BARADB_RAFT_PEERS` | | Follower linearizable reads | **Not supported** | best-effort след apply | +| Rolling upgrades | **Not supported** | рестарт на всички възли заедно — смесени v1.2/v1.3 binaries не трябва да работят в един клъстер | | ORC multi-thread shared LSM | **Not supported** | ARC по подразбиране | ## GA (single-node) Crash recovery с WAL, schema/index persist, `/health` + `/metrics`, offline backup/restore. -## Raft +## Raft (supported) -Виж [distributed.md](distributed.md). Staging/ops, **не** v1.2.0 HA продукт. +Виж [distributed.md](distributed.md). Поддържан обхват: 3-node, DML/DDL само върху `default`, failover под товар (acked writes оцеляват; in-flight writes → грешка, retry), raft TLS, cold-node recovery чрез InstallSnapshot. + +## Нови ограничения + +- **Legacy REP replication (без raft)** — пътят още извежда delete от празна стойност; insert в PK-only таблица се прилага грешно по него (редът изчезва). Използвай raft. +- **Snapshot-restore ctx** — след InstallSnapshot restore HTTP endpoints със startup-captured ctx може да сервират стари данни до рестарт (`/query` е свеж per-request); съществуващите клиентски връзки виждат pre-restore състояние — reconnect след restore. ## Виж също diff --git a/docs/en/distributed.md b/docs/en/distributed.md index 8a05c98..b04cb11 100644 --- a/docs/en/distributed.md +++ b/docs/en/distributed.md @@ -5,7 +5,7 @@ BaraDB supports distributed deployment with Raft consensus, sharding, and replic > ⚠️ **Multi-Database Limitation** > The distributed modules (Raft, sharding, and replication) are currently wired to the **`default`** database only. If you use multiple databases (`CREATE DATABASE`, `USE DATABASE`), distributed features do not yet span across them. Each database would need its own cluster setup. -> **Status (2026-07-30):** Raft C3a/C3b + DDL/forward/compact/metrics are **shipped**. Multi-node Raft is **experimental** for v1.2.0 GA (single-node is the production tier). See [known-limitations](known-limitations.md) and `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. +> **Status (2026-07-30, v1.3.0):** Raft C3a/C3b + DDL/forward/compact/metrics are **shipped**, and multi-node Raft is **supported** for the single-`default`-DB scope (failover under load, raft TLS, InstallSnapshot cold-node recovery — all e2e-proven). See [known-limitations](known-limitations.md) and `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`. ## Raft Consensus @@ -20,10 +20,23 @@ Leader election and log replication over TCP; SQL DML/DDL on the default DB go t | `BARADB_RAFT_WRITE_TIMEOUT_MS` | Max wait for majority commit on SQL writes (default 5000) | | `BARADB_RAFT_CLIENT_PEERS` | Optional `id@host:clientPort` map for leader write forwarding | | `BARADB_RAFT_LOG_MAX_ENTRIES` | Soft cap on in-memory raft log length (default 256); safe prefix compact | +| `BARADB_RAFT_SNAP_CHUNK_KB` | InstallSnapshot chunk size in KiB (default 256) | +| `BARADB_RAFT_PEER_STALE_MS` | Peer is stale after this many ms without an ack (default 30000); stale peers no longer pin log compaction | +| `BARADB_RAFT_TLS_ENABLED` | TLS on the raft TCP port (default false; fail-closed startup if cert/key missing) | +| `BARADB_RAFT_TLS_CERT_FILE` | Server certificate for the raft listener | +| `BARADB_RAFT_TLS_KEY_FILE` | Private key for the raft listener | +| `BARADB_RAFT_TLS_CA_FILE` | Optional CA bundle for peer verification | +| `BARADB_RAFT_TLS_VERIFY_PEER` | Mutual auth — verify client certificates (default false) | When Raft is enabled, SQL DML (`INSERT`/`UPDATE`/`DELETE`/`MERGE` and transactional `COMMIT`) and schema DDL (`CREATE`/`DROP`/`ALTER` table, index, view, graph, …) are accepted only on the leader of the **`default`** database. DML ships as put/delete log entries; DDL ships as a `ddl` entry with the original SQL and is re-executed on every node at apply. Followers that receive a write/DDL **forward** it to the leader when `BARADB_RAFT_CLIENT_PEERS` maps the leader id to a SQL client address; otherwise they return `not leader; leader is '…'`. Writes against any other database name are rejected (`raft writes only supported on the 'default' database`). `CREATE`/`DROP DATABASE` are not raft-replicated (multi-DB is out of scope for v1). Committed DML also updates secondary B-tree/FTS/HNSW indexes and in-memory graphs. -**Log compaction (v1):** after apply, each node may drop a fully-safe log prefix once `log.len` exceeds `BARADB_RAFT_LOG_MAX_ENTRIES`. The leader never discards past any peer's `matchIndex` (so lagging followers still catch up via AppendEntries). Snapshot metadata (`lastSnapshotIndex`/`Term`) is persisted in `raft_state.bin`; full InstallSnapshot state-machine payloads are not required while this safe-prefix policy holds. +**Log compaction:** after apply, each node may drop a fully-safe log prefix once `log.len` exceeds `BARADB_RAFT_LOG_MAX_ENTRIES`. On the leader, the safe prefix is computed only over peers that acked within `BARADB_RAFT_PEER_STALE_MS` — stale peers no longer pin compaction and are recovered by snapshot on return. Compaction never goes past `lastApplied`. Snapshot metadata (`lastSnapshotIndex`/`Term`) is persisted in `raft_state.bin`. + +**Snapshot recovery (InstallSnapshot, v1.3.0):** when a follower's lag is unrecoverable (the entries it needs were compacted away), the leader builds a `tar.gz` snapshot of the default DB and streams it as `BARADB_RAFT_SNAP_CHUNK_KB`-sized chunks. The follower restores it via the backup/restore path, adopts the snapshot base as its `commitIndex`/`lastApplied`, and resumes normal AppendEntries catch-up. A node that returns after a long outage — and a **wiped** node (data dir deleted, same node id) — both converge automatically through this path. Proven by `tests/raft_coldnode_e2e_test.nim`. + +**Client failover contract:** a write that is in flight when the leader dies **fails fast with an error** — the client must retry it (against the new leader, or any follower if `BARADB_RAFT_CLIENT_PEERS` forwarding is configured). Every write the server **acknowledged** survives the failover and is present on the new leader and all caught-up followers. Proven by `tests/raft_failover_load_e2e_test.nim` (leader killed under sustained INSERT load; all acked writes found on both survivors). + +**Raft TLS (v1.3.0):** set `BARADB_RAFT_TLS_ENABLED=true` plus `BARADB_RAFT_TLS_CERT_FILE`/`BARADB_RAFT_TLS_KEY_FILE` on every node; startup fails closed if the cert or key is missing. Add `BARADB_RAFT_TLS_CA_FILE` and `BARADB_RAFT_TLS_VERIFY_PEER=true` for mutual authentication. The whole cluster must run the same mode: a plaintext node cannot speak to a TLS port (its frames are undecryptable) and is excluded from the cluster — proven by `tests/raft_tls_e2e_test.nim`. Follower→leader SQL forwarding is TLS-wrapped automatically when the server's client wire port has TLS enabled. **Metrics:** with raft enabled, `GET /metrics` (HTTP port = `BARADB_PORT + 440`) includes Prometheus lines such as `baradb_raft_is_leader`, `baradb_raft_term`, `baradb_raft_log_entries`, `baradb_raft_apply_lag`, `baradb_raft_commit_wait_ms_total`, `baradb_raft_elections_total`, `baradb_raft_forwards_total`, and `baradb_raft_compactions_total`. `GET /health` embeds a `raft` object (`role`, `term`, `leader_id`, `commit_index`, `apply_lag`, …). @@ -67,6 +80,9 @@ let entry = n1.appendLog("SET key1 value1") |------|----------------| | `tests/raft_e2e_test.nim` | 3 real processes; election + kill-leader failover | | `tests/raft_writes_e2e_test.nim` | DDL/DML via raft, follower forward, index SELECT, failover writes | +| `tests/raft_failover_load_e2e_test.nim` | Leader killed under sustained write load; every acked write survives | +| `tests/raft_tls_e2e_test.nim` | Full-TLS 3-node cluster works; plaintext node excluded | +| `tests/raft_coldnode_e2e_test.nim` | Returning node and wiped node converge via InstallSnapshot | ## Sharding diff --git a/docs/en/known-limitations.md b/docs/en/known-limitations.md index bf64708..4c00bd0 100644 --- a/docs/en/known-limitations.md +++ b/docs/en/known-limitations.md @@ -1,6 +1,6 @@ -# Known Limitations — v1.2.0 Production GA +# Known Limitations — v1.3.0 -This page defines **what BaraDB promises** in the v1.2.0 production cut. +This page defines **what BaraDB promises** in the v1.3.0 production cut. | Tier | Meaning | |------|---------| @@ -10,20 +10,22 @@ This page defines **what BaraDB promises** in the v1.2.0 production cut. ## Support matrix -| Area | GA (v1.2.0) | Experimental / later | -|------|-------------|----------------------| +| Area | v1.3.0 | Notes | +|------|--------|-------| | Single-node SQL + LSM storage | **Supported** | — | | Schema persistence (tables, indexes) | **Supported** | — | | FTS / HNSW / graphs across restart | **Supported** | — | | Auth + JWT (when configured) | **Supported** | — | | Backup / restore (offline, all-databases) | **Supported** | — | | Multi-database (non-Raft) | **Supported** | — | -| Raft 3-node election + SQL/DDL | **Experimental** | InstallSnapshot SM payload, membership | +| Raft 3-node (single `default` DB) | **Supported** | failover under load, raft TLS, InstallSnapshot recovery — e2e-proven | +| Leader write forwarding | **Supported** | needs `BARADB_RAFT_CLIENT_PEERS` | | Raft multi-database | **Not supported** | only `default` | -| Leader write forwarding | **Experimental** | needs `BARADB_RAFT_CLIENT_PEERS` | +| `CREATE`/`DROP DATABASE` replication | **Not supported** | run per node | +| Raft membership changes (join/leave) | **Not supported** | fixed `BARADB_RAFT_PEERS` set | | Follower linearizable reads | **Not supported** | best-effort after apply | +| Rolling upgrades | **Not supported** | restart all nodes together — mixed v1.2/v1.3 binaries must not run in one cluster | | ORC multi-threaded shared LSM | **Not supported** | default is ARC (`nim.cfg`) | -| Zero-downtime rolling upgrade | **Not supported** | stop → backup → upgrade | | Postgres wire protocol | **Not supported** | Bara wire + HTTP | ## Single-node GA (what you can rely on) @@ -33,13 +35,19 @@ This page defines **what BaraDB promises** in the v1.2.0 production cut. - HTTP `/health` and `/metrics` for process liveness - Offline backup of `data/databases` and restore onto an empty data root -## Raft (experimental ops) +## Raft (supported, single-default-DB scope) -Documented in [distributed.md](distributed.md). Suitable for learning and careful staging; **not** the v1.2.0 HA product tier. +Documented in [distributed.md](distributed.md). Supported scope: -- SQL DML/DDL on **`default` only** -- Safe log prefix compact (not full InstallSnapshot) -- Failover proven in process e2e tests +- 3-node cluster, SQL DML/DDL on **`default` only** +- Failover under write load: every acknowledged write survives a leader kill; in-flight writes fail fast — clients must retry +- TLS on the raft port and on follower→leader forwarding (`BARADB_RAFT_TLS_*`) +- Cold-node recovery via InstallSnapshot (`BARADB_RAFT_SNAP_CHUNK_KB`, `BARADB_RAFT_PEER_STALE_MS`) + +## Newly documented limitations + +- **Legacy non-raft REP replication infers delete from empty value** — the non-raft replication path still treats an empty value as a delete, so inserts into a PK-only table are misapplied over that path (the row vanishes). Use raft replication instead. +- **Snapshot-restore ctx staleness** — after an InstallSnapshot restore, HTTP endpoints using the startup-captured ctx may serve stale data until the node is restarted; the `/query` path is fresh per-request. Pre-existing client connections likewise see pre-restore state — reconnect after a restore. ## Operational requirements diff --git a/docs/en/release-checklist.md b/docs/en/release-checklist.md index abce0ee..cd852d8 100644 --- a/docs/en/release-checklist.md +++ b/docs/en/release-checklist.md @@ -23,9 +23,12 @@ nim c -d:ssl --threads:on --path:src -r tests/test_schema_persist.nim ./scripts/backup-restore-drill.sh DRILL_PORT=19482 ./scripts/backup-restore-drill.sh -# Optional cluster e2e (experimental tier) +# Cluster e2e (raft supported tier — all five suites) ./tests/raft_e2e_test ./tests/raft_writes_e2e_test +./tests/raft_failover_load_e2e_test +./tests/raft_tls_e2e_test +./tests/raft_coldnode_e2e_test ``` ## Production compose diff --git a/docs/superpowers/plans/2026-07-30-v1.3.0-raft-supported.md b/docs/superpowers/plans/2026-07-30-v1.3.0-raft-supported.md new file mode 100644 index 0000000..0836cef --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-v1.3.0-raft-supported.md @@ -0,0 +1,598 @@ +# v1.3.0 Raft-Supported — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> Spec first: `docs/superpowers/specs/2026-07-30-raft-supported-design.md`. + +**Goal:** Move raft from experimental to supported: proven failover under +load, mandatory CI e2e, cold-node recovery via InstallSnapshot, raft-port TLS. + +**Architecture:** No changes to election/AppendEntries semantics. New +InstallSnapshot message pair rides the existing framed TCP transport +(backward-compatible trailing fields, `RaftProtoVersion` stays 1). TLS wraps +the existing transport via `protocol/ssl.nim`. Snapshot payload reuses +`core/backup.nim` tar.gz backup/restore. + +**Tech stack:** Nim 2.2.x, std/asyncnet + std/net SSL, existing +`tests/raft_*_e2e_test.nim` process harness, GitHub Actions. + +## Global Constraints + +- Spec: `docs/superpowers/specs/2026-07-30-raft-supported-design.md`. +- Do **not** change election safety, commit rules, or the DDL/DML + classification from C3b/C3c. +- Raft remains default-DB-only; snapshot payload covers the default DB only. +- Compile all touched Nim with `-d:ssl --threads:on --path:src`. +- Test baseline per task: `tests/test_all.nim` + `tests/bugfix_test.nim` must + stay green; raft e2e suites green where the binary is built. +- Branch: `main` (short-lived feature branches merged same day are fine). +- Version bump to 1.3.0 happens only in the final task (T12). + +--- + +## Phase map + +| Phase | Tasks | Outcome | +|-------|-------|---------| +| P1 Proof | T1 | Failover-under-load e2e | +| P2 CI | T2 | Mandatory raft e2e CI gate | +| P3 TLS | T3–T6 | Raft port TLS + mutual auth + TLS e2e | +| P4 Cold node | T7–T11 | InstallSnapshot + compaction unpin + cold-node e2e | +| P5 Release | T12 | Docs, limitations, version 1.3.0 | + +--- + +### Task 1: Failover-under-load E2E + +**Files:** +- Create: `tests/raft_failover_load_e2e_test.nim` +- Modify: `baradadb.nimble` (task `test`, line ~30-34: add `raft_failover_load_e2e_test` after `raft_writes_e2e_test`) + +**Interfaces:** +- Consumes: process harness conventions from `tests/raft_writes_e2e_test.nim` + (`NodeProc`, `drainOutput`, `portOpen`, `openClient`, `waitForRow` pattern); + client `adaptors/nim/baradb_sqlite`. +- Produces: suite `Raft failover under load E2E`. + +**Scenario (spec D1):** +- Port base `cbase = 50000 + (tstamp mod 4000)`, `rbase = cbase + 100` + (distinct from 35000/41000/46000 bases already in use). +- Boot 3 nodes with `BARADB_RAFT_*` env exactly as + `raft_writes_e2e_test.nim:150-171`. +- Wait for stable leader (same `maxLeader` logic). Leader DDL: + `CREATE TABLE load_test (id INT PRIMARY KEY)`. +- Load phase: spawn a Nim `Thread` that loops `n = 1, 2, ...`: + `INSERT INTO load_test (id) VALUES (n)` against the current leader's + client port; every **acknowledged** `n` appended to a + `seq[int]` guarded by a `Lock`. On exception: reopen client against a + survivor, continue (this models the documented client retry contract). +- At ≥ 50 acked writes: `killNode(leader)`. +- Assert A (availability): some survivor accepts an INSERT within 10 s of + the kill. +- Assert B (durability): after new leader is stable and the remaining + follower caught up (poll `SELECT count(*)` equality or 10 s deadline), + `SELECT id FROM load_test` on **both** survivors contains every acked id. +- Stop the writer thread in `finally`; reuse the `dumpAll`-on-fail + convention. + +- [x] **Step 1:** Write the suite skeleton: harness copied from + `raft_writes_e2e_test.nim` (drain/kill/leader-discovery helpers), writer + thread, kill at 50 acked, asserts A + B. +- [x] **Step 2:** Build binary and run: + `nim c -o:build/baradadb src/baradadb.nim && nim c -d:ssl --threads:on --path:src -r tests/raft_failover_load_e2e_test.nim` + Expected: PASS. +- [x] **Step 3:** Run 3 consecutive times (failover timing flakiness check). +- [x] **Step 4:** Add suite to `nimble test` list in `baradadb.nimble`. +- [x] **Step 5:** Commit + `test(raft): failover under sustained write load e2e` + +--- + +### Task 1a: Fix raft put/delete encoding for empty values (bug found in T1) + +**Bug:** `execInsert` (`src/barabadb/query/exec/dml.nim:60-90`) stores only +non-PK columns in the value, so a PK-only table row gets `valStr = ""` and +`kvPairs.add((fullKey, @[]))`. `appendWriteToRaft` +(`src/barabadb/core/server.nim:309-330`) encodes an empty value as a +`"delete"` log entry — but `execDelete` (`dml.nim:223-241`) uses the same +`(fullKey, @[])` shape for real deletes. Result: INSERT into a PK-only +table returns OK after majority commit, then every node (leader included, +on apply) **deletes the row**. Verified live in T1: 30 acked inserts → 0 +rows on all nodes. + +**Files:** +- Modify: `src/barabadb/query/exec/types.nim:139` — `ExecResult.keyValuePairs` +- Modify: `src/barabadb/query/exec/dml.nim` — 3 producer sites (insert ~90, + delete ~241, update ~316) +- Modify: `src/barabadb/core/server.nim` — `appendWriteToRaft` (~309) and + its call site (~441-464) +- Test: `tests/bugfix_test.nim` (new suite) + +**Interfaces:** +- Change the pair type to carry the op explicitly: + +```nim +# exec/types.nim +keyValuePairs*: seq[tuple[key: string, value: seq[byte], deleted: bool]] +``` + +- `execInsert`/`execUpdate` produce `deleted: false` (even when + `value.len == 0`); `execDelete` produces `deleted: true`. +- `appendWriteToRaft` encodes `deleted` → `"delete"`, else `"put"` + (empty value stays a put). Apply side (`baradadb.nim:358-368`) already + handles `put` with empty value correctly — no change needed there. +- Check other `keyValuePairs` consumers compile clean (replication path in + `server.nim`); keep `okResult(kvPairs=...)` call sites type-correct. + +**Steps:** +- [x] **Step 1:** Write the failing test in `tests/bugfix_test.nim`: + build `ExecResult` via the insert path for a PK-only table (or call + `appendWriteToRaft` semantics directly): assert a PK-only insert yields + a pair with `deleted == false` and encodes as `"put"`, while a delete + yields `deleted == true` and encodes as `"delete"`. +- [x] **Step 2:** Run, expect fail/compile error. +- [x] **Step 3:** Implement the type + producer/consumer changes. +- [x] **Step 4:** `bugfix_test` + `test_all` green; rebuild + `build/baradadb` and re-run `tests/raft_failover_load_e2e_test.nim` — + then **switch its table back** to the brief's original + `load_test (id INT PRIMARY KEY)` / `VALUES (n)` shape (remove the + two-column workaround and its header note) and re-run green. +- [x] **Step 5:** Commit + `fix(raft): distinguish put-with-empty-value from delete in write path` + +--- + +### Task 2: Mandatory raft e2e CI gate + +**Files:** +- Modify: `.github/workflows/ci.yml` +- Modify: `tests/raft_e2e_test.nim`, `tests/raft_writes_e2e_test.nim`, + `tests/raft_failover_load_e2e_test.nim` (skip→fail under CI) + +**Steps:** +- [x] **Step 1:** In each suite's binary-missing branch, replace plain + `skip()` with: + +```nim +if not fileExists(BinaryPath): + if getEnv("CI").len > 0: + echo "[FAIL] ", BinaryPath, " missing under CI — build step broken?" + fail() + else: + echo "[SKIP] ", BinaryPath, " missing — run `nimble test` (builds the server first)" + skip() +``` + +- [x] **Step 2:** Add a dedicated job to `.github/workflows/ci.yml` (after + the `test` job), modeled on its setup steps: + +```yaml + raft-e2e: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Setup Nim + uses: jiro4989/setup-nim-action@v1 + with: + nim-version: '2.2.10' + - name: Install system dependencies + run: sudo apt-get update -qq && sudo apt-get install -y -qq libssl-dev libpcre3-dev openssl ca-certificates + - name: Install Nim dependencies + run: nimble install --depsOnly -y + - name: Build server + run: nim c -d:ssl -o:build/baradadb src/baradadb.nim + - name: Raft e2e suites + env: + CI: "true" + run: | + nim c -d:ssl --threads:on --path:src -r tests/raft_e2e_test.nim + nim c -d:ssl --threads:on --path:src -r tests/raft_writes_e2e_test.nim + nim c -d:ssl --threads:on --path:src -r tests/raft_failover_load_e2e_test.nim +``` + +- [x] **Step 3:** Push on a branch; confirm the `raft-e2e` job appears and + is green; confirm deleting `build/` would fail (local run with `CI=true` + and no binary → FAIL). +- [x] **Step 4:** Commit + `ci(raft): dedicated mandatory raft e2e job; no silent skips under CI` + +--- + +### Task 3: Raft TLS config + fail-closed startup + +**Files:** +- Modify: `src/barabadb/core/config.nim` (fields after `raftLogMaxEntries` + at lines ~46/88; env parsing after line ~212) +- Modify: `src/baradadb.nim` (raft wiring block, lines 337-377) +- Test: `tests/bugfix_test.nim` (new suite) + +**Interfaces:** +- Produces config fields (used by T4/T6): + +```nim +raftTlsEnabled*: bool # default false +raftTlsCertFile*: string # default "" +raftTlsKeyFile*: string # default "" +raftTlsCaFile*: string # default "" +raftTlsVerifyPeer*: bool # default false +``` + +- Env parsing (mirror lines 184-212 style): + +```nim +cfg.raftTlsEnabled = parseEnvBool(getEnv("BARADB_RAFT_TLS_ENABLED", ""), cfg.raftTlsEnabled) +cfg.raftTlsCertFile = getEnv("BARADB_RAFT_TLS_CERT_FILE", cfg.raftTlsCertFile) +cfg.raftTlsKeyFile = getEnv("BARADB_RAFT_TLS_KEY_FILE", cfg.raftTlsKeyFile) +cfg.raftTlsCaFile = getEnv("BARADB_RAFT_TLS_CA_FILE", cfg.raftTlsCaFile) +cfg.raftTlsVerifyPeer = parseEnvBool(getEnv("BARADB_RAFT_TLS_VERIFY_PEER", ""), cfg.raftTlsVerifyPeer) +``` + +- Fail-closed in `baradadb.nim` before `newRaftNetwork`: + +```nim +if config.raftTlsEnabled: + if config.raftTlsCertFile.len == 0 or config.raftTlsKeyFile.len == 0 or + not fileExists(config.raftTlsCertFile) or not fileExists(config.raftTlsKeyFile): + raise newException(ValueError, + "BARADB_RAFT_TLS_ENABLED=true but cert/key missing " & + "(BARADB_RAFT_TLS_CERT_FILE / BARADB_RAFT_TLS_KEY_FILE)") +``` + +- [x] **Step 1:** Write failing test in `tests/bugfix_test.nim`: default + config has `raftTlsEnabled == false`; env `BARADB_RAFT_TLS_ENABLED=true` + + cert paths parse into config. +- [x] **Step 2:** Run test, expect compile/fail (fields don't exist). +- [x] **Step 3:** Implement fields + env parsing + startup check. +- [x] **Step 4:** Test green; `test_all` still green. +- [x] **Step 5:** Commit + `feat(raft): TLS config surface with fail-closed startup` + +--- + +### Task 4: TLS in RaftNetwork transport + +**Files:** +- Modify: `src/barabadb/core/raft.nim` (`RaftNetwork` type ~line 731, + `connectToPeer` ~748, `run` ~857) +- Modify: `src/baradadb.nim` (construct TLSContext, pass to network) +- Test: `tests/test_all.nim` (in-process TLS raft pair) + +**Interfaces:** +- Consumes: `protocol/ssl.nim` — `newTLSConfig(certFile, keyFile, caFile, + verifyPeer)`, `newTLSContext`, `wrapClient`, `wrapServer`. +- Produces: `RaftNetwork.tls*: TLSContext` (nil = plaintext, unchanged + default); `newRaftNetwork(node, tls = nil)`. + +**Implementation:** + +```nim +# raft.nim — in connectToPeer, after successful connect: +if net.tls != nil: + try: net.tls.wrapClient(sock) + except CatchableError: + try: sock.close() except CatchableError: discard + return + +# raft.nim — in run, after accept, before receiveLoop: +if net.tls != nil: + try: net.tls.wrapServer(client) + except CatchableError: + client.close() + continue +``` + +- `baradadb.nim`: build the context when enabled: + +```nim +var raftTls: TLSContext = nil +if config.raftTlsEnabled: + raftTls = newTLSContext(newTLSConfig( + config.raftTlsCertFile, config.raftTlsKeyFile, + caFile = config.raftTlsCaFile, verifyPeer = config.raftTlsVerifyPeer)) +... +raftNet = newRaftNetwork(raftNode, raftTls) +``` + +- [x] **Step 1:** Write failing in-process test (`test_all.nim`): two + `RaftNode`s over `RaftNetwork` with a self-signed cert from + `generateSelfSignedCert` (`protocol/ssl.nim:79`) — election completes + over TLS; plaintext dial to the TLS port produces no protocol effect + (no state change, connection dropped). +- [x] **Step 2:** Run, expect fail (no `tls` field). +- [x] **Step 3:** Implement transport changes + wiring. +- [x] **Step 4:** Test green; plaintext raft e2e suites still green + (regression: nil-TLS path untouched). +- [x] **Step 5:** Commit + `feat(raft): optional TLS on raft transport (server + dialer)` + +--- + +### Task 5: TLS for leader SQL forwarding + +**Files:** +- Modify: `src/barabadb/core/server.nim` (`forwardQueryToLeader`, lines + 210-289) + +**Interfaces:** +- Consumes: `protocol/ssl.nim` `wrapClient`; server config `tlsEnabled`, + `certFile`, `keyFile`. +- Produces: `forwardQueryToLeader(host, port, query, tls: TLSContext = nil, + ...)`. + +- [x] **Step 1:** When the server's client wire port has TLS on + (`server.tls != nil`), wrap the forwarding socket with a client-side + context before sending the wire header; on handshake failure return + `(false, QueryResult(), "leader forward TLS handshake failed")`. +- [x] **Step 2:** Manual check: TLS server + raft forwarding (follower + INSERT forwarded over TLS) works; non-TLS setup unchanged. +- [x] **Step 3:** Commit + `feat(raft): TLS on follower→leader SQL forwarding` + +--- + +### Task 6: Raft TLS E2E + +**Files:** +- Create: `tests/raft_tls_e2e_test.nim` +- Modify: `baradadb.nimble` (test list), `.github/workflows/ci.yml` + (raft-e2e job: add this suite) + +**Interfaces:** +- Consumes: T3/T4 implementation; harness from T1; openssl CLI for cert + generation (already used by `protocol/ssl.nim`). + +**Scenario:** +- Port base `54000 + (tstamp mod 4000)`. +- Generate one self-signed cert per node into the temp data dirs + (`openssl req -x509 ...`, or `generateSelfSignedCert`). +- Boot 3 nodes with `BARADB_RAFT_TLS_ENABLED=true` + per-node cert/key; + assert election + `CREATE TABLE` via raft DDL + one replicated INSERT + visible on a follower. +- Negative: start a 4th process with raft TLS **disabled** pointed at the + same peers; assert the TLS cluster still elects/operates among its 3 + members and the plaintext node never becomes leader (its frames are + undecryptable). +- Same `CI`-fail semantics as T2. + +- [x] **Step 1:** Write suite; Step 2: run green locally; Step 3: run 3× + (timing); Step 4: wire into `nimble test` + CI job; Step 5: Commit + `test(raft): 3-node TLS cluster e2e with plaintext rejection` + +--- + +### Task 7: InstallSnapshot protocol + +**Files:** +- Modify: `src/barabadb/core/raft.nim` (`RaftMessageKind` ~line 80, + `RaftMessage` ~86, `serialize` ~679, `deserializeRaftMessage` ~702) +- Test: `tests/test_all.nim` (serialize/deserialize round-trip) + +**Interfaces:** +- Produces: + +```nim +# RaftMessageKind += rmkInstallSnapshot, rmkInstallSnapshotReply +# RaftMessage new fields: +snapId*: uint64 # snapshot generation, matches leader's base at build time +snapOffset*: uint64 # byte offset of this chunk within the archive +snapData*: seq[byte] # chunk payload (<= snapChunkBytes) +snapDone*: bool # last chunk +# Reused for this kind: prevLogIndex = snapshot base index, +# prevLogTerm = snapshot base term. Reply uses success/matchIdx as usual. +``` + +- Serialization: append `snapId`, `snapOffset`, `snapData` (length-prefixed), + `snapDone` **after** `matchIdx`; deserialize each with `if not s.atEnd` + guards (pattern from `loadState`, raft.nim:163-167). Old binaries ignore + trailing bytes; new binaries default missing fields to zero/false. + `RaftProtoVersion` stays 1. + +- [x] **Step 1:** Write failing round-trip test: all new fields survive + serialize→deserialize; a buffer serialized by the *old* layout (no + trailing fields) deserializes with zero defaults. +- [x] **Step 2:** Run, expect fail. +- [x] **Step 3:** Implement. +- [x] **Step 4:** Green; existing raft suites still green. +- [x] **Step 5:** Commit + `feat(raft): InstallSnapshot wire protocol (backward-compatible)` + +--- + +### Task 8: Follower snapshot receive + restore + +**Files:** +- Modify: `src/barabadb/core/raft.nim` (`RaftNode` — new callback + + incoming-snapshot buffer; `processMessage` ~786) +- Modify: `src/barabadb/core/config.nim` (`raftSnapChunkKb: int`, env + `BARADB_RAFT_SNAP_CHUNK_KB`, default 256; parsing next to the other + `BARADB_RAFT_*` env reads) +- Modify: `src/baradadb.nim` (wire `restoreSnapshot` callback using + `core/backup.nim` + `DatabaseRegistry`; pass chunk size to the node) +- Test: `tests/test_all.nim` + +**Interfaces:** +- Produces on `RaftNode`: + +```nim +snapChunkBytes*: int # from BARADB_RAFT_SNAP_CHUNK_KB, default 262144 +restoreSnapshot*: proc(archivePath: string, baseIndex: uint64, + baseTerm: uint64): bool {.gcsafe.} +snapIncomingId*: uint64 +snapIncomingFile*: string # temp path under dataDir/raft/snap_incoming/ +``` + +- `processMessage` case `rmkInstallSnapshot`: append `snapData` at + `snapOffset` to the temp file (create/truncate when `snapId != + snapIncomingId`); on `snapDone`: call `restoreSnapshot`; on success set + `lastSnapshotIndex/Term = prevLogIndex/prevLogTerm`, + `commitIndex = lastApplied = lastSnapshotIndex`, clear `log`, + `saveState()`, reply success with `matchIdx = lastSnapshotIndex`; on + failure reply `success = false` and delete the temp file. +- `baradadb.nim` `restoreSnapshot` implementation: close default DB via + registry, `restoreDataDir(archivePath, defaultDbDir)` + (`backup.nim:263`), reopen, swap `ctx`. Return false on any exception. + +- [x] **Step 1:** Write failing test: feed a node two chunks + done with a + real tar.gz fixture; assert callback received the assembled file, state + fields updated, log cleared. +- [x] **Step 2:** Run, expect fail. **Step 3:** Implement. +- [x] **Step 4:** Green + regression suites. **Step 5:** Commit + `feat(raft): follower InstallSnapshot receive and restore` + +--- + +### Task 9: Leader snapshot send + +**Files:** +- Modify: `src/barabadb/core/raft.nim` (`handleAppendReply` floor branch + ~526-531, new `sendSnapshot` proc, per-peer reject counter) +- Modify: `src/baradadb.nim` (wire `buildSnapshot` callback) +- Test: `tests/test_all.nim` + +**Interfaces:** +- Consumes: T7 protocol, `backupDataDir` (`backup.nim:225`). +- Produces on `RaftNode`: + +```nim +buildSnapshot*: proc(destPath: string): bool {.gcsafe.} +snapRejectStreak*: Table[string, int] # consecutive floor-level rejects per peer +``` + +- Logic: in `handleAppendReply`, when a reject arrives **and** + `nextIndex[peerId] == lastSnapshotIndex + 1` (floor reached): increment + streak; at streak ≥ 2 the leader knows the follower needs a snapshot → + `asyncCheck sendSnapshot(peerId)`. Reset streak on any successful reply. +- `sendSnapshot`: `buildSnapshot` into `dataDir/raft/snap_out_.tar.gz` + (`snapId = lastSnapshotIndex`); stream chunks of `snapChunkBytes` as + `rmkInstallSnapshot`; on final success reply set + `matchIndex[peer] = lastSnapshotIndex`, + `nextIndex[peer] = lastSnapshotIndex + 1`; delete the temp archive. + +- [x] **Step 1:** Write failing test: leader with compacted log + (`lastSnapshotIndex = 100`) + peer at floor rejecting twice → snapshot + messages emitted; success reply advances `matchIndex`/`nextIndex`. +- [x] **Step 2:** Run, expect fail. **Step 3:** Implement. +- [x] **Step 4:** Green + regression. **Step 5:** Commit + `feat(raft): leader InstallSnapshot send on unrecoverable lag` + +--- + +### Task 10: Unpin compaction from dead peers + +**Files:** +- Modify: `src/barabadb/core/raft.nim` (`compactLog` ~244, `becomeLeader` + ~327, `handleAppendReply` ~487) +- Modify: `src/barabadb/core/config.nim` (`raftPeerStaleMs`, env + `BARADB_RAFT_PEER_STALE_MS`, default 30000) +- Test: `tests/test_all.nim` + +**Interfaces:** +- Produces: `matchIndexSeenMs*: Table[string, int64]` on `RaftNode` — + monotonic ms timestamp of the last successful reply per peer, updated in + `handleAppendReply` success branch and initialized to "now" in + `becomeLeader`. + +**Logic:** leader-side `compactLog` computes `minMatch` only over peers +with `now - matchIndexSeenMs[peer] <= raftPeerStaleMs`; peers stale longer +are excluded (they'll be snapshotted on return per T9). Follower path +unchanged. Guard: never compact past `lastApplied`. + +- [x] **Step 1:** Write failing test: leader, one peer never replies, + log > maxEntries → with default stale window, log compacts through + lastApplied anyway; with the peer responsive, compaction still pins at + its matchIndex (existing safety preserved). +- [x] **Step 2:** Run, expect fail. **Step 3:** Implement. +- [x] **Step 4:** Green + regression. **Step 5:** Commit + `feat(raft): compaction unpinned from stale peers (snapshot fallback)` + +--- + +### Task 11: Cold-node E2E + +**Files:** +- Create: `tests/raft_coldnode_e2e_test.nim` +- Modify: `baradadb.nimble`, `.github/workflows/ci.yml` (raft-e2e job) + +**Interfaces:** Consumes T7–T10; harness from T1. Port base +`58000 + (tstamp mod 4000)`. Small `BARADB_RAFT_LOG_MAX_ENTRIES=16` and +`BARADB_RAFT_PEER_STALE_MS=3000` to force compaction quickly. + +**Scenario A — node returns after compaction:** +1. 3-node cluster, create table, kill node n3. +2. Write 100 rows through the leader (forces compaction past n3's + matchIndex once n3 is stale). +3. Assert via `/metrics` on the leader HTTP port + (`baradb_raft_log_entries`) that the log stayed bounded. +4. Restart n3 with its intact data dir; assert it receives a snapshot + (leader log line / `baradb_raft_snapshot_index` advances on n3) and + within 15 s `SELECT count(*)` on n3 matches the leader. + +**Scenario B — wiped node joins:** +1. Stop n3, **delete its data dir**, restart with the same node id. +2. Assert it converges (snapshot → catch-up) and serves the full row set + within 20 s. + +- [x] **Step 1:** Write suite. **Step 2:** Green locally. **Step 3:** 3× + stability runs. **Step 4:** `nimble test` + CI wiring. **Step 5:** Commit + `test(raft): cold-node rejoin and wiped-node join e2e` + +--- + +### Task 12: Docs, limitations, version 1.3.0 + +**Files:** +- Modify: `docs/en/distributed.md`, `docs/bg/distributed.md` — client + failover contract (in-flight writes fail fast, retry; acked writes + durable), TLS setup (`BARADB_RAFT_TLS_*`), snapshot behavior/tunables + (`BARADB_RAFT_SNAP_CHUNK_KB`, `BARADB_RAFT_PEER_STALE_MS`) +- Modify: `docs/en/known-limitations.md`, `docs/bg/known-limitations.md` — + raft 3-node moves from "Experimental" to "Supported" for the covered + scope; remaining non-goals (multi-DB raft, membership changes, read + consistency levels) stay listed +- Modify: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md` — + status → v1.3.0 supported +- Modify: `CHANGELOG.md` — `## [1.3.0] — ` +- Modify: `baradadb.nimble` → `version = "1.3.0"`; README status lines +- Modify: `docs/en/release-checklist.md` — add raft TLS + cold-node suites + +- [x] **Step 1:** Doc edits. **Step 2:** Full `nimble test` green. +- [x] **Step 3:** Commit + `release: v1.3.0 raft-supported (failover load, CI gate, snapshot, TLS)` +- [ ] **Step 4 (human/controller):** tag `v1.3.0` after review. + +--- + +## Task dependency graph + +``` +T1 failover-load e2e ──→ T2 CI gate +T3 TLS config ──→ T4 transport TLS ──→ T5 forward TLS ──→ T6 TLS e2e +T7 snapshot protocol ──→ T8 follower restore ──→ T9 leader send ──→ T10 unpin ──→ T11 cold-node e2e +all ──→ T12 docs/version +``` + +## Explicit out-of-scope + +- Membership change (join/leave) protocol +- Multi-database raft; `CREATE`/`DROP DATABASE` replication +- Linearizable follower reads +- Rolling-upgrade compat shims beyond the trailing-field guard + (upgrade = restart all nodes) + +## Definition of Done + +- [x] All P1–P5 tasks complete, `nimble test` green +- [ ] `raft-e2e` CI job green and mandatory (no silent skip) +- [x] Failover-under-load e2e: every acked write survives leader kill +- [x] Cold-node e2e: returning node and wiped node converge automatically +- [x] Raft TLS e2e: full-TLS cluster works; plaintext node excluded +- [x] known-limitations updated: raft supported for the covered scope + +## Estimated effort + +| Phase | Effort | +|-------|--------| +| P1–P2 | 0.5–1 day | +| P3 | 1 day | +| P4 | 2–3 days | +| P5 | 0.5 day | +| **Total** | **~4–5 focused days** | diff --git a/docs/superpowers/specs/2026-07-30-raft-cluster-status.md b/docs/superpowers/specs/2026-07-30-raft-cluster-status.md index 6748992..76b4c9b 100644 --- a/docs/superpowers/specs/2026-07-30-raft-cluster-status.md +++ b/docs/superpowers/specs/2026-07-30-raft-cluster-status.md @@ -1,9 +1,38 @@ -# Raft Cluster Status — C3a / C3b / post-C3b +# Raft Cluster Status — C3a / C3b / post-C3b / v1.3.0 Date: 2026-07-30 -Status: **Shipped on `main`** (tip includes metrics). +Status: **v1.3.0 — Supported** for the single-`default`-DB scope (see the v1.3.0 section below). Branch: all work merged to `main` only (feature branch removed). +## v1.3.0 — raft-supported (2026-07-30) + +Raft moves from Experimental to **Supported** for a 3-node cluster on the +`default` database. Landed on top of the C3a/C3b base: + +- **Failover under load** — `tests/raft_failover_load_e2e_test.nim`: leader + killed under sustained writes; every acked write survives (client contract: + in-flight writes fail fast, retry). +- **Mandatory CI gate** — dedicated `raft-e2e` job runs all five raft e2e + suites; missing binary is a hard FAIL under CI. +- **Raft TLS** — `BARADB_RAFT_TLS_*` config, fail-closed startup, optional + mutual auth, TLS on follower→leader forwarding; + `tests/raft_tls_e2e_test.nim` (plaintext node excluded). +- **InstallSnapshot** — backward-compatible wire protocol, leader chunk send + (`BARADB_RAFT_SNAP_CHUNK_KB`, default 256), follower restore via + backup/restore, compaction unpinned from stale peers + (`BARADB_RAFT_PEER_STALE_MS`, default 30000); + `tests/raft_coldnode_e2e_test.nim` (returning + wiped node converge). +- **Fixes** — put/delete encoding (`deleted` flag), rejoin livelock (cached + peer sockets dropped on leadership), post-restore ctx repoint. + +Resolved non-goals from the list below: raft-port TLS, InstallSnapshot with +full SM payload. Still open: multi-database raft, `CREATE`/`DROP DATABASE` +replication, membership changes, linearizable follower reads, rolling +upgrades (restart all nodes together). + +Plan: `docs/superpowers/plans/2026-07-30-v1.3.0-raft-supported.md` · +Design: `docs/superpowers/specs/2026-07-30-raft-supported-design.md` + ## Phase map | Phase | Spec / plan | Status | What landed | diff --git a/docs/superpowers/specs/2026-07-30-raft-supported-design.md b/docs/superpowers/specs/2026-07-30-raft-supported-design.md new file mode 100644 index 0000000..8603597 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-raft-supported-design.md @@ -0,0 +1,212 @@ +# v1.3.0 Raft-Supported — Design Spec + +Date: 2026-07-30 +Status: Draft +Follows: `2026-07-30-raft-cluster-status.md` (C3a/C3b/C3c-lite shipped), +`2026-07-30-production-ga-design.md` ("After GA" section). + +## Goal + +Move the raft cluster from **experimental** to **supported** by closing the +four gaps named in the GA plan: failover under load (proven, not assumed), +CI e2e mandatory, a cold-node story, and raft-port TLS. + +Non-goals (unchanged from C3 status doc): multi-database raft, membership +change protocol, read consistency levels, `CREATE`/`DROP DATABASE` +replication. + +## Current state (verified 2026-07-30) + +- `src/barabadb/core/raft.nim` (919 lines): election, AppendEntries, + safe-prefix compaction, metrics, plain-TCP `RaftNetwork` transport. +- Wiring: `src/baradadb.nim:337-377` (env `BARADB_RAFT_*`, state in + `dataDir/raft/raft_state.bin`). +- Leader forwarding: `src/barabadb/core/server.nim:210-289` + (`forwardQueryToLeader`, plain TCP). +- TLS infra exists for the client wire port only: + `src/barabadb/protocol/ssl.nim` (`TLSConfig`, `TLSContext`, `wrapClient`, + `wrapServer`); server accept loop wraps at `core/server.nim:876-889`. +- E2E: `tests/raft_e2e_test.nim` (election + failover), + `tests/raft_writes_e2e_test.nim` (DDL/DML replication, forwarding, + failover write probe). Both run under `nimble test`, which CI runs. + +## Gap analysis + +### G1. Failover under load — unproven + +The existing failover test (`raft_writes_e2e_test.nim:349-405`) kills the +leader *while idle* and probes a single INSERT afterwards. Nothing tests a +write workload running *during* the leader crash, and nothing verifies that +every client-acknowledged write survives the failover (raft's core promise: +committed entries are never lost). + +### G2. CI e2e — present but silently skippable + +Both e2e suites `skip()` when `./build/baradadb` is missing +(`raft_writes_e2e_test.nim:413-417`). `nimble test` builds the binary first, +so CI runs them today — but a broken build step or a renamed binary turns a +raft regression into a silent green skip. There is also no dedicated CI job +that names raft e2e as a first-class gate. + +### G3. Cold node — two real failure modes + +1. **Log growth pinning.** Leader compaction + (`raft.nim:244-276`, `compactLog`) never discards past any peer's + `matchIndex`. A peer that is down keeps `matchIndex` stale, so the leader's + log grows without bound for as long as the node is down. +2. **Unrecoverable laggard.** Once the leader's log no longer contains a + follower's `nextIndex` (fresh/wiped node, or a node that was down through a + compaction), the follower rejects every AppendEntries + (`handleAppendEntries`, `raft.nim:380-394`) and the leader's + `nextIndex` decrement floor is `lastSnapshotIndex + 1` + (`handleAppendReply`, `raft.nim:526-531`). The pair is stuck forever: no + InstallSnapshot path exists. + +### G4. Raft port is plaintext + +`RaftNetwork` uses bare `newAsyncSocket()` (`raft.nim:748-765`, `857-871`). +Any host that can reach the raft port can inject RequestVote/AppendEntries +frames. The TLS machinery in `protocol/ssl.nim` is not used here; leader +forwarding (`forwardQueryToLeader`) is likewise plaintext. + +### G5. Raft write encoding loses empty-value puts (found 2026-07-30) + +`appendWriteToRaft` (`core/server.nim:309-330`) encodes an empty value as +`"delete"`, but PK-only-table inserts legitimately produce empty values +(`execInsert`, `query/exec/dml.nim:60-90`) — such inserts are acked after +majority commit and then deleted everywhere on apply. Fixed as plan Task 1a +(explicit `deleted` flag on `ExecResult.keyValuePairs`). + +## Design + +### D1. Failover-under-load E2E (test-only) + +New suite `tests/raft_failover_load_e2e_test.nim`, same process-management +conventions as `raft_writes_e2e_test.nim` (port base `50000 + tstamp mod +4000` to avoid collisions): + +1. Boot a 3-node cluster, elect a leader, create `load_test` table via raft + DDL. +2. Writer thread: sequential `INSERT INTO load_test (id) VALUES (n)`, + `n = 1, 2, ...`, recording every *acknowledged* id. On error ("not + leader", commit timeout, connection reset), probe both survivors and + resume on whichever accepts. +3. At ~50 acknowledged writes, kill the leader. +4. Assert: a survivor accepts a write within **10 s** of the kill + (availability bound). +5. Assert: after the new leader is stable and the remaining follower has + caught up, `SELECT id FROM load_test` on **both** survivors contains + **every acknowledged id** (committed writes never lost). Unacknowledged + writes may be present or absent — this is documented, not asserted. + +Also document the client-visible contract in `docs/en/distributed.md`: +in-flight writes during failover fail fast with an error; clients must +retry; acknowledged writes are durable across failover. + +### D2. CI e2e mandatory + +- New `raft-e2e` job in `.github/workflows/ci.yml`: setup Nim, build + `build/baradadb`, run the three raft e2e suites explicitly with `CI=true` + in the environment. +- Change skip semantics in all raft e2e suites: when `CI` env var is + non-empty and `./build/baradadb` is missing, **fail** instead of `skip()`. +- Add `raft_failover_load_e2e_test` to the `nimble test` list in + `baradadb.nimble`. + +### D3. Cold node — InstallSnapshot + +Extend the raft wire protocol and apply path: + +**Protocol.** New message kinds `rmkInstallSnapshot`, +`rmkInstallSnapshotReply`, and new fields on `RaftMessage`: +`snapId: uint64`, `snapOffset: uint64`, `snapData: seq[byte]`, +`snapDone: bool`. `lastSnapshotIndex`/`lastSnapshotTerm` ride on the +existing fields (`prevLogIndex`/`prevLogTerm` are reused as the snapshot +base for this kind). Serialization appends the new fields with `atEnd` +guards (same backward-compatible pattern as `loadState`, +`raft.nim:163-167`); `RaftProtoVersion` stays 1 — mixed-version clusters +simply never send the new kind (old leaders never trigger it). + +**Leader side.** Track consecutive AppendEntries rejections per peer. When +`nextIndex[peer]` has hit the `lastSnapshotIndex + 1` floor and the peer +still rejects, the peer is unrecoverably behind: + +1. Build a snapshot archive of the **default database** data dir with the + existing backup machinery (`backupDataDir` in + `src/barabadb/core/backup.nim:225`) into a temp file. +2. Stream it in chunks (`BARADB_RAFT_SNAP_CHUNK_KB`, default 256 KB) as + `rmkInstallSnapshot` messages over the existing peer socket. +3. On final ack, set `matchIndex[peer] = lastSnapshotIndex`, + `nextIndex[peer] = lastSnapshotIndex + 1`, resume normal AppendEntries. + +**Follower side.** On `rmkInstallSnapshot`: + +1. Buffer chunks to a temp file under `dataDir/raft/snap_incoming/`. +2. On `snapDone`, hand the archive to a new injected callback + `restoreSnapshot: proc(archivePath: string): bool {.gcsafe.}` (wired in + `baradadb.nim` where the `DatabaseRegistry` lives): close the default + DB, `restoreDataDir` (`backup.nim:263`) into the default DB dir, reopen, + and swap execution context. +3. Set `lastSnapshotIndex`/`lastSnapshotTerm`/`commitIndex`/`lastApplied` + from the message, clear the log, `saveState`. + +**Unpinning compaction.** Once InstallSnapshot exists, `compactLog` on the +leader compacts through `lastApplied` for peers whose `matchIndex` was +updated within the last `BARADB_RAFT_PEER_STALE_MS` (default 30 000); +long-dead peers no longer pin the log — they get a snapshot when they +return. Follower compaction is unchanged. + +**Fresh-node join** falls out for free: a wiped node rejects at the floor +and receives a snapshot. + +### D4. Raft TLS + +Config (env, mirroring existing `BARADB_TLS_*`): + +| Env | Config field | Default | +|-----|--------------|---------| +| `BARADB_RAFT_TLS_ENABLED` | `raftTlsEnabled: bool` | false | +| `BARADB_RAFT_TLS_CERT_FILE` | `raftTlsCertFile: string` | "" | +| `BARADB_RAFT_TLS_KEY_FILE` | `raftTlsKeyFile: string` | "" | +| `BARADB_RAFT_TLS_CA_FILE` | `raftTlsCaFile: string` | "" | +| `BARADB_RAFT_TLS_VERIFY_PEER` | `raftTlsVerifyPeer: bool` | false | + +- `RaftNetwork` gains `tls: TLSContext` (nil = plaintext, current + behavior). `connectToPeer` wraps with `wrapClient`; the accept loop in + `run` wraps with `wrapServer` before `receiveLoop` (same pattern as + `core/server.nim:876-889`). `verifyPeer` + CA file gives mutual auth. +- Fail closed: `raftEnabled and raftTlsEnabled` with missing cert/key → + refuse to start (raise at startup, like the JWT check in + `newServerWithRegistry`, `core/server.nim:58-63`). +- Leader SQL forwarding (`forwardQueryToLeader`) wraps its socket with the + **client** TLS context when `BARADB_TLS_ENABLED` is on (it dials the + client wire port, which is already TLS-capable). +- E2E: TLS variant cluster test — generate self-signed certs with + `generateSelfSignedCert` (`protocol/ssl.nim:79`), boot a 3-node cluster + with raft TLS on, assert election + one replicated write; assert a + plaintext peer cannot join (its frames are rejected and the cluster + elects among the TLS nodes). + +## Rollout / phases + +| Phase | Deliverable | Risk | +|-------|-------------|------| +| P1 | D1 failover-load e2e | none (test-only) | +| P2 | D2 CI e2e job | none | +| P3 | D4 raft TLS | medium (transport) | +| P4 | D3 InstallSnapshot + compaction unpin | high (protocol + apply) | + +P3 before P4 so snapshot transfer ships already-encryptable. Each phase is +independently mergeable; P4 is the v1.3.0 gate for calling raft +"supported". + +## Acceptance (v1.3.0) + +- Failover-under-load e2e green locally and in CI, in the mandatory gate. +- Killed-node-returns and wiped-node-join scenarios converge without manual + intervention (covered by new e2e phases). +- Leader log length stays bounded while a peer is down > `PEER_STALE_MS` + (assert via `baradb_raft_log_entries` metric in e2e). +- Raft port TLS: cluster runs fully over TLS; plaintext injection fails. +- `docs/en/distributed.md` + `known-limitations.md` updated: raft no longer + "experimental" for the covered scope; remaining non-goals listed.