docs: v1.3.0 raft-supported docs, limitations, runbook updates
This commit is contained in:
@@ -0,0 +1,598 @@
|
||||
# v1.3.0 Raft-Supported — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
> Spec first: `docs/superpowers/specs/2026-07-30-raft-supported-design.md`.
|
||||
|
||||
**Goal:** Move raft from experimental to supported: proven failover under
|
||||
load, mandatory CI e2e, cold-node recovery via InstallSnapshot, raft-port TLS.
|
||||
|
||||
**Architecture:** No changes to election/AppendEntries semantics. New
|
||||
InstallSnapshot message pair rides the existing framed TCP transport
|
||||
(backward-compatible trailing fields, `RaftProtoVersion` stays 1). TLS wraps
|
||||
the existing transport via `protocol/ssl.nim`. Snapshot payload reuses
|
||||
`core/backup.nim` tar.gz backup/restore.
|
||||
|
||||
**Tech stack:** Nim 2.2.x, std/asyncnet + std/net SSL, existing
|
||||
`tests/raft_*_e2e_test.nim` process harness, GitHub Actions.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Spec: `docs/superpowers/specs/2026-07-30-raft-supported-design.md`.
|
||||
- Do **not** change election safety, commit rules, or the DDL/DML
|
||||
classification from C3b/C3c.
|
||||
- Raft remains default-DB-only; snapshot payload covers the default DB only.
|
||||
- Compile all touched Nim with `-d:ssl --threads:on --path:src`.
|
||||
- Test baseline per task: `tests/test_all.nim` + `tests/bugfix_test.nim` must
|
||||
stay green; raft e2e suites green where the binary is built.
|
||||
- Branch: `main` (short-lived feature branches merged same day are fine).
|
||||
- Version bump to 1.3.0 happens only in the final task (T12).
|
||||
|
||||
---
|
||||
|
||||
## Phase map
|
||||
|
||||
| Phase | Tasks | Outcome |
|
||||
|-------|-------|---------|
|
||||
| P1 Proof | T1 | Failover-under-load e2e |
|
||||
| P2 CI | T2 | Mandatory raft e2e CI gate |
|
||||
| P3 TLS | T3–T6 | Raft port TLS + mutual auth + TLS e2e |
|
||||
| P4 Cold node | T7–T11 | InstallSnapshot + compaction unpin + cold-node e2e |
|
||||
| P5 Release | T12 | Docs, limitations, version 1.3.0 |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Failover-under-load E2E
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/raft_failover_load_e2e_test.nim`
|
||||
- Modify: `baradadb.nimble` (task `test`, line ~30-34: add `raft_failover_load_e2e_test` after `raft_writes_e2e_test`)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: process harness conventions from `tests/raft_writes_e2e_test.nim`
|
||||
(`NodeProc`, `drainOutput`, `portOpen`, `openClient`, `waitForRow` pattern);
|
||||
client `adaptors/nim/baradb_sqlite`.
|
||||
- Produces: suite `Raft failover under load E2E`.
|
||||
|
||||
**Scenario (spec D1):**
|
||||
- Port base `cbase = 50000 + (tstamp mod 4000)`, `rbase = cbase + 100`
|
||||
(distinct from 35000/41000/46000 bases already in use).
|
||||
- Boot 3 nodes with `BARADB_RAFT_*` env exactly as
|
||||
`raft_writes_e2e_test.nim:150-171`.
|
||||
- Wait for stable leader (same `maxLeader` logic). Leader DDL:
|
||||
`CREATE TABLE load_test (id INT PRIMARY KEY)`.
|
||||
- Load phase: spawn a Nim `Thread` that loops `n = 1, 2, ...`:
|
||||
`INSERT INTO load_test (id) VALUES (n)` against the current leader's
|
||||
client port; every **acknowledged** `n` appended to a
|
||||
`seq[int]` guarded by a `Lock`. On exception: reopen client against a
|
||||
survivor, continue (this models the documented client retry contract).
|
||||
- At ≥ 50 acked writes: `killNode(leader)`.
|
||||
- Assert A (availability): some survivor accepts an INSERT within 10 s of
|
||||
the kill.
|
||||
- Assert B (durability): after new leader is stable and the remaining
|
||||
follower caught up (poll `SELECT count(*)` equality or 10 s deadline),
|
||||
`SELECT id FROM load_test` on **both** survivors contains every acked id.
|
||||
- Stop the writer thread in `finally`; reuse the `dumpAll`-on-fail
|
||||
convention.
|
||||
|
||||
- [x] **Step 1:** Write the suite skeleton: harness copied from
|
||||
`raft_writes_e2e_test.nim` (drain/kill/leader-discovery helpers), writer
|
||||
thread, kill at 50 acked, asserts A + B.
|
||||
- [x] **Step 2:** Build binary and run:
|
||||
`nim c -o:build/baradadb src/baradadb.nim && nim c -d:ssl --threads:on --path:src -r tests/raft_failover_load_e2e_test.nim`
|
||||
Expected: PASS.
|
||||
- [x] **Step 3:** Run 3 consecutive times (failover timing flakiness check).
|
||||
- [x] **Step 4:** Add suite to `nimble test` list in `baradadb.nimble`.
|
||||
- [x] **Step 5:** Commit
|
||||
`test(raft): failover under sustained write load e2e`
|
||||
|
||||
---
|
||||
|
||||
### Task 1a: Fix raft put/delete encoding for empty values (bug found in T1)
|
||||
|
||||
**Bug:** `execInsert` (`src/barabadb/query/exec/dml.nim:60-90`) stores only
|
||||
non-PK columns in the value, so a PK-only table row gets `valStr = ""` and
|
||||
`kvPairs.add((fullKey, @[]))`. `appendWriteToRaft`
|
||||
(`src/barabadb/core/server.nim:309-330`) encodes an empty value as a
|
||||
`"delete"` log entry — but `execDelete` (`dml.nim:223-241`) uses the same
|
||||
`(fullKey, @[])` shape for real deletes. Result: INSERT into a PK-only
|
||||
table returns OK after majority commit, then every node (leader included,
|
||||
on apply) **deletes the row**. Verified live in T1: 30 acked inserts → 0
|
||||
rows on all nodes.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/query/exec/types.nim:139` — `ExecResult.keyValuePairs`
|
||||
- Modify: `src/barabadb/query/exec/dml.nim` — 3 producer sites (insert ~90,
|
||||
delete ~241, update ~316)
|
||||
- Modify: `src/barabadb/core/server.nim` — `appendWriteToRaft` (~309) and
|
||||
its call site (~441-464)
|
||||
- Test: `tests/bugfix_test.nim` (new suite)
|
||||
|
||||
**Interfaces:**
|
||||
- Change the pair type to carry the op explicitly:
|
||||
|
||||
```nim
|
||||
# exec/types.nim
|
||||
keyValuePairs*: seq[tuple[key: string, value: seq[byte], deleted: bool]]
|
||||
```
|
||||
|
||||
- `execInsert`/`execUpdate` produce `deleted: false` (even when
|
||||
`value.len == 0`); `execDelete` produces `deleted: true`.
|
||||
- `appendWriteToRaft` encodes `deleted` → `"delete"`, else `"put"`
|
||||
(empty value stays a put). Apply side (`baradadb.nim:358-368`) already
|
||||
handles `put` with empty value correctly — no change needed there.
|
||||
- Check other `keyValuePairs` consumers compile clean (replication path in
|
||||
`server.nim`); keep `okResult(kvPairs=...)` call sites type-correct.
|
||||
|
||||
**Steps:**
|
||||
- [x] **Step 1:** Write the failing test in `tests/bugfix_test.nim`:
|
||||
build `ExecResult` via the insert path for a PK-only table (or call
|
||||
`appendWriteToRaft` semantics directly): assert a PK-only insert yields
|
||||
a pair with `deleted == false` and encodes as `"put"`, while a delete
|
||||
yields `deleted == true` and encodes as `"delete"`.
|
||||
- [x] **Step 2:** Run, expect fail/compile error.
|
||||
- [x] **Step 3:** Implement the type + producer/consumer changes.
|
||||
- [x] **Step 4:** `bugfix_test` + `test_all` green; rebuild
|
||||
`build/baradadb` and re-run `tests/raft_failover_load_e2e_test.nim` —
|
||||
then **switch its table back** to the brief's original
|
||||
`load_test (id INT PRIMARY KEY)` / `VALUES (n)` shape (remove the
|
||||
two-column workaround and its header note) and re-run green.
|
||||
- [x] **Step 5:** Commit
|
||||
`fix(raft): distinguish put-with-empty-value from delete in write path`
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Mandatory raft e2e CI gate
|
||||
|
||||
**Files:**
|
||||
- Modify: `.github/workflows/ci.yml`
|
||||
- Modify: `tests/raft_e2e_test.nim`, `tests/raft_writes_e2e_test.nim`,
|
||||
`tests/raft_failover_load_e2e_test.nim` (skip→fail under CI)
|
||||
|
||||
**Steps:**
|
||||
- [x] **Step 1:** In each suite's binary-missing branch, replace plain
|
||||
`skip()` with:
|
||||
|
||||
```nim
|
||||
if not fileExists(BinaryPath):
|
||||
if getEnv("CI").len > 0:
|
||||
echo "[FAIL] ", BinaryPath, " missing under CI — build step broken?"
|
||||
fail()
|
||||
else:
|
||||
echo "[SKIP] ", BinaryPath, " missing — run `nimble test` (builds the server first)"
|
||||
skip()
|
||||
```
|
||||
|
||||
- [x] **Step 2:** Add a dedicated job to `.github/workflows/ci.yml` (after
|
||||
the `test` job), modeled on its setup steps:
|
||||
|
||||
```yaml
|
||||
raft-e2e:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Setup Nim
|
||||
uses: jiro4989/setup-nim-action@v1
|
||||
with:
|
||||
nim-version: '2.2.10'
|
||||
- name: Install system dependencies
|
||||
run: sudo apt-get update -qq && sudo apt-get install -y -qq libssl-dev libpcre3-dev openssl ca-certificates
|
||||
- name: Install Nim dependencies
|
||||
run: nimble install --depsOnly -y
|
||||
- name: Build server
|
||||
run: nim c -d:ssl -o:build/baradadb src/baradadb.nim
|
||||
- name: Raft e2e suites
|
||||
env:
|
||||
CI: "true"
|
||||
run: |
|
||||
nim c -d:ssl --threads:on --path:src -r tests/raft_e2e_test.nim
|
||||
nim c -d:ssl --threads:on --path:src -r tests/raft_writes_e2e_test.nim
|
||||
nim c -d:ssl --threads:on --path:src -r tests/raft_failover_load_e2e_test.nim
|
||||
```
|
||||
|
||||
- [x] **Step 3:** Push on a branch; confirm the `raft-e2e` job appears and
|
||||
is green; confirm deleting `build/` would fail (local run with `CI=true`
|
||||
and no binary → FAIL).
|
||||
- [x] **Step 4:** Commit
|
||||
`ci(raft): dedicated mandatory raft e2e job; no silent skips under CI`
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Raft TLS config + fail-closed startup
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/core/config.nim` (fields after `raftLogMaxEntries`
|
||||
at lines ~46/88; env parsing after line ~212)
|
||||
- Modify: `src/baradadb.nim` (raft wiring block, lines 337-377)
|
||||
- Test: `tests/bugfix_test.nim` (new suite)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces config fields (used by T4/T6):
|
||||
|
||||
```nim
|
||||
raftTlsEnabled*: bool # default false
|
||||
raftTlsCertFile*: string # default ""
|
||||
raftTlsKeyFile*: string # default ""
|
||||
raftTlsCaFile*: string # default ""
|
||||
raftTlsVerifyPeer*: bool # default false
|
||||
```
|
||||
|
||||
- Env parsing (mirror lines 184-212 style):
|
||||
|
||||
```nim
|
||||
cfg.raftTlsEnabled = parseEnvBool(getEnv("BARADB_RAFT_TLS_ENABLED", ""), cfg.raftTlsEnabled)
|
||||
cfg.raftTlsCertFile = getEnv("BARADB_RAFT_TLS_CERT_FILE", cfg.raftTlsCertFile)
|
||||
cfg.raftTlsKeyFile = getEnv("BARADB_RAFT_TLS_KEY_FILE", cfg.raftTlsKeyFile)
|
||||
cfg.raftTlsCaFile = getEnv("BARADB_RAFT_TLS_CA_FILE", cfg.raftTlsCaFile)
|
||||
cfg.raftTlsVerifyPeer = parseEnvBool(getEnv("BARADB_RAFT_TLS_VERIFY_PEER", ""), cfg.raftTlsVerifyPeer)
|
||||
```
|
||||
|
||||
- Fail-closed in `baradadb.nim` before `newRaftNetwork`:
|
||||
|
||||
```nim
|
||||
if config.raftTlsEnabled:
|
||||
if config.raftTlsCertFile.len == 0 or config.raftTlsKeyFile.len == 0 or
|
||||
not fileExists(config.raftTlsCertFile) or not fileExists(config.raftTlsKeyFile):
|
||||
raise newException(ValueError,
|
||||
"BARADB_RAFT_TLS_ENABLED=true but cert/key missing " &
|
||||
"(BARADB_RAFT_TLS_CERT_FILE / BARADB_RAFT_TLS_KEY_FILE)")
|
||||
```
|
||||
|
||||
- [x] **Step 1:** Write failing test in `tests/bugfix_test.nim`: default
|
||||
config has `raftTlsEnabled == false`; env `BARADB_RAFT_TLS_ENABLED=true`
|
||||
+ cert paths parse into config.
|
||||
- [x] **Step 2:** Run test, expect compile/fail (fields don't exist).
|
||||
- [x] **Step 3:** Implement fields + env parsing + startup check.
|
||||
- [x] **Step 4:** Test green; `test_all` still green.
|
||||
- [x] **Step 5:** Commit
|
||||
`feat(raft): TLS config surface with fail-closed startup`
|
||||
|
||||
---
|
||||
|
||||
### Task 4: TLS in RaftNetwork transport
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/core/raft.nim` (`RaftNetwork` type ~line 731,
|
||||
`connectToPeer` ~748, `run` ~857)
|
||||
- Modify: `src/baradadb.nim` (construct TLSContext, pass to network)
|
||||
- Test: `tests/test_all.nim` (in-process TLS raft pair)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `protocol/ssl.nim` — `newTLSConfig(certFile, keyFile, caFile,
|
||||
verifyPeer)`, `newTLSContext`, `wrapClient`, `wrapServer`.
|
||||
- Produces: `RaftNetwork.tls*: TLSContext` (nil = plaintext, unchanged
|
||||
default); `newRaftNetwork(node, tls = nil)`.
|
||||
|
||||
**Implementation:**
|
||||
|
||||
```nim
|
||||
# raft.nim — in connectToPeer, after successful connect:
|
||||
if net.tls != nil:
|
||||
try: net.tls.wrapClient(sock)
|
||||
except CatchableError:
|
||||
try: sock.close() except CatchableError: discard
|
||||
return
|
||||
|
||||
# raft.nim — in run, after accept, before receiveLoop:
|
||||
if net.tls != nil:
|
||||
try: net.tls.wrapServer(client)
|
||||
except CatchableError:
|
||||
client.close()
|
||||
continue
|
||||
```
|
||||
|
||||
- `baradadb.nim`: build the context when enabled:
|
||||
|
||||
```nim
|
||||
var raftTls: TLSContext = nil
|
||||
if config.raftTlsEnabled:
|
||||
raftTls = newTLSContext(newTLSConfig(
|
||||
config.raftTlsCertFile, config.raftTlsKeyFile,
|
||||
caFile = config.raftTlsCaFile, verifyPeer = config.raftTlsVerifyPeer))
|
||||
...
|
||||
raftNet = newRaftNetwork(raftNode, raftTls)
|
||||
```
|
||||
|
||||
- [x] **Step 1:** Write failing in-process test (`test_all.nim`): two
|
||||
`RaftNode`s over `RaftNetwork` with a self-signed cert from
|
||||
`generateSelfSignedCert` (`protocol/ssl.nim:79`) — election completes
|
||||
over TLS; plaintext dial to the TLS port produces no protocol effect
|
||||
(no state change, connection dropped).
|
||||
- [x] **Step 2:** Run, expect fail (no `tls` field).
|
||||
- [x] **Step 3:** Implement transport changes + wiring.
|
||||
- [x] **Step 4:** Test green; plaintext raft e2e suites still green
|
||||
(regression: nil-TLS path untouched).
|
||||
- [x] **Step 5:** Commit
|
||||
`feat(raft): optional TLS on raft transport (server + dialer)`
|
||||
|
||||
---
|
||||
|
||||
### Task 5: TLS for leader SQL forwarding
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/core/server.nim` (`forwardQueryToLeader`, lines
|
||||
210-289)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `protocol/ssl.nim` `wrapClient`; server config `tlsEnabled`,
|
||||
`certFile`, `keyFile`.
|
||||
- Produces: `forwardQueryToLeader(host, port, query, tls: TLSContext = nil,
|
||||
...)`.
|
||||
|
||||
- [x] **Step 1:** When the server's client wire port has TLS on
|
||||
(`server.tls != nil`), wrap the forwarding socket with a client-side
|
||||
context before sending the wire header; on handshake failure return
|
||||
`(false, QueryResult(), "leader forward TLS handshake failed")`.
|
||||
- [x] **Step 2:** Manual check: TLS server + raft forwarding (follower
|
||||
INSERT forwarded over TLS) works; non-TLS setup unchanged.
|
||||
- [x] **Step 3:** Commit
|
||||
`feat(raft): TLS on follower→leader SQL forwarding`
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Raft TLS E2E
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/raft_tls_e2e_test.nim`
|
||||
- Modify: `baradadb.nimble` (test list), `.github/workflows/ci.yml`
|
||||
(raft-e2e job: add this suite)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: T3/T4 implementation; harness from T1; openssl CLI for cert
|
||||
generation (already used by `protocol/ssl.nim`).
|
||||
|
||||
**Scenario:**
|
||||
- Port base `54000 + (tstamp mod 4000)`.
|
||||
- Generate one self-signed cert per node into the temp data dirs
|
||||
(`openssl req -x509 ...`, or `generateSelfSignedCert`).
|
||||
- Boot 3 nodes with `BARADB_RAFT_TLS_ENABLED=true` + per-node cert/key;
|
||||
assert election + `CREATE TABLE` via raft DDL + one replicated INSERT
|
||||
visible on a follower.
|
||||
- Negative: start a 4th process with raft TLS **disabled** pointed at the
|
||||
same peers; assert the TLS cluster still elects/operates among its 3
|
||||
members and the plaintext node never becomes leader (its frames are
|
||||
undecryptable).
|
||||
- Same `CI`-fail semantics as T2.
|
||||
|
||||
- [x] **Step 1:** Write suite; Step 2: run green locally; Step 3: run 3×
|
||||
(timing); Step 4: wire into `nimble test` + CI job; Step 5: Commit
|
||||
`test(raft): 3-node TLS cluster e2e with plaintext rejection`
|
||||
|
||||
---
|
||||
|
||||
### Task 7: InstallSnapshot protocol
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/core/raft.nim` (`RaftMessageKind` ~line 80,
|
||||
`RaftMessage` ~86, `serialize` ~679, `deserializeRaftMessage` ~702)
|
||||
- Test: `tests/test_all.nim` (serialize/deserialize round-trip)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
|
||||
```nim
|
||||
# RaftMessageKind += rmkInstallSnapshot, rmkInstallSnapshotReply
|
||||
# RaftMessage new fields:
|
||||
snapId*: uint64 # snapshot generation, matches leader's base at build time
|
||||
snapOffset*: uint64 # byte offset of this chunk within the archive
|
||||
snapData*: seq[byte] # chunk payload (<= snapChunkBytes)
|
||||
snapDone*: bool # last chunk
|
||||
# Reused for this kind: prevLogIndex = snapshot base index,
|
||||
# prevLogTerm = snapshot base term. Reply uses success/matchIdx as usual.
|
||||
```
|
||||
|
||||
- Serialization: append `snapId`, `snapOffset`, `snapData` (length-prefixed),
|
||||
`snapDone` **after** `matchIdx`; deserialize each with `if not s.atEnd`
|
||||
guards (pattern from `loadState`, raft.nim:163-167). Old binaries ignore
|
||||
trailing bytes; new binaries default missing fields to zero/false.
|
||||
`RaftProtoVersion` stays 1.
|
||||
|
||||
- [x] **Step 1:** Write failing round-trip test: all new fields survive
|
||||
serialize→deserialize; a buffer serialized by the *old* layout (no
|
||||
trailing fields) deserializes with zero defaults.
|
||||
- [x] **Step 2:** Run, expect fail.
|
||||
- [x] **Step 3:** Implement.
|
||||
- [x] **Step 4:** Green; existing raft suites still green.
|
||||
- [x] **Step 5:** Commit
|
||||
`feat(raft): InstallSnapshot wire protocol (backward-compatible)`
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Follower snapshot receive + restore
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/core/raft.nim` (`RaftNode` — new callback +
|
||||
incoming-snapshot buffer; `processMessage` ~786)
|
||||
- Modify: `src/barabadb/core/config.nim` (`raftSnapChunkKb: int`, env
|
||||
`BARADB_RAFT_SNAP_CHUNK_KB`, default 256; parsing next to the other
|
||||
`BARADB_RAFT_*` env reads)
|
||||
- Modify: `src/baradadb.nim` (wire `restoreSnapshot` callback using
|
||||
`core/backup.nim` + `DatabaseRegistry`; pass chunk size to the node)
|
||||
- Test: `tests/test_all.nim`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces on `RaftNode`:
|
||||
|
||||
```nim
|
||||
snapChunkBytes*: int # from BARADB_RAFT_SNAP_CHUNK_KB, default 262144
|
||||
restoreSnapshot*: proc(archivePath: string, baseIndex: uint64,
|
||||
baseTerm: uint64): bool {.gcsafe.}
|
||||
snapIncomingId*: uint64
|
||||
snapIncomingFile*: string # temp path under dataDir/raft/snap_incoming/
|
||||
```
|
||||
|
||||
- `processMessage` case `rmkInstallSnapshot`: append `snapData` at
|
||||
`snapOffset` to the temp file (create/truncate when `snapId !=
|
||||
snapIncomingId`); on `snapDone`: call `restoreSnapshot`; on success set
|
||||
`lastSnapshotIndex/Term = prevLogIndex/prevLogTerm`,
|
||||
`commitIndex = lastApplied = lastSnapshotIndex`, clear `log`,
|
||||
`saveState()`, reply success with `matchIdx = lastSnapshotIndex`; on
|
||||
failure reply `success = false` and delete the temp file.
|
||||
- `baradadb.nim` `restoreSnapshot` implementation: close default DB via
|
||||
registry, `restoreDataDir(archivePath, defaultDbDir)`
|
||||
(`backup.nim:263`), reopen, swap `ctx`. Return false on any exception.
|
||||
|
||||
- [x] **Step 1:** Write failing test: feed a node two chunks + done with a
|
||||
real tar.gz fixture; assert callback received the assembled file, state
|
||||
fields updated, log cleared.
|
||||
- [x] **Step 2:** Run, expect fail. **Step 3:** Implement.
|
||||
- [x] **Step 4:** Green + regression suites. **Step 5:** Commit
|
||||
`feat(raft): follower InstallSnapshot receive and restore`
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Leader snapshot send
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/core/raft.nim` (`handleAppendReply` floor branch
|
||||
~526-531, new `sendSnapshot` proc, per-peer reject counter)
|
||||
- Modify: `src/baradadb.nim` (wire `buildSnapshot` callback)
|
||||
- Test: `tests/test_all.nim`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: T7 protocol, `backupDataDir` (`backup.nim:225`).
|
||||
- Produces on `RaftNode`:
|
||||
|
||||
```nim
|
||||
buildSnapshot*: proc(destPath: string): bool {.gcsafe.}
|
||||
snapRejectStreak*: Table[string, int] # consecutive floor-level rejects per peer
|
||||
```
|
||||
|
||||
- Logic: in `handleAppendReply`, when a reject arrives **and**
|
||||
`nextIndex[peerId] == lastSnapshotIndex + 1` (floor reached): increment
|
||||
streak; at streak ≥ 2 the leader knows the follower needs a snapshot →
|
||||
`asyncCheck sendSnapshot(peerId)`. Reset streak on any successful reply.
|
||||
- `sendSnapshot`: `buildSnapshot` into `dataDir/raft/snap_out_<snapId>.tar.gz`
|
||||
(`snapId = lastSnapshotIndex`); stream chunks of `snapChunkBytes` as
|
||||
`rmkInstallSnapshot`; on final success reply set
|
||||
`matchIndex[peer] = lastSnapshotIndex`,
|
||||
`nextIndex[peer] = lastSnapshotIndex + 1`; delete the temp archive.
|
||||
|
||||
- [x] **Step 1:** Write failing test: leader with compacted log
|
||||
(`lastSnapshotIndex = 100`) + peer at floor rejecting twice → snapshot
|
||||
messages emitted; success reply advances `matchIndex`/`nextIndex`.
|
||||
- [x] **Step 2:** Run, expect fail. **Step 3:** Implement.
|
||||
- [x] **Step 4:** Green + regression. **Step 5:** Commit
|
||||
`feat(raft): leader InstallSnapshot send on unrecoverable lag`
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Unpin compaction from dead peers
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/barabadb/core/raft.nim` (`compactLog` ~244, `becomeLeader`
|
||||
~327, `handleAppendReply` ~487)
|
||||
- Modify: `src/barabadb/core/config.nim` (`raftPeerStaleMs`, env
|
||||
`BARADB_RAFT_PEER_STALE_MS`, default 30000)
|
||||
- Test: `tests/test_all.nim`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `matchIndexSeenMs*: Table[string, int64]` on `RaftNode` —
|
||||
monotonic ms timestamp of the last successful reply per peer, updated in
|
||||
`handleAppendReply` success branch and initialized to "now" in
|
||||
`becomeLeader`.
|
||||
|
||||
**Logic:** leader-side `compactLog` computes `minMatch` only over peers
|
||||
with `now - matchIndexSeenMs[peer] <= raftPeerStaleMs`; peers stale longer
|
||||
are excluded (they'll be snapshotted on return per T9). Follower path
|
||||
unchanged. Guard: never compact past `lastApplied`.
|
||||
|
||||
- [x] **Step 1:** Write failing test: leader, one peer never replies,
|
||||
log > maxEntries → with default stale window, log compacts through
|
||||
lastApplied anyway; with the peer responsive, compaction still pins at
|
||||
its matchIndex (existing safety preserved).
|
||||
- [x] **Step 2:** Run, expect fail. **Step 3:** Implement.
|
||||
- [x] **Step 4:** Green + regression. **Step 5:** Commit
|
||||
`feat(raft): compaction unpinned from stale peers (snapshot fallback)`
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Cold-node E2E
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/raft_coldnode_e2e_test.nim`
|
||||
- Modify: `baradadb.nimble`, `.github/workflows/ci.yml` (raft-e2e job)
|
||||
|
||||
**Interfaces:** Consumes T7–T10; harness from T1. Port base
|
||||
`58000 + (tstamp mod 4000)`. Small `BARADB_RAFT_LOG_MAX_ENTRIES=16` and
|
||||
`BARADB_RAFT_PEER_STALE_MS=3000` to force compaction quickly.
|
||||
|
||||
**Scenario A — node returns after compaction:**
|
||||
1. 3-node cluster, create table, kill node n3.
|
||||
2. Write 100 rows through the leader (forces compaction past n3's
|
||||
matchIndex once n3 is stale).
|
||||
3. Assert via `/metrics` on the leader HTTP port
|
||||
(`baradb_raft_log_entries`) that the log stayed bounded.
|
||||
4. Restart n3 with its intact data dir; assert it receives a snapshot
|
||||
(leader log line / `baradb_raft_snapshot_index` advances on n3) and
|
||||
within 15 s `SELECT count(*)` on n3 matches the leader.
|
||||
|
||||
**Scenario B — wiped node joins:**
|
||||
1. Stop n3, **delete its data dir**, restart with the same node id.
|
||||
2. Assert it converges (snapshot → catch-up) and serves the full row set
|
||||
within 20 s.
|
||||
|
||||
- [x] **Step 1:** Write suite. **Step 2:** Green locally. **Step 3:** 3×
|
||||
stability runs. **Step 4:** `nimble test` + CI wiring. **Step 5:** Commit
|
||||
`test(raft): cold-node rejoin and wiped-node join e2e`
|
||||
|
||||
---
|
||||
|
||||
### Task 12: Docs, limitations, version 1.3.0
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/en/distributed.md`, `docs/bg/distributed.md` — client
|
||||
failover contract (in-flight writes fail fast, retry; acked writes
|
||||
durable), TLS setup (`BARADB_RAFT_TLS_*`), snapshot behavior/tunables
|
||||
(`BARADB_RAFT_SNAP_CHUNK_KB`, `BARADB_RAFT_PEER_STALE_MS`)
|
||||
- Modify: `docs/en/known-limitations.md`, `docs/bg/known-limitations.md` —
|
||||
raft 3-node moves from "Experimental" to "Supported" for the covered
|
||||
scope; remaining non-goals (multi-DB raft, membership changes, read
|
||||
consistency levels) stay listed
|
||||
- Modify: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md` —
|
||||
status → v1.3.0 supported
|
||||
- Modify: `CHANGELOG.md` — `## [1.3.0] — <ship date>`
|
||||
- Modify: `baradadb.nimble` → `version = "1.3.0"`; README status lines
|
||||
- Modify: `docs/en/release-checklist.md` — add raft TLS + cold-node suites
|
||||
|
||||
- [x] **Step 1:** Doc edits. **Step 2:** Full `nimble test` green.
|
||||
- [x] **Step 3:** Commit
|
||||
`release: v1.3.0 raft-supported (failover load, CI gate, snapshot, TLS)`
|
||||
- [ ] **Step 4 (human/controller):** tag `v1.3.0` after review.
|
||||
|
||||
---
|
||||
|
||||
## Task dependency graph
|
||||
|
||||
```
|
||||
T1 failover-load e2e ──→ T2 CI gate
|
||||
T3 TLS config ──→ T4 transport TLS ──→ T5 forward TLS ──→ T6 TLS e2e
|
||||
T7 snapshot protocol ──→ T8 follower restore ──→ T9 leader send ──→ T10 unpin ──→ T11 cold-node e2e
|
||||
all ──→ T12 docs/version
|
||||
```
|
||||
|
||||
## Explicit out-of-scope
|
||||
|
||||
- Membership change (join/leave) protocol
|
||||
- Multi-database raft; `CREATE`/`DROP DATABASE` replication
|
||||
- Linearizable follower reads
|
||||
- Rolling-upgrade compat shims beyond the trailing-field guard
|
||||
(upgrade = restart all nodes)
|
||||
|
||||
## Definition of Done
|
||||
|
||||
- [x] All P1–P5 tasks complete, `nimble test` green
|
||||
- [ ] `raft-e2e` CI job green and mandatory (no silent skip)
|
||||
- [x] Failover-under-load e2e: every acked write survives leader kill
|
||||
- [x] Cold-node e2e: returning node and wiped node converge automatically
|
||||
- [x] Raft TLS e2e: full-TLS cluster works; plaintext node excluded
|
||||
- [x] known-limitations updated: raft supported for the covered scope
|
||||
|
||||
## Estimated effort
|
||||
|
||||
| Phase | Effort |
|
||||
|-------|--------|
|
||||
| P1–P2 | 0.5–1 day |
|
||||
| P3 | 1 day |
|
||||
| P4 | 2–3 days |
|
||||
| P5 | 0.5 day |
|
||||
| **Total** | **~4–5 focused days** |
|
||||
Reference in New Issue
Block a user