Files

T

dimgigov d303cc5658 docs: v1.3.0 raft-supported docs, limitations, runbook updates

2026-07-31 04:16:11 +03:00

24 KiB

Raw Blame History

v1.3.0 Raft-Supported — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking. Spec first: docs/superpowers/specs/2026-07-30-raft-supported-design.md.

Goal: Move raft from experimental to supported: proven failover under load, mandatory CI e2e, cold-node recovery via InstallSnapshot, raft-port TLS.

Architecture: No changes to election/AppendEntries semantics. New InstallSnapshot message pair rides the existing framed TCP transport (backward-compatible trailing fields, RaftProtoVersion stays 1). TLS wraps the existing transport via protocol/ssl.nim. Snapshot payload reuses core/backup.nim tar.gz backup/restore.

Tech stack: Nim 2.2.x, std/asyncnet + std/net SSL, existing tests/raft_*_e2e_test.nim process harness, GitHub Actions.

Global Constraints

Spec: docs/superpowers/specs/2026-07-30-raft-supported-design.md.
Do not change election safety, commit rules, or the DDL/DML classification from C3b/C3c.
Raft remains default-DB-only; snapshot payload covers the default DB only.
Compile all touched Nim with -d:ssl --threads:on --path:src.
Test baseline per task: tests/test_all.nim + tests/bugfix_test.nim must stay green; raft e2e suites green where the binary is built.
Branch: main (short-lived feature branches merged same day are fine).
Version bump to 1.3.0 happens only in the final task (T12).

Phase map

Phase	Tasks	Outcome
P1 Proof	T1	Failover-under-load e2e
P2 CI	T2	Mandatory raft e2e CI gate
P3 TLS	T3–T6	Raft port TLS + mutual auth + TLS e2e
P4 Cold node	T7–T11	InstallSnapshot + compaction unpin + cold-node e2e
P5 Release	T12	Docs, limitations, version 1.3.0

Task 1: Failover-under-load E2E

Files:

Create: tests/raft_failover_load_e2e_test.nim
Modify: baradadb.nimble (task test, line ~30-34: add raft_failover_load_e2e_test after raft_writes_e2e_test)

Interfaces:

Consumes: process harness conventions from tests/raft_writes_e2e_test.nim (NodeProc, drainOutput, portOpen, openClient, waitForRow pattern); client adaptors/nim/baradb_sqlite.
Produces: suite Raft failover under load E2E.

Scenario (spec D1):

Port base cbase = 50000 + (tstamp mod 4000), rbase = cbase + 100 (distinct from 35000/41000/46000 bases already in use).
Boot 3 nodes with BARADB_RAFT_* env exactly as raft_writes_e2e_test.nim:150-171.
Wait for stable leader (same maxLeader logic). Leader DDL: CREATE TABLE load_test (id INT PRIMARY KEY).
Load phase: spawn a Nim Thread that loops n = 1, 2, ...: INSERT INTO load_test (id) VALUES (n) against the current leader's client port; every acknowledged n appended to a seq[int] guarded by a Lock. On exception: reopen client against a survivor, continue (this models the documented client retry contract).
At ≥ 50 acked writes: killNode(leader).
Assert A (availability): some survivor accepts an INSERT within 10 s of the kill.
Assert B (durability): after new leader is stable and the remaining follower caught up (poll SELECT count(*) equality or 10 s deadline), SELECT id FROM load_test on both survivors contains every acked id.
Stop the writer thread in finally; reuse the dumpAll-on-fail convention.
Step 1: Write the suite skeleton: harness copied from raft_writes_e2e_test.nim (drain/kill/leader-discovery helpers), writer thread, kill at 50 acked, asserts A + B.
Step 2: Build binary and run: nim c -o:build/baradadb src/baradadb.nim && nim c -d:ssl --threads:on --path:src -r tests/raft_failover_load_e2e_test.nim Expected: PASS.
Step 3: Run 3 consecutive times (failover timing flakiness check).
Step 4: Add suite to nimble test list in baradadb.nimble.
Step 5: Commit test(raft): failover under sustained write load e2e

Task 1a: Fix raft put/delete encoding for empty values (bug found in T1)

Bug: execInsert (src/barabadb/query/exec/dml.nim:60-90) stores only non-PK columns in the value, so a PK-only table row gets valStr = "" and kvPairs.add((fullKey, @[])). appendWriteToRaft (src/barabadb/core/server.nim:309-330) encodes an empty value as a "delete" log entry — but execDelete (dml.nim:223-241) uses the same (fullKey, @[]) shape for real deletes. Result: INSERT into a PK-only table returns OK after majority commit, then every node (leader included, on apply) deletes the row. Verified live in T1: 30 acked inserts → 0 rows on all nodes.

Files:

Modify: src/barabadb/query/exec/types.nim:139 — ExecResult.keyValuePairs
Modify: src/barabadb/query/exec/dml.nim — 3 producer sites (insert ~90, delete ~241, update ~316)
Modify: src/barabadb/core/server.nim — appendWriteToRaft (~309) and its call site (~441-464)
Test: tests/bugfix_test.nim (new suite)

Interfaces:

Change the pair type to carry the op explicitly:

# exec/types.nim
keyValuePairs*: seq[tuple[key: string, value: seq[byte], deleted: bool]]

execInsert/execUpdate produce deleted: false (even when value.len == 0); execDelete produces deleted: true.
appendWriteToRaft encodes deleted → "delete", else "put" (empty value stays a put). Apply side (baradadb.nim:358-368) already handles put with empty value correctly — no change needed there.
Check other keyValuePairs consumers compile clean (replication path in server.nim); keep okResult(kvPairs=...) call sites type-correct.

Steps:

Step 1: Write the failing test in tests/bugfix_test.nim: build ExecResult via the insert path for a PK-only table (or call appendWriteToRaft semantics directly): assert a PK-only insert yields a pair with deleted == false and encodes as "put", while a delete yields deleted == true and encodes as "delete".
Step 2: Run, expect fail/compile error.
Step 3: Implement the type + producer/consumer changes.
Step 4: bugfix_test + test_all green; rebuild build/baradadb and re-run tests/raft_failover_load_e2e_test.nim — then switch its table back to the brief's original load_test (id INT PRIMARY KEY) / VALUES (n) shape (remove the two-column workaround and its header note) and re-run green.
Step 5: Commit fix(raft): distinguish put-with-empty-value from delete in write path

Task 2: Mandatory raft e2e CI gate

Files:

Modify: .github/workflows/ci.yml
Modify: tests/raft_e2e_test.nim, tests/raft_writes_e2e_test.nim, tests/raft_failover_load_e2e_test.nim (skip→fail under CI)

Steps:

Step 1: In each suite's binary-missing branch, replace plain skip() with:

if not fileExists(BinaryPath):
  if getEnv("CI").len > 0:
    echo "[FAIL] ", BinaryPath, " missing under CI — build step broken?"
    fail()
  else:
    echo "[SKIP] ", BinaryPath, " missing — run `nimble test` (builds the server first)"
    skip()

Step 2: Add a dedicated job to .github/workflows/ci.yml (after the test job), modeled on its setup steps:

  raft-e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup Nim
        uses: jiro4989/setup-nim-action@v1
        with:
          nim-version: '2.2.10'
      - name: Install system dependencies
        run: sudo apt-get update -qq && sudo apt-get install -y -qq libssl-dev libpcre3-dev openssl ca-certificates
      - name: Install Nim dependencies
        run: nimble install --depsOnly -y
      - name: Build server
        run: nim c -d:ssl -o:build/baradadb src/baradadb.nim
      - name: Raft e2e suites
        env:
          CI: "true"
        run: |
          nim c -d:ssl --threads:on --path:src -r tests/raft_e2e_test.nim
          nim c -d:ssl --threads:on --path:src -r tests/raft_writes_e2e_test.nim
          nim c -d:ssl --threads:on --path:src -r tests/raft_failover_load_e2e_test.nim

Step 3: Push on a branch; confirm the raft-e2e job appears and is green; confirm deleting build/ would fail (local run with CI=true and no binary → FAIL).
Step 4: Commit ci(raft): dedicated mandatory raft e2e job; no silent skips under CI

Task 3: Raft TLS config + fail-closed startup

Files:

Modify: src/barabadb/core/config.nim (fields after raftLogMaxEntries at lines ~46/88; env parsing after line ~212)
Modify: src/baradadb.nim (raft wiring block, lines 337-377)
Test: tests/bugfix_test.nim (new suite)

Interfaces:

Produces config fields (used by T4/T6):

raftTlsEnabled*: bool       # default false
raftTlsCertFile*: string    # default ""
raftTlsKeyFile*: string     # default ""
raftTlsCaFile*: string      # default ""
raftTlsVerifyPeer*: bool    # default false

Env parsing (mirror lines 184-212 style):

cfg.raftTlsEnabled = parseEnvBool(getEnv("BARADB_RAFT_TLS_ENABLED", ""), cfg.raftTlsEnabled)
cfg.raftTlsCertFile = getEnv("BARADB_RAFT_TLS_CERT_FILE", cfg.raftTlsCertFile)
cfg.raftTlsKeyFile = getEnv("BARADB_RAFT_TLS_KEY_FILE", cfg.raftTlsKeyFile)
cfg.raftTlsCaFile = getEnv("BARADB_RAFT_TLS_CA_FILE", cfg.raftTlsCaFile)
cfg.raftTlsVerifyPeer = parseEnvBool(getEnv("BARADB_RAFT_TLS_VERIFY_PEER", ""), cfg.raftTlsVerifyPeer)

Fail-closed in baradadb.nim before newRaftNetwork:

if config.raftTlsEnabled:
  if config.raftTlsCertFile.len == 0 or config.raftTlsKeyFile.len == 0 or
     not fileExists(config.raftTlsCertFile) or not fileExists(config.raftTlsKeyFile):
    raise newException(ValueError,
      "BARADB_RAFT_TLS_ENABLED=true but cert/key missing " &
      "(BARADB_RAFT_TLS_CERT_FILE / BARADB_RAFT_TLS_KEY_FILE)")

Step 1: Write failing test in tests/bugfix_test.nim: default config has raftTlsEnabled == false; env BARADB_RAFT_TLS_ENABLED=true
- cert paths parse into config.
Step 2: Run test, expect compile/fail (fields don't exist).
Step 3: Implement fields + env parsing + startup check.
Step 4: Test green; test_all still green.
Step 5: Commit feat(raft): TLS config surface with fail-closed startup

Task 4: TLS in RaftNetwork transport

Files:

Modify: src/barabadb/core/raft.nim (RaftNetwork type ~line 731, connectToPeer ~748, run ~857)
Modify: src/baradadb.nim (construct TLSContext, pass to network)
Test: tests/test_all.nim (in-process TLS raft pair)

Interfaces:

Consumes: protocol/ssl.nim — newTLSConfig(certFile, keyFile, caFile, verifyPeer), newTLSContext, wrapClient, wrapServer.
Produces: RaftNetwork.tls*: TLSContext (nil = plaintext, unchanged default); newRaftNetwork(node, tls = nil).

Implementation:

# raft.nim — in connectToPeer, after successful connect:
if net.tls != nil:
  try: net.tls.wrapClient(sock)
  except CatchableError:
    try: sock.close() except CatchableError: discard
    return

# raft.nim — in run, after accept, before receiveLoop:
if net.tls != nil:
  try: net.tls.wrapServer(client)
  except CatchableError:
    client.close()
    continue

baradadb.nim: build the context when enabled:

var raftTls: TLSContext = nil
if config.raftTlsEnabled:
  raftTls = newTLSContext(newTLSConfig(
    config.raftTlsCertFile, config.raftTlsKeyFile,
    caFile = config.raftTlsCaFile, verifyPeer = config.raftTlsVerifyPeer))
...
raftNet = newRaftNetwork(raftNode, raftTls)

Step 1: Write failing in-process test (test_all.nim): two RaftNodes over RaftNetwork with a self-signed cert from generateSelfSignedCert (protocol/ssl.nim:79) — election completes over TLS; plaintext dial to the TLS port produces no protocol effect (no state change, connection dropped).
Step 2: Run, expect fail (no tls field).
Step 3: Implement transport changes + wiring.
Step 4: Test green; plaintext raft e2e suites still green (regression: nil-TLS path untouched).
Step 5: Commit feat(raft): optional TLS on raft transport (server + dialer)

Task 5: TLS for leader SQL forwarding

Files:

Modify: src/barabadb/core/server.nim (forwardQueryToLeader, lines 210-289)

Interfaces:

Consumes: protocol/ssl.nim wrapClient; server config tlsEnabled, certFile, keyFile.
Produces: forwardQueryToLeader(host, port, query, tls: TLSContext = nil, ...).
Step 1: When the server's client wire port has TLS on (server.tls != nil), wrap the forwarding socket with a client-side context before sending the wire header; on handshake failure return (false, QueryResult(), "leader forward TLS handshake failed").
Step 2: Manual check: TLS server + raft forwarding (follower INSERT forwarded over TLS) works; non-TLS setup unchanged.
Step 3: Commit feat(raft): TLS on follower→leader SQL forwarding

Task 6: Raft TLS E2E

Files:

Create: tests/raft_tls_e2e_test.nim
Modify: baradadb.nimble (test list), .github/workflows/ci.yml (raft-e2e job: add this suite)

Interfaces:

Consumes: T3/T4 implementation; harness from T1; openssl CLI for cert generation (already used by protocol/ssl.nim).

Scenario:

Port base 54000 + (tstamp mod 4000).
Generate one self-signed cert per node into the temp data dirs (openssl req -x509 ..., or generateSelfSignedCert).
Boot 3 nodes with BARADB_RAFT_TLS_ENABLED=true + per-node cert/key; assert election + CREATE TABLE via raft DDL + one replicated INSERT visible on a follower.
Negative: start a 4th process with raft TLS disabled pointed at the same peers; assert the TLS cluster still elects/operates among its 3 members and the plaintext node never becomes leader (its frames are undecryptable).
Same CI-fail semantics as T2.
Step 1: Write suite; Step 2: run green locally; Step 3: run 3× (timing); Step 4: wire into nimble test + CI job; Step 5: Commit test(raft): 3-node TLS cluster e2e with plaintext rejection

Task 7: InstallSnapshot protocol

Files:

Modify: src/barabadb/core/raft.nim (RaftMessageKind ~line 80, RaftMessage ~86, serialize ~679, deserializeRaftMessage ~702)
Test: tests/test_all.nim (serialize/deserialize round-trip)

Interfaces:

Produces:

# RaftMessageKind += rmkInstallSnapshot, rmkInstallSnapshotReply
# RaftMessage new fields:
snapId*: uint64      # snapshot generation, matches leader's base at build time
snapOffset*: uint64  # byte offset of this chunk within the archive
snapData*: seq[byte] # chunk payload (<= snapChunkBytes)
snapDone*: bool      # last chunk
# Reused for this kind: prevLogIndex = snapshot base index,
# prevLogTerm = snapshot base term. Reply uses success/matchIdx as usual.

Serialization: append snapId, snapOffset, snapData (length-prefixed), snapDone after matchIdx; deserialize each with if not s.atEnd guards (pattern from loadState, raft.nim:163-167). Old binaries ignore trailing bytes; new binaries default missing fields to zero/false. RaftProtoVersion stays 1.
Step 1: Write failing round-trip test: all new fields survive serialize→deserialize; a buffer serialized by the old layout (no trailing fields) deserializes with zero defaults.
Step 2: Run, expect fail.
Step 3: Implement.
Step 4: Green; existing raft suites still green.
Step 5: Commit feat(raft): InstallSnapshot wire protocol (backward-compatible)

Task 8: Follower snapshot receive + restore

Files:

Modify: src/barabadb/core/raft.nim (RaftNode — new callback + incoming-snapshot buffer; processMessage ~786)
Modify: src/barabadb/core/config.nim (raftSnapChunkKb: int, env BARADB_RAFT_SNAP_CHUNK_KB, default 256; parsing next to the other BARADB_RAFT_* env reads)
Modify: src/baradadb.nim (wire restoreSnapshot callback using core/backup.nim + DatabaseRegistry; pass chunk size to the node)
Test: tests/test_all.nim

Interfaces:

Produces on RaftNode:

snapChunkBytes*: int          # from BARADB_RAFT_SNAP_CHUNK_KB, default 262144
restoreSnapshot*: proc(archivePath: string, baseIndex: uint64,
                       baseTerm: uint64): bool {.gcsafe.}
snapIncomingId*: uint64
snapIncomingFile*: string     # temp path under dataDir/raft/snap_incoming/

processMessage case rmkInstallSnapshot: append snapData at snapOffset to the temp file (create/truncate when snapId != snapIncomingId); on snapDone: call restoreSnapshot; on success set lastSnapshotIndex/Term = prevLogIndex/prevLogTerm, commitIndex = lastApplied = lastSnapshotIndex, clear log, saveState(), reply success with matchIdx = lastSnapshotIndex; on failure reply success = false and delete the temp file.
baradadb.nim restoreSnapshot implementation: close default DB via registry, restoreDataDir(archivePath, defaultDbDir) (backup.nim:263), reopen, swap ctx. Return false on any exception.
Step 1: Write failing test: feed a node two chunks + done with a real tar.gz fixture; assert callback received the assembled file, state fields updated, log cleared.
Step 2: Run, expect fail. Step 3: Implement.
Step 4: Green + regression suites. Step 5: Commit feat(raft): follower InstallSnapshot receive and restore

Task 9: Leader snapshot send

Files:

Modify: src/barabadb/core/raft.nim (handleAppendReply floor branch ~526-531, new sendSnapshot proc, per-peer reject counter)
Modify: src/baradadb.nim (wire buildSnapshot callback)
Test: tests/test_all.nim

Interfaces:

Consumes: T7 protocol, backupDataDir (backup.nim:225).
Produces on RaftNode:

buildSnapshot*: proc(destPath: string): bool {.gcsafe.}
snapRejectStreak*: Table[string, int]   # consecutive floor-level rejects per peer

Logic: in handleAppendReply, when a reject arrives and nextIndex[peerId] == lastSnapshotIndex + 1 (floor reached): increment streak; at streak ≥ 2 the leader knows the follower needs a snapshot → asyncCheck sendSnapshot(peerId). Reset streak on any successful reply.
sendSnapshot: buildSnapshot into dataDir/raft/snap_out_<snapId>.tar.gz (snapId = lastSnapshotIndex); stream chunks of snapChunkBytes as rmkInstallSnapshot; on final success reply set matchIndex[peer] = lastSnapshotIndex, nextIndex[peer] = lastSnapshotIndex + 1; delete the temp archive.
Step 1: Write failing test: leader with compacted log (lastSnapshotIndex = 100) + peer at floor rejecting twice → snapshot messages emitted; success reply advances matchIndex/nextIndex.
Step 2: Run, expect fail. Step 3: Implement.
Step 4: Green + regression. Step 5: Commit feat(raft): leader InstallSnapshot send on unrecoverable lag

Task 10: Unpin compaction from dead peers

Files:

Modify: src/barabadb/core/raft.nim (compactLog ~244, becomeLeader ~327, handleAppendReply ~487)
Modify: src/barabadb/core/config.nim (raftPeerStaleMs, env BARADB_RAFT_PEER_STALE_MS, default 30000)
Test: tests/test_all.nim

Interfaces:

Produces: matchIndexSeenMs*: Table[string, int64] on RaftNode — monotonic ms timestamp of the last successful reply per peer, updated in handleAppendReply success branch and initialized to "now" in becomeLeader.

Logic: leader-side compactLog computes minMatch only over peers with now - matchIndexSeenMs[peer] <= raftPeerStaleMs; peers stale longer are excluded (they'll be snapshotted on return per T9). Follower path unchanged. Guard: never compact past lastApplied.

Step 1: Write failing test: leader, one peer never replies, log > maxEntries → with default stale window, log compacts through lastApplied anyway; with the peer responsive, compaction still pins at its matchIndex (existing safety preserved).
Step 2: Run, expect fail. Step 3: Implement.
Step 4: Green + regression. Step 5: Commit feat(raft): compaction unpinned from stale peers (snapshot fallback)

Task 11: Cold-node E2E

Files:

Create: tests/raft_coldnode_e2e_test.nim
Modify: baradadb.nimble, .github/workflows/ci.yml (raft-e2e job)

Interfaces: Consumes T7–T10; harness from T1. Port base 58000 + (tstamp mod 4000). Small BARADB_RAFT_LOG_MAX_ENTRIES=16 and BARADB_RAFT_PEER_STALE_MS=3000 to force compaction quickly.

Scenario A — node returns after compaction:

3-node cluster, create table, kill node n3.
Write 100 rows through the leader (forces compaction past n3's matchIndex once n3 is stale).
Assert via /metrics on the leader HTTP port (baradb_raft_log_entries) that the log stayed bounded.
Restart n3 with its intact data dir; assert it receives a snapshot (leader log line / baradb_raft_snapshot_index advances on n3) and within 15 s SELECT count(*) on n3 matches the leader.

Scenario B — wiped node joins:

Stop n3, delete its data dir, restart with the same node id.
Assert it converges (snapshot → catch-up) and serves the full row set within 20 s.

Step 1: Write suite. Step 2: Green locally. Step 3: 3× stability runs. Step 4: nimble test + CI wiring. Step 5: Commit test(raft): cold-node rejoin and wiped-node join e2e

Task 12: Docs, limitations, version 1.3.0

Files:

Modify: docs/en/distributed.md, docs/bg/distributed.md — client failover contract (in-flight writes fail fast, retry; acked writes durable), TLS setup (BARADB_RAFT_TLS_*), snapshot behavior/tunables (BARADB_RAFT_SNAP_CHUNK_KB, BARADB_RAFT_PEER_STALE_MS)
Modify: docs/en/known-limitations.md, docs/bg/known-limitations.md — raft 3-node moves from "Experimental" to "Supported" for the covered scope; remaining non-goals (multi-DB raft, membership changes, read consistency levels) stay listed
Modify: docs/superpowers/specs/2026-07-30-raft-cluster-status.md — status → v1.3.0 supported
Modify: CHANGELOG.md — ## [1.3.0] — <ship date>
Modify: baradadb.nimble → version = "1.3.0"; README status lines
Modify: docs/en/release-checklist.md — add raft TLS + cold-node suites
Step 1: Doc edits. Step 2: Full nimble test green.
Step 3: Commit release: v1.3.0 raft-supported (failover load, CI gate, snapshot, TLS)
Step 4 (human/controller): tag v1.3.0 after review.

Task dependency graph

T1 failover-load e2e ──→ T2 CI gate
T3 TLS config ──→ T4 transport TLS ──→ T5 forward TLS ──→ T6 TLS e2e
T7 snapshot protocol ──→ T8 follower restore ──→ T9 leader send ──→ T10 unpin ──→ T11 cold-node e2e
all ──→ T12 docs/version

Explicit out-of-scope

Membership change (join/leave) protocol
Multi-database raft; CREATE/DROP DATABASE replication
Linearizable follower reads
Rolling-upgrade compat shims beyond the trailing-field guard (upgrade = restart all nodes)

Definition of Done

All P1–P5 tasks complete, nimble test green
raft-e2e CI job green and mandatory (no silent skip)
Failover-under-load e2e: every acked write survives leader kill
Cold-node e2e: returning node and wiped node converge automatically
Raft TLS e2e: full-TLS cluster works; plaintext node excluded
known-limitations updated: raft supported for the covered scope

Estimated effort

Phase	Effort
P1–P2	0.5–1 day
P3	1 day
P4	2–3 days
P5	0.5 day
Total	~4–5 focused days

24 KiB Raw Blame History Unescape Escape

v1.3.0 Raft-Supported — Implementation Plan

Global Constraints

Phase map

Task 1: Failover-under-load E2E

Task 1a: Fix raft put/delete encoding for empty values (bug found in T1)

Task 2: Mandatory raft e2e CI gate

Task 3: Raft TLS config + fail-closed startup

Task 4: TLS in RaftNetwork transport

Task 5: TLS for leader SQL forwarding

Task 6: Raft TLS E2E

Task 7: InstallSnapshot protocol

Task 8: Follower snapshot receive + restore

Task 9: Leader snapshot send

Task 10: Unpin compaction from dead peers

Task 11: Cold-node E2E

Task 12: Docs, limitations, version 1.3.0

Task dependency graph

Explicit out-of-scope

Definition of Done

Estimated effort

24 KiB

Raw Blame History