Baradb/docs/superpowers/specs/2026-07-30-raft-supported-design.md

v1.3.0 Raft-Supported — Design Spec

Date: 2026-07-30 Status: Draft Follows: 2026-07-30-raft-cluster-status.md (C3a/C3b/C3c-lite shipped), 2026-07-30-production-ga-design.md ("After GA" section).

Goal

Move the raft cluster from experimental to supported by closing the four gaps named in the GA plan: failover under load (proven, not assumed), CI e2e mandatory, a cold-node story, and raft-port TLS.

Non-goals (unchanged from C3 status doc): multi-database raft, membership change protocol, read consistency levels, CREATE/DROP DATABASE replication.

Current state (verified 2026-07-30)

• src/barabadb/core/raft.nim (919 lines): election, AppendEntries, safe-prefix compaction, metrics, plain-TCP RaftNetwork transport.
• Wiring: src/baradadb.nim:337-377 (env BARADB_RAFT_*, state in dataDir/raft/raft_state.bin).
• Leader forwarding: src/barabadb/core/server.nim:210-289 (forwardQueryToLeader, plain TCP).
• TLS infra exists for the client wire port only: src/barabadb/protocol/ssl.nim (TLSConfig, TLSContext, wrapClient, wrapServer); server accept loop wraps at core/server.nim:876-889.
• E2E: tests/raft_e2e_test.nim (election + failover), tests/raft_writes_e2e_test.nim (DDL/DML replication, forwarding, failover write probe). Both run under nimble test, which CI runs.

Gap analysis

G1. Failover under load — unproven

The existing failover test (raft_writes_e2e_test.nim:349-405) kills the leader while idle and probes a single INSERT afterwards. Nothing tests a write workload running during the leader crash, and nothing verifies that every client-acknowledged write survives the failover (raft's core promise: committed entries are never lost).

G2. CI e2e — present but silently skippable

Both e2e suites skip() when ./build/baradadb is missing (raft_writes_e2e_test.nim:413-417). nimble test builds the binary first, so CI runs them today — but a broken build step or a renamed binary turns a raft regression into a silent green skip. There is also no dedicated CI job that names raft e2e as a first-class gate.

G3. Cold node — two real failure modes

1. Log growth pinning. Leader compaction (raft.nim:244-276, compactLog) never discards past any peer's matchIndex. A peer that is down keeps matchIndex stale, so the leader's log grows without bound for as long as the node is down.
2. Unrecoverable laggard. Once the leader's log no longer contains a follower's nextIndex (fresh/wiped node, or a node that was down through a compaction), the follower rejects every AppendEntries (handleAppendEntries, raft.nim:380-394) and the leader's nextIndex decrement floor is lastSnapshotIndex + 1 (handleAppendReply, raft.nim:526-531). The pair is stuck forever: no InstallSnapshot path exists.

G4. Raft port is plaintext

RaftNetwork uses bare newAsyncSocket() (raft.nim:748-765, 857-871). Any host that can reach the raft port can inject RequestVote/AppendEntries frames. The TLS machinery in protocol/ssl.nim is not used here; leader forwarding (forwardQueryToLeader) is likewise plaintext.

G5. Raft write encoding loses empty-value puts (found 2026-07-30)

appendWriteToRaft (core/server.nim:309-330) encodes an empty value as "delete", but PK-only-table inserts legitimately produce empty values (execInsert, query/exec/dml.nim:60-90) — such inserts are acked after majority commit and then deleted everywhere on apply. Fixed as plan Task 1a (explicit deleted flag on ExecResult.keyValuePairs).

Design

D1. Failover-under-load E2E (test-only)

New suite tests/raft_failover_load_e2e_test.nim, same process-management conventions as raft_writes_e2e_test.nim (port base 50000 + tstamp mod 4000 to avoid collisions):

1. Boot a 3-node cluster, elect a leader, create load_test table via raft DDL.
2. Writer thread: sequential INSERT INTO load_test (id) VALUES (n), n = 1, 2, ..., recording every acknowledged id. On error ("not leader", commit timeout, connection reset), probe both survivors and resume on whichever accepts.
3. At ~50 acknowledged writes, kill the leader.
4. Assert: a survivor accepts a write within 10 s of the kill (availability bound).
5. Assert: after the new leader is stable and the remaining follower has caught up, SELECT id FROM load_test on both survivors contains every acknowledged id (committed writes never lost). Unacknowledged writes may be present or absent — this is documented, not asserted.

Also document the client-visible contract in docs/en/distributed.md: in-flight writes during failover fail fast with an error; clients must retry; acknowledged writes are durable across failover.

D2. CI e2e mandatory

• New raft-e2e job in .github/workflows/ci.yml: setup Nim, build build/baradadb, run the three raft e2e suites explicitly with CI=true in the environment.
• Change skip semantics in all raft e2e suites: when CI env var is non-empty and ./build/baradadb is missing, fail instead of skip().
• Add raft_failover_load_e2e_test to the nimble test list in baradadb.nimble.

D3. Cold node — InstallSnapshot

Extend the raft wire protocol and apply path:

Protocol. New message kinds rmkInstallSnapshot, rmkInstallSnapshotReply, and new fields on RaftMessage: snapId: uint64, snapOffset: uint64, snapData: seq[byte], snapDone: bool. lastSnapshotIndex/lastSnapshotTerm ride on the existing fields (prevLogIndex/prevLogTerm are reused as the snapshot base for this kind). Serialization appends the new fields with atEnd guards (same backward-compatible pattern as loadState, raft.nim:163-167); RaftProtoVersion stays 1 — mixed-version clusters simply never send the new kind (old leaders never trigger it).

Leader side. Track consecutive AppendEntries rejections per peer. When nextIndex[peer] has hit the lastSnapshotIndex + 1 floor and the peer still rejects, the peer is unrecoverably behind:

1. Build a snapshot archive of the default database data dir with the existing backup machinery (backupDataDir in src/barabadb/core/backup.nim:225) into a temp file.
2. Stream it in chunks (BARADB_RAFT_SNAP_CHUNK_KB, default 256 KB) as rmkInstallSnapshot messages over the existing peer socket.
3. On final ack, set matchIndex[peer] = lastSnapshotIndex, nextIndex[peer] = lastSnapshotIndex + 1, resume normal AppendEntries.

Follower side. On rmkInstallSnapshot:

1. Buffer chunks to a temp file under dataDir/raft/snap_incoming/.
2. On snapDone, hand the archive to a new injected callback restoreSnapshot: proc(archivePath: string): bool {.gcsafe.} (wired in baradadb.nim where the DatabaseRegistry lives): close the default DB, restoreDataDir (backup.nim:263) into the default DB dir, reopen, and swap execution context.
3. Set lastSnapshotIndex/lastSnapshotTerm/commitIndex/lastApplied from the message, clear the log, saveState.

Unpinning compaction. Once InstallSnapshot exists, compactLog on the leader compacts through lastApplied for peers whose matchIndex was updated within the last BARADB_RAFT_PEER_STALE_MS (default 30 000); long-dead peers no longer pin the log — they get a snapshot when they return. Follower compaction is unchanged.

Fresh-node join falls out for free: a wiped node rejects at the floor and receives a snapshot.

D4. Raft TLS

Config (env, mirroring existing BARADB_TLS_*):

Env	Config field	Default
BARADB_RAFT_TLS_ENABLED	raftTlsEnabled: bool	false
BARADB_RAFT_TLS_CERT_FILE	raftTlsCertFile: string	""
BARADB_RAFT_TLS_KEY_FILE	raftTlsKeyFile: string	""
BARADB_RAFT_TLS_CA_FILE	raftTlsCaFile: string	""
BARADB_RAFT_TLS_VERIFY_PEER	raftTlsVerifyPeer: bool	false

• RaftNetwork gains tls: TLSContext (nil = plaintext, current behavior). connectToPeer wraps with wrapClient; the accept loop in run wraps with wrapServer before receiveLoop (same pattern as core/server.nim:876-889). verifyPeer + CA file gives mutual auth.
• Fail closed: raftEnabled and raftTlsEnabled with missing cert/key → refuse to start (raise at startup, like the JWT check in newServerWithRegistry, core/server.nim:58-63).
• Leader SQL forwarding (forwardQueryToLeader) wraps its socket with the client TLS context when BARADB_TLS_ENABLED is on (it dials the client wire port, which is already TLS-capable).
• E2E: TLS variant cluster test — generate self-signed certs with generateSelfSignedCert (protocol/ssl.nim:79), boot a 3-node cluster with raft TLS on, assert election + one replicated write; assert a plaintext peer cannot join (its frames are rejected and the cluster elects among the TLS nodes).

Rollout / phases

Phase	Deliverable	Risk
P1	D1 failover-load e2e	none (test-only)
P2	D2 CI e2e job	none
P3	D4 raft TLS	medium (transport)
P4	D3 InstallSnapshot + compaction unpin	high (protocol + apply)

P3 before P4 so snapshot transfer ships already-encryptable. Each phase is independently mergeable; P4 is the v1.3.0 gate for calling raft "supported".

Acceptance (v1.3.0)

• Failover-under-load e2e green locally and in CI, in the mandatory gate.
• Killed-node-returns and wiped-node-join scenarios converge without manual intervention (covered by new e2e phases).
• Leader log length stays bounded while a peer is down > PEER_STALE_MS (assert via baradb_raft_log_entries metric in e2e).
• Raft port TLS: cluster runs fully over TLS; plaintext injection fails.
• docs/en/distributed.md + known-limitations.md updated: raft no longer "experimental" for the covered scope; remaining non-goals listed.