Baradb/docs/superpowers/plans/2026-07-30-raft-sql-writes.md

SQL Writes Through Raft (C3b) Implementation Plan

Status: DONE — shipped on main (plus post-C3b DDL/forward/compact/metrics). Historical plan; do not re-run tasks. Overview: docs/superpowers/specs/2026-07-30-raft-cluster-status.md.

Goal: When Raft is enabled, SQL writes commit through the Raft log before the client sees success; followers reject writes naming the leader and apply committed entries via the existing applyCommand loop.

Architecture: Intercept at the server-level executeQuery (core/server.nim:206-227) — same hook point and same data source (res.keyValuePairs) the legacy ReplicationManager uses. Leader: execute locally, append one raft entry per KV pair ("put" = key\x00value, "delete" = key — the exact format baradadb.nim's applyCommand already consumes), poll node.commitIndex until the last appended index commits (no raft.nim changes — state machine stays TLA-frozen). Follower: reject before execution.

Tech Stack: Nim 2.2.10, ARC, unittest, real server processes for E2E.

Global Constraints

• Spec: docs/superpowers/specs/2026-07-30-raft-sql-writes-design.md (read first).
• Test command per task: nim c -d:ssl --threads:on --path:src -o:tests/test_all tests/test_all.nim && ./tests/test_all — exit 0. Final task: full nimble test (673+ [OK]).
• raft.nim stays FROZEN (state machine, handlers, message format, serialization — TLA-faithfulness tests pin them). All changes in server.nim / config.nim / exec/params.nim / executor.nim / baradadb.nim / tests.
• Non-raft deployments (server.raftNode == nil): byte-identical behavior — verify the raft code path is fully gated.
• No new dependencies. Commits per task after green; source files only; no push (controller merges+pushes).
• Verified facts to use:
  • Server hook: core/server.nim:206-227 — parse at :213, executor.executeQuery at :218, legacy replication ship at :219-227 (replication.writeLsn(data) with key\0value). executeQuery(db, ctx, query, params, replication) is called from handleClient (:567 plain, :587 prepared).
  • Statement kinds: nkInsert/nkUpdate/nkDelete/nkMerge write directly; nkCommitTxn emits kvPairs at COMMIT (executor.nim:968-979); BEGIN/ROLLBACK are connection-local.
  • appendLog (raft.nim:355): appendLog*(node, command, data): LogEntry — returns empty entry (index 0) when not leader; replication ships on the next heartbeat (50ms); commit advances via handleAppendReply → applyCommitted (raft.nim:195-215, 407-421).
  • applyCommand format (baradadb.nim:336-343): cmd == "put" → data = key \x00 value; cmd == "delete" → data = key.
  • Delete kvPair convention: non-txn DELETE emits (fullKey, @[]) (exec/dml.nim:241). OPEN ITEM for Task 2: txn COMMIT emits (key, version.value) even when version.isDelete (executor.nim:971-976) — check whether version.value is empty for deletes (read core/mvcc.nim writeSet/VersionedValue); if not, change the COMMIT loop to emit (key, @[]) for deletes so the raft "empty value = delete" rule holds.
  • Config env parsing: core/config.nim:177-206 (BARADB_RAFT_* section).

---

Task 1: isWrite helper + Server.raftNode + follower rejection

Files:

• Modify: src/barabadb/query/exec/params.nim (isWrite helper next to isDDL)
• Modify: src/barabadb/core/server.nim (raftNode field + rejection in executeQuery)
• Modify: src/baradadb.nim (assign server.raftNode when raft enabled)
• Test: tests/bugfix_test.nim

Interfaces:

• Produces:

  • proc isWrite*(stmt: Node): bool in exec/params.nim — true for nkInsert, nkUpdate, nkDelete, nkMerge, nkCommitTxn (check exact enum names in query/ast.nim; nkCommitTxn included because COMMIT emits kvPairs).
  • Server.raftNode*: RaftNode (nil default) — server.nim imports core/raft (verify no cycle: raft.nim must not import server.nim).
  • Rejection in server-level executeQuery (core/server.nim:206+): right after parse and the empty-stmts check, BEFORE executor.executeQuery:

    if server.raftNode != nil and isWrite(astNode.stmts[0]):
      let node = server.raftNode
      if node.state != rsLeader:
        let who = if node.leaderId.len > 0: node.leaderId else: "none elected"
        return (false, QueryResult(), "not leader; leader is '" & who & "'")
    

    (Adjust to the proc's actual error-return convention and to how it accesses the Server — if executeQuery is not a method, thread the raft node the same way replication is threaded; check the handleClient call sites at :567/:587 first and pick the smaller change.)

• Step 1: Failing tests

New suite in tests/bugfix_test.nim:

suite "Raft write classification":
  test "isWrite classifies DML and COMMIT":
    check isWrite(parse("INSERT INTO t (id) VALUES (1)").stmts[0])
    check isWrite(parse("UPDATE t SET id = 2").stmts[0])
    check isWrite(parse("DELETE FROM t WHERE id = 1").stmts[0])
    check isWrite(parse("COMMIT").stmts[0])
    check not isWrite(parse("SELECT * FROM t").stmts[0])
    check not isWrite(parse("CREATE TABLE t (id INT)").stmts[0])
    check not isWrite(parse("BEGIN").stmts[0])
    check not isWrite(parse("ROLLBACK").stmts[0])

(Adapt to bugfix_test.nim's imports — it has parser already.)

• Step 2: Run, watch it fail

Expected: compile error — isWrite undeclared.

• Step 3: Implement

The helper, the Server field, the rejection, the baradadb.nim assignment (tcpServer.raftNode = raftNode — check the actual server variable name and that raftNet/raftNode are in scope; they were hoisted in the C3a fix wave).

• Step 4: Run, watch it pass

bugfix_test green; test_all green (461+ [OK]).

• Step 5: Commit

git add src/barabadb/query/exec/params.nim src/barabadb/core/server.nim src/baradadb.nim tests/bugfix_test.nim
git commit -m "feat(raft): classify writes, reject them on follower nodes"

---

Task 2: Leader write path — append + wait-for-commit

Files:

• Modify: src/barabadb/core/server.nim (raft append + commit wait, replacing the legacy ship when raft active)
• Modify: src/barabadb/core/config.nim (raftWriteTimeoutMs + env)
• Modify: possibly src/barabadb/query/executor.nim (COMMIT delete kvPair convention — see OPEN ITEM)
• Test: tests/test_all.nim (in-process leader path)

Interfaces:

• Consumes: Task 1's isWrite + server.raftNode; appendLog; applyCommand format.

• Produces:

  • In server executeQuery after res.success (replacing the legacy replication ship when raftNode != nil):

    if server.raftNode != nil and res.keyValuePairs.len > 0:
      let node = server.raftNode
      var lastIdx = 0'u64
      for (key, value) in res.keyValuePairs:
        let entry = if value.len > 0:
            node.appendLog("put", cast[seq[byte]](key & "\x00" & cast[string](value)))
          else:
            node.appendLog("delete", cast[seq[byte]](key))
        if entry.index == 0:
          return (false, QueryResult(), "lost leadership during raft append")
        lastIdx = entry.index
      # wait for commit
      let deadline = getMonoTime() + initDuration(milliseconds = config.raftWriteTimeoutMs)
      while node.commitIndex < lastIdx and getMonoTime() < deadline:
        await sleepAsync(10)   # or the sync equivalent — check whether executeQuery is async; if it is NOT async, use os.sleep in a bounded loop
      if node.commitIndex < lastIdx:
        return (false, QueryResult(), "raft commit timeout")
    else if replication != nil and res.keyValuePairs.len > 0:
      <existing legacy ship, unchanged>
    

    CRITICAL: check whether server-level executeQuery (server.nim:206) is a sync proc under withStorageGate — if sync, the wait must not block the async event loop; use short os.sleep polling and keep the timeout small, or document. Match the codebase's reality, not this sketch.
  • raftWriteTimeoutMs*: int on BaraConfig (default 5000) + BARADB_RAFT_WRITE_TIMEOUT_MS env parsing in the raft section of loadConfigFromEnv.
  • COMMIT delete convention resolved (OPEN ITEM above): version.value empty for deletes, or executor.nim COMMIT loop fixed to emit (key, @[]) for isDelete entries.

• Step 1: Failing test

In tests/test_all.nim (raft suites):

  test "leader append+commit wait round-trips through applyCommand":
    # 3 in-process nodes over TCP (reuse the "timerLoop elects a leader" setup from C3a Task 2)
    # wait for a leader; on the leader: appendLog("put", "users.1\x00alice")
    # poll leader.commitIndex until >= entry.index (deadline 3s)
    # assert a follower's applyCommand got invoked with ("put", data)
    #   (wire applyCommand on each node to record calls — check how baradadb.nim wires it)
  test "txn COMMIT delete kvPairs are empty-valued":
    # embedded ctx: BEGIN; INSERT; DELETE same row; COMMIT
    # assert res.keyValuePairs for the deleted key has value.len == 0

• Step 2: Run, watch them fail

Expected: first — no server-side helper exists yet (compile error or assertion); second — may already pass or fail depending on the OPEN ITEM finding; record which.

• Step 3: Implement

Per Produces. Resolve the sync/async question by reading server.nim:206-227 and its callers first; keep the wait bounded and simple.

• Step 4: Run, watch them pass

test_all green.

• Step 5: Commit

git add src/barabadb/core/server.nim src/barabadb/core/config.nim src/barabadb/query/executor.nim tests/test_all.nim
git commit -m "feat(raft): leader appends writes to raft log and waits for commit"

---

Task 3: E2E — replicated writes across a real 3-node cluster

Files:

• Create: tests/raft_writes_e2e_test.nim (or extend tests/raft_e2e_test.nim — pick the cleaner; new file preferred to keep runtimes isolated)
• Modify: baradadb.nimble (wire into test task)

Interfaces:

• Consumes: Tasks 1-2; the E2E process harness from tests/raft_e2e_test.nim (port base, env setup, leader detection via the "became leader" log line, teardown); the Nim client via adaptors/nim/baradb_sqlite (pattern from tests/nimforum_smoke_test.nim: open("127.0.0.1:" & $port, "", "", "default"), db.exec(sql"..."), db.getAllRows).

• Produces: suite "Raft replicated writes E2E":

  1. Boot 3 nodes (raft enabled), wait for leader (log line).
  2. On the LEADER's client port: CREATE TABLE + INSERT a row (expect success).
  3. On a FOLLOWER's client port: poll SELECT until the row appears (deadline 5s; follower applies committed entries via applyCommand to its default DB).
  4. On a FOLLOWER: INSERT → expect an error containing "not leader".
  5. Kill the leader; wait for new leader; INSERT on the new leader succeeds; the row becomes visible on the remaining follower.
  6. Teardown all processes; dump captured output on any failure.

• Step 1: Write the test

Follow raft_e2e_test.nim conventions (O_NONBLOCK output drains, deadlines, skip-with-reason if build/baradadb missing, different port base from both nimforum_smoke_test and raft_e2e_test). Note: applyCommand applies to the DEFAULT database — use database "default" in the client.

• Step 2: Run standalone (twice)

nim c -d:ssl --threads:on --path:src -o:tests/raft_writes_e2e_test tests/raft_writes_e2e_test.nim && ./tests/raft_writes_e2e_test Expected: PASS twice consecutively. If the follower never sees the row, debug order: raft commit wait (Task 2) → applyCommand wiring → default-DB targeting. Dump output on failure.

• Step 3: Wire into nimble test + full suite

Add to baradadb.nimble test list after raft_e2e_test; add the binary name to .gitignore next to tests/raft_e2e_test. Run nimble test — exit 0, 673+ [OK].

• Step 4: Commit

git add tests/raft_writes_e2e_test.nim baradadb.nimble .gitignore
git commit -m "test(raft): E2E replicated writes, follower rejection, failover writes"

---

Task 4: Docs + close-out

Files:

• Modify: docs/superpowers/specs/2026-07-30-raft-sql-writes-design.md (status → done)

• Modify: README.md only if it claims raft/replication behavior that changed (check grep -n -i 'raft\|replicat' README.md | head -20 — update only now-false lines, minimally)

• Step 1: Update docs

• Step 2: Commit

git add docs/ README.md
git commit -m "docs: SQL writes through raft (C3b) done"

---

Self-Review Notes

• Spec coverage: write classification → T1; follower rejection → T1; leader append+wait → T2; config → T2; E2E → T3; docs → T4. Non-goals honored (no forwarding, no DDL replication, raft.nim frozen).
• The COMMIT delete-convention open item is the main correctness risk (a deleted key resurrected on followers) — T2's second test pins it before it can ship.
• Double application on the leader is idempotent by KV semantics; T3's follower-visibility assertions prove the real path end-to-end.