docs: raft cluster status, plans closed, CHANGELOG/README/monitoring
- Add raft-cluster-status overview (C3a/C3b/post-C3b shipped on main) - Mark C3a/C3b design+plans done; refresh operator docs en/bg - CHANGELOG 1.2.0 Raft section; README cluster example and status line - monitoring.md health/metrics match real HTTP port+440 and raft series
This commit is contained in:
@@ -1,116 +1,34 @@
|
||||
# Networked Raft Bootstrap (C3a) — Design
|
||||
|
||||
Date: 2026-07-30
|
||||
Status: Approved direction (user: "продължи"); implementation follows.
|
||||
Date: 2026-07-30
|
||||
Status: **Done** (merged to `main`). See also `2026-07-30-raft-cluster-status.md`.
|
||||
|
||||
## Problem
|
||||
|
||||
Raft in BaraDB is half-wired for real networking. `core/raft.nim` already has
|
||||
a TCP transport (`RaftNetwork`, binary framing, serialization, election-over-
|
||||
TCP proven by a test), but in production:
|
||||
|
||||
1. `node.peerAddrs` is **never populated** (`baradadb.nim` creates the node
|
||||
from `config.raftPeers` but never parses addresses) — all sends silently
|
||||
no-op (`raft.nim:560-561`).
|
||||
2. **No election timer runs** — `tick` is only called from tests; a deployed
|
||||
node never starts an election.
|
||||
3. `handleAppendEntries` on the wire path does **not reset the election
|
||||
timer** — even if timers ran, followers would start elections despite a
|
||||
healthy leader.
|
||||
4. Raft **state persistence is disabled** in server startup (`dataDir` not
|
||||
passed to `newRaftNode`, `baradadb.nim:333`).
|
||||
5. Framing reads assume full TCP reads (`recv(4)`/`recv(payloadLen)`,
|
||||
`raft.nim:604-611`) — partial reads corrupt the stream.
|
||||
|
||||
Result: `BARADB_RAFT_ENABLED=true` today starts a listening socket that can
|
||||
never elect anyone. This phase wires what exists; it does NOT change the SQL
|
||||
write path (that is C3b) or add membership/snapshots (C3c).
|
||||
Raft in BaraDB was half-wired for real networking. `core/raft.nim` already had
|
||||
a TCP transport, but production never populated `peerAddrs`, never ran an
|
||||
election timer, did not reset timers on AppendEntries, skipped state
|
||||
persistence, and used unsafe partial frame reads.
|
||||
|
||||
## Goal
|
||||
|
||||
A 3-node BaraDB cluster started with ordinary config elects a leader over
|
||||
TCP, maintains it with heartbeats, and re-elects after the leader is killed —
|
||||
verified end-to-end with real server processes.
|
||||
A 3-node BaraDB cluster elects a leader over TCP, maintains it with heartbeats,
|
||||
and re-elects after the leader is killed — verified with real processes
|
||||
(`tests/raft_e2e_test.nim`).
|
||||
|
||||
Non-goals: SQL writes through Raft (C3b), membership changes, snapshots,
|
||||
reconnect/backoff hardening, TLS/auth on the raft port (C3c).
|
||||
## Delivered
|
||||
|
||||
## Design
|
||||
| Item | Implementation |
|
||||
|------|----------------|
|
||||
| Peer addresses | `BARADB_RAFT_PEERS=id@host:port` → `raftPeerAddrs` |
|
||||
| Election timer | `timerLoop` inside `RaftNetwork.run` |
|
||||
| Heartbeat reset | `processMessage` resets timer on AppendEntries with valid term |
|
||||
| State persistence | `dataDir` → `raft_state.bin` |
|
||||
| Partial reads | `recvExact` frame reassembly |
|
||||
| E2E | `tests/raft_e2e_test.nim` (election + failover) |
|
||||
|
||||
### 1. Peer address configuration
|
||||
## Non-goals (handled later)
|
||||
|
||||
- Env (existing mechanism): `BARADB_RAFT_PEERS` entries extended from bare
|
||||
`nodeId` to `nodeId@host:port`. Comma-separated, e.g.
|
||||
`BARADB_RAFT_PEERS="n1@127.0.0.1:9473,n2@127.0.0.1:9474,n3@127.0.0.1:9475"`.
|
||||
Bare entries (no `@`) keep current meaning (peer id, no address).
|
||||
- Parsing lives in `core/config.nim` next to the existing `BARADB_RAFT_*`
|
||||
env handling (config.nim:174-179), producing `raftPeerAddrs: Table[string,
|
||||
(string, int)]` on BaraConfig. Malformed entries fail startup with a clear
|
||||
error (config-time, not runtime).
|
||||
- `baradadb.nim` startup copies `config.raftPeerAddrs` into
|
||||
`node.peerAddrs` and passes the data dir (see §3).
|
||||
|
||||
### 2. Election timer in production
|
||||
|
||||
- A `timerLoop` async proc in `core/raft.nim` (next to `heartbeatLoop`,
|
||||
raft.nim:625): every 50ms calls `tick(node)`; on election timeout calls
|
||||
the existing `startElection` path (raft.nim:659-685), which sends
|
||||
RequestVote over `RaftNetwork.send`.
|
||||
- `RaftNetwork.run` (raft.nim:633) starts `timerLoop` alongside
|
||||
`heartbeatLoop` via `asyncCheck` — single place, no baradadb.nim changes
|
||||
beyond startup wiring.
|
||||
- Wire-path timer reset: in the message-receive handling of `RaftNetwork`,
|
||||
after a valid AppendEntries (or heartbeat) is processed, reset the
|
||||
follower's election timer (`ElectionTimer.lastHeartbeat = now`) — wherever
|
||||
the existing handler processes inbound AppendEntries, matching what the
|
||||
in-process tests do manually.
|
||||
|
||||
### 3. State persistence on
|
||||
|
||||
- `baradadb.nim:333`: pass `config.dataDir` (or a raft subdirectory of it —
|
||||
check what `newRaftNode` expects; `saveState`/`loadState` write
|
||||
`raft_state.bin`) so currentTerm/votedFor/log survive restarts, per the
|
||||
Raft spec (and the TLA models).
|
||||
|
||||
### 4. Framing robustness
|
||||
|
||||
- Replace the `recv(4)` / `recv(payloadLen)` assumptions with the existing
|
||||
`recvExact`-style helper pattern used by the main server
|
||||
(`core/server.nim:312-329` has `recvExact`/`recvExactWithTimeout` — mirror
|
||||
the approach inside raft.nim; do NOT import server.nim into raft.nim).
|
||||
|
||||
### 5. What does NOT change
|
||||
|
||||
- SQL write path (ReplicationManager, server.nim) — untouched.
|
||||
- Raft state machine, message format, serialization — untouched (the TLA-
|
||||
faithfulness tests pin them).
|
||||
- `applyCommand` hook in baradadb.nim — stays as-is; committed entries
|
||||
(from tests / future C3b) still apply to the default DB.
|
||||
|
||||
## Testing
|
||||
|
||||
TDD where feasible; the headline test is end-to-end:
|
||||
|
||||
1. **Config parsing** (unit, `tests/test_all.nim` or bugfix_test): peers
|
||||
with and without `@host:port`, malformed entries error clearly.
|
||||
2. **Framing** (unit): partial-read feed of a serialized message through the
|
||||
new read path (socketpair or chunked strings) — message reassembles.
|
||||
3. **E2E cluster** (new file `tests/raft_e2e_test.nim`): start 3 real
|
||||
`build/baradadb` processes with raft enabled on distinct ports
|
||||
(client/raft ports offset per node), wait for a leader (queryable how?
|
||||
— simplest: each node logs its role; or a `RAFT STATUS` text command on
|
||||
the client port if one exists cheaply — decide in the plan), kill the
|
||||
leader, assert a new leader is elected within ~5s. Clean teardown.
|
||||
4. Existing suites stay green (`nimble test`, 659+ `[OK]`) — especially the
|
||||
in-process raft suites and the 3-node election TCP test.
|
||||
|
||||
## Risks
|
||||
|
||||
- Election-timer/async interactions with the main server's async loop —
|
||||
`RaftNetwork.run` already runs under `asyncCheck`; timerLoop follows the
|
||||
same pattern. Watch for CPU spin (50ms sleep, not busy loop).
|
||||
- Port allocation in the E2E test (parallel CI) — use the same
|
||||
time-derived port offset pattern as `tests/nimforum_smoke_test.nim`.
|
||||
- Timer reset wiring point: the inbound message path must reach the node
|
||||
the timer ticks — same `RaftNode` instance, verified by the E2E test
|
||||
(no spurious elections under a healthy leader).
|
||||
SQL writes (C3b), DDL/forward/compact/metrics (post-C3b / C3c-lite) — all
|
||||
shipped; see cluster status doc. Still open: membership, InstallSnapshot SM
|
||||
payload, multi-DB raft.
|
||||
|
||||
Reference in New Issue
Block a user