docs: raft cluster status, plans closed, CHANGELOG/README/monitoring
- Add raft-cluster-status overview (C3a/C3b/post-C3b shipped on main) - Mark C3a/C3b design+plans done; refresh operator docs en/bg - CHANGELOG 1.2.0 Raft section; README cluster example and status line - monitoring.md health/metrics match real HTTP port+440 and raft series
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
# Networked Raft Bootstrap (C3a) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
> **Status: DONE** — shipped on `main`. Historical plan; do not re-run tasks.
|
||||
> Overview: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`.
|
||||
|
||||
**Goal:** A 3-node BaraDB cluster started with ordinary config elects a leader over TCP, maintains it with heartbeats, and re-elects after the leader is killed.
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# SQL Writes Through Raft (C3b) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
> **Status: DONE** — shipped on `main` (plus post-C3b DDL/forward/compact/metrics).
|
||||
> Historical plan; do not re-run tasks.
|
||||
> Overview: `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`.
|
||||
|
||||
**Goal:** When Raft is enabled, SQL writes commit through the Raft log before the client sees success; followers reject writes naming the leader and apply committed entries via the existing applyCommand loop.
|
||||
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Raft Cluster Status — C3a / C3b / post-C3b
|
||||
|
||||
Date: 2026-07-30
|
||||
Status: **Shipped on `main`** (tip includes metrics).
|
||||
Branch: all work merged to `main` only (feature branch removed).
|
||||
|
||||
## Phase map
|
||||
|
||||
| Phase | Spec / plan | Status | What landed |
|
||||
|-------|-------------|--------|-------------|
|
||||
| **C3a** Network bootstrap | `raft-network-bootstrap-design.md` + plan | **Done** | `id@host:port` peers, election timer in production, AppendEntries resets timer, `dataDir` persistence, partial-read frames, `tests/raft_e2e_test.nim` |
|
||||
| **C3b** SQL writes | `raft-sql-writes-design.md` + plan | **Done** | `isWrite`, follower reject / forward, leader append+wait-commit, rich apply (LSM+index+graph), multi-stmt gate, default-DB only, `tests/raft_writes_e2e_test.nim` |
|
||||
| **C3c-lite** DDL + ops | (this status doc) | **Done** | DDL via `ddl` log entries; leader forwarding (`BARADB_RAFT_CLIENT_PEERS`); safe log compact + snapshot metadata; Prometheus + `/health` raft |
|
||||
|
||||
## Key commits (main)
|
||||
|
||||
| Commit (short) | Summary |
|
||||
|----------------|---------|
|
||||
| C3a series | peers, timer, frames, e2e election |
|
||||
| `38c1c01`…`a462d21` | C3b classification → append → e2e → docs |
|
||||
| `0d51497` | multi-stmt gate, rich apply, delete kv |
|
||||
| `50f827f` | graph apply, non-default DB reject |
|
||||
| `095698b` | DDL through raft |
|
||||
| `9df8316` | leader write/DDL forwarding |
|
||||
| `53704e1` | safe log compaction + snapshot base |
|
||||
| `1b3c261` | raft metrics on `/metrics` + `/health` |
|
||||
|
||||
## Production behavior (summary)
|
||||
|
||||
1. Enable with `BARADB_RAFT_*` env (see `docs/en/distributed.md`).
|
||||
2. Cluster elects a leader over TCP; state in `dataDir/raft/raft_state.bin`.
|
||||
3. DML/DDL on **default** DB only: leader appends, waits for majority, returns.
|
||||
4. Followers forward to leader if `BARADB_RAFT_CLIENT_PEERS` is set; else `not leader`.
|
||||
5. Apply updates LSM + secondary engines; DDL re-executes SQL on each node.
|
||||
6. Log soft-cap via safe prefix compact; metrics on HTTP port `BARADB_PORT+440`.
|
||||
|
||||
## Explicit non-goals still open
|
||||
|
||||
- Multi-database raft (only `default`)
|
||||
- `CREATE`/`DROP DATABASE` replication
|
||||
- Membership change (join/leave) protocol
|
||||
- InstallSnapshot with full SM / LSM payload (v1 uses safe-prefix compact only)
|
||||
- Raft port TLS / mutual auth
|
||||
- Read consistency levels (read-your-writes, linearizable reads on followers)
|
||||
- Automatic client redirect without `CLIENT_PEERS`
|
||||
|
||||
## Tests
|
||||
|
||||
| Suite | Covers |
|
||||
|-------|--------|
|
||||
| `tests/raft_e2e_test.nim` | 3-process election + failover |
|
||||
| `tests/raft_writes_e2e_test.nim` | DDL + DML replicate, forward, index SELECT, failover writes |
|
||||
| `tests/test_all.nim` | in-process raft, append/wait, compact, metrics |
|
||||
| `tests/tla_faithfulness.nim` | ElectionSafety, LogMatching, … |
|
||||
| `tests/bugfix_test.nim` | peers / client peers / isWrite / isRaftDdl |
|
||||
|
||||
## Docs to keep in sync
|
||||
|
||||
- `docs/en/distributed.md` / `docs/bg/distributed.md` — operator guide
|
||||
- `docs/en/monitoring.md` — health/metrics (raft section)
|
||||
- `CHANGELOG.md` — `[1.2.0] Unreleased` Raft section
|
||||
- README raft status line
|
||||
@@ -1,116 +1,34 @@
|
||||
# Networked Raft Bootstrap (C3a) — Design
|
||||
|
||||
Date: 2026-07-30
|
||||
Status: Approved direction (user: "продължи"); implementation follows.
|
||||
Date: 2026-07-30
|
||||
Status: **Done** (merged to `main`). See also `2026-07-30-raft-cluster-status.md`.
|
||||
|
||||
## Problem
|
||||
|
||||
Raft in BaraDB is half-wired for real networking. `core/raft.nim` already has
|
||||
a TCP transport (`RaftNetwork`, binary framing, serialization, election-over-
|
||||
TCP proven by a test), but in production:
|
||||
|
||||
1. `node.peerAddrs` is **never populated** (`baradadb.nim` creates the node
|
||||
from `config.raftPeers` but never parses addresses) — all sends silently
|
||||
no-op (`raft.nim:560-561`).
|
||||
2. **No election timer runs** — `tick` is only called from tests; a deployed
|
||||
node never starts an election.
|
||||
3. `handleAppendEntries` on the wire path does **not reset the election
|
||||
timer** — even if timers ran, followers would start elections despite a
|
||||
healthy leader.
|
||||
4. Raft **state persistence is disabled** in server startup (`dataDir` not
|
||||
passed to `newRaftNode`, `baradadb.nim:333`).
|
||||
5. Framing reads assume full TCP reads (`recv(4)`/`recv(payloadLen)`,
|
||||
`raft.nim:604-611`) — partial reads corrupt the stream.
|
||||
|
||||
Result: `BARADB_RAFT_ENABLED=true` today starts a listening socket that can
|
||||
never elect anyone. This phase wires what exists; it does NOT change the SQL
|
||||
write path (that is C3b) or add membership/snapshots (C3c).
|
||||
Raft in BaraDB was half-wired for real networking. `core/raft.nim` already had
|
||||
a TCP transport, but production never populated `peerAddrs`, never ran an
|
||||
election timer, did not reset timers on AppendEntries, skipped state
|
||||
persistence, and used unsafe partial frame reads.
|
||||
|
||||
## Goal
|
||||
|
||||
A 3-node BaraDB cluster started with ordinary config elects a leader over
|
||||
TCP, maintains it with heartbeats, and re-elects after the leader is killed —
|
||||
verified end-to-end with real server processes.
|
||||
A 3-node BaraDB cluster elects a leader over TCP, maintains it with heartbeats,
|
||||
and re-elects after the leader is killed — verified with real processes
|
||||
(`tests/raft_e2e_test.nim`).
|
||||
|
||||
Non-goals: SQL writes through Raft (C3b), membership changes, snapshots,
|
||||
reconnect/backoff hardening, TLS/auth on the raft port (C3c).
|
||||
## Delivered
|
||||
|
||||
## Design
|
||||
| Item | Implementation |
|
||||
|------|----------------|
|
||||
| Peer addresses | `BARADB_RAFT_PEERS=id@host:port` → `raftPeerAddrs` |
|
||||
| Election timer | `timerLoop` inside `RaftNetwork.run` |
|
||||
| Heartbeat reset | `processMessage` resets timer on AppendEntries with valid term |
|
||||
| State persistence | `dataDir` → `raft_state.bin` |
|
||||
| Partial reads | `recvExact` frame reassembly |
|
||||
| E2E | `tests/raft_e2e_test.nim` (election + failover) |
|
||||
|
||||
### 1. Peer address configuration
|
||||
## Non-goals (handled later)
|
||||
|
||||
- Env (existing mechanism): `BARADB_RAFT_PEERS` entries extended from bare
|
||||
`nodeId` to `nodeId@host:port`. Comma-separated, e.g.
|
||||
`BARADB_RAFT_PEERS="n1@127.0.0.1:9473,n2@127.0.0.1:9474,n3@127.0.0.1:9475"`.
|
||||
Bare entries (no `@`) keep current meaning (peer id, no address).
|
||||
- Parsing lives in `core/config.nim` next to the existing `BARADB_RAFT_*`
|
||||
env handling (config.nim:174-179), producing `raftPeerAddrs: Table[string,
|
||||
(string, int)]` on BaraConfig. Malformed entries fail startup with a clear
|
||||
error (config-time, not runtime).
|
||||
- `baradadb.nim` startup copies `config.raftPeerAddrs` into
|
||||
`node.peerAddrs` and passes the data dir (see §3).
|
||||
|
||||
### 2. Election timer in production
|
||||
|
||||
- A `timerLoop` async proc in `core/raft.nim` (next to `heartbeatLoop`,
|
||||
raft.nim:625): every 50ms calls `tick(node)`; on election timeout calls
|
||||
the existing `startElection` path (raft.nim:659-685), which sends
|
||||
RequestVote over `RaftNetwork.send`.
|
||||
- `RaftNetwork.run` (raft.nim:633) starts `timerLoop` alongside
|
||||
`heartbeatLoop` via `asyncCheck` — single place, no baradadb.nim changes
|
||||
beyond startup wiring.
|
||||
- Wire-path timer reset: in the message-receive handling of `RaftNetwork`,
|
||||
after a valid AppendEntries (or heartbeat) is processed, reset the
|
||||
follower's election timer (`ElectionTimer.lastHeartbeat = now`) — wherever
|
||||
the existing handler processes inbound AppendEntries, matching what the
|
||||
in-process tests do manually.
|
||||
|
||||
### 3. State persistence on
|
||||
|
||||
- `baradadb.nim:333`: pass `config.dataDir` (or a raft subdirectory of it —
|
||||
check what `newRaftNode` expects; `saveState`/`loadState` write
|
||||
`raft_state.bin`) so currentTerm/votedFor/log survive restarts, per the
|
||||
Raft spec (and the TLA models).
|
||||
|
||||
### 4. Framing robustness
|
||||
|
||||
- Replace the `recv(4)` / `recv(payloadLen)` assumptions with the existing
|
||||
`recvExact`-style helper pattern used by the main server
|
||||
(`core/server.nim:312-329` has `recvExact`/`recvExactWithTimeout` — mirror
|
||||
the approach inside raft.nim; do NOT import server.nim into raft.nim).
|
||||
|
||||
### 5. What does NOT change
|
||||
|
||||
- SQL write path (ReplicationManager, server.nim) — untouched.
|
||||
- Raft state machine, message format, serialization — untouched (the TLA-
|
||||
faithfulness tests pin them).
|
||||
- `applyCommand` hook in baradadb.nim — stays as-is; committed entries
|
||||
(from tests / future C3b) still apply to the default DB.
|
||||
|
||||
## Testing
|
||||
|
||||
TDD where feasible; the headline test is end-to-end:
|
||||
|
||||
1. **Config parsing** (unit, `tests/test_all.nim` or bugfix_test): peers
|
||||
with and without `@host:port`, malformed entries error clearly.
|
||||
2. **Framing** (unit): partial-read feed of a serialized message through the
|
||||
new read path (socketpair or chunked strings) — message reassembles.
|
||||
3. **E2E cluster** (new file `tests/raft_e2e_test.nim`): start 3 real
|
||||
`build/baradadb` processes with raft enabled on distinct ports
|
||||
(client/raft ports offset per node), wait for a leader (queryable how?
|
||||
— simplest: each node logs its role; or a `RAFT STATUS` text command on
|
||||
the client port if one exists cheaply — decide in the plan), kill the
|
||||
leader, assert a new leader is elected within ~5s. Clean teardown.
|
||||
4. Existing suites stay green (`nimble test`, 659+ `[OK]`) — especially the
|
||||
in-process raft suites and the 3-node election TCP test.
|
||||
|
||||
## Risks
|
||||
|
||||
- Election-timer/async interactions with the main server's async loop —
|
||||
`RaftNetwork.run` already runs under `asyncCheck`; timerLoop follows the
|
||||
same pattern. Watch for CPU spin (50ms sleep, not busy loop).
|
||||
- Port allocation in the E2E test (parallel CI) — use the same
|
||||
time-derived port offset pattern as `tests/nimforum_smoke_test.nim`.
|
||||
- Timer reset wiring point: the inbound message path must reach the node
|
||||
the timer ticks — same `RaftNode` instance, verified by the E2E test
|
||||
(no spurious elections under a healthy leader).
|
||||
SQL writes (C3b), DDL/forward/compact/metrics (post-C3b / C3c-lite) — all
|
||||
shipped; see cluster status doc. Still open: membership, InstallSnapshot SM
|
||||
payload, multi-DB raft.
|
||||
|
||||
@@ -1,109 +1,55 @@
|
||||
# SQL Writes Through Raft (C3b) — Design
|
||||
|
||||
Date: 2026-07-30
|
||||
Status: Done (implemented on `feat/raft-sql-writes`).
|
||||
Date: 2026-07-30
|
||||
Status: **Done** (merged to `main`). Extended by post-C3b work (DDL, forward,
|
||||
compact, metrics) — see `2026-07-30-raft-cluster-status.md`.
|
||||
|
||||
## Problem
|
||||
|
||||
With C3a a real Raft cluster elects a leader over TCP, but SQL writes still
|
||||
bypass Raft entirely: they go straight to the local LSM and (optionally) to
|
||||
the legacy `ReplicationManager` (unconnected in production). Committed Raft
|
||||
entries can be applied (`applyCommand` is wired and invoked on commit), yet
|
||||
nothing ever calls `appendLog` from the write path.
|
||||
bypassed Raft: they went straight to the local LSM. Committed Raft entries
|
||||
could be applied (`applyCommand`), yet nothing called `appendLog` from the
|
||||
write path.
|
||||
|
||||
## Goal
|
||||
## Goal (delivered)
|
||||
|
||||
When Raft is enabled, SQL writes are committed through the Raft log before
|
||||
the client sees success:
|
||||
When Raft is enabled, SQL writes commit through the Raft log before the client
|
||||
sees success:
|
||||
|
||||
- Leader: execute locally (constraints validated as today), append the
|
||||
resulting KV pairs to the Raft log, wait for majority commit, then return.
|
||||
- Followers: reject write statements with a clear "not leader" error naming
|
||||
the known leader; apply committed entries via the existing `applyCommand`
|
||||
loop (raft.nim:195-215).
|
||||
- Non-raft deployments: byte-identical behavior to today.
|
||||
- **Leader:** execute locally, append KV pairs (`put` / `delete`), wait for
|
||||
majority `commitIndex`, return.
|
||||
- **Followers:** originally reject with `not leader; leader is '…'`; with
|
||||
`BARADB_RAFT_CLIENT_PEERS` they **forward** to the leader (post-C3b).
|
||||
- **Apply:** LSM + secondary B-tree/FTS/HNSW + in-memory graphs
|
||||
(`applyReplicatedPut` / `Delete`).
|
||||
- **Non-raft:** behavior gated on `raftNode == nil`.
|
||||
|
||||
Non-goals (C3c+): leader forwarding/proxy, read consistency levels,
|
||||
schema/DDL replication (DDL produces no keyValuePairs today — out of scope),
|
||||
membership, snapshots.
|
||||
## Design (as shipped)
|
||||
|
||||
## Design
|
||||
### Write interception
|
||||
|
||||
### Write interception point
|
||||
`core/server.nim` `executeQuery`:
|
||||
|
||||
`core/server.nim:206-227` — the server-level `executeQuery` parses the AST
|
||||
(server.nim:213) and, after successful execution, already ships
|
||||
`res.keyValuePairs` to the ReplicationManager (server.nim:219-227). The Raft
|
||||
path slots into the same place with the same data source:
|
||||
|
||||
1. Classify the statement right after parse: write = `stmt.kind in {nkInsert,
|
||||
nkUpdate, nkDelete, nkMerge}` (DDL is out of scope; SELECT unaffected).
|
||||
New helper `isWrite(stmt)` in `exec/params.nim` next to `isDDL`.
|
||||
2. If Raft is active for this server (`server.raftNode != nil`):
|
||||
- Not leader (`node.state != rsLeader`): return error
|
||||
`not leader; leader is '<node.leaderId>'` (or `no leader elected`).
|
||||
The statement is NOT executed.
|
||||
- Leader: execute as today. If `res.success` and
|
||||
`res.keyValuePairs.len > 0`: for each `(key, value)` append one log
|
||||
entry — `cmd = "put"` with data `key \x00 value`, or `cmd = "delete"`
|
||||
with data `key` when value is empty (this is EXACTLY the format the
|
||||
existing `applyCommand` in baradadb.nim:336-343 already consumes — no
|
||||
format changes, frozen). Then wait until `node.commitIndex` reaches the
|
||||
last appended index (see below) before returning `res`.
|
||||
- The legacy `replication.writeLsn` hook is skipped when the Raft write
|
||||
path handled the statement (no double shipping).
|
||||
3. Single-node raft (enabled, no peers): majority is self — appends commit
|
||||
immediately; behavior is correct with negligible overhead.
|
||||
1. Classify every statement: `isWrite` / `isRaftDdl` (not only `stmts[0]`).
|
||||
2. If raft active and write/DDL: require **default** database + leader (or
|
||||
forward).
|
||||
3. After success: pure DML → `appendWriteToRaft`; any DDL in batch →
|
||||
`appendDdlToRaft` (full SQL re-exec on apply).
|
||||
|
||||
### Wait-for-commit
|
||||
|
||||
No raft.nim changes (the state machine stays TLA-frozen). Poll from
|
||||
server.nim: after appending, `while node.commitIndex < lastIdx`: sleep 10ms,
|
||||
up to `config.raftWriteTimeoutMs` (new env `BARADB_RAFT_WRITE_TIMEOUT_MS`,
|
||||
default 5000). Commit advances on the 50ms heartbeat cadence via the
|
||||
existing `handleAppendReply`/`applyCommitted` path, so typical latency is
|
||||
one heartbeat. On timeout: return an error (`raft commit timeout`) — the
|
||||
entry may still commit later; documented limitation. Local execution already
|
||||
happened (it validates constraints and produces kvPairs) — leader double-
|
||||
applies via applyCommand on commit, which is idempotent for put/delete KV.
|
||||
Poll `commitIndex` outside the storage gate (`appendWriteToRaft` /
|
||||
`waitRaftCommit`); timeout → `raft commit timeout`.
|
||||
|
||||
### Server wiring
|
||||
### Known v1 limitations (still true)
|
||||
|
||||
`Server` gains `raftNode*: RaftNode` (nil by default) in core/server.nim;
|
||||
baradadb.nim assigns it when raft is enabled (the node already exists there).
|
||||
server.nim imports core/raft (no cycle: raft.nim does not import server).
|
||||
- Multi-DB raft not supported (`raft writes only supported on the 'default' database`).
|
||||
- `CREATE`/`DROP DATABASE` not raft-replicated.
|
||||
- Leader local execute before majority (timeout leaves local write; documented).
|
||||
- No InstallSnapshot full SM dump (safe log compact only).
|
||||
- No membership changes.
|
||||
|
||||
### Known v1 limitations (documented in code)
|
||||
## Tests
|
||||
|
||||
- Leader losing leadership between local execution and appendLog (narrow
|
||||
race): appendLog returns the empty entry (index 0) — surfaced as a
|
||||
`lost leadership` error; the local write already applied (uncommitted).
|
||||
- Transactional writes: kvPairs are emitted at COMMIT (executor.nim:970-979)
|
||||
— the COMMIT statement is the raft-replicated unit (classify nkCommitTxn?
|
||||
— resolve in planning: COMMIT's kvPairs flow through the same hook since
|
||||
the existing replication hook already catches them).
|
||||
- DDL is not replicated (no kvPairs today).
|
||||
|
||||
## Testing
|
||||
|
||||
1. Unit (`tests/bugfix_test.nim`): `isWrite` classification for
|
||||
INSERT/UPDATE/DELETE/MERGE vs SELECT/DDL.
|
||||
2. Follower rejection (in-process or E2E): write on a follower returns the
|
||||
"not leader" error naming the leader; the statement is not executed.
|
||||
3. E2E (extend `tests/raft_e2e_test.nim` or sibling): 3 real nodes; write a
|
||||
row on the leader via the Nim client (adaptors/nim/baradb_sqlite pattern
|
||||
from nimforum_smoke_test); poll a SELECT on a follower until the row
|
||||
appears (deadline ~5s — follower applies on commit via applyCommand);
|
||||
write on a follower → rejected; kill leader → re-election → writes work
|
||||
on the new leader.
|
||||
4. Full `nimble test` green (673+ `[OK]`).
|
||||
|
||||
## Risks
|
||||
|
||||
- Double application on the leader (executor + applyCommand): idempotent
|
||||
KV semantics — verify put/delete idempotency holds for the value encoding
|
||||
(same key → same value; delete of existing key).
|
||||
- `node.log[idx-1]` positional indexing in the leader commit loop
|
||||
(raft.nim:407) — pre-existing, out of scope unless the E2E trips it.
|
||||
- Apply lag on followers (heartbeat cadence): tests poll with deadlines,
|
||||
never fixed sleeps.
|
||||
- Unit: classification, append/timeout, apply indexes/graphs, compact, metrics
|
||||
- E2E: `tests/raft_writes_e2e_test.nim` (schema, forward, index SELECT, failover)
|
||||
|
||||
Reference in New Issue
Block a user