docs: v1.3.0 raft-supported docs, limitations, runbook updates
This commit is contained in:
+18
-2
@@ -5,7 +5,7 @@ BaraDB supports distributed deployment with Raft consensus, sharding, and replic
|
||||
> ⚠️ **Multi-Database Limitation**
|
||||
> The distributed modules (Raft, sharding, and replication) are currently wired to the **`default`** database only. If you use multiple databases (`CREATE DATABASE`, `USE DATABASE`), distributed features do not yet span across them. Each database would need its own cluster setup.
|
||||
|
||||
> **Status (2026-07-30):** Raft C3a/C3b + DDL/forward/compact/metrics are **shipped**. Multi-node Raft is **experimental** for v1.2.0 GA (single-node is the production tier). See [known-limitations](known-limitations.md) and `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`.
|
||||
> **Status (2026-07-30, v1.3.0):** Raft C3a/C3b + DDL/forward/compact/metrics are **shipped**, and multi-node Raft is **supported** for the single-`default`-DB scope (failover under load, raft TLS, InstallSnapshot cold-node recovery — all e2e-proven). See [known-limitations](known-limitations.md) and `docs/superpowers/specs/2026-07-30-raft-cluster-status.md`.
|
||||
|
||||
## Raft Consensus
|
||||
|
||||
@@ -20,10 +20,23 @@ Leader election and log replication over TCP; SQL DML/DDL on the default DB go t
|
||||
| `BARADB_RAFT_WRITE_TIMEOUT_MS` | Max wait for majority commit on SQL writes (default 5000) |
|
||||
| `BARADB_RAFT_CLIENT_PEERS` | Optional `id@host:clientPort` map for leader write forwarding |
|
||||
| `BARADB_RAFT_LOG_MAX_ENTRIES` | Soft cap on in-memory raft log length (default 256); safe prefix compact |
|
||||
| `BARADB_RAFT_SNAP_CHUNK_KB` | InstallSnapshot chunk size in KiB (default 256) |
|
||||
| `BARADB_RAFT_PEER_STALE_MS` | Peer is stale after this many ms without an ack (default 30000); stale peers no longer pin log compaction |
|
||||
| `BARADB_RAFT_TLS_ENABLED` | TLS on the raft TCP port (default false; fail-closed startup if cert/key missing) |
|
||||
| `BARADB_RAFT_TLS_CERT_FILE` | Server certificate for the raft listener |
|
||||
| `BARADB_RAFT_TLS_KEY_FILE` | Private key for the raft listener |
|
||||
| `BARADB_RAFT_TLS_CA_FILE` | Optional CA bundle for peer verification |
|
||||
| `BARADB_RAFT_TLS_VERIFY_PEER` | Mutual auth — verify client certificates (default false) |
|
||||
|
||||
When Raft is enabled, SQL DML (`INSERT`/`UPDATE`/`DELETE`/`MERGE` and transactional `COMMIT`) and schema DDL (`CREATE`/`DROP`/`ALTER` table, index, view, graph, …) are accepted only on the leader of the **`default`** database. DML ships as put/delete log entries; DDL ships as a `ddl` entry with the original SQL and is re-executed on every node at apply. Followers that receive a write/DDL **forward** it to the leader when `BARADB_RAFT_CLIENT_PEERS` maps the leader id to a SQL client address; otherwise they return `not leader; leader is '…'`. Writes against any other database name are rejected (`raft writes only supported on the 'default' database`). `CREATE`/`DROP DATABASE` are not raft-replicated (multi-DB is out of scope for v1). Committed DML also updates secondary B-tree/FTS/HNSW indexes and in-memory graphs.
|
||||
|
||||
**Log compaction (v1):** after apply, each node may drop a fully-safe log prefix once `log.len` exceeds `BARADB_RAFT_LOG_MAX_ENTRIES`. The leader never discards past any peer's `matchIndex` (so lagging followers still catch up via AppendEntries). Snapshot metadata (`lastSnapshotIndex`/`Term`) is persisted in `raft_state.bin`; full InstallSnapshot state-machine payloads are not required while this safe-prefix policy holds.
|
||||
**Log compaction:** after apply, each node may drop a fully-safe log prefix once `log.len` exceeds `BARADB_RAFT_LOG_MAX_ENTRIES`. On the leader, the safe prefix is computed only over peers that acked within `BARADB_RAFT_PEER_STALE_MS` — stale peers no longer pin compaction and are recovered by snapshot on return. Compaction never goes past `lastApplied`. Snapshot metadata (`lastSnapshotIndex`/`Term`) is persisted in `raft_state.bin`.
|
||||
|
||||
**Snapshot recovery (InstallSnapshot, v1.3.0):** when a follower's lag is unrecoverable (the entries it needs were compacted away), the leader builds a `tar.gz` snapshot of the default DB and streams it as `BARADB_RAFT_SNAP_CHUNK_KB`-sized chunks. The follower restores it via the backup/restore path, adopts the snapshot base as its `commitIndex`/`lastApplied`, and resumes normal AppendEntries catch-up. A node that returns after a long outage — and a **wiped** node (data dir deleted, same node id) — both converge automatically through this path. Proven by `tests/raft_coldnode_e2e_test.nim`.
|
||||
|
||||
**Client failover contract:** a write that is in flight when the leader dies **fails fast with an error** — the client must retry it (against the new leader, or any follower if `BARADB_RAFT_CLIENT_PEERS` forwarding is configured). Every write the server **acknowledged** survives the failover and is present on the new leader and all caught-up followers. Proven by `tests/raft_failover_load_e2e_test.nim` (leader killed under sustained INSERT load; all acked writes found on both survivors).
|
||||
|
||||
**Raft TLS (v1.3.0):** set `BARADB_RAFT_TLS_ENABLED=true` plus `BARADB_RAFT_TLS_CERT_FILE`/`BARADB_RAFT_TLS_KEY_FILE` on every node; startup fails closed if the cert or key is missing. Add `BARADB_RAFT_TLS_CA_FILE` and `BARADB_RAFT_TLS_VERIFY_PEER=true` for mutual authentication. The whole cluster must run the same mode: a plaintext node cannot speak to a TLS port (its frames are undecryptable) and is excluded from the cluster — proven by `tests/raft_tls_e2e_test.nim`. Follower→leader SQL forwarding is TLS-wrapped automatically when the server's client wire port has TLS enabled.
|
||||
|
||||
**Metrics:** with raft enabled, `GET /metrics` (HTTP port = `BARADB_PORT + 440`) includes Prometheus lines such as `baradb_raft_is_leader`, `baradb_raft_term`, `baradb_raft_log_entries`, `baradb_raft_apply_lag`, `baradb_raft_commit_wait_ms_total`, `baradb_raft_elections_total`, `baradb_raft_forwards_total`, and `baradb_raft_compactions_total`. `GET /health` embeds a `raft` object (`role`, `term`, `leader_id`, `commit_index`, `apply_lag`, …).
|
||||
|
||||
@@ -67,6 +80,9 @@ let entry = n1.appendLog("SET key1 value1")
|
||||
|------|----------------|
|
||||
| `tests/raft_e2e_test.nim` | 3 real processes; election + kill-leader failover |
|
||||
| `tests/raft_writes_e2e_test.nim` | DDL/DML via raft, follower forward, index SELECT, failover writes |
|
||||
| `tests/raft_failover_load_e2e_test.nim` | Leader killed under sustained write load; every acked write survives |
|
||||
| `tests/raft_tls_e2e_test.nim` | Full-TLS 3-node cluster works; plaintext node excluded |
|
||||
| `tests/raft_coldnode_e2e_test.nim` | Returning node and wiped node converge via InstallSnapshot |
|
||||
|
||||
## Sharding
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Known Limitations — v1.2.0 Production GA
|
||||
# Known Limitations — v1.3.0
|
||||
|
||||
This page defines **what BaraDB promises** in the v1.2.0 production cut.
|
||||
This page defines **what BaraDB promises** in the v1.3.0 production cut.
|
||||
|
||||
| Tier | Meaning |
|
||||
|------|---------|
|
||||
@@ -10,20 +10,22 @@ This page defines **what BaraDB promises** in the v1.2.0 production cut.
|
||||
|
||||
## Support matrix
|
||||
|
||||
| Area | GA (v1.2.0) | Experimental / later |
|
||||
|------|-------------|----------------------|
|
||||
| Area | v1.3.0 | Notes |
|
||||
|------|--------|-------|
|
||||
| Single-node SQL + LSM storage | **Supported** | — |
|
||||
| Schema persistence (tables, indexes) | **Supported** | — |
|
||||
| FTS / HNSW / graphs across restart | **Supported** | — |
|
||||
| Auth + JWT (when configured) | **Supported** | — |
|
||||
| Backup / restore (offline, all-databases) | **Supported** | — |
|
||||
| Multi-database (non-Raft) | **Supported** | — |
|
||||
| Raft 3-node election + SQL/DDL | **Experimental** | InstallSnapshot SM payload, membership |
|
||||
| Raft 3-node (single `default` DB) | **Supported** | failover under load, raft TLS, InstallSnapshot recovery — e2e-proven |
|
||||
| Leader write forwarding | **Supported** | needs `BARADB_RAFT_CLIENT_PEERS` |
|
||||
| Raft multi-database | **Not supported** | only `default` |
|
||||
| Leader write forwarding | **Experimental** | needs `BARADB_RAFT_CLIENT_PEERS` |
|
||||
| `CREATE`/`DROP DATABASE` replication | **Not supported** | run per node |
|
||||
| Raft membership changes (join/leave) | **Not supported** | fixed `BARADB_RAFT_PEERS` set |
|
||||
| Follower linearizable reads | **Not supported** | best-effort after apply |
|
||||
| Rolling upgrades | **Not supported** | restart all nodes together — mixed v1.2/v1.3 binaries must not run in one cluster |
|
||||
| ORC multi-threaded shared LSM | **Not supported** | default is ARC (`nim.cfg`) |
|
||||
| Zero-downtime rolling upgrade | **Not supported** | stop → backup → upgrade |
|
||||
| Postgres wire protocol | **Not supported** | Bara wire + HTTP |
|
||||
|
||||
## Single-node GA (what you can rely on)
|
||||
@@ -33,13 +35,19 @@ This page defines **what BaraDB promises** in the v1.2.0 production cut.
|
||||
- HTTP `/health` and `/metrics` for process liveness
|
||||
- Offline backup of `data/databases` and restore onto an empty data root
|
||||
|
||||
## Raft (experimental ops)
|
||||
## Raft (supported, single-default-DB scope)
|
||||
|
||||
Documented in [distributed.md](distributed.md). Suitable for learning and careful staging; **not** the v1.2.0 HA product tier.
|
||||
Documented in [distributed.md](distributed.md). Supported scope:
|
||||
|
||||
- SQL DML/DDL on **`default` only**
|
||||
- Safe log prefix compact (not full InstallSnapshot)
|
||||
- Failover proven in process e2e tests
|
||||
- 3-node cluster, SQL DML/DDL on **`default` only**
|
||||
- Failover under write load: every acknowledged write survives a leader kill; in-flight writes fail fast — clients must retry
|
||||
- TLS on the raft port and on follower→leader forwarding (`BARADB_RAFT_TLS_*`)
|
||||
- Cold-node recovery via InstallSnapshot (`BARADB_RAFT_SNAP_CHUNK_KB`, `BARADB_RAFT_PEER_STALE_MS`)
|
||||
|
||||
## Newly documented limitations
|
||||
|
||||
- **Legacy non-raft REP replication infers delete from empty value** — the non-raft replication path still treats an empty value as a delete, so inserts into a PK-only table are misapplied over that path (the row vanishes). Use raft replication instead.
|
||||
- **Snapshot-restore ctx staleness** — after an InstallSnapshot restore, HTTP endpoints using the startup-captured ctx may serve stale data until the node is restarted; the `/query` path is fresh per-request. Pre-existing client connections likewise see pre-restore state — reconnect after a restore.
|
||||
|
||||
## Operational requirements
|
||||
|
||||
|
||||
@@ -23,9 +23,12 @@ nim c -d:ssl --threads:on --path:src -r tests/test_schema_persist.nim
|
||||
./scripts/backup-restore-drill.sh
|
||||
DRILL_PORT=19482 ./scripts/backup-restore-drill.sh
|
||||
|
||||
# Optional cluster e2e (experimental tier)
|
||||
# Cluster e2e (raft supported tier — all five suites)
|
||||
./tests/raft_e2e_test
|
||||
./tests/raft_writes_e2e_test
|
||||
./tests/raft_failover_load_e2e_test
|
||||
./tests/raft_tls_e2e_test
|
||||
./tests/raft_coldnode_e2e_test
|
||||
```
|
||||
|
||||
## Production compose
|
||||
|
||||
Reference in New Issue
Block a user