docs: B-tree index persistence plan

docs/superpowers/plans/2026-07-30-btree-index-persistence.md

@@ -0,0 +1,86 @@
+# B-tree Index Persistence Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Standalone `CREATE [UNIQUE] INDEX` (B-tree) indexes survive a server restart — same pattern as the C1 engine persistence (FTS/HNSW/graph, commits 6703ca2..8df0e02).
+
+**Background (verified facts):**
+- CREATE INDEX B-tree branch: `src/barabadb/query/executor.nim` (~line 1376-1390, after the FTS/HNSW branches): builds `ctx.btrees[colKey]` from an execScan, persists NOTHING. After restart the index is gone (planner silently falls back to full scans — performance gap, results stay correct).
+- PK/UNIQUE-from-table-DDL B-trees DO survive: their definitions live in persisted table DDL and `rebuildSecondaryIndexes` (exec/schema.nim:134-168) repopulates them at startup.
+- `_schema:indexes:` prefix exists ONLY as a never-written delete in DROP INDEX (executor.nim:1437) — do not reuse it; use a new consistent prefix.
+- C1 established: `restoreEngines(ctx)` in executor.nim scans `_schema:ftsidx:`/`_schema:vecidx:` prefixes and replays DDL via `executeQueryImpl` (per-key try/except + warn); graphs via loader. Prefix consts live in exec/schema.nim (SchemaFtsIndexPrefix etc.).
+- Replay of the B-tree CREATE INDEX branch is idempotent: it replaces `ctx.btrees[colKey]` with a fresh index and repopulates from scan — no double-insert risk against rebuildSecondaryIndexes.
+- Parser facts: `parseCreateIndex` (parser.nim:1302+) — name optional (`CREATE INDEX ON t (c)` legal, ciName empty → colKey default), UNIQUE optional (`CREATE UNIQUE INDEX ...`).
+- DROP INDEX btree branch (executor.nim:~1392-1439): matches `key == stmt.diName or key.endsWith("." & stmt.diName)` in ctx.btrees, deletes in-memory entry; else-branch deletes `_schema:indexes:<name>` (dead path).
+- DROP TABLE sweep (executor.nim:~863-886) already deletes in-memory fts/vec entries + their `_schema:` keys for the table prefix — extend identically.
+
+**Global constraints:**
+- TDD: failing test FIRST in `tests/test_schema_persist.nim` (follow its conventions), watch it fail for the right reason, then implement.
+- Test commands: `nim c -d:ssl --threads:on --path:src -o:tests/test_schema_persist tests/test_schema_persist.nim && ./tests/test_schema_persist` AND `nim c -d:ssl --threads:on --path:src -o:tests/test_all tests/test_all.nim && ./tests/test_all` — both exit 0.
+- No behavior changes otherwise; old DBs unaffected (zero-key scan no-op); startup never fails (restoreEngines' existing per-key try/except covers the new prefix).
+- Commits after green; source files only; no push (controller merges+pushes).
+
+---
+
+### Task 1: B-tree index persistence
+
+**Files:**
+- Modify: `src/barabadb/query/exec/schema.nim` (new prefix const sibling)
+- Modify: `src/barabadb/query/executor.nim` (persist in btree branch, restoreEngines scan, DROP INDEX + DROP TABLE cleanup)
+- Test: `tests/test_schema_persist.nim`
+
+**Interfaces:**
+- Key format: `_schema:btreeidx:<colKey>` → reconstructed DDL:
+  - named: `CREATE [UNIQUE ]INDEX <ciName> ON <table> (<cols join ", ">)`
+  - unnamed: `CREATE [UNIQUE ]INDEX ON <table> (<cols join ", ">)` (nameless replayable form, same as the C1 fix)
+  - UNIQUE preserved iff `stmt.ciUnique` (check the actual AST field name in query/ast.nim).
+- `SchemaBtreeIndexPrefix* = "_schema:btreeidx:"` in exec/schema.nim next to the C1 consts.
+
+- [ ] **Step 1: Failing tests (write all three)**
+
+```nim
+  test "B-tree index survives reopen":
+    # create table users(id INTEGER PRIMARY KEY, name TEXT, age INTEGER)
+    # insert 3 rows; CREATE INDEX users_age ON users (age)
+    # verify index works pre-restart: check "users.age" in ctx.btrees
+    # close; reopen; check "users.age" in ctx2.btrees
+    # verify the planner/lookup can use it: SELECT with WHERE age = ... succeeds
+    # insert another row post-reopen, verify ctx2.btrees["users.age"] lookup sees it
+
+  test "Unnamed B-tree index survives reopen":
+    # same flow with CREATE INDEX ON users (age) — no name
+
+  test "DROP INDEX removes B-tree index and its schema key":
+    # create + index + DROP INDEX users_age; close; reopen
+    # check "users.age" notin ctx2.btrees (no ghost rebuild)
+```
+
+- [ ] **Step 2: Run, watch them fail**
+
+Expected: first two FAIL (`"users.age" in ctx2.btrees` false after reopen); third may pass or fail depending on current DROP behavior — record which.
+
+- [ ] **Step 3: Implement**
+
+1. exec/schema.nim: add `SchemaBtreeIndexPrefix* = "_schema:btreeidx:"` next to the C1 consts.
+2. executor.nim btree CREATE INDEX branch (after the population loop, before return): persist the reconstructed DDL per the format above (UNIQUE iff the AST says so; name clause only when ciName non-empty).
+3. restoreEngines: add the `_schema:btreeidx:` prefix to the scan/replay (same executeQueryImpl path).
+4. DROP INDEX: in the btree found-branch, also `ctx.db.delete(SchemaBtreeIndexPrefix & targetKey)`; leave the dead `_schema:indexes:` else-branch untouched.
+5. DROP TABLE: extend the existing engine sweep to also delete `_schema:btreeidx:` keys with the `dropName & "."` prefix.
+
+- [ ] **Step 4: Run, watch them pass**
+
+Both test commands; expected PASS + test_all green (461+ [OK]).
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/barabadb/query/exec/schema.nim src/barabadb/query/executor.nim tests/test_schema_persist.nim
+git commit -m "feat(persist): standalone B-tree indexes survive restart"
+```
+
+---
+
+## Self-Review Notes
+
+- The pattern is proven (C1 fts/vec tasks); the only new surface is the btree DROP-branch key deletion and UNIQUE flag handling — tests pin both.
+- UNIQUE flag: verify the AST field name before writing code (grep `ciUnique\|ciKind` src/barabadb/query/ast.nim); if no unique flag exists on the index AST, note it and persist without UNIQUE (and say so in the report).