diff --git a/docs/superpowers/plans/2026-07-30-btree-index-persistence.md b/docs/superpowers/plans/2026-07-30-btree-index-persistence.md new file mode 100644 index 0000000..4cd3edf --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-btree-index-persistence.md @@ -0,0 +1,86 @@ +# B-tree Index Persistence Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax. + +**Goal:** Standalone `CREATE [UNIQUE] INDEX` (B-tree) indexes survive a server restart — same pattern as the C1 engine persistence (FTS/HNSW/graph, commits 6703ca2..8df0e02). + +**Background (verified facts):** +- CREATE INDEX B-tree branch: `src/barabadb/query/executor.nim` (~line 1376-1390, after the FTS/HNSW branches): builds `ctx.btrees[colKey]` from an execScan, persists NOTHING. After restart the index is gone (planner silently falls back to full scans — performance gap, results stay correct). +- PK/UNIQUE-from-table-DDL B-trees DO survive: their definitions live in persisted table DDL and `rebuildSecondaryIndexes` (exec/schema.nim:134-168) repopulates them at startup. +- `_schema:indexes:` prefix exists ONLY as a never-written delete in DROP INDEX (executor.nim:1437) — do not reuse it; use a new consistent prefix. +- C1 established: `restoreEngines(ctx)` in executor.nim scans `_schema:ftsidx:`/`_schema:vecidx:` prefixes and replays DDL via `executeQueryImpl` (per-key try/except + warn); graphs via loader. Prefix consts live in exec/schema.nim (SchemaFtsIndexPrefix etc.). +- Replay of the B-tree CREATE INDEX branch is idempotent: it replaces `ctx.btrees[colKey]` with a fresh index and repopulates from scan — no double-insert risk against rebuildSecondaryIndexes. +- Parser facts: `parseCreateIndex` (parser.nim:1302+) — name optional (`CREATE INDEX ON t (c)` legal, ciName empty → colKey default), UNIQUE optional (`CREATE UNIQUE INDEX ...`). +- DROP INDEX btree branch (executor.nim:~1392-1439): matches `key == stmt.diName or key.endsWith("." & stmt.diName)` in ctx.btrees, deletes in-memory entry; else-branch deletes `_schema:indexes:` (dead path). +- DROP TABLE sweep (executor.nim:~863-886) already deletes in-memory fts/vec entries + their `_schema:` keys for the table prefix — extend identically. + +**Global constraints:** +- TDD: failing test FIRST in `tests/test_schema_persist.nim` (follow its conventions), watch it fail for the right reason, then implement. +- Test commands: `nim c -d:ssl --threads:on --path:src -o:tests/test_schema_persist tests/test_schema_persist.nim && ./tests/test_schema_persist` AND `nim c -d:ssl --threads:on --path:src -o:tests/test_all tests/test_all.nim && ./tests/test_all` — both exit 0. +- No behavior changes otherwise; old DBs unaffected (zero-key scan no-op); startup never fails (restoreEngines' existing per-key try/except covers the new prefix). +- Commits after green; source files only; no push (controller merges+pushes). + +--- + +### Task 1: B-tree index persistence + +**Files:** +- Modify: `src/barabadb/query/exec/schema.nim` (new prefix const sibling) +- Modify: `src/barabadb/query/executor.nim` (persist in btree branch, restoreEngines scan, DROP INDEX + DROP TABLE cleanup) +- Test: `tests/test_schema_persist.nim` + +**Interfaces:** +- Key format: `_schema:btreeidx:` → reconstructed DDL: + - named: `CREATE [UNIQUE ]INDEX ON ()` + - unnamed: `CREATE [UNIQUE ]INDEX ON
()` (nameless replayable form, same as the C1 fix) + - UNIQUE preserved iff `stmt.ciUnique` (check the actual AST field name in query/ast.nim). +- `SchemaBtreeIndexPrefix* = "_schema:btreeidx:"` in exec/schema.nim next to the C1 consts. + +- [ ] **Step 1: Failing tests (write all three)** + +```nim + test "B-tree index survives reopen": + # create table users(id INTEGER PRIMARY KEY, name TEXT, age INTEGER) + # insert 3 rows; CREATE INDEX users_age ON users (age) + # verify index works pre-restart: check "users.age" in ctx.btrees + # close; reopen; check "users.age" in ctx2.btrees + # verify the planner/lookup can use it: SELECT with WHERE age = ... succeeds + # insert another row post-reopen, verify ctx2.btrees["users.age"] lookup sees it + + test "Unnamed B-tree index survives reopen": + # same flow with CREATE INDEX ON users (age) — no name + + test "DROP INDEX removes B-tree index and its schema key": + # create + index + DROP INDEX users_age; close; reopen + # check "users.age" notin ctx2.btrees (no ghost rebuild) +``` + +- [ ] **Step 2: Run, watch them fail** + +Expected: first two FAIL (`"users.age" in ctx2.btrees` false after reopen); third may pass or fail depending on current DROP behavior — record which. + +- [ ] **Step 3: Implement** + +1. exec/schema.nim: add `SchemaBtreeIndexPrefix* = "_schema:btreeidx:"` next to the C1 consts. +2. executor.nim btree CREATE INDEX branch (after the population loop, before return): persist the reconstructed DDL per the format above (UNIQUE iff the AST says so; name clause only when ciName non-empty). +3. restoreEngines: add the `_schema:btreeidx:` prefix to the scan/replay (same executeQueryImpl path). +4. DROP INDEX: in the btree found-branch, also `ctx.db.delete(SchemaBtreeIndexPrefix & targetKey)`; leave the dead `_schema:indexes:` else-branch untouched. +5. DROP TABLE: extend the existing engine sweep to also delete `_schema:btreeidx:` keys with the `dropName & "."` prefix. + +- [ ] **Step 4: Run, watch them pass** + +Both test commands; expected PASS + test_all green (461+ [OK]). + +- [ ] **Step 5: Commit** + +```bash +git add src/barabadb/query/exec/schema.nim src/barabadb/query/executor.nim tests/test_schema_persist.nim +git commit -m "feat(persist): standalone B-tree indexes survive restart" +``` + +--- + +## Self-Review Notes + +- The pattern is proven (C1 fts/vec tasks); the only new surface is the btree DROP-branch key deletion and UNIQUE flag handling — tests pin both. +- UNIQUE flag: verify the AST field name before writing code (grep `ciUnique\|ciKind` src/barabadb/query/ast.nim); if no unique flag exists on the index AST, note it and persist without UNIQUE (and say so in the report).