Baradb/docs/superpowers/plans/2026-07-30-btree-index-persistence.md

B-tree Index Persistence Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans. Steps use checkbox (- [ ]) syntax.

Goal: Standalone CREATE [UNIQUE] INDEX (B-tree) indexes survive a server restart — same pattern as the C1 engine persistence (FTS/HNSW/graph, commits 6703ca2..8df0e02).

Background (verified facts):

• CREATE INDEX B-tree branch: src/barabadb/query/executor.nim (~line 1376-1390, after the FTS/HNSW branches): builds ctx.btrees[colKey] from an execScan, persists NOTHING. After restart the index is gone (planner silently falls back to full scans — performance gap, results stay correct).
• PK/UNIQUE-from-table-DDL B-trees DO survive: their definitions live in persisted table DDL and rebuildSecondaryIndexes (exec/schema.nim:134-168) repopulates them at startup.
• _schema:indexes: prefix exists ONLY as a never-written delete in DROP INDEX (executor.nim:1437) — do not reuse it; use a new consistent prefix.
• C1 established: restoreEngines(ctx) in executor.nim scans _schema:ftsidx:/_schema:vecidx: prefixes and replays DDL via executeQueryImpl (per-key try/except + warn); graphs via loader. Prefix consts live in exec/schema.nim (SchemaFtsIndexPrefix etc.).
• Replay of the B-tree CREATE INDEX branch is idempotent: it replaces ctx.btrees[colKey] with a fresh index and repopulates from scan — no double-insert risk against rebuildSecondaryIndexes.
• Parser facts: parseCreateIndex (parser.nim:1302+) — name optional (CREATE INDEX ON t (c) legal, ciName empty → colKey default), UNIQUE optional (CREATE UNIQUE INDEX ...).
• DROP INDEX btree branch (executor.nim:~1392-1439): matches key == stmt.diName or key.endsWith("." & stmt.diName) in ctx.btrees, deletes in-memory entry; else-branch deletes _schema:indexes:<name> (dead path).
• DROP TABLE sweep (executor.nim:~863-886) already deletes in-memory fts/vec entries + their _schema: keys for the table prefix — extend identically.

Global constraints:

• TDD: failing test FIRST in tests/test_schema_persist.nim (follow its conventions), watch it fail for the right reason, then implement.
• Test commands: nim c -d:ssl --threads:on --path:src -o:tests/test_schema_persist tests/test_schema_persist.nim && ./tests/test_schema_persist AND nim c -d:ssl --threads:on --path:src -o:tests/test_all tests/test_all.nim && ./tests/test_all — both exit 0.
• No behavior changes otherwise; old DBs unaffected (zero-key scan no-op); startup never fails (restoreEngines' existing per-key try/except covers the new prefix).
• Commits after green; source files only; no push (controller merges+pushes).

---

Task 1: B-tree index persistence

Files:

• Modify: src/barabadb/query/exec/schema.nim (new prefix const sibling)
• Modify: src/barabadb/query/executor.nim (persist in btree branch, restoreEngines scan, DROP INDEX + DROP TABLE cleanup)
• Test: tests/test_schema_persist.nim

Interfaces:

• Key format: _schema:btreeidx:<colKey> → reconstructed DDL:

  • named: CREATE [UNIQUE ]INDEX <ciName> ON <table> (<cols join ", ">)
  • unnamed: CREATE [UNIQUE ]INDEX ON <table> (<cols join ", ">) (nameless replayable form, same as the C1 fix)
  • UNIQUE preserved iff stmt.ciUnique (check the actual AST field name in query/ast.nim).

• SchemaBtreeIndexPrefix* = "_schema:btreeidx:" in exec/schema.nim next to the C1 consts.

• Step 1: Failing tests (write all three)

  test "B-tree index survives reopen":
    # create table users(id INTEGER PRIMARY KEY, name TEXT, age INTEGER)
    # insert 3 rows; CREATE INDEX users_age ON users (age)
    # verify index works pre-restart: check "users.age" in ctx.btrees
    # close; reopen; check "users.age" in ctx2.btrees
    # verify the planner/lookup can use it: SELECT with WHERE age = ... succeeds
    # insert another row post-reopen, verify ctx2.btrees["users.age"] lookup sees it

  test "Unnamed B-tree index survives reopen":
    # same flow with CREATE INDEX ON users (age) — no name

  test "DROP INDEX removes B-tree index and its schema key":
    # create + index + DROP INDEX users_age; close; reopen
    # check "users.age" notin ctx2.btrees (no ghost rebuild)

• Step 2: Run, watch them fail

Expected: first two FAIL ("users.age" in ctx2.btrees false after reopen); third may pass or fail depending on current DROP behavior — record which.

• Step 3: Implement

1. exec/schema.nim: add SchemaBtreeIndexPrefix* = "_schema:btreeidx:" next to the C1 consts.
2. executor.nim btree CREATE INDEX branch (after the population loop, before return): persist the reconstructed DDL per the format above (UNIQUE iff the AST says so; name clause only when ciName non-empty).
3. restoreEngines: add the _schema:btreeidx: prefix to the scan/replay (same executeQueryImpl path).
4. DROP INDEX: in the btree found-branch, also ctx.db.delete(SchemaBtreeIndexPrefix & targetKey); leave the dead _schema:indexes: else-branch untouched.
5. DROP TABLE: extend the existing engine sweep to also delete _schema:btreeidx: keys with the dropName & "." prefix.

• Step 4: Run, watch them pass

Both test commands; expected PASS + test_all green (461+ [OK]).

• Step 5: Commit

git add src/barabadb/query/exec/schema.nim src/barabadb/query/executor.nim tests/test_schema_persist.nim
git commit -m "feat(persist): standalone B-tree indexes survive restart"

---

Self-Review Notes

• The pattern is proven (C1 fts/vec tasks); the only new surface is the btree DROP-branch key deletion and UNIQUE flag handling — tests pin both.
• UNIQUE flag: verify the AST field name before writing code (grep ciUnique\|ciKind src/barabadb/query/ast.nim); if no unique flag exists on the index AST, note it and persist without UNIQUE (and say so in the report).