87 lines
5.8 KiB
Markdown
87 lines
5.8 KiB
Markdown
# B-tree Index Persistence Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax.
|
|
|
|
**Goal:** Standalone `CREATE [UNIQUE] INDEX` (B-tree) indexes survive a server restart — same pattern as the C1 engine persistence (FTS/HNSW/graph, commits 6703ca2..8df0e02).
|
|
|
|
**Background (verified facts):**
|
|
- CREATE INDEX B-tree branch: `src/barabadb/query/executor.nim` (~line 1376-1390, after the FTS/HNSW branches): builds `ctx.btrees[colKey]` from an execScan, persists NOTHING. After restart the index is gone (planner silently falls back to full scans — performance gap, results stay correct).
|
|
- PK/UNIQUE-from-table-DDL B-trees DO survive: their definitions live in persisted table DDL and `rebuildSecondaryIndexes` (exec/schema.nim:134-168) repopulates them at startup.
|
|
- `_schema:indexes:` prefix exists ONLY as a never-written delete in DROP INDEX (executor.nim:1437) — do not reuse it; use a new consistent prefix.
|
|
- C1 established: `restoreEngines(ctx)` in executor.nim scans `_schema:ftsidx:`/`_schema:vecidx:` prefixes and replays DDL via `executeQueryImpl` (per-key try/except + warn); graphs via loader. Prefix consts live in exec/schema.nim (SchemaFtsIndexPrefix etc.).
|
|
- Replay of the B-tree CREATE INDEX branch is idempotent: it replaces `ctx.btrees[colKey]` with a fresh index and repopulates from scan — no double-insert risk against rebuildSecondaryIndexes.
|
|
- Parser facts: `parseCreateIndex` (parser.nim:1302+) — name optional (`CREATE INDEX ON t (c)` legal, ciName empty → colKey default), UNIQUE optional (`CREATE UNIQUE INDEX ...`).
|
|
- DROP INDEX btree branch (executor.nim:~1392-1439): matches `key == stmt.diName or key.endsWith("." & stmt.diName)` in ctx.btrees, deletes in-memory entry; else-branch deletes `_schema:indexes:<name>` (dead path).
|
|
- DROP TABLE sweep (executor.nim:~863-886) already deletes in-memory fts/vec entries + their `_schema:` keys for the table prefix — extend identically.
|
|
|
|
**Global constraints:**
|
|
- TDD: failing test FIRST in `tests/test_schema_persist.nim` (follow its conventions), watch it fail for the right reason, then implement.
|
|
- Test commands: `nim c -d:ssl --threads:on --path:src -o:tests/test_schema_persist tests/test_schema_persist.nim && ./tests/test_schema_persist` AND `nim c -d:ssl --threads:on --path:src -o:tests/test_all tests/test_all.nim && ./tests/test_all` — both exit 0.
|
|
- No behavior changes otherwise; old DBs unaffected (zero-key scan no-op); startup never fails (restoreEngines' existing per-key try/except covers the new prefix).
|
|
- Commits after green; source files only; no push (controller merges+pushes).
|
|
|
|
---
|
|
|
|
### Task 1: B-tree index persistence
|
|
|
|
**Files:**
|
|
- Modify: `src/barabadb/query/exec/schema.nim` (new prefix const sibling)
|
|
- Modify: `src/barabadb/query/executor.nim` (persist in btree branch, restoreEngines scan, DROP INDEX + DROP TABLE cleanup)
|
|
- Test: `tests/test_schema_persist.nim`
|
|
|
|
**Interfaces:**
|
|
- Key format: `_schema:btreeidx:<colKey>` → reconstructed DDL:
|
|
- named: `CREATE [UNIQUE ]INDEX <ciName> ON <table> (<cols join ", ">)`
|
|
- unnamed: `CREATE [UNIQUE ]INDEX ON <table> (<cols join ", ">)` (nameless replayable form, same as the C1 fix)
|
|
- UNIQUE preserved iff `stmt.ciUnique` (check the actual AST field name in query/ast.nim).
|
|
- `SchemaBtreeIndexPrefix* = "_schema:btreeidx:"` in exec/schema.nim next to the C1 consts.
|
|
|
|
- [ ] **Step 1: Failing tests (write all three)**
|
|
|
|
```nim
|
|
test "B-tree index survives reopen":
|
|
# create table users(id INTEGER PRIMARY KEY, name TEXT, age INTEGER)
|
|
# insert 3 rows; CREATE INDEX users_age ON users (age)
|
|
# verify index works pre-restart: check "users.age" in ctx.btrees
|
|
# close; reopen; check "users.age" in ctx2.btrees
|
|
# verify the planner/lookup can use it: SELECT with WHERE age = ... succeeds
|
|
# insert another row post-reopen, verify ctx2.btrees["users.age"] lookup sees it
|
|
|
|
test "Unnamed B-tree index survives reopen":
|
|
# same flow with CREATE INDEX ON users (age) — no name
|
|
|
|
test "DROP INDEX removes B-tree index and its schema key":
|
|
# create + index + DROP INDEX users_age; close; reopen
|
|
# check "users.age" notin ctx2.btrees (no ghost rebuild)
|
|
```
|
|
|
|
- [ ] **Step 2: Run, watch them fail**
|
|
|
|
Expected: first two FAIL (`"users.age" in ctx2.btrees` false after reopen); third may pass or fail depending on current DROP behavior — record which.
|
|
|
|
- [ ] **Step 3: Implement**
|
|
|
|
1. exec/schema.nim: add `SchemaBtreeIndexPrefix* = "_schema:btreeidx:"` next to the C1 consts.
|
|
2. executor.nim btree CREATE INDEX branch (after the population loop, before return): persist the reconstructed DDL per the format above (UNIQUE iff the AST says so; name clause only when ciName non-empty).
|
|
3. restoreEngines: add the `_schema:btreeidx:` prefix to the scan/replay (same executeQueryImpl path).
|
|
4. DROP INDEX: in the btree found-branch, also `ctx.db.delete(SchemaBtreeIndexPrefix & targetKey)`; leave the dead `_schema:indexes:` else-branch untouched.
|
|
5. DROP TABLE: extend the existing engine sweep to also delete `_schema:btreeidx:` keys with the `dropName & "."` prefix.
|
|
|
|
- [ ] **Step 4: Run, watch them pass**
|
|
|
|
Both test commands; expected PASS + test_all green (461+ [OK]).
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/barabadb/query/exec/schema.nim src/barabadb/query/executor.nim tests/test_schema_persist.nim
|
|
git commit -m "feat(persist): standalone B-tree indexes survive restart"
|
|
```
|
|
|
|
---
|
|
|
|
## Self-Review Notes
|
|
|
|
- The pattern is proven (C1 fts/vec tasks); the only new surface is the btree DROP-branch key deletion and UNIQUE flag handling — tests pin both.
|
|
- UNIQUE flag: verify the AST field name before writing code (grep `ciUnique\|ciKind` src/barabadb/query/ast.nim); if no unique flag exists on the index AST, note it and persist without UNIQUE (and say so in the report).
|