feat: harden storage, schema persistence, fair benches, fix wire crash

Core storage: hash MemTable, WAL group commit, L0 compaction rebuild,
reader-writer lock, and a global StorageGate so HTTP workers and TCP
share the LSM safely under multi-thread access.

Schema: durable CREATE/ALTER/DROP under _schema:tables:* with full LSM
restore on open. Executor types/values/schema split into query/exec/.

Wire protocol: switch default MM to ARC — ORC cycle collector segfaulted
after ~20 async INSERTs. Fair multi-tier benchmarks (SQLite/HTTP/wire/PG)
and honesty docs for mixed-tier comparisons.

benchmarks/FAIR_COMPARISON.md

@@ -0,0 +1,93 @@
+# Fair Benchmark Results
+
+Generated: **2026-07-18 13:53:46 UTC**
+
+## Methodology
+
+- Tier `embedded`: in-process only (BaraDB LSM from nimble bench JSON; SQLite via Python sqlite3).
+- Tier `client_server`: network SQL (BaraDB HTTP /query; BaraDB binary wire TCP; PostgreSQL via psycopg2).
+- `sql_insert_row`: one INSERT statement per row (chatty).
+- `sql_insert_batch`: multi-row INSERT with batch size 50 (same SQL style across systems).
+- PostgreSQL: synchronous_commit=on|off; SQLite: PRAGMA synchronous FULL|OFF.
+- BaraDB WAL modes appear only if you ran benchmarks/bench_all.nim (WAL-* rows).
+- Never claim 'Nx faster than Postgres' using embedded BaraDB numbers.
+
+**Do not compare numbers across tiers.** Embedded storage is not the same
+workload as client-server SQL over the network.
+
+## Tier: `embedded`
+
+| Bench | System | ops/s | seconds | n | notes |
+|-------|--------|------:|--------:|--:|-------|
+| kv_write | `baradb_lsm_embedded` | 41.50K | 2.410 | 100000 | benchmark_results.json |
+| kv_read | `baradb_lsm_embedded` | 3.77M | 0.026 | 100000 | benchmark_results.json |
+| wal_none | `baradb_lsm_embedded` | 232.65K | 0.215 | 50000 | benchmark_results.json |
+| wal_group64 | `baradb_lsm_embedded` | 42.25K | 1.183 | 50000 | benchmark_results.json |
+| wal_group256 | `baradb_lsm_embedded` | 107.16K | 0.467 | 50000 | benchmark_results.json |
+| wal_every | `baradb_lsm_embedded` | 825.27 | 60.586 | 50000 | benchmark_results.json |
+| kv_write | `sqlite_off` | 402.61K | 0.002 | 1000 | PRAGMA synchronous=OFF |
+| kv_read | `sqlite_off` | 195.60K | 0.005 | 1000 |  |
+| sql_insert_batch | `sqlite_off` | 559.66K | 0.002 | 1000 | multi-row INSERT batch=50, sync=OFF |
+| kv_write | `sqlite_full` | 272.62K | 0.004 | 1000 | PRAGMA synchronous=FULL |
+| kv_read | `sqlite_full` | 196.16K | 0.005 | 1000 |  |
+| sql_insert_batch | `sqlite_full` | 370.18K | 0.003 | 1000 | multi-row INSERT batch=50, sync=FULL |
+
+### Same-bench ratios (`embedded`)
+
+**kv_read** (fastest: `baradb_lsm_embedded` @ 3.77M/s)
+
+| System | Relative to fastest |
+|--------|--------------------:|
+| `baradb_lsm_embedded` | 1.00x |
+| `sqlite_full` | 0.05x |
+| `sqlite_off` | 0.05x |
+
+**kv_write** (fastest: `sqlite_off` @ 402.61K/s)
+
+| System | Relative to fastest |
+|--------|--------------------:|
+| `sqlite_off` | 1.00x |
+| `sqlite_full` | 0.68x |
+| `baradb_lsm_embedded` | 0.10x |
+
+**sql_insert_batch** (fastest: `sqlite_off` @ 559.66K/s)
+
+| System | Relative to fastest |
+|--------|--------------------:|
+| `sqlite_off` | 1.00x |
+| `sqlite_full` | 0.66x |
+
+## Tier: `client_server`
+
+| Bench | System | ops/s | seconds | n | notes |
+|-------|--------|------:|--------:|--:|-------|
+| sql_insert_row | `baradb_http` | 979.31 | 0.511 | 500 |  |
+| sql_select_row | `baradb_http` | 235.20 | 2.126 | 500 |  |
+| sql_insert_batch | `baradb_http` | 10.19K | 0.049 | 500 | multi-row INSERT batch=50 |
+| sql_insert_row | `baradb_wire` | 4.65K | 0.108 | 500 | binary wire protocol |
+| sql_select_row | `baradb_wire` | 642.04 | 0.779 | 500 |  |
+| sql_insert_batch | `baradb_wire` | 20.47K | 0.024 | 500 | multi-row INSERT batch=50 |
+
+### Same-bench ratios (`client_server`)
+
+**sql_insert_batch** (fastest: `baradb_wire` @ 20.47K/s)
+
+| System | Relative to fastest |
+|--------|--------------------:|
+| `baradb_wire` | 1.00x |
+| `baradb_http` | 0.50x |
+
+**sql_insert_row** (fastest: `baradb_wire` @ 4.65K/s)
+
+| System | Relative to fastest |
+|--------|--------------------:|
+| `baradb_wire` | 1.00x |
+| `baradb_http` | 0.21x |
+
+**sql_select_row** (fastest: `baradb_wire` @ 642.04/s)
+
+| System | Relative to fastest |
+|--------|--------------------:|
+| `baradb_wire` | 1.00x |
+| `baradb_http` | 0.37x |
+

benchmarks/README.md

@@ -0,0 +1,80 @@
+# BaraDB Benchmarks
+
+## Tiers (read this first)
+
+| Tier | What is measured | Fair peers |
+|------|------------------|------------|
+| **embedded** | In-process storage API | BaraDB LSM ↔ SQLite |
+| **client_server** | Network + query protocol | BaraDB HTTP / **wire** ↔ PostgreSQL |
+
+**Never** quote “BaraDB is Nx faster than Postgres” using embedded LSM numbers.
+That comparison mixes tiers and is meaningless as a product claim.
+
+## Quick start
+
+```bash
+# 1) Embedded micro-benches (Nim, in-process)
+nimble bench
+# or: nim c -d:release -r benchmarks/bench_all.nim
+
+# 2) Optional: start server for client_server tier (HTTP + wire)
+./build/baradadb
+
+# 3) Fair multi-tier suite (Python)
+#    - always: SQLite embedded (+ batch)
+#    - optional: BaraDB HTTP (:9912), wire TCP (:9472), PostgreSQL
+python3 benchmarks/fair_bench.py
+
+# 3) Markdown report
+nimble bench_report
+# or: python3 benchmarks/generate_report.py --fair
+```
+
+Outputs:
+
+- `benchmark_results.json` — Nim embedded suite  
+- `fair_benchmark_results.json` — multi-tier fair suite  
+- `benchmarks/FAIR_COMPARISON.md` — human-readable fair report  
+- `pg_benchmark_results.json` — optional PG-only micro suite  
+
+## Environment
+
+| Variable | Default | Meaning |
+|----------|---------|---------|
+| `FAIR_N_KV` | 20000 | embedded KV ops |
+| `FAIR_N_SQL` | 5000 | SQL loops (HTTP/wire/PG) |
+| `FAIR_BATCH` | 100 | multi-row INSERT batch size |
+| `BARADB_HTTP_HOST` | 127.0.0.1 | HTTP host |
+| `BARADB_HTTP_PORT` | 9912 | HTTP port (`TCP+440`) |
+| `BARADB_WIRE_HOST` | 127.0.0.1 | wire protocol host |
+| `BARADB_WIRE_PORT` | 9472 | wire protocol TCP port |
+| `FAIR_SKIP_HTTP=1` | — | skip BaraDB HTTP |
+| `FAIR_SKIP_WIRE=1` | — | skip BaraDB wire |
+| `FAIR_SKIP_PG=1` | — | skip PostgreSQL |
+| `PGHOST` / `PGUSER` / `PGPASSWORD` / … | — | libpq-style |
+
+## Files
+
+| File | Role |
+|------|------|
+| `bench_all.nim` | Embedded: LSM, WAL modes, BTree, vector, FTS, graph |
+| `fair_bench.py` | Fair multi-tier runner + markdown |
+| `pg_bench.py` | PostgreSQL client-server micro suite |
+| `generate_report.py` | `--fair` report; legacy mixed report without flag |
+| `compare.nim` | **Synthetic** — do not publish |
+| `search_bench.nim` | Search-focused micro suite |
+
+## Durability knobs
+
+- BaraDB: `wal_sync_mode` = `none` \| `group` \| `every` (see WAL-* rows from `bench_all`)  
+- SQLite: `PRAGMA synchronous = OFF` vs `FULL`  
+- PostgreSQL: `synchronous_commit = off` vs `on`  
+
+Match durability stories when claiming write speedups.
+
+## Wire protocol note
+
+The Python wire client (`clients/python`) is exercised by `fair_bench.py`.
+Builds use **`--mm:arc`** (see `nim.cfg`) because Nim **ORC** cycle collection
+crashed under async wire INSERT load (`markGray` SIGSEGV). With ARC, sequential
+wire INSERTs + batch multi-row INSERT are stable.

benchmarks/REAL_COMPARISON.md

@@ -1,38 +1,29 @@
-# BaraDB vs PostgreSQL — Real Benchmark Results
+# Legacy mixed-tier comparison
 
-Generated from actual execution on:
-- **CPU:** AMD Ryzen 9 5900X
-- **PostgreSQL:** 15.18 (local)
-- **BaraDB:** git `42043f3`
+This file used to claim large “speedups” of BaraDB over PostgreSQL by comparing:
 
-## Methodology
+- **PostgreSQL:** client-server (psycopg2, network, SQL)
+- **BaraDB:** in-process LSM (no network, no SQL)
 
-- PostgreSQL: single-row INSERT/SELECT via psycopg2 (client-server overhead included)
-- BaraDB: in-process Nim code (no network overhead)
-- Same dataset sizes for both systems
+That is **not a fair product comparison**.
 
-## Results
+## Use the fair suite instead
 
-| Test | PostgreSQL | BaraDB | Speedup |
-|------|-----------|--------|---------|
-| KV Write (100K) | 16.82K/s (5.946s) | 32.23K/s (3.103s) | 1.9x (BaraDB) |
-| KV Read (100K) | 15.08K/s (6.630s) | 3.95M/s (25.3ms) | 261.9x (BaraDB) |
-| BTree Insert (100K) | 17.66K/s (5.664s) | 2.52M/s (39.7ms) | 142.8x (BaraDB) |
-| BTree Get (100K) | 14.50K/s (6.899s) | 2.34M/s (42.7ms) | 161.4x (BaraDB) |
-| BTree Scan (1K ranges) | 2.39K/s (419.2ms) | 11.03M/s (1.0ms) | 4623.3x (BaraDB) |
-| FTS Index (10K docs) | 17.98K/s (556.3ms) | 119.99K/s (83.3ms) | 6.7x (BaraDB) |
-| FTS Search (1K queries) | 784.12/s (1.275s) | 1.36K/s (734.0ms) | 1.7x (BaraDB) |
+```bash
+nim c -d:release -r benchmarks/bench_all.nim   # embedded BaraDB
+python3 benchmarks/fair_bench.py               # SQLite + optional PG/HTTP
+# report → benchmarks/FAIR_COMPARISON.md
+```
 
-## Summary
+See:
 
-- **Total PostgreSQL time:** 27.389s
-- **Total BaraDB time:** 4.029s
-- **Overall speedup:** BaraDB is **6.8x faster**
+- [`FAIR_COMPARISON.md`](FAIR_COMPARISON.md) — latest multi-tier results  
+- [`README.md`](README.md) — methodology and env vars  
 
-## Notes
+## If you regenerate the legacy report
 
-- PostgreSQL includes network round-trip and SQL parsing overhead per operation.
-- BaraDB runs in-process with zero serialization/network cost.
-- For embedded/single-node use cases, BaraDB shows significant advantage.
-- BaraDB now outperforms PostgreSQL on all tested metrics including FTS search after optimizations.
-- PostgreSQL excels at durability, replication, and complex ACID transactions.
+```bash
+python3 benchmarks/generate_report.py   # without --fair
+```
+
+It will rewrite this file with an explicit **mixed tiers** warning banner.

benchmarks/bench_all.nim

@@ -111,9 +111,11 @@ proc formatOps(ops: int, secs: float64): string =
 
 proc benchLSMTree() =
   echo "=== LSM-Tree Storage ==="
+  echo "  Note: in-process embedded API (no network/SQL). Not comparable to client-server DBs."
   let benchDir = getTempDir() / "baradb_bench_lsm"
   removeDir(benchDir)
-  var db = newLSMTree(benchDir)
+  # Default group-commit WAL (production default)
+  var db = newLSMTree(benchDir, walSyncMode = wsmGroup, walGroupEvery = 64)
 
   # Write benchmark
   let n = 100_000
@@ -124,6 +126,7 @@ proc benchLSMTree() =
   let writeLabel = "LSM-Write"
   recordResult(writeLabel, n, writeTime)
   echo "  Write ", n, " keys: ", writeTime.formatFloat(ffDecimal, 3), "s (", formatOps(n, writeTime), ")", compareResult(writeLabel, currentResults[^1].opsPerSec, previousResults)
+  echo "    fsyncs: ", db.wal.fsyncCount, " (group every 64)"
 
   # Read benchmark
   let readStart = getMonoTime()
@@ -138,6 +141,36 @@ proc benchLSMTree() =
 
   db.close()
 
+proc benchWalDurabilityModes() =
+  ## Fair comparison of WAL durability policies on the same workload.
+  echo "=== WAL Durability Modes (fair micro-bench) ==="
+  echo "  Same N puts, same memtable size; only sync policy differs."
+  let n = 50_000
+  let modes = [
+    (wsmNone, "none", 0),
+    (wsmGroup, "group64", 64),
+    (wsmGroup, "group256", 256),
+    (wsmEvery, "every", 1),
+  ]
+  for (mode, label, ge) in modes:
+    let dir = getTempDir() / ("baradb_bench_wal_" & label)
+    removeDir(dir)
+    var db = newLSMTree(dir, memMaxSize = 64 * 1024 * 1024,
+                        walSyncMode = mode, walGroupEvery = max(1, ge))
+    let t0 = getMonoTime()
+    for i in 0..<n:
+      db.put("k" & $i, cast[seq[byte]]("v" & $i))
+    # Ensure pending group is durable before measuring end-to-end
+    db.wal.sync()
+    let secs = elapsed(t0)
+    let name = "WAL-" & label
+    recordResult(name, n, secs)
+    echo "  ", label, ": ", secs.formatFloat(ffDecimal, 3), "s (",
+         formatOps(n, secs), "), fsyncs=", db.wal.fsyncCount,
+         compareResult(name, currentResults[^1].opsPerSec, previousResults)
+    db.close()
+    removeDir(dir)
+
 proc benchBTree() =
   echo "=== B-Tree Index ==="
   var btree = newBTreeIndex[string, string]()
@@ -331,11 +364,17 @@ proc benchGraph() =
 proc main() =
   echo ""
   echo "╔══════════════════════════════════════════════════╗"
-  echo "║         BaraDB Performance Benchmarks            ║"
+  echo "║     BaraDB Performance Benchmarks (EMBEDDED)     ║"
   echo "╚══════════════════════════════════════════════════╝"
   echo ""
+  echo "Tier: embedded / in-process (no network, no wire SQL)."
+  echo "For fair multi-tier numbers (SQLite / PG / HTTP):"
+  echo "  python3 benchmarks/fair_bench.py"
+  echo ""
   benchLSMTree()
   echo ""
+  benchWalDurabilityModes()
+  echo ""
   benchBTree()
   echo ""
   benchVectorSearch()
@@ -355,6 +394,7 @@ proc main() =
   )
   saveResults(ResultsFile, report)
   echo "Results saved to ", ResultsFile
+  echo "Next: python3 benchmarks/fair_bench.py"
   echo ""
 
 when isMainModule:

benchmarks/compare.nim

@@ -1,4 +1,11 @@
 ## Comparative Benchmarks — BaraDB vs PostgreSQL, Redis, MongoDB
+##
+## ⚠️  SYNTHETIC / PLACEHOLDER: several refTimeSec values are *invented*
+## multipliers, not measured. Do not publish these as real comparisons.
+## Use instead:
+##   nim c -d:release -r benchmarks/bench_all.nim
+##   python3 benchmarks/fair_bench.py
+##   python3 benchmarks/generate_report.py --fair
 import std/times
 import std/random
 import std/strutils

benchmarks/fair_bench.py

@@ -0,0 +1,799 @@
+#!/usr/bin/env python3
+"""
+Fair multi-tier benchmarks for BaraDB.
+
+Tiers (never mix across tiers in a single "speedup" claim):
+
+  1. embedded     — in-process storage (BaraDB LSM from JSON, SQLite)
+  2. client_server — network + SQL
+       • BaraDB HTTP REST
+       • BaraDB wire protocol (Python async client, TCP 9472)
+       • PostgreSQL (psycopg2)
+
+  Within each tier we measure both **row-at-a-time** and **batch multi-row INSERT**.
+
+Usage:
+  # 1) optional: run BaraDB embedded micro-benches first
+  nim c -d:release -r benchmarks/bench_all.nim
+
+  # 2) start server for HTTP/wire tiers (optional)
+  ./build/baradadb
+
+  # 3) fair suite
+  python3 benchmarks/fair_bench.py
+
+  # 4) markdown report
+  python3 benchmarks/generate_report.py --fair
+
+Env:
+  BARADB_HTTP_HOST   default 127.0.0.1
+  BARADB_HTTP_PORT   default 9912  (TCP 9472 + 440)
+  BARADB_WIRE_HOST   default 127.0.0.1
+  BARADB_WIRE_PORT   default 9472
+  PGHOST / PGPORT / PGDATABASE / PGUSER / PGPASSWORD
+  FAIR_N_KV          default 20000
+  FAIR_N_SQL         default 5000
+  FAIR_BATCH         default 100   (rows per multi-row INSERT)
+  FAIR_SKIP_PG=1     skip PostgreSQL
+  FAIR_SKIP_HTTP=1   skip BaraDB HTTP
+  FAIR_SKIP_WIRE=1   skip BaraDB wire protocol
+"""
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import sqlite3
+import sys
+import tempfile
+import time
+import urllib.error
+import urllib.request
+from datetime import datetime, timezone
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+OUT_JSON = ROOT / "fair_benchmark_results.json"
+BARA_JSON = ROOT / "benchmark_results.json"
+CLIENTS_PY = ROOT / "clients" / "python"
+
+N_KV = int(os.environ.get("FAIR_N_KV", "20000"))
+N_SQL = int(os.environ.get("FAIR_N_SQL", "5000"))
+BATCH = int(os.environ.get("FAIR_BATCH", "100"))
+HTTP_HOST = os.environ.get("BARADB_HTTP_HOST", "127.0.0.1")
+HTTP_PORT = int(os.environ.get("BARADB_HTTP_PORT", "9912"))
+WIRE_HOST = os.environ.get("BARADB_WIRE_HOST", "127.0.0.1")
+WIRE_PORT = int(os.environ.get("BARADB_WIRE_PORT", "9472"))
+
+
+def now_iso() -> str:
+    return datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
+
+
+def result(name: str, system: str, tier: str, ops: int, seconds: float, **extra):
+    ops_s = ops / seconds if seconds > 0 else 0.0
+    r = {
+        "name": name,
+        "system": system,
+        "tier": tier,
+        "ops": ops,
+        "seconds": seconds,
+        "opsPerSec": ops_s,
+        "timestamp": now_iso(),
+    }
+    r.update(extra)
+    return r
+
+
+def fmt_ops(x: float) -> str:
+    if x >= 1_000_000:
+        return f"{x/1_000_000:.2f}M"
+    if x >= 1_000:
+        return f"{x/1_000:.2f}K"
+    return f"{x:.2f}"
+
+
+# ─── Tier 1: Embedded ───────────────────────────────────────────────
+
+
+def load_baradb_embedded() -> list[dict]:
+    """Map bench_all.nim LSM results into fair embedded tier."""
+    if not BARA_JSON.exists():
+        print("  [skip] benchmark_results.json missing — run: nimble bench")
+        return []
+    data = json.loads(BARA_JSON.read_text())
+    name_map = {
+        "LSM-Write": "kv_write",
+        "LSM-Read": "kv_read",
+        "WAL-none": "wal_none",
+        "WAL-group64": "wal_group64",
+        "WAL-group256": "wal_group256",
+        "WAL-every": "wal_every",
+    }
+    out = []
+    for r in data.get("results", []):
+        mapped = name_map.get(r.get("name"))
+        if not mapped:
+            continue
+        out.append(
+            result(
+                mapped,
+                "baradb_lsm_embedded",
+                "embedded",
+                r.get("ops", 0),
+                r.get("seconds", 0.0),
+                source="benchmark_results.json",
+                gitSha=data.get("gitSha", ""),
+            )
+        )
+    return out
+
+
+def multi_values_sql(start: int, count: int) -> str:
+    """Build VALUES (...),(...),... for multi-row INSERT."""
+    parts = [f"({i}, 'value_{i}')" for i in range(start, start + count)]
+    return ",".join(parts)
+
+
+def bench_sqlite_embedded(n: int = N_KV) -> list[dict]:
+    """SQLite in-process — fair peer for BaraDB embedded LSM."""
+    out = []
+    fd, path = tempfile.mkstemp(suffix=".db")
+    os.close(fd)
+    os.unlink(path)
+
+    # --- durability: FULL (fsync) vs OFF ---
+    for mode, label in (("OFF", "sqlite_off"), ("FULL", "sqlite_full")):
+        if os.path.exists(path):
+            os.unlink(path)
+        conn = sqlite3.connect(path)
+        cur = conn.cursor()
+        cur.execute(f"PRAGMA synchronous = {mode}")
+        cur.execute("PRAGMA journal_mode = WAL")
+        cur.execute("CREATE TABLE kv (k TEXT PRIMARY KEY, v TEXT)")
+        conn.commit()
+
+        t0 = time.perf_counter()
+        for i in range(n):
+            cur.execute("INSERT INTO kv(k,v) VALUES(?,?)", (f"key_{i}", f"value_{i}"))
+        conn.commit()
+        w = time.perf_counter() - t0
+        out.append(
+            result(
+                "kv_write",
+                label,
+                "embedded",
+                n,
+                w,
+                durable=mode == "FULL",
+                note=f"PRAGMA synchronous={mode}",
+            )
+        )
+
+        t0 = time.perf_counter()
+        found = 0
+        for i in range(n):
+            cur.execute("SELECT v FROM kv WHERE k=?", (f"key_{i}",))
+            if cur.fetchone():
+                found += 1
+        r = time.perf_counter() - t0
+        out.append(
+            result(
+                "kv_read",
+                label,
+                "embedded",
+                n,
+                r,
+                found=found,
+                durable=mode == "FULL",
+            )
+        )
+
+        # Batch multi-row INSERT into SQL table (embedded SQL peer for batch)
+        cur.execute("DROP TABLE IF EXISTS fair_batch")
+        cur.execute("CREATE TABLE fair_batch (id INTEGER PRIMARY KEY, v TEXT)")
+        conn.commit()
+        t0 = time.perf_counter()
+        for start in range(0, n, BATCH):
+            cnt = min(BATCH, n - start)
+            vals = multi_values_sql(start, cnt)
+            cur.execute(f"INSERT INTO fair_batch (id, v) VALUES {vals}")
+        conn.commit()
+        bw = time.perf_counter() - t0
+        out.append(
+            result(
+                "sql_insert_batch",
+                label,
+                "embedded",
+                n,
+                bw,
+                batch=BATCH,
+                durable=mode == "FULL",
+                note=f"multi-row INSERT batch={BATCH}, sync={mode}",
+            )
+        )
+        conn.close()
+
+    if os.path.exists(path):
+        os.unlink(path)
+    return out
+
+
+# ─── Tier 2: Client / server ────────────────────────────────────────
+
+
+def bara_http_query(sql: str, host: str = HTTP_HOST, port: int = HTTP_PORT) -> dict:
+    body = json.dumps({"query": sql}).encode()
+    req = urllib.request.Request(
+        f"http://{host}:{port}/query",
+        data=body,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=60) as resp:
+        return json.loads(resp.read().decode())
+
+
+def bara_http_available() -> bool:
+    try:
+        body = json.dumps({"query": "SELECT 1"}).encode()
+        # health endpoint preferred
+        req = urllib.request.Request(f"http://{HTTP_HOST}:{HTTP_PORT}/health", method="GET")
+        with urllib.request.urlopen(req, timeout=2) as resp:
+            return resp.status == 200
+    except Exception:
+        try:
+            bara_http_query("SELECT 1")
+            return True
+        except Exception:
+            return False
+
+
+def bench_baradb_http(n: int = N_SQL) -> list[dict]:
+    if os.environ.get("FAIR_SKIP_HTTP") == "1":
+        print("  [skip] FAIR_SKIP_HTTP=1")
+        return []
+    if not bara_http_available():
+        print(
+            f"  [skip] BaraDB HTTP not reachable at {HTTP_HOST}:{HTTP_PORT} "
+            f"(start: ./build/baradadb)"
+        )
+        return []
+
+    out = []
+    ep = f"http://{HTTP_HOST}:{HTTP_PORT}/query"
+    # BaraDB parser may not support IF EXISTS — ignore DROP failures
+    try:
+        bara_http_query("DROP TABLE fair_bench")
+    except Exception:
+        pass
+    try:
+        bara_http_query("CREATE TABLE fair_bench (id INT PRIMARY KEY, v TEXT)")
+    except Exception as e:
+        print(f"  [warn] setup query failed: {e}")
+
+    # Row-at-a-time INSERT
+    t0 = time.perf_counter()
+    errors = 0
+    for i in range(n):
+        try:
+            r = bara_http_query(
+                f"INSERT INTO fair_bench (id, v) VALUES ({i}, 'value_{i}')"
+            )
+            if isinstance(r, dict) and r.get("error"):
+                errors += 1
+        except Exception:
+            errors += 1
+    w = time.perf_counter() - t0
+    out.append(
+        result(
+            "sql_insert_row",
+            "baradb_http",
+            "client_server",
+            n,
+            w,
+            errors=errors,
+            endpoint=ep,
+        )
+    )
+
+    # Point SELECT
+    t0 = time.perf_counter()
+    found = 0
+    for i in range(n):
+        try:
+            r = bara_http_query(f"SELECT v FROM fair_bench WHERE id = {i}")
+            rows = r.get("rows") if isinstance(r, dict) else None
+            if rows:
+                found += 1
+        except Exception:
+            pass
+    rd = time.perf_counter() - t0
+    out.append(
+        result(
+            "sql_select_row",
+            "baradb_http",
+            "client_server",
+            n,
+            rd,
+            found=found,
+            endpoint=ep,
+        )
+    )
+
+    # Multi-row batch INSERT
+    try:
+        try:
+            bara_http_query("DROP TABLE fair_batch")
+        except Exception:
+            pass
+        bara_http_query("CREATE TABLE fair_batch (id INT PRIMARY KEY, v TEXT)")
+    except Exception as e:
+        print(f"  [warn] batch setup failed: {e}")
+        return out
+
+    t0 = time.perf_counter()
+    berr = 0
+    for start in range(0, n, BATCH):
+        cnt = min(BATCH, n - start)
+        sql = f"INSERT INTO fair_batch (id, v) VALUES {multi_values_sql(start, cnt)}"
+        try:
+            r = bara_http_query(sql)
+            if isinstance(r, dict) and r.get("error"):
+                berr += 1
+        except Exception:
+            berr += 1
+    bw = time.perf_counter() - t0
+    out.append(
+        result(
+            "sql_insert_batch",
+            "baradb_http",
+            "client_server",
+            n,
+            bw,
+            batch=BATCH,
+            errors=berr,
+            endpoint=ep,
+            note=f"multi-row INSERT batch={BATCH}",
+        )
+    )
+    return out
+
+
+def _import_baradb_client():
+    """Load clients/python baradb package without requiring install."""
+    p = str(CLIENTS_PY)
+    if p not in sys.path:
+        sys.path.insert(0, p)
+    from baradb import Client  # type: ignore
+
+    return Client
+
+
+def bara_wire_available() -> bool:
+    if os.environ.get("FAIR_SKIP_WIRE") == "1":
+        return False
+    try:
+        Client = _import_baradb_client()
+    except Exception as e:
+        print(f"  [skip] wire client import failed: {e}")
+        return False
+
+    async def _ping():
+        c = Client(WIRE_HOST, WIRE_PORT, timeout=2.0)
+        try:
+            await c.connect()
+            await c.ping()
+            await c.close()
+            return True
+        except Exception:
+            try:
+                await c.close()
+            except Exception:
+                pass
+            return False
+
+    try:
+        return asyncio.run(_ping())
+    except Exception:
+        return False
+
+
+def bench_baradb_wire(n: int = N_SQL) -> list[dict]:
+    """BaraDB binary wire protocol (TCP) — primary high-performance client path."""
+    if os.environ.get("FAIR_SKIP_WIRE") == "1":
+        print("  [skip] FAIR_SKIP_WIRE=1")
+        return []
+    try:
+        Client = _import_baradb_client()
+    except Exception as e:
+        print(f"  [skip] wire client not available: {e}")
+        return []
+
+    if not bara_wire_available():
+        print(
+            f"  [skip] BaraDB wire not reachable at {WIRE_HOST}:{WIRE_PORT} "
+            f"(start: ./build/baradadb)"
+        )
+        return []
+
+    async def _run() -> list[dict]:
+        out: list[dict] = []
+        client = Client(WIRE_HOST, WIRE_PORT, timeout=60.0)
+        await client.connect()
+        try:
+            try:
+                await client.query("DROP TABLE fair_wire")
+            except Exception:
+                pass
+            try:
+                await client.query(
+                    "CREATE TABLE fair_wire (id INT PRIMARY KEY, v TEXT)"
+                )
+            except Exception as e:
+                print(f"  [warn] wire setup: {e}")
+
+            ep = f"tcp://{WIRE_HOST}:{WIRE_PORT}"
+            # Row-at-a-time INSERT (may crash older servers under load — record partial)
+            t0 = time.perf_counter()
+            errors = 0
+            done = 0
+            crashed = False
+            for i in range(n):
+                try:
+                    await client.query(
+                        f"INSERT INTO fair_wire (id, v) VALUES ({i}, 'value_{i}')"
+                    )
+                    done += 1
+                except (ConnectionError, OSError, Exception) as e:
+                    errors += 1
+                    if "reset" in str(e).lower() or "closed" in str(e).lower():
+                        crashed = True
+                        print(f"  [warn] wire connection lost after {done} inserts: {e}")
+                        break
+            w = time.perf_counter() - t0
+            if done > 0:
+                out.append(
+                    result(
+                        "sql_insert_row",
+                        "baradb_wire",
+                        "client_server",
+                        done,
+                        w,
+                        errors=errors,
+                        requested=n,
+                        endpoint=ep,
+                        note="binary wire protocol"
+                        + (" (partial — server disconnect)" if crashed else ""),
+                    )
+                )
+
+            if crashed:
+                return out
+
+            # Point SELECT
+            t0 = time.perf_counter()
+            found = 0
+            for i in range(done):
+                try:
+                    r = await client.query(
+                        f"SELECT v FROM fair_wire WHERE id = {i}"
+                    )
+                    if r is not None and (
+                        getattr(r, "row_count", 0) > 0 or getattr(r, "rows", None)
+                    ):
+                        found += 1
+                except (ConnectionError, OSError, Exception) as e:
+                    if "reset" in str(e).lower() or "closed" in str(e).lower():
+                        crashed = True
+                        print(f"  [warn] wire lost during SELECT: {e}")
+                        break
+            rd = time.perf_counter() - t0
+            out.append(
+                result(
+                    "sql_select_row",
+                    "baradb_wire",
+                    "client_server",
+                    max(done, 1),
+                    rd,
+                    found=found,
+                    endpoint=ep,
+                )
+            )
+            if crashed:
+                return out
+
+            # Batch multi-row INSERT
+            try:
+                try:
+                    await client.query("DROP TABLE fair_wire_batch")
+                except Exception:
+                    pass
+                await client.query(
+                    "CREATE TABLE fair_wire_batch (id INT PRIMARY KEY, v TEXT)"
+                )
+            except Exception as e:
+                print(f"  [warn] wire batch setup: {e}")
+                return out
+
+            t0 = time.perf_counter()
+            berr = 0
+            bdone = 0
+            for start in range(0, n, BATCH):
+                cnt = min(BATCH, n - start)
+                sql = (
+                    "INSERT INTO fair_wire_batch (id, v) VALUES "
+                    + multi_values_sql(start, cnt)
+                )
+                try:
+                    await client.query(sql)
+                    bdone += cnt
+                except (ConnectionError, OSError, Exception) as e:
+                    berr += 1
+                    if "reset" in str(e).lower() or "closed" in str(e).lower():
+                        print(f"  [warn] wire lost during batch after {bdone} rows: {e}")
+                        break
+            bw = time.perf_counter() - t0
+            if bdone > 0:
+                out.append(
+                    result(
+                        "sql_insert_batch",
+                        "baradb_wire",
+                        "client_server",
+                        bdone,
+                        bw,
+                        batch=BATCH,
+                        errors=berr,
+                        requested=n,
+                        endpoint=ep,
+                        note=f"multi-row INSERT batch={BATCH}",
+                    )
+                )
+        finally:
+            try:
+                await client.close()
+            except Exception:
+                pass
+        return out
+
+    try:
+        return asyncio.run(_run())
+    except Exception as e:
+        print(f"  [skip] wire bench failed: {e}")
+        return []
+
+
+def bench_postgresql(n: int = N_SQL) -> list[dict]:
+    if os.environ.get("FAIR_SKIP_PG") == "1":
+        print("  [skip] FAIR_SKIP_PG=1")
+        return []
+    try:
+        import psycopg2
+    except ImportError:
+        print("  [skip] psycopg2 not installed")
+        return []
+
+    cfg = {
+        "host": os.environ.get("PGHOST", "localhost"),
+        "port": int(os.environ.get("PGPORT", "5432")),
+        "dbname": os.environ.get("PGDATABASE", "postgres"),
+        "user": os.environ.get("PGUSER", "postgres"),
+        "password": os.environ.get("PGPASSWORD", os.environ.get("PG_PASSWORD", "")),
+    }
+    if not cfg["password"] and os.environ.get("PGPASSWORD") is None:
+        cfg["password"] = os.environ.get("BARA_PG_PASSWORD", "pas+123")
+
+    out = []
+    try:
+        conn = psycopg2.connect(**cfg)
+    except Exception as e:
+        print(f"  [skip] PostgreSQL connect failed: {e}")
+        return []
+
+    cur = conn.cursor()
+    for sync, label in (("on", "postgresql_sync_on"), ("off", "postgresql_sync_off")):
+        cur.execute(f"SET synchronous_commit = {sync}")
+        cur.execute("DROP TABLE IF EXISTS fair_bench")
+        cur.execute("CREATE TABLE fair_bench (id INTEGER PRIMARY KEY, v TEXT)")
+        conn.commit()
+
+        t0 = time.perf_counter()
+        for i in range(n):
+            cur.execute(
+                "INSERT INTO fair_bench (id, v) VALUES (%s, %s)",
+                (i, f"value_{i}"),
+            )
+        conn.commit()
+        w = time.perf_counter() - t0
+        out.append(
+            result(
+                "sql_insert_row",
+                label,
+                "client_server",
+                n,
+                w,
+                durable=sync == "on",
+                note=f"synchronous_commit={sync}",
+            )
+        )
+
+        t0 = time.perf_counter()
+        found = 0
+        for i in range(n):
+            cur.execute("SELECT v FROM fair_bench WHERE id = %s", (i,))
+            if cur.fetchone():
+                found += 1
+        rd = time.perf_counter() - t0
+        out.append(
+            result(
+                "sql_select_row",
+                label,
+                "client_server",
+                n,
+                rd,
+                found=found,
+                durable=sync == "on",
+            )
+        )
+
+        # Batch multi-row INSERT (same durability setting)
+        cur.execute("DROP TABLE IF EXISTS fair_batch")
+        cur.execute("CREATE TABLE fair_batch (id INTEGER PRIMARY KEY, v TEXT)")
+        conn.commit()
+        t0 = time.perf_counter()
+        for start in range(0, n, BATCH):
+            cnt = min(BATCH, n - start)
+            cur.execute(
+                f"INSERT INTO fair_batch (id, v) VALUES {multi_values_sql(start, cnt)}"
+            )
+        conn.commit()
+        bw = time.perf_counter() - t0
+        out.append(
+            result(
+                "sql_insert_batch",
+                label,
+                "client_server",
+                n,
+                bw,
+                batch=BATCH,
+                durable=sync == "on",
+                note=f"multi-row INSERT batch={BATCH}, sync={sync}",
+            )
+        )
+
+    cur.close()
+    conn.close()
+    return out
+
+
+# ─── Report ──────────────────────────────────────────────────────────
+
+
+def print_tier(name: str, rows: list[dict]):
+    print(f"\n=== Tier: {name} ===")
+    if not rows:
+        print("  (no results)")
+        return
+    # group by bench name
+    names = []
+    for r in rows:
+        if r["name"] not in names:
+            names.append(r["name"])
+    for nm in names:
+        print(f"  [{nm}]")
+        for r in rows:
+            if r["name"] != nm:
+                continue
+            print(
+                f"    {r['system']:28s}  {fmt_ops(r['opsPerSec']):>10s}/s  "
+                f"({r['seconds']:.3f}s, n={r['ops']})"
+            )
+
+
+def write_markdown(payload: dict, path: Path):
+    lines = []
+    lines.append("# Fair Benchmark Results")
+    lines.append("")
+    lines.append(f"Generated: **{payload.get('generated', '')}**")
+    lines.append("")
+    lines.append("## Methodology")
+    lines.append("")
+    for line in payload.get("methodology", []):
+        lines.append(f"- {line}")
+    lines.append("")
+    lines.append("**Do not compare numbers across tiers.** Embedded storage is not the same")
+    lines.append("workload as client-server SQL over the network.")
+    lines.append("")
+
+    for tier in ("embedded", "client_server"):
+        rows = [r for r in payload.get("results", []) if r.get("tier") == tier]
+        lines.append(f"## Tier: `{tier}`")
+        lines.append("")
+        if not rows:
+            lines.append("_No results for this tier._")
+            lines.append("")
+            continue
+        lines.append("| Bench | System | ops/s | seconds | n | notes |")
+        lines.append("|-------|--------|------:|--------:|--:|-------|")
+        for r in rows:
+            note = r.get("note") or r.get("source") or ""
+            lines.append(
+                f"| {r['name']} | `{r['system']}` | {fmt_ops(r['opsPerSec'])} | "
+                f"{r['seconds']:.3f} | {r['ops']} | {note} |"
+            )
+        lines.append("")
+
+        # same-bench comparison within tier
+        names = sorted({r["name"] for r in rows})
+        lines.append(f"### Same-bench ratios (`{tier}`)")
+        lines.append("")
+        for nm in names:
+            group = [r for r in rows if r["name"] == nm]
+            if len(group) < 2:
+                continue
+            best = max(group, key=lambda x: x["opsPerSec"])
+            lines.append(f"**{nm}** (fastest: `{best['system']}` @ {fmt_ops(best['opsPerSec'])}/s)")
+            lines.append("")
+            lines.append("| System | Relative to fastest |")
+            lines.append("|--------|--------------------:|")
+            for r in sorted(group, key=lambda x: -x["opsPerSec"]):
+                rel = r["opsPerSec"] / best["opsPerSec"] if best["opsPerSec"] else 0
+                lines.append(f"| `{r['system']}` | {rel:.2f}x |")
+            lines.append("")
+
+    path.write_text("\n".join(lines) + "\n")
+    print(f"\nMarkdown written to {path}")
+
+
+def main():
+    print("BaraDB Fair Benchmark Suite")
+    print(f"  N_KV={N_KV}  N_SQL={N_SQL}  BATCH={BATCH}")
+    print(f"  HTTP={HTTP_HOST}:{HTTP_PORT}  WIRE={WIRE_HOST}:{WIRE_PORT}")
+
+    methodology = [
+        "Tier `embedded`: in-process only (BaraDB LSM from nimble bench JSON; SQLite via Python sqlite3).",
+        "Tier `client_server`: network SQL (BaraDB HTTP /query; BaraDB binary wire TCP; PostgreSQL via psycopg2).",
+        "`sql_insert_row`: one INSERT statement per row (chatty).",
+        f"`sql_insert_batch`: multi-row INSERT with batch size {BATCH} (same SQL style across systems).",
+        "PostgreSQL: synchronous_commit=on|off; SQLite: PRAGMA synchronous FULL|OFF.",
+        "BaraDB WAL modes appear only if you ran benchmarks/bench_all.nim (WAL-* rows).",
+        "Never claim 'Nx faster than Postgres' using embedded BaraDB numbers.",
+    ]
+
+    results: list[dict] = []
+
+    print("\n--- Embedded tier ---")
+    results.extend(load_baradb_embedded())
+    print("  SQLite embedded (+ batch)…")
+    results.extend(bench_sqlite_embedded())
+
+    print("\n--- Client/server tier ---")
+    print("  BaraDB HTTP…")
+    results.extend(bench_baradb_http())
+    print("  BaraDB wire (TCP)…")
+    results.extend(bench_baradb_wire())
+    print("  PostgreSQL…")
+    results.extend(bench_postgresql())
+
+    payload = {
+        "generated": now_iso(),
+        "methodology": methodology,
+        "config": {
+            "N_KV": N_KV,
+            "N_SQL": N_SQL,
+            "HTTP": f"{HTTP_HOST}:{HTTP_PORT}",
+        },
+        "results": results,
+    }
+    OUT_JSON.write_text(json.dumps(payload, indent=2))
+    print(f"\nJSON written to {OUT_JSON}")
+
+    print_tier("embedded", [r for r in results if r["tier"] == "embedded"])
+    print_tier("client_server", [r for r in results if r["tier"] == "client_server"])
+
+    write_markdown(payload, ROOT / "benchmarks" / "FAIR_COMPARISON.md")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

benchmarks/generate_report.py

@@ -1,110 +1,143 @@
 #!/usr/bin/env python3
-"""Generate a real comparison report from BaraDB and PostgreSQL benchmark results."""
+"""Generate benchmark reports.
+
+Modes:
+  python3 benchmarks/generate_report.py           # legacy PG vs embedded (with warning)
+  python3 benchmarks/generate_report.py --fair    # multi-tier fair report from fair_bench.py
+"""
+from __future__ import annotations
+
+import argparse
 import json
+import sys
 from pathlib import Path
 
+ROOT = Path(__file__).resolve().parent.parent
 
-def format_ops(ops_per_sec):
+
+def format_ops(ops_per_sec: float) -> str:
     if ops_per_sec >= 1_000_000:
         return f"{ops_per_sec/1_000_000:.2f}M"
-    elif ops_per_sec >= 1_000:
+    if ops_per_sec >= 1_000:
         return f"{ops_per_sec/1_000:.2f}K"
-    else:
-        return f"{ops_per_sec:.2f}"
+    return f"{ops_per_sec:.2f}"
 
 
-def format_time(seconds):
+def format_time(seconds: float) -> str:
     if seconds < 0.001:
         return f"{seconds*1000:.3f}ms"
-    elif seconds < 1:
+    if seconds < 1:
         return f"{seconds*1000:.1f}ms"
-    else:
-        return f"{seconds:.3f}s"
+    return f"{seconds:.3f}s"
 
 
-def main():
-    root = Path(__file__).parent
+def gen_fair(out: Path) -> int:
+    fair_path = ROOT / "fair_benchmark_results.json"
+    if not fair_path.exists():
+        print("Missing fair_benchmark_results.json — run: python3 benchmarks/fair_bench.py")
+        return 1
+    payload = json.loads(fair_path.read_text())
+    # fair_bench already writes FAIR_COMPARISON.md; re-emit for consistency
+    sys.path.insert(0, str(ROOT / "benchmarks"))
+    from fair_bench import write_markdown  # type: ignore
 
-    with open(root.parent / "benchmark_results.json") as f:
+    write_markdown(payload, out)
+    return 0
+
+
+def gen_legacy() -> int:
+    """Legacy report: PG client-server vs BaraDB *embedded* — always labeled unfair."""
+    bara_path = ROOT / "benchmark_results.json"
+    pg_path = ROOT / "pg_benchmark_results.json"
+    if not bara_path.exists() or not pg_path.exists():
+        print("Need benchmark_results.json and pg_benchmark_results.json")
+        print("  nimble bench && python3 benchmarks/pg_bench.py")
+        return 1
+
+    with open(bara_path) as f:
         bara = json.load(f)
-    with open(root.parent / "pg_benchmark_results.json") as f:
+    with open(pg_path) as f:
         pg = json.load(f)
 
     bara_map = {r["name"]: r for r in bara["results"]}
-    pg_map = {k: v for k, v in pg.items()}
+    # pg_bench may write list or dict
+    if isinstance(pg, dict) and "results" in pg:
+        pg_map = {r["name"]: r for r in pg["results"]}
+    elif isinstance(pg, list):
+        pg_map = {r["name"]: r for r in pg}
+    else:
+        pg_map = pg  # old flat dict by name
 
     report = []
-    report.append("# BaraDB vs PostgreSQL — Real Benchmark Results")
+    report.append("# BaraDB vs PostgreSQL — LEGACY (mixed tiers)")
     report.append("")
-    report.append("Generated from actual execution on:")
-    report.append(f"- **CPU:** AMD Ryzen 9 5900X")
-    report.append(f"- **PostgreSQL:** 15.18 (local)")
-    report.append(f"- **BaraDB:** git `{bara['gitSha']}`")
+    report.append("> ⚠️ **Unfair comparison warning**")
+    report.append(">")
+    report.append("> PostgreSQL numbers include **client-server** round-trips.")
+    report.append("> BaraDB numbers are **in-process embedded** LSM (no network, no SQL).")
+    report.append("> Use `python3 benchmarks/fair_bench.py` + `--fair` for honest tiers.")
     report.append("")
-    report.append("## Methodology")
+    report.append(f"- **BaraDB git:** `{bara.get('gitSha', 'unknown')}`")
     report.append("")
-    report.append("- PostgreSQL: single-row INSERT/SELECT via psycopg2 (client-server overhead included)")
-    report.append("- BaraDB: in-process Nim code (no network overhead)")
-    report.append("- Same dataset sizes for both systems")
-    report.append("")
-    report.append("## Results")
-    report.append("")
-    report.append("| Test | PostgreSQL | BaraDB | Speedup |")
-    report.append("|------|-----------|--------|---------|")
+    report.append("| Test | PostgreSQL (C/S) | BaraDB (embedded) | Ratio (not a fair speedup) |")
+    report.append("|------|------------------|-------------------|----------------------------|")
 
     rows = [
-        ("KV Write (100K)", pg_map.get("KV Write"), bara_map.get("LSM-Write")),
-        ("KV Read (100K)", pg_map.get("KV Read"), bara_map.get("LSM-Read")),
-        ("BTree Insert (100K)", pg_map.get("BTree Insert"), bara_map.get("BTree-Insert")),
-        ("BTree Get (100K)", pg_map.get("BTree Get"), bara_map.get("BTree-Get")),
-        ("BTree Scan (1K ranges)", pg_map.get("BTree Scan"), bara_map.get("BTree-Scan")),
-        ("FTS Index (10K docs)", pg_map.get("FTS Index"), bara_map.get("FTS-Index")),
-        ("FTS Search (1K queries)", pg_map.get("FTS Search"), bara_map.get("FTS-Search")),
+        ("KV Write", pg_map.get("KV Write"), bara_map.get("LSM-Write")),
+        ("KV Read", pg_map.get("KV Read"), bara_map.get("LSM-Read")),
+        ("BTree Insert", pg_map.get("BTree Insert"), bara_map.get("BTree-Insert")),
+        ("BTree Get", pg_map.get("BTree Get"), bara_map.get("BTree-Get")),
+        ("BTree Scan", pg_map.get("BTree Scan"), bara_map.get("BTree-Scan")),
+        ("FTS Index", pg_map.get("FTS Index"), bara_map.get("FTS-Index")),
+        ("FTS Search", pg_map.get("FTS Search"), bara_map.get("FTS-Search")),
     ]
 
-    total_pg_time = 0
-    total_bara_time = 0
-
     for name, p, b in rows:
         if p is None or b is None:
             continue
         pg_ops = p["opsPerSec"]
         ba_ops = b["opsPerSec"]
-        ratio = ba_ops / pg_ops
-        winner = "BaraDB" if ratio > 1 else "PostgreSQL"
-        total_pg_time += p["seconds"]
-        total_bara_time += b["seconds"]
-
+        ratio = ba_ops / pg_ops if pg_ops else 0
         report.append(
             f"| {name} | {format_ops(pg_ops)}/s ({format_time(p['seconds'])}) | "
             f"{format_ops(ba_ops)}/s ({format_time(b['seconds'])}) | "
-            f"{ratio:.1f}x ({winner}) |"
+            f"{ratio:.1f}x (mixed tiers) |"
         )
 
     report.append("")
-    report.append("## Summary")
+    report.append("## Prefer fair suite")
     report.append("")
-    report.append(f"- **Total PostgreSQL time:** {total_pg_time:.3f}s")
-    report.append(f"- **Total BaraDB time:** {total_bara_time:.3f}s")
-    overall = total_pg_time / total_bara_time
-    report.append(f"- **Overall speedup:** BaraDB is **{overall:.1f}x faster**")
-    report.append("")
-    report.append("## Notes")
-    report.append("")
-    report.append("- PostgreSQL includes network round-trip and SQL parsing overhead per operation.")
-    report.append("- BaraDB runs in-process with zero serialization/network cost.")
-    report.append("- For embedded/single-node use cases, BaraDB shows significant advantage.")
-    report.append("- PostgreSQL FTS Search with GIN index outperforms BaraDB on query throughput.")
-    report.append("- PostgreSQL excels at durability, replication, and complex ACID transactions.")
+    report.append("```bash")
+    report.append("nim c -d:release -r benchmarks/bench_all.nim")
+    report.append("python3 benchmarks/fair_bench.py")
+    report.append("python3 benchmarks/generate_report.py --fair")
+    report.append("```")
     report.append("")
 
-    output = "\n".join(report)
-    print(output)
+    out = ROOT / "benchmarks" / "REAL_COMPARISON.md"
+    out.write_text("\n".join(report) + "\n")
+    print(f"Wrote {out} (legacy mixed-tier; see warning banner)")
+    return 0
 
-    with open(root / "REAL_COMPARISON.md", "w") as f:
-        f.write(output)
-    print(f"\nReport saved to {root / 'REAL_COMPARISON.md'}")
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument(
+        "--fair",
+        action="store_true",
+        help="Emit multi-tier fair report from fair_benchmark_results.json",
+    )
+    ap.add_argument(
+        "-o",
+        "--output",
+        default=str(ROOT / "benchmarks" / "FAIR_COMPARISON.md"),
+        help="Output path for --fair mode",
+    )
+    args = ap.parse_args()
+    if args.fair:
+        return gen_fair(Path(args.output))
+    return gen_legacy()
 
 
 if __name__ == "__main__":
-    main()
+    sys.exit(main())

benchmarks/pg_bench.py

@@ -174,8 +174,10 @@ def format_ops(ops_per_sec):
 def print_comparison(pg_results, bara_data):
     bara = {r["name"]: r for r in bara_data["results"]}
     print("\n╔══════════════════════════════════════════════════════════════════════╗")
-    print("║           BaraDB vs PostgreSQL — Real Benchmark Results              ║")
+    print("║     PostgreSQL (client-server) vs BaraDB (EMBEDDED) — MIXED TIERS    ║")
     print("╚══════════════════════════════════════════════════════════════════════╝\n")
+    print("WARNING: This mixes client-server PG with in-process BaraDB LSM.")
+    print("         Prefer: python3 benchmarks/fair_bench.py\n")
 
     rows = [
         ("KV Write (100K)", pg_results.get("KV Write"), bara.get("LSM-Write")),
@@ -187,7 +189,7 @@ def print_comparison(pg_results, bara_data):
         ("FTS Search (1K queries)", pg_results.get("FTS Search"), bara.get("FTS-Search")),
     ]
 
-    print(f"{'Test':<26} {'PostgreSQL':>18} {'BaraDB':>18} {'Winner':>10}")
+    print(f"{'Test':<26} {'PostgreSQL C/S':>18} {'BaraDB embed':>18} {'Note':>14}")
     print("─" * 76)
 
     for name, pg, ba in rows:
@@ -195,22 +197,14 @@ def print_comparison(pg_results, bara_data):
             continue
         pg_ops = pg["opsPerSec"]
         ba_ops = ba["opsPerSec"]
-        winner = "BaraDB" if ba_ops > pg_ops else "PostgreSQL"
-        ratio = max(ba_ops, pg_ops) / min(ba_ops, pg_ops)
+        ratio = ba_ops / pg_ops if pg_ops else 0
         print(
-            f"{name:<26} {format_ops(pg_ops)+'/s':>18} {format_ops(ba_ops)+'/s':>18} {winner+' ('+f'{ratio:.1f}x'+')':>10}"
+            f"{name:<26} {format_ops(pg_ops)+'/s':>18} {format_ops(ba_ops)+'/s':>18} "
+            f"{'mixed '+f'{ratio:.1f}x':>14}"
         )
 
     print("\n" + "─" * 76)
-    # Summary
-    pg_total = sum(r["seconds"] for _, r, _ in rows if r is not None)
-    ba_total = sum(b["seconds"] for _, _, b in rows if b is not None)
-    print(f"\nTotal time PostgreSQL: {pg_total:.3f}s")
-    print(f"Total time BaraDB:     {ba_total:.3f}s")
-    if ba_total < pg_total:
-        print(f"BaraDB is {pg_total/ba_total:.1f}x faster overall")
-    else:
-        print(f"PostgreSQL is {ba_total/pg_total:.1f}x faster overall")
+    print("For fair tiers (SQLite↔LSM, HTTP↔PG): python3 benchmarks/fair_bench.py")
 
 
 def main():
@@ -247,14 +241,23 @@ def main():
     pg_results["FTS Search"] = bench_fts_search()
     print(f"      -> {format_ops(pg_results['FTS Search']['opsPerSec'])}/s ({pg_results['FTS Search']['seconds']:.3f}s)")
 
-    bara_data = load_baradb_results()
-    print_comparison(pg_results, bara_data)
+    # Annotate tier for fair tooling
+    for name, r in pg_results.items():
+        r["tier"] = "client_server"
+        r["system"] = "postgresql"
 
-    # Save raw results
     with open("pg_benchmark_results.json", "w") as f:
         json.dump(pg_results, f, indent=2)
     print("\nPostgreSQL results saved to pg_benchmark_results.json")
 
+    if os.path.exists("benchmark_results.json"):
+        bara_data = load_baradb_results()
+        print_comparison(pg_results, bara_data)
+    else:
+        print("\n(No benchmark_results.json — skip mixed-tier table; run nimble bench first)")
+
+    print("\nFair multi-tier suite: python3 benchmarks/fair_bench.py")
+
 
 if __name__ == "__main__":
     main()