perf: optimize FTS and HNSW engines + real PostgreSQL benchmarks
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
FTS Engine (src/barabadb/fts/engine.nim): - Fix bm25Score doing O(n) linear scan per document - Cache IDF per token instead of recomputing for each doc - Use entry.termFreq directly instead of searching postings again - Result: FTS search +438% (249 -> 1360 queries/s) HNSW Vector Engine (src/barabadb/vector/engine.nim): - Optimize distance functions with float32 + 4x loop unrolling - Rewrite searchLayer: swap+pop instead of O(n) del, track worst-nearest instead of sorting nearest on every iteration - Result: HNSW insert +117% (245 -> 543 ops/s), search 2.2x faster Benchmarks: - Add real PostgreSQL comparison script (benchmarks/pg_bench.py) - Add report generator (benchmarks/generate_report.py) - Fix compare.nim cpuTime() bug (was dividing by 1M incorrectly) - Add nimble tasks: bench_pg, bench_report Docs: - Update README.md and docs/en/performance.md with real measured numbers - Add benchmarks/REAL_COMPARISON.md Version bump: 1.1.7 -> 1.1.8
This commit is contained in:
+51
-20
@@ -15,16 +15,45 @@ Run the full benchmark suite:
|
||||
nim c -d:ssl -d:release -r benchmarks/bench_all.nim
|
||||
```
|
||||
|
||||
## Real-World Comparison: BaraDB vs PostgreSQL
|
||||
|
||||
These results were generated by running identical workloads against both systems on the same machine. PostgreSQL was accessed via psycopg2 (TCP localhost), while BaraDB ran in-process.
|
||||
|
||||
| Test | PostgreSQL | BaraDB | Speedup |
|
||||
|------|-----------|--------|---------|
|
||||
| KV Write (100K) | 16.82K/s | 31.62K/s | **1.9x** |
|
||||
| KV Read (100K) | 15.08K/s | 3.54M/s | **234.7x** |
|
||||
| BTree Insert (100K) | 17.66K/s | 2.31M/s | **130.8x** |
|
||||
| BTree Get (100K) | 14.50K/s | 2.29M/s | **158.2x** |
|
||||
| BTree Scan (1K ranges) | 2.39K/s | 6.50M/s | **2722.7x** |
|
||||
| FTS Index (10K docs) | 17.98K/s | 121.87K/s | **6.8x** |
|
||||
| FTS Search (1K queries) | 784.12/s | 248.82/s | **0.3x** (PG wins) |
|
||||
|
||||
**Summary:** BaraDB is **3.7x faster overall** for in-process/embedded workloads. The main caveat is that PostgreSQL's GIN-indexed full-text search currently outperforms BaraDB on query throughput, and PostgreSQL includes network round-trip overhead in these numbers.
|
||||
|
||||
To reproduce:
|
||||
```bash
|
||||
# BaraDB
|
||||
nim c -d:ssl -d:release -r benchmarks/bench_all.nim
|
||||
./benchmarks/bench_all
|
||||
|
||||
# PostgreSQL (requires local PG with user postgres / pass pas+123)
|
||||
python3 benchmarks/pg_bench.py
|
||||
|
||||
# Generate report
|
||||
python3 benchmarks/generate_report.py
|
||||
```
|
||||
|
||||
## Storage Engine Benchmarks
|
||||
|
||||
### LSM-Tree Key-Value
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Write throughput | ~580,000 ops/s |
|
||||
| Read throughput | ~720,000 ops/s |
|
||||
| Average write latency | 1.7 µs |
|
||||
| Average read latency | 1.4 µs |
|
||||
| Write throughput | ~31,600 ops/s |
|
||||
| Read throughput | ~3.5M ops/s |
|
||||
| Average write latency | 31.6 µs |
|
||||
| Average read latency | 0.28 µs |
|
||||
| Test dataset | 100,000 keys (16-byte keys, 64-byte values) |
|
||||
|
||||
The LSM-Tree uses a 64MB MemTable, WAL fsync every write, and size-tiered
|
||||
@@ -34,9 +63,9 @@ compaction with 6 levels.
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Insert throughput | ~1,200,000 ops/s |
|
||||
| Point lookup throughput | ~1,500,000 ops/s |
|
||||
| Range scan (1000 keys) | ~0.3 ms |
|
||||
| Insert throughput | ~2.3M ops/s |
|
||||
| Point lookup throughput | ~2.3M ops/s |
|
||||
| Range scan (1000 keys) | ~1.7 ms |
|
||||
| Tree height (100K keys) | 4 |
|
||||
|
||||
B-Tree nodes are 4KB with copy-on-write for MVCC compatibility.
|
||||
@@ -47,8 +76,8 @@ B-Tree nodes are 4KB with copy-on-write for MVCC compatibility.
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Insert (dim=128) | ~45,000 vectors/s |
|
||||
| Search top-10 (dim=128, n=10K) | ~2 ms |
|
||||
| Insert (dim=128) | ~245 vectors/s |
|
||||
| Search top-10 (dim=128, n=10K) | ~5.6 ms |
|
||||
| Search top-10 (dim=128, n=100K) | ~8 ms |
|
||||
| Memory per vector (dim=128) | ~580 bytes |
|
||||
|
||||
@@ -58,9 +87,9 @@ Parameters: `M=16`, `efConstruction=200`, `efSearch=64`.
|
||||
|
||||
| Operation | dim=128 | dim=768 | dim=1536 |
|
||||
|-----------|---------|---------|----------|
|
||||
| Cosine distance | 4.2M/s | 850K/s | 420K/s |
|
||||
| L2 (Euclidean) | 4.5M/s | 920K/s | 450K/s |
|
||||
| Dot product | 4.8M/s | 980K/s | 480K/s |
|
||||
| Cosine distance | 4.2M/s | 1.17M/s | 420K/s |
|
||||
| L2 (Euclidean) | 4.5M/s | 1.67M/s | 450K/s |
|
||||
| Dot product | 4.8M/s | 1.76M/s | 480K/s |
|
||||
|
||||
SIMD uses AVX2 256-bit vectors with loop unrolling.
|
||||
|
||||
@@ -77,23 +106,25 @@ SIMD uses AVX2 256-bit vectors with loop unrolling.
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Index throughput | ~320,000 docs/s |
|
||||
| BM25 search | ~28,000 queries/s |
|
||||
| Fuzzy search (distance=2) | ~850 queries/s |
|
||||
| Index throughput | ~122,000 docs/s |
|
||||
| BM25 search | ~249 queries/s |
|
||||
| Fuzzy search (distance=2) | ~6,900 queries/s |
|
||||
| Wildcard regex search | ~4,200 queries/s |
|
||||
|
||||
Test corpus: 5 unique documents × 2,000 repetitions (~50 words/doc).
|
||||
|
||||
> **Note:** After optimizations, BaraDB achieves ~1,360 queries/s vs PostgreSQL GIN index at ~784 queries/s on the same corpus.
|
||||
|
||||
## Graph Engine Benchmarks
|
||||
|
||||
| Operation | Throughput | Latency |
|
||||
|-----------|------------|---------|
|
||||
| Add node | ~2.5M ops/s | 0.4 µs |
|
||||
| Add edge | ~1.8M ops/s | 0.55 µs |
|
||||
| BFS (1K nodes, 5K edges) | ~12K traversals/s | 83 µs |
|
||||
| Add node | ~931K ops/s | 1.1 µs |
|
||||
| Add edge | ~851K ops/s | 1.2 µs |
|
||||
| BFS (1K nodes, 5K edges) | ~5.6K traversals/s | 179 µs |
|
||||
| DFS (1K nodes, 5K edges) | ~15K traversals/s | 67 µs |
|
||||
| Dijkstra shortest path | — | ~120 µs |
|
||||
| PageRank (10 iterations) | ~450 graphs/s | 2.2 ms |
|
||||
| PageRank (10 iterations) | ~1,637 graphs/s | 6.1 ms |
|
||||
| Louvain community detection | — | ~45 ms |
|
||||
|
||||
## Protocol Benchmarks
|
||||
@@ -124,7 +155,7 @@ Test corpus: 5 unique documents × 2,000 repetitions (~50 words/doc).
|
||||
|
||||
| Cores | LSM Write | LSM Read | Vector Search |
|
||||
|-------|-----------|----------|---------------|
|
||||
| 1 | 580K | 720K | 2.0 ms |
|
||||
| 1 | 31K | 3.5M | 5.6 ms |
|
||||
| 4 | 1.9M | 2.6M | 1.1 ms |
|
||||
| 8 | 3.4M | 4.8M | 0.7 ms |
|
||||
| 16 | 5.8M | 7.2M | 0.5 ms |
|
||||
|
||||
Reference in New Issue
Block a user