docs(en): Update English docs for Vector SQL Integration
- docs/en/vector.md — add SQL usage section (CREATE TABLE VECTOR, distance functions, <-> operator, CREATE INDEX USING hnsw) - docs/en/baraql.md — update vector search section with real SQL syntax, add VECTOR(n) to data types, update keyword table - docs/en/changelog.md — add Vector SQL Integration and bugfixes to [Unreleased] - docs/ARCHITECTURE.md — add SQL Integration bullet to Vector Engine - README.md — update vector engine section with SQL examples, add Vector SQL to roadmap, bump test count to 340+
This commit is contained in:
@@ -285,8 +285,23 @@ let range = btree.scan("key_a", "key_z")
|
|||||||
|
|
||||||
### Vector Engine
|
### Vector Engine
|
||||||
|
|
||||||
Native HNSW and IVF-PQ indexes for similarity search.
|
Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- SQL vector search
|
||||||
|
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
|
||||||
|
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
|
||||||
|
|
||||||
|
-- Nearest neighbor search
|
||||||
|
SELECT id FROM items
|
||||||
|
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, ...]') ASC
|
||||||
|
LIMIT 10;
|
||||||
|
|
||||||
|
-- With HNSW index
|
||||||
|
CREATE INDEX idx_vec ON items(embedding) USING hnsw;
|
||||||
|
```
|
||||||
|
|
||||||
|
Native Nim API:
|
||||||
```nim
|
```nim
|
||||||
import barabadb/vector/engine
|
import barabadb/vector/engine
|
||||||
|
|
||||||
@@ -301,7 +316,10 @@ let filtered = idx.searchWithFilter(queryVector, k = 10,
|
|||||||
```
|
```
|
||||||
|
|
||||||
Features:
|
Features:
|
||||||
- **HNSW** — hierarchical navigable small world graph
|
- **SQL vector types** — `VECTOR(n)` with dimension validation
|
||||||
|
- **SQL distance functions** — `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
|
||||||
|
- **`<->` operator** — Euclidean distance nearest-neighbor shorthand
|
||||||
|
- **HNSW index** — `CREATE INDEX ... USING hnsw` with automatic maintenance
|
||||||
- **IVF-PQ** — inverted file index with product quantization
|
- **IVF-PQ** — inverted file index with product quantization
|
||||||
- **Distance metrics** — cosine, euclidean, dot product, Manhattan
|
- **Distance metrics** — cosine, euclidean, dot product, Manhattan
|
||||||
- **Quantization** — scalar 8-bit/4-bit, product, binary
|
- **Quantization** — scalar 8-bit/4-bit, product, binary
|
||||||
@@ -1231,7 +1249,7 @@ src/barabadb/
|
|||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Run all tests (262 tests, 56 suites)
|
# Run all tests (340+ tests, 60+ suites)
|
||||||
nim c --path:src -r tests/test_all.nim
|
nim c --path:src -r tests/test_all.nim
|
||||||
|
|
||||||
# Run benchmarks
|
# Run benchmarks
|
||||||
@@ -1249,6 +1267,7 @@ nim c -d:release -r benchmarks/bench_all.nim
|
|||||||
| Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 |
|
| Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 |
|
||||||
| Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 |
|
| Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 |
|
||||||
| Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 |
|
| Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 |
|
||||||
|
| Vector SQL Integration (VECTOR type, distance functions, <->, HNSW indexes) | ✅ | 100% | v1.1.0 |
|
||||||
| Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 |
|
| Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 |
|
||||||
| FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 |
|
| FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 |
|
||||||
| CLI shell | ✅ | 100% | v1.0.0 |
|
| CLI shell | ✅ | 100% | v1.0.0 |
|
||||||
|
|||||||
@@ -90,6 +90,12 @@ The query layer processes BaraQL — a SQL-compatible query language with extens
|
|||||||
- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression.
|
- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression.
|
||||||
- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan).
|
- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan).
|
||||||
- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput.
|
- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput.
|
||||||
|
- **SQL Integration** (`query/executor.nim`):
|
||||||
|
- `VECTOR(n)` column type with dimension validation
|
||||||
|
- `CREATE INDEX ... USING hnsw` / `USING ivfpq`
|
||||||
|
- Distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
|
||||||
|
- `<->` nearest-neighbor operator
|
||||||
|
- Automatic index maintenance on INSERT/UPDATE
|
||||||
|
|
||||||
### Graph Engine (`graph/`)
|
### Graph Engine (`graph/`)
|
||||||
- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency.
|
- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency.
|
||||||
|
|||||||
+61
-20
@@ -18,6 +18,7 @@ BaraQL is a SQL-compatible query language with extensions for graph, vector, and
|
|||||||
| `bytes` | Raw bytes | `0xDEADBEEF` |
|
| `bytes` | Raw bytes | `0xDEADBEEF` |
|
||||||
| `array<T>` | Homogeneous array | `[1, 2, 3]` |
|
| `array<T>` | Homogeneous array | `[1, 2, 3]` |
|
||||||
| `vector` | Float32 vector | `[0.1, 0.2, 0.3]` |
|
| `vector` | Float32 vector | `[0.1, 0.2, 0.3]` |
|
||||||
|
| `vector(n)` | Fixed-dimension float32 vector (SQL) | `VECTOR(768)` |
|
||||||
| `object` | Key-value object | `{"a": 1}` |
|
| `object` | Key-value object | `{"a": 1}` |
|
||||||
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
|
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
|
||||||
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
|
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
|
||||||
@@ -352,6 +353,7 @@ CREATE TYPE Cat EXTENDING Animal {
|
|||||||
CREATE INDEX idx_users_name ON users(name);
|
CREATE INDEX idx_users_name ON users(name);
|
||||||
CREATE UNIQUE INDEX idx_users_email ON users(email);
|
CREATE UNIQUE INDEX idx_users_email ON users(email);
|
||||||
CREATE INDEX idx_users_age ON users(age) USING btree;
|
CREATE INDEX idx_users_age ON users(age) USING btree;
|
||||||
|
CREATE INDEX idx_vectors ON items(embedding) USING hnsw;
|
||||||
```
|
```
|
||||||
|
|
||||||
### DROP
|
### DROP
|
||||||
@@ -387,37 +389,76 @@ SELECT * FROM articles WHERE body @@ 'machine learning';
|
|||||||
RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
|
RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
|
||||||
```
|
```
|
||||||
|
|
||||||
## Vector Search
|
## Vector Search (SQL)
|
||||||
|
|
||||||
|
### Creating Vector Columns
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
-- Insert with vector
|
CREATE TABLE items (
|
||||||
INSERT articles {
|
id INT PRIMARY KEY,
|
||||||
title := 'Nim Programming',
|
embedding VECTOR(768)
|
||||||
embedding := [0.1, 0.2, 0.3, 0.4]
|
);
|
||||||
};
|
```
|
||||||
|
|
||||||
-- Similarity search (cosine distance)
|
### Inserting Vectors
|
||||||
SELECT title FROM articles
|
|
||||||
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
|
|
||||||
LIMIT 5;
|
|
||||||
|
|
||||||
-- Euclidean distance
|
```sql
|
||||||
SELECT title FROM articles
|
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, 0.4]');
|
||||||
ORDER BY l2_distance(embedding, [0.1, 0.2, 0.3, 0.4])
|
```
|
||||||
LIMIT 5;
|
|
||||||
|
|
||||||
-- Dot product
|
### Distance Functions
|
||||||
SELECT title FROM articles
|
|
||||||
ORDER BY dot_product(embedding, [0.1, 0.2, 0.3, 0.4]) DESC
|
```sql
|
||||||
|
-- Cosine distance (0 = identical, 2 = opposite)
|
||||||
|
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- Euclidean / L2 distance
|
||||||
|
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- L2 distance with <-> operator
|
||||||
|
SELECT id, embedding <-> '[0.1, 0.2, 0.3, 0.4]' AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- Inner product (negative dot product)
|
||||||
|
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- Manhattan / L1 distance
|
||||||
|
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
|
||||||
|
FROM items;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nearest Neighbor Search
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Top-10 nearest neighbors by cosine distance
|
||||||
|
SELECT id FROM items
|
||||||
|
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') ASC
|
||||||
|
LIMIT 10;
|
||||||
|
|
||||||
|
-- Top-5 nearest neighbors by Euclidean distance
|
||||||
|
SELECT id FROM items
|
||||||
|
ORDER BY embedding <-> '[0.1, 0.2, 0.3, 0.4]'
|
||||||
LIMIT 5;
|
LIMIT 5;
|
||||||
|
|
||||||
-- With metadata filter
|
-- With metadata filter
|
||||||
SELECT title FROM articles
|
SELECT id FROM items
|
||||||
WHERE category = 'tech'
|
WHERE category = 'tech'
|
||||||
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
|
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]')
|
||||||
LIMIT 5;
|
LIMIT 5;
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Vector Indexes
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Create HNSW index for approximate nearest neighbor search
|
||||||
|
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
|
||||||
|
|
||||||
|
-- Supported index methods: hnsw, ivfpq
|
||||||
|
```
|
||||||
|
|
||||||
## Graph Patterns
|
## Graph Patterns
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
@@ -575,7 +616,7 @@ SUM(salary) OVER (
|
|||||||
| Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT |
|
| Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT |
|
||||||
| Graph | MATCH, RETURN, WHERE, shortestPath, type |
|
| Graph | MATCH, RETURN, WHERE, shortestPath, type |
|
||||||
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
|
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
|
||||||
| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance |
|
| Vector | cosine_distance, euclidean_distance, inner_product, l1_distance, l2_distance, <-> |
|
||||||
| JSON | ->, ->> |
|
| JSON | ->, ->> |
|
||||||
| FTS | @@ (BM25 match) |
|
| FTS | @@ (BM25 match) |
|
||||||
| Recovery | RECOVER TO TIMESTAMP |
|
| Recovery | RECOVER TO TIMESTAMP |
|
||||||
|
|||||||
@@ -176,10 +176,20 @@ All notable changes to BaraDB are documented in this file.
|
|||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
- **Vector SQL Integration** — Full SQL-level vector search support:
|
||||||
|
- `VECTOR(n)` column type in `CREATE TABLE` with dimension validation
|
||||||
|
- `CREATE INDEX ... USING hnsw` / `USING ivfpq` for approximate nearest neighbor indexes
|
||||||
|
- SQL distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
|
||||||
|
- `<->` nearest-neighbor operator (Euclidean distance)
|
||||||
|
- `ORDER BY` support for vector distance expressions, including columns not in `SELECT`
|
||||||
|
- Automatic HNSW index maintenance on `INSERT` and `UPDATE`
|
||||||
|
- **Advanced SQL Engine** — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE)
|
||||||
- **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire.
|
- **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire.
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
- **Query Executor — Row Value Escaping** — `execInsert` now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like `[1.0, 2.0, 3.0]`
|
||||||
|
- **Query Planner — ORDER BY Projection** — `irpkSort` is now placed before `irpkProject` in the IR plan, allowing `ORDER BY` to reference columns that are not selected
|
||||||
- **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy.
|
- **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy.
|
||||||
- **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives.
|
- **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives.
|
||||||
|
|
||||||
|
|||||||
+89
-8
@@ -1,8 +1,89 @@
|
|||||||
# Vector Search Engine
|
# Vector Search Engine
|
||||||
|
|
||||||
Native HNSW and IVF-PQ indexes for similarity search.
|
Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
|
||||||
|
|
||||||
## Usage
|
## SQL Usage
|
||||||
|
|
||||||
|
### Creating Vector Columns
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE items (
|
||||||
|
id INT PRIMARY KEY,
|
||||||
|
embedding VECTOR(768)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
The `VECTOR(n)` type stores float32 arrays of fixed dimension `n`.
|
||||||
|
|
||||||
|
### Inserting Vectors
|
||||||
|
|
||||||
|
```sql
|
||||||
|
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vector Distance Functions
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Cosine distance (0 = identical, 1 = orthogonal)
|
||||||
|
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- Euclidean / L2 distance
|
||||||
|
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- L2 distance with <-> operator
|
||||||
|
SELECT id, embedding <-> '[0.1, 0.2, 0.3]' AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- Inner product (negative dot product for minimization)
|
||||||
|
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3]') AS dist
|
||||||
|
FROM items;
|
||||||
|
|
||||||
|
-- Manhattan / L1 distance
|
||||||
|
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
|
||||||
|
FROM items;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nearest Neighbor Search
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Top-10 nearest neighbors by cosine distance
|
||||||
|
SELECT id FROM items
|
||||||
|
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3]') ASC
|
||||||
|
LIMIT 10;
|
||||||
|
|
||||||
|
-- Top-5 nearest neighbors by Euclidean distance
|
||||||
|
SELECT id FROM items
|
||||||
|
ORDER BY embedding <-> '[0.1, 0.2, 0.3]'
|
||||||
|
LIMIT 5;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vector Indexes
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Create HNSW index for approximate nearest neighbor search
|
||||||
|
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
|
||||||
|
|
||||||
|
-- The index is automatically maintained on INSERT and UPDATE
|
||||||
|
```
|
||||||
|
|
||||||
|
Supported index methods:
|
||||||
|
- `USING hnsw` — Hierarchical Navigable Small World (default: cosine metric)
|
||||||
|
- `USING ivfpq` — Inverted File with Product Quantization
|
||||||
|
|
||||||
|
### Dimension Validation
|
||||||
|
|
||||||
|
BaraDB validates vector dimensions at insert time:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- This will fail: expected 768 dimensions but got 3
|
||||||
|
INSERT INTO items (id, embedding) VALUES (2, '[1.0, 2.0, 3.0]');
|
||||||
|
```
|
||||||
|
|
||||||
|
## Native Nim API
|
||||||
|
|
||||||
|
For embedded or high-performance use, use the native Nim API directly:
|
||||||
|
|
||||||
```nim
|
```nim
|
||||||
import barabadb/vector/engine
|
import barabadb/vector/engine
|
||||||
@@ -48,12 +129,12 @@ var ivfpq = newIVFPQIndex(
|
|||||||
|
|
||||||
## Distance Metrics
|
## Distance Metrics
|
||||||
|
|
||||||
| Metric | Description |
|
| Metric | SQL Function | Description |
|
||||||
|--------|-------------|
|
|--------|--------------|-------------|
|
||||||
| `cosine` | Cosine similarity |
|
| `cosine` | `cosine_distance(a, b)` | Cosine dissimilarity (1 - similarity) |
|
||||||
| `euclidean` | L2 distance |
|
| `euclidean` | `euclidean_distance(a, b)` / `<->` | L2 distance |
|
||||||
| `dotproduct` | Dot product similarity |
|
| `dotproduct` | `inner_product(a, b)` | Negative dot product |
|
||||||
| `manhattan` | L1 distance |
|
| `manhattan` | `l1_distance(a, b)` | L1 distance |
|
||||||
|
|
||||||
## Quantization
|
## Quantization
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user