docs(en): Update English docs for Vector SQL Integration

- docs/en/vector.md — add SQL usage section (CREATE TABLE VECTOR,
  distance functions, <-> operator, CREATE INDEX USING hnsw)
- docs/en/baraql.md — update vector search section with real SQL syntax,
  add VECTOR(n) to data types, update keyword table
- docs/en/changelog.md — add Vector SQL Integration and bugfixes to [Unreleased]
- docs/ARCHITECTURE.md — add SQL Integration bullet to Vector Engine
- README.md — update vector engine section with SQL examples,
  add Vector SQL to roadmap, bump test count to 340+
This commit is contained in:
2026-05-14 14:20:57 +03:00
parent d076cfde3b
commit b0978812cb
5 changed files with 188 additions and 31 deletions
+22 -3
View File
@@ -285,8 +285,23 @@ let range = btree.scan("key_a", "key_z")
### Vector Engine ### Vector Engine
Native HNSW and IVF-PQ indexes for similarity search. Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
```sql
-- SQL vector search
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
-- Nearest neighbor search
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, ...]') ASC
LIMIT 10;
-- With HNSW index
CREATE INDEX idx_vec ON items(embedding) USING hnsw;
```
Native Nim API:
```nim ```nim
import barabadb/vector/engine import barabadb/vector/engine
@@ -301,7 +316,10 @@ let filtered = idx.searchWithFilter(queryVector, k = 10,
``` ```
Features: Features:
- **HNSW** — hierarchical navigable small world graph - **SQL vector types** — `VECTOR(n)` with dimension validation
- **SQL distance functions** — `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- **`<->` operator** — Euclidean distance nearest-neighbor shorthand
- **HNSW index** — `CREATE INDEX ... USING hnsw` with automatic maintenance
- **IVF-PQ** — inverted file index with product quantization - **IVF-PQ** — inverted file index with product quantization
- **Distance metrics** — cosine, euclidean, dot product, Manhattan - **Distance metrics** — cosine, euclidean, dot product, Manhattan
- **Quantization** — scalar 8-bit/4-bit, product, binary - **Quantization** — scalar 8-bit/4-bit, product, binary
@@ -1231,7 +1249,7 @@ src/barabadb/
## Tests ## Tests
```bash ```bash
# Run all tests (262 tests, 56 suites) # Run all tests (340+ tests, 60+ suites)
nim c --path:src -r tests/test_all.nim nim c --path:src -r tests/test_all.nim
# Run benchmarks # Run benchmarks
@@ -1249,6 +1267,7 @@ nim c -d:release -r benchmarks/bench_all.nim
| Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 | | Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 |
| Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 | | Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 |
| Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 | | Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 |
| Vector SQL Integration (VECTOR type, distance functions, <->, HNSW indexes) | ✅ | 100% | v1.1.0 |
| Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 | | Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 |
| FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 | | FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 |
| CLI shell | ✅ | 100% | v1.0.0 | | CLI shell | ✅ | 100% | v1.0.0 |
+6
View File
@@ -90,6 +90,12 @@ The query layer processes BaraQL — a SQL-compatible query language with extens
- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression. - **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression.
- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan). - **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan).
- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput. - **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput.
- **SQL Integration** (`query/executor.nim`):
- `VECTOR(n)` column type with dimension validation
- `CREATE INDEX ... USING hnsw` / `USING ivfpq`
- Distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- `<->` nearest-neighbor operator
- Automatic index maintenance on INSERT/UPDATE
### Graph Engine (`graph/`) ### Graph Engine (`graph/`)
- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency. - **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency.
+61 -20
View File
@@ -18,6 +18,7 @@ BaraQL is a SQL-compatible query language with extensions for graph, vector, and
| `bytes` | Raw bytes | `0xDEADBEEF` | | `bytes` | Raw bytes | `0xDEADBEEF` |
| `array<T>` | Homogeneous array | `[1, 2, 3]` | | `array<T>` | Homogeneous array | `[1, 2, 3]` |
| `vector` | Float32 vector | `[0.1, 0.2, 0.3]` | | `vector` | Float32 vector | `[0.1, 0.2, 0.3]` |
| `vector(n)` | Fixed-dimension float32 vector (SQL) | `VECTOR(768)` |
| `object` | Key-value object | `{"a": 1}` | | `object` | Key-value object | `{"a": 1}` |
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` | | `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` | | `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
@@ -352,6 +353,7 @@ CREATE TYPE Cat EXTENDING Animal {
CREATE INDEX idx_users_name ON users(name); CREATE INDEX idx_users_name ON users(name);
CREATE UNIQUE INDEX idx_users_email ON users(email); CREATE UNIQUE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_age ON users(age) USING btree; CREATE INDEX idx_users_age ON users(age) USING btree;
CREATE INDEX idx_vectors ON items(embedding) USING hnsw;
``` ```
### DROP ### DROP
@@ -387,37 +389,76 @@ SELECT * FROM articles WHERE body @@ 'machine learning';
RECOVER TO TIMESTAMP '2026-05-07T12:00:00'; RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
``` ```
## Vector Search ## Vector Search (SQL)
### Creating Vector Columns
```sql ```sql
-- Insert with vector CREATE TABLE items (
INSERT articles { id INT PRIMARY KEY,
title := 'Nim Programming', embedding VECTOR(768)
embedding := [0.1, 0.2, 0.3, 0.4] );
}; ```
-- Similarity search (cosine distance) ### Inserting Vectors
SELECT title FROM articles
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
LIMIT 5;
-- Euclidean distance ```sql
SELECT title FROM articles INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, 0.4]');
ORDER BY l2_distance(embedding, [0.1, 0.2, 0.3, 0.4]) ```
LIMIT 5;
-- Dot product ### Distance Functions
SELECT title FROM articles
ORDER BY dot_product(embedding, [0.1, 0.2, 0.3, 0.4]) DESC ```sql
-- Cosine distance (0 = identical, 2 = opposite)
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- Euclidean / L2 distance
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- L2 distance with <-> operator
SELECT id, embedding <-> '[0.1, 0.2, 0.3, 0.4]' AS dist
FROM items;
-- Inner product (negative dot product)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- Manhattan / L1 distance
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
```
### Nearest Neighbor Search
```sql
-- Top-10 nearest neighbors by cosine distance
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') ASC
LIMIT 10;
-- Top-5 nearest neighbors by Euclidean distance
SELECT id FROM items
ORDER BY embedding <-> '[0.1, 0.2, 0.3, 0.4]'
LIMIT 5; LIMIT 5;
-- With metadata filter -- With metadata filter
SELECT title FROM articles SELECT id FROM items
WHERE category = 'tech' WHERE category = 'tech'
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4]) ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]')
LIMIT 5; LIMIT 5;
``` ```
### Vector Indexes
```sql
-- Create HNSW index for approximate nearest neighbor search
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
-- Supported index methods: hnsw, ivfpq
```
## Graph Patterns ## Graph Patterns
```sql ```sql
@@ -575,7 +616,7 @@ SUM(salary) OVER (
| Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT | | Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT |
| Graph | MATCH, RETURN, WHERE, shortestPath, type | | Graph | MATCH, RETURN, WHERE, shortestPath, type |
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS | | FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance | | Vector | cosine_distance, euclidean_distance, inner_product, l1_distance, l2_distance, <-> |
| JSON | ->, ->> | | JSON | ->, ->> |
| FTS | @@ (BM25 match) | | FTS | @@ (BM25 match) |
| Recovery | RECOVER TO TIMESTAMP | | Recovery | RECOVER TO TIMESTAMP |
+10
View File
@@ -176,10 +176,20 @@ All notable changes to BaraDB are documented in this file.
### Added ### Added
- **Vector SQL Integration** — Full SQL-level vector search support:
- `VECTOR(n)` column type in `CREATE TABLE` with dimension validation
- `CREATE INDEX ... USING hnsw` / `USING ivfpq` for approximate nearest neighbor indexes
- SQL distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- `<->` nearest-neighbor operator (Euclidean distance)
- `ORDER BY` support for vector distance expressions, including columns not in `SELECT`
- Automatic HNSW index maintenance on `INSERT` and `UPDATE`
- **Advanced SQL Engine** — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE)
- **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire. - **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire.
### Fixed ### Fixed
- **Query Executor — Row Value Escaping** — `execInsert` now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like `[1.0, 2.0, 3.0]`
- **Query Planner — ORDER BY Projection** — `irpkSort` is now placed before `irpkProject` in the IR plan, allowing `ORDER BY` to reference columns that are not selected
- **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy. - **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy.
- **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives. - **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives.
+89 -8
View File
@@ -1,8 +1,89 @@
# Vector Search Engine # Vector Search Engine
Native HNSW and IVF-PQ indexes for similarity search. Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
## Usage ## SQL Usage
### Creating Vector Columns
```sql
CREATE TABLE items (
id INT PRIMARY KEY,
embedding VECTOR(768)
);
```
The `VECTOR(n)` type stores float32 arrays of fixed dimension `n`.
### Inserting Vectors
```sql
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
```
### Vector Distance Functions
```sql
-- Cosine distance (0 = identical, 1 = orthogonal)
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- Euclidean / L2 distance
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- L2 distance with <-> operator
SELECT id, embedding <-> '[0.1, 0.2, 0.3]' AS dist
FROM items;
-- Inner product (negative dot product for minimization)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- Manhattan / L1 distance
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
```
### Nearest Neighbor Search
```sql
-- Top-10 nearest neighbors by cosine distance
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3]') ASC
LIMIT 10;
-- Top-5 nearest neighbors by Euclidean distance
SELECT id FROM items
ORDER BY embedding <-> '[0.1, 0.2, 0.3]'
LIMIT 5;
```
### Vector Indexes
```sql
-- Create HNSW index for approximate nearest neighbor search
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
-- The index is automatically maintained on INSERT and UPDATE
```
Supported index methods:
- `USING hnsw` — Hierarchical Navigable Small World (default: cosine metric)
- `USING ivfpq` — Inverted File with Product Quantization
### Dimension Validation
BaraDB validates vector dimensions at insert time:
```sql
-- This will fail: expected 768 dimensions but got 3
INSERT INTO items (id, embedding) VALUES (2, '[1.0, 2.0, 3.0]');
```
## Native Nim API
For embedded or high-performance use, use the native Nim API directly:
```nim ```nim
import barabadb/vector/engine import barabadb/vector/engine
@@ -48,12 +129,12 @@ var ivfpq = newIVFPQIndex(
## Distance Metrics ## Distance Metrics
| Metric | Description | | Metric | SQL Function | Description |
|--------|-------------| |--------|--------------|-------------|
| `cosine` | Cosine similarity | | `cosine` | `cosine_distance(a, b)` | Cosine dissimilarity (1 - similarity) |
| `euclidean` | L2 distance | | `euclidean` | `euclidean_distance(a, b)` / `<->` | L2 distance |
| `dotproduct` | Dot product similarity | | `dotproduct` | `inner_product(a, b)` | Negative dot product |
| `manhattan` | L1 distance | | `manhattan` | `l1_distance(a, b)` | L1 distance |
## Quantization ## Quantization