From b0978812cb87feeb2d584788500155e2fadc4293 Mon Sep 17 00:00:00 2001 From: dimgigov Date: Thu, 14 May 2026 14:20:57 +0300 Subject: [PATCH] docs(en): Update English docs for Vector SQL Integration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/en/vector.md — add SQL usage section (CREATE TABLE VECTOR, distance functions, <-> operator, CREATE INDEX USING hnsw) - docs/en/baraql.md — update vector search section with real SQL syntax, add VECTOR(n) to data types, update keyword table - docs/en/changelog.md — add Vector SQL Integration and bugfixes to [Unreleased] - docs/ARCHITECTURE.md — add SQL Integration bullet to Vector Engine - README.md — update vector engine section with SQL examples, add Vector SQL to roadmap, bump test count to 340+ --- README.md | 25 ++++++++++-- docs/ARCHITECTURE.md | 6 +++ docs/en/baraql.md | 81 +++++++++++++++++++++++++++--------- docs/en/changelog.md | 10 +++++ docs/en/vector.md | 97 ++++++++++++++++++++++++++++++++++++++++---- 5 files changed, 188 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index d91a87a..0474744 100644 --- a/README.md +++ b/README.md @@ -285,8 +285,23 @@ let range = btree.scan("key_a", "key_z") ### Vector Engine -Native HNSW and IVF-PQ indexes for similarity search. +Native HNSW and IVF-PQ indexes for similarity search with full SQL integration. +```sql +-- SQL vector search +CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768)); +INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]'); + +-- Nearest neighbor search +SELECT id FROM items +ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, ...]') ASC +LIMIT 10; + +-- With HNSW index +CREATE INDEX idx_vec ON items(embedding) USING hnsw; +``` + +Native Nim API: ```nim import barabadb/vector/engine @@ -301,7 +316,10 @@ let filtered = idx.searchWithFilter(queryVector, k = 10, ``` Features: -- **HNSW** — hierarchical navigable small world graph +- **SQL vector types** — `VECTOR(n)` with dimension validation +- **SQL distance functions** — `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()` +- **`<->` operator** — Euclidean distance nearest-neighbor shorthand +- **HNSW index** — `CREATE INDEX ... USING hnsw` with automatic maintenance - **IVF-PQ** — inverted file index with product quantization - **Distance metrics** — cosine, euclidean, dot product, Manhattan - **Quantization** — scalar 8-bit/4-bit, product, binary @@ -1231,7 +1249,7 @@ src/barabadb/ ## Tests ```bash -# Run all tests (262 tests, 56 suites) +# Run all tests (340+ tests, 60+ suites) nim c --path:src -r tests/test_all.nim # Run benchmarks @@ -1249,6 +1267,7 @@ nim c -d:release -r benchmarks/bench_all.nim | Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 | | Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 | | Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 | +| Vector SQL Integration (VECTOR type, distance functions, <->, HNSW indexes) | ✅ | 100% | v1.1.0 | | Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 | | FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 | | CLI shell | ✅ | 100% | v1.0.0 | diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index b67bdbd..e5192f9 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -90,6 +90,12 @@ The query layer processes BaraQL — a SQL-compatible query language with extens - **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression. - **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan). - **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput. +- **SQL Integration** (`query/executor.nim`): + - `VECTOR(n)` column type with dimension validation + - `CREATE INDEX ... USING hnsw` / `USING ivfpq` + - Distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()` + - `<->` nearest-neighbor operator + - Automatic index maintenance on INSERT/UPDATE ### Graph Engine (`graph/`) - **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency. diff --git a/docs/en/baraql.md b/docs/en/baraql.md index daf118c..f4b9061 100644 --- a/docs/en/baraql.md +++ b/docs/en/baraql.md @@ -18,6 +18,7 @@ BaraQL is a SQL-compatible query language with extensions for graph, vector, and | `bytes` | Raw bytes | `0xDEADBEEF` | | `array` | Homogeneous array | `[1, 2, 3]` | | `vector` | Float32 vector | `[0.1, 0.2, 0.3]` | +| `vector(n)` | Fixed-dimension float32 vector (SQL) | `VECTOR(768)` | | `object` | Key-value object | `{"a": 1}` | | `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` | | `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` | @@ -352,6 +353,7 @@ CREATE TYPE Cat EXTENDING Animal { CREATE INDEX idx_users_name ON users(name); CREATE UNIQUE INDEX idx_users_email ON users(email); CREATE INDEX idx_users_age ON users(age) USING btree; +CREATE INDEX idx_vectors ON items(embedding) USING hnsw; ``` ### DROP @@ -387,37 +389,76 @@ SELECT * FROM articles WHERE body @@ 'machine learning'; RECOVER TO TIMESTAMP '2026-05-07T12:00:00'; ``` -## Vector Search +## Vector Search (SQL) + +### Creating Vector Columns ```sql --- Insert with vector -INSERT articles { - title := 'Nim Programming', - embedding := [0.1, 0.2, 0.3, 0.4] -}; +CREATE TABLE items ( + id INT PRIMARY KEY, + embedding VECTOR(768) +); +``` --- Similarity search (cosine distance) -SELECT title FROM articles -ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4]) -LIMIT 5; +### Inserting Vectors --- Euclidean distance -SELECT title FROM articles -ORDER BY l2_distance(embedding, [0.1, 0.2, 0.3, 0.4]) -LIMIT 5; +```sql +INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, 0.4]'); +``` --- Dot product -SELECT title FROM articles -ORDER BY dot_product(embedding, [0.1, 0.2, 0.3, 0.4]) DESC +### Distance Functions + +```sql +-- Cosine distance (0 = identical, 2 = opposite) +SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist +FROM items; + +-- Euclidean / L2 distance +SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist +FROM items; + +-- L2 distance with <-> operator +SELECT id, embedding <-> '[0.1, 0.2, 0.3, 0.4]' AS dist +FROM items; + +-- Inner product (negative dot product) +SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist +FROM items; + +-- Manhattan / L1 distance +SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist +FROM items; +``` + +### Nearest Neighbor Search + +```sql +-- Top-10 nearest neighbors by cosine distance +SELECT id FROM items +ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') ASC +LIMIT 10; + +-- Top-5 nearest neighbors by Euclidean distance +SELECT id FROM items +ORDER BY embedding <-> '[0.1, 0.2, 0.3, 0.4]' LIMIT 5; -- With metadata filter -SELECT title FROM articles +SELECT id FROM items WHERE category = 'tech' -ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4]) +ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') LIMIT 5; ``` +### Vector Indexes + +```sql +-- Create HNSW index for approximate nearest neighbor search +CREATE INDEX idx_items_vec ON items(embedding) USING hnsw; + +-- Supported index methods: hnsw, ivfpq +``` + ## Graph Patterns ```sql @@ -575,7 +616,7 @@ SUM(salary) OVER ( | Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT | | Graph | MATCH, RETURN, WHERE, shortestPath, type | | FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS | -| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance | +| Vector | cosine_distance, euclidean_distance, inner_product, l1_distance, l2_distance, <-> | | JSON | ->, ->> | | FTS | @@ (BM25 match) | | Recovery | RECOVER TO TIMESTAMP | diff --git a/docs/en/changelog.md b/docs/en/changelog.md index 14cad68..9356412 100644 --- a/docs/en/changelog.md +++ b/docs/en/changelog.md @@ -176,10 +176,20 @@ All notable changes to BaraDB are documented in this file. ### Added +- **Vector SQL Integration** — Full SQL-level vector search support: + - `VECTOR(n)` column type in `CREATE TABLE` with dimension validation + - `CREATE INDEX ... USING hnsw` / `USING ivfpq` for approximate nearest neighbor indexes + - SQL distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()` + - `<->` nearest-neighbor operator (Euclidean distance) + - `ORDER BY` support for vector distance expressions, including columns not in `SELECT` + - Automatic HNSW index maintenance on `INSERT` and `UPDATE` +- **Advanced SQL Engine** — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE) - **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire. ### Fixed +- **Query Executor — Row Value Escaping** — `execInsert` now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like `[1.0, 2.0, 3.0]` +- **Query Planner — ORDER BY Projection** — `irpkSort` is now placed before `irpkProject` in the IR plan, allowing `ORDER BY` to reference columns that are not selected - **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy. - **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives. diff --git a/docs/en/vector.md b/docs/en/vector.md index b575cd5..50ce572 100644 --- a/docs/en/vector.md +++ b/docs/en/vector.md @@ -1,8 +1,89 @@ # Vector Search Engine -Native HNSW and IVF-PQ indexes for similarity search. +Native HNSW and IVF-PQ indexes for similarity search with full SQL integration. -## Usage +## SQL Usage + +### Creating Vector Columns + +```sql +CREATE TABLE items ( + id INT PRIMARY KEY, + embedding VECTOR(768) +); +``` + +The `VECTOR(n)` type stores float32 arrays of fixed dimension `n`. + +### Inserting Vectors + +```sql +INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]'); +``` + +### Vector Distance Functions + +```sql +-- Cosine distance (0 = identical, 1 = orthogonal) +SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3]') AS dist +FROM items; + +-- Euclidean / L2 distance +SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3]') AS dist +FROM items; + +-- L2 distance with <-> operator +SELECT id, embedding <-> '[0.1, 0.2, 0.3]' AS dist +FROM items; + +-- Inner product (negative dot product for minimization) +SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3]') AS dist +FROM items; + +-- Manhattan / L1 distance +SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3]') AS dist +FROM items; +``` + +### Nearest Neighbor Search + +```sql +-- Top-10 nearest neighbors by cosine distance +SELECT id FROM items +ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3]') ASC +LIMIT 10; + +-- Top-5 nearest neighbors by Euclidean distance +SELECT id FROM items +ORDER BY embedding <-> '[0.1, 0.2, 0.3]' +LIMIT 5; +``` + +### Vector Indexes + +```sql +-- Create HNSW index for approximate nearest neighbor search +CREATE INDEX idx_items_vec ON items(embedding) USING hnsw; + +-- The index is automatically maintained on INSERT and UPDATE +``` + +Supported index methods: +- `USING hnsw` — Hierarchical Navigable Small World (default: cosine metric) +- `USING ivfpq` — Inverted File with Product Quantization + +### Dimension Validation + +BaraDB validates vector dimensions at insert time: + +```sql +-- This will fail: expected 768 dimensions but got 3 +INSERT INTO items (id, embedding) VALUES (2, '[1.0, 2.0, 3.0]'); +``` + +## Native Nim API + +For embedded or high-performance use, use the native Nim API directly: ```nim import barabadb/vector/engine @@ -48,12 +129,12 @@ var ivfpq = newIVFPQIndex( ## Distance Metrics -| Metric | Description | -|--------|-------------| -| `cosine` | Cosine similarity | -| `euclidean` | L2 distance | -| `dotproduct` | Dot product similarity | -| `manhattan` | L1 distance | +| Metric | SQL Function | Description | +|--------|--------------|-------------| +| `cosine` | `cosine_distance(a, b)` | Cosine dissimilarity (1 - similarity) | +| `euclidean` | `euclidean_distance(a, b)` / `<->` | L2 distance | +| `dotproduct` | `inner_product(a, b)` | Negative dot product | +| `manhattan` | `l1_distance(a, b)` | L1 distance | ## Quantization