baraba/Baradb
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(en): Update English docs for Vector SQL Integration

Browse Source 
- docs/en/vector.md — add SQL usage section (CREATE TABLE VECTOR,
  distance functions, <-> operator, CREATE INDEX USING hnsw)
- docs/en/baraql.md — update vector search section with real SQL syntax,
  add VECTOR(n) to data types, update keyword table
- docs/en/changelog.md — add Vector SQL Integration and bugfixes to [Unreleased]
- docs/ARCHITECTURE.md — add SQL Integration bullet to Vector Engine
- README.md — update vector engine section with SQL examples,
  add Vector SQL to roadmap, bump test count to 340+
This commit is contained in:
dimgigov
2026-05-14 14:20:57 +03:00
parent d076cfde3b
commit b0978812cb
5 changed files with 188 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+22 -3
View File
@@ -285,8 +285,23 @@ let range = btree.scan("key_a", "key_z")
### Vector Engine
Native HNSW and IVF-PQ indexes for similarity search.
Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
```sql
-- SQL vector search
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
-- Nearest neighbor search
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, ...]') ASC
LIMIT 10;
-- With HNSW index
CREATE INDEX idx_vec ON items(embedding) USING hnsw;
```
Native Nim API:
```nim
import barabadb/vector/engine
@@ -301,7 +316,10 @@ let filtered = idx.searchWithFilter(queryVector, k = 10,
```
Features:
- **HNSW** — hierarchical navigable small world graph
- **SQL vector types** — `VECTOR(n)` with dimension validation
- **SQL distance functions** — `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- **`<->` operator** — Euclidean distance nearest-neighbor shorthand
- **HNSW index** — `CREATE INDEX ... USING hnsw` with automatic maintenance
- **IVF-PQ** — inverted file index with product quantization
- **Distance metrics** — cosine, euclidean, dot product, Manhattan
- **Quantization** — scalar 8-bit/4-bit, product, binary
@@ -1231,7 +1249,7 @@ src/barabadb/
## Tests
```bash
# Run all tests (262 tests, 56 suites)
# Run all tests (340+ tests, 60+ suites)
nim c --path:src -r tests/test_all.nim
# Run benchmarks
@@ -1249,6 +1267,7 @@ nim c -d:release -r benchmarks/bench_all.nim
| Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 |
| Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 |
| Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 |
| Vector SQL Integration (VECTOR type, distance functions, <->, HNSW indexes) | ✅ | 100% | v1.1.0 |
| Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 |
| FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 |
| CLI shell | ✅ | 100% | v1.0.0 |
docs/ARCHITECTURE.md
+6
View File
@@ -90,6 +90,12 @@ The query layer processes BaraQL — a SQL-compatible query language with extens
- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression.
- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan).
- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput.
- **SQL Integration** (`query/executor.nim`):
- `VECTOR(n)` column type with dimension validation
- `CREATE INDEX ... USING hnsw` / `USING ivfpq`
- Distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- `<->` nearest-neighbor operator
- Automatic index maintenance on INSERT/UPDATE
### Graph Engine (`graph/`)
- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency.
docs/en/baraql.md
+61 -20
View File
@@ -18,6 +18,7 @@ BaraQL is a SQL-compatible query language with extensions for graph, vector, and
| `bytes` | Raw bytes | `0xDEADBEEF` |
| `array<T>` | Homogeneous array | `[1, 2, 3]` |
| `vector` | Float32 vector | `[0.1, 0.2, 0.3]` |
| `vector(n)` | Fixed-dimension float32 vector (SQL) | `VECTOR(768)` |
| `object` | Key-value object | `{"a": 1}` |
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
@@ -352,6 +353,7 @@ CREATE TYPE Cat EXTENDING Animal {
CREATE INDEX idx_users_name ON users(name);
CREATE UNIQUE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_age ON users(age) USING btree;
CREATE INDEX idx_vectors ON items(embedding) USING hnsw;
```
### DROP
@@ -387,37 +389,76 @@ SELECT * FROM articles WHERE body @@ 'machine learning';
RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
```
## Vector Search
## Vector Search (SQL)
### Creating Vector Columns
```sql
-- Insert with vector
INSERT articles {
title := 'Nim Programming',
embedding := [0.1, 0.2, 0.3, 0.4]
};
CREATE TABLE items (
id INT PRIMARY KEY,
embedding VECTOR(768)
);
```
-- Similarity search (cosine distance)
SELECT title FROM articles
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
LIMIT 5;
### Inserting Vectors
-- Euclidean distance
SELECT title FROM articles
ORDER BY l2_distance(embedding, [0.1, 0.2, 0.3, 0.4])
LIMIT 5;
```sql
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, 0.4]');
```
-- Dot product
SELECT title FROM articles
ORDER BY dot_product(embedding, [0.1, 0.2, 0.3, 0.4]) DESC
### Distance Functions
```sql
-- Cosine distance (0 = identical, 2 = opposite)
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- Euclidean / L2 distance
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- L2 distance with <-> operator
SELECT id, embedding <-> '[0.1, 0.2, 0.3, 0.4]' AS dist
FROM items;
-- Inner product (negative dot product)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- Manhattan / L1 distance
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
```
### Nearest Neighbor Search
```sql
-- Top-10 nearest neighbors by cosine distance
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') ASC
LIMIT 10;
-- Top-5 nearest neighbors by Euclidean distance
SELECT id FROM items
ORDER BY embedding <-> '[0.1, 0.2, 0.3, 0.4]'
LIMIT 5;
-- With metadata filter
SELECT title FROM articles
SELECT id FROM items
WHERE category = 'tech'
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]')
LIMIT 5;
```
### Vector Indexes
```sql
-- Create HNSW index for approximate nearest neighbor search
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
-- Supported index methods: hnsw, ivfpq
```
## Graph Patterns
```sql
@@ -575,7 +616,7 @@ SUM(salary) OVER (
| Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT |
| Graph | MATCH, RETURN, WHERE, shortestPath, type |
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance |
| Vector | cosine_distance, euclidean_distance, inner_product, l1_distance, l2_distance, <-> |
| JSON | ->, ->> |
| FTS | @@ (BM25 match) |
| Recovery | RECOVER TO TIMESTAMP |
docs/en/changelog.md
+10
View File
@@ -176,10 +176,20 @@ All notable changes to BaraDB are documented in this file.
### Added
- **Vector SQL Integration** — Full SQL-level vector search support:
- `VECTOR(n)` column type in `CREATE TABLE` with dimension validation
- `CREATE INDEX ... USING hnsw` / `USING ivfpq` for approximate nearest neighbor indexes
- SQL distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- `<->` nearest-neighbor operator (Euclidean distance)
- `ORDER BY` support for vector distance expressions, including columns not in `SELECT`
- Automatic HNSW index maintenance on `INSERT` and `UPDATE`
- **Advanced SQL Engine** — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE)
- **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire.
### Fixed
- **Query Executor — Row Value Escaping** — `execInsert` now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like `[1.0, 2.0, 3.0]`
- **Query Planner — ORDER BY Projection** — `irpkSort` is now placed before `irpkProject` in the IR plan, allowing `ORDER BY` to reference columns that are not selected
- **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy.
- **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives.
docs/en/vector.md
+89 -8
View File
@@ -1,8 +1,89 @@
# Vector Search Engine
Native HNSW and IVF-PQ indexes for similarity search.
Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
## Usage
## SQL Usage
### Creating Vector Columns
```sql
CREATE TABLE items (
id INT PRIMARY KEY,
embedding VECTOR(768)
);
```
The `VECTOR(n)` type stores float32 arrays of fixed dimension `n`.
### Inserting Vectors
```sql
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
```
### Vector Distance Functions
```sql
-- Cosine distance (0 = identical, 1 = orthogonal)
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- Euclidean / L2 distance
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- L2 distance with <-> operator
SELECT id, embedding <-> '[0.1, 0.2, 0.3]' AS dist
FROM items;
-- Inner product (negative dot product for minimization)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- Manhattan / L1 distance
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
```
### Nearest Neighbor Search
```sql
-- Top-10 nearest neighbors by cosine distance
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3]') ASC
LIMIT 10;
-- Top-5 nearest neighbors by Euclidean distance
SELECT id FROM items
ORDER BY embedding <-> '[0.1, 0.2, 0.3]'
LIMIT 5;
```
### Vector Indexes
```sql
-- Create HNSW index for approximate nearest neighbor search
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
-- The index is automatically maintained on INSERT and UPDATE
```
Supported index methods:
- `USING hnsw` — Hierarchical Navigable Small World (default: cosine metric)
- `USING ivfpq` — Inverted File with Product Quantization
### Dimension Validation
BaraDB validates vector dimensions at insert time:
```sql
-- This will fail: expected 768 dimensions but got 3
INSERT INTO items (id, embedding) VALUES (2, '[1.0, 2.0, 3.0]');
```
## Native Nim API
For embedded or high-performance use, use the native Nim API directly:
```nim
import barabadb/vector/engine
@@ -48,12 +129,12 @@ var ivfpq = newIVFPQIndex(
## Distance Metrics
| Metric | Description |
|--------|-------------|
| `cosine` | Cosine similarity |
| `euclidean` | L2 distance |
| `dotproduct` | Dot product similarity |
| `manhattan` | L1 distance |
| Metric | SQL Function | Description |
|--------|--------------|-------------|
| `cosine` | `cosine_distance(a, b)` | Cosine dissimilarity (1 - similarity) |
| `euclidean` | `euclidean_distance(a, b)` / `<->` | L2 distance |
| `dotproduct` | `inner_product(a, b)` | Negative dot product |
| `manhattan` | `l1_distance(a, b)` | L1 distance |
## Quantization