baraba/Baradb
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add German (DE) documentation + update all docs for Sessions 10-12
CI / test (push) Has been cancelled
Details
CI / verify (push) Has been cancelled
Details

Browse Source 
New German documentation (docs/de/):
- index.md, quickstart.md, installation.md
- baraql.md, graph.md, vector.md, mcp.md

Updated English documentation:
- changelog.md: all Sessions 10-12 features
- graph.md: SQL GRAPH_TABLE, CREATE GRAPH, all 8 algorithms, Cypher, similarity_nodes, node2vec
- vector.md: hybrid RAG, chunk(), embed_text(), auto-embed, nl_to_sql(), schema_prompt()
- baraql.md: new AI & Cross-Modal Functions section, updated keyword tables
- mcp.md: MCP Server documentation (new file)
- index.md: added German (DE) language link
This commit is contained in:
dimgigov
2026-05-17 16:15:45 +03:00
parent e783215276
commit a5d34c001a
13 changed files with 1194 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/en/vector.md
+91 -71
View File
@@ -1,6 +1,7 @@
# Vector Search Engine
Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
Includes AI pipeline for chunking, embedding, and hybrid RAG search.
## SQL Usage
@@ -8,8 +9,8 @@ Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
```sql
CREATE TABLE items (
id INT PRIMARY KEY,
embedding VECTOR(768)
id INT PRIMARY KEY,
embedding VECTOR(768)
);
```
@@ -25,24 +26,17 @@ INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
```sql
-- Cosine distance (0 = identical, 1 = orthogonal)
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3]') AS dist FROM items;
-- Euclidean / L2 distance
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3]') AS dist FROM items;
SELECT id, embedding <-> '[0.1, 0.2, 0.3]' AS dist FROM items;
-- L2 distance with <-> operator
SELECT id, embedding <-> '[0.1, 0.2, 0.3]' AS dist
FROM items;
-- Inner product (negative dot product for minimization)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- Inner product (negative for minimization)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3]') AS dist FROM items;
-- Manhattan / L1 distance
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3]') AS dist FROM items;
```
### Nearest Neighbor Search
@@ -52,79 +46,84 @@ FROM items;
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3]') ASC
LIMIT 10;
-- Top-5 nearest neighbors by Euclidean distance
SELECT id FROM items
ORDER BY embedding <-> '[0.1, 0.2, 0.3]'
LIMIT 5;
```
### Vector Indexes
```sql
-- Create HNSW index for approximate nearest neighbor search
-- Create HNSW index
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
-- The index is automatically maintained on INSERT and UPDATE
-- Index is automatically maintained on INSERT and UPDATE
```
Supported index methods:
- `USING hnsw` — Hierarchical Navigable Small World (default: cosine metric)
- `USING ivfpq` — Inverted File with Product Quantization
### Dimension Validation
BaraDB validates vector dimensions at insert time:
## Hybrid RAG Search
```sql
-- This will fail: expected 768 dimensions but got 3
INSERT INTO items (id, embedding) VALUES (2, '[1.0, 2.0, 3.0]');
-- Combined vector + FTS search with Reciprocal Rank Fusion reranking
SELECT hybrid_search('AI query', embedding, content, 10) AS result;
-- Filtered hybrid search
SELECT hybrid_search_filtered('AI query', embedding, content, 10, 'category', 'news') AS result;
-- Comma-separated IDs only
SELECT hybrid_search_ids('AI query', embedding, content, 10) AS result;
```
## Native Nim API
## AI Pipeline
For embedded or high-performance use, use the native Nim API directly:
### Text Chunking
```nim
import barabadb/vector/engine
```sql
-- Split text into overlapping chunks (max 1024 chars, 128 overlap)
SELECT chunk('Long text content here...', 1024, 128) AS result;
var idx = newHNSWIndex(dimensions = 128)
idx.insert(1, @[1.0'f32, 0.0'f32, ...], {"category": "A"}.toTable)
# Search
let results = idx.search(queryVector, k = 10)
# With metadata filtering
let filtered = idx.searchWithFilter(queryVector, k = 10,
filter = proc(meta: Table[string, string]): bool =
return meta.getOrDefault("category") == "A")
-- Returns: [{"index":0, "text":"...", "size":124}, ...]
```
## Index Types
Strategies: `paragraph`, `sentence`, `fixed`, `recursive` (default).
### HNSW
### Embedding Generation
Hierarchical Navigable Small World graph for approximate nearest neighbor search.
```nim
var hnsw = newHNSWIndex(
dimensions = 128,
m = 16, # connections per layer
efConstruction = 200, # search width during construction
efSearch = 100 # search width during query
)
```sql
-- Call external embedding service for a query vector
SELECT embed_text('query text here') AS result;
```
### IVF-PQ
Configure the embedder via environment variables:
```bash
export BARADB_EMBED_ENDPOINT=http://localhost:11434/api/embeddings
export BARADB_EMBED_MODEL=nomic-embed-text
export BARADB_EMBED_API_KEY=sk-... # optional, for OpenAI
```
Inverted File Index with Product Quantization for compression.
### Auto-Embedding on INSERT
```nim
var ivfpq = newIVFPQIndex(
dimensions = 128,
numCentroids = 256,
subQuantizers = 8
)
When a VECTOR column is NULL on INSERT but a TEXT column has content, the embedding
is automatically generated (if an embedder is configured):
```sql
CREATE TABLE docs (id INTEGER PRIMARY KEY, content TEXT, embedding VECTOR(768));
CREATE INDEX docs_vec ON docs(embedding) USING hnsw;
-- embedding is automatically populated
INSERT INTO docs (id, content) VALUES (1, 'This text will be auto-embedded');
```
## Natural Language → SQL
```sql
-- Generate schema prompt for LLM context
SELECT schema_prompt('users') AS result;
-- Natural language to SQL (requires configured LLM)
SELECT nl_to_sql('Show all users over 25 years old', 'users') AS result;
```
LLM configuration:
```bash
export BARADB_LLM_ENDPOINT=http://localhost:11434/api/generate
export BARADB_LLM_MODEL=llama3
export BARADB_LLM_API_KEY=sk-... # optional
```
## Distance Metrics
@@ -136,15 +135,37 @@ var ivfpq = newIVFPQIndex(
| `dotproduct` | `inner_product(a, b)` | Negative dot product |
| `manhattan` | `l1_distance(a, b)` | L1 distance |
## Native Nim API
```nim
import barabadb/vector/engine
var idx = newHNSWIndex(dimensions = 128)
idx.insert(1, @[1.0'f32, 0.0'f32, ...], {"category": "A"}.toTable)
let results = idx.search(queryVector, k = 10)
let filtered = idx.searchWithFilter(queryVector, k = 10,
filter = proc(meta: Table[string, string]): bool = "category" in meta)
```
## Index Types
### HNSW (Default)
```nim
var hnsw = newHNSWIndex(dimensions = 128, m = 16, efConstruction = 200)
```
### IVF-PQ
```nim
var ivfpq = newIVFPQIndex(dimensions = 128, numCentroids = 256, subQuantizers = 8)
```
## Quantization
```nim
import barabadb/vector/quant
# Scalar quantization
let scalar = scalarQuantize(data, bits = 8)
# Product quantization
let pq = productQuantize(data, subVectors = 8, bits = 8)
```
@@ -152,6 +173,5 @@ let pq = productQuantize(data, subVectors = 8, bits = 8)
```nim
import barabadb/vector/simd
let dist = simdCosineDistance(vec1, vec2)
```
```