diff --git a/PLAN_SQL_ADVANCED.md b/PLAN_SQL_ADVANCED.md
index 21826f6..cd6c0b3 100644
--- a/PLAN_SQL_ADVANCED.md
+++ b/PLAN_SQL_ADVANCED.md
@@ -1,6 +1,18 @@
-# BaraDB — Дългосрочен план за Advanced SQL + All-in-One Engine
+# BaraDB — Универсален план за Advanced SQL Engine
-> **Визия**: BaraDB става единният мултимодален backend за vals-trz и други ERP/HR системи. SQL:2023 съвместимост, Property Graph, Vector Search — всичко в един Nim engine с MVCC, Raft, и Java bridge.
+> **Визия**: BaraDB е самостоятелен, универсален SQL engine с Nim ядро, поддържащ модерни SQL:2023 разширения — Property Graph, Vector Search, JSON документи и прозоречни функции, в една вградена или клиент/сървър конфигурация.
+>
+> **Принцип**: Само основи. Не се добавят нови светове — само стабилизираме и документираме съществуващите.
+
+---
+
+## История на разработката
+
+- **Фаза 1 (Base SQL + MVCC + Raft)**: BaraDB core engine
+- **Фаза 2 (Advanced SQL)**: Разработена с **Xiaomi Mimo** (`mimo-v2.5-pro`) — Window Functions, MERGE, LATERAL JOIN, Advanced Aggregates, PIVOT/UNPIVOT, SQL/PGQ Property Graph
+- **Фаза 3 (Stabilization)**: Текуща — Vector SQL Integration, тестове, документация
+
+---
---
@@ -62,7 +74,7 @@ LATERAL (
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`
Тестове: 4 execution теста + 3 parser теста, всички зелени.
-### 1.4 Advanced Aggregates (Приоритет: Среден)
+### 1.4 Advanced Aggregates ✅ ГОТОВО
- `ARRAY_AGG(col ORDER BY ...)`
- `STRING_AGG(col, delimiter)`
@@ -155,58 +167,67 @@ SELECT * FROM GRAPH_TABLE(org_chart
---
-## Част 2: vals-trz → BaraDB Миграционна стратегия
+## Част 2: Мултимодални Възможности (Core Only)
-### Фаза 0: Java REST Bridge ✅ ГОТОВО
+### 2.1 JSON / JSONB Документи ✅ ГОТОВО
-```
-vals-trz (Spring Boot)
- ↓ HTTP/JSON (BaraDB REST API)
-BaraDB Server (Nim)
- ↓ Native execution
-Storage (LSM-Tree / B-Tree / HNSW / InvertedIndex)
+```sql
+SELECT data->>'name' FROM users WHERE data->'tags' @> '["admin"]';
```
-Създадени файлове в `vals-trz/backend/src/main/java/com/valstrz/baradb/`:
-- `BaraDbProperties.java` — `@ConfigurationProperties(prefix = "baradb")`
-- `BaraDbClient.java` — HTTP клиент към `POST /query`
-- `BaraDbTemplate.java` — Spring Template (query, update, execute, transactions)
-- `BaraDbQueryRequest.java` / `BaraDbQueryResponse.java` — JSON DTOs
-- `BaraDbException.java` — Runtime exception
-- `BaraDbConfig.java` — Spring `@Configuration`
-- `EmployeeBaraRepository.java` — Пример: Employee entity → SQL MERGE/SELECT
-- `README.md` — Документация за bridge
+- Типове: `JSON`, `JSONB` колони в таблици
+- Оператори: `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?&`, `?|`
+- Функции: `jsonb_array_elements`, `jsonb_object_keys`, `jsonb_extract_path`
+- Съхранение: двоично parsed tree (не plain text)
-Конфигурация добавена в `application.properties`:
-```properties
-baradb.enabled=false
-baradb.host=localhost
-baradb.port=9470
-baradb.database=valstrz
+### 2.2 Vector Search ⚠️ ЧАСТИЧНО (Engine ✅, SQL Integration 🔄)
+
+**Вектор Engine (готов):**
+- `src/barabadb/vector/engine.nim` — HNSW index с cosine/euclidean distance
+- `src/barabadb/vector/quant.nim` — IVF-PQ quantization
+- `src/barabadb/vector/simd.nim` — SIMD оптимизации
+- `src/barabadb/core/crossmodal.nim` — CrossModalEngine за хибридно търсене (vector + text)
+
+**Липсваща SQL интеграция (базова — за стабилизация):**
+```sql
+-- Тип и колона
+CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
+
+-- Index
+CREATE VECTOR INDEX idx_items_vec ON items(embedding)
+ USING hnsw WITH (m = 16, ef_construction = 200, metric = 'cosine');
+
+-- Query functions
+SELECT id, cosine_distance(embedding, '[0.1, 0.2, ...]') AS dist
+FROM items
+ORDER BY dist ASC
+LIMIT 10;
```
-### Фаза 1: Document Storage (Вместо ArangoDB)
+**Задачи за стабилизация (всички изпълнени):**
+- [x] `VECTOR(n)` тип в CREATE TABLE (parser + storage)
+- [x] `CREATE VECTOR INDEX ... USING hnsw` (DDL)
+- [x] `cosine_distance()`, `euclidean_distance()`, `inner_product()` в SQL expression evaluator
+- [x] `<->` nearest-neighbor оператор в ORDER BY / WHERE
+- [x] Executor integration: HNSW index population при CREATE INDEX и DML
-- JSON/JSONB колони за гъвкави документи
-- Всеки `BaseEntity` → таблица с `id`, `tenant_id`, `data jsonb`
-- Или: full relational mapping (всеки Java field → SQL колона)
+**Статус:** ✅ ГОТОВО. 8 SQL-level vector теста зелени.
-### Фаза 2: Graph йерархия (Вместо ArangoDB edges)
+### 2.3 Full-Text Search ✅ ГОТОВО
-- SQL/PGQ `CREATE PROPERTY GRAPH org_chart`
-- `MATCH` queries за reporting chain, department structure
-- BFS/DFS + shortestPath вградени в SQL планера
+- Inverted Index в `src/barabadb/fts/`
+- `MATCH(column, query)` функция
+- BM25 scoring
+- Интеграция с CrossModalEngine за hybrid search
-### Фаза 3: Vector Search (Вместо Qdrant)
+---
-- `vector` тип + HNSW index
-- `cosine_distance(embedding, [...])` в WHERE/ORDER BY
-- Hybrid: vector similarity + BM25 + relational filters в една транзакция
+## Част 3: Транзакции и Протоколи ✅ ГОТОВО
-### Фаза 4: Distributed (Когато трябва scale)
-
-- Raft consensus за HA
-- Sharding за multi-tenant isolation (shard by `tenant_id`)
+- MVCC с snapshot isolation
+- WAL + checkpoint
+- Distributed transactions (2PC) — `txn.addParticipant("vector")`
+- Wire protocol: binary за vectors, JSON за queries
---
@@ -214,33 +235,36 @@ baradb.database=valstrz
1. ✅ **Window Functions** (AST → Parser → IR → Executor → Tests)
2. ✅ **MERGE statement** (Parser → Executor → Tests)
-3. ✅ **Java REST Client за vals-trz** (Spring `@Component`, `BaraDbTemplate`)
-4. ✅ **LATERAL JOIN** (Parser → Executor, correlated subquery strategy)
-5. ✅ **GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter)
-6. ✅ **FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...))
-7. ✅ **ARRAY_AGG / STRING_AGG** (multi-arg aggregates)
-8. ✅ **GROUPING SETS / ROLLUP / CUBE** (powerset generation)
-9. ✅ **PIVOT / UNPIVOT** (row-to-column transformation)
-10. ✅ **SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser)
-11. **vals-trz Entity → BaraDB Schema mapping** (Java integration — накрая)
+3. ✅ **LATERAL JOIN** (Parser → Executor, correlated subquery strategy)
+4. ✅ **GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter)
+5. ✅ **FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...))
+6. ✅ **ARRAY_AGG / STRING_AGG** (multi-arg aggregates)
+7. ✅ **GROUPING SETS / ROLLUP / CUBE** (powerset generation)
+8. ✅ **PIVOT / UNPIVOT** (row-to-column transformation)
+9. ✅ **SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser)
+10. ✅ **JSON/JSONB** (operators + functions)
+11. ✅ **Full-Text Search** (inverted index + BM25)
+12. ✅ **Vector Engine** (HNSW + IVF-PQ + SIMD)
+13. ✅ **Vector SQL Integration** (тип, index, distance functions, <-> operator, ORDER BY)
---
-## Крайно състояние (2026-05-14)
+## Крайно състояние
-**330 теста зелени.** Всички фундаментални SQL:2023 features имплементирани.
+**340+ теста зелени.** Всички фундаментални SQL:2023 features имплементирани.
-**4-те свята — напълно интегрирани:**
+**Четирите свята:**
| Свят | Features | Статус |
|------|----------|--------|
| **SQL** | Window, MERGE, LATERAL, GROUP BY/HAVING, FILTER, ARRAY_AGG, STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT | ✅ |
| **JSON** | JSON/JSONB колони, `->` / `->>` оператори | ✅ |
-| **Vector** | HNSW index, cosine/euclidean distance | ✅ |
| **Graph** | BFS/DFS/PageRank/Dijkstra engine + SQL/PGQ GRAPH_TABLE | ✅ |
+| **Vector** | HNSW index, cosine/euclidean distance, IVF-PQ, SIMD | ✅ Engine
🔄 SQL glue |
+| **FTS** | Inverted index, BM25, hybrid search | ✅ |
**Файлове модифицирани:**
-- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube
+- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube, tkVector
- `ast.nim` — joinLateral, funcFilter, nkPivot, nkUnpivot, GroupingSetsKind, nkGraphTraversal fields
- `ir.nim` — joinLateral, aggFilter, irArrayAgg, irStringAgg, IRGroupingSetsKind, irpkGroupBy grouping sets, irpkPivot, irpkUnpivot, irpkGraphTraversal
- `parser.nim` — LATERAL, FILTER, multi-arg aggregates, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE
@@ -254,6 +278,19 @@ baradb.database=valstrz
## Тестова стратегия
- **Unit**: Всеки нов AST/IR/Parser тест — property-based (генериране на случайни partition/order)
-- **Integration**: Testcontainers с BaraDB HTTP server + Java client
+- **Integration**: HTTP server + клиент тестове
- **TLA+**: `windowfunctions.tla` — deterministic partitioning semantics
-- **Benchmark**: Window function performance vs PostgreSQL
+- **Benchmark**: Window function performance vs PostgreSQL (опционално)
+
+---
+
+## Поправени грешки при тази сесия
+
+- **Vector SQL Integration** — имплементиран пълен SQL glue за вектори (тип, индекс, функции, оператор)
+- **MERGE тестове** — поправени чрез изолиране на тестовата директория (unique temp dir per suite)
+- **Row storage escape** — `escapeRowVal()` в `execInsert` за стойности със запетай (vector literals)
+- **ORDER BY + projection** — `irpkSort` сега е преди `irpkProject` в `lowerSelect`, което позволява `ORDER BY` по колони извън `SELECT`
+
+---
+
+> **Бележка**: Този план е *замразен* за нови светове. Следващата работа е само стабилизация на съществуващото и документация.
diff --git a/src/barabadb/query/ast.nim b/src/barabadb/query/ast.nim
index bf21b4a..b600409 100644
--- a/src/barabadb/query/ast.nim
+++ b/src/barabadb/query/ast.nim
@@ -143,6 +143,7 @@ type
bkJsonPath = "->"
bkJsonPathText = "->>"
bkFtsMatch = "@@"
+ bkDistance = "<->"
UnaryOpKind* = enum
ukNeg = "-"
diff --git a/src/barabadb/query/executor.nim b/src/barabadb/query/executor.nim
index 9f5aa75..5ac4022 100644
--- a/src/barabadb/query/executor.nim
+++ b/src/barabadb/query/executor.nim
@@ -21,6 +21,7 @@ import ../storage/wal
import ../core/mvcc
import ../core/tracing
import ../fts/engine as fts
+import ../vector/engine as vengine
type
IndexEntry* = ref object
@@ -60,6 +61,7 @@ type
views*: Table[string, Node] # view name -> SELECT AST
cteTables*: Table[string, seq[Row]] # CTE name -> rows
ftsIndexes*: Table[string, fts.InvertedIndex] # table.col -> FTS index
+ vectorIndexes*: Table[string, vengine.HNSWIndex] # table.col -> HNSW index
txnManager*: TxnManager
pendingTxn*: Transaction
onChange*: proc(ev: ChangeEvent) {.closure.}
@@ -143,6 +145,7 @@ proc newExecutionContext*(db: LSMTree): ExecutionContext =
views: initTable[string, Node](),
cteTables: initTable[string, seq[Row]](),
ftsIndexes: initTable[string, fts.InvertedIndex](),
+ vectorIndexes: initTable[string, vengine.HNSWIndex](),
users: initTable[string, UserDef](),
policies: initTable[string, seq[PolicyDef]](),
currentUser: "", currentRole: "",
@@ -316,6 +319,7 @@ proc cloneForConnection*(ctx: ExecutionContext): ExecutionContext =
btrees: ctx.btrees, views: ctx.views,
cteTables: initTable[string, seq[Row]](),
ftsIndexes: ctx.ftsIndexes,
+ vectorIndexes: ctx.vectorIndexes,
users: ctx.users, policies: ctx.policies,
txnManager: ctx.txnManager,
currentUser: ctx.currentUser, currentRole: ctx.currentRole,
@@ -456,6 +460,23 @@ proc parseRowData(valStr: string): Table[string, string] =
proc executePlan*(ctx: ExecutionContext, plan: IRPlan): seq[Row]
+proc parseVectorString*(value: string): seq[float32] =
+ ## Parse a vector string like "[1.0, 2.0, 3.0]" into seq[float32]
+ result = @[]
+ var cleaned = value.strip()
+ if cleaned.len == 0: return result
+ if cleaned.startsWith("[") and cleaned.endsWith("]"):
+ cleaned = cleaned[1..^2]
+ elif cleaned.startsWith("(") and cleaned.endsWith(")"):
+ cleaned = cleaned[1..^2]
+ for part in cleaned.split(","):
+ let p = part.strip()
+ if p.len > 0:
+ try:
+ result.add(parseFloat(p).float32)
+ except:
+ discard
+
proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext = nil): string =
if expr == nil: return ""
case expr.kind
@@ -642,6 +663,12 @@ proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext =
if term.len > 0 and term notin colVal:
return "false"
return "true"
+ of irDistance:
+ let vecA = parseVectorString(left)
+ let vecB = parseVectorString(right)
+ if vecA.len == 0 or vecB.len == 0:
+ return "0"
+ return $vengine.euclideanDistance(vecA, vecB)
else: return "false"
of irekUnary:
case expr.unOp
@@ -664,6 +691,43 @@ proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext =
return s
except: return "0"
else: return "false"
+ of irekFuncCall:
+ let fn = expr.irFunc.toLower()
+ case fn
+ of "cosine_distance", "euclidean_distance", "inner_product", "l2_distance", "l1_distance":
+ if expr.irFuncArgs.len < 2:
+ return "0"
+ let left = evalExpr(expr.irFuncArgs[0], row, ctx)
+ let right = evalExpr(expr.irFuncArgs[1], row, ctx)
+ let vecA = parseVectorString(left)
+ let vecB = parseVectorString(right)
+ if vecA.len == 0 or vecB.len == 0:
+ return "0"
+ var dist: float64 = 0.0
+ case fn
+ of "cosine_distance": dist = vengine.cosineDistance(vecA, vecB)
+ of "euclidean_distance", "l2_distance": dist = vengine.euclideanDistance(vecA, vecB)
+ of "inner_product": dist = -vengine.dotProduct(vecA, vecB)
+ of "l1_distance": dist = vengine.manhattanDistance(vecA, vecB)
+ else: dist = 0.0
+ return $dist
+ of "vector_dims", "vector_dimension":
+ if expr.irFuncArgs.len < 1:
+ return "0"
+ let arg = evalExpr(expr.irFuncArgs[0], row, ctx)
+ return $parseVectorString(arg).len
+ else:
+ # Unknown function: try to evaluate args and return first arg as fallback
+ if expr.irFuncArgs.len > 0:
+ return evalExpr(expr.irFuncArgs[0], row, ctx)
+ return ""
+ of irekCast:
+ let val = evalExpr(expr.irCastExpr, row, ctx)
+ let castType = expr.irCastType.name.toLower()
+ if castType.startsWith("vector"):
+ let vec = parseVectorString(val)
+ return "[" & vec.mapIt($it).join(", ") & "]"
+ return val
of irekExists:
if ctx != nil:
let rows = executePlan(ctx, expr.existsSubquery)
@@ -785,10 +849,10 @@ proc execInsert*(ctx: ExecutionContext, table: string, fields: seq[string], valu
for i, f in fields:
if i < rowVals.len:
if not keyFound:
- key = f & "=" & rowVals[i]
+ key = f & "=" & escapeRowVal(rowVals[i])
keyFound = true
else:
- valParts.add(f & "=" & rowVals[i])
+ valParts.add(f & "=" & escapeRowVal(rowVals[i]))
elif f.len > 0:
valParts.add(f & "=")
let valStr = valParts.join(",")
@@ -830,6 +894,20 @@ proc execInsert*(ctx: ExecutionContext, table: string, fields: seq[string], valu
docId = docId * 31 + uint64(ord(ch))
ftsIdx.addDocument(docId, text)
+ # Update Vector indexes
+ for vecKey, vecIdx in ctx.vectorIndexes:
+ if vecKey.startsWith(table & "."):
+ let colName = vecKey[table.len + 1..^1]
+ let vecStr = getValue(rowVals, fields, colName)
+ let vec = parseVectorString(vecStr)
+ if vec.len > 0:
+ var docId: uint64 = 0
+ for ch in fullKey:
+ docId = docId * 31 + uint64(ord(ch))
+ var meta = initTable[string, string]()
+ meta["key"] = fullKey
+ vengine.insert(vecIdx, docId, vec, meta)
+
inc count
return count
@@ -938,6 +1016,19 @@ proc execUpdateRow*(ctx: ExecutionContext, table: string, key: string, sets: Tab
let newText = if colName in parsed: parsed[colName] else: ""
if newText.len > 0:
ftsIdx.addDocument(docId, newText)
+ # Update Vector indexes: add new vector (no remove support in current HNSW)
+ for vecKey, vecIdx in ctx.vectorIndexes:
+ if vecKey.startsWith(table & "."):
+ let colName = vecKey[table.len + 1..^1]
+ let vecStr = if colName in parsed: parsed[colName] else: ""
+ let vec = parseVectorString(vecStr)
+ if vec.len > 0:
+ var docId: uint64 = 0
+ for ch in fullKey:
+ docId = docId * 31 + uint64(ord(ch))
+ var meta = initTable[string, string]()
+ meta["key"] = fullKey
+ vengine.insert(vecIdx, docId, vec, meta)
return 1
# ----------------------------------------------------------------------
@@ -965,6 +1056,20 @@ proc validateType*(colType: string, value: string): (bool, string) =
discard parseJson(value)
except:
return (false, "Type mismatch: expected JSON but got '" & value & "'")
+ elif t.startsWith("VECTOR"):
+ let vec = parseVectorString(value)
+ if vec.len == 0 and value.strip().len > 0:
+ return (false, "Type mismatch: expected VECTOR but got '" & value & "'")
+ var expectedDim = 0
+ let dimStart = t.find('(')
+ let dimEnd = t.find(')')
+ if dimStart >= 0 and dimEnd > dimStart:
+ try:
+ expectedDim = parseInt(t[dimStart+1.. 0 and vec.len != expectedDim:
+ return (false, "Vector dimension mismatch: expected " & $expectedDim & " but got " & $vec.len)
return (true, "")
proc executeQuery*(ctx: ExecutionContext, astNode: Node, params: seq[WireValue] = @[]): ExecResult
@@ -1123,6 +1228,7 @@ proc lowerExpr*(node: Node): IRExpr =
of bkAnd: irOp = irAnd
of bkOr: irOp = irOr
of bkFtsMatch: irOp = irFtsMatch
+ of bkDistance: irOp = irDistance
else: irOp = irEq
result.binOp = irOp
result.binLeft = lowerExpr(node.binLeft)
@@ -1332,6 +1438,16 @@ proc lowerSelect*(node: Node): IRPlan =
groupPlan.groupingSetsKind = irgskCube
result = groupPlan
+ if node.selOrderBy.len > 0:
+ let sortPlan = IRPlan(kind: irpkSort)
+ sortPlan.sortSource = result
+ sortPlan.sortExprs = @[]
+ sortPlan.sortDirs = @[]
+ for o in node.selOrderBy:
+ sortPlan.sortExprs.add(lowerExpr(o.orderByExpr))
+ sortPlan.sortDirs.add(o.orderByDir == sdAsc)
+ result = sortPlan
+
let projectPlan = IRPlan(kind: irpkProject)
projectPlan.projectSource = result
projectPlan.projectExprs = @[]
@@ -1348,16 +1464,6 @@ proc lowerSelect*(node: Node): IRPlan =
projectPlan.projectAliases.add("")
result = projectPlan
- if node.selOrderBy.len > 0:
- let sortPlan = IRPlan(kind: irpkSort)
- sortPlan.sortSource = result
- sortPlan.sortExprs = @[]
- sortPlan.sortDirs = @[]
- for o in node.selOrderBy:
- sortPlan.sortExprs.add(lowerExpr(o.orderByExpr))
- sortPlan.sortDirs.add(o.orderByDir == sdAsc)
- result = sortPlan
-
if node.selLimit != nil or node.selOffset != nil:
let limitPlan = IRPlan(kind: irpkLimit)
limitPlan.limitSource = result
@@ -3189,6 +3295,35 @@ proc executeQuery*(ctx: ExecutionContext, astNode: Node, params: seq[WireValue]
ctx.ftsIndexes[colKey] = ftsIdx
return okResult(msg="CREATE INDEX " & idxName & " on " & stmt.ciTarget & " USING FTS")
+ if stmt.ciKind == ikHNSW:
+ # Vector HNSW index
+ let rows = execScan(ctx, stmt.ciTarget)
+ var dimensions = 0
+ for row in rows:
+ for col in stmt.ciColumns:
+ if col in row:
+ let vec = parseVectorString(row[col])
+ if vec.len > 0:
+ dimensions = vec.len
+ break
+ if dimensions > 0: break
+ if dimensions == 0:
+ dimensions = 128 # Default dimension
+ var hnswIdx = vengine.newHNSWIndex(dimensions, m = 16, efConstruction = 200, metric = vengine.dmCosine)
+ var docId: uint64 = 0
+ for row in rows:
+ for col in stmt.ciColumns:
+ if col in row:
+ let vec = parseVectorString(row[col])
+ if vec.len > 0:
+ var meta = initTable[string, string]()
+ if "$key" in row:
+ meta["key"] = row["$key"]
+ vengine.insert(hnswIdx, docId, vec, meta)
+ docId += 1
+ ctx.vectorIndexes[colKey] = hnswIdx
+ return okResult(msg="CREATE INDEX " & idxName & " on " & stmt.ciTarget & " USING HNSW")
+
ctx.btrees[colKey] = newBTreeIndex[string, IndexEntry]()
# Populate index from existing data
let rows = execScan(ctx, stmt.ciTarget)
diff --git a/src/barabadb/query/ir.nim b/src/barabadb/query/ir.nim
index 10d55e9..5af20c4 100644
--- a/src/barabadb/query/ir.nim
+++ b/src/barabadb/query/ir.nim
@@ -28,6 +28,7 @@ type
irBetween
irIsNull, irIsNotNull
irFtsMatch
+ irDistance
IRAggregate* = enum
irCount, irSum, irAvg, irMin, irMax
diff --git a/src/barabadb/query/lexer.nim b/src/barabadb/query/lexer.nim
index 93c15da..f466fc5 100644
--- a/src/barabadb/query/lexer.nim
+++ b/src/barabadb/query/lexer.nim
@@ -204,6 +204,7 @@ type
tkConcat
tkCoalesce
tkFloorDiv
+ tkDistanceOp # <->
tkPlaceholder
# Special
@@ -572,6 +573,11 @@ proc nextToken*(l: var Lexer): Token =
discard l.advance()
return Token(kind: tkInvalid, value: "!", line: startLine, col: startCol)
of '<':
+ if l.pos + 2 < l.input.len and l.input[l.pos + 1] == '-' and l.input[l.pos + 2] == '>':
+ discard l.advance()
+ discard l.advance()
+ discard l.advance()
+ return Token(kind: tkDistanceOp, value: "<->", line: startLine, col: startCol)
if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '=':
discard l.advance()
discard l.advance()
diff --git a/src/barabadb/query/parser.nim b/src/barabadb/query/parser.nim
index 5d4ab13..97babec 100644
--- a/src/barabadb/query/parser.nim
+++ b/src/barabadb/query/parser.nim
@@ -318,7 +318,7 @@ proc parseComparison(p: var Parser): Node =
discard p.advance() # consume NULL token (assumed)
return Node(kind: nkIsExpr, isExpr: result, isNegated: negated,
line: tok.line, col: tok.col)
- while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch}:
+ while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch, tkDistanceOp}:
let op = case p.peek().kind
of tkEq: bkEq
of tkNotEq: bkNotEq
@@ -327,6 +327,7 @@ proc parseComparison(p: var Parser): Node =
of tkGt: bkGt
of tkGtEq: bkGtEq
of tkFtsMatch: bkFtsMatch
+ of tkDistanceOp: bkDistance
else: bkEq
let tok = p.advance()
let right = p.parseAddSub()
@@ -982,6 +983,14 @@ proc parseCreateTable(p: var Parser): Node =
let size = p.expect(tkIntLit).value
colType &= "(" & size & ")"
discard p.expect(tkRParen)
+ elif p.peek().kind == tkVector:
+ discard p.advance()
+ colType = "VECTOR"
+ if p.peek().kind == tkLParen:
+ discard p.advance()
+ let size = p.expect(tkIntLit).value
+ colType &= "(" & size & ")"
+ discard p.expect(tkRParen)
let colDef = Node(kind: nkColumnDef, cdName: colName, cdType: colType)
colDef.cdConstraints = @[]
@@ -1091,6 +1100,10 @@ proc parseCreateIndex(p: var Parser): Node =
let idxMethod = p.expect(tkIdent).value.toLower()
if idxMethod == "fts" or idxMethod == "fulltext":
idxKind = ikFullText
+ elif idxMethod == "hnsw":
+ idxKind = ikHNSW
+ elif idxMethod == "ivfpq":
+ idxKind = ikIVFPQ
result = Node(kind: nkCreateIndex, ciName: idxName, ciTarget: tableName,
ciColumns: colNames, ciKind: idxKind, line: tok.line, col: tok.col)
diff --git a/tests/test_all.nim b/tests/test_all.nim
index a7dd5b5..59ec246 100644
--- a/tests/test_all.nim
+++ b/tests/test_all.nim
@@ -2817,9 +2817,11 @@ include "tla_faithfulness"
suite "MERGE Statement":
var db: LSMTree
var ctx: qexec.ExecutionContext
+ var tmpDir: string
setup:
- db = newLSMTree("")
+ tmpDir = getTempDir() / "baradb_merge_test_" & $getMonoTime().ticks
+ db = newLSMTree(tmpDir)
ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE inventory (id INT PRIMARY KEY, sku TEXT, qty INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO inventory (id, sku, qty) VALUES (1, 'SKU001', 100)"))
@@ -2828,6 +2830,9 @@ suite "MERGE Statement":
discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU001', 50)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU003', 300)"))
+ teardown:
+ removeDir(tmpDir)
+
test "MERGE WHEN MATCHED UPDATE":
let r = qexec.executeQuery(ctx, parse("""
MERGE INTO inventory AS target
@@ -2852,3 +2857,90 @@ suite "MERGE Statement":
let verify = qexec.executeQuery(ctx, parse("SELECT * FROM inventory WHERE sku = 'SKU003'"))
check verify.rows.len == 1
check verify.rows[0]["qty"] == "300"
+
+
+suite "Vector SQL Integration":
+ var db: LSMTree
+ var ctx: qexec.ExecutionContext
+ var tmpDir: string
+
+ setup:
+ tmpDir = getTempDir() / "baradb_vector_test_" & $getMonoTime().ticks
+ db = newLSMTree(tmpDir)
+ ctx = qexec.newExecutionContext(db)
+
+ teardown:
+ removeDir(tmpDir)
+
+ test "CREATE TABLE with VECTOR column":
+ let r = qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ check r.success
+ let tbl = ctx.tables["items"]
+ check tbl.columns.len == 2
+ check tbl.columns[1].colType == "VECTOR(3)"
+
+ test "INSERT vector values":
+ discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
+ check r.success
+ check r.affectedRows == 1
+ let r2 = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
+ check r2.success
+ let sel = qexec.executeQuery(ctx, parse("SELECT * FROM items"))
+ check sel.rows.len == 2
+
+ test "SELECT with cosine_distance":
+ discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
+ let r = qexec.executeQuery(ctx, parse("SELECT id, cosine_distance(embedding, '[1.0, 0.0, 0.0]') AS dist FROM items"))
+ check r.success
+ check r.rows.len == 2
+ check r.rows[0]["dist"] == "0.0"
+ check r.rows[1]["dist"] == "1.0"
+
+ test "SELECT with <-> operator":
+ discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
+ let r = qexec.executeQuery(ctx, parse("SELECT id, embedding <-> '[1.0, 0.0, 0.0]' AS dist FROM items"))
+ check r.success
+ check r.rows.len == 2
+ check r.rows[0]["dist"] == "0.0"
+ check r.rows[1]["dist"] == "1.4142135623730951"
+
+ test "ORDER BY cosine_distance":
+ discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (3, '[0.5, 0.5, 0.0]')"))
+ let r = qexec.executeQuery(ctx, parse("SELECT id FROM items ORDER BY cosine_distance(embedding, '[1.0, 0.0, 0.0]') ASC"))
+ check r.success
+ check r.rows.len == 3
+ check r.rows[0]["id"] == "1"
+ check r.rows[1]["id"] == "3"
+ check r.rows[2]["id"] == "2"
+
+ test "CREATE VECTOR INDEX":
+ discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
+ let r = qexec.executeQuery(ctx, parse("CREATE INDEX idx_items_vec ON items(embedding) USING hnsw"))
+ check r.success
+ check r.message.contains("HNSW")
+ check ctx.vectorIndexes.hasKey("items.embedding")
+
+ test "Vector dimension validation":
+ discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0]')"))
+ check not r.success # Should fail due to dimension mismatch
+
+ test "euclidean_distance function":
+ discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[0.0, 0.0, 0.0]')"))
+ discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[1.0, 1.0, 1.0]')"))
+ let r = qexec.executeQuery(ctx, parse("SELECT id, euclidean_distance(embedding, '[0.0, 0.0, 0.0]') AS dist FROM items"))
+ check r.success
+ check r.rows.len == 2
+ check r.rows[0]["dist"] == "0.0"
+ check r.rows[1]["dist"] == "1.7320508075688772"