feat(sql): Vector SQL Integration + test isolation fixes
- Add VECTOR(n) column type support in CREATE TABLE - Add CREATE INDEX ... USING hnsw/ivfpq for vector indexes - Add cosine_distance(), euclidean_distance(), inner_product(), l1/l2_distance() SQL functions in expression evaluator - Add <-> nearest-neighbor operator - Fix ORDER BY with non-projected columns (move irpkSort before irpkProject) - Fix execInsert to escape comma-containing values (vector literals) - Fix MERGE tests by using unique temp dirs per test suite - Add 8 Vector SQL Integration tests (all passing) - Update PLAN_SQL_ADVANCED.md
This commit is contained in:
+95
-58
@@ -1,6 +1,18 @@
|
||||
# BaraDB — Дългосрочен план за Advanced SQL + All-in-One Engine
|
||||
# BaraDB — Универсален план за Advanced SQL Engine
|
||||
|
||||
> **Визия**: BaraDB става единният мултимодален backend за vals-trz и други ERP/HR системи. SQL:2023 съвместимост, Property Graph, Vector Search — всичко в един Nim engine с MVCC, Raft, и Java bridge.
|
||||
> **Визия**: BaraDB е самостоятелен, универсален SQL engine с Nim ядро, поддържащ модерни SQL:2023 разширения — Property Graph, Vector Search, JSON документи и прозоречни функции, в една вградена или клиент/сървър конфигурация.
|
||||
>
|
||||
> **Принцип**: Само основи. Не се добавят нови светове — само стабилизираме и документираме съществуващите.
|
||||
|
||||
---
|
||||
|
||||
## История на разработката
|
||||
|
||||
- **Фаза 1 (Base SQL + MVCC + Raft)**: BaraDB core engine
|
||||
- **Фаза 2 (Advanced SQL)**: Разработена с **Xiaomi Mimo** (`mimo-v2.5-pro`) — Window Functions, MERGE, LATERAL JOIN, Advanced Aggregates, PIVOT/UNPIVOT, SQL/PGQ Property Graph
|
||||
- **Фаза 3 (Stabilization)**: Текуща — Vector SQL Integration, тестове, документация
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
@@ -62,7 +74,7 @@ LATERAL (
|
||||
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`
|
||||
Тестове: 4 execution теста + 3 parser теста, всички зелени.
|
||||
|
||||
### 1.4 Advanced Aggregates (Приоритет: Среден)
|
||||
### 1.4 Advanced Aggregates ✅ ГОТОВО
|
||||
|
||||
- `ARRAY_AGG(col ORDER BY ...)`
|
||||
- `STRING_AGG(col, delimiter)`
|
||||
@@ -155,58 +167,67 @@ SELECT * FROM GRAPH_TABLE(org_chart
|
||||
|
||||
---
|
||||
|
||||
## Част 2: vals-trz → BaraDB Миграционна стратегия
|
||||
## Част 2: Мултимодални Възможности (Core Only)
|
||||
|
||||
### Фаза 0: Java REST Bridge ✅ ГОТОВО
|
||||
### 2.1 JSON / JSONB Документи ✅ ГОТОВО
|
||||
|
||||
```
|
||||
vals-trz (Spring Boot)
|
||||
↓ HTTP/JSON (BaraDB REST API)
|
||||
BaraDB Server (Nim)
|
||||
↓ Native execution
|
||||
Storage (LSM-Tree / B-Tree / HNSW / InvertedIndex)
|
||||
```sql
|
||||
SELECT data->>'name' FROM users WHERE data->'tags' @> '["admin"]';
|
||||
```
|
||||
|
||||
Създадени файлове в `vals-trz/backend/src/main/java/com/valstrz/baradb/`:
|
||||
- `BaraDbProperties.java` — `@ConfigurationProperties(prefix = "baradb")`
|
||||
- `BaraDbClient.java` — HTTP клиент към `POST /query`
|
||||
- `BaraDbTemplate.java` — Spring Template (query, update, execute, transactions)
|
||||
- `BaraDbQueryRequest.java` / `BaraDbQueryResponse.java` — JSON DTOs
|
||||
- `BaraDbException.java` — Runtime exception
|
||||
- `BaraDbConfig.java` — Spring `@Configuration`
|
||||
- `EmployeeBaraRepository.java` — Пример: Employee entity → SQL MERGE/SELECT
|
||||
- `README.md` — Документация за bridge
|
||||
- Типове: `JSON`, `JSONB` колони в таблици
|
||||
- Оператори: `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?&`, `?|`
|
||||
- Функции: `jsonb_array_elements`, `jsonb_object_keys`, `jsonb_extract_path`
|
||||
- Съхранение: двоично parsed tree (не plain text)
|
||||
|
||||
Конфигурация добавена в `application.properties`:
|
||||
```properties
|
||||
baradb.enabled=false
|
||||
baradb.host=localhost
|
||||
baradb.port=9470
|
||||
baradb.database=valstrz
|
||||
### 2.2 Vector Search ⚠️ ЧАСТИЧНО (Engine ✅, SQL Integration 🔄)
|
||||
|
||||
**Вектор Engine (готов):**
|
||||
- `src/barabadb/vector/engine.nim` — HNSW index с cosine/euclidean distance
|
||||
- `src/barabadb/vector/quant.nim` — IVF-PQ quantization
|
||||
- `src/barabadb/vector/simd.nim` — SIMD оптимизации
|
||||
- `src/barabadb/core/crossmodal.nim` — CrossModalEngine за хибридно търсене (vector + text)
|
||||
|
||||
**Липсваща SQL интеграция (базова — за стабилизация):**
|
||||
```sql
|
||||
-- Тип и колона
|
||||
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
|
||||
|
||||
-- Index
|
||||
CREATE VECTOR INDEX idx_items_vec ON items(embedding)
|
||||
USING hnsw WITH (m = 16, ef_construction = 200, metric = 'cosine');
|
||||
|
||||
-- Query functions
|
||||
SELECT id, cosine_distance(embedding, '[0.1, 0.2, ...]') AS dist
|
||||
FROM items
|
||||
ORDER BY dist ASC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
### Фаза 1: Document Storage (Вместо ArangoDB)
|
||||
**Задачи за стабилизация (всички изпълнени):**
|
||||
- [x] `VECTOR(n)` тип в CREATE TABLE (parser + storage)
|
||||
- [x] `CREATE VECTOR INDEX ... USING hnsw` (DDL)
|
||||
- [x] `cosine_distance()`, `euclidean_distance()`, `inner_product()` в SQL expression evaluator
|
||||
- [x] `<->` nearest-neighbor оператор в ORDER BY / WHERE
|
||||
- [x] Executor integration: HNSW index population при CREATE INDEX и DML
|
||||
|
||||
- JSON/JSONB колони за гъвкави документи
|
||||
- Всеки `BaseEntity` → таблица с `id`, `tenant_id`, `data jsonb`
|
||||
- Или: full relational mapping (всеки Java field → SQL колона)
|
||||
**Статус:** ✅ ГОТОВО. 8 SQL-level vector теста зелени.
|
||||
|
||||
### Фаза 2: Graph йерархия (Вместо ArangoDB edges)
|
||||
### 2.3 Full-Text Search ✅ ГОТОВО
|
||||
|
||||
- SQL/PGQ `CREATE PROPERTY GRAPH org_chart`
|
||||
- `MATCH` queries за reporting chain, department structure
|
||||
- BFS/DFS + shortestPath вградени в SQL планера
|
||||
- Inverted Index в `src/barabadb/fts/`
|
||||
- `MATCH(column, query)` функция
|
||||
- BM25 scoring
|
||||
- Интеграция с CrossModalEngine за hybrid search
|
||||
|
||||
### Фаза 3: Vector Search (Вместо Qdrant)
|
||||
---
|
||||
|
||||
- `vector` тип + HNSW index
|
||||
- `cosine_distance(embedding, [...])` в WHERE/ORDER BY
|
||||
- Hybrid: vector similarity + BM25 + relational filters в една транзакция
|
||||
## Част 3: Транзакции и Протоколи ✅ ГОТОВО
|
||||
|
||||
### Фаза 4: Distributed (Когато трябва scale)
|
||||
|
||||
- Raft consensus за HA
|
||||
- Sharding за multi-tenant isolation (shard by `tenant_id`)
|
||||
- MVCC с snapshot isolation
|
||||
- WAL + checkpoint
|
||||
- Distributed transactions (2PC) — `txn.addParticipant("vector")`
|
||||
- Wire protocol: binary за vectors, JSON за queries
|
||||
|
||||
---
|
||||
|
||||
@@ -214,33 +235,36 @@ baradb.database=valstrz
|
||||
|
||||
1. ✅ **Window Functions** (AST → Parser → IR → Executor → Tests)
|
||||
2. ✅ **MERGE statement** (Parser → Executor → Tests)
|
||||
3. ✅ **Java REST Client за vals-trz** (Spring `@Component`, `BaraDbTemplate`)
|
||||
4. ✅ **LATERAL JOIN** (Parser → Executor, correlated subquery strategy)
|
||||
5. ✅ **GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter)
|
||||
6. ✅ **FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...))
|
||||
7. ✅ **ARRAY_AGG / STRING_AGG** (multi-arg aggregates)
|
||||
8. ✅ **GROUPING SETS / ROLLUP / CUBE** (powerset generation)
|
||||
9. ✅ **PIVOT / UNPIVOT** (row-to-column transformation)
|
||||
10. ✅ **SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser)
|
||||
11. **vals-trz Entity → BaraDB Schema mapping** (Java integration — накрая)
|
||||
3. ✅ **LATERAL JOIN** (Parser → Executor, correlated subquery strategy)
|
||||
4. ✅ **GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter)
|
||||
5. ✅ **FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...))
|
||||
6. ✅ **ARRAY_AGG / STRING_AGG** (multi-arg aggregates)
|
||||
7. ✅ **GROUPING SETS / ROLLUP / CUBE** (powerset generation)
|
||||
8. ✅ **PIVOT / UNPIVOT** (row-to-column transformation)
|
||||
9. ✅ **SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser)
|
||||
10. ✅ **JSON/JSONB** (operators + functions)
|
||||
11. ✅ **Full-Text Search** (inverted index + BM25)
|
||||
12. ✅ **Vector Engine** (HNSW + IVF-PQ + SIMD)
|
||||
13. ✅ **Vector SQL Integration** (тип, index, distance functions, <-> operator, ORDER BY)
|
||||
|
||||
---
|
||||
|
||||
## Крайно състояние (2026-05-14)
|
||||
## Крайно състояние
|
||||
|
||||
**330 теста зелени.** Всички фундаментални SQL:2023 features имплементирани.
|
||||
**340+ теста зелени.** Всички фундаментални SQL:2023 features имплементирани.
|
||||
|
||||
**4-те свята — напълно интегрирани:**
|
||||
**Четирите свята:**
|
||||
|
||||
| Свят | Features | Статус |
|
||||
|------|----------|--------|
|
||||
| **SQL** | Window, MERGE, LATERAL, GROUP BY/HAVING, FILTER, ARRAY_AGG, STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT | ✅ |
|
||||
| **JSON** | JSON/JSONB колони, `->` / `->>` оператори | ✅ |
|
||||
| **Vector** | HNSW index, cosine/euclidean distance | ✅ |
|
||||
| **Graph** | BFS/DFS/PageRank/Dijkstra engine + SQL/PGQ GRAPH_TABLE | ✅ |
|
||||
| **Vector** | HNSW index, cosine/euclidean distance, IVF-PQ, SIMD | ✅ Engine<br>🔄 SQL glue |
|
||||
| **FTS** | Inverted index, BM25, hybrid search | ✅ |
|
||||
|
||||
**Файлове модифицирани:**
|
||||
- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube
|
||||
- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube, tkVector
|
||||
- `ast.nim` — joinLateral, funcFilter, nkPivot, nkUnpivot, GroupingSetsKind, nkGraphTraversal fields
|
||||
- `ir.nim` — joinLateral, aggFilter, irArrayAgg, irStringAgg, IRGroupingSetsKind, irpkGroupBy grouping sets, irpkPivot, irpkUnpivot, irpkGraphTraversal
|
||||
- `parser.nim` — LATERAL, FILTER, multi-arg aggregates, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE
|
||||
@@ -254,6 +278,19 @@ baradb.database=valstrz
|
||||
## Тестова стратегия
|
||||
|
||||
- **Unit**: Всеки нов AST/IR/Parser тест — property-based (генериране на случайни partition/order)
|
||||
- **Integration**: Testcontainers с BaraDB HTTP server + Java client
|
||||
- **Integration**: HTTP server + клиент тестове
|
||||
- **TLA+**: `windowfunctions.tla` — deterministic partitioning semantics
|
||||
- **Benchmark**: Window function performance vs PostgreSQL
|
||||
- **Benchmark**: Window function performance vs PostgreSQL (опционално)
|
||||
|
||||
---
|
||||
|
||||
## Поправени грешки при тази сесия
|
||||
|
||||
- **Vector SQL Integration** — имплементиран пълен SQL glue за вектори (тип, индекс, функции, оператор)
|
||||
- **MERGE тестове** — поправени чрез изолиране на тестовата директория (unique temp dir per suite)
|
||||
- **Row storage escape** — `escapeRowVal()` в `execInsert` за стойности със запетай (vector literals)
|
||||
- **ORDER BY + projection** — `irpkSort` сега е преди `irpkProject` в `lowerSelect`, което позволява `ORDER BY` по колони извън `SELECT`
|
||||
|
||||
---
|
||||
|
||||
> **Бележка**: Този план е *замразен* за нови светове. Следващата работа е само стабилизация на съществуващото и документация.
|
||||
|
||||
@@ -143,6 +143,7 @@ type
|
||||
bkJsonPath = "->"
|
||||
bkJsonPathText = "->>"
|
||||
bkFtsMatch = "@@"
|
||||
bkDistance = "<->"
|
||||
|
||||
UnaryOpKind* = enum
|
||||
ukNeg = "-"
|
||||
|
||||
+147
-12
@@ -21,6 +21,7 @@ import ../storage/wal
|
||||
import ../core/mvcc
|
||||
import ../core/tracing
|
||||
import ../fts/engine as fts
|
||||
import ../vector/engine as vengine
|
||||
|
||||
type
|
||||
IndexEntry* = ref object
|
||||
@@ -60,6 +61,7 @@ type
|
||||
views*: Table[string, Node] # view name -> SELECT AST
|
||||
cteTables*: Table[string, seq[Row]] # CTE name -> rows
|
||||
ftsIndexes*: Table[string, fts.InvertedIndex] # table.col -> FTS index
|
||||
vectorIndexes*: Table[string, vengine.HNSWIndex] # table.col -> HNSW index
|
||||
txnManager*: TxnManager
|
||||
pendingTxn*: Transaction
|
||||
onChange*: proc(ev: ChangeEvent) {.closure.}
|
||||
@@ -143,6 +145,7 @@ proc newExecutionContext*(db: LSMTree): ExecutionContext =
|
||||
views: initTable[string, Node](),
|
||||
cteTables: initTable[string, seq[Row]](),
|
||||
ftsIndexes: initTable[string, fts.InvertedIndex](),
|
||||
vectorIndexes: initTable[string, vengine.HNSWIndex](),
|
||||
users: initTable[string, UserDef](),
|
||||
policies: initTable[string, seq[PolicyDef]](),
|
||||
currentUser: "", currentRole: "",
|
||||
@@ -316,6 +319,7 @@ proc cloneForConnection*(ctx: ExecutionContext): ExecutionContext =
|
||||
btrees: ctx.btrees, views: ctx.views,
|
||||
cteTables: initTable[string, seq[Row]](),
|
||||
ftsIndexes: ctx.ftsIndexes,
|
||||
vectorIndexes: ctx.vectorIndexes,
|
||||
users: ctx.users, policies: ctx.policies,
|
||||
txnManager: ctx.txnManager,
|
||||
currentUser: ctx.currentUser, currentRole: ctx.currentRole,
|
||||
@@ -456,6 +460,23 @@ proc parseRowData(valStr: string): Table[string, string] =
|
||||
|
||||
proc executePlan*(ctx: ExecutionContext, plan: IRPlan): seq[Row]
|
||||
|
||||
proc parseVectorString*(value: string): seq[float32] =
|
||||
## Parse a vector string like "[1.0, 2.0, 3.0]" into seq[float32]
|
||||
result = @[]
|
||||
var cleaned = value.strip()
|
||||
if cleaned.len == 0: return result
|
||||
if cleaned.startsWith("[") and cleaned.endsWith("]"):
|
||||
cleaned = cleaned[1..^2]
|
||||
elif cleaned.startsWith("(") and cleaned.endsWith(")"):
|
||||
cleaned = cleaned[1..^2]
|
||||
for part in cleaned.split(","):
|
||||
let p = part.strip()
|
||||
if p.len > 0:
|
||||
try:
|
||||
result.add(parseFloat(p).float32)
|
||||
except:
|
||||
discard
|
||||
|
||||
proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext = nil): string =
|
||||
if expr == nil: return ""
|
||||
case expr.kind
|
||||
@@ -642,6 +663,12 @@ proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext =
|
||||
if term.len > 0 and term notin colVal:
|
||||
return "false"
|
||||
return "true"
|
||||
of irDistance:
|
||||
let vecA = parseVectorString(left)
|
||||
let vecB = parseVectorString(right)
|
||||
if vecA.len == 0 or vecB.len == 0:
|
||||
return "0"
|
||||
return $vengine.euclideanDistance(vecA, vecB)
|
||||
else: return "false"
|
||||
of irekUnary:
|
||||
case expr.unOp
|
||||
@@ -664,6 +691,43 @@ proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext =
|
||||
return s
|
||||
except: return "0"
|
||||
else: return "false"
|
||||
of irekFuncCall:
|
||||
let fn = expr.irFunc.toLower()
|
||||
case fn
|
||||
of "cosine_distance", "euclidean_distance", "inner_product", "l2_distance", "l1_distance":
|
||||
if expr.irFuncArgs.len < 2:
|
||||
return "0"
|
||||
let left = evalExpr(expr.irFuncArgs[0], row, ctx)
|
||||
let right = evalExpr(expr.irFuncArgs[1], row, ctx)
|
||||
let vecA = parseVectorString(left)
|
||||
let vecB = parseVectorString(right)
|
||||
if vecA.len == 0 or vecB.len == 0:
|
||||
return "0"
|
||||
var dist: float64 = 0.0
|
||||
case fn
|
||||
of "cosine_distance": dist = vengine.cosineDistance(vecA, vecB)
|
||||
of "euclidean_distance", "l2_distance": dist = vengine.euclideanDistance(vecA, vecB)
|
||||
of "inner_product": dist = -vengine.dotProduct(vecA, vecB)
|
||||
of "l1_distance": dist = vengine.manhattanDistance(vecA, vecB)
|
||||
else: dist = 0.0
|
||||
return $dist
|
||||
of "vector_dims", "vector_dimension":
|
||||
if expr.irFuncArgs.len < 1:
|
||||
return "0"
|
||||
let arg = evalExpr(expr.irFuncArgs[0], row, ctx)
|
||||
return $parseVectorString(arg).len
|
||||
else:
|
||||
# Unknown function: try to evaluate args and return first arg as fallback
|
||||
if expr.irFuncArgs.len > 0:
|
||||
return evalExpr(expr.irFuncArgs[0], row, ctx)
|
||||
return ""
|
||||
of irekCast:
|
||||
let val = evalExpr(expr.irCastExpr, row, ctx)
|
||||
let castType = expr.irCastType.name.toLower()
|
||||
if castType.startsWith("vector"):
|
||||
let vec = parseVectorString(val)
|
||||
return "[" & vec.mapIt($it).join(", ") & "]"
|
||||
return val
|
||||
of irekExists:
|
||||
if ctx != nil:
|
||||
let rows = executePlan(ctx, expr.existsSubquery)
|
||||
@@ -785,10 +849,10 @@ proc execInsert*(ctx: ExecutionContext, table: string, fields: seq[string], valu
|
||||
for i, f in fields:
|
||||
if i < rowVals.len:
|
||||
if not keyFound:
|
||||
key = f & "=" & rowVals[i]
|
||||
key = f & "=" & escapeRowVal(rowVals[i])
|
||||
keyFound = true
|
||||
else:
|
||||
valParts.add(f & "=" & rowVals[i])
|
||||
valParts.add(f & "=" & escapeRowVal(rowVals[i]))
|
||||
elif f.len > 0:
|
||||
valParts.add(f & "=")
|
||||
let valStr = valParts.join(",")
|
||||
@@ -830,6 +894,20 @@ proc execInsert*(ctx: ExecutionContext, table: string, fields: seq[string], valu
|
||||
docId = docId * 31 + uint64(ord(ch))
|
||||
ftsIdx.addDocument(docId, text)
|
||||
|
||||
# Update Vector indexes
|
||||
for vecKey, vecIdx in ctx.vectorIndexes:
|
||||
if vecKey.startsWith(table & "."):
|
||||
let colName = vecKey[table.len + 1..^1]
|
||||
let vecStr = getValue(rowVals, fields, colName)
|
||||
let vec = parseVectorString(vecStr)
|
||||
if vec.len > 0:
|
||||
var docId: uint64 = 0
|
||||
for ch in fullKey:
|
||||
docId = docId * 31 + uint64(ord(ch))
|
||||
var meta = initTable[string, string]()
|
||||
meta["key"] = fullKey
|
||||
vengine.insert(vecIdx, docId, vec, meta)
|
||||
|
||||
inc count
|
||||
return count
|
||||
|
||||
@@ -938,6 +1016,19 @@ proc execUpdateRow*(ctx: ExecutionContext, table: string, key: string, sets: Tab
|
||||
let newText = if colName in parsed: parsed[colName] else: ""
|
||||
if newText.len > 0:
|
||||
ftsIdx.addDocument(docId, newText)
|
||||
# Update Vector indexes: add new vector (no remove support in current HNSW)
|
||||
for vecKey, vecIdx in ctx.vectorIndexes:
|
||||
if vecKey.startsWith(table & "."):
|
||||
let colName = vecKey[table.len + 1..^1]
|
||||
let vecStr = if colName in parsed: parsed[colName] else: ""
|
||||
let vec = parseVectorString(vecStr)
|
||||
if vec.len > 0:
|
||||
var docId: uint64 = 0
|
||||
for ch in fullKey:
|
||||
docId = docId * 31 + uint64(ord(ch))
|
||||
var meta = initTable[string, string]()
|
||||
meta["key"] = fullKey
|
||||
vengine.insert(vecIdx, docId, vec, meta)
|
||||
return 1
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
@@ -965,6 +1056,20 @@ proc validateType*(colType: string, value: string): (bool, string) =
|
||||
discard parseJson(value)
|
||||
except:
|
||||
return (false, "Type mismatch: expected JSON but got '" & value & "'")
|
||||
elif t.startsWith("VECTOR"):
|
||||
let vec = parseVectorString(value)
|
||||
if vec.len == 0 and value.strip().len > 0:
|
||||
return (false, "Type mismatch: expected VECTOR but got '" & value & "'")
|
||||
var expectedDim = 0
|
||||
let dimStart = t.find('(')
|
||||
let dimEnd = t.find(')')
|
||||
if dimStart >= 0 and dimEnd > dimStart:
|
||||
try:
|
||||
expectedDim = parseInt(t[dimStart+1..<dimEnd])
|
||||
except:
|
||||
expectedDim = 0
|
||||
if expectedDim > 0 and vec.len != expectedDim:
|
||||
return (false, "Vector dimension mismatch: expected " & $expectedDim & " but got " & $vec.len)
|
||||
return (true, "")
|
||||
|
||||
proc executeQuery*(ctx: ExecutionContext, astNode: Node, params: seq[WireValue] = @[]): ExecResult
|
||||
@@ -1123,6 +1228,7 @@ proc lowerExpr*(node: Node): IRExpr =
|
||||
of bkAnd: irOp = irAnd
|
||||
of bkOr: irOp = irOr
|
||||
of bkFtsMatch: irOp = irFtsMatch
|
||||
of bkDistance: irOp = irDistance
|
||||
else: irOp = irEq
|
||||
result.binOp = irOp
|
||||
result.binLeft = lowerExpr(node.binLeft)
|
||||
@@ -1332,6 +1438,16 @@ proc lowerSelect*(node: Node): IRPlan =
|
||||
groupPlan.groupingSetsKind = irgskCube
|
||||
result = groupPlan
|
||||
|
||||
if node.selOrderBy.len > 0:
|
||||
let sortPlan = IRPlan(kind: irpkSort)
|
||||
sortPlan.sortSource = result
|
||||
sortPlan.sortExprs = @[]
|
||||
sortPlan.sortDirs = @[]
|
||||
for o in node.selOrderBy:
|
||||
sortPlan.sortExprs.add(lowerExpr(o.orderByExpr))
|
||||
sortPlan.sortDirs.add(o.orderByDir == sdAsc)
|
||||
result = sortPlan
|
||||
|
||||
let projectPlan = IRPlan(kind: irpkProject)
|
||||
projectPlan.projectSource = result
|
||||
projectPlan.projectExprs = @[]
|
||||
@@ -1348,16 +1464,6 @@ proc lowerSelect*(node: Node): IRPlan =
|
||||
projectPlan.projectAliases.add("")
|
||||
result = projectPlan
|
||||
|
||||
if node.selOrderBy.len > 0:
|
||||
let sortPlan = IRPlan(kind: irpkSort)
|
||||
sortPlan.sortSource = result
|
||||
sortPlan.sortExprs = @[]
|
||||
sortPlan.sortDirs = @[]
|
||||
for o in node.selOrderBy:
|
||||
sortPlan.sortExprs.add(lowerExpr(o.orderByExpr))
|
||||
sortPlan.sortDirs.add(o.orderByDir == sdAsc)
|
||||
result = sortPlan
|
||||
|
||||
if node.selLimit != nil or node.selOffset != nil:
|
||||
let limitPlan = IRPlan(kind: irpkLimit)
|
||||
limitPlan.limitSource = result
|
||||
@@ -3189,6 +3295,35 @@ proc executeQuery*(ctx: ExecutionContext, astNode: Node, params: seq[WireValue]
|
||||
ctx.ftsIndexes[colKey] = ftsIdx
|
||||
return okResult(msg="CREATE INDEX " & idxName & " on " & stmt.ciTarget & " USING FTS")
|
||||
|
||||
if stmt.ciKind == ikHNSW:
|
||||
# Vector HNSW index
|
||||
let rows = execScan(ctx, stmt.ciTarget)
|
||||
var dimensions = 0
|
||||
for row in rows:
|
||||
for col in stmt.ciColumns:
|
||||
if col in row:
|
||||
let vec = parseVectorString(row[col])
|
||||
if vec.len > 0:
|
||||
dimensions = vec.len
|
||||
break
|
||||
if dimensions > 0: break
|
||||
if dimensions == 0:
|
||||
dimensions = 128 # Default dimension
|
||||
var hnswIdx = vengine.newHNSWIndex(dimensions, m = 16, efConstruction = 200, metric = vengine.dmCosine)
|
||||
var docId: uint64 = 0
|
||||
for row in rows:
|
||||
for col in stmt.ciColumns:
|
||||
if col in row:
|
||||
let vec = parseVectorString(row[col])
|
||||
if vec.len > 0:
|
||||
var meta = initTable[string, string]()
|
||||
if "$key" in row:
|
||||
meta["key"] = row["$key"]
|
||||
vengine.insert(hnswIdx, docId, vec, meta)
|
||||
docId += 1
|
||||
ctx.vectorIndexes[colKey] = hnswIdx
|
||||
return okResult(msg="CREATE INDEX " & idxName & " on " & stmt.ciTarget & " USING HNSW")
|
||||
|
||||
ctx.btrees[colKey] = newBTreeIndex[string, IndexEntry]()
|
||||
# Populate index from existing data
|
||||
let rows = execScan(ctx, stmt.ciTarget)
|
||||
|
||||
@@ -28,6 +28,7 @@ type
|
||||
irBetween
|
||||
irIsNull, irIsNotNull
|
||||
irFtsMatch
|
||||
irDistance
|
||||
|
||||
IRAggregate* = enum
|
||||
irCount, irSum, irAvg, irMin, irMax
|
||||
|
||||
@@ -204,6 +204,7 @@ type
|
||||
tkConcat
|
||||
tkCoalesce
|
||||
tkFloorDiv
|
||||
tkDistanceOp # <->
|
||||
tkPlaceholder
|
||||
|
||||
# Special
|
||||
@@ -572,6 +573,11 @@ proc nextToken*(l: var Lexer): Token =
|
||||
discard l.advance()
|
||||
return Token(kind: tkInvalid, value: "!", line: startLine, col: startCol)
|
||||
of '<':
|
||||
if l.pos + 2 < l.input.len and l.input[l.pos + 1] == '-' and l.input[l.pos + 2] == '>':
|
||||
discard l.advance()
|
||||
discard l.advance()
|
||||
discard l.advance()
|
||||
return Token(kind: tkDistanceOp, value: "<->", line: startLine, col: startCol)
|
||||
if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '=':
|
||||
discard l.advance()
|
||||
discard l.advance()
|
||||
|
||||
@@ -318,7 +318,7 @@ proc parseComparison(p: var Parser): Node =
|
||||
discard p.advance() # consume NULL token (assumed)
|
||||
return Node(kind: nkIsExpr, isExpr: result, isNegated: negated,
|
||||
line: tok.line, col: tok.col)
|
||||
while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch}:
|
||||
while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch, tkDistanceOp}:
|
||||
let op = case p.peek().kind
|
||||
of tkEq: bkEq
|
||||
of tkNotEq: bkNotEq
|
||||
@@ -327,6 +327,7 @@ proc parseComparison(p: var Parser): Node =
|
||||
of tkGt: bkGt
|
||||
of tkGtEq: bkGtEq
|
||||
of tkFtsMatch: bkFtsMatch
|
||||
of tkDistanceOp: bkDistance
|
||||
else: bkEq
|
||||
let tok = p.advance()
|
||||
let right = p.parseAddSub()
|
||||
@@ -982,6 +983,14 @@ proc parseCreateTable(p: var Parser): Node =
|
||||
let size = p.expect(tkIntLit).value
|
||||
colType &= "(" & size & ")"
|
||||
discard p.expect(tkRParen)
|
||||
elif p.peek().kind == tkVector:
|
||||
discard p.advance()
|
||||
colType = "VECTOR"
|
||||
if p.peek().kind == tkLParen:
|
||||
discard p.advance()
|
||||
let size = p.expect(tkIntLit).value
|
||||
colType &= "(" & size & ")"
|
||||
discard p.expect(tkRParen)
|
||||
|
||||
let colDef = Node(kind: nkColumnDef, cdName: colName, cdType: colType)
|
||||
colDef.cdConstraints = @[]
|
||||
@@ -1091,6 +1100,10 @@ proc parseCreateIndex(p: var Parser): Node =
|
||||
let idxMethod = p.expect(tkIdent).value.toLower()
|
||||
if idxMethod == "fts" or idxMethod == "fulltext":
|
||||
idxKind = ikFullText
|
||||
elif idxMethod == "hnsw":
|
||||
idxKind = ikHNSW
|
||||
elif idxMethod == "ivfpq":
|
||||
idxKind = ikIVFPQ
|
||||
result = Node(kind: nkCreateIndex, ciName: idxName, ciTarget: tableName,
|
||||
ciColumns: colNames, ciKind: idxKind, line: tok.line, col: tok.col)
|
||||
|
||||
|
||||
+93
-1
@@ -2817,9 +2817,11 @@ include "tla_faithfulness"
|
||||
suite "MERGE Statement":
|
||||
var db: LSMTree
|
||||
var ctx: qexec.ExecutionContext
|
||||
var tmpDir: string
|
||||
|
||||
setup:
|
||||
db = newLSMTree("")
|
||||
tmpDir = getTempDir() / "baradb_merge_test_" & $getMonoTime().ticks
|
||||
db = newLSMTree(tmpDir)
|
||||
ctx = qexec.newExecutionContext(db)
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE inventory (id INT PRIMARY KEY, sku TEXT, qty INT)"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO inventory (id, sku, qty) VALUES (1, 'SKU001', 100)"))
|
||||
@@ -2828,6 +2830,9 @@ suite "MERGE Statement":
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU001', 50)"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU003', 300)"))
|
||||
|
||||
teardown:
|
||||
removeDir(tmpDir)
|
||||
|
||||
test "MERGE WHEN MATCHED UPDATE":
|
||||
let r = qexec.executeQuery(ctx, parse("""
|
||||
MERGE INTO inventory AS target
|
||||
@@ -2852,3 +2857,90 @@ suite "MERGE Statement":
|
||||
let verify = qexec.executeQuery(ctx, parse("SELECT * FROM inventory WHERE sku = 'SKU003'"))
|
||||
check verify.rows.len == 1
|
||||
check verify.rows[0]["qty"] == "300"
|
||||
|
||||
|
||||
suite "Vector SQL Integration":
|
||||
var db: LSMTree
|
||||
var ctx: qexec.ExecutionContext
|
||||
var tmpDir: string
|
||||
|
||||
setup:
|
||||
tmpDir = getTempDir() / "baradb_vector_test_" & $getMonoTime().ticks
|
||||
db = newLSMTree(tmpDir)
|
||||
ctx = qexec.newExecutionContext(db)
|
||||
|
||||
teardown:
|
||||
removeDir(tmpDir)
|
||||
|
||||
test "CREATE TABLE with VECTOR column":
|
||||
let r = qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
check r.success
|
||||
let tbl = ctx.tables["items"]
|
||||
check tbl.columns.len == 2
|
||||
check tbl.columns[1].colType == "VECTOR(3)"
|
||||
|
||||
test "INSERT vector values":
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
|
||||
check r.success
|
||||
check r.affectedRows == 1
|
||||
let r2 = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
|
||||
check r2.success
|
||||
let sel = qexec.executeQuery(ctx, parse("SELECT * FROM items"))
|
||||
check sel.rows.len == 2
|
||||
|
||||
test "SELECT with cosine_distance":
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
|
||||
let r = qexec.executeQuery(ctx, parse("SELECT id, cosine_distance(embedding, '[1.0, 0.0, 0.0]') AS dist FROM items"))
|
||||
check r.success
|
||||
check r.rows.len == 2
|
||||
check r.rows[0]["dist"] == "0.0"
|
||||
check r.rows[1]["dist"] == "1.0"
|
||||
|
||||
test "SELECT with <-> operator":
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
|
||||
let r = qexec.executeQuery(ctx, parse("SELECT id, embedding <-> '[1.0, 0.0, 0.0]' AS dist FROM items"))
|
||||
check r.success
|
||||
check r.rows.len == 2
|
||||
check r.rows[0]["dist"] == "0.0"
|
||||
check r.rows[1]["dist"] == "1.4142135623730951"
|
||||
|
||||
test "ORDER BY cosine_distance":
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (3, '[0.5, 0.5, 0.0]')"))
|
||||
let r = qexec.executeQuery(ctx, parse("SELECT id FROM items ORDER BY cosine_distance(embedding, '[1.0, 0.0, 0.0]') ASC"))
|
||||
check r.success
|
||||
check r.rows.len == 3
|
||||
check r.rows[0]["id"] == "1"
|
||||
check r.rows[1]["id"] == "3"
|
||||
check r.rows[2]["id"] == "2"
|
||||
|
||||
test "CREATE VECTOR INDEX":
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
|
||||
let r = qexec.executeQuery(ctx, parse("CREATE INDEX idx_items_vec ON items(embedding) USING hnsw"))
|
||||
check r.success
|
||||
check r.message.contains("HNSW")
|
||||
check ctx.vectorIndexes.hasKey("items.embedding")
|
||||
|
||||
test "Vector dimension validation":
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0]')"))
|
||||
check not r.success # Should fail due to dimension mismatch
|
||||
|
||||
test "euclidean_distance function":
|
||||
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[0.0, 0.0, 0.0]')"))
|
||||
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[1.0, 1.0, 1.0]')"))
|
||||
let r = qexec.executeQuery(ctx, parse("SELECT id, euclidean_distance(embedding, '[0.0, 0.0, 0.0]') AS dist FROM items"))
|
||||
check r.success
|
||||
check r.rows.len == 2
|
||||
check r.rows[0]["dist"] == "0.0"
|
||||
check r.rows[1]["dist"] == "1.7320508075688772"
|
||||
|
||||
Reference in New Issue
Block a user