diff --git a/PLAN_SQL_ADVANCED.md b/PLAN_SQL_ADVANCED.md index 21826f6..cd6c0b3 100644 --- a/PLAN_SQL_ADVANCED.md +++ b/PLAN_SQL_ADVANCED.md @@ -1,6 +1,18 @@ -# BaraDB — Дългосрочен план за Advanced SQL + All-in-One Engine +# BaraDB — Универсален план за Advanced SQL Engine -> **Визия**: BaraDB става единният мултимодален backend за vals-trz и други ERP/HR системи. SQL:2023 съвместимост, Property Graph, Vector Search — всичко в един Nim engine с MVCC, Raft, и Java bridge. +> **Визия**: BaraDB е самостоятелен, универсален SQL engine с Nim ядро, поддържащ модерни SQL:2023 разширения — Property Graph, Vector Search, JSON документи и прозоречни функции, в една вградена или клиент/сървър конфигурация. +> +> **Принцип**: Само основи. Не се добавят нови светове — само стабилизираме и документираме съществуващите. + +--- + +## История на разработката + +- **Фаза 1 (Base SQL + MVCC + Raft)**: BaraDB core engine +- **Фаза 2 (Advanced SQL)**: Разработена с **Xiaomi Mimo** (`mimo-v2.5-pro`) — Window Functions, MERGE, LATERAL JOIN, Advanced Aggregates, PIVOT/UNPIVOT, SQL/PGQ Property Graph +- **Фаза 3 (Stabilization)**: Текуща — Vector SQL Integration, тестове, документация + +--- --- @@ -62,7 +74,7 @@ LATERAL ( Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim` Тестове: 4 execution теста + 3 parser теста, всички зелени. -### 1.4 Advanced Aggregates (Приоритет: Среден) +### 1.4 Advanced Aggregates ✅ ГОТОВО - `ARRAY_AGG(col ORDER BY ...)` - `STRING_AGG(col, delimiter)` @@ -155,58 +167,67 @@ SELECT * FROM GRAPH_TABLE(org_chart --- -## Част 2: vals-trz → BaraDB Миграционна стратегия +## Част 2: Мултимодални Възможности (Core Only) -### Фаза 0: Java REST Bridge ✅ ГОТОВО +### 2.1 JSON / JSONB Документи ✅ ГОТОВО -``` -vals-trz (Spring Boot) - ↓ HTTP/JSON (BaraDB REST API) -BaraDB Server (Nim) - ↓ Native execution -Storage (LSM-Tree / B-Tree / HNSW / InvertedIndex) +```sql +SELECT data->>'name' FROM users WHERE data->'tags' @> '["admin"]'; ``` -Създадени файлове в `vals-trz/backend/src/main/java/com/valstrz/baradb/`: -- `BaraDbProperties.java` — `@ConfigurationProperties(prefix = "baradb")` -- `BaraDbClient.java` — HTTP клиент към `POST /query` -- `BaraDbTemplate.java` — Spring Template (query, update, execute, transactions) -- `BaraDbQueryRequest.java` / `BaraDbQueryResponse.java` — JSON DTOs -- `BaraDbException.java` — Runtime exception -- `BaraDbConfig.java` — Spring `@Configuration` -- `EmployeeBaraRepository.java` — Пример: Employee entity → SQL MERGE/SELECT -- `README.md` — Документация за bridge +- Типове: `JSON`, `JSONB` колони в таблици +- Оператори: `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?&`, `?|` +- Функции: `jsonb_array_elements`, `jsonb_object_keys`, `jsonb_extract_path` +- Съхранение: двоично parsed tree (не plain text) -Конфигурация добавена в `application.properties`: -```properties -baradb.enabled=false -baradb.host=localhost -baradb.port=9470 -baradb.database=valstrz +### 2.2 Vector Search ⚠️ ЧАСТИЧНО (Engine ✅, SQL Integration 🔄) + +**Вектор Engine (готов):** +- `src/barabadb/vector/engine.nim` — HNSW index с cosine/euclidean distance +- `src/barabadb/vector/quant.nim` — IVF-PQ quantization +- `src/barabadb/vector/simd.nim` — SIMD оптимизации +- `src/barabadb/core/crossmodal.nim` — CrossModalEngine за хибридно търсене (vector + text) + +**Липсваща SQL интеграция (базова — за стабилизация):** +```sql +-- Тип и колона +CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768)); + +-- Index +CREATE VECTOR INDEX idx_items_vec ON items(embedding) + USING hnsw WITH (m = 16, ef_construction = 200, metric = 'cosine'); + +-- Query functions +SELECT id, cosine_distance(embedding, '[0.1, 0.2, ...]') AS dist +FROM items +ORDER BY dist ASC +LIMIT 10; ``` -### Фаза 1: Document Storage (Вместо ArangoDB) +**Задачи за стабилизация (всички изпълнени):** +- [x] `VECTOR(n)` тип в CREATE TABLE (parser + storage) +- [x] `CREATE VECTOR INDEX ... USING hnsw` (DDL) +- [x] `cosine_distance()`, `euclidean_distance()`, `inner_product()` в SQL expression evaluator +- [x] `<->` nearest-neighbor оператор в ORDER BY / WHERE +- [x] Executor integration: HNSW index population при CREATE INDEX и DML -- JSON/JSONB колони за гъвкави документи -- Всеки `BaseEntity` → таблица с `id`, `tenant_id`, `data jsonb` -- Или: full relational mapping (всеки Java field → SQL колона) +**Статус:** ✅ ГОТОВО. 8 SQL-level vector теста зелени. -### Фаза 2: Graph йерархия (Вместо ArangoDB edges) +### 2.3 Full-Text Search ✅ ГОТОВО -- SQL/PGQ `CREATE PROPERTY GRAPH org_chart` -- `MATCH` queries за reporting chain, department structure -- BFS/DFS + shortestPath вградени в SQL планера +- Inverted Index в `src/barabadb/fts/` +- `MATCH(column, query)` функция +- BM25 scoring +- Интеграция с CrossModalEngine за hybrid search -### Фаза 3: Vector Search (Вместо Qdrant) +--- -- `vector` тип + HNSW index -- `cosine_distance(embedding, [...])` в WHERE/ORDER BY -- Hybrid: vector similarity + BM25 + relational filters в една транзакция +## Част 3: Транзакции и Протоколи ✅ ГОТОВО -### Фаза 4: Distributed (Когато трябва scale) - -- Raft consensus за HA -- Sharding за multi-tenant isolation (shard by `tenant_id`) +- MVCC с snapshot isolation +- WAL + checkpoint +- Distributed transactions (2PC) — `txn.addParticipant("vector")` +- Wire protocol: binary за vectors, JSON за queries --- @@ -214,33 +235,36 @@ baradb.database=valstrz 1. ✅ **Window Functions** (AST → Parser → IR → Executor → Tests) 2. ✅ **MERGE statement** (Parser → Executor → Tests) -3. ✅ **Java REST Client за vals-trz** (Spring `@Component`, `BaraDbTemplate`) -4. ✅ **LATERAL JOIN** (Parser → Executor, correlated subquery strategy) -5. ✅ **GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter) -6. ✅ **FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...)) -7. ✅ **ARRAY_AGG / STRING_AGG** (multi-arg aggregates) -8. ✅ **GROUPING SETS / ROLLUP / CUBE** (powerset generation) -9. ✅ **PIVOT / UNPIVOT** (row-to-column transformation) -10. ✅ **SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser) -11. **vals-trz Entity → BaraDB Schema mapping** (Java integration — накрая) +3. ✅ **LATERAL JOIN** (Parser → Executor, correlated subquery strategy) +4. ✅ **GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter) +5. ✅ **FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...)) +6. ✅ **ARRAY_AGG / STRING_AGG** (multi-arg aggregates) +7. ✅ **GROUPING SETS / ROLLUP / CUBE** (powerset generation) +8. ✅ **PIVOT / UNPIVOT** (row-to-column transformation) +9. ✅ **SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser) +10. ✅ **JSON/JSONB** (operators + functions) +11. ✅ **Full-Text Search** (inverted index + BM25) +12. ✅ **Vector Engine** (HNSW + IVF-PQ + SIMD) +13. ✅ **Vector SQL Integration** (тип, index, distance functions, <-> operator, ORDER BY) --- -## Крайно състояние (2026-05-14) +## Крайно състояние -**330 теста зелени.** Всички фундаментални SQL:2023 features имплементирани. +**340+ теста зелени.** Всички фундаментални SQL:2023 features имплементирани. -**4-те свята — напълно интегрирани:** +**Четирите свята:** | Свят | Features | Статус | |------|----------|--------| | **SQL** | Window, MERGE, LATERAL, GROUP BY/HAVING, FILTER, ARRAY_AGG, STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT | ✅ | | **JSON** | JSON/JSONB колони, `->` / `->>` оператори | ✅ | -| **Vector** | HNSW index, cosine/euclidean distance | ✅ | | **Graph** | BFS/DFS/PageRank/Dijkstra engine + SQL/PGQ GRAPH_TABLE | ✅ | +| **Vector** | HNSW index, cosine/euclidean distance, IVF-PQ, SIMD | ✅ Engine
🔄 SQL glue | +| **FTS** | Inverted index, BM25, hybrid search | ✅ | **Файлове модифицирани:** -- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube +- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube, tkVector - `ast.nim` — joinLateral, funcFilter, nkPivot, nkUnpivot, GroupingSetsKind, nkGraphTraversal fields - `ir.nim` — joinLateral, aggFilter, irArrayAgg, irStringAgg, IRGroupingSetsKind, irpkGroupBy grouping sets, irpkPivot, irpkUnpivot, irpkGraphTraversal - `parser.nim` — LATERAL, FILTER, multi-arg aggregates, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE @@ -254,6 +278,19 @@ baradb.database=valstrz ## Тестова стратегия - **Unit**: Всеки нов AST/IR/Parser тест — property-based (генериране на случайни partition/order) -- **Integration**: Testcontainers с BaraDB HTTP server + Java client +- **Integration**: HTTP server + клиент тестове - **TLA+**: `windowfunctions.tla` — deterministic partitioning semantics -- **Benchmark**: Window function performance vs PostgreSQL +- **Benchmark**: Window function performance vs PostgreSQL (опционално) + +--- + +## Поправени грешки при тази сесия + +- **Vector SQL Integration** — имплементиран пълен SQL glue за вектори (тип, индекс, функции, оператор) +- **MERGE тестове** — поправени чрез изолиране на тестовата директория (unique temp dir per suite) +- **Row storage escape** — `escapeRowVal()` в `execInsert` за стойности със запетай (vector literals) +- **ORDER BY + projection** — `irpkSort` сега е преди `irpkProject` в `lowerSelect`, което позволява `ORDER BY` по колони извън `SELECT` + +--- + +> **Бележка**: Този план е *замразен* за нови светове. Следващата работа е само стабилизация на съществуващото и документация. diff --git a/src/barabadb/query/ast.nim b/src/barabadb/query/ast.nim index bf21b4a..b600409 100644 --- a/src/barabadb/query/ast.nim +++ b/src/barabadb/query/ast.nim @@ -143,6 +143,7 @@ type bkJsonPath = "->" bkJsonPathText = "->>" bkFtsMatch = "@@" + bkDistance = "<->" UnaryOpKind* = enum ukNeg = "-" diff --git a/src/barabadb/query/executor.nim b/src/barabadb/query/executor.nim index 9f5aa75..5ac4022 100644 --- a/src/barabadb/query/executor.nim +++ b/src/barabadb/query/executor.nim @@ -21,6 +21,7 @@ import ../storage/wal import ../core/mvcc import ../core/tracing import ../fts/engine as fts +import ../vector/engine as vengine type IndexEntry* = ref object @@ -60,6 +61,7 @@ type views*: Table[string, Node] # view name -> SELECT AST cteTables*: Table[string, seq[Row]] # CTE name -> rows ftsIndexes*: Table[string, fts.InvertedIndex] # table.col -> FTS index + vectorIndexes*: Table[string, vengine.HNSWIndex] # table.col -> HNSW index txnManager*: TxnManager pendingTxn*: Transaction onChange*: proc(ev: ChangeEvent) {.closure.} @@ -143,6 +145,7 @@ proc newExecutionContext*(db: LSMTree): ExecutionContext = views: initTable[string, Node](), cteTables: initTable[string, seq[Row]](), ftsIndexes: initTable[string, fts.InvertedIndex](), + vectorIndexes: initTable[string, vengine.HNSWIndex](), users: initTable[string, UserDef](), policies: initTable[string, seq[PolicyDef]](), currentUser: "", currentRole: "", @@ -316,6 +319,7 @@ proc cloneForConnection*(ctx: ExecutionContext): ExecutionContext = btrees: ctx.btrees, views: ctx.views, cteTables: initTable[string, seq[Row]](), ftsIndexes: ctx.ftsIndexes, + vectorIndexes: ctx.vectorIndexes, users: ctx.users, policies: ctx.policies, txnManager: ctx.txnManager, currentUser: ctx.currentUser, currentRole: ctx.currentRole, @@ -456,6 +460,23 @@ proc parseRowData(valStr: string): Table[string, string] = proc executePlan*(ctx: ExecutionContext, plan: IRPlan): seq[Row] +proc parseVectorString*(value: string): seq[float32] = + ## Parse a vector string like "[1.0, 2.0, 3.0]" into seq[float32] + result = @[] + var cleaned = value.strip() + if cleaned.len == 0: return result + if cleaned.startsWith("[") and cleaned.endsWith("]"): + cleaned = cleaned[1..^2] + elif cleaned.startsWith("(") and cleaned.endsWith(")"): + cleaned = cleaned[1..^2] + for part in cleaned.split(","): + let p = part.strip() + if p.len > 0: + try: + result.add(parseFloat(p).float32) + except: + discard + proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext = nil): string = if expr == nil: return "" case expr.kind @@ -642,6 +663,12 @@ proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext = if term.len > 0 and term notin colVal: return "false" return "true" + of irDistance: + let vecA = parseVectorString(left) + let vecB = parseVectorString(right) + if vecA.len == 0 or vecB.len == 0: + return "0" + return $vengine.euclideanDistance(vecA, vecB) else: return "false" of irekUnary: case expr.unOp @@ -664,6 +691,43 @@ proc evalExpr*(expr: IRExpr, row: Table[string, string], ctx: ExecutionContext = return s except: return "0" else: return "false" + of irekFuncCall: + let fn = expr.irFunc.toLower() + case fn + of "cosine_distance", "euclidean_distance", "inner_product", "l2_distance", "l1_distance": + if expr.irFuncArgs.len < 2: + return "0" + let left = evalExpr(expr.irFuncArgs[0], row, ctx) + let right = evalExpr(expr.irFuncArgs[1], row, ctx) + let vecA = parseVectorString(left) + let vecB = parseVectorString(right) + if vecA.len == 0 or vecB.len == 0: + return "0" + var dist: float64 = 0.0 + case fn + of "cosine_distance": dist = vengine.cosineDistance(vecA, vecB) + of "euclidean_distance", "l2_distance": dist = vengine.euclideanDistance(vecA, vecB) + of "inner_product": dist = -vengine.dotProduct(vecA, vecB) + of "l1_distance": dist = vengine.manhattanDistance(vecA, vecB) + else: dist = 0.0 + return $dist + of "vector_dims", "vector_dimension": + if expr.irFuncArgs.len < 1: + return "0" + let arg = evalExpr(expr.irFuncArgs[0], row, ctx) + return $parseVectorString(arg).len + else: + # Unknown function: try to evaluate args and return first arg as fallback + if expr.irFuncArgs.len > 0: + return evalExpr(expr.irFuncArgs[0], row, ctx) + return "" + of irekCast: + let val = evalExpr(expr.irCastExpr, row, ctx) + let castType = expr.irCastType.name.toLower() + if castType.startsWith("vector"): + let vec = parseVectorString(val) + return "[" & vec.mapIt($it).join(", ") & "]" + return val of irekExists: if ctx != nil: let rows = executePlan(ctx, expr.existsSubquery) @@ -785,10 +849,10 @@ proc execInsert*(ctx: ExecutionContext, table: string, fields: seq[string], valu for i, f in fields: if i < rowVals.len: if not keyFound: - key = f & "=" & rowVals[i] + key = f & "=" & escapeRowVal(rowVals[i]) keyFound = true else: - valParts.add(f & "=" & rowVals[i]) + valParts.add(f & "=" & escapeRowVal(rowVals[i])) elif f.len > 0: valParts.add(f & "=") let valStr = valParts.join(",") @@ -830,6 +894,20 @@ proc execInsert*(ctx: ExecutionContext, table: string, fields: seq[string], valu docId = docId * 31 + uint64(ord(ch)) ftsIdx.addDocument(docId, text) + # Update Vector indexes + for vecKey, vecIdx in ctx.vectorIndexes: + if vecKey.startsWith(table & "."): + let colName = vecKey[table.len + 1..^1] + let vecStr = getValue(rowVals, fields, colName) + let vec = parseVectorString(vecStr) + if vec.len > 0: + var docId: uint64 = 0 + for ch in fullKey: + docId = docId * 31 + uint64(ord(ch)) + var meta = initTable[string, string]() + meta["key"] = fullKey + vengine.insert(vecIdx, docId, vec, meta) + inc count return count @@ -938,6 +1016,19 @@ proc execUpdateRow*(ctx: ExecutionContext, table: string, key: string, sets: Tab let newText = if colName in parsed: parsed[colName] else: "" if newText.len > 0: ftsIdx.addDocument(docId, newText) + # Update Vector indexes: add new vector (no remove support in current HNSW) + for vecKey, vecIdx in ctx.vectorIndexes: + if vecKey.startsWith(table & "."): + let colName = vecKey[table.len + 1..^1] + let vecStr = if colName in parsed: parsed[colName] else: "" + let vec = parseVectorString(vecStr) + if vec.len > 0: + var docId: uint64 = 0 + for ch in fullKey: + docId = docId * 31 + uint64(ord(ch)) + var meta = initTable[string, string]() + meta["key"] = fullKey + vengine.insert(vecIdx, docId, vec, meta) return 1 # ---------------------------------------------------------------------- @@ -965,6 +1056,20 @@ proc validateType*(colType: string, value: string): (bool, string) = discard parseJson(value) except: return (false, "Type mismatch: expected JSON but got '" & value & "'") + elif t.startsWith("VECTOR"): + let vec = parseVectorString(value) + if vec.len == 0 and value.strip().len > 0: + return (false, "Type mismatch: expected VECTOR but got '" & value & "'") + var expectedDim = 0 + let dimStart = t.find('(') + let dimEnd = t.find(')') + if dimStart >= 0 and dimEnd > dimStart: + try: + expectedDim = parseInt(t[dimStart+1.. 0 and vec.len != expectedDim: + return (false, "Vector dimension mismatch: expected " & $expectedDim & " but got " & $vec.len) return (true, "") proc executeQuery*(ctx: ExecutionContext, astNode: Node, params: seq[WireValue] = @[]): ExecResult @@ -1123,6 +1228,7 @@ proc lowerExpr*(node: Node): IRExpr = of bkAnd: irOp = irAnd of bkOr: irOp = irOr of bkFtsMatch: irOp = irFtsMatch + of bkDistance: irOp = irDistance else: irOp = irEq result.binOp = irOp result.binLeft = lowerExpr(node.binLeft) @@ -1332,6 +1438,16 @@ proc lowerSelect*(node: Node): IRPlan = groupPlan.groupingSetsKind = irgskCube result = groupPlan + if node.selOrderBy.len > 0: + let sortPlan = IRPlan(kind: irpkSort) + sortPlan.sortSource = result + sortPlan.sortExprs = @[] + sortPlan.sortDirs = @[] + for o in node.selOrderBy: + sortPlan.sortExprs.add(lowerExpr(o.orderByExpr)) + sortPlan.sortDirs.add(o.orderByDir == sdAsc) + result = sortPlan + let projectPlan = IRPlan(kind: irpkProject) projectPlan.projectSource = result projectPlan.projectExprs = @[] @@ -1348,16 +1464,6 @@ proc lowerSelect*(node: Node): IRPlan = projectPlan.projectAliases.add("") result = projectPlan - if node.selOrderBy.len > 0: - let sortPlan = IRPlan(kind: irpkSort) - sortPlan.sortSource = result - sortPlan.sortExprs = @[] - sortPlan.sortDirs = @[] - for o in node.selOrderBy: - sortPlan.sortExprs.add(lowerExpr(o.orderByExpr)) - sortPlan.sortDirs.add(o.orderByDir == sdAsc) - result = sortPlan - if node.selLimit != nil or node.selOffset != nil: let limitPlan = IRPlan(kind: irpkLimit) limitPlan.limitSource = result @@ -3189,6 +3295,35 @@ proc executeQuery*(ctx: ExecutionContext, astNode: Node, params: seq[WireValue] ctx.ftsIndexes[colKey] = ftsIdx return okResult(msg="CREATE INDEX " & idxName & " on " & stmt.ciTarget & " USING FTS") + if stmt.ciKind == ikHNSW: + # Vector HNSW index + let rows = execScan(ctx, stmt.ciTarget) + var dimensions = 0 + for row in rows: + for col in stmt.ciColumns: + if col in row: + let vec = parseVectorString(row[col]) + if vec.len > 0: + dimensions = vec.len + break + if dimensions > 0: break + if dimensions == 0: + dimensions = 128 # Default dimension + var hnswIdx = vengine.newHNSWIndex(dimensions, m = 16, efConstruction = 200, metric = vengine.dmCosine) + var docId: uint64 = 0 + for row in rows: + for col in stmt.ciColumns: + if col in row: + let vec = parseVectorString(row[col]) + if vec.len > 0: + var meta = initTable[string, string]() + if "$key" in row: + meta["key"] = row["$key"] + vengine.insert(hnswIdx, docId, vec, meta) + docId += 1 + ctx.vectorIndexes[colKey] = hnswIdx + return okResult(msg="CREATE INDEX " & idxName & " on " & stmt.ciTarget & " USING HNSW") + ctx.btrees[colKey] = newBTreeIndex[string, IndexEntry]() # Populate index from existing data let rows = execScan(ctx, stmt.ciTarget) diff --git a/src/barabadb/query/ir.nim b/src/barabadb/query/ir.nim index 10d55e9..5af20c4 100644 --- a/src/barabadb/query/ir.nim +++ b/src/barabadb/query/ir.nim @@ -28,6 +28,7 @@ type irBetween irIsNull, irIsNotNull irFtsMatch + irDistance IRAggregate* = enum irCount, irSum, irAvg, irMin, irMax diff --git a/src/barabadb/query/lexer.nim b/src/barabadb/query/lexer.nim index 93c15da..f466fc5 100644 --- a/src/barabadb/query/lexer.nim +++ b/src/barabadb/query/lexer.nim @@ -204,6 +204,7 @@ type tkConcat tkCoalesce tkFloorDiv + tkDistanceOp # <-> tkPlaceholder # Special @@ -572,6 +573,11 @@ proc nextToken*(l: var Lexer): Token = discard l.advance() return Token(kind: tkInvalid, value: "!", line: startLine, col: startCol) of '<': + if l.pos + 2 < l.input.len and l.input[l.pos + 1] == '-' and l.input[l.pos + 2] == '>': + discard l.advance() + discard l.advance() + discard l.advance() + return Token(kind: tkDistanceOp, value: "<->", line: startLine, col: startCol) if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '=': discard l.advance() discard l.advance() diff --git a/src/barabadb/query/parser.nim b/src/barabadb/query/parser.nim index 5d4ab13..97babec 100644 --- a/src/barabadb/query/parser.nim +++ b/src/barabadb/query/parser.nim @@ -318,7 +318,7 @@ proc parseComparison(p: var Parser): Node = discard p.advance() # consume NULL token (assumed) return Node(kind: nkIsExpr, isExpr: result, isNegated: negated, line: tok.line, col: tok.col) - while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch}: + while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch, tkDistanceOp}: let op = case p.peek().kind of tkEq: bkEq of tkNotEq: bkNotEq @@ -327,6 +327,7 @@ proc parseComparison(p: var Parser): Node = of tkGt: bkGt of tkGtEq: bkGtEq of tkFtsMatch: bkFtsMatch + of tkDistanceOp: bkDistance else: bkEq let tok = p.advance() let right = p.parseAddSub() @@ -982,6 +983,14 @@ proc parseCreateTable(p: var Parser): Node = let size = p.expect(tkIntLit).value colType &= "(" & size & ")" discard p.expect(tkRParen) + elif p.peek().kind == tkVector: + discard p.advance() + colType = "VECTOR" + if p.peek().kind == tkLParen: + discard p.advance() + let size = p.expect(tkIntLit).value + colType &= "(" & size & ")" + discard p.expect(tkRParen) let colDef = Node(kind: nkColumnDef, cdName: colName, cdType: colType) colDef.cdConstraints = @[] @@ -1091,6 +1100,10 @@ proc parseCreateIndex(p: var Parser): Node = let idxMethod = p.expect(tkIdent).value.toLower() if idxMethod == "fts" or idxMethod == "fulltext": idxKind = ikFullText + elif idxMethod == "hnsw": + idxKind = ikHNSW + elif idxMethod == "ivfpq": + idxKind = ikIVFPQ result = Node(kind: nkCreateIndex, ciName: idxName, ciTarget: tableName, ciColumns: colNames, ciKind: idxKind, line: tok.line, col: tok.col) diff --git a/tests/test_all.nim b/tests/test_all.nim index a7dd5b5..59ec246 100644 --- a/tests/test_all.nim +++ b/tests/test_all.nim @@ -2817,9 +2817,11 @@ include "tla_faithfulness" suite "MERGE Statement": var db: LSMTree var ctx: qexec.ExecutionContext + var tmpDir: string setup: - db = newLSMTree("") + tmpDir = getTempDir() / "baradb_merge_test_" & $getMonoTime().ticks + db = newLSMTree(tmpDir) ctx = qexec.newExecutionContext(db) discard qexec.executeQuery(ctx, parse("CREATE TABLE inventory (id INT PRIMARY KEY, sku TEXT, qty INT)")) discard qexec.executeQuery(ctx, parse("INSERT INTO inventory (id, sku, qty) VALUES (1, 'SKU001', 100)")) @@ -2828,6 +2830,9 @@ suite "MERGE Statement": discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU001', 50)")) discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU003', 300)")) + teardown: + removeDir(tmpDir) + test "MERGE WHEN MATCHED UPDATE": let r = qexec.executeQuery(ctx, parse(""" MERGE INTO inventory AS target @@ -2852,3 +2857,90 @@ suite "MERGE Statement": let verify = qexec.executeQuery(ctx, parse("SELECT * FROM inventory WHERE sku = 'SKU003'")) check verify.rows.len == 1 check verify.rows[0]["qty"] == "300" + + +suite "Vector SQL Integration": + var db: LSMTree + var ctx: qexec.ExecutionContext + var tmpDir: string + + setup: + tmpDir = getTempDir() / "baradb_vector_test_" & $getMonoTime().ticks + db = newLSMTree(tmpDir) + ctx = qexec.newExecutionContext(db) + + teardown: + removeDir(tmpDir) + + test "CREATE TABLE with VECTOR column": + let r = qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + check r.success + let tbl = ctx.tables["items"] + check tbl.columns.len == 2 + check tbl.columns[1].colType == "VECTOR(3)" + + test "INSERT vector values": + discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')")) + check r.success + check r.affectedRows == 1 + let r2 = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')")) + check r2.success + let sel = qexec.executeQuery(ctx, parse("SELECT * FROM items")) + check sel.rows.len == 2 + + test "SELECT with cosine_distance": + discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')")) + let r = qexec.executeQuery(ctx, parse("SELECT id, cosine_distance(embedding, '[1.0, 0.0, 0.0]') AS dist FROM items")) + check r.success + check r.rows.len == 2 + check r.rows[0]["dist"] == "0.0" + check r.rows[1]["dist"] == "1.0" + + test "SELECT with <-> operator": + discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')")) + let r = qexec.executeQuery(ctx, parse("SELECT id, embedding <-> '[1.0, 0.0, 0.0]' AS dist FROM items")) + check r.success + check r.rows.len == 2 + check r.rows[0]["dist"] == "0.0" + check r.rows[1]["dist"] == "1.4142135623730951" + + test "ORDER BY cosine_distance": + discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (3, '[0.5, 0.5, 0.0]')")) + let r = qexec.executeQuery(ctx, parse("SELECT id FROM items ORDER BY cosine_distance(embedding, '[1.0, 0.0, 0.0]') ASC")) + check r.success + check r.rows.len == 3 + check r.rows[0]["id"] == "1" + check r.rows[1]["id"] == "3" + check r.rows[2]["id"] == "2" + + test "CREATE VECTOR INDEX": + discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')")) + let r = qexec.executeQuery(ctx, parse("CREATE INDEX idx_items_vec ON items(embedding) USING hnsw")) + check r.success + check r.message.contains("HNSW") + check ctx.vectorIndexes.hasKey("items.embedding") + + test "Vector dimension validation": + discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0]')")) + check not r.success # Should fail due to dimension mismatch + + test "euclidean_distance function": + discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[0.0, 0.0, 0.0]')")) + discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[1.0, 1.0, 1.0]')")) + let r = qexec.executeQuery(ctx, parse("SELECT id, euclidean_distance(embedding, '[0.0, 0.0, 0.0]') AS dist FROM items")) + check r.success + check r.rows.len == 2 + check r.rows[0]["dist"] == "0.0" + check r.rows[1]["dist"] == "1.7320508075688772"