Compare commits

...

11 Commits

Author SHA1 Message Date
dimgigov f7d4961125 feat: Multi-tenant ERP support via session variables + RLS
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- SET var = value / current_setting('var') for session-scoped variables
- current_user / current_role SQL keywords with auth bridge
- server.nim + httpserver.nim populate ExecutionContext.currentUser/currentRole
- RLS policies can reference current_setting('app.tenant_id') for tenant isolation
- Fixed evalExpr to propagate ctx recursively (fixes current_user in sub-expressions)
- Fixed GROUPING SETS execution (lowerSelect checks selGroupingSetsKind)
- Fixed FTS CREATE INDEX docId mismatch (hash of tableName.$key)
- Fixed all test suites to use isolated temp directories
- Added 5 multi-tenant tests (355 total, all green)
- Updated docs: PLAN_SQL_ADVANCED.md, baraql.md, changelog.md
2026-05-14 16:28:41 +03:00
dimgigov b0978812cb docs(en): Update English docs for Vector SQL Integration
- docs/en/vector.md — add SQL usage section (CREATE TABLE VECTOR,
  distance functions, <-> operator, CREATE INDEX USING hnsw)
- docs/en/baraql.md — update vector search section with real SQL syntax,
  add VECTOR(n) to data types, update keyword table
- docs/en/changelog.md — add Vector SQL Integration and bugfixes to [Unreleased]
- docs/ARCHITECTURE.md — add SQL Integration bullet to Vector Engine
- README.md — update vector engine section with SQL examples,
  add Vector SQL to roadmap, bump test count to 340+
2026-05-14 14:20:57 +03:00
dimgigov d076cfde3b feat(sql): Vector SQL Integration + test isolation fixes
- Add VECTOR(n) column type support in CREATE TABLE
- Add CREATE INDEX ... USING hnsw/ivfpq for vector indexes
- Add cosine_distance(), euclidean_distance(), inner_product(), l1/l2_distance()
  SQL functions in expression evaluator
- Add <-> nearest-neighbor operator
- Fix ORDER BY with non-projected columns (move irpkSort before irpkProject)
- Fix execInsert to escape comma-containing values (vector literals)
- Fix MERGE tests by using unique temp dirs per test suite
- Add 8 Vector SQL Integration tests (all passing)
- Update PLAN_SQL_ADVANCED.md
2026-05-14 14:14:13 +03:00
dimgigov 96dfaaecb1 feat(sql): Advanced SQL — LATERAL, GROUP BY/HAVING, FILTER, aggregates, PIVOT, SQL/PGQ
- LATERAL JOIN: correlated subquery strategy (scan + merge + filter/sort/limit)
- GROUP BY: SUM/AVG/MIN/MAX evaluation inside groups, HAVING filter
- FILTER (WHERE ...): conditional aggregates for COUNT/SUM/AVG
- ARRAY_AGG / STRING_AGG: multi-argument aggregate functions
- GROUPING SETS / ROLLUP / CUBE: powerset generation for multi-level aggregation
- PIVOT / UNPIVOT: row-to-column and column-to-row transformation
- SQL/PGQ Property Graph: GRAPH_TABLE MATCH parser + executor skeleton
- 330 tests passing, all 4 modalities (SQL/JSON/Vector/Graph) integrated
2026-05-14 13:14:10 +03:00
dimgigov e2a526df6f feat(sql): Window Functions + MERGE statement + REST bridge plan
- Add Window Functions: ROW_NUMBER, RANK, DENSE_RANK, LEAD, LAG,
  FIRST_VALUE, LAST_VALUE, NTILE with PARTITION BY, ORDER BY,
  ROWS/RANGE frame specifications
- Add MERGE/UPSERT statement with WHEN MATCHED UPDATE and
  WHEN NOT MATCHED INSERT
- Add SQL/PGQ Property Graph long-term plan in PLAN_SQL_ADVANCED.md
- Add 7 new tests for Window Functions and MERGE
- Update baraql.md documentation
2026-05-14 11:02:09 +03:00
dimgigov 18f3c16b2a Fix/nodebara compatibility (#2) 2026-05-14 02:33:46 +03:00
dimgigov 71dcffecce fix(gossip): use async UDP socket to avoid blocking the event loop
The gossip listener used synchronous newSocket + recvFrom,
which blocks the async event loop and prevents other async
operations from running while waiting for UDP packets.

Switch to newAsyncSocket + async recvFrom so the loop can
continue processing other tasks between gossip rounds.
2026-05-14 02:22:07 +03:00
dimgigov 398769ff97 feat(client): add TCP request queue for safe concurrency
The previous implementation allowed multiple concurrent async calls
(query, execute, ping) to interleave writes on the same socket,
which corrupted the binary protocol framing when NodeBB fired
parallel database operations.

Add an internal _requestQueue and _requestLock so that all TCP
requests are serialized: each async operation enqueues a task,
and tasks are drained one at a time via setImmediate().
2026-05-14 02:22:07 +03:00
dimgigov 8fb5dde858 fix(protocol): serialize float32/float64/fkVector in big-endian
The JavaScript client reads floats via readFloatBE/readDoubleBE,
which expect IEEE-754 values in big-endian byte order.
The Nim server was writing them with copyMem in native byte order,
so on little-endian machines (x86_64) the JS side deserialized
FLOAT64 values as garbage (e.g. 1.0 became 3.03865e-319).

Fix serialization by casting to int32/int64 and using bigEndian32/
bigEndian64, mirroring the existing big-endian handling for ints.
Apply the same fix to deserialization and to fkVector elements.
2026-05-14 02:22:07 +03:00
dimgigov a0d2ca7776 Merge release/v1.1.0 into main 2026-05-13 15:07:44 +03:00
dimgigov 4e7e568525 release: v1.1.0 — bug fixes, Docker, clients, SCRAM auth
- Bump version to 1.1.0 across all components
- Remove debug CI artifacts (debug.yml, test_all.nim.bak, test_lock)
- Update CHANGELOG with v1.1.0 release notes
- Bug fixes: irNeg, distributed txn, sharding, Raft majority, MVCC,
  LSM-Tree, auth (JWT + SCRAM-SHA-256), wire protocol, SQL injection,
  ReDoS, graph, vector, FTS, build warnings
- Client SDKs: JS/TS, Python, Nim, Rust with tests and examples
- Docker: production + source + test compose configs
- TLA+: crossmodal, backup, recovery specs with symmetry reduction
2026-05-13 15:05:59 +03:00
36 changed files with 3241 additions and 2884 deletions
-59
View File
@@ -1,59 +0,0 @@
name: Debug Tests
on:
push:
branches: [main]
jobs:
debug:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Nim
uses: jiro4989/setup-nim-action@v1
with:
nim-version: '2.2.10'
- name: Install system dependencies
run: sudo apt-get update -qq && sudo apt-get install -y -qq libssl-dev libpcre3-dev openssl ca-certificates
- name: Install Nim dependencies
run: nimble install --depsOnly -y
- name: Check nim binary
run: |
which nim
nim --version
echo "NIM_OK=1"
- name: Compile and run lock test
run: nim c --threads:on --path:src -r tests/test_lock.nim
- name: Run join_tests
run: timeout 30 nim c -d:ssl --threads:on --path:src -r tests/join_tests.nim
- name: Run tla_faithfulness
run: nim c -d:ssl --threads:on --path:src -r tests/tla_faithfulness.nim
- name: Compile test_all
run: nim c -d:ssl --threads:on --path:src tests/test_all.nim
- name: Run test_all binary
run: ./tests/test_all > test_all.log 2>&1
- name: Upload test_all log to repo
if: failure()
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
URL=$(curl -s -F'file=@test_all.log' https://0x0.st)
echo "::notice::test_all.log uploaded to $URL"
sudo apt-get install -y -qq git
git config user.name "CI Bot"
git config user.email "ci@baradb"
git checkout -b ci-logs-${{ github.run_id }} || true
cp test_all.log ci_log.txt
git add ci_log.txt
git commit -m "CI log for run ${{ github.run_id }}" || true
git push https://$GITHUB_TOKEN@github.com/${{ github.repository }}.git ci-logs-${{ github.run_id }} || echo "PUSH_FAILED=1"
+2
View File
@@ -38,3 +38,5 @@ clients/rust/target/
clients/nim/src/baradb/client clients/nim/src/baradb/client
src/barabadb/storage/compaction src/barabadb/storage/compaction
docker-compose.test.yml docker-compose.test.yml
src/barabadb/query/executor
tests/join_tests
+1 -1
View File
@@ -16,7 +16,7 @@ FROM debian:bookworm-slim
LABEL maintainer="BaraDB Team" LABEL maintainer="BaraDB Team"
LABEL description="BaraDB — Multimodal Database Engine" LABEL description="BaraDB — Multimodal Database Engine"
LABEL version="1.0.0" LABEL version="1.1.0"
# Инсталираме runtime зависимости # Инсталираме runtime зависимости
# libpcre3 — нужна за Nim regex (зарежда се динамично) # libpcre3 — нужна за Nim regex (зарежда се динамично)
+1 -1
View File
@@ -40,7 +40,7 @@ FROM alpine:3.19
LABEL maintainer="BaraDB Team" LABEL maintainer="BaraDB Team"
LABEL description="BaraDB — Multimodal Database Engine (source build)" LABEL description="BaraDB — Multimodal Database Engine (source build)"
LABEL version="1.0.0" LABEL version="1.1.0"
# Инсталираме runtime зависимости # Инсталираме runtime зависимости
RUN apk add --no-cache ca-certificates su-exec wget pcre RUN apk add --no-cache ca-certificates su-exec wget pcre
+351
View File
@@ -0,0 +1,351 @@
# BaraDB — Универсален план за Advanced SQL Engine
> **Визия**: BaraDB е самостоятелен, универсален SQL engine с Nim ядро, поддържащ модерни SQL:2023 разширения — Property Graph, Vector Search, JSON документи и прозоречни функции, в една вградена или клиент/сървър конфигурация.
>
> **Принцип**: Само основи. Не се добавят нови светове — само стабилизираме и документираме съществуващите.
>
> **Multi-Tenant фокус**: BaraDB е проектирана да поддържа ERP сценарии с много фирми (tenants) в една база данни. Всеки tenant се изолира чрез Row-Level Security (RLS) + session variables (`SET app.tenant_id = 'X'`), а не чрез отделни бази.
---
## История на разработката
- **Фаза 1 (Base SQL + MVCC + Raft)**: BaraDB core engine
- **Фаза 2 (Advanced SQL)**: Разработена с **Xiaomi Mimo** (`mimo-v2.5-pro`) — Window Functions, MERGE, LATERAL JOIN, Advanced Aggregates, PIVOT/UNPIVOT, SQL/PGQ Property Graph
- **Фаза 3 (Stabilization + Multi-Tenant)**: Текуща — Vector SQL Integration, Session Variables, `current_user`/`current_role`, RLS tenant isolation, тестове, документация
---
---
## Част 1: BaraDB Advanced SQL Engine
### 1.1 Window Functions ✅ ГОТОВО
Нови AST nodes: `nkWindowExpr`, `nkOverClause`, `nkFrameSpec`. Нов IR plan: `irpkWindow`.
| Функция | Описание | Статус |
|---------|----------|--------|
| `ROW_NUMBER()` | Пореден номер в партишъна | ✅ |
| `RANK()` / `DENSE_RANK()` | Класиране с/без gaps | ✅ |
| `LEAD(col, n, default)` / `LAG(col, n, default)` | Достъп до съседни редове | ✅ |
| `FIRST_VALUE(col)` / `LAST_VALUE(col)` | Краен елемент във frame | ✅ |
| `NTILE(n)` | Bucket-ване в n части | ✅ |
Frame поддръжка: `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`, `codegen.nim`
Тестове: 5 теста в `tests/test_all.nim`, всички зелени.
### 1.2 MERGE / UPSERT ✅ ГОТОВО
```sql
MERGE INTO inventory AS target
USING updates AS source
ON target.sku = source.sku
WHEN MATCHED THEN UPDATE SET qty = target.qty + source.delta
WHEN NOT MATCHED THEN INSERT (sku, qty) VALUES (source.sku, source.delta);
```
- Поддържа таблица или subquery като source
- WHEN MATCHED UPDATE с eval на изрази (target.col + source.col)
- WHEN NOT MATCHED INSERT с eval на value изрази
- Trigger support (BEFORE/AFTER UPDATE/INSERT)
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`, `codegen.nim`
Тестове: 2 теста в `tests/test_all.nim`, всички зелени.
### 1.3 LATERAL JOIN / CROSS APPLY ✅ ГОТОВО
Позволява correlated subquery във FROM clause с достъп до лявата таблица.
```sql
SELECT u.name, recent_orders.*
FROM users u,
LATERAL (
SELECT order_id, total FROM orders o
WHERE o.user_id = u.id ORDER BY created_at DESC LIMIT 3
) recent_orders;
```
- Поддържа `JOIN LATERAL`, `LEFT JOIN LATERAL`, `CROSS JOIN LATERAL`
- Correlated references (e.g. `u.id`) чрез scan + merge + filter стратегия
- Sort и Limit от subquery се прилагат след merge
- LEFT LATERAL запазва unmatched редове с NULL padding
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`
Тестове: 4 execution теста + 3 parser теста, всички зелени.
### 1.4 Advanced Aggregates ✅ ГОТОВО
- `ARRAY_AGG(col ORDER BY ...)`
- `STRING_AGG(col, delimiter)`
- `COUNT(*) FILTER (WHERE ...)`
- `GROUPING SETS`, `CUBE`, `ROLLUP`
#### GROUP BY + HAVING ✅ ГОТОВО
- SUM/AVG/MIN/MAX оценяват се в групите
- HAVING филтрира групите по aggregate условия
- Pre-computed aggregates се съхраняват в group rows
- evalExpr поддържа irekAggregate lookup
Тестове: 6 теста в `tests/test_all.nim`, всички зелени.
#### FILTER (WHERE ...) ✅ ГОТОВО
```sql
SELECT COUNT(*) FILTER (WHERE active = true) FROM users;
SELECT dept, SUM(amount) FILTER (WHERE amount > 100) FROM sales GROUP BY dept;
```
- Parser: `FILTER (WHERE ...)` след aggregate function call
- AST: `funcFilter*: Node` на `nkFuncCall`
- IR: `aggFilter*: IRExpr` на `irekAggregate`
- Executor: филтрира редове преди aggregate computation
Тестове: 2 execution теста + 1 parser тест, всички зелени.
#### ARRAY_AGG / STRING_AGG ✅ ГОТОВО
```sql
SELECT dept, ARRAY_AGG(amount) AS amounts FROM sales GROUP BY dept;
SELECT dept, STRING_AGG(name, ', ') AS names FROM employees GROUP BY dept;
```
- Нови IR aggregate ops: `irArrayAgg`, `irStringAgg`
- Multi-argument aggregate parsing (delimiter за STRING_AGG)
- FILTER support за двете функции
Тестове: 2 теста, всички зелени.
#### GROUPING SETS / ROLLUP / CUBE ✅ ГОТОВО
```sql
SELECT dept, SUM(amount) FROM sales GROUP BY ROLLUP (dept);
SELECT dept, job, SUM(amount) FROM sales GROUP BY CUBE (dept, job);
SELECT dept, job, SUM(amount) FROM sales GROUP BY GROUPING SETS ((dept), (job), ());
```
- ROLLUP(a, b) → GROUPING SETS ((a,b), (a), ())
- CUBE(a, b) → GROUPING SETS ((a,b), (a), (b), ())
- Генериране на subsets за CUBE чрез powerset алгоритъм
Тестове: 4 parser теста + 1 execution тест, всички зелени.
### 1.5 PIVOT / UNPIVOT ✅ ГОТОВО
```sql
SELECT * FROM (SELECT name, dept, salary FROM emp)
PIVOT (SUM(salary) FOR dept IN ('Eng', 'Sales'));
SELECT * FROM emp
UNPIVOT (salary FOR dept IN (eng_salary, sales_salary));
```
- Parser: PIVOT/UNPIVOT в FROM clause
- IR: `irpkPivot`, `irpkUnpivot`
- Executor: group by identity cols → aggregate per pivot value → create columns
- Subquery storage в `nkFrom.fromSubquery`
Тестове: 1 parser + 1 execution тест, всички зелени.
### 1.6 SQL:2023 Property Graph (SQL/PGQ) ✅ ГОТОВО (Parser)
```sql
SELECT * FROM GRAPH_TABLE(org_chart
MATCH (e)-[r]->(d)
COLUMNS (e.name, d.name)
);
```
- Lexer: `tkVertex`, `tkEdge`, `tkLabels`, `tkGraphTable`, `tkMatch`, `tkColumns`, `tkSrc`, `tkDst`
- AST: `nkGraphTraversal` с `gtGraphName`, `gtReturnCols`
- IR: `irpkGraphTraversal` с `graphName`, `graphAlgo`, `graphReturnCols`
- Executor: table-based graph storage (`graph_nodes`, `graph_edges`)
- Parser: `GRAPH_TABLE(name MATCH (pattern) COLUMNS (cols))`
Тестове: 1 parser тест, всички зелени.
---
## Част 1.5: Multi-Tenant ERP Support ✅ ГОТОВО
BaraDB поддържа multi-tenant архитектура, при която множество фирми (tenants) работят в една физическа база данни. Това е критично за ERP сценарии, където поддръжката на "сто бази" не е опция.
### Механизъм
| Компонент | Описание |
|-----------|----------|
| **Session Variables** | `SET app.tenant_id = 'company-123'` — задава tenant за текущата сесия |
| **current_setting()** | `current_setting('app.tenant_id')` — чете session променлива в SQL израз |
| **current_user** | `current_user` — връща автентикирания потребител от JWT/SCRAM |
| **current_role** | `current_role` — връща ролята на автентикирания потребител |
| **RLS Policies** | `CREATE POLICY tenant_isolation ON invoices FOR SELECT USING (tenant_id = current_setting('app.tenant_id'))` |
| **Auth Bridge** | `server.nim` и `httpserver.nim` попълват `ExecutionContext.currentUser`/`currentRole` след верификация |
### Пример
```sql
-- Една таблица за всички фирми
CREATE TABLE invoices (
id SERIAL PRIMARY KEY,
tenant_id TEXT NOT NULL,
data JSONB
);
-- Изолация чрез RLS
CREATE POLICY tenant_isolation ON invoices
FOR SELECT USING (tenant_id = current_setting('app.tenant_id'));
-- Всяка сесия вижда само своя tenant
SET app.tenant_id = 'company-a';
SELECT * FROM invoices; -- → само фактури на company-a
```
### Архитектурни предимства
- **JSONB документи** — schema-flexible, лесно се добавят нови полета без миграции (като ArangoDB)
- **RLS изолация** — базата данни гарантира, че всеки tenant вижда само своите данни
- **Един instance** — един BaraDB сървър обслужва всички tenants, вместо сто отделни бази
- **Auth integration** — JWT/SCRAM токените носят `sub` (user) и `role`, които се пропагират до executor-а
---
## Част 2: Мултимодални Възможности (Core Only)
### 2.1 JSON / JSONB Документи ✅ ГОТОВО
```sql
SELECT data->>'name' FROM users WHERE data->'tags' @> '["admin"]';
```
- Типове: `JSON`, `JSONB` колони в таблици
- Оператори: `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?&`, `?|`
- Функции: `jsonb_array_elements`, `jsonb_object_keys`, `jsonb_extract_path`
- Съхранение: двоично parsed tree (не plain text)
### 2.2 Vector Search ⚠️ ЧАСТИЧНО (Engine ✅, SQL Integration 🔄)
**Вектор Engine (готов):**
- `src/barabadb/vector/engine.nim` — HNSW index с cosine/euclidean distance
- `src/barabadb/vector/quant.nim` — IVF-PQ quantization
- `src/barabadb/vector/simd.nim` — SIMD оптимизации
- `src/barabadb/core/crossmodal.nim` — CrossModalEngine за хибридно търсене (vector + text)
**Липсваща SQL интеграция (базова — за стабилизация):**
```sql
-- Тип и колона
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
-- Index
CREATE VECTOR INDEX idx_items_vec ON items(embedding)
USING hnsw WITH (m = 16, ef_construction = 200, metric = 'cosine');
-- Query functions
SELECT id, cosine_distance(embedding, '[0.1, 0.2, ...]') AS dist
FROM items
ORDER BY dist ASC
LIMIT 10;
```
**Задачи за стабилизация (всички изпълнени):**
- [x] `VECTOR(n)` тип в CREATE TABLE (parser + storage)
- [x] `CREATE VECTOR INDEX ... USING hnsw` (DDL)
- [x] `cosine_distance()`, `euclidean_distance()`, `inner_product()` в SQL expression evaluator
- [x] `<->` nearest-neighbor оператор в ORDER BY / WHERE
- [x] Executor integration: HNSW index population при CREATE INDEX и DML
**Статус:** ✅ ГОТОВО. 8 SQL-level vector теста зелени.
### 2.3 Full-Text Search ✅ ГОТОВО
- Inverted Index в `src/barabadb/fts/`
- `MATCH(column, query)` функция
- BM25 scoring
- Интеграция с CrossModalEngine за hybrid search
---
## Част 3: Транзакции и Протоколи ✅ ГОТОВО
- MVCC с snapshot isolation
- WAL + checkpoint
- Distributed transactions (2PC) — `txn.addParticipant("vector")`
- Wire protocol: binary за vectors, JSON за queries
---
## Имплементационен ред (финален статус)
1.**Window Functions** (AST → Parser → IR → Executor → Tests)
2.**MERGE statement** (Parser → Executor → Tests)
3.**LATERAL JOIN** (Parser → Executor, correlated subquery strategy)
4.**GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter)
5.**FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...))
6.**ARRAY_AGG / STRING_AGG** (multi-arg aggregates)
7.**GROUPING SETS / ROLLUP / CUBE** (powerset generation)
8.**PIVOT / UNPIVOT** (row-to-column transformation)
9.**SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser)
10.**JSON/JSONB** (operators + functions)
11.**Full-Text Search** (inverted index + BM25)
12.**Vector Engine** (HNSW + IVF-PQ + SIMD)
13.**Vector SQL Integration** (тип, index, distance functions, <-> operator, ORDER BY)
---
## Крайно състояние
**340+ теста зелени.** Всички фундаментални SQL:2023 features имплементирани.
**Четирите свята:**
| Свят | Features | Статус |
|------|----------|--------|
| **SQL** | Window, MERGE, LATERAL, GROUP BY/HAVING, FILTER, ARRAY_AGG, STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT | ✅ |
| **JSON** | JSON/JSONB колони, `->` / `->>` оператори | ✅ |
| **Graph** | BFS/DFS/PageRank/Dijkstra engine + SQL/PGQ GRAPH_TABLE | ✅ |
| **Vector** | HNSW index, cosine/euclidean distance, IVF-PQ, SIMD | ✅ Engine<br>🔄 SQL glue |
| **FTS** | Inverted index, BM25, hybrid search | ✅ |
**Файлове модифицирани:**
- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube, tkVector
- `ast.nim` — joinLateral, funcFilter, nkPivot, nkUnpivot, GroupingSetsKind, nkGraphTraversal fields
- `ir.nim` — joinLateral, aggFilter, irArrayAgg, irStringAgg, IRGroupingSetsKind, irpkGroupBy grouping sets, irpkPivot, irpkUnpivot, irpkGraphTraversal
- `parser.nim` — LATERAL, FILTER, multi-arg aggregates, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE
- `executor.nim` — LATERAL correlated strategy, GROUP BY aggregates + HAVING, FILTER in aggregates, ARRAY_AGG/STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE, fromTable kind checks
- `codegen.nim` — irpkPivot, irpkUnpivot, irpkGraphTraversal
- `tests/test_all.nim` — 25+ нови теста
- `tests/join_tests.nim` — 4 LATERAL теста
---
## Тестова стратегия
- **Unit**: Всеки нов AST/IR/Parser тест — property-based (генериране на случайни partition/order)
- **Integration**: HTTP server + клиент тестове
- **TLA+**: `windowfunctions.tla` — deterministic partitioning semantics
- **Benchmark**: Window function performance vs PostgreSQL (опционално)
---
## Поправени грешки при тази сесия
- **Vector SQL Integration** — имплементиран пълен SQL glue за вектори (тип, индекс, функции, оператор)
- **MERGE тестове** — поправени чрез изолиране на тестовата директория (unique temp dir per suite)
- **Row storage escape** — `escapeRowVal()` в `execInsert` за стойности със запетай (vector literals)
- **ORDER BY + projection** — `irpkSort` сега е преди `irpkProject` в `lowerSelect`, което позволява `ORDER BY` по колони извън `SELECT`
- **GROUPING SETS execution** — `lowerSelect` сега проверява `selGroupingSetsKind != gskNone` освен `selGroupBy.len > 0`, което позволява изпълнение на GROUPING SETS без традиционен GROUP BY
- **FTS CREATE INDEX docId** — поправено несъответствие в изчислението на `docId` при `CREATE INDEX ... USING FTS` (сега използва хеш на `tableName.$key`, съвместим с DML операциите)
- **Тестова изолация (всички suite-ове)** — всички `newLSMTree("")` заменени с уникални temp директории; setup/teardown за suite-ове с изолирана state
- **Multi-tenant ERP support** — имплементирани критични градивни елементи:
- `SET var = value` — session variables за tenant isolation
- `current_setting('var')` — четене на session променливи в SQL изрази
- `current_user` / `current_role` — SQL keywords, които се оценяват от `ExecutionContext`
- Auth bridge — `server.nim` и `httpserver.nim` попълват `currentUser`/`currentRole` след JWT/SCRAM верификация
- RLS tenant isolation тест — `CREATE POLICY` + `current_setting('app.tenant_id')` работи за multi-tenant филтрация
- `evalExpr` вече предава `ctx` рекурсивно — поправен бъг, при който `current_user`/`current_setting` връщаха празни стойности в под-изрази
---
> **Бележка**: Този план е *замразен* за нови светове. Следващата работа е само стабилизация на съществуващото и документация.
+60
View File
@@ -0,0 +1,60 @@
# Nodebara Compatibility Fixes
This PR bundles three independent fixes discovered while integrating BaraDB with [NodeBB](https://nodebb.org/) (a large Node.js forum application). Each commit is self-contained and can be reviewed separately.
---
## 1. fix(protocol): serialize float32/float64 in big-endian
**Problem:**
The JavaScript client reads `FLOAT32`/`FLOAT64` wire values with `readFloatBE()` / `readDoubleBE()` (big-endian), but the Nim server was writing them with `copyMem(..., unsafeAddr fl, N)` — i.e. **native byte order**. On little-endian machines (virtually all x86_64 servers) the client deserializes garbage. Example: a zset `score = 1.0` becomes `3.03865e-319`, which breaks any application relying on numeric scores (user IDs, timestamps, rankings, etc.).
**Fix:**
Cast floats to `int32`/`int64` and route them through the existing `bigEndian32`/`bigEndian64` helpers, exactly like the integer paths already do. Same change applied to deserialization.
**Impact:**
Breaking fix for cross-platform wire compatibility. No API changes.
---
## 2. feat(client): add TCP request queue for safe concurrency
**Problem:**
`Client.query()`, `execute()`, and `ping()` were async methods that wrote directly to the TCP socket. When NodeBB fired multiple parallel DB operations (common on startup), their binary frames interleaved on the wire, causing parse errors, wrong request/response pairing, and random crashes.
**Fix:**
Introduce an internal `_requestQueue` + `_requestLock`. Every public async method enqueues a closure; a tiny drain loop processes them one at a time via `setImmediate()`.
**Impact:**
No breaking API change. Existing single-request usage is unchanged; concurrent usage now works safely.
---
## 3. fix(gossip): use async UDP socket to avoid blocking the event loop
**Problem:**
`startGossipListener` created a **synchronous** UDP socket (`newSocket`) and called blocking `recvFrom` inside an `async` proc. This freezes the entire async event loop until a UDP packet arrives, stalling all other async I/O.
**Fix:**
Replace `newSocket` with `newAsyncSocket` and `recvFrom` with `await recvFrom`.
**Impact:**
Non-breaking. Gossip remains optional; when enabled it no longer blocks the main loop.
---
## Testing
- [x] NodeBB v4.11.2 setup completes end-to-end
- [x] Admin login works (relies on correct FLOAT64 score deserialization)
- [x] Concurrent DB queries during startup no longer corrupt frames
- [x] Gossip listener no longer blocks other async tasks
---
## Checklist
- [x] Each commit is atomic and compiles on its own
- [x] No Nim compiler warnings introduced
- [x] JS client backward-compatible for single-request callers
- [x] Existing tests (`nimble test`) still pass
+40 -4
View File
@@ -4,7 +4,7 @@
**A multimodal database engine written in Nim — 100% native, zero dependencies.** **A multimodal database engine written in Nim — 100% native, zero dependencies.**
[![Version](https://img.shields.io/badge/version-1.0.0-blue.svg)](baradadb.nimble) [![Version](https://img.shields.io/badge/version-1.1.0-blue.svg)](baradadb.nimble)
[![Documentation](https://img.shields.io/badge/docs-7_languages-blue.svg)](docs/index.md) [![Documentation](https://img.shields.io/badge/docs-7_languages-blue.svg)](docs/index.md)
## Documentation ## Documentation
@@ -285,8 +285,23 @@ let range = btree.scan("key_a", "key_z")
### Vector Engine ### Vector Engine
Native HNSW and IVF-PQ indexes for similarity search. Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
```sql
-- SQL vector search
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
-- Nearest neighbor search
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, ...]') ASC
LIMIT 10;
-- With HNSW index
CREATE INDEX idx_vec ON items(embedding) USING hnsw;
```
Native Nim API:
```nim ```nim
import barabadb/vector/engine import barabadb/vector/engine
@@ -301,7 +316,10 @@ let filtered = idx.searchWithFilter(queryVector, k = 10,
``` ```
Features: Features:
- **HNSW** — hierarchical navigable small world graph - **SQL vector types** — `VECTOR(n)` with dimension validation
- **SQL distance functions** — `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- **`<->` operator** — Euclidean distance nearest-neighbor shorthand
- **HNSW index** — `CREATE INDEX ... USING hnsw` with automatic maintenance
- **IVF-PQ** — inverted file index with product quantization - **IVF-PQ** — inverted file index with product quantization
- **Distance metrics** — cosine, euclidean, dot product, Manhattan - **Distance metrics** — cosine, euclidean, dot product, Manhattan
- **Quantization** — scalar 8-bit/4-bit, product, binary - **Quantization** — scalar 8-bit/4-bit, product, binary
@@ -606,6 +624,23 @@ See [docs/en/docker.md](docs/en/docker.md) for full Docker documentation.
| `BARADB_CERT_FILE` | — | TLS certificate path | | `BARADB_CERT_FILE` | — | TLS certificate path |
| `BARADB_KEY_FILE` | — | TLS private key path | | `BARADB_KEY_FILE` | — | TLS private key path |
## Built with BaraDB
### NodeBara
**[NodeBara](https://codeberg.org/baraDB/nodebara)** is the first large-scale application running on BaraDB — a modern forum platform forked from NodeBB and fully adapted for BaraDB's native multimodal engine.
- **Concurrent query safety** — TCP request queue in the JS client handles NodeBara's parallel startup queries without frame corruption
- **Numeric accuracy** — Big-endian float serialization guarantees correct zset scores, timestamps, and rankings across platforms
- **Non-blocking cluster gossip** — Async UDP sockets keep the event loop free under load
```bash
git clone https://codeberg.org/baraDB/nodebara
cd nodebara
npm install
npm run setup # uses BaraDB as the default database
```
## Client SDKs ## Client SDKs
BaraDB provides official clients for multiple languages: BaraDB provides official clients for multiple languages:
@@ -1214,7 +1249,7 @@ src/barabadb/
## Tests ## Tests
```bash ```bash
# Run all tests (262 tests, 56 suites) # Run all tests (340+ tests, 60+ suites)
nim c --path:src -r tests/test_all.nim nim c --path:src -r tests/test_all.nim
# Run benchmarks # Run benchmarks
@@ -1232,6 +1267,7 @@ nim c -d:release -r benchmarks/bench_all.nim
| Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 | | Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 |
| Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 | | Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 |
| Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 | | Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 |
| Vector SQL Integration (VECTOR type, distance functions, <->, HNSW indexes) | ✅ | 100% | v1.1.0 |
| Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 | | Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 |
| FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 | | FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 |
| CLI shell | ✅ | 100% | v1.0.0 | | CLI shell | ✅ | 100% | v1.0.0 |
+1 -1
View File
@@ -1,5 +1,5 @@
# Package # Package
version = "1.0.0" version = "1.1.0"
author = "BaraDB Team" author = "BaraDB Team"
description = "BaraDB — Multimodal database written in Nim" description = "BaraDB — Multimodal database written in Nim"
license = "Apache-2.0" license = "Apache-2.0"
+78 -48
View File
@@ -244,6 +244,8 @@ class Client {
this.requestId = 0; this.requestId = 0;
this._buffer = Buffer.alloc(0); this._buffer = Buffer.alloc(0);
this._pendingResolve = null; this._pendingResolve = null;
this._requestQueue = [];
this._requestLock = false;
} }
async connect() { async connect() {
@@ -478,70 +480,98 @@ class Client {
throw new Error(`Unexpected auth response: 0x${header.kind.toString(16)}`); throw new Error(`Unexpected auth response: 0x${header.kind.toString(16)}`);
} }
async _processQueue() {
if (this._requestLock || this._requestQueue.length === 0) return;
this._requestLock = true;
const { task, resolve, reject } = this._requestQueue.shift();
try {
const result = await task();
resolve(result);
} catch (err) {
reject(err);
} finally {
this._requestLock = false;
setImmediate(() => this._processQueue());
}
}
_enqueue(task) {
return new Promise((resolve, reject) => {
this._requestQueue.push({ task, resolve, reject });
this._processQueue();
});
}
async ping() { async ping() {
if (!this.connected) throw new Error('Not connected'); if (!this.connected) throw new Error('Not connected');
const msg = this._build(MsgKind.PING, Buffer.alloc(0)); return this._enqueue(async () => {
this.socket.write(msg); const msg = this._build(MsgKind.PING, Buffer.alloc(0));
this.socket.write(msg);
const header = await this._readHeader(); const header = await this._readHeader();
if (header.kind === MsgKind.PONG) return true; if (header.kind === MsgKind.PONG) return true;
if (header.kind === MsgKind.ERROR) throw await this._readError(header.length); if (header.kind === MsgKind.ERROR) throw await this._readError(header.length);
return false; return false;
});
} }
async query(sql) { async query(sql) {
if (!this.connected) throw new Error('Not connected'); if (!this.connected) throw new Error('Not connected');
const queryBuf = Buffer.from(sql, 'utf-8'); return this._enqueue(async () => {
const payload = Buffer.alloc(4 + queryBuf.length + 1); const queryBuf = Buffer.from(sql, 'utf-8');
payload.writeUInt32BE(queryBuf.length, 0); const payload = Buffer.alloc(4 + queryBuf.length + 1);
queryBuf.copy(payload, 4); payload.writeUInt32BE(queryBuf.length, 0);
payload[4 + queryBuf.length] = ResultFormat.BINARY; queryBuf.copy(payload, 4);
payload[4 + queryBuf.length] = ResultFormat.BINARY;
const msg = this._build(MsgKind.QUERY, payload); const msg = this._build(MsgKind.QUERY, payload);
this.socket.write(msg); this.socket.write(msg);
const header = await this._readHeader(); const header = await this._readHeader();
if (header.kind === MsgKind.ERROR) throw await this._readError(header.length); if (header.kind === MsgKind.ERROR) throw await this._readError(header.length);
if (header.kind === MsgKind.DATA) return await this._readDataResponse(header.length); if (header.kind === MsgKind.DATA) return await this._readDataResponse(header.length);
if (header.kind === MsgKind.COMPLETE) { if (header.kind === MsgKind.COMPLETE) {
const data = await this._recvExact(header.length); const data = await this._recvExact(header.length);
const result = new QueryResult(); const result = new QueryResult();
result.affectedRows = data.readUInt32BE(0); result.affectedRows = data.readUInt32BE(0);
return result; return result;
} }
return new QueryResult(); return new QueryResult();
});
} }
async queryParams(sql, params = []) { async queryParams(sql, params = []) {
if (!this.connected) throw new Error('Not connected'); if (!this.connected) throw new Error('Not connected');
const queryBuf = Buffer.from(sql, 'utf-8'); return this._enqueue(async () => {
const paramParts = []; const queryBuf = Buffer.from(sql, 'utf-8');
for (const p of params) { const paramParts = [];
paramParts.push(p.serialize()); for (const p of params) {
} paramParts.push(p.serialize());
const paramsBuf = Buffer.concat(paramParts); }
const paramsBuf = Buffer.concat(paramParts);
const payload = Buffer.alloc(4 + queryBuf.length + 1 + 4 + paramsBuf.length); const payload = Buffer.alloc(4 + queryBuf.length + 1 + 4 + paramsBuf.length);
let pos = 0; let pos = 0;
payload.writeUInt32BE(queryBuf.length, pos); pos += 4; payload.writeUInt32BE(queryBuf.length, pos); pos += 4;
queryBuf.copy(payload, pos); pos += queryBuf.length; queryBuf.copy(payload, pos); pos += queryBuf.length;
payload[pos] = ResultFormat.BINARY; pos++; payload[pos] = ResultFormat.BINARY; pos++;
payload.writeUInt32BE(params.length, pos); pos += 4; payload.writeUInt32BE(params.length, pos); pos += 4;
paramsBuf.copy(payload, pos); paramsBuf.copy(payload, pos);
const msg = this._build(MsgKind.QUERY_PARAMS, payload); const msg = this._build(MsgKind.QUERY_PARAMS, payload);
this.socket.write(msg); this.socket.write(msg);
const header = await this._readHeader(); const header = await this._readHeader();
if (header.kind === MsgKind.ERROR) throw await this._readError(header.length); if (header.kind === MsgKind.ERROR) throw await this._readError(header.length);
if (header.kind === MsgKind.DATA) return await this._readDataResponse(header.length); if (header.kind === MsgKind.DATA) return await this._readDataResponse(header.length);
if (header.kind === MsgKind.COMPLETE) { if (header.kind === MsgKind.COMPLETE) {
const data = await this._recvExact(header.length); const data = await this._recvExact(header.length);
const result = new QueryResult(); const result = new QueryResult();
result.affectedRows = data.readUInt32BE(0); result.affectedRows = data.readUInt32BE(0);
return result; return result;
} }
return new QueryResult(); return new QueryResult();
});
} }
async execute(sql) { async execute(sql) {
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "baradb", "name": "baradb",
"version": "1.0.0", "version": "1.1.0",
"description": "Official JavaScript/Node.js client for BaraDB — Multimodal Database Engine", "description": "Official JavaScript/Node.js client for BaraDB — Multimodal Database Engine",
"main": "baradb.js", "main": "baradb.js",
"types": "baradb.d.ts", "types": "baradb.d.ts",
+1 -1
View File
@@ -1,6 +1,6 @@
# Package # Package
version = "1.0.0" version = "1.1.0"
author = "BaraDB Team" author = "BaraDB Team"
description = "Official Nim client for BaraDB — async binary protocol client" description = "Official Nim client for BaraDB — async binary protocol client"
license = "Apache-2.0" license = "Apache-2.0"
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project] [project]
name = "baradb" name = "baradb"
version = "1.0.0" version = "1.1.0"
description = "Official Python client for BaraDB — Multimodal Database Engine" description = "Official Python client for BaraDB — Multimodal Database Engine"
readme = "README.md" readme = "README.md"
license = { text = "Apache-2.0" } license = { text = "Apache-2.0" }
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "baradb" name = "baradb"
version = "1.0.0" version = "1.1.0"
edition = "2021" edition = "2021"
authors = ["BaraDB Team <team@baradb.dev>"] authors = ["BaraDB Team <team@baradb.dev>"]
description = "Official Rust client for BaraDB — binary protocol client" description = "Official Rust client for BaraDB — binary protocol client"
+6
View File
@@ -90,6 +90,12 @@ The query layer processes BaraQL — a SQL-compatible query language with extens
- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression. - **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression.
- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan). - **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan).
- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput. - **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput.
- **SQL Integration** (`query/executor.nim`):
- `VECTOR(n)` column type with dimension validation
- `CREATE INDEX ... USING hnsw` / `USING ivfpq`
- Distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- `<->` nearest-neighbor operator
- Automatic index maintenance on INSERT/UPDATE
### Graph Engine (`graph/`) ### Graph Engine (`graph/`)
- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency. - **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency.
+147 -21
View File
@@ -18,6 +18,7 @@ BaraQL is a SQL-compatible query language with extensions for graph, vector, and
| `bytes` | Raw bytes | `0xDEADBEEF` | | `bytes` | Raw bytes | `0xDEADBEEF` |
| `array<T>` | Homogeneous array | `[1, 2, 3]` | | `array<T>` | Homogeneous array | `[1, 2, 3]` |
| `vector` | Float32 vector | `[0.1, 0.2, 0.3]` | | `vector` | Float32 vector | `[0.1, 0.2, 0.3]` |
| `vector(n)` | Fixed-dimension float32 vector (SQL) | `VECTOR(768)` |
| `object` | Key-value object | `{"a": 1}` | | `object` | Key-value object | `{"a": 1}` |
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` | | `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` | | `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
@@ -352,6 +353,7 @@ CREATE TYPE Cat EXTENDING Animal {
CREATE INDEX idx_users_name ON users(name); CREATE INDEX idx_users_name ON users(name);
CREATE UNIQUE INDEX idx_users_email ON users(email); CREATE UNIQUE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_age ON users(age) USING btree; CREATE INDEX idx_users_age ON users(age) USING btree;
CREATE INDEX idx_vectors ON items(embedding) USING hnsw;
``` ```
### DROP ### DROP
@@ -387,37 +389,76 @@ SELECT * FROM articles WHERE body @@ 'machine learning';
RECOVER TO TIMESTAMP '2026-05-07T12:00:00'; RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
``` ```
## Vector Search ## Vector Search (SQL)
### Creating Vector Columns
```sql ```sql
-- Insert with vector CREATE TABLE items (
INSERT articles { id INT PRIMARY KEY,
title := 'Nim Programming', embedding VECTOR(768)
embedding := [0.1, 0.2, 0.3, 0.4] );
}; ```
-- Similarity search (cosine distance) ### Inserting Vectors
SELECT title FROM articles
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
LIMIT 5;
-- Euclidean distance ```sql
SELECT title FROM articles INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, 0.4]');
ORDER BY l2_distance(embedding, [0.1, 0.2, 0.3, 0.4]) ```
LIMIT 5;
-- Dot product ### Distance Functions
SELECT title FROM articles
ORDER BY dot_product(embedding, [0.1, 0.2, 0.3, 0.4]) DESC ```sql
-- Cosine distance (0 = identical, 2 = opposite)
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- Euclidean / L2 distance
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- L2 distance with <-> operator
SELECT id, embedding <-> '[0.1, 0.2, 0.3, 0.4]' AS dist
FROM items;
-- Inner product (negative dot product)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
-- Manhattan / L1 distance
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') AS dist
FROM items;
```
### Nearest Neighbor Search
```sql
-- Top-10 nearest neighbors by cosine distance
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]') ASC
LIMIT 10;
-- Top-5 nearest neighbors by Euclidean distance
SELECT id FROM items
ORDER BY embedding <-> '[0.1, 0.2, 0.3, 0.4]'
LIMIT 5; LIMIT 5;
-- With metadata filter -- With metadata filter
SELECT title FROM articles SELECT id FROM items
WHERE category = 'tech' WHERE category = 'tech'
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4]) ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, 0.4]')
LIMIT 5; LIMIT 5;
``` ```
### Vector Indexes
```sql
-- Create HNSW index for approximate nearest neighbor search
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
-- Supported index methods: hnsw, ivfpq
```
## Graph Patterns ## Graph Patterns
```sql ```sql
@@ -517,6 +558,88 @@ LIMIT 10;
SELECT /*+ PARALLEL(4) */ * FROM large_table; SELECT /*+ PARALLEL(4) */ * FROM large_table;
``` ```
## Window Functions
```sql
-- Ranking functions
SELECT
name,
department,
ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) AS rn,
RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS r,
DENSE_RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS dr
FROM employees;
-- Value functions
SELECT
name,
salary,
LAG(salary, 1, 0) OVER (ORDER BY salary) AS prev_salary,
LEAD(salary, 1, 0) OVER (ORDER BY salary) AS next_salary,
FIRST_VALUE(name) OVER (PARTITION BY department ORDER BY salary) AS cheapest,
LAST_VALUE(name) OVER (PARTITION BY department ORDER BY salary) AS most_expensive
FROM employees;
-- Distribution functions
SELECT name, NTILE(4) OVER (ORDER BY salary) AS quartile FROM employees;
```
### Frame Specifications
```sql
-- ROWS frame
SUM(salary) OVER (
PARTITION BY department
ORDER BY hire_date
ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
)
-- RANGE frame
SUM(salary) OVER (
PARTITION BY department
ORDER BY hire_date
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
)
```
## Multi-Tenant ERP
BaraDB supports running multiple companies (tenants) in a single database instance, using **Row-Level Security (RLS)** combined with **session variables**.
### Session Variables
```sql
SET app.tenant_id = 'company-123';
SELECT current_setting('app.tenant_id') AS tenant;
```
### Current User / Role
```sql
SELECT current_user AS me, current_role AS my_role;
```
### RLS Tenant Isolation
```sql
-- Enable RLS on a table
ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
-- Create a policy that filters by tenant
CREATE POLICY tenant_isolation ON invoices
FOR SELECT USING (tenant_id = current_setting('app.tenant_id'));
-- Each session only sees its own data
SET app.tenant_id = 'company-a';
SELECT * FROM invoices; -- only company-a rows
```
### Why Multi-Tenant?
- **One instance, many tenants** — no need to run 100 separate databases
- **JSONB documents** — schema-flexible storage, easy to add fields per tenant
- **RLS guarantees isolation** — the database enforces tenant boundaries, not just the application
## Supported Keywords ## Supported Keywords
| Category | Keywords | | Category | Keywords |
@@ -531,8 +654,11 @@ SELECT /*+ PARALLEL(4) */ * FROM large_table;
| Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT | | Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT |
| Graph | MATCH, RETURN, WHERE, shortestPath, type | | Graph | MATCH, RETURN, WHERE, shortestPath, type |
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS | | FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance | | Vector | cosine_distance, euclidean_distance, inner_product, l1_distance, l2_distance, <-> |
| JSON | ->, ->> | | JSON | ->, ->> |
| FTS | @@ (BM25 match) | | FTS | @@ (BM25 match) |
| Recovery | RECOVER TO TIMESTAMP | | Recovery | RECOVER TO TIMESTAMP |
| Functions | count, sum, avg, min, max, stddev, variance, abs, sqrt, lower, upper, len, trim, substr, now, last_insert_id | | Functions | count, sum, avg, min, max, stddev, variance, abs, sqrt, lower, upper, len, trim, substr, now, last_insert_id, current_setting |
| Session | SET, current_setting, current_user, current_role |
| Window | OVER, PARTITION BY, ROWS, RANGE, UNBOUNDED PRECEDING, CURRENT ROW, FOLLOWING |
| Window Functions | ROW_NUMBER, RANK, DENSE_RANK, LEAD, LAG, FIRST_VALUE, LAST_VALUE, NTILE |
+78
View File
@@ -2,6 +2,65 @@
All notable changes to BaraDB are documented in this file. All notable changes to BaraDB are documented in this file.
## [Unreleased] — SQL:2023 Stabilization
### Fixed
- **GROUPING SETS execution** — `lowerSelect` now creates `irpkGroupBy` when `selGroupingSetsKind != gskNone`, even if `selGroupBy` is empty. Previously, queries like `GROUP BY GROUPING SETS ((dept), ())` bypassed the grouping executor entirely.
- **FTS CREATE INDEX docId mismatch** — `CREATE INDEX ... USING FTS` now computes `docId` as a hash of `tableName.$key`, consistent with DML operations (`INSERT`/`UPDATE`/`DELETE`). Previously, index creation used sequential IDs (0, 1, 2...), causing `@@` queries to never match indexed documents.
- **Test isolation (all suites)** — All `newLSMTree("")` calls replaced with unique temporary directories per suite. Eliminates WAL accumulation issues and flaky tests caused by shared database state between test runs.
- **Window frame parser** — `parseFrameBoundary` no longer consumes `tkRow` after `tkCurrent` incorrectly (was using `tkRows`). Also fixed `tkRow` keyword conflict with `ENABLE ROW LEVEL SECURITY` parsing.
- **ORDER BY + SELECT projection** — `lowerSelect` now places `irpkSort` before `irpkProject`, enabling `ORDER BY` on columns not present in the `SELECT` list.
- **UNPIVOT execution** — Verified and fixed missing test coverage for UNPIVOT transformation.
### Added
- **JSON operators** — `@>` (contains), `<@` (contained by), `?` (has key), `?|` (has any), `?&` (has all) now supported in lexer, parser, and executor.
- **Window frame execution** — `ROWS BETWEEN X PRECEDING AND Y FOLLOWING` / `CURRENT ROW` frame boundaries now respected by `FIRST_VALUE` and `LAST_VALUE`.
- **Session variables** — `SET var_name = value` and `current_setting('var_name')` for connection-scoped key/value storage.
- **Current user/role** — `current_user` and `current_role` SQL keywords evaluate to the authenticated session's user and role.
- **Auth-executor bridge** — Wire server and HTTP server now populate `ExecutionContext.currentUser` and `ExecutionContext.currentRole` after JWT/SCRAM authentication.
- **Multi-tenant RLS** — Row-Level Security policies can now reference `current_user`, `current_role`, and `current_setting('app.tenant_id')` for per-tenant data isolation.
## [1.1.0] — 2026-05-13
### Added
- **Client SDKs v1.1.0** — Full-featured clients for all languages:
- JavaScript: TypeScript definitions, package.json, examples, unit & integration tests
- Python: Restructured as proper package (`baradb/` with `__init__.py` and `core.py`), pyproject.toml, examples, tests (query builder, wire protocol, integration)
- Nim: Examples, integration tests, README
- Rust: Examples, integration tests, improved Cargo.toml
- **SCRAM-SHA-256 Authentication** — RFC 7677 compliant authentication with PBKDF2 + HMAC + SHA-256 + nonce/salt generation
- **HTTP SCRAM Endpoints** — `/auth/scram/start` + `/auth/scram/finish` in HTTP server
- **Docker Compose Test Configuration** — `docker-compose.test.yml` for test environments
- **CI/CD Clients Pipeline** — `.github/workflows/clients-ci.yml` for automated client testing
### Fixed
- **Query Executor** — Unary minus (`irNeg`) evaluation now works correctly in SELECT and WHERE clauses
- **Distributed Transactions** — Rollback after commit attempt no longer violates atomicity
- **Sharding** — Data migration protocol with TCP + `scanAll` on LSM
- **Raft** — Majority calculation for even number of nodes fixed
- **MVCC** — Aborted transactions no longer become visible
- **LSM-Tree** — Data loss on immutable memtable overwrite fixed; SSTable lookup sorting fixed
- **Auth** — JWT signature changed to HMAC-SHA256 (no longer trivially forgeable); token expiration (`exp`/`nbf`/`iat`) now validated; signature comparison is now constant-time
- **Recovery** — `summary()` no longer mutates the database
- **Wire Protocol** — 64MB limit + bounds checking + max depth to prevent OOM/DoS
- **SQL Injection** — `exprToSql` now escapes single quotes
- **ReDoS** — `irLike`/`irILike` now escape regex metacharacters
- **Graph** — `addEdge` now checks node existence
- **Vector** — Dimension mismatch validation + HNSW locking
- **FTS** — UTF-8 tokenization now uses runes instead of bytes
- **Build** — `nim.cfg` adds `-d:ssl` so `nimble build` works without flags; `--threads:on` added to all CI commands
### Changed
- **Version bumped to 1.1.0** across all components (server, Docker images, clients, CLI)
- **README** — Version badge updated; all feature tables now reference v1.1.0
- **TLA+ Formal Verification** — Added `crossmodal.tla`, `backup.tla`, `recovery.tla`; symmetry reduction in all 9 specs
- **Clean build** — 0 compiler warnings on Nim 2.2.10
## [0.1.0] — 2025-01-15 ## [0.1.0] — 2025-01-15
### Added ### Added
@@ -135,6 +194,25 @@ All notable changes to BaraDB are documented in this file.
## [Unreleased] ## [Unreleased]
### Added
- **Vector SQL Integration** — Full SQL-level vector search support:
- `VECTOR(n)` column type in `CREATE TABLE` with dimension validation
- `CREATE INDEX ... USING hnsw` / `USING ivfpq` for approximate nearest neighbor indexes
- SQL distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- `<->` nearest-neighbor operator (Euclidean distance)
- `ORDER BY` support for vector distance expressions, including columns not in `SELECT`
- Automatic HNSW index maintenance on `INSERT` and `UPDATE`
- **Advanced SQL Engine** — Window functions, MERGE/UPSERT, LATERAL JOIN, PIVOT/UNPIVOT, SQL/PGQ Property Graph, Advanced Aggregates (ARRAY_AGG, STRING_AGG, FILTER, GROUPING SETS/ROLLUP/CUBE)
- **JavaScript Client — TCP Request Queue** — Internal `_requestQueue` + `_requestLock` for safe concurrent queries. Multiple parallel `query()` / `execute()` / `ping()` calls no longer interleave binary frames on the wire.
### Fixed
- **Query Executor — Row Value Escaping** — `execInsert` now properly escapes commas and equals signs in column values, fixing storage corruption for vector literals like `[1.0, 2.0, 3.0]`
- **Query Planner — ORDER BY Projection** — `irpkSort` is now placed before `irpkProject` in the IR plan, allowing `ORDER BY` to reference columns that are not selected
- **Wire Protocol — Big-Endian Float Serialization** — `FLOAT32`/`FLOAT64` and vector float values are now serialized in big-endian byte order, matching the client's `readFloatBE()` / `readDoubleBE()` and ensuring cross-platform numeric accuracy.
- **Gossip Protocol — Async UDP Socket** — Replaced synchronous `newSocket` + blocking `recvFrom` with `newAsyncSocket` + `await recvFrom`, preventing the async event loop from freezing until a UDP packet arrives.
### Planned ### Planned
- Query plan caching - Query plan caching
+12
View File
@@ -45,6 +45,18 @@ await client.commit();
await client.close(); await client.close();
``` ```
### Concurrent Queries
The JavaScript client automatically serializes concurrent requests over a single TCP connection via an internal request queue. You can safely fire multiple parallel operations — their binary frames will not interleave on the wire:
```typescript
const [users, orders, stats] = await Promise.all([
client.query('SELECT * FROM users'),
client.query('SELECT * FROM orders'),
client.query('SELECT count(*) FROM visits')
]);
```
### WebSocket Streaming ### WebSocket Streaming
```typescript ```typescript
+3 -3
View File
@@ -98,13 +98,13 @@ Every message starts with a 8-byte header:
| INT16 | 0x03 | 2 | Signed 16-bit integer | | INT16 | 0x03 | 2 | Signed 16-bit integer |
| INT32 | 0x04 | 4 | Signed 32-bit integer | | INT32 | 0x04 | 4 | Signed 32-bit integer |
| INT64 | 0x05 | 8 | Signed 64-bit integer | | INT64 | 0x05 | 8 | Signed 64-bit integer |
| FLOAT32 | 0x06 | 4 | IEEE 754 single precision | | FLOAT32 | 0x06 | 4 | IEEE 754 single precision (big-endian) |
| FLOAT64 | 0x07 | 8 | IEEE 754 double precision | | FLOAT64 | 0x07 | 8 | IEEE 754 double precision (big-endian) |
| STRING | 0x08 | variable | UTF-8 string (4-byte length prefix) | | STRING | 0x08 | variable | UTF-8 string (4-byte length prefix) |
| BYTES | 0x09 | variable | Raw bytes (4-byte length prefix) | | BYTES | 0x09 | variable | Raw bytes (4-byte length prefix) |
| ARRAY | 0x0A | variable | Array of values | | ARRAY | 0x0A | variable | Array of values |
| OBJECT | 0x0B | variable | Key-value object | | OBJECT | 0x0B | variable | Key-value object |
| VECTOR | 0x0C | variable | Float32 array (4-byte length prefix) | | VECTOR | 0x0C | variable | Float32 array (4-byte length prefix, big-endian floats) |
### Error Message Payload ### Error Message Payload
+89 -8
View File
@@ -1,8 +1,89 @@
# Vector Search Engine # Vector Search Engine
Native HNSW and IVF-PQ indexes for similarity search. Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
## Usage ## SQL Usage
### Creating Vector Columns
```sql
CREATE TABLE items (
id INT PRIMARY KEY,
embedding VECTOR(768)
);
```
The `VECTOR(n)` type stores float32 arrays of fixed dimension `n`.
### Inserting Vectors
```sql
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
```
### Vector Distance Functions
```sql
-- Cosine distance (0 = identical, 1 = orthogonal)
SELECT id, cosine_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- Euclidean / L2 distance
SELECT id, euclidean_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- L2 distance with <-> operator
SELECT id, embedding <-> '[0.1, 0.2, 0.3]' AS dist
FROM items;
-- Inner product (negative dot product for minimization)
SELECT id, inner_product(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
-- Manhattan / L1 distance
SELECT id, l1_distance(embedding, '[0.1, 0.2, 0.3]') AS dist
FROM items;
```
### Nearest Neighbor Search
```sql
-- Top-10 nearest neighbors by cosine distance
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3]') ASC
LIMIT 10;
-- Top-5 nearest neighbors by Euclidean distance
SELECT id FROM items
ORDER BY embedding <-> '[0.1, 0.2, 0.3]'
LIMIT 5;
```
### Vector Indexes
```sql
-- Create HNSW index for approximate nearest neighbor search
CREATE INDEX idx_items_vec ON items(embedding) USING hnsw;
-- The index is automatically maintained on INSERT and UPDATE
```
Supported index methods:
- `USING hnsw` — Hierarchical Navigable Small World (default: cosine metric)
- `USING ivfpq` — Inverted File with Product Quantization
### Dimension Validation
BaraDB validates vector dimensions at insert time:
```sql
-- This will fail: expected 768 dimensions but got 3
INSERT INTO items (id, embedding) VALUES (2, '[1.0, 2.0, 3.0]');
```
## Native Nim API
For embedded or high-performance use, use the native Nim API directly:
```nim ```nim
import barabadb/vector/engine import barabadb/vector/engine
@@ -48,12 +129,12 @@ var ivfpq = newIVFPQIndex(
## Distance Metrics ## Distance Metrics
| Metric | Description | | Metric | SQL Function | Description |
|--------|-------------| |--------|--------------|-------------|
| `cosine` | Cosine similarity | | `cosine` | `cosine_distance(a, b)` | Cosine dissimilarity (1 - similarity) |
| `euclidean` | L2 distance | | `euclidean` | `euclidean_distance(a, b)` / `<->` | L2 distance |
| `dotproduct` | Dot product similarity | | `dotproduct` | `inner_product(a, b)` | Negative dot product |
| `manhattan` | L1 distance | | `manhattan` | `l1_distance(a, b)` | L1 distance |
## Quantization ## Quantization
+1 -1
View File
@@ -5,7 +5,7 @@ import ../query/lexer
import ../query/parser import ../query/parser
const const
Version = "0.1.0" Version = "1.1.0"
Prompt = "bara> " Prompt = "bara> "
ContinuationPrompt = " .. > " ContinuationPrompt = " .. > "
+6 -9
View File
@@ -4,6 +4,7 @@ import std/random
import std/monotimes import std/monotimes
import std/asyncdispatch import std/asyncdispatch
import std/net import std/net
import std/asyncnet
import std/strutils import std/strutils
import std/streams import std/streams
import std/os import std/os
@@ -36,7 +37,7 @@ type
fanout*: int # number of nodes to gossip to per round fanout*: int # number of nodes to gossip to per round
gossipPort*: int gossipPort*: int
running*: bool running*: bool
sock*: Socket sock*: AsyncSocket
onJoin*: proc(node: GossipNode) {.gcsafe.} onJoin*: proc(node: GossipNode) {.gcsafe.}
onLeave*: proc(nodeId: string) {.gcsafe.} onLeave*: proc(nodeId: string) {.gcsafe.}
onSuspect*: proc(nodeId: string) {.gcsafe.} onSuspect*: proc(nodeId: string) {.gcsafe.}
@@ -305,25 +306,21 @@ proc startGossipRound*(gp: GossipProtocol, intervalMs: int = 2000) {.async.} =
gp.broadcastGossip() gp.broadcastGossip()
proc startGossipListener*(gp: GossipProtocol) {.async.} = proc startGossipListener*(gp: GossipProtocol) {.async.} =
gp.sock = newSocket(AF_INET, SOCK_DGRAM, IPPROTO_UDP) gp.sock = newAsyncSocket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)
gp.sock.setSockOpt(OptReuseAddr, true) gp.sock.setSockOpt(OptReuseAddr, true)
gp.sock.bindAddr(Port(gp.gossipPort)) gp.sock.bindAddr(Port(gp.gossipPort))
gp.running = true gp.running = true
while gp.running: while gp.running:
try: try:
var data = newString(65535) let (data, senderAddr, senderPort) = await gp.sock.recvFrom(65535)
var senderAddr = "" if data.len > 0:
var senderPort: Port
let bytesRead = gp.sock.recvFrom(data, 65535, senderAddr, senderPort)
if bytesRead > 0:
data.setLen(bytesRead)
gp.handleIncomingGossip(data, senderAddr) gp.handleIncomingGossip(data, senderAddr)
except: except:
# Small sleep on error to avoid spin # Small sleep on error to avoid spin
gp.sock.close() gp.sock.close()
# Recreate socket for next iteration # Recreate socket for next iteration
try: try:
gp.sock = newSocket(AF_INET, SOCK_DGRAM, IPPROTO_UDP) gp.sock = newAsyncSocket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)
gp.sock.setSockOpt(OptReuseAddr, true) gp.sock.setSockOpt(OptReuseAddr, true)
gp.sock.bindAddr(Port(gp.gossipPort)) gp.sock.bindAddr(Port(gp.gossipPort))
except: except:
+9 -3
View File
@@ -108,6 +108,8 @@ proc queryHandler(server: HttpServer): RequestHandler =
let ctx = newContext(request) let ctx = newContext(request)
server.metrics.queriesTotal += 1 server.metrics.queriesTotal += 1
var userId = ""
var role = ""
# Auth check # Auth check
if server.config.authEnabled: if server.config.authEnabled:
let authHeader = request.headers["Authorization"] let authHeader = request.headers["Authorization"]
@@ -115,10 +117,12 @@ proc queryHandler(server: HttpServer): RequestHandler =
ctx.json(%*{"error": "Unauthorized"}, 401) ctx.json(%*{"error": "Unauthorized"}, 401)
return return
let tokenStr = authHeader[7..^1] let tokenStr = authHeader[7..^1]
let (valid, userId, role) = server.verifyToken(tokenStr) let (valid, uid, r) = server.verifyToken(tokenStr)
if not valid: if not valid:
ctx.json(%*{"error": "Unauthorized"}, 401) ctx.json(%*{"error": "Unauthorized"}, 401)
return return
userId = uid
role = r
let body = parseJson(request.body) let body = parseJson(request.body)
if body == nil or "query" notin body: if body == nil or "query" notin body:
@@ -131,6 +135,8 @@ proc queryHandler(server: HttpServer): RequestHandler =
return return
var reqCtx = cloneForConnection(server.ctx) var reqCtx = cloneForConnection(server.ctx)
reqCtx.currentUser = userId
reqCtx.currentRole = role
let tokens = tokenize(queryStr) let tokens = tokenize(queryStr)
let astNode = parse(tokens) let astNode = parse(tokens)
@@ -178,7 +184,7 @@ proc queryHandler(server: HttpServer): RequestHandler =
proc healthHandler(): RequestHandler = proc healthHandler(): RequestHandler =
return proc(request: Request) {.gcsafe.} = return proc(request: Request) {.gcsafe.} =
let ctx = newContext(request) let ctx = newContext(request)
ctx.json(%*{"status": "ok", "version": "0.1.0"}) ctx.json(%*{"status": "ok", "version": "1.1.0"})
proc metricsHandler(server: HttpServer): RequestHandler = proc metricsHandler(server: HttpServer): RequestHandler =
return proc(request: Request) {.gcsafe.} = return proc(request: Request) {.gcsafe.} =
@@ -514,7 +520,7 @@ function showTab(idx){
} }
setInterval(() => { if(document.querySelectorAll('.panel')[4].classList.contains('active')) loadMetrics() }, 5000) setInterval(() => { if(document.querySelectorAll('.panel')[4].classList.contains('active')) loadMetrics() }, 5000)
</script> </script>
<div class='status' style='text-align:center;padding:10px'>BaraDB v1.0.0 — Multimodal Database Engine</div> <div class='status' style='text-align:center;padding:10px'>BaraDB v1.1.0 — Multimodal Database Engine</div>
</body></html>""" </body></html>"""
request.respond(200, @[("Content-Type", "text/html; charset=utf-8")], html) request.respond(200, @[("Content-Type", "text/html; charset=utf-8")], html)
+3 -1
View File
@@ -436,9 +436,11 @@ proc handleClient(server: Server, client: AsyncSocket, clientId: int) {.async.}
case header.kind case header.kind
of mkAuth: of mkAuth:
let tokenStr = parseAuthMessage(cast[seq[byte]](payload)) let tokenStr = parseAuthMessage(cast[seq[byte]](payload))
let (valid, userId, _) = verifyToken(secret, tokenStr) let (valid, userId, role) = verifyToken(secret, tokenStr)
if valid: if valid:
authenticated = true authenticated = true
connCtx.currentUser = userId
connCtx.currentRole = role
let okMsg = makeAuthOkMessage(header.requestId) let okMsg = makeAuthOkMessage(header.requestId)
await client.send(cast[string](okMsg)) await client.send(cast[string](okMsg))
info("Client " & $clientId & " authenticated as " & userId) info("Client " & $clientId & " authenticated as " & userId)
+1 -1
View File
@@ -116,7 +116,7 @@ proc exportOtlp*(tracer: Tracer, endpoint: string = "http://localhost:4318/v1/tr
{"key": "service.name", "value": {"stringValue": "baradadb"}} {"key": "service.name", "value": {"stringValue": "baradadb"}}
]}, ]},
"scopeSpans": [{ "scopeSpans": [{
"scope": {"name": "baradadb-tracer", "version": "0.1.0"}, "scope": {"name": "baradadb-tracer", "version": "1.1.0"},
"spans": otlpSpans "spans": otlpSpans
}] }]
}] }]
+15 -6
View File
@@ -169,12 +169,14 @@ proc serializeValue*(buf: var seq[byte], val: WireValue) =
of fkFloat32: of fkFloat32:
var fl = val.float32Val var fl = val.float32Val
var bytes: array[4, byte] var bytes: array[4, byte]
copyMem(addr bytes, unsafeAddr fl, 4) var i32 = cast[int32](fl)
bigEndian32(addr bytes, unsafeAddr i32)
buf.add(bytes) buf.add(bytes)
of fkFloat64: of fkFloat64:
var fl = val.float64Val var fl = val.float64Val
var bytes: array[8, byte] var bytes: array[8, byte]
copyMem(addr bytes, unsafeAddr fl, 8) var i64 = cast[int64](fl)
bigEndian64(addr bytes, unsafeAddr i64)
buf.add(bytes) buf.add(bytes)
of fkString: buf.writeString(val.strVal) of fkString: buf.writeString(val.strVal)
of fkBytes: buf.writeBytes(val.bytesVal) of fkBytes: buf.writeBytes(val.bytesVal)
@@ -192,7 +194,8 @@ proc serializeValue*(buf: var seq[byte], val: WireValue) =
for f in val.vecVal: for f in val.vecVal:
var fl = f var fl = f
var bytes: array[4, byte] var bytes: array[4, byte]
copyMem(addr bytes, unsafeAddr fl, 4) var i32 = cast[int32](fl)
bigEndian32(addr bytes, unsafeAddr i32)
buf.add(bytes) buf.add(bytes)
of fkJson: of fkJson:
buf.writeString(val.jsonVal) buf.writeString(val.jsonVal)
@@ -227,14 +230,18 @@ proc deserializeValue*(buf: openArray[byte], pos: var int, depth: int = 0): Wire
var fl: float32 var fl: float32
var bytes: array[4, byte] var bytes: array[4, byte]
for i in 0..3: bytes[i] = buf[pos + i] for i in 0..3: bytes[i] = buf[pos + i]
copyMem(addr fl, unsafeAddr bytes, 4) var i32: int32
bigEndian32(addr i32, unsafeAddr bytes)
fl = cast[float32](i32)
pos += 4 pos += 4
result = WireValue(kind: fkFloat32, float32Val: fl) result = WireValue(kind: fkFloat32, float32Val: fl)
of fkFloat64: of fkFloat64:
var fl: float64 var fl: float64
var bytes: array[8, byte] var bytes: array[8, byte]
for i in 0..7: bytes[i] = buf[pos + i] for i in 0..7: bytes[i] = buf[pos + i]
copyMem(addr fl, unsafeAddr bytes, 8) var i64: int64
bigEndian64(addr i64, unsafeAddr bytes)
fl = cast[float64](i64)
pos += 8 pos += 8
result = WireValue(kind: fkFloat64, float64Val: fl) result = WireValue(kind: fkFloat64, float64Val: fl)
of fkString: of fkString:
@@ -270,7 +277,9 @@ proc deserializeValue*(buf: openArray[byte], pos: var int, depth: int = 0): Wire
var fl: float32 var fl: float32
var bytes: array[4, byte] var bytes: array[4, byte]
for j in 0..3: bytes[j] = buf[pos + j] for j in 0..3: bytes[j] = buf[pos + j]
copyMem(addr fl, unsafeAddr bytes, 4) var i32: int32
bigEndian32(addr i32, unsafeAddr bytes)
fl = cast[float32](i32)
pos += 4 pos += 4
vec.add(fl) vec.add(fl)
result = WireValue(kind: fkVector, vecVal: vec) result = WireValue(kind: fkVector, vecVal: vec)
+82
View File
@@ -8,6 +8,7 @@ type
nkInsert nkInsert
nkUpdate nkUpdate
nkDelete nkDelete
nkMerge
nkCreateType nkCreateType
nkDropType nkDropType
nkAlterType nkAlterType
@@ -39,6 +40,7 @@ type
nkDisableRLS nkDisableRLS
nkGrant nkGrant
nkRevoke nkRevoke
nkSetVar
# Clauses # Clauses
nkFrom nkFrom
@@ -77,6 +79,8 @@ type
nkIsExpr nkIsExpr
nkStar nkStar
nkPlaceholder nkPlaceholder
nkCurrentUser
nkCurrentRole
# Graph-specific # Graph-specific
nkGraphTraversal nkGraphTraversal
@@ -89,6 +93,10 @@ type
nkVectorSimilar nkVectorSimilar
nkVectorNearest nkVectorNearest
# Pivot
nkPivot
nkUnpivot
# Set operations # Set operations
nkSetOp nkSetOp
@@ -102,6 +110,12 @@ type
nkColumnDef nkColumnDef
nkConstraintDef nkConstraintDef
# Window functions
nkWindowExpr
nkOverClause
nkFrameSpec
nkFrameBoundary
# Top-level # Top-level
nkStatementList nkStatementList
@@ -132,6 +146,11 @@ type
bkJsonPath = "->" bkJsonPath = "->"
bkJsonPathText = "->>" bkJsonPathText = "->>"
bkFtsMatch = "@@" bkFtsMatch = "@@"
bkDistance = "<->"
bkJsonContains = "@>"
bkJsonContainedBy = "<@"
bkJsonHasAny = "?|"
bkJsonHasAll = "?&"
UnaryOpKind* = enum UnaryOpKind* = enum
ukNeg = "-" ukNeg = "-"
@@ -151,6 +170,12 @@ type
sdkIntersect sdkIntersect
sdkExcept sdkExcept
GroupingSetsKind* = enum
gskNone
gskGroupingSets
gskRollup
gskCube
SortDir* = enum SortDir* = enum
sdAsc sdAsc
sdDesc sdDesc
@@ -168,6 +193,8 @@ type
selJoins*: seq[Node] selJoins*: seq[Node]
selWhere*: Node selWhere*: Node
selGroupBy*: seq[Node] selGroupBy*: seq[Node]
selGroupingSetsKind*: GroupingSetsKind
selGroupingSets*: seq[seq[Node]]
selHaving*: Node selHaving*: Node
selOrderBy*: seq[Node] selOrderBy*: seq[Node]
selLimit*: Node selLimit*: Node
@@ -189,6 +216,15 @@ type
delAlias*: string delAlias*: string
delWhere*: Node delWhere*: Node
delReturning*: seq[Node] delReturning*: seq[Node]
of nkMerge:
mergeTarget*: string
mergeTargetAlias*: string
mergeSource*: Node
mergeSourceAlias*: string
mergeOn*: Node
mergeMatchedUpdate*: seq[Node] # SET assignments, empty if no UPDATE
mergeNotMatchedInsert*: seq[Node] # column list for INSERT
mergeNotMatchedValues*: seq[Node] # value expressions for INSERT
of nkCreateType: of nkCreateType:
ctName*: string ctName*: string
ctBases*: seq[string] ctBases*: seq[string]
@@ -288,6 +324,9 @@ type
rvPrivilege*: string rvPrivilege*: string
rvTable*: string rvTable*: string
rvGrantee*: string rvGrantee*: string
of nkSetVar:
svName*: string
svValue*: string
of nkApplyMigration: of nkApplyMigration:
amName*: string amName*: string
of nkMigrationStatus: of nkMigrationStatus:
@@ -309,6 +348,7 @@ type
of nkFrom: of nkFrom:
fromTable*: string fromTable*: string
fromAlias*: string fromAlias*: string
fromSubquery*: Node
of nkWhere: of nkWhere:
whereExpr*: Node whereExpr*: Node
of nkOrderBy: of nkOrderBy:
@@ -340,6 +380,7 @@ type
of nkFuncCall: of nkFuncCall:
funcName*: string funcName*: string
funcArgs*: seq[Node] funcArgs*: seq[Node]
funcFilter*: Node
of nkTypeCast: of nkTypeCast:
castType*: string castType*: string
castExpr*: Node castExpr*: Node
@@ -391,11 +432,13 @@ type
isType*: string isType*: string
isNegated*: bool isNegated*: bool
of nkGraphTraversal: of nkGraphTraversal:
gtGraphName*: string
gtStart*: Node gtStart*: Node
gtEdge*: string gtEdge*: string
gtDirection*: string gtDirection*: string
gtEnd*: Node gtEnd*: Node
gtMaxDepth*: int gtMaxDepth*: int
gtReturnCols*: seq[string]
of nkBfsQuery: of nkBfsQuery:
bfsStart*: Node bfsStart*: Node
bfsTarget*: Node bfsTarget*: Node
@@ -426,11 +469,22 @@ type
vnVector*: Node vnVector*: Node
vnLimit*: int vnLimit*: int
vnMetric*: string vnMetric*: string
of nkPivot:
pivotSource*: Node
pivotAgg*: Node
pivotForCol*: string
pivotInValues*: seq[string]
of nkUnpivot:
unpivotSource*: Node
unpivotValueCol*: string
unpivotForCol*: string
unpivotInCols*: seq[string]
of nkJoin: of nkJoin:
joinKind*: JoinKind joinKind*: JoinKind
joinTarget*: Node joinTarget*: Node
joinOn*: Node joinOn*: Node
joinAlias*: string joinAlias*: string
joinLateral*: bool
of nkSetOp: of nkSetOp:
setOpKind*: SetOpKind setOpKind*: SetOpKind
setOpAll*: bool setOpAll*: bool
@@ -440,6 +494,10 @@ type
discard discard
of nkPlaceholder: of nkPlaceholder:
discard discard
of nkCurrentUser:
discard
of nkCurrentRole:
discard
of nkPropertyDef: of nkPropertyDef:
pdName*: string pdName*: string
pdType*: string pdType*: string
@@ -456,6 +514,24 @@ type
idName*: string idName*: string
idExpr*: Node idExpr*: Node
idKind*: IndexKind idKind*: IndexKind
of nkWindowExpr:
winFunc*: string
winArgs*: seq[Node]
winOver*: Node
of nkOverClause:
overPartition*: seq[Node]
overOrderBy*: seq[Node]
overFrame*: Node
of nkFrameSpec:
frameMode*: string # "ROWS" or "RANGE"
frameStartType*: string # "UNBOUNDED PRECEDING", "CURRENT ROW", "EXPR PRECEDING", "EXPR FOLLOWING"
frameStartExpr*: Node
frameEndType*: string
frameEndExpr*: Node
frameExclude*: string # "NO OTHERS", "CURRENT ROW", "GROUP", "TIES"
of nkFrameBoundary:
fbType*: string # "UNBOUNDED PRECEDING", "CURRENT ROW", "EXPR PRECEDING", "EXPR FOLLOWING", "UNBOUNDED FOLLOWING"
fbExpr*: Node
of nkStatementList: of nkStatementList:
stmts*: seq[Node] stmts*: seq[Node]
@@ -470,5 +546,11 @@ proc newNode*(kind: NodeKind, line: int = 0, col: int = 0): Node =
of nkAlterTable: result.altOps = @[] of nkAlterTable: result.altOps = @[]
of nkColumnDef: result.cdConstraints = @[] of nkColumnDef: result.cdConstraints = @[]
of nkConstraintDef: result.cstColumns = @[]; result.cstRefColumns = @[] of nkConstraintDef: result.cstColumns = @[]; result.cstRefColumns = @[]
of nkWindowExpr: result.winArgs = @[]
of nkOverClause: result.overPartition = @[]; result.overOrderBy = @[]
of nkMerge:
result.mergeMatchedUpdate = @[]
result.mergeNotMatchedInsert = @[]
result.mergeNotMatchedValues = @[]
of nkStatementList: result.stmts = @[] of nkStatementList: result.stmts = @[]
else: discard else: discard
+22
View File
@@ -194,6 +194,28 @@ proc codegenPlan*(plan: IRPlan): StorageOp =
of irpkExplain: of irpkExplain:
return codegenPlan(plan.explainPlan) return codegenPlan(plan.explainPlan)
of irpkWindow:
let sourceOp = codegenPlan(plan.winSource)
let op = newStorageOp(sokProject)
for wf in plan.winFuncs:
op.columns.add(wf.wfName)
if sourceOp != nil:
op.children.add(sourceOp)
return op
of irpkMerge:
return newStorageOp(sokScan)
of irpkPivot:
let sourceOp = codegenPlan(plan.pivotSource)
let op = newStorageOp(sokScan)
if sourceOp != nil: op.children.add(sourceOp)
return op
of irpkUnpivot:
let sourceOp = codegenPlan(plan.unpivotSource)
let op = newStorageOp(sokScan)
if sourceOp != nil: op.children.add(sourceOp)
return op
of irpkGraphTraversal:
return newStorageOp(sokScan)
proc estimateCost*(op: StorageOp): float64 = proc estimateCost*(op: StorageOp): float64 =
if op == nil: if op == nil:
File diff suppressed because it is too large Load Diff
+83
View File
@@ -1,5 +1,6 @@
## BaraQL IR — Intermediate Representation for compilation ## BaraQL IR — Intermediate Representation for compilation
import std/tables import std/tables
import std/strutils
import ../core/types import ../core/types
type type
@@ -27,9 +28,15 @@ type
irBetween irBetween
irIsNull, irIsNotNull irIsNull, irIsNotNull
irFtsMatch irFtsMatch
irDistance
irJsonContains
irJsonContainedBy
irJsonHasAny
irJsonHasAll
IRAggregate* = enum IRAggregate* = enum
irCount, irSum, irAvg, irMin, irMax irCount, irSum, irAvg, irMin, irMax
irArrayAgg, irStringAgg
IRLiteral* = object IRLiteral* = object
case kind*: ValueKind case kind*: ValueKind
@@ -52,6 +59,7 @@ type
irekExists irekExists
irekStar irekStar
irekJsonPath irekJsonPath
irekWindowFunc
IRJoinKind* = enum IRJoinKind* = enum
irjkInner irjkInner
@@ -71,11 +79,22 @@ type
irpkInsert irpkInsert
irpkUpdate irpkUpdate
irpkDelete irpkDelete
irpkMerge
irpkCreateType irpkCreateType
irpkUnion irpkUnion
irpkCTE irpkCTE
irpkValues irpkValues
irpkExplain irpkExplain
irpkWindow
irpkPivot
irpkUnpivot
irpkGraphTraversal
IRGroupingSetsKind* = enum
irgskNone
irgskGroupingSets
irgskRollup
irgskCube
IRPlan* = ref object IRPlan* = ref object
case kind*: IRPlanKind case kind*: IRPlanKind
@@ -94,12 +113,15 @@ type
groupKeys*: seq[IRExpr] groupKeys*: seq[IRExpr]
groupAggs*: seq[IRExpr] groupAggs*: seq[IRExpr]
groupHaving*: IRExpr groupHaving*: IRExpr
groupingSetsKind*: IRGroupingSetsKind
groupingSets*: seq[seq[IRExpr]]
of irpkJoin: of irpkJoin:
joinKind*: IRJoinKind joinKind*: IRJoinKind
joinLeft*: IRPlan joinLeft*: IRPlan
joinRight*: IRPlan joinRight*: IRPlan
joinCond*: IRExpr joinCond*: IRExpr
joinAlias*: string joinAlias*: string
joinLateral*: bool
of irpkSort: of irpkSort:
sortSource*: IRPlan sortSource*: IRPlan
sortExprs*: seq[IRExpr] sortExprs*: seq[IRExpr]
@@ -121,6 +143,14 @@ type
deleteTable*: string deleteTable*: string
deleteAlias*: string deleteAlias*: string
deleteSource*: IRPlan deleteSource*: IRPlan
of irpkMerge:
mergeTarget*: string
mergeTargetAlias*: string
mergeSourcePlan*: IRPlan
mergeOnCond*: IRExpr
mergeUpdateSets*: seq[(string, IRExpr)]
mergeInsertFields*: seq[string]
mergeInsertValues*: seq[seq[IRExpr]]
of irpkCreateType: of irpkCreateType:
createTypeName*: string createTypeName*: string
createTypeDef*: IRType createTypeDef*: IRType
@@ -136,6 +166,34 @@ type
valuesRows*: seq[seq[IRExpr]] valuesRows*: seq[seq[IRExpr]]
of irpkExplain: of irpkExplain:
explainPlan*: IRPlan explainPlan*: IRPlan
of irpkWindow:
winSource*: IRPlan
winFuncs*: seq[IRExpr]
winPartition*: seq[IRExpr]
winOrderBy*: seq[IRExpr]
winOrderDirs*: seq[bool]
winFrameMode*: string
winFrameStart*: string
winFrameEnd*: string
of irpkPivot:
pivotSource*: IRPlan
pivotAgg*: IRExpr
pivotForCol*: string
pivotInValues*: seq[string]
of irpkUnpivot:
unpivotSource*: IRPlan
unpivotValueCol*: string
unpivotForCol*: string
unpivotInCols*: seq[string]
of irpkGraphTraversal:
graphName*: string
graphAlgo*: string # "bfs", "dfs", "shortest", "pagerank"
graphStartNode*: string
graphEndNode*: string
graphEdgeLabel*: string
graphMaxDepth*: int
graphFilter*: IRExpr
graphReturnCols*: seq[string]
IRExpr* = ref object IRExpr* = ref object
case kind*: IRExprKind case kind*: IRExprKind
@@ -154,6 +212,7 @@ type
aggOp*: IRAggregate aggOp*: IRAggregate
aggArgs*: seq[IRExpr] aggArgs*: seq[IRExpr]
aggDistinct*: bool aggDistinct*: bool
aggFilter*: IRExpr
of irekFuncCall: of irekFuncCall:
irFunc*: string irFunc*: string
irFuncArgs*: seq[IRExpr] irFuncArgs*: seq[IRExpr]
@@ -172,6 +231,15 @@ type
jpExpr*: IRExpr jpExpr*: IRExpr
jpKey*: string jpKey*: string
jpAsText*: bool jpAsText*: bool
of irekWindowFunc:
wfName*: string
wfArgs*: seq[IRExpr]
wfPartition*: seq[IRExpr]
wfOrderBy*: seq[IRExpr]
wfOrderDirs*: seq[bool]
wfFrameMode*: string
wfFrameStart*: string
wfFrameEnd*: string
type type
TypeChecker* = ref object TypeChecker* = ref object
@@ -241,6 +309,10 @@ proc inferExpr*(tc: TypeChecker, expr: IRExpr, context: Table[string, IRType]):
if expr.aggArgs.len > 0: if expr.aggArgs.len > 0:
return tc.inferExpr(expr.aggArgs[0], context) return tc.inferExpr(expr.aggArgs[0], context)
return nil return nil
of irArrayAgg:
return IRType(name: "array", kind: itkArray)
of irStringAgg:
return IRType(name: "text", kind: itkScalar)
of irekFuncCall: of irekFuncCall:
return IRType(name: "unknown", kind: itkScalar) return IRType(name: "unknown", kind: itkScalar)
of irekCast: of irekCast:
@@ -255,3 +327,14 @@ proc inferExpr*(tc: TypeChecker, expr: IRExpr, context: Table[string, IRType]):
of irekJsonPath: of irekJsonPath:
if expr.jpAsText: return IRType(name: "str", kind: itkScalar) if expr.jpAsText: return IRType(name: "str", kind: itkScalar)
return IRType(name: "json", kind: itkScalar) return IRType(name: "json", kind: itkScalar)
of irekWindowFunc:
# Window functions return int64 for ranking, or the type of the argument for value functions
case expr.wfName.toLower()
of "row_number", "rank", "dense_rank", "ntile":
return IRType(name: "int64", kind: itkScalar)
of "lead", "lag", "first_value", "last_value":
if expr.wfArgs.len > 0:
return tc.inferExpr(expr.wfArgs[0], context)
return nil
else:
return IRType(name: "unknown", kind: itkScalar)
+111
View File
@@ -35,6 +35,7 @@ type
tkOuter tkOuter
tkFull tkFull
tkCross tkCross
tkLateral
tkOrder tkOrder
tkBy tkBy
tkAsc tkAsc
@@ -77,6 +78,7 @@ type
tkBetween tkBetween
tkLike tkLike
tkILike tkILike
tkFilter
tkReturning tkReturning
tkPrimary tkPrimary
tkKey tkKey
@@ -120,6 +122,24 @@ type
tkAvg tkAvg
tkMin tkMin
tkMax tkMax
tkArrayAgg
tkStringAgg
tkGrouping
tkSets
tkRollup
tkCube
tkPivot
tkUnpivot
tkVertex
tkEdge
tkLabels
tkGraphTable
tkMatch
tkColumns
tkSrc
tkDst
tkMerge
tkMatched
tkArray tkArray
tkVector tkVector
tkGraph tkGraph
@@ -127,6 +147,10 @@ type
tkArrowR # -> tkArrowR # ->
tkArrowRR # ->> tkArrowRR # ->>
tkFtsMatch # @@ tkFtsMatch # @@
tkJsonContains # @>
tkJsonContainedBy # <@
tkJsonHasAny # ?|
tkJsonHasAll # ?&
tkSimilar tkSimilar
tkNearest tkNearest
tkTo tkTo
@@ -134,6 +158,27 @@ type
tkDfs tkDfs
tkPath tkPath
# Window functions
tkOver
tkPartition
tkRow
tkRows
tkRange
tkUnbounded
tkPreceding
tkFollowing
tkCurrent
tkCurrentUser
tkCurrentRole
tkRowNumber
tkRank
tkDenseRank
tkLead
tkLag
tkFirstValue
tkLastValue
tkNtile
# Operators # Operators
tkPlus tkPlus
tkMinus tkMinus
@@ -166,6 +211,7 @@ type
tkConcat tkConcat
tkCoalesce tkCoalesce
tkFloorDiv tkFloorDiv
tkDistanceOp # <->
tkPlaceholder tkPlaceholder
# Special # Special
@@ -206,6 +252,7 @@ const keywords*: Table[string, TokenKind] = {
"outer": tkOuter, "outer": tkOuter,
"full": tkFull, "full": tkFull,
"cross": tkCross, "cross": tkCross,
"lateral": tkLateral,
"order": tkOrder, "order": tkOrder,
"by": tkBy, "by": tkBy,
"asc": tkAsc, "asc": tkAsc,
@@ -248,6 +295,7 @@ const keywords*: Table[string, TokenKind] = {
"between": tkBetween, "between": tkBetween,
"like": tkLike, "like": tkLike,
"ilike": tkILike, "ilike": tkILike,
"filter": tkFilter,
"returning": tkReturning, "returning": tkReturning,
"primary": tkPrimary, "primary": tkPrimary,
"key": tkKey, "key": tkKey,
@@ -291,6 +339,27 @@ const keywords*: Table[string, TokenKind] = {
"avg": tkAvg, "avg": tkAvg,
"min": tkMin, "min": tkMin,
"max": tkMax, "max": tkMax,
"array_agg": tkArrayAgg,
"string_agg": tkStringAgg,
"grouping": tkGrouping,
"sets": tkSets,
"rollup": tkRollup,
"cube": tkCube,
"pivot": tkPivot,
"unpivot": tkUnpivot,
"vertex": tkVertex,
"vertices": tkVertex,
"edge": tkEdge,
"edges": tkEdge,
"label": tkLabels,
"labels": tkLabels,
"graph_table": tkGraphTable,
"match": tkMatch,
"columns": tkColumns,
"src": tkSrc,
"dst": tkDst,
"merge": tkMerge,
"matched": tkMatched,
"array": tkArray, "array": tkArray,
"vector": tkVector, "vector": tkVector,
"graph": tkGraph, "graph": tkGraph,
@@ -301,6 +370,25 @@ const keywords*: Table[string, TokenKind] = {
"bfs": tkBfs, "bfs": tkBfs,
"dfs": tkDfs, "dfs": tkDfs,
"path": tkPath, "path": tkPath,
"over": tkOver,
"partition": tkPartition,
"row": tkRow,
"rows": tkRows,
"range": tkRange,
"unbounded": tkUnbounded,
"preceding": tkPreceding,
"following": tkFollowing,
"current": tkCurrent,
"current_user": tkCurrentUser,
"current_role": tkCurrentRole,
"row_number": tkRowNumber,
"rank": tkRank,
"dense_rank": tkDenseRank,
"lead": tkLead,
"lag": tkLag,
"first_value": tkFirstValue,
"last_value": tkLastValue,
"ntile": tkNtile,
}.toTable }.toTable
proc newLexer*(input: string): Lexer = proc newLexer*(input: string): Lexer =
@@ -495,6 +583,11 @@ proc nextToken*(l: var Lexer): Token =
discard l.advance() discard l.advance()
return Token(kind: tkInvalid, value: "!", line: startLine, col: startCol) return Token(kind: tkInvalid, value: "!", line: startLine, col: startCol)
of '<': of '<':
if l.pos + 2 < l.input.len and l.input[l.pos + 1] == '-' and l.input[l.pos + 2] == '>':
discard l.advance()
discard l.advance()
discard l.advance()
return Token(kind: tkDistanceOp, value: "<->", line: startLine, col: startCol)
if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '=': if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '=':
discard l.advance() discard l.advance()
discard l.advance() discard l.advance()
@@ -503,6 +596,10 @@ proc nextToken*(l: var Lexer): Token =
discard l.advance() discard l.advance()
discard l.advance() discard l.advance()
return Token(kind: tkNotEq, value: "<>", line: startLine, col: startCol) return Token(kind: tkNotEq, value: "<>", line: startLine, col: startCol)
if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '@':
discard l.advance()
discard l.advance()
return Token(kind: tkJsonContainedBy, value: "<@", line: startLine, col: startCol)
discard l.advance() discard l.advance()
return Token(kind: tkLt, value: "<", line: startLine, col: startCol) return Token(kind: tkLt, value: "<", line: startLine, col: startCol)
of '>': of '>':
@@ -513,6 +610,16 @@ proc nextToken*(l: var Lexer): Token =
discard l.advance() discard l.advance()
return Token(kind: tkGt, value: ">", line: startLine, col: startCol) return Token(kind: tkGt, value: ">", line: startLine, col: startCol)
of '?': of '?':
if l.pos + 2 < l.input.len and l.input[l.pos + 1] == '|':
discard l.advance()
discard l.advance()
discard l.advance()
return Token(kind: tkJsonHasAny, value: "?|", line: startLine, col: startCol)
if l.pos + 2 < l.input.len and l.input[l.pos + 1] == '&':
discard l.advance()
discard l.advance()
discard l.advance()
return Token(kind: tkJsonHasAll, value: "?&", line: startLine, col: startCol)
if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '?': if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '?':
discard l.advance() discard l.advance()
discard l.advance() discard l.advance()
@@ -524,6 +631,10 @@ proc nextToken*(l: var Lexer): Token =
discard l.advance() discard l.advance()
discard l.advance() discard l.advance()
return Token(kind: tkFtsMatch, value: "@@", line: startLine, col: startCol) return Token(kind: tkFtsMatch, value: "@@", line: startLine, col: startCol)
if l.pos + 1 < l.input.len and l.input[l.pos + 1] == '>':
discard l.advance()
discard l.advance()
return Token(kind: tkJsonContains, value: "@>", line: startLine, col: startCol)
discard l.advance() discard l.advance()
return Token(kind: tkInvalid, value: "@", line: startLine, col: startCol) return Token(kind: tkInvalid, value: "@", line: startLine, col: startCol)
of '.': of '.':
+390 -24
View File
@@ -38,6 +38,10 @@ proc parseExpr(p: var Parser): Node
proc parseSelect(p: var Parser): Node proc parseSelect(p: var Parser): Node
proc parseStatement*(p: var Parser): Node proc parseStatement*(p: var Parser): Node
proc parseCreateType(p: var Parser): Node proc parseCreateType(p: var Parser): Node
proc parseOverClause(p: var Parser): Node
proc parseFrameSpec(p: var Parser): Node
proc parseFrameBoundary(p: var Parser): string
proc parseSetVar(p: var Parser): Node
proc parsePrimary(p: var Parser): Node = proc parsePrimary(p: var Parser): Node =
let tok = p.peek() let tok = p.peek()
@@ -57,8 +61,15 @@ proc parsePrimary(p: var Parser): Node =
of tkNull: of tkNull:
discard p.advance() discard p.advance()
Node(kind: nkNullLit, line: tok.line, col: tok.col) Node(kind: nkNullLit, line: tok.line, col: tok.col)
of tkIdent: of tkCurrentUser:
discard p.advance() discard p.advance()
Node(kind: nkCurrentUser, line: tok.line, col: tok.col)
of tkCurrentRole:
discard p.advance()
Node(kind: nkCurrentRole, line: tok.line, col: tok.col)
of tkIdent, tkRowNumber, tkRank, tkDenseRank, tkLead, tkLag, tkFirstValue, tkLastValue, tkNtile:
discard p.advance()
let funcName = tok.value
# Check for function call: ident(...) # Check for function call: ident(...)
if p.peek().kind == tkLParen: if p.peek().kind == tkLParen:
discard p.advance() # consume ( discard p.advance() # consume (
@@ -68,15 +79,21 @@ proc parsePrimary(p: var Parser): Node =
while p.match(tkComma): while p.match(tkComma):
args.add(p.parseExpr()) args.add(p.parseExpr())
discard p.expect(tkRParen) discard p.expect(tkRParen)
return Node(kind: nkFuncCall, funcName: tok.value, funcArgs: args, var funcNode = Node(kind: nkFuncCall, funcName: funcName, funcArgs: args,
line: tok.line, col: tok.col) line: tok.line, col: tok.col)
# Check for window function: func(...) OVER (...)
if p.peek().kind == tkOver:
let overClause = p.parseOverClause()
return Node(kind: nkWindowExpr, winFunc: funcName, winArgs: args,
winOver: overClause, line: tok.line, col: tok.col)
return funcNode
# Check for dotted path: ident.ident.ident # Check for dotted path: ident.ident.ident
var parts = @[tok.value] var parts = @[funcName]
while p.peek().kind == tkDot: while p.peek().kind == tkDot:
discard p.advance() # consume . discard p.advance() # consume .
parts.add(p.expect(tkIdent).value) parts.add(p.expect(tkIdent).value)
if parts.len == 1: if parts.len == 1:
return Node(kind: nkIdent, identName: tok.value, line: tok.line, col: tok.col) return Node(kind: nkIdent, identName: funcName, line: tok.line, col: tok.col)
return Node(kind: nkPath, pathParts: parts, line: tok.line, col: tok.col) return Node(kind: nkPath, pathParts: parts, line: tok.line, col: tok.col)
of tkLParen: of tkLParen:
discard p.advance() discard p.advance()
@@ -119,7 +136,7 @@ proc parsePrimary(p: var Parser): Node =
discard p.advance() discard p.advance()
let operand = p.parsePrimary() let operand = p.parsePrimary()
Node(kind: nkUnaryOp, unOp: ukNeg, unOperand: operand, line: tok.line, col: tok.col) Node(kind: nkUnaryOp, unOp: ukNeg, unOperand: operand, line: tok.line, col: tok.col)
of tkCount, tkSum, tkAvg, tkMin, tkMax: of tkCount, tkSum, tkAvg, tkMin, tkMax, tkArrayAgg, tkStringAgg:
let funcName = tok.value let funcName = tok.value
discard p.advance() discard p.advance()
discard p.expect(tkLParen) discard p.expect(tkLParen)
@@ -131,9 +148,17 @@ proc parsePrimary(p: var Parser): Node =
hasDistinct = true hasDistinct = true
if p.peek().kind != tkRParen: if p.peek().kind != tkRParen:
args.add(p.parseExpr()) args.add(p.parseExpr())
while p.match(tkComma):
args.add(p.parseExpr())
discard p.expect(tkRParen) discard p.expect(tkRParen)
var node = Node(kind: nkFuncCall, funcName: funcName.toLower(), funcArgs: args, var node = Node(kind: nkFuncCall, funcName: funcName.toLower(), funcArgs: args,
line: tok.line, col: tok.col) line: tok.line, col: tok.col)
# Handle FILTER (WHERE ...)
if p.match(tkFilter):
discard p.expect(tkLParen)
discard p.expect(tkWhere)
node.funcFilter = p.parseExpr()
discard p.expect(tkRParen)
return node return node
of tkCase: of tkCase:
discard p.advance() discard p.advance()
@@ -157,6 +182,80 @@ proc parsePrimary(p: var Parser): Node =
discard p.advance() discard p.advance()
Node(kind: nkNullLit, line: tok.line, col: tok.col) Node(kind: nkNullLit, line: tok.line, col: tok.col)
proc parseOverClause(p: var Parser): Node =
## Parse OVER ( [PARTITION BY expr, ...] [ORDER BY expr [ASC|DESC], ...] [frame] )
discard p.expect(tkOver)
discard p.expect(tkLParen)
result = Node(kind: nkOverClause)
# PARTITION BY
if p.match(tkPartition):
discard p.expect(tkBy)
result.overPartition.add(p.parseExpr())
while p.match(tkComma):
result.overPartition.add(p.parseExpr())
# ORDER BY
if p.match(tkOrder):
discard p.expect(tkBy)
var expr = p.parseExpr()
var dir = sdAsc
if p.match(tkDesc):
dir = sdDesc
elif p.match(tkAsc):
dir = sdAsc
result.overOrderBy.add(Node(kind: nkOrderBy, orderByExpr: expr, orderByDir: dir))
while p.match(tkComma):
expr = p.parseExpr()
dir = sdAsc
if p.match(tkDesc):
dir = sdDesc
elif p.match(tkAsc):
dir = sdAsc
result.overOrderBy.add(Node(kind: nkOrderBy, orderByExpr: expr, orderByDir: dir))
# Frame specification
if p.peek().kind in {tkRows, tkRange}:
result.overFrame = p.parseFrameSpec()
discard p.expect(tkRParen)
proc parseFrameSpec(p: var Parser): Node =
## Parse ROWS|RANGE frame
let modeTok = p.advance() # tkRows or tkRange
result = Node(kind: nkFrameSpec, frameMode: modeTok.value)
if p.match(tkBetween):
result.frameStartType = p.parseFrameBoundary()
discard p.expect(tkAnd)
result.frameEndType = p.parseFrameBoundary()
else:
result.frameStartType = p.parseFrameBoundary()
result.frameEndType = "CURRENT ROW"
proc parseFrameBoundary(p: var Parser): string =
## Returns boundary type string like "UNBOUNDED PRECEDING", "CURRENT ROW", "1 PRECEDING"
if p.match(tkUnbounded):
if p.match(tkPreceding):
return "UNBOUNDED PRECEDING"
elif p.match(tkFollowing):
return "UNBOUNDED FOLLOWING"
else:
raise newException(ValueError, "Expected PRECEDING or FOLLOWING after UNBOUNDED")
elif p.match(tkCurrent):
discard p.expect(tkRow)
return "CURRENT ROW"
else:
# Expect a simple integer literal for offset boundaries
let offsetExpr = p.parsePrimary()
var offsetStr = ""
case offsetExpr.kind:
of nkIntLit: offsetStr = $offsetExpr.intVal
of nkIdent: offsetStr = offsetExpr.identName
else:
raise newException(ValueError, "Expected integer literal or identifier in frame boundary, got " & $offsetExpr.kind)
if p.match(tkPreceding):
return offsetStr & " PRECEDING"
elif p.match(tkFollowing):
return offsetStr & " FOLLOWING"
else:
raise newException(ValueError, "Expected PRECEDING or FOLLOWING after frame offset")
proc parsePostfix(p: var Parser): Node = proc parsePostfix(p: var Parser): Node =
result = p.parsePrimary() result = p.parsePrimary()
while p.peek().kind in {tkArrowR, tkArrowRR}: while p.peek().kind in {tkArrowR, tkArrowRR}:
@@ -226,7 +325,7 @@ proc parseComparison(p: var Parser): Node =
discard p.advance() # consume NULL token (assumed) discard p.advance() # consume NULL token (assumed)
return Node(kind: nkIsExpr, isExpr: result, isNegated: negated, return Node(kind: nkIsExpr, isExpr: result, isNegated: negated,
line: tok.line, col: tok.col) line: tok.line, col: tok.col)
while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch}: while p.peek().kind in {tkEq, tkNotEq, tkLt, tkLtEq, tkGt, tkGtEq, tkFtsMatch, tkDistanceOp, tkJsonContains, tkJsonContainedBy, tkJsonHasAny, tkJsonHasAll}:
let op = case p.peek().kind let op = case p.peek().kind
of tkEq: bkEq of tkEq: bkEq
of tkNotEq: bkNotEq of tkNotEq: bkNotEq
@@ -235,6 +334,11 @@ proc parseComparison(p: var Parser): Node =
of tkGt: bkGt of tkGt: bkGt
of tkGtEq: bkGtEq of tkGtEq: bkGtEq
of tkFtsMatch: bkFtsMatch of tkFtsMatch: bkFtsMatch
of tkDistanceOp: bkDistance
of tkJsonContains: bkJsonContains
of tkJsonContainedBy: bkJsonContainedBy
of tkJsonHasAny: bkJsonHasAny
of tkJsonHasAll: bkJsonHasAll
else: bkEq else: bkEq
let tok = p.advance() let tok = p.advance()
let right = p.parseAddSub() let right = p.parseAddSub()
@@ -354,7 +458,66 @@ proc parseSelect(p: var Parser): Node =
elif p.peek().kind == tkIdent: elif p.peek().kind == tkIdent:
alias = p.advance().value alias = p.advance().value
result.selFrom = Node(kind: nkFrom, fromTable: "(subquery)", result.selFrom = Node(kind: nkFrom, fromTable: "(subquery)",
fromAlias: alias, line: tok.line, col: tok.col) fromAlias: alias, fromSubquery: subquery, line: tok.line, col: tok.col)
elif p.peek().kind == tkGraphTable:
# GRAPH_TABLE(name MATCH (pattern) COLUMNS (cols))
discard p.advance()
discard p.expect(tkLParen)
let graphName = p.expect(tkIdent).value
discard p.expect(tkMatch)
# Parse pattern: (node)-[edge]->(node)
var patternNodes: seq[string]
var patternEdges: seq[string]
# First node
discard p.expect(tkLParen)
if p.peek().kind == tkIdent:
patternNodes.add(p.advance().value)
discard p.expect(tkRParen)
# Edge and next node(s)
while p.peek().kind == tkMinus or p.peek().kind == tkArrowR:
if p.match(tkArrowR):
discard
elif p.match(tkMinus):
if p.peek().kind == tkLBracket:
discard p.advance()
if p.peek().kind == tkIdent:
patternEdges.add(p.advance().value)
discard p.expect(tkRBracket)
discard p.expect(tkArrowR)
else:
discard
if p.peek().kind == tkLParen:
discard p.advance()
if p.peek().kind == tkIdent:
patternNodes.add(p.advance().value)
discard p.expect(tkRParen)
# COLUMNS (col1, col2, ...)
var returnCols: seq[string]
if p.match(tkColumns):
discard p.expect(tkLParen)
if p.peek().kind == tkIdent:
var colName = p.advance().value
# Handle dotted names: e.name
while p.peek().kind == tkDot:
discard p.advance() # skip dot
colName &= "." & p.expect(tkIdent).value
returnCols.add(colName)
while p.match(tkComma):
if p.peek().kind == tkIdent:
colName = p.advance().value
while p.peek().kind == tkDot:
discard p.advance()
colName &= "." & p.expect(tkIdent).value
returnCols.add(colName)
if p.match(tkAs):
discard p.advance() # skip alias
discard p.expect(tkRParen)
discard p.expect(tkRParen)
# Create a graph traversal node
result.selFrom = Node(kind: nkGraphTraversal, gtGraphName: graphName,
gtStart: nil, gtEdge: if patternEdges.len > 0: patternEdges[0] else: "",
gtDirection: "out", gtEnd: nil, gtMaxDepth: -1,
gtReturnCols: returnCols, line: tok.line, col: tok.col)
else: else:
let tableTok = p.expect(tkIdent) let tableTok = p.expect(tkIdent)
var alias = "" var alias = ""
@@ -365,26 +528,89 @@ proc parseSelect(p: var Parser): Node =
result.selFrom = Node(kind: nkFrom, fromTable: tableTok.value, result.selFrom = Node(kind: nkFrom, fromTable: tableTok.value,
fromAlias: alias, line: tableTok.line, col: tableTok.col) fromAlias: alias, line: tableTok.line, col: tableTok.col)
# Parse PIVOT / UNPIVOT after FROM source
if p.peek().kind == tkPivot:
discard p.advance()
discard p.expect(tkLParen)
let aggFunc = p.parseExpr() # e.g. SUM(salary)
discard p.expect(tkFor)
let forCol = p.expect(tkIdent).value
discard p.expect(tkIn)
discard p.expect(tkLParen)
var inValues: seq[string] = @[]
inValues.add(p.expect(tkStringLit).value)
while p.match(tkComma):
inValues.add(p.expect(tkStringLit).value)
discard p.expect(tkRParen)
discard p.expect(tkRParen)
result.selFrom = Node(kind: nkPivot, pivotSource: result.selFrom,
pivotAgg: aggFunc, pivotForCol: forCol,
pivotInValues: inValues, line: tok.line, col: tok.col)
elif p.peek().kind == tkUnpivot:
discard p.advance()
discard p.expect(tkLParen)
let valCol = p.expect(tkIdent).value
discard p.expect(tkFor)
let forCol = p.expect(tkIdent).value
discard p.expect(tkIn)
discard p.expect(tkLParen)
var inCols: seq[string] = @[]
inCols.add(p.expect(tkIdent).value)
while p.match(tkComma):
inCols.add(p.expect(tkIdent).value)
discard p.expect(tkRParen)
discard p.expect(tkRParen)
result.selFrom = Node(kind: nkUnpivot, unpivotSource: result.selFrom,
unpivotValueCol: valCol, unpivotForCol: forCol,
unpivotInCols: inCols, line: tok.line, col: tok.col)
# Parse JOINs # Parse JOINs
while p.peek().kind == tkJoin or while p.peek().kind == tkJoin or
p.peek().kind == tkLateral or
(p.peek().kind in {tkInner, tkLeft, tkRight, tkFull, tkCross} and (p.peek().kind in {tkInner, tkLeft, tkRight, tkFull, tkCross} and
p.pos + 1 < p.tokens.len and p.tokens[p.pos + 1].kind == tkJoin): p.pos + 1 < p.tokens.len and p.tokens[p.pos + 1].kind in {tkJoin, tkLateral}):
let jk = p.parseJoinType() var isLateral = false
discard p.expect(tkJoin) var jk = jkInner
let joinTable = p.expect(tkIdent) # Check for LATERAL before join type
if p.match(tkLateral):
isLateral = true
# Could be standalone LATERAL or LATERAL JOIN
if p.peek().kind == tkJoin:
discard p.advance()
else:
jk = p.parseJoinType()
discard p.expect(tkJoin)
# Check for LATERAL after JOIN keyword
if p.match(tkLateral):
isLateral = true
var joinTarget: Node
var joinAlias = "" var joinAlias = ""
if p.match(tkAs): if isLateral:
joinAlias = p.expect(tkIdent).value # LATERAL (subquery) AS alias
elif p.peek().kind == tkIdent: discard p.expect(tkLParen)
joinAlias = p.advance().value let subquery = p.parseSelect()
discard p.expect(tkRParen)
if p.match(tkAs):
joinAlias = p.expect(tkIdent).value
elif p.peek().kind == tkIdent:
joinAlias = p.advance().value
joinTarget = Node(kind: nkSubquery, subQuery: subquery,
line: tok.line, col: tok.col)
else:
let joinTable = p.expect(tkIdent)
if p.match(tkAs):
joinAlias = p.expect(tkIdent).value
elif p.peek().kind == tkIdent:
joinAlias = p.advance().value
joinTarget = Node(kind: nkFrom, fromTable: joinTable.value,
fromAlias: joinAlias, line: joinTable.line, col: joinTable.col)
var joinCond: Node = nil var joinCond: Node = nil
if p.match(tkOn): if p.match(tkOn):
joinCond = p.parseExpr() joinCond = p.parseExpr()
let joinTarget = Node(kind: nkFrom, fromTable: joinTable.value,
fromAlias: joinAlias, line: joinTable.line, col: joinTable.col)
result.selJoins.add(Node(kind: nkJoin, joinKind: jk, joinTarget: joinTarget, result.selJoins.add(Node(kind: nkJoin, joinKind: jk, joinTarget: joinTarget,
joinOn: joinCond, joinAlias: joinAlias, joinOn: joinCond, joinAlias: joinAlias, joinLateral: isLateral,
line: joinTable.line, col: joinTable.col)) line: tok.line, col: tok.col))
# Parse WHERE # Parse WHERE
if p.match(tkWhere): if p.match(tkWhere):
@@ -394,9 +620,47 @@ proc parseSelect(p: var Parser): Node =
if p.match(tkGroup): if p.match(tkGroup):
discard p.expect(tkBy) discard p.expect(tkBy)
result.selGroupBy = @[] result.selGroupBy = @[]
result.selGroupBy.add(p.parseExpr()) result.selGroupingSetsKind = gskNone
while p.match(tkComma): result.selGroupingSets = @[]
# Check for GROUPING SETS, ROLLUP, CUBE
if p.peek().kind == tkGrouping:
discard p.advance()
discard p.expect(tkSets)
result.selGroupingSetsKind = gskGroupingSets
discard p.expect(tkLParen)
# Parse each set: (col1, col2) or ()
while true:
discard p.expect(tkLParen)
var setExprs: seq[Node] = @[]
if p.peek().kind != tkRParen:
setExprs.add(p.parseExpr())
while p.match(tkComma):
setExprs.add(p.parseExpr())
discard p.expect(tkRParen)
result.selGroupingSets.add(setExprs)
if not p.match(tkComma): break
discard p.expect(tkRParen)
elif p.peek().kind == tkRollup:
discard p.advance()
result.selGroupingSetsKind = gskRollup
discard p.expect(tkLParen)
result.selGroupBy.add(p.parseExpr()) result.selGroupBy.add(p.parseExpr())
while p.match(tkComma):
result.selGroupBy.add(p.parseExpr())
discard p.expect(tkRParen)
elif p.peek().kind == tkCube:
discard p.advance()
result.selGroupingSetsKind = gskCube
discard p.expect(tkLParen)
result.selGroupBy.add(p.parseExpr())
while p.match(tkComma):
result.selGroupBy.add(p.parseExpr())
discard p.expect(tkRParen)
else:
# Regular GROUP BY
result.selGroupBy.add(p.parseExpr())
while p.match(tkComma):
result.selGroupBy.add(p.parseExpr())
# Parse HAVING # Parse HAVING
if p.match(tkHaving): if p.match(tkHaving):
@@ -540,6 +804,64 @@ proc parseDelete(p: var Parser): Node =
while p.match(tkComma): while p.match(tkComma):
result.delReturning.add(p.parseExpr()) result.delReturning.add(p.parseExpr())
proc parseMerge(p: var Parser): Node =
let tok = p.expect(tkMerge)
discard p.match(tkInto) # optional INTO
result = Node(kind: nkMerge, line: tok.line, col: tok.col)
result.mergeTarget = p.expect(tkIdent).value
if p.match(tkAs):
result.mergeTargetAlias = p.expect(tkIdent).value
elif p.peek().kind == tkIdent:
result.mergeTargetAlias = p.advance().value
discard p.expect(tkUsing)
# Source can be a table name or a subquery
if p.peek().kind == tkLParen:
discard p.advance() # consume (
result.mergeSource = p.parseSelect()
discard p.expect(tkRParen)
else:
let srcTable = p.expect(tkIdent).value
result.mergeSource = Node(kind: nkIdent, identName: srcTable, line: tok.line, col: tok.col)
if p.match(tkAs):
result.mergeSourceAlias = p.expect(tkIdent).value
elif p.peek().kind == tkIdent:
result.mergeSourceAlias = p.advance().value
discard p.expect(tkOn)
result.mergeOn = p.parseExpr()
# WHEN MATCHED THEN UPDATE SET ...
if p.match(tkWhen):
if p.match(tkNot):
discard p.expect(tkMatched)
discard p.expect(tkThen)
discard p.expect(tkInsert)
discard p.expect(tkLParen)
result.mergeNotMatchedInsert.add(Node(kind: nkIdent, identName: p.expect(tkIdent).value))
while p.match(tkComma):
result.mergeNotMatchedInsert.add(Node(kind: nkIdent, identName: p.expect(tkIdent).value))
discard p.expect(tkRParen)
discard p.expect(tkValues)
discard p.expect(tkLParen)
result.mergeNotMatchedValues.add(p.parseExpr())
while p.match(tkComma):
result.mergeNotMatchedValues.add(p.parseExpr())
discard p.expect(tkRParen)
else:
discard p.expect(tkMatched)
discard p.expect(tkThen)
discard p.expect(tkUpdate)
discard p.expect(tkSet)
let col = p.expect(tkIdent).value
discard p.expect(tkEq)
result.mergeMatchedUpdate.add(Node(kind: nkBinOp, binOp: bkAssign,
binLeft: Node(kind: nkIdent, identName: col),
binRight: p.parseExpr()))
while p.match(tkComma):
let col2 = p.expect(tkIdent).value
discard p.expect(tkEq)
result.mergeMatchedUpdate.add(Node(kind: nkBinOp, binOp: bkAssign,
binLeft: Node(kind: nkIdent, identName: col2),
binRight: p.parseExpr()))
proc parseCreateType(p: var Parser): Node = proc parseCreateType(p: var Parser): Node =
let tok = p.expect(tkCreate) let tok = p.expect(tkCreate)
discard p.expect(tkType) discard p.expect(tkType)
@@ -672,6 +994,14 @@ proc parseCreateTable(p: var Parser): Node =
let size = p.expect(tkIntLit).value let size = p.expect(tkIntLit).value
colType &= "(" & size & ")" colType &= "(" & size & ")"
discard p.expect(tkRParen) discard p.expect(tkRParen)
elif p.peek().kind == tkVector:
discard p.advance()
colType = "VECTOR"
if p.peek().kind == tkLParen:
discard p.advance()
let size = p.expect(tkIntLit).value
colType &= "(" & size & ")"
discard p.expect(tkRParen)
let colDef = Node(kind: nkColumnDef, cdName: colName, cdType: colType) let colDef = Node(kind: nkColumnDef, cdName: colName, cdType: colType)
colDef.cdConstraints = @[] colDef.cdConstraints = @[]
@@ -737,13 +1067,13 @@ proc parseAlterTable(p: var Parser): Node =
# Check for ENABLE/DISABLE ROW LEVEL SECURITY # Check for ENABLE/DISABLE ROW LEVEL SECURITY
if p.peek().kind == tkEnable: if p.peek().kind == tkEnable:
discard p.advance() discard p.advance()
discard p.expect(tkIdent) # ROW discard p.expect(tkRow) # ROW
discard p.expect(tkIdent) # LEVEL discard p.expect(tkIdent) # LEVEL
discard p.expect(tkIdent) # SECURITY discard p.expect(tkIdent) # SECURITY
return Node(kind: nkEnableRLS, erlsTable: tableName, line: tok.line, col: tok.col) return Node(kind: nkEnableRLS, erlsTable: tableName, line: tok.line, col: tok.col)
elif p.peek().kind == tkDisable: elif p.peek().kind == tkDisable:
discard p.advance() discard p.advance()
discard p.expect(tkIdent) # ROW discard p.expect(tkRow) # ROW
discard p.expect(tkIdent) # LEVEL discard p.expect(tkIdent) # LEVEL
discard p.expect(tkIdent) # SECURITY discard p.expect(tkIdent) # SECURITY
return Node(kind: nkDisableRLS, drlsTable: tableName, line: tok.line, col: tok.col) return Node(kind: nkDisableRLS, drlsTable: tableName, line: tok.line, col: tok.col)
@@ -781,6 +1111,10 @@ proc parseCreateIndex(p: var Parser): Node =
let idxMethod = p.expect(tkIdent).value.toLower() let idxMethod = p.expect(tkIdent).value.toLower()
if idxMethod == "fts" or idxMethod == "fulltext": if idxMethod == "fts" or idxMethod == "fulltext":
idxKind = ikFullText idxKind = ikFullText
elif idxMethod == "hnsw":
idxKind = ikHNSW
elif idxMethod == "ivfpq":
idxKind = ikIVFPQ
result = Node(kind: nkCreateIndex, ciName: idxName, ciTarget: tableName, result = Node(kind: nkCreateIndex, ciName: idxName, ciTarget: tableName,
ciColumns: colNames, ciKind: idxKind, line: tok.line, col: tok.col) ciColumns: colNames, ciKind: idxKind, line: tok.line, col: tok.col)
@@ -1099,12 +1433,42 @@ proc parseRevoke(p: var Parser): Node =
result = Node(kind: nkRevoke, rvPrivilege: priv, rvTable: tableName, result = Node(kind: nkRevoke, rvPrivilege: priv, rvTable: tableName,
rvGrantee: grantee, line: tok.line, col: tok.col) rvGrantee: grantee, line: tok.line, col: tok.col)
proc parseSetVar(p: var Parser): Node =
let tok = p.expect(tkSet)
var varName = p.expect(tkIdent).value
while p.peek().kind == tkDot:
discard p.advance()
varName.add(".")
varName.add(p.expect(tkIdent).value)
if p.match(tkEq) or p.match(tkTo):
discard
let valTok = p.peek()
var valStr = ""
case valTok.kind
of tkStringLit:
valStr = valTok.value
discard p.advance()
of tkIntLit:
valStr = valTok.value
discard p.advance()
of tkFloatLit:
valStr = valTok.value
discard p.advance()
of tkIdent:
valStr = valTok.value
discard p.advance()
else:
raise newException(ValueError, "Expected value after SET " & varName)
result = Node(kind: nkSetVar, svName: varName, svValue: valStr,
line: tok.line, col: tok.col)
proc parseStatement*(p: var Parser): Node = proc parseStatement*(p: var Parser): Node =
case p.peek().kind case p.peek().kind
of tkWith, tkSelect: p.parseSelect() of tkWith, tkSelect: p.parseSelect()
of tkInsert: p.parseInsert() of tkInsert: p.parseInsert()
of tkUpdate: p.parseUpdate() of tkUpdate: p.parseUpdate()
of tkDelete: p.parseDelete() of tkDelete: p.parseDelete()
of tkMerge: p.parseMerge()
of tkCreate: of tkCreate:
if p.pos + 1 < p.tokens.len: if p.pos + 1 < p.tokens.len:
let next = p.tokens[p.pos + 1] let next = p.tokens[p.pos + 1]
@@ -1159,6 +1523,8 @@ proc parseStatement*(p: var Parser): Node =
p.parseGrant() p.parseGrant()
of tkRevoke: of tkRevoke:
p.parseRevoke() p.parseRevoke()
of tkSet:
p.parseSetVar()
of tkApply: of tkApply:
if p.pos + 1 < p.tokens.len and p.tokens[p.pos + 1].kind == tkMigration: if p.pos + 1 < p.tokens.len and p.tokens[p.pos + 1].kind == tkMigration:
p.parseApplyMigration() p.parseApplyMigration()
+1 -1
View File
@@ -86,7 +86,7 @@ proc main() =
# Init structured logger from config # Init structured logger from config
let logLvl = parseEnum[LogLevel]("ll" & capitalizeAscii(config.logLevel)) let logLvl = parseEnum[LogLevel]("ll" & capitalizeAscii(config.logLevel))
defaultLogger = newLogger(logLvl, config.logFile) defaultLogger = newLogger(logLvl, config.logFile)
info("BaraDB v1.0.0 — Multimodal Database Engine") info("BaraDB v1.1.0 — Multimodal Database Engine")
# Security check: warn if JWT secret is not configured # Security check: warn if JWT secret is not configured
if config.jwtSecret.len == 0: if config.jwtSecret.len == 0:
+29
View File
@@ -79,3 +79,32 @@ suite "JOIN execution":
test "count after CROSS JOIN": test "count after CROSS JOIN":
let r = execSql(ctx, "SELECT COUNT(*) AS cnt FROM users u CROSS JOIN orders o") let r = execSql(ctx, "SELECT COUNT(*) AS cnt FROM users u CROSS JOIN orders o")
check r.rows[0]["cnt"] == "6" check r.rows[0]["cnt"] == "6"
test "LATERAL JOIN with correlated subquery":
let r = execSql(ctx,
"SELECT u.name, recent.total FROM users u JOIN LATERAL (SELECT o.total FROM orders o WHERE o.user_id = u.id ORDER BY o.total DESC LIMIT 1) AS recent ON 1=1")
check r.rows.len == 1
check r.rows[0]["name"] == "Alice"
check r.rows[0]["total"] == "99.5"
test "LATERAL JOIN returns no rows when subquery empty":
let r = execSql(ctx,
"SELECT u.name, x.total FROM users u JOIN LATERAL (SELECT o.total FROM orders o WHERE o.user_id = u.id AND o.total > 1000) AS x ON 1=1")
check r.rows.len == 0
test "LEFT LATERAL JOIN keeps unmatched rows":
let r = execSql(ctx,
"SELECT u.name, x.total FROM users u LEFT JOIN LATERAL (SELECT o.total FROM orders o WHERE o.user_id = u.id ORDER BY o.total DESC LIMIT 1) AS x ON 1=1")
check r.rows.len == 2
# Alice has match (99.5), Bob has no orders -> NULL
var foundBob = false
for row in r.rows:
if row["name"] == "Bob":
check row["total"] == ""
foundBob = true
check foundBob
test "CROSS JOIN LATERAL":
let r = execSql(ctx,
"SELECT u.name, x.total FROM users u CROSS JOIN LATERAL (SELECT o.total FROM orders o WHERE o.user_id = u.id) AS x")
check r.rows.len == 2 # Alice has 2 orders, Bob has 0
+526 -9
View File
@@ -969,8 +969,40 @@ suite "BaraQL Parser — Extended":
test "Parse GROUP BY with HAVING": test "Parse GROUP BY with HAVING":
let ast = parse("SELECT dept, count(*) FROM employees GROUP BY dept HAVING count(*) > 5") let ast = parse("SELECT dept, count(*) FROM employees GROUP BY dept HAVING count(*) > 5")
check ast.stmts[0].selGroupBy.len == 1
check ast.stmts[0].selHaving != nil check ast.stmts[0].selHaving != nil
test "Parse FILTER clause":
let ast = parse("SELECT COUNT(*) FILTER (WHERE active = true) FROM users")
check ast.stmts[0].selResult.len == 1
check ast.stmts[0].selResult[0].funcFilter != nil
test "Parse ROLLUP":
let ast = parse("SELECT dept, SUM(amount) FROM sales GROUP BY ROLLUP (dept)")
check ast.stmts[0].selGroupingSetsKind == gskRollup
check ast.stmts[0].selGroupBy.len == 1
test "Parse CUBE":
let ast = parse("SELECT dept, job, SUM(amount) FROM sales GROUP BY CUBE (dept, job)")
check ast.stmts[0].selGroupingSetsKind == gskCube
check ast.stmts[0].selGroupBy.len == 2
test "Parse GROUPING SETS":
let ast = parse("SELECT dept, job, SUM(amount) FROM sales GROUP BY GROUPING SETS ((dept), (job), ())")
check ast.stmts[0].selGroupingSetsKind == gskGroupingSets
check ast.stmts[0].selGroupingSets.len == 3
test "Parse PIVOT":
let ast = parse("SELECT * FROM (SELECT dept, salary FROM emp) PIVOT (SUM(salary) FOR dept IN ('Eng', 'Sales'))")
check ast.stmts[0].selFrom.kind == nkPivot
check ast.stmts[0].selFrom.pivotForCol == "dept"
check ast.stmts[0].selFrom.pivotInValues.len == 2
test "Parse GRAPH_TABLE":
let ast = parse("SELECT * FROM GRAPH_TABLE(org_chart MATCH (e)-[r]->(d) COLUMNS (e.name, d.name))")
check ast.stmts[0].selFrom.kind == nkGraphTraversal
check ast.stmts[0].selFrom.gtGraphName == "org_chart"
test "Parse ORDER BY with direction": test "Parse ORDER BY with direction":
let ast = parse("SELECT name FROM users ORDER BY age DESC") let ast = parse("SELECT name FROM users ORDER BY age DESC")
check ast.stmts[0].selOrderBy.len == 1 check ast.stmts[0].selOrderBy.len == 1
@@ -994,6 +1026,25 @@ suite "BaraQL Parser — Extended":
let ast = parse("SELECT * FROM a JOIN b ON a.id = b.aid JOIN c ON b.id = c.bid") let ast = parse("SELECT * FROM a JOIN b ON a.id = b.aid JOIN c ON b.id = c.bid")
check ast.stmts[0].selJoins.len == 2 check ast.stmts[0].selJoins.len == 2
test "Parse LATERAL JOIN":
let ast = parse("SELECT u.name, x.total FROM users u JOIN LATERAL (SELECT o.total FROM orders o WHERE o.user_id = u.id) AS x ON 1=1")
check ast.stmts[0].selJoins.len == 1
check ast.stmts[0].selJoins[0].joinLateral == true
check ast.stmts[0].selJoins[0].joinTarget.kind == nkSubquery
check ast.stmts[0].selJoins[0].joinAlias == "x"
test "Parse LEFT LATERAL JOIN":
let ast = parse("SELECT u.name FROM users u LEFT JOIN LATERAL (SELECT 1) AS x ON 1=1")
check ast.stmts[0].selJoins.len == 1
check ast.stmts[0].selJoins[0].joinKind == jkLeft
check ast.stmts[0].selJoins[0].joinLateral == true
test "Parse CROSS JOIN LATERAL":
let ast = parse("SELECT * FROM users u CROSS JOIN LATERAL (SELECT 1) AS x")
check ast.stmts[0].selJoins.len == 1
check ast.stmts[0].selJoins[0].joinKind == jkCross
check ast.stmts[0].selJoins[0].joinLateral == true
test "Parse CTE (WITH)": test "Parse CTE (WITH)":
let ast = parse("WITH active AS (SELECT * FROM users WHERE active = true) SELECT * FROM active") let ast = parse("WITH active AS (SELECT * FROM users WHERE active = true) SELECT * FROM active")
check ast.stmts[0].selWith.len == 1 check ast.stmts[0].selWith.len == 1
@@ -2236,6 +2287,69 @@ suite "Row-Level Security":
check res.success check res.success
check res.rows.len == 1 # superuser sees all (only 1 row exists) check res.rows.len == 1 # superuser sees all (only 1 row exists)
suite "Session Variables and Multi-Tenant":
test "SET statement parse":
let ast = parse("SET app.tenant_id = 'company-123'")
check ast.stmts.len == 1
check ast.stmts[0].kind == nkSetVar
check ast.stmts[0].svName == "app.tenant_id"
check ast.stmts[0].svValue == "company-123"
test "SET and current_setting":
let tmpDir = getTempDir() / "baradb_session_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("SET app.tenant_id = 'company-123'"))
let r = qexec.executeQuery(ctx, parse("SELECT current_setting('app.tenant_id') AS tenant"))
check r.success
check r.rows.len == 1
check r.rows[0]["tenant"] == "company-123"
test "current_user in SELECT":
let tmpDir = getTempDir() / "baradb_user_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db)
ctx.currentUser = "alice"
let r = qexec.executeQuery(ctx, parse("SELECT current_user AS me"))
check r.success
check r.rows.len == 1
check r.rows[0]["me"] == "alice"
test "current_role in SELECT":
let tmpDir = getTempDir() / "baradb_role_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db)
ctx.currentRole = "admin"
let r = qexec.executeQuery(ctx, parse("SELECT current_role AS role"))
check r.success
check r.rows.len == 1
check r.rows[0]["role"] == "admin"
test "Multi-tenant RLS with current_setting":
let tmpDir = getTempDir() / "baradb_multitenant_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE invoices (id INTEGER, tenant_id TEXT, amount INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO invoices (id, tenant_id, amount) VALUES (1, 'company-a', 100)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO invoices (id, tenant_id, amount) VALUES (2, 'company-b', 200)"))
# Set session variable for tenant
discard qexec.executeQuery(ctx, parse("SET app.tenant_id = 'company-a'"))
# Create policy that uses current_setting
ctx.users["app"] = qexec.UserDef(name: "app", passwordHash: "", isSuperuser: false, roles: @[])
ctx.currentUser = "app"
ctx.policies["invoices"] = @[
qexec.PolicyDef(name: "tenant_isolation", tableName: "invoices", command: "SELECT",
usingExpr: Node(kind: nkBinOp, binOp: bkEq,
binLeft: Node(kind: nkIdent, identName: "tenant_id"),
binRight: Node(kind: nkFuncCall, funcName: "current_setting",
funcArgs: @[Node(kind: nkStringLit, strVal: "app.tenant_id")])),
withCheckExpr: nil)
]
let r = qexec.executeQuery(ctx, parse("SELECT id, tenant_id FROM invoices"))
check r.success
check r.rows.len == 1
check r.rows[0]["tenant_id"] == "company-a"
suite "UTF-8 Support": suite "UTF-8 Support":
test "Tokenize UTF-8 identifiers": test "Tokenize UTF-8 identifiers":
let tokens = lex.tokenize("SELECT имя FROM потребители") let tokens = lex.tokenize("SELECT имя FROM потребители")
@@ -2252,7 +2366,8 @@ suite "UTF-8 Support":
check ast.stmts[0].selWhere.whereExpr.binRight.strVal == "София" check ast.stmts[0].selWhere.whereExpr.binRight.strVal == "София"
test "Execute query with UTF-8 data": test "Execute query with UTF-8 data":
var db = newLSMTree("") let tmpDir = getTempDir() / "baradb_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db) var ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE потребители (имя TEXT, град TEXT)")) discard qexec.executeQuery(ctx, parse("CREATE TABLE потребители (имя TEXT, град TEXT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO потребители (имя, град) VALUES ('Иван', 'София'), ('Мария', 'Пловдив')")) discard qexec.executeQuery(ctx, parse("INSERT INTO потребители (имя, град) VALUES ('Иван', 'София'), ('Мария', 'Пловдив')"))
@@ -2264,7 +2379,8 @@ suite "UTF-8 Support":
suite "B-Tree Range Scan": suite "B-Tree Range Scan":
test "BETWEEN uses index range scan": test "BETWEEN uses index range scan":
var db = newLSMTree("") let tmpDir = getTempDir() / "baradb_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db) var ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE products (id INTEGER, name TEXT)")) discard qexec.executeQuery(ctx, parse("CREATE TABLE products (id INTEGER, name TEXT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO products (id, name) VALUES (1, 'apple'), (2, 'banana'), (3, 'cherry'), (4, 'date'), (5, 'elderberry')")) discard qexec.executeQuery(ctx, parse("INSERT INTO products (id, name) VALUES (1, 'apple'), (2, 'banana'), (3, 'cherry'), (4, 'date'), (5, 'elderberry')"))
@@ -2274,7 +2390,8 @@ suite "B-Tree Range Scan":
check res.rows.len == 3 check res.rows.len == 3
test "Greater than uses index range scan": test "Greater than uses index range scan":
var db = newLSMTree("") let tmpDir = getTempDir() / "baradb_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db) var ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE nums (id INTEGER, val TEXT)")) discard qexec.executeQuery(ctx, parse("CREATE TABLE nums (id INTEGER, val TEXT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO nums (id, val) VALUES (1, '10'), (2, '20'), (3, '30'), (4, '40'), (5, '50')")) discard qexec.executeQuery(ctx, parse("INSERT INTO nums (id, val) VALUES (1, '10'), (2, '20'), (3, '30'), (4, '40'), (5, '50')"))
@@ -2284,7 +2401,8 @@ suite "B-Tree Range Scan":
check res.rows.len == 3 check res.rows.len == 3
test "Less than or equal uses index range scan": test "Less than or equal uses index range scan":
var db = newLSMTree("") let tmpDir = getTempDir() / "baradb_test_" & $getMonoTime().ticks
var db = newLSMTree(tmpDir)
var ctx = qexec.newExecutionContext(db) var ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE nums2 (id INTEGER, val TEXT)")) discard qexec.executeQuery(ctx, parse("CREATE TABLE nums2 (id INTEGER, val TEXT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO nums2 (id, val) VALUES (1, '10'), (2, '20'), (3, '30'), (4, '40'), (5, '50')")) discard qexec.executeQuery(ctx, parse("INSERT INTO nums2 (id, val) VALUES (1, '10'), (2, '20'), (3, '30'), (4, '40'), (5, '50')"))
@@ -2407,13 +2525,18 @@ suite "Enhanced Migrations":
suite "Parameterized queries": suite "Parameterized queries":
var db: LSMTree var db: LSMTree
var ctx: qexec.ExecutionContext var ctx: qexec.ExecutionContext
var tmpDir: string
var paramInitialized = false
setup: setup:
db = newLSMTree("") if not paramInitialized:
ctx = qexec.newExecutionContext(db) tmpDir = getTempDir() / "baradb_param_test_" & $getMonoTime().ticks
discard qexec.executeQuery(ctx, parse("CREATE TABLE users (id INT, name TEXT, age INT)")) db = newLSMTree(tmpDir)
discard qexec.executeQuery(ctx, parse("INSERT INTO users (id, name, age) VALUES (1, 'Alice', 30)")) ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("INSERT INTO users (id, name, age) VALUES (2, 'Bob', 25)")) discard qexec.executeQuery(ctx, parse("CREATE TABLE users (id INT, name TEXT, age INT, active BOOLEAN)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO users (id, name, age, active) VALUES (1, 'Alice', 30, true)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO users (id, name, age, active) VALUES (2, 'Bob', 25, false)"))
paramInitialized = true
test "SELECT with placeholder params": test "SELECT with placeholder params":
let sql = "SELECT * FROM users WHERE id = ?" let sql = "SELECT * FROM users WHERE id = ?"
@@ -2532,6 +2655,34 @@ suite "Parameterized queries":
check r.success check r.success
check r.rows.len >= 1 check r.rows.len >= 1
test "JSON contains @>":
discard qexec.executeQuery(ctx, parse("CREATE TABLE IF NOT EXISTS jsontest2 (id INT PRIMARY KEY, data JSON)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO jsontest2 (id, data) VALUES (1, '{\"name\": \"Alice\", \"age\": 30}')"))
let r = qexec.executeQuery(ctx, parse("SELECT id FROM jsontest2 WHERE data @> '{\"name\": \"Alice\"}'"))
check r.success
check r.rows.len == 1
check r.rows[0]["id"] == "1"
test "JSON contained by <@":
let r = qexec.executeQuery(ctx, parse("SELECT id FROM jsontest2 WHERE '{\"name\": \"Alice\"}' <@ data"))
check r.success
check r.rows.len == 1
test "JSON has key json_has_key":
let r = qexec.executeQuery(ctx, parse("SELECT id FROM jsontest2 WHERE json_has_key(data, 'name') = 'true'"))
check r.success
check r.rows.len == 1
test "JSON has any key ?|":
let r = qexec.executeQuery(ctx, parse("SELECT id FROM jsontest2 WHERE data ?| '[\"name\", \"missing\"]'"))
check r.success
check r.rows.len == 1
test "JSON has all keys ?&":
let r = qexec.executeQuery(ctx, parse("SELECT id FROM jsontest2 WHERE data ?& '[\"name\", \"age\"]'"))
check r.success
check r.rows.len == 1
test "FTS match operator @@ parse": test "FTS match operator @@ parse":
let ast = parse("SELECT * FROM docs WHERE content @@ 'hello'") let ast = parse("SELECT * FROM docs WHERE content @@ 'hello'")
check ast.stmts[0].kind == nkSelect check ast.stmts[0].kind == nkSelect
@@ -2576,8 +2727,374 @@ suite "Parameterized queries":
check r.rows.len == 1 check r.rows.len == 1
check r.rows[0]["name"] == "Bob" check r.rows[0]["name"] == "Bob"
suite "Window Functions":
var db: LSMTree
var ctx: qexec.ExecutionContext
var tmpDir: string
setup:
tmpDir = getTempDir() / "baradb_window_test_" & $getMonoTime().ticks
db = newLSMTree(tmpDir)
ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE employees (id INT, name TEXT, department TEXT, salary INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO employees (id, name, department, salary) VALUES (1, 'Alice', 'Engineering', 90000)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO employees (id, name, department, salary) VALUES (2, 'Bob', 'Engineering', 80000)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO employees (id, name, department, salary) VALUES (3, 'Charlie', 'Sales', 70000)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO employees (id, name, department, salary) VALUES (4, 'Diana', 'Sales', 75000)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO employees (id, name, department, salary) VALUES (5, 'Eve', 'Engineering', 95000)"))
teardown:
removeDir(tmpDir)
test "ROW_NUMBER with PARTITION BY and ORDER BY":
let r = qexec.executeQuery(ctx, parse("SELECT name, department, ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) AS rn FROM employees"))
check r.success
check r.rows.len == 5
# Engineering: Eve(95000)=1, Alice(90000)=2, Bob(80000)=3
# Sales: Diana(75000)=1, Charlie(70000)=2
var found = initTable[string, string]()
for row in r.rows:
found[row["name"]] = row["rn"]
check found["Eve"] == "1"
check found["Alice"] == "2"
check found["Bob"] == "3"
check found["Diana"] == "1"
check found["Charlie"] == "2"
test "RANK and DENSE_RANK":
let r = qexec.executeQuery(ctx, parse("SELECT name, salary, RANK() OVER (ORDER BY salary DESC) AS r, DENSE_RANK() OVER (ORDER BY salary DESC) AS dr FROM employees"))
check r.success
check r.rows.len == 5
for row in r.rows:
if row["name"] == "Eve":
check row["r"] == "1"
check row["dr"] == "1"
if row["name"] == "Alice":
check row["r"] == "2"
check row["dr"] == "2"
test "LEAD and LAG":
let r = qexec.executeQuery(ctx, parse("SELECT name, salary, LAG(salary, 1, 0) OVER (ORDER BY salary) AS prev, LEAD(salary, 1, 0) OVER (ORDER BY salary) AS next FROM employees"))
check r.success
check r.rows.len == 5
for row in r.rows:
if row["name"] == "Charlie":
check row["prev"] == "0"
check row["next"] == "75000"
if row["name"] == "Diana":
check row["prev"] == "70000"
check row["next"] == "80000"
test "FIRST_VALUE and LAST_VALUE":
let r = qexec.executeQuery(ctx, parse("SELECT department, FIRST_VALUE(name) OVER (PARTITION BY department ORDER BY salary) AS first, LAST_VALUE(name) OVER (PARTITION BY department ORDER BY salary) AS last FROM employees"))
check r.success
check r.rows.len == 5
test "NTILE":
let r = qexec.executeQuery(ctx, parse("SELECT name, NTILE(2) OVER (ORDER BY salary) AS bucket FROM employees"))
check r.success
check r.rows.len == 5
for row in r.rows:
check row["bucket"] in @["1", "2"]
test "FIRST_VALUE with frame":
let r = qexec.executeQuery(ctx, parse("SELECT name, salary, FIRST_VALUE(salary) OVER (ORDER BY salary ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS first_sal FROM employees"))
check r.success
check r.rows.len == 5
for row in r.rows:
if row["name"] == "Charlie":
check row["first_sal"] == "70000"
if row["name"] == "Diana":
check row["first_sal"] == "70000"
if row["name"] == "Bob":
check row["first_sal"] == "75000"
test "LAST_VALUE with frame":
let r = qexec.executeQuery(ctx, parse("SELECT name, salary, LAST_VALUE(salary) OVER (ORDER BY salary ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING) AS last_sal FROM employees"))
check r.success
check r.rows.len == 5
for row in r.rows:
if row["name"] == "Charlie":
check row["last_sal"] == "75000"
if row["name"] == "Diana":
check row["last_sal"] == "80000"
if row["name"] == "Bob":
check row["last_sal"] == "90000"
suite "GROUP BY Aggregates":
var db: LSMTree
var ctx: qexec.ExecutionContext
var tmpDir: string
setup:
tmpDir = getTempDir() / "baradb_group_test_" & $getMonoTime().ticks
db = newLSMTree(tmpDir)
ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE sales (id INT, dept TEXT, amount INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO sales (id, dept, amount) VALUES (1, 'A', 100)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO sales (id, dept, amount) VALUES (2, 'A', 200)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO sales (id, dept, amount) VALUES (3, 'B', 150)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO sales (id, dept, amount) VALUES (4, 'B', 50)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO sales (id, dept, amount) VALUES (5, 'C', 300)"))
teardown:
removeDir(tmpDir)
test "GROUP BY with COUNT(*)":
let r = qexec.executeQuery(ctx, parse("SELECT dept, COUNT(*) AS cnt FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
test "GROUP BY with SUM":
let r = qexec.executeQuery(ctx, parse("SELECT dept, SUM(amount) AS total FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
var foundA = false
for row in r.rows:
if row["dept"] == "A":
check row["total"] == "300.0"
foundA = true
check foundA
test "GROUP BY with AVG":
let r = qexec.executeQuery(ctx, parse("SELECT dept, AVG(amount) AS avg_amt FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
var foundB = false
for row in r.rows:
if row["dept"] == "B":
check row["avg_amt"] == "100.0"
foundB = true
check foundB
test "GROUP BY with MIN and MAX":
let r = qexec.executeQuery(ctx, parse("SELECT dept, MIN(amount) AS lo, MAX(amount) AS hi FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
var foundA = false
for row in r.rows:
if row["dept"] == "A":
check row["lo"] == "100"
check row["hi"] == "200"
foundA = true
check foundA
test "GROUP BY with HAVING":
let r = qexec.executeQuery(ctx, parse("SELECT dept, SUM(amount) AS total FROM sales GROUP BY dept HAVING SUM(amount) > 200"))
check r.success
check r.rows.len == 2 # A (300) and C (300)
for row in r.rows:
check row["dept"] in @["A", "C"]
test "GROUP BY with multiple aggregates":
let r = qexec.executeQuery(ctx, parse("SELECT dept, COUNT(*) AS cnt, SUM(amount) AS total, AVG(amount) AS avg_amt FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
test "COUNT with FILTER (WHERE ...)":
let r = qexec.executeQuery(ctx, parse("SELECT COUNT(*) AS total, COUNT(*) FILTER (WHERE amount > 100) AS big FROM sales"))
check r.success
check r.rows.len == 1
check r.rows[0]["total"] == "5"
check r.rows[0]["big"] == "3"
test "SUM with FILTER (WHERE ...)":
let r = qexec.executeQuery(ctx, parse("SELECT dept, SUM(amount) FILTER (WHERE amount > 100) AS big_total FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
var foundA = false
for row in r.rows:
if row["dept"] == "A":
check row["big_total"] == "200.0"
foundA = true
check foundA
test "ARRAY_AGG":
let r = qexec.executeQuery(ctx, parse("SELECT dept, ARRAY_AGG(amount) AS amounts FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
var foundA = false
for row in r.rows:
if row["dept"] == "A":
check "100" in row["amounts"]
check "200" in row["amounts"]
foundA = true
check foundA
test "STRING_AGG":
let r = qexec.executeQuery(ctx, parse("SELECT dept, STRING_AGG(amount, ',') AS vals FROM sales GROUP BY dept"))
check r.success
check r.rows.len == 3
test "ROLLUP":
let r = qexec.executeQuery(ctx, parse("SELECT dept, SUM(amount) AS total FROM sales GROUP BY ROLLUP (dept)"))
check r.success
# 3 dept groups + 1 grand total = 4 rows
check r.rows.len == 4
test "PIVOT":
discard qexec.executeQuery(ctx, parse("CREATE TABLE emp (name TEXT, dept TEXT, salary INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO emp (name, dept, salary) VALUES ('Alice', 'Eng', 90000)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO emp (name, dept, salary) VALUES ('Bob', 'Eng', 80000)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO emp (name, dept, salary) VALUES ('Charlie', 'Sales', 70000)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO emp (name, dept, salary) VALUES ('Diana', 'Sales', 75000)"))
let r = qexec.executeQuery(ctx, parse("SELECT * FROM (SELECT name, dept, salary FROM emp) PIVOT (SUM(salary) FOR dept IN ('Eng', 'Sales'))"))
check r.success
check r.rows.len == 4 # one row per employee
test "CUBE execution":
let r = qexec.executeQuery(ctx, parse("SELECT dept, SUM(amount) AS total FROM sales GROUP BY CUBE (dept)"))
check r.success
# 3 dept groups + 1 grand total = 4 rows
check r.rows.len == 4
test "GROUPING SETS execution":
let r = qexec.executeQuery(ctx, parse("SELECT dept, SUM(amount) AS total FROM sales GROUP BY GROUPING SETS ((dept), ())"))
check r.success
# 3 dept groups + 1 grand total = 4 rows
check r.rows.len == 4
test "UNPIVOT execution":
discard qexec.executeQuery(ctx, parse("CREATE TABLE pivoted (name TEXT, eng_salary INT, sales_salary INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO pivoted (name, eng_salary, sales_salary) VALUES ('Alice', 90000, 0)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO pivoted (name, eng_salary, sales_salary) VALUES ('Bob', 0, 70000)"))
let r = qexec.executeQuery(ctx, parse("SELECT * FROM pivoted UNPIVOT (salary FOR dept IN (eng_salary, sales_salary))"))
check r.success
check r.rows.len == 4
# JOIN tests # JOIN tests
include "join_tests" include "join_tests"
# TLA+ faithfulness tests # TLA+ faithfulness tests
include "tla_faithfulness" include "tla_faithfulness"
suite "MERGE Statement":
var db: LSMTree
var ctx: qexec.ExecutionContext
var tmpDir: string
setup:
tmpDir = getTempDir() / "baradb_merge_test_" & $getMonoTime().ticks
db = newLSMTree(tmpDir)
ctx = qexec.newExecutionContext(db)
discard qexec.executeQuery(ctx, parse("CREATE TABLE inventory (id INT PRIMARY KEY, sku TEXT, qty INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO inventory (id, sku, qty) VALUES (1, 'SKU001', 100)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO inventory (id, sku, qty) VALUES (2, 'SKU002', 200)"))
discard qexec.executeQuery(ctx, parse("CREATE TABLE updates (sku TEXT, delta INT)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU001', 50)"))
discard qexec.executeQuery(ctx, parse("INSERT INTO updates (sku, delta) VALUES ('SKU003', 300)"))
teardown:
removeDir(tmpDir)
test "MERGE WHEN MATCHED UPDATE":
let r = qexec.executeQuery(ctx, parse("""
MERGE INTO inventory AS target
USING updates AS source
ON target.sku = source.sku
WHEN MATCHED THEN UPDATE SET qty = target.qty + source.delta
"""))
check r.success
check r.affectedRows == 1
let verify = qexec.executeQuery(ctx, parse("SELECT * FROM inventory WHERE sku = 'SKU001'"))
check verify.rows[0]["qty"] == "150.0"
test "MERGE WHEN NOT MATCHED INSERT":
let r = qexec.executeQuery(ctx, parse("""
MERGE INTO inventory AS target
USING updates AS source
ON target.sku = source.sku
WHEN NOT MATCHED THEN INSERT (id, sku, qty) VALUES (3, source.sku, source.delta)
"""))
check r.success
check r.affectedRows == 1
let verify = qexec.executeQuery(ctx, parse("SELECT * FROM inventory WHERE sku = 'SKU003'"))
check verify.rows.len == 1
check verify.rows[0]["qty"] == "300"
suite "Vector SQL Integration":
var db: LSMTree
var ctx: qexec.ExecutionContext
var tmpDir: string
setup:
tmpDir = getTempDir() / "baradb_vector_test_" & $getMonoTime().ticks
db = newLSMTree(tmpDir)
ctx = qexec.newExecutionContext(db)
teardown:
removeDir(tmpDir)
test "CREATE TABLE with VECTOR column":
let r = qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
check r.success
let tbl = ctx.tables["items"]
check tbl.columns.len == 2
check tbl.columns[1].colType == "VECTOR(3)"
test "INSERT vector values":
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
check r.success
check r.affectedRows == 1
let r2 = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
check r2.success
let sel = qexec.executeQuery(ctx, parse("SELECT * FROM items"))
check sel.rows.len == 2
test "SELECT with cosine_distance":
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
let r = qexec.executeQuery(ctx, parse("SELECT id, cosine_distance(embedding, '[1.0, 0.0, 0.0]') AS dist FROM items"))
check r.success
check r.rows.len == 2
check r.rows[0]["dist"] == "0.0"
check r.rows[1]["dist"] == "1.0"
test "SELECT with <-> operator":
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
let r = qexec.executeQuery(ctx, parse("SELECT id, embedding <-> '[1.0, 0.0, 0.0]' AS dist FROM items"))
check r.success
check r.rows.len == 2
check r.rows[0]["dist"] == "0.0"
check r.rows[1]["dist"] == "1.4142135623730951"
test "ORDER BY cosine_distance":
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (3, '[0.5, 0.5, 0.0]')"))
let r = qexec.executeQuery(ctx, parse("SELECT id FROM items ORDER BY cosine_distance(embedding, '[1.0, 0.0, 0.0]') ASC"))
check r.success
check r.rows.len == 3
check r.rows[0]["id"] == "1"
check r.rows[1]["id"] == "3"
check r.rows[2]["id"] == "2"
test "CREATE VECTOR INDEX":
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0, 0.0]')"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[0.0, 1.0, 0.0]')"))
let r = qexec.executeQuery(ctx, parse("CREATE INDEX idx_items_vec ON items(embedding) USING hnsw"))
check r.success
check r.message.contains("HNSW")
check ctx.vectorIndexes.hasKey("items.embedding")
test "Vector dimension validation":
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
let r = qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[1.0, 0.0]')"))
check not r.success # Should fail due to dimension mismatch
test "euclidean_distance function":
discard qexec.executeQuery(ctx, parse("CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(3))"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (1, '[0.0, 0.0, 0.0]')"))
discard qexec.executeQuery(ctx, parse("INSERT INTO items (id, embedding) VALUES (2, '[1.0, 1.0, 1.0]')"))
let r = qexec.executeQuery(ctx, parse("SELECT id, euclidean_distance(embedding, '[0.0, 0.0, 0.0]') AS dist FROM items"))
check r.success
check r.rows.len == 2
check r.rows[0]["dist"] == "0.0"
check r.rows[1]["dist"] == "1.7320508075688772"
File diff suppressed because it is too large Load Diff
BIN
View File
Binary file not shown.