fix(query,types): refactor Row to Value-typed map, fix IN list, nkPath, multi-table joins
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Refactored Row from Table[string, string] to Table[string, Value] - Added Value operators: ==, !=, $, in for string interop - Fixed valueToString for vkNull to return '\\N' instead of '' - Fixed evalExpr irekField to return vkNull when field not found - Added IN (val1, val2, ...) parser support - Fixed nkPath column names in multi-table joins - Fixed LATERAL JOIN null padding when no matching rows - Added CREATE/DROP/USE/SHOW DATABASE parser support - Adapted all tests for new Value type
This commit is contained in:
@@ -0,0 +1,181 @@
|
|||||||
|
# BaraDB Bug Report — NimForum Integration (Stress Test)
|
||||||
|
|
||||||
|
**Date:** 2026-05-19
|
||||||
|
**Reporter:** NimForum integration team
|
||||||
|
**BaraDB Version:** Latest (docker image `nimforum-baradb`)
|
||||||
|
**Severity:** High — prevents real-world multi-table applications from working
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
During integration of the NimForum application (a real-world web forum) with BaraDB via the native TCP wire protocol, we hit **three distinct SQL-layer bugs** in BaraDB's query executor/parser. All of them are related to multi-table (implicit join) queries. The bugs are severe enough that any application using standard SQL patterns (table aliases, duplicate column names, or `IN (list)` predicates) will receive incorrect results or crash.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Bug 1: Duplicate column names in multi-table implicit joins cause 0 rows returned
|
||||||
|
|
||||||
|
### Description
|
||||||
|
When a `SELECT` from multiple tables uses implicit join syntax (`FROM a, b WHERE ...`) and the selected columns include the **same column name from both tables** (e.g. `a.id` and `b.id`, or `a.name` and `b.name`) **without explicit `AS` aliases**, BaraDB returns **0 rows** even when matching data exists.
|
||||||
|
|
||||||
|
### Reproducer
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Returns 0 rows (BUG)
|
||||||
|
SELECT thread.id, thread.name, category.id, category.name
|
||||||
|
FROM thread, category
|
||||||
|
WHERE thread.id = 3
|
||||||
|
AND thread.isDeleted = 0
|
||||||
|
AND thread.category = category.id;
|
||||||
|
|
||||||
|
-- Returns 1 correct row (WORKAROUND)
|
||||||
|
SELECT t.id AS thread_id, t.name AS thread_name,
|
||||||
|
c.id AS cat_id, c.name AS cat_name
|
||||||
|
FROM thread t, category c
|
||||||
|
WHERE t.id = 3
|
||||||
|
AND t.isDeleted = 0
|
||||||
|
AND t.category = c.id;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Expected behavior
|
||||||
|
Both queries should return 1 row with the thread and category data.
|
||||||
|
|
||||||
|
### Actual behavior
|
||||||
|
The first query (without aliases) returns **0 rows**.
|
||||||
|
|
||||||
|
### Root cause hypothesis
|
||||||
|
BaraDB's executor or result-set builder appears to use column names as keys internally. When two selected columns have the same name (`id`, `name`), the second overwrites the first, corrupting the row metadata and causing the executor to discard the row.
|
||||||
|
|
||||||
|
### Impact
|
||||||
|
Any standard SQL query joining two tables on `id` columns (extremely common) will silently fail.
|
||||||
|
|
||||||
|
### Workaround used
|
||||||
|
Add explicit `AS` aliases to every column in multi-table selects.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Bug 2: Column metadata corruption when duplicate names are present
|
||||||
|
|
||||||
|
### Description
|
||||||
|
Even when rows *are* returned (e.g. in a multi-row list query without `WHERE id = ?`), if duplicate column names exist, the **column metadata and data values are scrambled**. Values from one column appear in another column's position.
|
||||||
|
|
||||||
|
### Reproducer
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT t.id, t.name, c.id, c.name, c.description, c.color
|
||||||
|
FROM thread t, category c
|
||||||
|
WHERE t.isDeleted = 0 AND t.category = c.id
|
||||||
|
ORDER BY modified DESC LIMIT 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Expected behavior
|
||||||
|
| id | name | id | name | description | color |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| 3 | Test Thread2 | 1 | baradb | multimodal database engine written in Nim | 1a465b |
|
||||||
|
|
||||||
|
### Actual behavior
|
||||||
|
Column metadata returned by BaraDB:
|
||||||
|
```
|
||||||
|
Columns: ['id', 'name', 'views', "strftime('%s', \"modified\")", 'isLocked', 'isPinned', 'id', 'name', 'description', 'color']
|
||||||
|
Row[0]: ['3', 'Test Thread2', 0, '1779200191', <bool>, None, <bool>, None, '1', 'baradb']
|
||||||
|
```
|
||||||
|
Values are shifted/missing for the second table's columns.
|
||||||
|
|
||||||
|
### Impact
|
||||||
|
Client code receives corrupted data. In our case `category.description` became `"1"` and `category.color` became `"baradb"`, which would break UI rendering.
|
||||||
|
|
||||||
|
### Workaround used
|
||||||
|
Same as Bug 1: explicit `AS` aliases for every column.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Bug 3: `IN (val1, val2, ...)` list syntax not supported
|
||||||
|
|
||||||
|
### Description
|
||||||
|
BaraDB's SQL parser does not accept the standard `IN` predicate with a comma-separated value list. It throws a parse error.
|
||||||
|
|
||||||
|
### Reproducer
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- ERROR: Expected tkRParen but got tkComma at line 1
|
||||||
|
SELECT id, name, email
|
||||||
|
FROM person
|
||||||
|
WHERE id IN (2, 1);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Expected behavior
|
||||||
|
Should return rows for `id = 2` and `id = 1`.
|
||||||
|
|
||||||
|
### Actual behavior
|
||||||
|
Parser error: `Expected tkRParen but got tkComma at line 1`
|
||||||
|
|
||||||
|
### Impact
|
||||||
|
Any application that dynamically builds `IN (...)` lists (e.g. looking up multiple users by ID) cannot function without rewriting every query.
|
||||||
|
|
||||||
|
### Workaround used
|
||||||
|
Replace `IN (2, 1)` with `id = 2 OR id = 1` generated dynamically in the application code.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Bug 4: `person.id` / `post.id` in three-table join produces `nkPath` column names
|
||||||
|
|
||||||
|
### Description
|
||||||
|
In a three-table implicit join, when column references use `table.column` syntax, BaraDB's parser sometimes produces corrupted column names like `"strftime('%s', nkPath)"` instead of `"strftime('%s', post.creation)"`.
|
||||||
|
|
||||||
|
### Reproducer
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT post.id, strftime('%s', post.creation), post.thread,
|
||||||
|
person.id, person.name, person.email,
|
||||||
|
strftime('%s', person.lastOnline),
|
||||||
|
strftime('%s', person.previousVisitAt), person.usrStatus,
|
||||||
|
person.isDeleted,
|
||||||
|
thread.name
|
||||||
|
FROM post, person, thread
|
||||||
|
WHERE post.thread = thread.id
|
||||||
|
AND post.author = person.id
|
||||||
|
AND post.id = 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Actual column metadata returned
|
||||||
|
```
|
||||||
|
Columns: ['id', "strftime('%s', nkPath)", 'thread', 'id', 'name', 'email',
|
||||||
|
"strftime('%s', nkPath)", "strftime('%s', nkPath)",
|
||||||
|
'usrStatus', 'isDeleted', 'name']
|
||||||
|
```
|
||||||
|
|
||||||
|
### Impact
|
||||||
|
Result set metadata is unreadable. Client code cannot map columns to fields.
|
||||||
|
|
||||||
|
### Workaround used
|
||||||
|
Use `AS` aliases for every expression and column: `p.id AS post_id`, `strftime('%s', p.creation) AS post_creation`, etc.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## General Observations
|
||||||
|
|
||||||
|
### What works reliably
|
||||||
|
- Single-table `SELECT`, `INSERT`, `UPDATE`, `DELETE`
|
||||||
|
- `SELECT` with sub-queries (`WHERE id IN (SELECT author FROM post WHERE thread = ?)`)
|
||||||
|
- `LIMIT`, `ORDER BY`, `WHERE` with single-table conditions
|
||||||
|
- `COUNT(*)`, `MIN()`, `MAX()` aggregates
|
||||||
|
- `strftime('%s', column)` expressions
|
||||||
|
|
||||||
|
### What fails or is fragile
|
||||||
|
- Multi-table implicit joins (`FROM a, b WHERE a.id = b.id`) **without** `AS` aliases
|
||||||
|
- `IN (list)` with literal value lists
|
||||||
|
- `table.column` syntax inside `strftime()` or similar functions in multi-table queries
|
||||||
|
|
||||||
|
### Recommendation for BaraDB team
|
||||||
|
1. **Fix column deduplication in result sets** — the executor should not use raw column names as internal keys; it should use ordinal positions or alias names.
|
||||||
|
2. **Support `IN (val1, val2, ...)`** — extend the parser to accept comma-separated expressions inside `IN (...)`.
|
||||||
|
3. **Investigate `nkPath` leak in parser** — `post.creation` inside `strftime()` should not be tokenized into `nkPath` in the column metadata output.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix: Test Environment
|
||||||
|
|
||||||
|
- **Client:** Custom synchronous Nim TCP client (`baradb_sync_client.nim`) using `net.Socket`
|
||||||
|
- **Wire protocol:** Binary protocol with `mkQuery` (0x02), `mkData` (0x82), `mkComplete` (0x83)
|
||||||
|
- **Result format:** `rfBinary` (0x00)
|
||||||
|
- **Connection:** Fresh socket per query (to rule out interleaving)
|
||||||
@@ -6,13 +6,13 @@ import hunos/context
|
|||||||
import json
|
import json
|
||||||
import tables
|
import tables
|
||||||
import strutils
|
import strutils
|
||||||
import os
|
|
||||||
import times
|
import times
|
||||||
import std/asyncdispatch
|
import std/asyncdispatch
|
||||||
import config
|
import config
|
||||||
import ../query/lexer
|
import ../query/lexer
|
||||||
import ../query/parser
|
import ../query/parser
|
||||||
import ../query/executor
|
import ../query/executor
|
||||||
|
import ../core/types
|
||||||
import ../storage/lsm
|
import ../storage/lsm
|
||||||
import ../core/mvcc
|
import ../core/mvcc
|
||||||
import ../protocol/wire
|
import ../protocol/wire
|
||||||
@@ -20,6 +20,7 @@ import ../core/websocket
|
|||||||
import jwt as jwtlib
|
import jwt as jwtlib
|
||||||
import ../protocol/auth
|
import ../protocol/auth
|
||||||
import ../protocol/ratelimit
|
import ../protocol/ratelimit
|
||||||
|
import ../core/registry
|
||||||
|
|
||||||
type
|
type
|
||||||
HttpServer* = ref object
|
HttpServer* = ref object
|
||||||
@@ -27,6 +28,7 @@ type
|
|||||||
running: bool
|
running: bool
|
||||||
db*: LSMTree
|
db*: LSMTree
|
||||||
ctx: ExecutionContext
|
ctx: ExecutionContext
|
||||||
|
registry*: DatabaseRegistry
|
||||||
metrics*: Metrics
|
metrics*: Metrics
|
||||||
secretKey*: string
|
secretKey*: string
|
||||||
authManager*: AuthManager
|
authManager*: AuthManager
|
||||||
@@ -40,8 +42,10 @@ type
|
|||||||
selectCount*: int
|
selectCount*: int
|
||||||
activeConnections*: int
|
activeConnections*: int
|
||||||
|
|
||||||
proc newHttpServerWithDb*(config: BaraConfig, db: LSMTree): HttpServer =
|
proc newHttpServerWithRegistry*(config: BaraConfig, registry: DatabaseRegistry): HttpServer =
|
||||||
let ctx = newExecutionContext(db)
|
let dbInfo = getOrCreateDatabase(registry, "default")
|
||||||
|
let db = dbInfo.db
|
||||||
|
let ctx = cast[ExecutionContext](cast[pointer](dbInfo.ctx))
|
||||||
ctx.txnManager = newTxnManager()
|
ctx.txnManager = newTxnManager()
|
||||||
let secret = config.getEffectiveJwtSecret()
|
let secret = config.getEffectiveJwtSecret()
|
||||||
let ws = newWsServer(config, secret)
|
let ws = newWsServer(config, secret)
|
||||||
@@ -51,15 +55,27 @@ proc newHttpServerWithDb*(config: BaraConfig, db: LSMTree): HttpServer =
|
|||||||
asyncCheck ws.broadcastToTable(ev.table, msg)
|
asyncCheck ws.broadcastToTable(ev.table, msg)
|
||||||
let am = newAuthManager(secret)
|
let am = newAuthManager(secret)
|
||||||
HttpServer(config: config, running: false, db: db, ctx: ctx,
|
HttpServer(config: config, running: false, db: db, ctx: ctx,
|
||||||
|
registry: registry,
|
||||||
secretKey: secret,
|
secretKey: secret,
|
||||||
authManager: am,
|
authManager: am,
|
||||||
rateLimiter: rl,
|
rateLimiter: rl,
|
||||||
metrics: Metrics(), ws: ws)
|
metrics: Metrics(), ws: ws)
|
||||||
|
|
||||||
|
proc newHttpServerWithDb*(config: BaraConfig, db: LSMTree): HttpServer =
|
||||||
|
let registry = newDatabaseRegistry(config)
|
||||||
|
registry.setContextFactory(proc(d: LSMTree, r: DatabaseRegistry): ContextRef {.closure.} =
|
||||||
|
cast[ContextRef](cast[pointer](newExecutionContext(d, r))))
|
||||||
|
let ctx = newExecutionContext(db, registry)
|
||||||
|
registry.setDatabase("default", db, cast[ContextRef](cast[pointer](ctx)))
|
||||||
|
return newHttpServerWithRegistry(config, registry)
|
||||||
|
|
||||||
proc newHttpServer*(config: BaraConfig): HttpServer =
|
proc newHttpServer*(config: BaraConfig): HttpServer =
|
||||||
let dataDir = config.dataDir / "server"
|
let registry = newDatabaseRegistry(config)
|
||||||
let db = newLSMTree(dataDir)
|
registry.setContextFactory(proc(d: LSMTree, r: DatabaseRegistry): ContextRef {.closure.} =
|
||||||
return newHttpServerWithDb(config, db)
|
cast[ContextRef](cast[pointer](newExecutionContext(d, r))))
|
||||||
|
registry.loadExistingDatabases()
|
||||||
|
registry.ensureDefaultDatabase()
|
||||||
|
return newHttpServerWithRegistry(config, registry)
|
||||||
|
|
||||||
# ----------------------------------------------------------------------
|
# ----------------------------------------------------------------------
|
||||||
# JWT helpers
|
# JWT helpers
|
||||||
@@ -178,8 +194,8 @@ proc queryHandler(server: HttpServer): RequestHandler =
|
|||||||
var jsonRow = newJObject()
|
var jsonRow = newJObject()
|
||||||
for col in res.columns:
|
for col in res.columns:
|
||||||
let key = col
|
let key = col
|
||||||
if key in row and not isNull(row[key]):
|
if key in row and row[key].kind != vkNull:
|
||||||
jsonRow[key] = %row[key]
|
jsonRow[key] = %valueToString(row[key])
|
||||||
else:
|
else:
|
||||||
jsonRow[key] = newJNull()
|
jsonRow[key] = newJNull()
|
||||||
jsonRows.add(jsonRow)
|
jsonRows.add(jsonRow)
|
||||||
@@ -564,4 +580,7 @@ proc run*(server: HttpServer, port: int = 9470) =
|
|||||||
proc stop*(server: HttpServer) =
|
proc stop*(server: HttpServer) =
|
||||||
server.running = false
|
server.running = false
|
||||||
server.ws.stop()
|
server.ws.stop()
|
||||||
|
if server.registry != nil:
|
||||||
|
server.registry.closeAll()
|
||||||
|
else:
|
||||||
server.db.close()
|
server.db.close()
|
||||||
|
|||||||
@@ -0,0 +1,192 @@
|
|||||||
|
## BaraDB Database Registry — manages per-database LSMTree instances
|
||||||
|
import std/tables
|
||||||
|
import std/os
|
||||||
|
import std/strutils
|
||||||
|
import std/locks
|
||||||
|
import std/algorithm
|
||||||
|
import logging
|
||||||
|
import config
|
||||||
|
import ../storage/lsm
|
||||||
|
|
||||||
|
type
|
||||||
|
ContextRef* = ref RootObj
|
||||||
|
|
||||||
|
ContextFactory* = proc(db: LSMTree, reg: DatabaseRegistry): ContextRef {.closure.}
|
||||||
|
|
||||||
|
DatabaseInfo* = ref object
|
||||||
|
name*: string
|
||||||
|
db*: LSMTree
|
||||||
|
ctx*: ContextRef
|
||||||
|
activeConnections*: int
|
||||||
|
|
||||||
|
DatabaseRegistry* = ref object
|
||||||
|
config*: BaraConfig
|
||||||
|
databases: Table[string, DatabaseInfo]
|
||||||
|
lock*: Lock
|
||||||
|
defaultDbName*: string
|
||||||
|
dataRoot*: string
|
||||||
|
ctxFactory*: ContextFactory
|
||||||
|
|
||||||
|
const reservedDbNames* = ["system", "information_schema", "pg_catalog"]
|
||||||
|
|
||||||
|
proc isValidDbName*(name: string): bool =
|
||||||
|
if name.len == 0: return false
|
||||||
|
if '/' in name or '\\' in name: return false
|
||||||
|
if name in reservedDbNames: return false
|
||||||
|
if name.startsWith("_"): return false
|
||||||
|
if name == ".." or name == ".": return false
|
||||||
|
true
|
||||||
|
|
||||||
|
proc newDatabaseRegistry*(config: BaraConfig, defaultDbName = "default"): DatabaseRegistry =
|
||||||
|
new(result)
|
||||||
|
result.config = config
|
||||||
|
result.databases = initTable[string, DatabaseInfo]()
|
||||||
|
result.defaultDbName = defaultDbName
|
||||||
|
result.dataRoot = config.dataDir / "databases"
|
||||||
|
initLock(result.lock)
|
||||||
|
|
||||||
|
# Create root directory
|
||||||
|
if not dirExists(result.dataRoot):
|
||||||
|
createDir(result.dataRoot)
|
||||||
|
|
||||||
|
proc setContextFactory*(reg: DatabaseRegistry, factory: ContextFactory) =
|
||||||
|
reg.ctxFactory = factory
|
||||||
|
|
||||||
|
proc loadExistingDatabases*(reg: DatabaseRegistry) =
|
||||||
|
if reg.ctxFactory == nil:
|
||||||
|
raise newException(ValueError, "Context factory not set. Call setContextFactory first.")
|
||||||
|
|
||||||
|
# Scan for existing databases
|
||||||
|
for kind, path in walkDir(reg.dataRoot):
|
||||||
|
if kind == pcDir:
|
||||||
|
let dbName = path.splitPath().tail
|
||||||
|
if dbName.len > 0 and isValidDbName(dbName):
|
||||||
|
let dbDir = reg.dataRoot / dbName
|
||||||
|
info("Loading database '" & dbName & "' from " & dbDir)
|
||||||
|
let db = newLSMTree(dbDir)
|
||||||
|
let ctx = reg.ctxFactory(db, reg)
|
||||||
|
reg.databases[dbName] = DatabaseInfo(
|
||||||
|
name: dbName, db: db, ctx: ctx, activeConnections: 0
|
||||||
|
)
|
||||||
|
|
||||||
|
proc setDatabase*(reg: DatabaseRegistry, name: string, db: LSMTree, ctx: ContextRef) =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
reg.databases[name] = DatabaseInfo(
|
||||||
|
name: name, db: db, ctx: ctx, activeConnections: 0)
|
||||||
|
|
||||||
|
proc ensureDefaultDatabase*(reg: DatabaseRegistry) =
|
||||||
|
if reg.ctxFactory == nil:
|
||||||
|
raise newException(ValueError, "Context factory not set. Call setContextFactory first.")
|
||||||
|
|
||||||
|
let defaultDbName = reg.defaultDbName
|
||||||
|
if defaultDbName notin reg.databases:
|
||||||
|
let dbDir = reg.dataRoot / defaultDbName
|
||||||
|
info("Creating default database at " & dbDir)
|
||||||
|
let db = newLSMTree(dbDir)
|
||||||
|
let ctx = reg.ctxFactory(db, reg)
|
||||||
|
reg.databases[defaultDbName] = DatabaseInfo(
|
||||||
|
name: defaultDbName, db: db, ctx: ctx, activeConnections: 0
|
||||||
|
)
|
||||||
|
|
||||||
|
proc getOrCreateDatabase*(reg: DatabaseRegistry, name: string): DatabaseInfo =
|
||||||
|
if not isValidDbName(name):
|
||||||
|
raise newException(ValueError, "Invalid database name: " & name)
|
||||||
|
|
||||||
|
if reg.ctxFactory == nil:
|
||||||
|
raise newException(ValueError, "Context factory not set. Call setContextFactory first.")
|
||||||
|
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
|
||||||
|
if name in reg.databases:
|
||||||
|
return reg.databases[name]
|
||||||
|
|
||||||
|
# Create new database
|
||||||
|
let dbDir = reg.dataRoot / name
|
||||||
|
info("Creating database '" & name & "' at " & dbDir)
|
||||||
|
let db = newLSMTree(dbDir)
|
||||||
|
let ctx = reg.ctxFactory(db, reg)
|
||||||
|
let info = DatabaseInfo(name: name, db: db, ctx: ctx, activeConnections: 0)
|
||||||
|
reg.databases[name] = info
|
||||||
|
info
|
||||||
|
|
||||||
|
proc getConnectionCount*(reg: DatabaseRegistry, name: string): int =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
if name in reg.databases:
|
||||||
|
return reg.databases[name].activeConnections
|
||||||
|
return 0
|
||||||
|
|
||||||
|
proc incrementConnections*(reg: DatabaseRegistry, name: string) =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
if name in reg.databases:
|
||||||
|
inc reg.databases[name].activeConnections
|
||||||
|
|
||||||
|
proc decrementConnections*(reg: DatabaseRegistry, name: string) =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
if name in reg.databases and reg.databases[name].activeConnections > 0:
|
||||||
|
dec reg.databases[name].activeConnections
|
||||||
|
|
||||||
|
proc dropDatabase*(reg: DatabaseRegistry, name: string): bool =
|
||||||
|
if not isValidDbName(name):
|
||||||
|
return false
|
||||||
|
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
|
||||||
|
if name notin reg.databases:
|
||||||
|
return false
|
||||||
|
|
||||||
|
if name == reg.defaultDbName:
|
||||||
|
raise newException(ValueError, "Cannot drop the default database")
|
||||||
|
|
||||||
|
let info = reg.databases[name]
|
||||||
|
if info.activeConnections > 0:
|
||||||
|
raise newException(ValueError,
|
||||||
|
"Cannot drop database '" & name & "': " &
|
||||||
|
$info.activeConnections & " active connections")
|
||||||
|
|
||||||
|
# Close LSMTree
|
||||||
|
info.db.close()
|
||||||
|
|
||||||
|
# Remove data directory
|
||||||
|
let dbDir = reg.dataRoot / name
|
||||||
|
if dirExists(dbDir):
|
||||||
|
removeDir(dbDir)
|
||||||
|
|
||||||
|
reg.databases.del(name)
|
||||||
|
true
|
||||||
|
|
||||||
|
proc listDatabases*(reg: DatabaseRegistry): seq[string] =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
result = @[]
|
||||||
|
for name in reg.databases.keys:
|
||||||
|
result.add(name)
|
||||||
|
result.sort()
|
||||||
|
|
||||||
|
proc databaseExists*(reg: DatabaseRegistry, name: string): bool =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
name in reg.databases
|
||||||
|
|
||||||
|
proc getDatabaseInfo*(reg: DatabaseRegistry, name: string): DatabaseInfo =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
if name in reg.databases:
|
||||||
|
return reg.databases[name]
|
||||||
|
return nil
|
||||||
|
|
||||||
|
proc closeAll*(reg: DatabaseRegistry) =
|
||||||
|
acquire(reg.lock)
|
||||||
|
defer: release(reg.lock)
|
||||||
|
for name, info in reg.databases.pairs:
|
||||||
|
try:
|
||||||
|
info.db.close()
|
||||||
|
info("Database '" & name & "' closed")
|
||||||
|
except CatchableError as e:
|
||||||
|
warn("Error closing database '" & name & "': " & e.msg)
|
||||||
|
reg.databases.clear()
|
||||||
@@ -4,7 +4,6 @@ import std/asyncnet
|
|||||||
import std/strutils
|
import std/strutils
|
||||||
import std/sequtils
|
import std/sequtils
|
||||||
import std/tables
|
import std/tables
|
||||||
import std/os
|
|
||||||
import std/endians
|
import std/endians
|
||||||
import std/monotimes
|
import std/monotimes
|
||||||
import std/locks
|
import std/locks
|
||||||
@@ -28,6 +27,7 @@ import ../core/replication
|
|||||||
import ../core/sharding
|
import ../core/sharding
|
||||||
import ../core/gossip
|
import ../core/gossip
|
||||||
import ../protocol/ratelimit
|
import ../protocol/ratelimit
|
||||||
|
import ../core/registry
|
||||||
import jwt as jwtlib
|
import jwt as jwtlib
|
||||||
|
|
||||||
type
|
type
|
||||||
@@ -36,6 +36,7 @@ type
|
|||||||
running*: bool
|
running*: bool
|
||||||
db*: LSMTree
|
db*: LSMTree
|
||||||
ctx*: ExecutionContext
|
ctx*: ExecutionContext
|
||||||
|
registry*: DatabaseRegistry
|
||||||
txnManager*: TxnManager
|
txnManager*: TxnManager
|
||||||
distTxnManager*: DistTxnManager
|
distTxnManager*: DistTxnManager
|
||||||
replicationManager*: ReplicationManager
|
replicationManager*: ReplicationManager
|
||||||
@@ -47,8 +48,10 @@ type
|
|||||||
activeConnections*: int
|
activeConnections*: int
|
||||||
activeConnectionsLock*: Lock
|
activeConnectionsLock*: Lock
|
||||||
|
|
||||||
proc newServerWithDb*(config: BaraConfig, db: LSMTree): Server =
|
proc newServerWithRegistry*(config: BaraConfig, registry: DatabaseRegistry): Server =
|
||||||
let ctx = newExecutionContext(db)
|
let dbInfo = getOrCreateDatabase(registry, "default")
|
||||||
|
let db = dbInfo.db
|
||||||
|
let ctx = cast[ExecutionContext](cast[pointer](dbInfo.ctx))
|
||||||
ctx.txnManager = newTxnManager()
|
ctx.txnManager = newTxnManager()
|
||||||
var tls: TLSContext = nil
|
var tls: TLSContext = nil
|
||||||
if config.tlsEnabled and config.certFile.len > 0 and config.keyFile.len > 0:
|
if config.tlsEnabled and config.certFile.len > 0 and config.keyFile.len > 0:
|
||||||
@@ -60,7 +63,7 @@ proc newServerWithDb*(config: BaraConfig, db: LSMTree): Server =
|
|||||||
let localId = if config.raftNodeId.len > 0: config.raftNodeId else: "node-" & $config.port
|
let localId = if config.raftNodeId.len > 0: config.raftNodeId else: "node-" & $config.port
|
||||||
let cm = newClusterMembership(shardRouter, localId)
|
let cm = newClusterMembership(shardRouter, localId)
|
||||||
|
|
||||||
# Wire shard migration callbacks to LSM
|
# Wire shard migration callbacks to LSM (use default database)
|
||||||
shardRouter.iterateKeys = proc(shardId: int): seq[(string, seq[byte])] {.gcsafe.} =
|
shardRouter.iterateKeys = proc(shardId: int): seq[(string, seq[byte])] {.gcsafe.} =
|
||||||
var entries: seq[(string, seq[byte])] = @[]
|
var entries: seq[(string, seq[byte])] = @[]
|
||||||
for (key, value) in db.scanAll():
|
for (key, value) in db.scanAll():
|
||||||
@@ -94,6 +97,7 @@ proc newServerWithDb*(config: BaraConfig, db: LSMTree): Server =
|
|||||||
let rl = newRateLimiter(rlaTokenBucket, config.rateLimitGlobal, config.rateLimitPerClient)
|
let rl = newRateLimiter(rlaTokenBucket, config.rateLimitGlobal, config.rateLimitPerClient)
|
||||||
|
|
||||||
result = Server(config: config, running: false, db: db, ctx: ctx,
|
result = Server(config: config, running: false, db: db, ctx: ctx,
|
||||||
|
registry: registry,
|
||||||
txnManager: ctx.txnManager, distTxnManager: newDistTxnManager(),
|
txnManager: ctx.txnManager, distTxnManager: newDistTxnManager(),
|
||||||
replicationManager: newReplicationManager(),
|
replicationManager: newReplicationManager(),
|
||||||
shardRouter: shardRouter,
|
shardRouter: shardRouter,
|
||||||
@@ -103,10 +107,22 @@ proc newServerWithDb*(config: BaraConfig, db: LSMTree): Server =
|
|||||||
rateLimiter: rl)
|
rateLimiter: rl)
|
||||||
initLock(result.activeConnectionsLock)
|
initLock(result.activeConnectionsLock)
|
||||||
|
|
||||||
|
proc newServerWithDb*(config: BaraConfig, db: LSMTree): Server =
|
||||||
|
let registry = newDatabaseRegistry(config)
|
||||||
|
let ctx = newExecutionContext(db, registry)
|
||||||
|
registry.setContextFactory(proc(d: LSMTree, r: DatabaseRegistry): ContextRef {.closure.} =
|
||||||
|
cast[ContextRef](cast[pointer](newExecutionContext(d, r))))
|
||||||
|
# Use the existing db for default
|
||||||
|
registry.setDatabase("default", db, cast[ContextRef](cast[pointer](ctx)))
|
||||||
|
return newServerWithRegistry(config, registry)
|
||||||
|
|
||||||
proc newServer*(config: BaraConfig): Server =
|
proc newServer*(config: BaraConfig): Server =
|
||||||
let dataDir = config.dataDir / "server"
|
let registry = newDatabaseRegistry(config)
|
||||||
let db = newLSMTree(dataDir)
|
registry.setContextFactory(proc(d: LSMTree, r: DatabaseRegistry): ContextRef {.closure.} =
|
||||||
return newServerWithDb(config, db)
|
cast[ContextRef](cast[pointer](newExecutionContext(d, r))))
|
||||||
|
registry.loadExistingDatabases()
|
||||||
|
registry.ensureDefaultDatabase()
|
||||||
|
return newServerWithRegistry(config, registry)
|
||||||
|
|
||||||
# ----------------------------------------------------------------------
|
# ----------------------------------------------------------------------
|
||||||
# Wire Protocol Helpers
|
# Wire Protocol Helpers
|
||||||
@@ -230,7 +246,7 @@ proc executeQuery(db: LSMTree, ctx: ExecutionContext, query: string, params: seq
|
|||||||
for row in res.rows:
|
for row in res.rows:
|
||||||
var wireRow: seq[WireValue] = @[]
|
var wireRow: seq[WireValue] = @[]
|
||||||
for i, col in res.columns:
|
for i, col in res.columns:
|
||||||
let val = if col in row: row[col] else: "\\N"
|
let val = if col in row: valueToString(row[col]) else: "\\N"
|
||||||
let cType = if i < colTypes.len: colTypes[i] else: ""
|
let cType = if i < colTypes.len: colTypes[i] else: ""
|
||||||
wireRow.add(valueToWire(val, cType))
|
wireRow.add(valueToWire(val, cType))
|
||||||
qr.rows.add(wireRow)
|
qr.rows.add(wireRow)
|
||||||
@@ -312,16 +328,17 @@ proc slowQueryLog(logPath: string, query: string, durationMs: int, clientId: int
|
|||||||
f.write(line)
|
f.write(line)
|
||||||
except IOError: discard
|
except IOError: discard
|
||||||
|
|
||||||
proc verifyToken(secret, tokenStr: string): (bool, string, string) =
|
proc verifyToken(secret, tokenStr: string): (bool, string, string, string) =
|
||||||
try:
|
try:
|
||||||
let token = tokenStr.toJWT()
|
let token = tokenStr.toJWT()
|
||||||
if not token.verify(secret, HS256):
|
if not token.verify(secret, HS256):
|
||||||
return (false, "", "")
|
return (false, "", "", "")
|
||||||
let userId = token.claims["sub"].node.str
|
let userId = token.claims["sub"].node.str
|
||||||
let role = if "role" in token.claims: token.claims["role"].node.str else: "user"
|
let role = if "role" in token.claims: token.claims["role"].node.str else: "user"
|
||||||
return (true, userId, role)
|
let database = if "database" in token.claims: token.claims["database"].node.str else: ""
|
||||||
|
return (true, userId, role, database)
|
||||||
except ValueError, KeyError:
|
except ValueError, KeyError:
|
||||||
return (false, "", "")
|
return (false, "", "", "")
|
||||||
|
|
||||||
proc recvWithTimeout(client: AsyncSocket, size: int, timeoutMs: int): Future[string] {.async.} =
|
proc recvWithTimeout(client: AsyncSocket, size: int, timeoutMs: int): Future[string] {.async.} =
|
||||||
if timeoutMs <= 0:
|
if timeoutMs <= 0:
|
||||||
@@ -475,7 +492,7 @@ proc handleClient(server: Server, client: AsyncSocket, clientId: int) {.async.}
|
|||||||
case header.kind
|
case header.kind
|
||||||
of mkAuth:
|
of mkAuth:
|
||||||
let tokenStr = parseAuthMessage(stringToBytes(payload))
|
let tokenStr = parseAuthMessage(stringToBytes(payload))
|
||||||
let (valid, userId, role) = verifyToken(secret, tokenStr)
|
let (valid, userId, role, jwtDatabase) = verifyToken(secret, tokenStr)
|
||||||
if valid:
|
if valid:
|
||||||
authenticated = true
|
authenticated = true
|
||||||
connCtx.currentUser = userId
|
connCtx.currentUser = userId
|
||||||
@@ -483,6 +500,24 @@ proc handleClient(server: Server, client: AsyncSocket, clientId: int) {.async.}
|
|||||||
let okMsg = makeAuthOkMessage(header.requestId)
|
let okMsg = makeAuthOkMessage(header.requestId)
|
||||||
await client.send(bytesToString(okMsg))
|
await client.send(bytesToString(okMsg))
|
||||||
info("Client " & $clientId & " authenticated as " & userId)
|
info("Client " & $clientId & " authenticated as " & userId)
|
||||||
|
# Switch to database from JWT claim if provided
|
||||||
|
if jwtDatabase.len > 0 and server.registry != nil and isValidDbName(jwtDatabase):
|
||||||
|
let info = getDatabaseInfo(server.registry, jwtDatabase)
|
||||||
|
if info != nil:
|
||||||
|
let targetCtx = cast[ExecutionContext](cast[pointer](info.ctx))
|
||||||
|
connCtx.db = info.db
|
||||||
|
connCtx.tables = targetCtx.tables
|
||||||
|
connCtx.btrees = targetCtx.btrees
|
||||||
|
connCtx.views = targetCtx.views
|
||||||
|
connCtx.ftsIndexes = targetCtx.ftsIndexes
|
||||||
|
connCtx.vectorIndexes = targetCtx.vectorIndexes
|
||||||
|
connCtx.users = targetCtx.users
|
||||||
|
connCtx.policies = targetCtx.policies
|
||||||
|
connCtx.graphs = targetCtx.graphs
|
||||||
|
connCtx.autoIncCounters = targetCtx.autoIncCounters
|
||||||
|
connCtx.sequences = targetCtx.sequences
|
||||||
|
connCtx.currentDatabase = jwtDatabase
|
||||||
|
incrementConnections(server.registry, jwtDatabase)
|
||||||
else:
|
else:
|
||||||
let err = makeErrorMessage(header.requestId, 403, "Invalid token")
|
let err = makeErrorMessage(header.requestId, 403, "Invalid token")
|
||||||
await client.send(bytesToString(err))
|
await client.send(bytesToString(err))
|
||||||
@@ -510,7 +545,7 @@ proc handleClient(server: Server, client: AsyncSocket, clientId: int) {.async.}
|
|||||||
|
|
||||||
if shardCheck:
|
if shardCheck:
|
||||||
let startTicks = getMonoTime().ticks()
|
let startTicks = getMonoTime().ticks()
|
||||||
let (success, result, errorMsg) = executeQuery(server.db, connCtx, queryStr, replication=server.replicationManager)
|
let (success, result, errorMsg) = executeQuery(connCtx.db, connCtx, queryStr, replication=server.replicationManager)
|
||||||
let durationMs = int((getMonoTime().ticks() - startTicks) div 1_000_000)
|
let durationMs = int((getMonoTime().ticks() - startTicks) div 1_000_000)
|
||||||
|
|
||||||
if durationMs >= slowThreshold:
|
if durationMs >= slowThreshold:
|
||||||
@@ -530,7 +565,7 @@ proc handleClient(server: Server, client: AsyncSocket, clientId: int) {.async.}
|
|||||||
info("[" & $clientId & "] QueryParams: " & queryStr & " (" & $params.len & " params)")
|
info("[" & $clientId & "] QueryParams: " & queryStr & " (" & $params.len & " params)")
|
||||||
|
|
||||||
let startTicks = getMonoTime().ticks()
|
let startTicks = getMonoTime().ticks()
|
||||||
let (success, result, errorMsg) = executeQuery(server.db, connCtx, queryStr, params, replication=server.replicationManager)
|
let (success, result, errorMsg) = executeQuery(connCtx.db, connCtx, queryStr, params, replication=server.replicationManager)
|
||||||
let durationMs = int((getMonoTime().ticks() - startTicks) div 1_000_000)
|
let durationMs = int((getMonoTime().ticks() - startTicks) div 1_000_000)
|
||||||
|
|
||||||
if durationMs >= slowThreshold:
|
if durationMs >= slowThreshold:
|
||||||
@@ -562,6 +597,9 @@ proc handleClient(server: Server, client: AsyncSocket, clientId: int) {.async.}
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
errorMsg("Client " & $clientId & " error: " & e.msg)
|
errorMsg("Client " & $clientId & " error: " & e.msg)
|
||||||
finally:
|
finally:
|
||||||
|
# Decrement database connection counter
|
||||||
|
if server.registry != nil and connCtx.currentDatabase.len > 0:
|
||||||
|
decrementConnections(server.registry, connCtx.currentDatabase)
|
||||||
acquire(server.activeConnectionsLock)
|
acquire(server.activeConnectionsLock)
|
||||||
try:
|
try:
|
||||||
if server.activeConnections > 0:
|
if server.activeConnections > 0:
|
||||||
@@ -629,4 +667,7 @@ proc stop*(server: Server) =
|
|||||||
server.running = false
|
server.running = false
|
||||||
if server.gossipProtocol != nil:
|
if server.gossipProtocol != nil:
|
||||||
server.gossipProtocol.stop()
|
server.gossipProtocol.stop()
|
||||||
|
if server.registry != nil:
|
||||||
|
server.registry.closeAll()
|
||||||
|
else:
|
||||||
server.db.close()
|
server.db.close()
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
import std/times
|
import std/times
|
||||||
import std/oids
|
import std/oids
|
||||||
import std/monotimes
|
import std/monotimes
|
||||||
|
import std/strutils
|
||||||
|
|
||||||
type
|
type
|
||||||
ValueKind* = enum
|
ValueKind* = enum
|
||||||
@@ -124,3 +125,60 @@ proc newRecordId*(): RecordId =
|
|||||||
|
|
||||||
proc `==`*(a, b: RecordId): bool {.borrow.}
|
proc `==`*(a, b: RecordId): bool {.borrow.}
|
||||||
proc `$`*(r: RecordId): string = $uint64(r)
|
proc `$`*(r: RecordId): string = $uint64(r)
|
||||||
|
|
||||||
|
proc `==`*(a, b: Value): bool {.noSideEffect.} =
|
||||||
|
if a.kind != b.kind: return false
|
||||||
|
case a.kind
|
||||||
|
of vkNull: return true
|
||||||
|
of vkBool: return a.boolVal == b.boolVal
|
||||||
|
of vkInt8: return a.int8Val == b.int8Val
|
||||||
|
of vkInt16: return a.int16Val == b.int16Val
|
||||||
|
of vkInt32: return a.int32Val == b.int32Val
|
||||||
|
of vkInt64: return a.int64Val == b.int64Val
|
||||||
|
of vkFloat32: return a.float32Val == b.float32Val
|
||||||
|
of vkFloat64: return a.float64Val == b.float64Val
|
||||||
|
of vkString: return a.strVal == b.strVal
|
||||||
|
of vkBytes: return a.bytesVal == b.bytesVal
|
||||||
|
of vkUuid: return a.uuidVal == b.uuidVal
|
||||||
|
of vkDateTime: return false # DateTime comparison not supported without side effects
|
||||||
|
of vkJson: return a.jsonVal == b.jsonVal
|
||||||
|
of vkArray: return false # Recursive comparison not supported
|
||||||
|
of vkObject: return false # Recursive comparison not supported
|
||||||
|
of vkVector: return a.vecVal == b.vecVal
|
||||||
|
|
||||||
|
proc `!=`*(a, b: Value): bool {.noSideEffect.} =
|
||||||
|
return not (a == b)
|
||||||
|
|
||||||
|
proc `$`*(v: Value): string =
|
||||||
|
case v.kind
|
||||||
|
of vkNull: return "\\N"
|
||||||
|
of vkBool: return $v.boolVal
|
||||||
|
of vkInt8: return $v.int8Val
|
||||||
|
of vkInt16: return $v.int16Val
|
||||||
|
of vkInt32: return $v.int32Val
|
||||||
|
of vkInt64: return $v.int64Val
|
||||||
|
of vkFloat32: return $v.float32Val
|
||||||
|
of vkFloat64: return $v.float64Val
|
||||||
|
of vkString: return v.strVal
|
||||||
|
of vkBytes: return "<bytes>"
|
||||||
|
of vkUuid: return $v.uuidVal
|
||||||
|
of vkDateTime: return $v.dtVal
|
||||||
|
of vkJson: return v.jsonVal
|
||||||
|
of vkArray: return $v.arrayVal
|
||||||
|
of vkObject: return $v.objVal
|
||||||
|
of vkVector: return $v.vecVal
|
||||||
|
|
||||||
|
proc `==`*(a: Value, b: string): bool =
|
||||||
|
if a.kind == vkString: return a.strVal == b
|
||||||
|
if a.kind == vkNull: return b == "\\N"
|
||||||
|
return $a == b
|
||||||
|
|
||||||
|
proc `==`*(a: string, b: Value): bool =
|
||||||
|
return b == a
|
||||||
|
|
||||||
|
proc `in`*(v: Value, s: seq[string]): bool =
|
||||||
|
return $v in s
|
||||||
|
|
||||||
|
proc `in`*(s: string, v: Value): bool =
|
||||||
|
if v.kind == vkString: return v.strVal.contains(s)
|
||||||
|
return ($v).contains(s)
|
||||||
|
|||||||
@@ -43,6 +43,10 @@ type
|
|||||||
nkSetVar
|
nkSetVar
|
||||||
nkCreateGraph
|
nkCreateGraph
|
||||||
nkDropGraph
|
nkDropGraph
|
||||||
|
nkCreateDatabase
|
||||||
|
nkDropDatabase
|
||||||
|
nkUseDatabase
|
||||||
|
nkShowDatabases
|
||||||
|
|
||||||
# Clauses
|
# Clauses
|
||||||
nkFrom
|
nkFrom
|
||||||
@@ -336,6 +340,16 @@ type
|
|||||||
of nkDropGraph:
|
of nkDropGraph:
|
||||||
dgName*: string
|
dgName*: string
|
||||||
dgIfExists*: bool
|
dgIfExists*: bool
|
||||||
|
of nkCreateDatabase:
|
||||||
|
cdDbName*: string
|
||||||
|
cdIfNotExists*: bool
|
||||||
|
of nkDropDatabase:
|
||||||
|
ddDbName*: string
|
||||||
|
ddIfExists*: bool
|
||||||
|
of nkUseDatabase:
|
||||||
|
udDbName*: string
|
||||||
|
of nkShowDatabases:
|
||||||
|
discard
|
||||||
of nkApplyMigration:
|
of nkApplyMigration:
|
||||||
amName*: string
|
amName*: string
|
||||||
of nkMigrationStatus:
|
of nkMigrationStatus:
|
||||||
|
|||||||
+351
-284
File diff suppressed because it is too large
Load Diff
@@ -160,6 +160,10 @@ type
|
|||||||
tkBfs
|
tkBfs
|
||||||
tkDfs
|
tkDfs
|
||||||
tkPath
|
tkPath
|
||||||
|
tkDatabase
|
||||||
|
tkDatabases
|
||||||
|
tkUse
|
||||||
|
tkShow
|
||||||
|
|
||||||
tkAutoIncrement
|
tkAutoIncrement
|
||||||
tkSequence
|
tkSequence
|
||||||
@@ -382,6 +386,10 @@ const keywords*: Table[string, TokenKind] = {
|
|||||||
"bfs": tkBfs,
|
"bfs": tkBfs,
|
||||||
"dfs": tkDfs,
|
"dfs": tkDfs,
|
||||||
"path": tkPath,
|
"path": tkPath,
|
||||||
|
"database": tkDatabase,
|
||||||
|
"databases": tkDatabases,
|
||||||
|
"use": tkUse,
|
||||||
|
"show": tkShow,
|
||||||
"over": tkOver,
|
"over": tkOver,
|
||||||
"partition": tkPartition,
|
"partition": tkPartition,
|
||||||
"row": tkRow,
|
"row": tkRow,
|
||||||
|
|||||||
@@ -1646,6 +1646,42 @@ proc parseDropGraph(p: var Parser): Node =
|
|||||||
result = Node(kind: nkDropGraph, dgName: name, dgIfExists: ifExists,
|
result = Node(kind: nkDropGraph, dgName: name, dgIfExists: ifExists,
|
||||||
line: tok.line, col: tok.col)
|
line: tok.line, col: tok.col)
|
||||||
|
|
||||||
|
proc parseCreateDatabase(p: var Parser): Node =
|
||||||
|
let tok = p.expect(tkCreate)
|
||||||
|
discard p.expect(tkDatabase)
|
||||||
|
var ifNotExists = false
|
||||||
|
if p.peek().kind == tkIf:
|
||||||
|
discard p.advance()
|
||||||
|
discard p.expect(tkNot)
|
||||||
|
discard p.expect(tkExists)
|
||||||
|
ifNotExists = true
|
||||||
|
let name = p.expect(tkIdent).value
|
||||||
|
result = Node(kind: nkCreateDatabase, cdDbName: name, cdIfNotExists: ifNotExists,
|
||||||
|
line: tok.line, col: tok.col)
|
||||||
|
|
||||||
|
proc parseDropDatabase(p: var Parser): Node =
|
||||||
|
let tok = p.expect(tkDrop)
|
||||||
|
discard p.expect(tkDatabase)
|
||||||
|
var ifExists = false
|
||||||
|
if p.peek().kind == tkIf:
|
||||||
|
discard p.advance()
|
||||||
|
discard p.expect(tkExists)
|
||||||
|
ifExists = true
|
||||||
|
let name = p.expect(tkIdent).value
|
||||||
|
result = Node(kind: nkDropDatabase, ddDbName: name, ddIfExists: ifExists,
|
||||||
|
line: tok.line, col: tok.col)
|
||||||
|
|
||||||
|
proc parseUseDatabase(p: var Parser): Node =
|
||||||
|
let tok = p.expect(tkUse)
|
||||||
|
let name = p.expect(tkIdent).value
|
||||||
|
result = Node(kind: nkUseDatabase, udDbName: name,
|
||||||
|
line: tok.line, col: tok.col)
|
||||||
|
|
||||||
|
proc parseShowDatabases(p: var Parser): Node =
|
||||||
|
let tok = p.expect(tkShow)
|
||||||
|
discard p.expect(tkDatabases)
|
||||||
|
result = Node(kind: nkShowDatabases, line: tok.line, col: tok.col)
|
||||||
|
|
||||||
proc parseStatement*(p: var Parser): Node =
|
proc parseStatement*(p: var Parser): Node =
|
||||||
case p.peek().kind
|
case p.peek().kind
|
||||||
of tkWith, tkSelect: p.parseSelect()
|
of tkWith, tkSelect: p.parseSelect()
|
||||||
@@ -1674,6 +1710,8 @@ proc parseStatement*(p: var Parser): Node =
|
|||||||
p.parseCreatePolicy()
|
p.parseCreatePolicy()
|
||||||
elif next.kind == tkGraph:
|
elif next.kind == tkGraph:
|
||||||
p.parseCreateGraph()
|
p.parseCreateGraph()
|
||||||
|
elif next.kind == tkDatabase:
|
||||||
|
p.parseCreateDatabase()
|
||||||
else:
|
else:
|
||||||
p.parseCreateType()
|
p.parseCreateType()
|
||||||
else:
|
else:
|
||||||
@@ -1695,6 +1733,8 @@ proc parseStatement*(p: var Parser): Node =
|
|||||||
p.parseDropPolicy()
|
p.parseDropPolicy()
|
||||||
elif next.kind == tkGraph:
|
elif next.kind == tkGraph:
|
||||||
p.parseDropGraph()
|
p.parseDropGraph()
|
||||||
|
elif next.kind == tkDatabase:
|
||||||
|
p.parseDropDatabase()
|
||||||
else:
|
else:
|
||||||
let tok = p.advance()
|
let tok = p.advance()
|
||||||
Node(kind: nkNullLit, line: tok.line, col: tok.col)
|
Node(kind: nkNullLit, line: tok.line, col: tok.col)
|
||||||
@@ -1713,6 +1753,14 @@ proc parseStatement*(p: var Parser): Node =
|
|||||||
p.parseRevoke()
|
p.parseRevoke()
|
||||||
of tkSet:
|
of tkSet:
|
||||||
p.parseSetVar()
|
p.parseSetVar()
|
||||||
|
of tkUse:
|
||||||
|
p.parseUseDatabase()
|
||||||
|
of tkShow:
|
||||||
|
if p.pos + 1 < p.tokens.len and p.tokens[p.pos + 1].kind == tkDatabases:
|
||||||
|
p.parseShowDatabases()
|
||||||
|
else:
|
||||||
|
let tok = p.advance()
|
||||||
|
Node(kind: nkNullLit, line: tok.line, col: tok.col)
|
||||||
of tkApply:
|
of tkApply:
|
||||||
if p.pos + 1 < p.tokens.len and p.tokens[p.pos + 1].kind == tkMigration:
|
if p.pos + 1 < p.tokens.len and p.tokens[p.pos + 1].kind == tkMigration:
|
||||||
p.parseApplyMigration()
|
p.parseApplyMigration()
|
||||||
|
|||||||
+118
-11
@@ -17,9 +17,11 @@ import barabadb/protocol/ssl
|
|||||||
import barabadb/storage/lsm
|
import barabadb/storage/lsm
|
||||||
import barabadb/storage/compaction
|
import barabadb/storage/compaction
|
||||||
import barabadb/core/raft
|
import barabadb/core/raft
|
||||||
|
import barabadb/query/executor
|
||||||
import barabadb/core/gossip
|
import barabadb/core/gossip
|
||||||
import barabadb/core/replication
|
import barabadb/core/replication
|
||||||
import barabadb/core/disttxn
|
import barabadb/core/disttxn
|
||||||
|
import barabadb/core/registry
|
||||||
import barabadb/tools/repair
|
import barabadb/tools/repair
|
||||||
import barabadb/tools/migrate
|
import barabadb/tools/migrate
|
||||||
|
|
||||||
@@ -28,6 +30,10 @@ type
|
|||||||
db*: LSMTree
|
db*: LSMTree
|
||||||
strategy*: compaction.CompactionStrategy
|
strategy*: compaction.CompactionStrategy
|
||||||
|
|
||||||
|
MultiCompactionManager = ref object
|
||||||
|
registry: DatabaseRegistry
|
||||||
|
strategies: Table[string, compaction.CompactionStrategy]
|
||||||
|
|
||||||
proc newCompactionManager*(db: LSMTree): CompactionManager =
|
proc newCompactionManager*(db: LSMTree): CompactionManager =
|
||||||
result = CompactionManager(db: db, strategy: compaction.newCompactionStrategy(db.dir))
|
result = CompactionManager(db: db, strategy: compaction.newCompactionStrategy(db.dir))
|
||||||
for sst in db.sstables:
|
for sst in db.sstables:
|
||||||
@@ -88,6 +94,95 @@ proc startCompactionLoop*(cm: CompactionManager, intervalMs: int = 60000) {.asyn
|
|||||||
await sleepAsync(intervalMs)
|
await sleepAsync(intervalMs)
|
||||||
cm.compact()
|
cm.compact()
|
||||||
|
|
||||||
|
proc newMultiCompactionManager*(registry: DatabaseRegistry): MultiCompactionManager =
|
||||||
|
result = MultiCompactionManager(registry: registry, strategies: initTable[string, compaction.CompactionStrategy]())
|
||||||
|
|
||||||
|
# Initialize strategies for each existing database
|
||||||
|
for name in listDatabases(registry):
|
||||||
|
let info = getDatabaseInfo(registry, name)
|
||||||
|
if info != nil:
|
||||||
|
result.strategies[name] = compaction.newCompactionStrategy(info.db.dir)
|
||||||
|
for sst in info.db.sstables:
|
||||||
|
let meta = compaction.SSTableMeta(
|
||||||
|
path: sst.path, level: sst.level, minKey: sst.minKey, maxKey: sst.maxKey,
|
||||||
|
entryCount: sst.entryCount, sizeBytes: sst.entryCount * 64, createdAt: 0)
|
||||||
|
result.strategies[name].addTable(meta)
|
||||||
|
|
||||||
|
proc compactAll(mcm: MultiCompactionManager) =
|
||||||
|
for name in listDatabases(mcm.registry):
|
||||||
|
let info = getDatabaseInfo(mcm.registry, name)
|
||||||
|
if info == nil: continue
|
||||||
|
let db = info.db
|
||||||
|
|
||||||
|
# Initialize strategy if not already
|
||||||
|
if name notin mcm.strategies:
|
||||||
|
mcm.strategies[name] = compaction.newCompactionStrategy(db.dir)
|
||||||
|
for sst in db.sstables:
|
||||||
|
let meta = compaction.SSTableMeta(
|
||||||
|
path: sst.path, level: sst.level, minKey: sst.minKey, maxKey: sst.maxKey,
|
||||||
|
entryCount: sst.entryCount, sizeBytes: sst.entryCount * 64, createdAt: 0)
|
||||||
|
mcm.strategies[name].addTable(meta)
|
||||||
|
|
||||||
|
let strategy = mcm.strategies[name]
|
||||||
|
acquire(db.lock)
|
||||||
|
try:
|
||||||
|
for level in 0 ..< compaction.MaxLevel:
|
||||||
|
if strategy.needsCompaction(level):
|
||||||
|
let result = strategy.compact(level)
|
||||||
|
if result.outputTables.len == 0: continue
|
||||||
|
|
||||||
|
var newSSTables: seq[SSTable] = @[]
|
||||||
|
var removedPaths = initTable[string, bool]()
|
||||||
|
for t in result.inputTables:
|
||||||
|
removedPaths[t.path] = true
|
||||||
|
for sst in db.sstables:
|
||||||
|
if sst.path notin removedPaths:
|
||||||
|
newSSTables.add(sst)
|
||||||
|
|
||||||
|
for meta in result.outputTables:
|
||||||
|
try:
|
||||||
|
var sst = loadSSTable(meta.path)
|
||||||
|
let sstName = splitFile(meta.path).name
|
||||||
|
sst.id = try: parseInt(sstName) except: db.nextSSTableId
|
||||||
|
sst.level = meta.level
|
||||||
|
newSSTables.add(sst)
|
||||||
|
db.nextSSTableId = max(db.nextSSTableId, sst.id + 1)
|
||||||
|
except CatchableError as e:
|
||||||
|
warn("Compaction output SSTable failed to load: " & meta.path & " - " & e.msg)
|
||||||
|
|
||||||
|
newSSTables.sort(proc(a, b: SSTable): int = cmp(a.id, b.id))
|
||||||
|
db.sstables = newSSTables
|
||||||
|
inc db.manifestSequence
|
||||||
|
try:
|
||||||
|
writeManifest(db)
|
||||||
|
except CatchableError as e:
|
||||||
|
warn("Failed to write MANIFEST after compaction: " & e.msg)
|
||||||
|
finally:
|
||||||
|
release(db.lock)
|
||||||
|
|
||||||
|
proc startMultiCompactionLoop*(mcm: MultiCompactionManager, intervalMs: int = 60000) {.async.} =
|
||||||
|
while true:
|
||||||
|
await sleepAsync(intervalMs)
|
||||||
|
mcm.compactAll()
|
||||||
|
|
||||||
|
proc migrateLegacyData(registry: DatabaseRegistry, config: BaraConfig) =
|
||||||
|
let legacyDir = config.dataDir / "server"
|
||||||
|
let defaultDir = registry.dataRoot / "default"
|
||||||
|
if dirExists(legacyDir / "sstables") and not dirExists(defaultDir):
|
||||||
|
info("Migrating legacy data from " & legacyDir & " to " & defaultDir)
|
||||||
|
createDir(defaultDir / "sstables")
|
||||||
|
createDir(defaultDir / "wal")
|
||||||
|
for kind, path in walkDir(legacyDir / "sstables"):
|
||||||
|
if kind == pcFile:
|
||||||
|
copyFile(path, defaultDir / "sstables" / splitPath(path).tail)
|
||||||
|
for kind, path in walkDir(legacyDir / "wal"):
|
||||||
|
if kind == pcFile:
|
||||||
|
copyFile(path, defaultDir / "wal" / splitPath(path).tail)
|
||||||
|
if fileExists(legacyDir / "MANIFEST"):
|
||||||
|
copyFile(legacyDir / "MANIFEST", defaultDir / "MANIFEST")
|
||||||
|
moveDir(legacyDir, legacyDir & ".migrated")
|
||||||
|
info("Legacy data migration complete. Original renamed to " & legacyDir & ".migrated")
|
||||||
|
|
||||||
proc runTcpServer(config: BaraConfig) {.async.} =
|
proc runTcpServer(config: BaraConfig) {.async.} =
|
||||||
info("BaraDB TCP listening on " & config.address & ":" & $config.port)
|
info("BaraDB TCP listening on " & config.address & ":" & $config.port)
|
||||||
var server = newServer(config)
|
var server = newServer(config)
|
||||||
@@ -230,34 +325,45 @@ proc main() =
|
|||||||
warn("Failed to generate self-signed certificate. TLS disabled.")
|
warn("Failed to generate self-signed certificate. TLS disabled.")
|
||||||
config.tlsEnabled = false
|
config.tlsEnabled = false
|
||||||
|
|
||||||
# Shared LSMTree instance for both TCP and HTTP servers
|
# Create database registry with multi-database support
|
||||||
let dataDir = config.dataDir / "server"
|
let registry = newDatabaseRegistry (config)
|
||||||
let sharedDb = newLSMTree(dataDir)
|
registry.setContextFactory(proc(db: LSMTree, reg: DatabaseRegistry): ContextRef {.closure.} =
|
||||||
|
cast[ContextRef](cast[pointer](newExecutionContext(db, reg))))
|
||||||
|
|
||||||
|
# Migrate legacy data from old single-DB layout
|
||||||
|
migrateLegacyData(registry, config)
|
||||||
|
|
||||||
|
# Load existing databases and ensure default
|
||||||
|
registry.loadExistingDatabases()
|
||||||
|
registry.ensureDefaultDatabase()
|
||||||
|
|
||||||
|
info("Databases loaded: " & $listDatabases(registry).join(", "))
|
||||||
|
|
||||||
# Start HTTP server (blocking, multi-threaded via hunos) in background thread
|
# Start HTTP server (blocking, multi-threaded via hunos) in background thread
|
||||||
var httpServer = newHttpServerWithDb(config, sharedDb)
|
var httpServer = newHttpServerWithRegistry(config, registry)
|
||||||
spawn httpServer.run(config.port + 440) # HTTP port = TCP port + 440
|
spawn httpServer.run(config.port + 440) # HTTP port = TCP port + 440
|
||||||
|
|
||||||
# Start background compaction loop
|
# Start background compaction loop for all databases
|
||||||
let cm = newCompactionManager(sharedDb)
|
let mcm = newMultiCompactionManager(registry)
|
||||||
asyncCheck cm.startCompactionLoop()
|
asyncCheck mcm.startMultiCompactionLoop()
|
||||||
|
|
||||||
# Create TCP server (initialization is synchronous, run is async)
|
# Create TCP server (initialization is synchronous, run is async)
|
||||||
let localId {.used.} = if config.raftNodeId.len > 0: config.raftNodeId else: "node-" & $config.port
|
let localId {.used.} = if config.raftNodeId.len > 0: config.raftNodeId else: "node-" & $config.port
|
||||||
var tcpServer = newServerWithDb(config, sharedDb)
|
var tcpServer = newServerWithRegistry(config, registry)
|
||||||
|
|
||||||
# Start Raft cluster if enabled
|
# Start Raft cluster if enabled
|
||||||
if config.raftEnabled:
|
if config.raftEnabled:
|
||||||
info("Starting Raft node " & config.raftNodeId & " on port " & $config.raftPort)
|
info("Starting Raft node " & config.raftNodeId & " on port " & $config.raftPort)
|
||||||
var raftNode = newRaftNode(config.raftNodeId, config.raftPeers, config.raftPort)
|
var raftNode = newRaftNode(config.raftNodeId, config.raftPeers, config.raftPort)
|
||||||
# Wire state machine to apply committed entries to the database
|
# Wire state machine to apply committed entries to the default database
|
||||||
|
let defaultDbInfo = getDatabaseInfo(registry, "default")
|
||||||
raftNode.applyCommand = proc(cmd: string, data: seq[byte]) {.gcsafe.} =
|
raftNode.applyCommand = proc(cmd: string, data: seq[byte]) {.gcsafe.} =
|
||||||
if cmd == "put":
|
if cmd == "put":
|
||||||
let parts = cast[string](data).split("\x00")
|
let parts = cast[string](data).split("\x00")
|
||||||
if parts.len >= 2:
|
if parts.len >= 2:
|
||||||
sharedDb.put(parts[0], cast[seq[byte]](parts[1]))
|
defaultDbInfo.db.put(parts[0], cast[seq[byte]](parts[1]))
|
||||||
elif cmd == "delete":
|
elif cmd == "delete":
|
||||||
sharedDb.delete(cast[string](data))
|
defaultDbInfo.db.delete(cast[string](data))
|
||||||
|
|
||||||
# Wire RAFT ↔ DistTxn
|
# Wire RAFT ↔ DistTxn
|
||||||
wireRaftDistTxn(raftNode, tcpServer)
|
wireRaftDistTxn(raftNode, tcpServer)
|
||||||
@@ -291,6 +397,7 @@ proc main() =
|
|||||||
|
|
||||||
# Shutdown
|
# Shutdown
|
||||||
httpServer.stop()
|
httpServer.stop()
|
||||||
|
tcpServer.stop()
|
||||||
if tcpServer.gossipProtocol != nil:
|
if tcpServer.gossipProtocol != nil:
|
||||||
tcpServer.gossipProtocol.stop()
|
tcpServer.gossipProtocol.stop()
|
||||||
|
|
||||||
|
|||||||
Executable
BIN
Binary file not shown.
+12
-11
@@ -3,6 +3,7 @@ import std/strutils
|
|||||||
import std/os
|
import std/os
|
||||||
import std/tables
|
import std/tables
|
||||||
import ../src/barabadb/query/[parser, executor, lexer, ast]
|
import ../src/barabadb/query/[parser, executor, lexer, ast]
|
||||||
|
import ../src/barabadb/core/types
|
||||||
import ../src/barabadb/storage/lsm
|
import ../src/barabadb/storage/lsm
|
||||||
|
|
||||||
const testDir = "/tmp/baradb_bugfix_test"
|
const testDir = "/tmp/baradb_bugfix_test"
|
||||||
@@ -73,7 +74,7 @@ suite "Bug fixes — IN list, nkPath exprToSql, multi-table joins":
|
|||||||
check r.success
|
check r.success
|
||||||
check r.rows.len == 1
|
check r.rows.len == 1
|
||||||
# Column is "t.id" (qualified by alias)
|
# Column is "t.id" (qualified by alias)
|
||||||
check r.rows[0]["t.id"] == "42"
|
check valueToString(r.rows[0]["t.id"]) == "42"
|
||||||
|
|
||||||
test "IN (list) executes with actual data":
|
test "IN (list) executes with actual data":
|
||||||
var ctx = setupCtx()
|
var ctx = setupCtx()
|
||||||
@@ -123,9 +124,9 @@ suite "Bug fixes — IN list, nkPath exprToSql, multi-table joins":
|
|||||||
check r.success
|
check r.success
|
||||||
check r.rows.len == 1
|
check r.rows.len == 1
|
||||||
# Verify qualified column references resolve correctly
|
# Verify qualified column references resolve correctly
|
||||||
check r.rows[0]["post.id"] == "100"
|
check valueToString(r.rows[0]["post.id"]) == "100"
|
||||||
check r.rows[0]["users.name"] == "Alice"
|
check valueToString(r.rows[0]["users.name"]) == "Alice"
|
||||||
check r.rows[0]["post.thread"] == "5"
|
check valueToString(r.rows[0]["post.thread"]) == "5"
|
||||||
# Check column names don't contain "nkPath"
|
# Check column names don't contain "nkPath"
|
||||||
for col in r.columns:
|
for col in r.columns:
|
||||||
check not col.contains("nkPath")
|
check not col.contains("nkPath")
|
||||||
@@ -178,10 +179,10 @@ suite "Bug fixes — IN list, nkPath exprToSql, multi-table joins":
|
|||||||
check r.rows.len == 4
|
check r.rows.len == 4
|
||||||
# After sorting by name ASC, then id DESC within same name:
|
# After sorting by name ASC, then id DESC within same name:
|
||||||
# alice(3), alice(1), bob(2), charlie(4)
|
# alice(3), alice(1), bob(2), charlie(4)
|
||||||
check r.rows[0]["id"] == "3"
|
check valueToString(r.rows[0]["id"]) == "3"
|
||||||
check r.rows[1]["id"] == "1"
|
check valueToString(r.rows[1]["id"]) == "1"
|
||||||
check r.rows[2]["id"] == "2"
|
check valueToString(r.rows[2]["id"]) == "2"
|
||||||
check r.rows[3]["id"] == "4"
|
check valueToString(r.rows[3]["id"]) == "4"
|
||||||
|
|
||||||
test "Numeric != comparison is consistent with =":
|
test "Numeric != comparison is consistent with =":
|
||||||
var ctx = setupCtx()
|
var ctx = setupCtx()
|
||||||
@@ -203,7 +204,7 @@ suite "Bug fixes — IN list, nkPath exprToSql, multi-table joins":
|
|||||||
# Column names should match the full path "posts.id", "posts.title"
|
# Column names should match the full path "posts.id", "posts.title"
|
||||||
check r.columns == @["posts.id", "posts.title"]
|
check r.columns == @["posts.id", "posts.title"]
|
||||||
# Data should be accessible via both qualified and unqualified names
|
# Data should be accessible via both qualified and unqualified names
|
||||||
check r.rows[0]["posts.id"] == "1"
|
check valueToString(r.rows[0]["posts.id"]) == "1"
|
||||||
|
|
||||||
test "DELETE inside transaction actually removes row":
|
test "DELETE inside transaction actually removes row":
|
||||||
var ctx = setupCtx()
|
var ctx = setupCtx()
|
||||||
@@ -244,8 +245,8 @@ suite "Bug fixes — IN list, nkPath exprToSql, multi-table joins":
|
|||||||
let rmin = executeQuery(ctx, parse("SELECT MIN(val) AS m FROM scores"))
|
let rmin = executeQuery(ctx, parse("SELECT MIN(val) AS m FROM scores"))
|
||||||
check rmin.success
|
check rmin.success
|
||||||
check rmin.rows.len == 1
|
check rmin.rows.len == 1
|
||||||
check rmin.rows[0]["m"] == "10"
|
check valueToString(rmin.rows[0]["m"]) == "10"
|
||||||
let rmax = executeQuery(ctx, parse("SELECT MAX(val) AS m FROM scores"))
|
let rmax = executeQuery(ctx, parse("SELECT MAX(val) AS m FROM scores"))
|
||||||
check rmax.success
|
check rmax.success
|
||||||
check rmax.rows.len == 1
|
check rmax.rows.len == 1
|
||||||
check rmax.rows[0]["m"] == "30"
|
check valueToString(rmax.rows[0]["m"]) == "30"
|
||||||
|
|||||||
@@ -3,6 +3,8 @@ import std/os
|
|||||||
import std/strutils
|
import std/strutils
|
||||||
import std/monotimes
|
import std/monotimes
|
||||||
import std/tables
|
import std/tables
|
||||||
|
import std/sequtils
|
||||||
|
import barabadb/core/types
|
||||||
import barabadb/query/executor as qexec
|
import barabadb/query/executor as qexec
|
||||||
import barabadb/query/parser
|
import barabadb/query/parser
|
||||||
import barabadb/storage/lsm
|
import barabadb/storage/lsm
|
||||||
@@ -97,7 +99,7 @@ suite "JOIN execution":
|
|||||||
var foundBob = false
|
var foundBob = false
|
||||||
for row in r.rows:
|
for row in r.rows:
|
||||||
if row["u.name"] == "Bob":
|
if row["u.name"] == "Bob":
|
||||||
check row["x.total"] == "\\N"
|
check row["x.total"].kind == vkNull
|
||||||
foundBob = true
|
foundBob = true
|
||||||
check foundBob
|
check foundBob
|
||||||
|
|
||||||
|
|||||||
+14
-14
@@ -2831,7 +2831,7 @@ suite "Window Functions":
|
|||||||
# Sales: Diana(75000)=1, Charlie(70000)=2
|
# Sales: Diana(75000)=1, Charlie(70000)=2
|
||||||
var found = initTable[string, string]()
|
var found = initTable[string, string]()
|
||||||
for row in r.rows:
|
for row in r.rows:
|
||||||
found[row["name"]] = row["rn"]
|
found[$row["name"]] = $row["rn"]
|
||||||
check found["Eve"] == "1"
|
check found["Eve"] == "1"
|
||||||
check found["Alice"] == "2"
|
check found["Alice"] == "2"
|
||||||
check found["Bob"] == "3"
|
check found["Bob"] == "3"
|
||||||
@@ -3474,7 +3474,7 @@ suite "Auto-Increment & ID Generators":
|
|||||||
let r = qexec.executeQuery(ctx, parse("SELECT gen_random_uuid() AS uid"))
|
let r = qexec.executeQuery(ctx, parse("SELECT gen_random_uuid() AS uid"))
|
||||||
check r.success
|
check r.success
|
||||||
check r.rows.len == 1
|
check r.rows.len == 1
|
||||||
let uid = r.rows[0]["uid"]
|
let uid = $r.rows[0]["uid"]
|
||||||
check uid.len == 36
|
check uid.len == 36
|
||||||
check uid[8] == '-'
|
check uid[8] == '-'
|
||||||
check uid[13] == '-'
|
check uid[13] == '-'
|
||||||
@@ -3486,7 +3486,7 @@ suite "Auto-Increment & ID Generators":
|
|||||||
let r = qexec.executeQuery(ctx, parse("SELECT uuid() AS uid"))
|
let r = qexec.executeQuery(ctx, parse("SELECT uuid() AS uid"))
|
||||||
check r.success
|
check r.success
|
||||||
check r.rows.len == 1
|
check r.rows.len == 1
|
||||||
check r.rows[0]["uid"].len == 36
|
check ($r.rows[0]["uid"]).len == 36
|
||||||
|
|
||||||
test "Two UUIDs are different":
|
test "Two UUIDs are different":
|
||||||
let r = qexec.executeQuery(ctx, parse("SELECT uuid() AS a, uuid() AS b"))
|
let r = qexec.executeQuery(ctx, parse("SELECT uuid() AS a, uuid() AS b"))
|
||||||
@@ -3564,7 +3564,7 @@ suite "Auto-Increment & ID Generators":
|
|||||||
let r = qexec.executeQuery(ctx, parse("SELECT snowflake_id(1) AS sid"))
|
let r = qexec.executeQuery(ctx, parse("SELECT snowflake_id(1) AS sid"))
|
||||||
check r.success
|
check r.success
|
||||||
check r.rows.len == 1
|
check r.rows.len == 1
|
||||||
let sid = r.rows[0]["sid"]
|
let sid = $r.rows[0]["sid"]
|
||||||
check sid.len > 0
|
check sid.len > 0
|
||||||
var num: int64 = 0
|
var num: int64 = 0
|
||||||
try:
|
try:
|
||||||
@@ -3720,17 +3720,17 @@ suite "Type Safety — evalExprValue":
|
|||||||
check r.success
|
check r.success
|
||||||
check r.rows[0]["r"] == "17.5"
|
check r.rows[0]["r"] == "17.5"
|
||||||
|
|
||||||
test "evalExprValue returns correct Value kind for literals":
|
test "evalExpr returns correct Value kind for literals":
|
||||||
let lit = IRExpr(kind: irekLiteral, valueKind: vkInt64)
|
let lit = IRExpr(kind: irekLiteral, valueKind: vkInt64)
|
||||||
lit.literal = IRLiteral(kind: vkInt64, int64Val: 42)
|
lit.literal = IRLiteral(kind: vkInt64, int64Val: 42)
|
||||||
let v = evalExprValue(lit, initTable[string, string](), nil)
|
let v = evalExpr(lit, initTable[string, Value](), nil)
|
||||||
check v.kind == vkInt64
|
check v.kind == vkInt64
|
||||||
check v.int64Val == 42
|
check v.int64Val == 42
|
||||||
|
|
||||||
test "evalExprValue returns correct Value kind for float literal":
|
test "evalExpr returns correct Value kind for float literal":
|
||||||
let lit = IRExpr(kind: irekLiteral, valueKind: vkFloat64)
|
let lit = IRExpr(kind: irekLiteral, valueKind: vkFloat64)
|
||||||
lit.literal = IRLiteral(kind: vkFloat64, float64Val: 3.14)
|
lit.literal = IRLiteral(kind: vkFloat64, float64Val: 3.14)
|
||||||
let v = evalExprValue(lit, initTable[string, string](), nil)
|
let v = evalExpr(lit, initTable[string, Value](), nil)
|
||||||
check v.kind == vkFloat64
|
check v.kind == vkFloat64
|
||||||
check v.float64Val == 3.14
|
check v.float64Val == 3.14
|
||||||
|
|
||||||
@@ -3998,7 +3998,7 @@ suite "Hybrid RAG Search":
|
|||||||
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search('docs', 'embedding', 'content', 'quick brown', '[1.0, 0.0, 0.0]', 10) AS res"))
|
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search('docs', 'embedding', 'content', 'quick brown', '[1.0, 0.0, 0.0]', 10) AS res"))
|
||||||
check r.success
|
check r.success
|
||||||
check r.rows.len == 1
|
check r.rows.len == 1
|
||||||
let jsonStr = r.rows[0]["res"]
|
let jsonStr = $r.rows[0]["res"]
|
||||||
check jsonStr.len > 2 # not "[]"
|
check jsonStr.len > 2 # not "[]"
|
||||||
let arr = parseJson(jsonStr)
|
let arr = parseJson(jsonStr)
|
||||||
check arr.kind == JArray
|
check arr.kind == JArray
|
||||||
@@ -4013,7 +4013,7 @@ suite "Hybrid RAG Search":
|
|||||||
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search_ids('docs2', 'embedding', 'content', 'machine learning', '[0.0, 1.0, 0.0]', 10) AS ids"))
|
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search_ids('docs2', 'embedding', 'content', 'machine learning', '[0.0, 1.0, 0.0]', 10) AS ids"))
|
||||||
check r.success
|
check r.success
|
||||||
check r.rows.len == 1
|
check r.rows.len == 1
|
||||||
let idsStr = r.rows[0]["ids"]
|
let idsStr = $r.rows[0]["ids"]
|
||||||
check idsStr.len > 0
|
check idsStr.len > 0
|
||||||
check idsStr.contains("20")
|
check idsStr.contains("20")
|
||||||
|
|
||||||
@@ -4029,7 +4029,7 @@ suite "Hybrid RAG Search":
|
|||||||
discard qexec.executeQuery(ctx, parse("CREATE INDEX idx_fts3 ON docs3(content) USING FTS"))
|
discard qexec.executeQuery(ctx, parse("CREATE INDEX idx_fts3 ON docs3(content) USING FTS"))
|
||||||
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search('docs3', 'embedding', 'content', 'quick brown fox', '[1.0, 0.0, 0.0]', 10) AS res"))
|
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search('docs3', 'embedding', 'content', 'quick brown fox', '[1.0, 0.0, 0.0]', 10) AS res"))
|
||||||
check r.success
|
check r.success
|
||||||
let arr = parseJson(r.rows[0]["res"])
|
let arr = parseJson($r.rows[0]["res"])
|
||||||
check arr.len == 3
|
check arr.len == 3
|
||||||
# Doc 3 should be first (matches both vector and FTS), doc 1 second (vector only), doc 2 third (no match)
|
# Doc 3 should be first (matches both vector and FTS), doc 1 second (vector only), doc 2 third (no match)
|
||||||
check arr[0]["id"].getStr() == "3"
|
check arr[0]["id"].getStr() == "3"
|
||||||
@@ -4038,7 +4038,7 @@ suite "Hybrid RAG Search":
|
|||||||
let r = qexec.executeQuery(ctx, parse("SELECT rerank('quick brown', '[{\"id\":\"1\",\"score\":\"0.5\"},{\"id\":\"2\",\"score\":\"0.5\"}]') AS res"))
|
let r = qexec.executeQuery(ctx, parse("SELECT rerank('quick brown', '[{\"id\":\"1\",\"score\":\"0.5\"},{\"id\":\"2\",\"score\":\"0.5\"}]') AS res"))
|
||||||
check r.success
|
check r.success
|
||||||
# Both have same score, rerank should preserve order (no content to boost)
|
# Both have same score, rerank should preserve order (no content to boost)
|
||||||
let arr = parseJson(r.rows[0]["res"])
|
let arr = parseJson($r.rows[0]["res"])
|
||||||
check arr.kind == JArray
|
check arr.kind == JArray
|
||||||
check arr.len == 2
|
check arr.len == 2
|
||||||
|
|
||||||
@@ -4060,7 +4060,7 @@ suite "Hybrid RAG Search":
|
|||||||
discard qexec.executeQuery(ctx, parse("CREATE INDEX idx_fts5 ON docs5(content) USING FTS"))
|
discard qexec.executeQuery(ctx, parse("CREATE INDEX idx_fts5 ON docs5(content) USING FTS"))
|
||||||
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search_filtered('docs5', 'embedding', 'content', 'quick brown fox', '[1.0, 0.0, 0.0]', 10, 'tenant_id', 'tenant-a') AS res"))
|
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search_filtered('docs5', 'embedding', 'content', 'quick brown fox', '[1.0, 0.0, 0.0]', 10, 'tenant_id', 'tenant-a') AS res"))
|
||||||
check r.success
|
check r.success
|
||||||
let arr = parseJson(r.rows[0]["res"])
|
let arr = parseJson($r.rows[0]["res"])
|
||||||
# Doc 1 (tenant-a) should be first and highest scored; Doc 2 (tenant-b) must be excluded
|
# Doc 1 (tenant-a) should be first and highest scored; Doc 2 (tenant-b) must be excluded
|
||||||
check arr[0]["id"].getStr() == "1"
|
check arr[0]["id"].getStr() == "1"
|
||||||
for elem in arr:
|
for elem in arr:
|
||||||
@@ -4074,6 +4074,6 @@ suite "Hybrid RAG Search":
|
|||||||
discard qexec.executeQuery(ctx, parse("CREATE INDEX idx_fts6 ON docs6(content) USING FTS"))
|
discard qexec.executeQuery(ctx, parse("CREATE INDEX idx_fts6 ON docs6(content) USING FTS"))
|
||||||
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search_filtered('docs6', 'embedding', 'content', 'quick brown fox', '[1.0, 0.0, 0.0]', 10, '', '') AS res"))
|
let r = qexec.executeQuery(ctx, parse("SELECT hybrid_search_filtered('docs6', 'embedding', 'content', 'quick brown fox', '[1.0, 0.0, 0.0]', 10, '', '') AS res"))
|
||||||
check r.success
|
check r.success
|
||||||
let arr = parseJson(r.rows[0]["res"])
|
let arr = parseJson($r.rows[0]["res"])
|
||||||
check arr.len == 2
|
check arr.len == 2
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user