Compare commits

...

37 Commits

Author SHA1 Message Date
dimgigov 4c6c72dc5b docs(plan): mark 9.1.2 as complete
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-17 03:50:57 +03:00
dimgigov 2e1d982b9f fix(executor): include arguments in aggregate column names
Issue #6: getSelectColumns() produced count() for count(*)
and max() for max(id). Now uses exprToSql() on funcArgs
to generate consistent names matching the actual SQL expression.

Before: columns: @[count()], rows: @[{count(*): 2}]
After:  columns: @[count(*)], rows: @[{count(*): 2}]
2026-05-17 03:50:43 +03:00
dimgigov cd62f98641 docs(plan): mark 9.1.1 as complete 2026-05-17 03:47:30 +03:00
dimgigov 07a22d305d chore: fix all build warnings and add stabilization roadmap
- Fix 3× ResultShadowed in cypher.nim, recovery.nim, shell.nim
- Remove 6× UnusedImport in test files
- Build is now clean: 0 warnings, 294 tests pass
- Add 4-week stabilization plan to PLAN.md (Sessions 9.1–9.4)
2026-05-17 03:47:17 +03:00
dimgigov 5231724e77 fix(executor): handle subqueries in comparisons and per-row UPDATE expressions
- Add missing nkSubquery case in lowerExpr so scalar subqueries
  (e.g. budget > (SELECT AVG(budget) FROM projects)) are lowered
  correctly instead of being treated as NULL literals.
- Move UPDATE SET expression evaluation inside the row loop so
  column references (e.g. salary + 5000) are resolved against the
  current row, fixing type validation failures.
- Make irAdd/irSub/irMul/irDiv return integer strings when the
  result is a whole number, avoiding spurious float output for INT
  columns (e.g. MERGE UPDATE qty = 100 + 50 now yields 150).

These fix regressions introduced by switching NULL representation
from empty string to \N, which exposed the pre-existing bugs above.

Refs: issue #9, issue #10
2026-05-17 03:29:25 +03:00
dimgigov 073ec41652 fix: issue #9 reserved keyword 'key' + issue #10 empty string treated as NULL
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Issue 9: Add backtick-quoted identifier support in lexer.nim
- Users can now write `key` to use reserved keywords as column/table names

Issue 10: Use \N as internal NULL marker instead of empty string
- isNull() now checks for \N instead of value.len == 0
- Empty strings can now be stored in NOT NULL columns
- NULL values properly distinguished from empty strings in storage, wire protocol, and HTTP JSON responses
2026-05-17 01:59:25 +03:00
dimgigov 58b7598155 fix(dev): streamline build config for new contributors
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
- Add --path:src and --threads:on to nim.cfg so tests compile without manual flags
- Add tests/config.nims for automatic path resolution in test directory
- Simplify nimble task definitions (remove redundant flags now in nim.cfg)
- Fix Dockerfile.source comment: remove obsolete nimmax reference

This makes 'nim c tests/test_all.nim' and 'nimble test' work out of the box.
2026-05-16 03:20:43 +03:00
dimgigov 3fce374aba fix(tests): correct vkInt -> vkInt64 in test_minimal, remove tracked binary from git
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
2026-05-16 03:13:19 +03:00
dimgigov 9756d93cb8 docs: add deficiency audit and ID generators plan
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
2026-05-16 03:07:11 +03:00
dimgigov 652ed1b477 feat: add ID generators, fix 6 forum-discovered deficiencies
ID Generators:
- AUTO_INCREMENT for INTEGER columns
- SERIAL / BIGSERIAL as syntactic sugar
- gen_random_uuid() / uuid() for UUID v4 generation
- nextval() / currval() for sequence support
- snowflake_id(node_id) for distributed 64-bit IDs
- RETURNING clause for INSERT

Deficiency Fixes:
- Comma join: FROM t1, t2 now works as implicit CROSS JOIN
- DEFAULT in restoreSchema: defaults survive server restart
- Duplicate column names: t.id instead of id in joins
- Empty result sets: always send column metadata to client
- GROUP BY non-agg columns: return first row value instead of empty
- Aggregate column names: count(*) instead of count()

Bug fix: PK uniqueness check now uses correct key format
2026-05-16 02:57:30 +03:00
dimgigov 406dcf4f4e docs(ormin): clarify transaction and returning support
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Top-level BEGIN/COMMIT/ROLLBACK work (server-side)
- SAVEPOINT (nested tx) does not work — BaraDB server limitation
- RETURNING parsed but ignored by BaraDB executor — server limitation
2026-05-15 22:56:45 +03:00
dimgigov 4ac147dc6c Add Docker support for Ormin + BaraDB dev stack
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- docker-compose.ormin.yml: BaraDB + Ormin dev environment
- clients/nim/ormin/docker/Dockerfile: dev image with Nim 2.2.10
- clients/nim/ormin/docker/entrypoint.sh: auto-install deps
- docker-compose.test.yml: add Ormin compile test
2026-05-15 22:23:45 +03:00
dimgigov 4e54075078 Add Ormin ORM support for BaraDB (Nim client)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-15 21:49:08 +03:00
dimgigov 2e945c1dcb release: bump version to 1.1.2 (forum-tested, nested SQL fixes)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-15 18:34:47 +03:00
dimgigov 2fbafb11bd docs: add GitHub stars badge and thank you note
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
2026-05-15 18:30:10 +03:00
dimgigov 3453d3cae7 feat: add forum Nim client improvements and SQLite adaptor
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-15 18:12:01 +03:00
dimgigov 00e9ea7484 Merge branch 'main' of ssh://git.invoicing.top:222/baraba/Baradb
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-15 18:01:25 +03:00
dimgigov c5751b4a0c fix: clean up compiler warnings and unused variables 2026-05-15 18:01:11 +03:00
dimgigov bf374b28f7 chore: ignore docker export tarballs
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
2026-05-15 17:46:53 +03:00
dimgigov 92e6b99df2 Merge pull request 'feat: subquery IN/NOT IN, datetime functions, scanAll refactor, INSERT/UPDATE eval improvements' (#1) from master into main
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Reviewed-on: #1
2026-05-15 14:35:06 +00:00
dimgigov f8d466d8f3 feat: subquery IN/NOT IN, datetime functions, scanAll refactor, INSERT/UPDATE eval improvements
CI / test (pull_request) Has been cancelled
CI / verify (pull_request) Has been cancelled
Clients CI / build-server (pull_request) Has been cancelled
Clients CI / test-python (pull_request) Has been cancelled
Clients CI / test-javascript (pull_request) Has been cancelled
Clients CI / test-nim (pull_request) Has been cancelled
Clients CI / test-rust (pull_request) Has been cancelled
2026-05-15 17:33:54 +03:00
dimgigov 40616ad3c0 Update repository URLs from GitHub to Codeberg
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-15 12:11:00 +03:00
dimgigov 8c29e700c7 Revert "Update repository URLs from GitHub to Codeberg"
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
This reverts commit 956066631e.
2026-05-15 12:10:36 +03:00
dimgigov 956066631e Update repository URLs from GitHub to Codeberg
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-15 12:03:11 +03:00
dimgigov d11e99ab5a ci(clients): install Python dev dependencies for pytest-asyncio
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
2026-05-14 23:41:19 +03:00
dimgigov 359f945170 fix(clients): repair Python & Rust tests, add container test orchestration
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- Python: fix wire protocol test method names (bool_val -> bool, etc.)
- Python: make integration tests and example fully async with pytest-asyncio
- Rust: add tokio dev-dependency and convert integration tests to async/await
- Rust: update ping_test example to async
- Nim: remove committed ELF build artifact
- Docker: add BARADB_HOST/BARADB_PORT env vars to test containers
- Docker: fix docker-compose.test.yml usage (remove --abort-on-container-exit)
- Add scripts/test-clients.sh for sequential client test runs
- Remove docker-compose.test.yml from .gitignore so it is tracked
- Fix repository URLs in Python and Rust READMEs
2026-05-14 23:35:45 +03:00
dimgigov c55d3080cf Update documentation and clients for v1.1.0
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Documentation updates:
- Fix v0.1.0 → v1.1.0 version numbers in en, ru, fa, zh docs
- Add missing Window Functions, Multi-Tenant ERP, Supported Keywords sections
  to ru, fa, zh baraql.md (~105 lines each)
- Expand Turkish and Arabic baraql.md (110 → 268 lines)
- Expand Turkish and Arabic installation.md (62 → 307 lines)
- Add new Bulgarian documentation files (18 new files)

Client updates:
- Python: Full async/await rewrite with asyncio, request queueing
- Rust: Full async/await rewrite with tokio, async examples
- Nim: Update README to v1.1.0
- All clients now support async patterns consistently
2026-05-14 23:05:47 +03:00
dimgigov f7d4961125 feat: Multi-tenant ERP support via session variables + RLS
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
- SET var = value / current_setting('var') for session-scoped variables
- current_user / current_role SQL keywords with auth bridge
- server.nim + httpserver.nim populate ExecutionContext.currentUser/currentRole
- RLS policies can reference current_setting('app.tenant_id') for tenant isolation
- Fixed evalExpr to propagate ctx recursively (fixes current_user in sub-expressions)
- Fixed GROUPING SETS execution (lowerSelect checks selGroupingSetsKind)
- Fixed FTS CREATE INDEX docId mismatch (hash of tableName.$key)
- Fixed all test suites to use isolated temp directories
- Added 5 multi-tenant tests (355 total, all green)
- Updated docs: PLAN_SQL_ADVANCED.md, baraql.md, changelog.md
2026-05-14 16:28:41 +03:00
dimgigov b0978812cb docs(en): Update English docs for Vector SQL Integration
- docs/en/vector.md — add SQL usage section (CREATE TABLE VECTOR,
  distance functions, <-> operator, CREATE INDEX USING hnsw)
- docs/en/baraql.md — update vector search section with real SQL syntax,
  add VECTOR(n) to data types, update keyword table
- docs/en/changelog.md — add Vector SQL Integration and bugfixes to [Unreleased]
- docs/ARCHITECTURE.md — add SQL Integration bullet to Vector Engine
- README.md — update vector engine section with SQL examples,
  add Vector SQL to roadmap, bump test count to 340+
2026-05-14 14:20:57 +03:00
dimgigov d076cfde3b feat(sql): Vector SQL Integration + test isolation fixes
- Add VECTOR(n) column type support in CREATE TABLE
- Add CREATE INDEX ... USING hnsw/ivfpq for vector indexes
- Add cosine_distance(), euclidean_distance(), inner_product(), l1/l2_distance()
  SQL functions in expression evaluator
- Add <-> nearest-neighbor operator
- Fix ORDER BY with non-projected columns (move irpkSort before irpkProject)
- Fix execInsert to escape comma-containing values (vector literals)
- Fix MERGE tests by using unique temp dirs per test suite
- Add 8 Vector SQL Integration tests (all passing)
- Update PLAN_SQL_ADVANCED.md
2026-05-14 14:14:13 +03:00
dimgigov 96dfaaecb1 feat(sql): Advanced SQL — LATERAL, GROUP BY/HAVING, FILTER, aggregates, PIVOT, SQL/PGQ
- LATERAL JOIN: correlated subquery strategy (scan + merge + filter/sort/limit)
- GROUP BY: SUM/AVG/MIN/MAX evaluation inside groups, HAVING filter
- FILTER (WHERE ...): conditional aggregates for COUNT/SUM/AVG
- ARRAY_AGG / STRING_AGG: multi-argument aggregate functions
- GROUPING SETS / ROLLUP / CUBE: powerset generation for multi-level aggregation
- PIVOT / UNPIVOT: row-to-column and column-to-row transformation
- SQL/PGQ Property Graph: GRAPH_TABLE MATCH parser + executor skeleton
- 330 tests passing, all 4 modalities (SQL/JSON/Vector/Graph) integrated
2026-05-14 13:14:10 +03:00
dimgigov e2a526df6f feat(sql): Window Functions + MERGE statement + REST bridge plan
- Add Window Functions: ROW_NUMBER, RANK, DENSE_RANK, LEAD, LAG,
  FIRST_VALUE, LAST_VALUE, NTILE with PARTITION BY, ORDER BY,
  ROWS/RANGE frame specifications
- Add MERGE/UPSERT statement with WHEN MATCHED UPDATE and
  WHEN NOT MATCHED INSERT
- Add SQL/PGQ Property Graph long-term plan in PLAN_SQL_ADVANCED.md
- Add 7 new tests for Window Functions and MERGE
- Update baraql.md documentation
2026-05-14 11:02:09 +03:00
dimgigov 18f3c16b2a Fix/nodebara compatibility (#2) 2026-05-14 02:33:46 +03:00
dimgigov 71dcffecce fix(gossip): use async UDP socket to avoid blocking the event loop
The gossip listener used synchronous newSocket + recvFrom,
which blocks the async event loop and prevents other async
operations from running while waiting for UDP packets.

Switch to newAsyncSocket + async recvFrom so the loop can
continue processing other tasks between gossip rounds.
2026-05-14 02:22:07 +03:00
dimgigov 398769ff97 feat(client): add TCP request queue for safe concurrency
The previous implementation allowed multiple concurrent async calls
(query, execute, ping) to interleave writes on the same socket,
which corrupted the binary protocol framing when NodeBB fired
parallel database operations.

Add an internal _requestQueue and _requestLock so that all TCP
requests are serialized: each async operation enqueues a task,
and tasks are drained one at a time via setImmediate().
2026-05-14 02:22:07 +03:00
dimgigov 8fb5dde858 fix(protocol): serialize float32/float64/fkVector in big-endian
The JavaScript client reads floats via readFloatBE/readDoubleBE,
which expect IEEE-754 values in big-endian byte order.
The Nim server was writing them with copyMem in native byte order,
so on little-endian machines (x86_64) the JS side deserialized
FLOAT64 values as garbage (e.g. 1.0 became 3.03865e-319).

Fix serialization by casting to int32/int64 and using bigEndian32/
bigEndian64, mirroring the existing big-endian handling for ints.
Apply the same fix to deserialization and to fkVector elements.
2026-05-14 02:22:07 +03:00
dimgigov a0d2ca7776 Merge release/v1.1.0 into main 2026-05-13 15:07:44 +03:00
174 changed files with 20566 additions and 1052 deletions
+1 -2
View File
@@ -80,8 +80,7 @@ jobs:
working-directory: clients/python working-directory: clients/python
run: | run: |
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install pytest pip install -e ".[dev]"
pip install -e .
- name: Run Python unit tests - name: Run Python unit tests
working-directory: clients/python working-directory: clients/python
+6 -1
View File
@@ -7,6 +7,9 @@ nimblecache/
# Binaries (compiled test executables) # Binaries (compiled test executables)
tests/test_all tests/test_all
tests/stress_test tests/stress_test
tests/test_lock
tests/test_minimal
tests/tla_faithfulness
benchmarks/bench_all benchmarks/bench_all
benchmarks/compare benchmarks/compare
clients/nim/tests/test_client clients/nim/tests/test_client
@@ -37,4 +40,6 @@ clients/rust/target/
# Nim compiled binaries # Nim compiled binaries
clients/nim/src/baradb/client clients/nim/src/baradb/client
src/barabadb/storage/compaction src/barabadb/storage/compaction
docker-compose.test.yml src/barabadb/query/executor
tests/join_tests
*.tar.gz
+231
View File
@@ -0,0 +1,231 @@
# BaraDB Deficiencies Fixed During NimForum Migration
This document summarizes the bugs and design deficiencies discovered in BaraDB while porting NimForum from SQLite to BaraDB's TCP wire protocol.
---
## 1. Comma Join Not Supported in Parser
**Problem:** The BaraQL parser did not recognize comma-separated table lists in the `FROM` clause.
```sql
SELECT * FROM thread t, category c WHERE t.category = c.id
```
This would fail with "unknown tokens" after the first comma.
**Root Cause:** `parseSelect()` only parsed a single table after `FROM` and ignored everything after a comma.
**Fix:** Modified `database/src/barabadb/query/parser.nim` to loop over comma-separated tables and treat each additional table as an implicit `CROSS JOIN`.
```nim
while p.peek().kind == tkComma:
discard p.advance()
let nextTableTok = p.expect(tkIdent)
# ... build joinNode with jkCross and add to result.selJoins
```
---
## 2. DEFAULT Constraints Not Evaluated at Schema Creation Time
**Problem:** `CREATE TABLE` with `DEFAULT <expr>` stored the raw AST node instead of the evaluated string value. During `INSERT`, `applyDefaultValues()` checked `colDef.defaultVal.len > 0`, but `defaultVal` was empty because the AST was never evaluated to a string.
**Result:** Explicitly omitted columns in `INSERT` raised `NOT NULL constraint violation` even when a `DEFAULT` was defined.
**Fix:** Added `evalNodeToString()` in `database/src/barabadb/query/executor.nim` to evaluate `DEFAULT` AST nodes to their string representation during schema restoration.
```nim
proc evalNodeToString(node: Node): string =
let ir = lowerExpr(node)
return evalExpr(ir, initTable[string, string](), nil)
```
---
## 3. Join Column Resolution Overwrites Duplicate Column Names
**Problem:** `Row` is defined as `Table[string, string]`. When a query selects columns with identical names from joined tables:
```sql
SELECT t.id, c.id FROM thread t INNER JOIN category c ON t.category = c.id
```
The `Project` operator in `executePlan()` builds `newRow` as a `Table`. The second `id` overwrites the first, so both columns end up with the value from the right-hand table.
**Result:** `t.id` returns `0` (category id) instead of `1` (thread id).
**Fix:** Two changes in `database/src/barabadb/query/executor.nim`:
1. `lowerSelect()` — for `nkPath` expressions, use the full path (`t.id`) as the alias instead of just the last segment (`id`).
2. Added uniqueness logic for all aliases: if a duplicate alias is detected, append `_1`, `_2`, etc. This also fixes `strftime()` appearing multiple times in the same select list.
```nim
# Before
projectPlan.projectAliases.add(e.pathParts[^1]) # -> "id"
# After
projectPlan.projectAliases.add(e.pathParts.join(".")) # -> "t.id"
```
---
## 4. Empty Result Sets Do Not Send Column Metadata
**Problem:** In `database/src/barabadb/core/server.nim`, the server only sent the `mkData` message when `result.rows.len > 0`:
```nim
if result.rows.len > 0:
let dataMsg = serializeResult(result, header.requestId)
await client.send(...)
```
For queries with zero matching rows (e.g. `WHERE id IN (SELECT ...)` with no subquery matches), the client received only `mkComplete` and no `mkData`.
**Result:** The client saw `columns: @[]` and `rows: @[]`. When `getRow()` fell back to `newSeq[string](qr.columns.len)`, it returned an empty seq, causing "index out of bounds" errors when the application tried to access `row[0]`.
**Fix:** Removed the `if result.rows.len > 0` guard. `serializeResult()` is now always sent, including the correct column names even when `rowCount = 0`.
---
## 5. GROUP BY Returns Empty Values for Non-Aggregated Columns
**Problem:** Unlike SQLite, BaraDB does not automatically pick an arbitrary row value for columns that are neither in `GROUP BY` nor inside an aggregate function.
```sql
SELECT u.id, u.name, count(*)
FROM person u, post p
WHERE p.author = u.id AND p.thread = ?
GROUP BY name
```
**Result:** `u.id`, `u.email`, `u.usrStatus`, etc. return empty strings, while `name` and `count(*)` are correct.
**Fix:** (Workaround in forum code) Replaced `GROUP BY` queries with `DISTINCT` + separate subqueries where ordering by count is not critical:
```sql
SELECT DISTINCT u.id, u.name, u.email, ...
FROM person u, post p
WHERE p.author = u.id AND p.thread = ?
LIMIT 5
```
---
## 6. Inconsistent Aggregate Column Names
**Problem:** Aggregate functions produce column names that omit the argument expression:
- `SELECT count(*)` → column name `count()`
- `SELECT max(id)` → column name `max()`
- `SELECT min(creation)` → column name `min()`
Code that relies on exact column names (e.g. `getValue` looking up `count(*)`) can be confused.
**Fix:** (Workaround in forum code) Avoided name-dependent lookup and rewrote queries to use positional access via `getRow()` / `getAllRows()`.
---
## 7. Async Client + waitFor in Async Context = Connection Instability
**Problem:** The original `baradb_client.nim` used `AsyncSocket` wrapped in `SyncClient` via `waitFor`. When called from inside Jester's async handlers, nested `waitFor` + `poll()` created race conditions on the single socket. `recv(12)` occasionally returned 0 bytes, causing "Connection closed" exceptions.
**Result:** Login and other routes crashed intermittently with 502 Bad Gateway.
**Fix:** Built a new synchronous client (`forum/src/baradb_sync_client.nim`) using blocking `net.Socket` from Nim's standard library. This eliminates all async event loop interactions.
Key differences from the async client:
- Uses `net.Socket` instead of `asyncnet.AsyncSocket`
- Uses blocking `recv()` with an explicit `recvExact()` helper
- No dependency on `asyncdispatch` or `waitFor`
- Fully compatible with the existing wire protocol
---
## 8. Lack of Thread Safety in the Adapter
**Problem:** A single `SyncClient` instance was shared across all HTTP requests. NimForum runs on Jester's async event loop, which can interleave request handlers in the same thread. Without synchronization, two handlers could send queries over the same socket simultaneously, corrupting the wire protocol stream.
**Fix:** Added a global `Lock` and `withDbLock` template in `forum/src/baradb_sqlite.nim`:
```nim
var dbLock: Lock
initLock(dbLock)
template withDbLock(body: untyped) =
acquire(dbLock)
try: body
finally: release(dbLock)
```
All DB operations (`query`, `getRow`, `exec`, etc.) are wrapped in this lock.
---
## 9. `key` is a Reserved SQL Keyword
**Problem:** BaraDB's SQL parser treats `key` as a reserved keyword. Using it as a column or table name causes parse errors:
```
Expected tkIdent but got tkKey
```
**Result:** The `settings` table could not use `key varchar(100) primary key`.
**Fix:** Added backtick-quoted identifier support in `query/lexer.nim`. Users can now escape reserved keywords using backticks:
```sql
CREATE TABLE settings(
`key` varchar(100) primary key,
value varchar(500) default ''
);
```
Changes made:
- Added `readBacktickIdent()` proc in `lexer.nim`
- Added backtick case in `nextToken()` to recognize `` `identifier` `` as `tkIdent`
---
## 10. Empty String (`""`) Treated as NULL
**Problem:** BaraDB normalized empty strings to `NULL` internally. A column defined as `NOT NULL` rejected empty string inserts, even when the application explicitly passed `''`.
**Result:** SMTP settings saved via the admin panel failed with constraint violations when fields were left blank:
```sql
INSERT INTO settings (skey, value) VALUES ('smtpUser', '');
-- ERROR: NOT NULL constraint violation
```
**Fix:** Changed internal NULL representation from empty string `""` to `\N` in `query/executor.nim`:
1. `isNull()` now checks for `\N` instead of `value.len == 0`
2. NULL literals in INSERT/UPDATE store `\N` instead of `""`
3. Missing columns return `\N` instead of `""`
4. `getValue()` returns `\N` for missing fields
5. JOIN padding uses `\N` for unmatched rows
6. Server wire protocol uses `\N` sentinel to send `fkNull`
7. HTTP server sends JSON `null` for NULL values
This allows empty strings to be stored in NOT NULL columns while still properly supporting NULL values.
---
## Summary Table
| # | Issue | Location | Type |
|---|-------|----------|------|
| 1 | Comma join parsing | `query/parser.nim` | Parser bug |
| 2 | DEFAULT not evaluated | `query/executor.nim` | Schema bug |
| 3 | Duplicate column overwrite | `query/executor.nim` | Data structure bug |
| 4 | Empty result = no columns | `core/server.nim` | Protocol bug |
| 5 | GROUP BY empty values | N/A (engine behavior) | Semantic difference |
| 6 | Inconsistent agg names | N/A (engine behavior) | Naming convention |
| 7 | Async client unstable | `forum/src/baradb_client.nim` | Client design |
| 8 | No thread safety | `forum/src/baradb_sqlite.nim` | Adapter design |
| 9 | `key` reserved keyword | `query/lexer.nim` | Parser limitation |
| 10 | Empty string = NULL | `query/executor.nim` | Data handling bug |
---
*Document version: 2026-05-17*
+3 -3
View File
@@ -4,9 +4,9 @@
# └─────────────────────────────────────────────────────────┘ # └─────────────────────────────────────────────────────────┘
# #
# Този Dockerfile build-ва BaraDB от source код. # Този Dockerfile build-ва BaraDB от source код.
# ВНИМАНИЕ: Изисква локални nimble пакети (hunos, nimmax), които # ВНИМАНИЕ: Изисква локален nimble пакет (hunos), който
# не са в публичния nimble repository. Преди build трябва да: # не е в публичния nimble repository. Преди build трябва да:
# 1. symlink-нете или копирате hunos/ и nimmax/ в проекта, или # 1. symlink-нете или копирате hunos/ в проекта, или
# 2. копирате ~/.nimble/pkgs2 в builder стейджа. # 2. копирате ~/.nimble/pkgs2 в builder стейджа.
# #
# Build: # Build:
+79
View File
@@ -123,3 +123,82 @@
**BaraDB v1.0.0 е production-ready за blogs, e-commerce и small ERP системи.** **BaraDB v1.0.0 е production-ready за blogs, e-commerce и small ERP системи.**
**Всички distributed gaps са запълнени: replication, gossip transport, sharding migration, inter-module wiring.** **Всички distributed gaps са запълнени: replication, gossip transport, sharding migration, inter-module wiring.**
---
## 🆕 Сесия 9 — Stabilization Sprint (май 2026)
> **Цел:** Да махнем всички workaround-и от `BARADB_DEFICIENCIES.md`, да почистим build-а и да подготвим почвата за типова система.
> **Принцип:** Без нови светове — само stabilizaция на съществуващото.
### Седмица 1: Deficiency Hunt + Build Cleanup
| # | Задача | Оценка | Статус |
|---|--------|--------|--------|
| 9.1.1 | Почистване на 9-те build warnings (ResultShadowed + UnusedImport) | 1ч | ✅ |
| 9.1.2 | Issue #6: Aggregate column names (`count(*)``count(*)`, `max(id)``max(id)`) | 2ч | ✅ |
| 9.1.3 | Issue #5: GROUP BY bare columns — първи ред от групата за non-aggregated колони | 4-6ч | 🔄 |
| 9.1.4 | Issue #7+8: Решение за async vs sync client + thread safety | 2ч | 🔄 |
| 9.1.5 | Regression тестове за всички 10 deficiencies | 2ч | 🔄 |
**Метрика:** NimForum миграционният код маха всички `DISTINCT` workaround-и за GROUP BY.
---
### Седмица 2: Type Safety in Execution Layer
| # | Задача | Оценка | Статус |
|---|--------|--------|--------|
| 9.2.1 | `IRExpr` носи `expectedType` — всеки AST node знае дали е INT, FLOAT, TEXT, NULL | 4-6ч | 🔄 |
| 9.2.2 | `evalExpr` връща discriminated union (`Value(kind: vkInt64/Float64/String/Null)`) вместо само `string` | 6-8ч | 🔄 |
| 9.2.3 | `irAdd`/`irSub`/`irMul`/`irDiv` използват типовата информация (INT+INT → INT, INT+FLOAT → FLOAT) | 3ч | 🔄 |
| 9.2.4 | `validateType` използва `Value.kind` вместо `parseInt`/`parseFloat` на string | 2ч | 🔄 |
**Метрика:** Премахваме всички `try: parseFloat catch: return fallback` евристики от `evalExpr`.
---
### Седмица 3: JOIN Performance
| # | Задача | Оценка | Статус |
|---|--------|--------|--------|
| 9.3.1 | Hash Join: `ON a.col = b.col` с hash table върху по-малката страна | 6ч | 🔄 |
| 9.3.2 | Index Nested Loop Join: ако има B-Tree индекс на join колоната | 4ч | 🔄 |
| 9.3.3 | Benchmark: `thread JOIN category` с 10K/100K/1M редове | 2ч | 🔄 |
| 9.3.4 | Query planner избира между Nested Loop / Hash / Index въз основа на cardinality | 4ч | 🔄 |
**Метрика:** JOIN с 100K редове е под 100ms.
---
### Седмица 4: Production Hardening
| # | Задача | Оценка | Статус |
|---|--------|--------|--------|
| 9.4.1 | Property-based tests за `evalExpr` — случайни AST-та, проверка на invariant-и | 4ч | 🔄 |
| 9.4.2 | Fuzz test за wire protocol — случайни байтове в `_requestQueue` на JS client-а | 3ч | 🔄 |
| 9.4.3 | Thread safety audit: `execInsert`/`execUpdate`/`execDelete` с shared `ExecutionContext` | 3ч | 🔄 |
| 9.4.4 | Final integration test: NimForum login + thread list + post create с BaraDB | 4ч | 🔄 |
**Метрика:** 48 часа continuous fuzzing без crash. NimForum smoke test минава end-to-end.
---
### Рискове и митигации
| Риск | Митигация |
|------|-----------|
| GROUP BY bare columns е по-сложен от очакваното | Fallback: SQLite behavior (първи ред), не PostgreSQL (грешка) |
| Type safety рефакторингът чупи 294 теста | Правим го на отделен branch, не в `main` |
| Hash Join изисква памет proportional на N | Добавяме `work_mem` лимит като PostgreSQL |
---
### Текущи метрики (преди сесия 9)
| Метрика | Стойност |
|---------|----------|
| **Тестове** | 294 — 0 failures |
| **Build warnings** | 9 (3× ResultShadowed + 6× UnusedImport) |
| **BARADB_DEFICIENCIES** | 4 непоправени (#5, #6, #7, #8) |
| **Workaround-и в NimForum** | 2 (GROUP BY → DISTINCT, aggregate positional access) |
+71
View File
@@ -0,0 +1,71 @@
# Plan: ID Generators & Sequence System
## Goal
Add auto-generated ID support to BaraDB so users don't need to manually supply IDs on INSERT.
## Phase 1: ID Generators
### 1.1 AUTO_INCREMENT on INTEGER columns
- Add `AUTO_INCREMENT` keyword to lexer
- Parse in CREATE TABLE: `id INTEGER PRIMARY KEY AUTO_INCREMENT`
- Store auto-increment state per table in ExecutionContext (counter)
- On INSERT without explicit ID → auto-populate with next value
- Thread-safe counter (atomic increment)
### 1.2 SERIAL / BIGSERIAL as syntactic sugar
- `SERIAL` = `INTEGER AUTO_INCREMENT`
- `BIGSERIAL` = `BIGINT AUTO_INCREMENT`
- Already partially parsed — wire to auto-increment logic
### 1.3 UUID generation
- Add `gen_random_uuid()` or `uuid()` as built-in function
- Can be used in INSERT: `INSERT INTO t (id) VALUES (uuid())`
- Also usable as DEFAULT: `id UUID DEFAULT uuid()`
- Use Nim's std/oids or crypto random
### 1.4 RETURNING clause
- After INSERT, return generated values
- `INSERT INTO t (name) VALUES ('x') RETURNING id`
- Already partially parsed — wire to execution
### 1.5 CREATE SEQUENCE / nextval / currval
- `CREATE SEQUENCE seq_name START 1 INCREMENT 1`
- `nextval('seq_name')` → returns next value
- `currval('seq_name')` → returns current value
- Store sequences in ExecutionContext
### 1.6 Snowflake ID (distributed)
- 64-bit ID = timestamp(41) + node_id(10) + sequence(12)
- `snowflake_id(node_id)` function
- For future distributed use
## Phase 2: JOIN Optimizations (future)
### 2.1 Hash Join
- For equi-join ON a.col = b.col
- Build hash table on smaller side, probe with larger
- O(N+M) instead of O(N*M)
### 2.2 Index Nested Loop Join
- If index exists on join column → probe index per left row
- O(N * log M) instead of O(N*M)
### 2.3 Merge Join
- For sorted inputs
- Two-pointer sweep O(N+M)
## Phase 3: Foreign Key Enforcement (future)
### 3.1 CASCADE DELETE
### 3.2 SET NULL on delete
### 3.3 RESTRICT on delete
### 3.4 ON UPDATE CASCADE
### 3.5 FK check on UPDATE (not just INSERT)
## Implementation Order
1. AUTO_INCREMENT (lexer + parser + executor)
2. SERIAL/BIGSERIAL sugar
3. UUID function
4. RETURNING clause
5. Sequences (CREATE SEQUENCE / nextval / currval)
6. Snowflake ID function
+351
View File
@@ -0,0 +1,351 @@
# BaraDB — Универсален план за Advanced SQL Engine
> **Визия**: BaraDB е самостоятелен, универсален SQL engine с Nim ядро, поддържащ модерни SQL:2023 разширения — Property Graph, Vector Search, JSON документи и прозоречни функции, в една вградена или клиент/сървър конфигурация.
>
> **Принцип**: Само основи. Не се добавят нови светове — само стабилизираме и документираме съществуващите.
>
> **Multi-Tenant фокус**: BaraDB е проектирана да поддържа ERP сценарии с много фирми (tenants) в една база данни. Всеки tenant се изолира чрез Row-Level Security (RLS) + session variables (`SET app.tenant_id = 'X'`), а не чрез отделни бази.
---
## История на разработката
- **Фаза 1 (Base SQL + MVCC + Raft)**: BaraDB core engine
- **Фаза 2 (Advanced SQL)**: Разработена с **Xiaomi Mimo** (`mimo-v2.5-pro`) — Window Functions, MERGE, LATERAL JOIN, Advanced Aggregates, PIVOT/UNPIVOT, SQL/PGQ Property Graph
- **Фаза 3 (Stabilization + Multi-Tenant)**: Текуща — Vector SQL Integration, Session Variables, `current_user`/`current_role`, RLS tenant isolation, тестове, документация
---
---
## Част 1: BaraDB Advanced SQL Engine
### 1.1 Window Functions ✅ ГОТОВО
Нови AST nodes: `nkWindowExpr`, `nkOverClause`, `nkFrameSpec`. Нов IR plan: `irpkWindow`.
| Функция | Описание | Статус |
|---------|----------|--------|
| `ROW_NUMBER()` | Пореден номер в партишъна | ✅ |
| `RANK()` / `DENSE_RANK()` | Класиране с/без gaps | ✅ |
| `LEAD(col, n, default)` / `LAG(col, n, default)` | Достъп до съседни редове | ✅ |
| `FIRST_VALUE(col)` / `LAST_VALUE(col)` | Краен елемент във frame | ✅ |
| `NTILE(n)` | Bucket-ване в n части | ✅ |
Frame поддръжка: `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`, `codegen.nim`
Тестове: 5 теста в `tests/test_all.nim`, всички зелени.
### 1.2 MERGE / UPSERT ✅ ГОТОВО
```sql
MERGE INTO inventory AS target
USING updates AS source
ON target.sku = source.sku
WHEN MATCHED THEN UPDATE SET qty = target.qty + source.delta
WHEN NOT MATCHED THEN INSERT (sku, qty) VALUES (source.sku, source.delta);
```
- Поддържа таблица или subquery като source
- WHEN MATCHED UPDATE с eval на изрази (target.col + source.col)
- WHEN NOT MATCHED INSERT с eval на value изрази
- Trigger support (BEFORE/AFTER UPDATE/INSERT)
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`, `codegen.nim`
Тестове: 2 теста в `tests/test_all.nim`, всички зелени.
### 1.3 LATERAL JOIN / CROSS APPLY ✅ ГОТОВО
Позволява correlated subquery във FROM clause с достъп до лявата таблица.
```sql
SELECT u.name, recent_orders.*
FROM users u,
LATERAL (
SELECT order_id, total FROM orders o
WHERE o.user_id = u.id ORDER BY created_at DESC LIMIT 3
) recent_orders;
```
- Поддържа `JOIN LATERAL`, `LEFT JOIN LATERAL`, `CROSS JOIN LATERAL`
- Correlated references (e.g. `u.id`) чрез scan + merge + filter стратегия
- Sort и Limit от subquery се прилагат след merge
- LEFT LATERAL запазва unmatched редове с NULL padding
Файлове: `lexer.nim`, `ast.nim`, `ir.nim`, `parser.nim`, `executor.nim`
Тестове: 4 execution теста + 3 parser теста, всички зелени.
### 1.4 Advanced Aggregates ✅ ГОТОВО
- `ARRAY_AGG(col ORDER BY ...)`
- `STRING_AGG(col, delimiter)`
- `COUNT(*) FILTER (WHERE ...)`
- `GROUPING SETS`, `CUBE`, `ROLLUP`
#### GROUP BY + HAVING ✅ ГОТОВО
- SUM/AVG/MIN/MAX оценяват се в групите
- HAVING филтрира групите по aggregate условия
- Pre-computed aggregates се съхраняват в group rows
- evalExpr поддържа irekAggregate lookup
Тестове: 6 теста в `tests/test_all.nim`, всички зелени.
#### FILTER (WHERE ...) ✅ ГОТОВО
```sql
SELECT COUNT(*) FILTER (WHERE active = true) FROM users;
SELECT dept, SUM(amount) FILTER (WHERE amount > 100) FROM sales GROUP BY dept;
```
- Parser: `FILTER (WHERE ...)` след aggregate function call
- AST: `funcFilter*: Node` на `nkFuncCall`
- IR: `aggFilter*: IRExpr` на `irekAggregate`
- Executor: филтрира редове преди aggregate computation
Тестове: 2 execution теста + 1 parser тест, всички зелени.
#### ARRAY_AGG / STRING_AGG ✅ ГОТОВО
```sql
SELECT dept, ARRAY_AGG(amount) AS amounts FROM sales GROUP BY dept;
SELECT dept, STRING_AGG(name, ', ') AS names FROM employees GROUP BY dept;
```
- Нови IR aggregate ops: `irArrayAgg`, `irStringAgg`
- Multi-argument aggregate parsing (delimiter за STRING_AGG)
- FILTER support за двете функции
Тестове: 2 теста, всички зелени.
#### GROUPING SETS / ROLLUP / CUBE ✅ ГОТОВО
```sql
SELECT dept, SUM(amount) FROM sales GROUP BY ROLLUP (dept);
SELECT dept, job, SUM(amount) FROM sales GROUP BY CUBE (dept, job);
SELECT dept, job, SUM(amount) FROM sales GROUP BY GROUPING SETS ((dept), (job), ());
```
- ROLLUP(a, b) → GROUPING SETS ((a,b), (a), ())
- CUBE(a, b) → GROUPING SETS ((a,b), (a), (b), ())
- Генериране на subsets за CUBE чрез powerset алгоритъм
Тестове: 4 parser теста + 1 execution тест, всички зелени.
### 1.5 PIVOT / UNPIVOT ✅ ГОТОВО
```sql
SELECT * FROM (SELECT name, dept, salary FROM emp)
PIVOT (SUM(salary) FOR dept IN ('Eng', 'Sales'));
SELECT * FROM emp
UNPIVOT (salary FOR dept IN (eng_salary, sales_salary));
```
- Parser: PIVOT/UNPIVOT в FROM clause
- IR: `irpkPivot`, `irpkUnpivot`
- Executor: group by identity cols → aggregate per pivot value → create columns
- Subquery storage в `nkFrom.fromSubquery`
Тестове: 1 parser + 1 execution тест, всички зелени.
### 1.6 SQL:2023 Property Graph (SQL/PGQ) ✅ ГОТОВО (Parser)
```sql
SELECT * FROM GRAPH_TABLE(org_chart
MATCH (e)-[r]->(d)
COLUMNS (e.name, d.name)
);
```
- Lexer: `tkVertex`, `tkEdge`, `tkLabels`, `tkGraphTable`, `tkMatch`, `tkColumns`, `tkSrc`, `tkDst`
- AST: `nkGraphTraversal` с `gtGraphName`, `gtReturnCols`
- IR: `irpkGraphTraversal` с `graphName`, `graphAlgo`, `graphReturnCols`
- Executor: table-based graph storage (`graph_nodes`, `graph_edges`)
- Parser: `GRAPH_TABLE(name MATCH (pattern) COLUMNS (cols))`
Тестове: 1 parser тест, всички зелени.
---
## Част 1.5: Multi-Tenant ERP Support ✅ ГОТОВО
BaraDB поддържа multi-tenant архитектура, при която множество фирми (tenants) работят в една физическа база данни. Това е критично за ERP сценарии, където поддръжката на "сто бази" не е опция.
### Механизъм
| Компонент | Описание |
|-----------|----------|
| **Session Variables** | `SET app.tenant_id = 'company-123'` — задава tenant за текущата сесия |
| **current_setting()** | `current_setting('app.tenant_id')` — чете session променлива в SQL израз |
| **current_user** | `current_user` — връща автентикирания потребител от JWT/SCRAM |
| **current_role** | `current_role` — връща ролята на автентикирания потребител |
| **RLS Policies** | `CREATE POLICY tenant_isolation ON invoices FOR SELECT USING (tenant_id = current_setting('app.tenant_id'))` |
| **Auth Bridge** | `server.nim` и `httpserver.nim` попълват `ExecutionContext.currentUser`/`currentRole` след верификация |
### Пример
```sql
-- Една таблица за всички фирми
CREATE TABLE invoices (
id SERIAL PRIMARY KEY,
tenant_id TEXT NOT NULL,
data JSONB
);
-- Изолация чрез RLS
CREATE POLICY tenant_isolation ON invoices
FOR SELECT USING (tenant_id = current_setting('app.tenant_id'));
-- Всяка сесия вижда само своя tenant
SET app.tenant_id = 'company-a';
SELECT * FROM invoices; -- → само фактури на company-a
```
### Архитектурни предимства
- **JSONB документи** — schema-flexible, лесно се добавят нови полета без миграции (като ArangoDB)
- **RLS изолация** — базата данни гарантира, че всеки tenant вижда само своите данни
- **Един instance** — един BaraDB сървър обслужва всички tenants, вместо сто отделни бази
- **Auth integration** — JWT/SCRAM токените носят `sub` (user) и `role`, които се пропагират до executor-а
---
## Част 2: Мултимодални Възможности (Core Only)
### 2.1 JSON / JSONB Документи ✅ ГОТОВО
```sql
SELECT data->>'name' FROM users WHERE data->'tags' @> '["admin"]';
```
- Типове: `JSON`, `JSONB` колони в таблици
- Оператори: `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?&`, `?|`
- Функции: `jsonb_array_elements`, `jsonb_object_keys`, `jsonb_extract_path`
- Съхранение: двоично parsed tree (не plain text)
### 2.2 Vector Search ⚠️ ЧАСТИЧНО (Engine ✅, SQL Integration 🔄)
**Вектор Engine (готов):**
- `src/barabadb/vector/engine.nim` — HNSW index с cosine/euclidean distance
- `src/barabadb/vector/quant.nim` — IVF-PQ quantization
- `src/barabadb/vector/simd.nim` — SIMD оптимизации
- `src/barabadb/core/crossmodal.nim` — CrossModalEngine за хибридно търсене (vector + text)
**Липсваща SQL интеграция (базова — за стабилизация):**
```sql
-- Тип и колона
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
-- Index
CREATE VECTOR INDEX idx_items_vec ON items(embedding)
USING hnsw WITH (m = 16, ef_construction = 200, metric = 'cosine');
-- Query functions
SELECT id, cosine_distance(embedding, '[0.1, 0.2, ...]') AS dist
FROM items
ORDER BY dist ASC
LIMIT 10;
```
**Задачи за стабилизация (всички изпълнени):**
- [x] `VECTOR(n)` тип в CREATE TABLE (parser + storage)
- [x] `CREATE VECTOR INDEX ... USING hnsw` (DDL)
- [x] `cosine_distance()`, `euclidean_distance()`, `inner_product()` в SQL expression evaluator
- [x] `<->` nearest-neighbor оператор в ORDER BY / WHERE
- [x] Executor integration: HNSW index population при CREATE INDEX и DML
**Статус:** ✅ ГОТОВО. 8 SQL-level vector теста зелени.
### 2.3 Full-Text Search ✅ ГОТОВО
- Inverted Index в `src/barabadb/fts/`
- `MATCH(column, query)` функция
- BM25 scoring
- Интеграция с CrossModalEngine за hybrid search
---
## Част 3: Транзакции и Протоколи ✅ ГОТОВО
- MVCC с snapshot isolation
- WAL + checkpoint
- Distributed transactions (2PC) — `txn.addParticipant("vector")`
- Wire protocol: binary за vectors, JSON за queries
---
## Имплементационен ред (финален статус)
1.**Window Functions** (AST → Parser → IR → Executor → Tests)
2.**MERGE statement** (Parser → Executor → Tests)
3.**LATERAL JOIN** (Parser → Executor, correlated subquery strategy)
4.**GROUP BY + HAVING** (SUM/AVG/MIN/MAX, HAVING filter)
5.**FILTER clause** (COUNT/SUM/AVG FILTER (WHERE ...))
6.**ARRAY_AGG / STRING_AGG** (multi-arg aggregates)
7.**GROUPING SETS / ROLLUP / CUBE** (powerset generation)
8.**PIVOT / UNPIVOT** (row-to-column transformation)
9.**SQL/PGQ Property Graph** (GRAPH_TABLE MATCH parser)
10.**JSON/JSONB** (operators + functions)
11.**Full-Text Search** (inverted index + BM25)
12.**Vector Engine** (HNSW + IVF-PQ + SIMD)
13.**Vector SQL Integration** (тип, index, distance functions, <-> operator, ORDER BY)
---
## Крайно състояние
**340+ теста зелени.** Всички фундаментални SQL:2023 features имплементирани.
**Четирите свята:**
| Свят | Features | Статус |
|------|----------|--------|
| **SQL** | Window, MERGE, LATERAL, GROUP BY/HAVING, FILTER, ARRAY_AGG, STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT | ✅ |
| **JSON** | JSON/JSONB колони, `->` / `->>` оператори | ✅ |
| **Graph** | BFS/DFS/PageRank/Dijkstra engine + SQL/PGQ GRAPH_TABLE | ✅ |
| **Vector** | HNSW index, cosine/euclidean distance, IVF-PQ, SIMD | ✅ Engine<br>🔄 SQL glue |
| **FTS** | Inverted index, BM25, hybrid search | ✅ |
**Файлове модифицирани:**
- `lexer.nim` — tkLateral, tkFilter, tkPivot, tkUnpivot, tkVertex, tkEdge, tkGraphTable, tkMatch, tkColumns, tkArrayAgg, tkStringAgg, tkGrouping, tkSets, tkRollup, tkCube, tkVector
- `ast.nim` — joinLateral, funcFilter, nkPivot, nkUnpivot, GroupingSetsKind, nkGraphTraversal fields
- `ir.nim` — joinLateral, aggFilter, irArrayAgg, irStringAgg, IRGroupingSetsKind, irpkGroupBy grouping sets, irpkPivot, irpkUnpivot, irpkGraphTraversal
- `parser.nim` — LATERAL, FILTER, multi-arg aggregates, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE
- `executor.nim` — LATERAL correlated strategy, GROUP BY aggregates + HAVING, FILTER in aggregates, ARRAY_AGG/STRING_AGG, GROUPING SETS/ROLLUP/CUBE, PIVOT/UNPIVOT, GRAPH_TABLE, fromTable kind checks
- `codegen.nim` — irpkPivot, irpkUnpivot, irpkGraphTraversal
- `tests/test_all.nim` — 25+ нови теста
- `tests/join_tests.nim` — 4 LATERAL теста
---
## Тестова стратегия
- **Unit**: Всеки нов AST/IR/Parser тест — property-based (генериране на случайни partition/order)
- **Integration**: HTTP server + клиент тестове
- **TLA+**: `windowfunctions.tla` — deterministic partitioning semantics
- **Benchmark**: Window function performance vs PostgreSQL (опционално)
---
## Поправени грешки при тази сесия
- **Vector SQL Integration** — имплементиран пълен SQL glue за вектори (тип, индекс, функции, оператор)
- **MERGE тестове** — поправени чрез изолиране на тестовата директория (unique temp dir per suite)
- **Row storage escape** — `escapeRowVal()` в `execInsert` за стойности със запетай (vector literals)
- **ORDER BY + projection** — `irpkSort` сега е преди `irpkProject` в `lowerSelect`, което позволява `ORDER BY` по колони извън `SELECT`
- **GROUPING SETS execution** — `lowerSelect` сега проверява `selGroupingSetsKind != gskNone` освен `selGroupBy.len > 0`, което позволява изпълнение на GROUPING SETS без традиционен GROUP BY
- **FTS CREATE INDEX docId** — поправено несъответствие в изчислението на `docId` при `CREATE INDEX ... USING FTS` (сега използва хеш на `tableName.$key`, съвместим с DML операциите)
- **Тестова изолация (всички suite-ове)** — всички `newLSMTree("")` заменени с уникални temp директории; setup/teardown за suite-ове с изолирана state
- **Multi-tenant ERP support** — имплементирани критични градивни елементи:
- `SET var = value` — session variables за tenant isolation
- `current_setting('var')` — четене на session променливи в SQL изрази
- `current_user` / `current_role` — SQL keywords, които се оценяват от `ExecutionContext`
- Auth bridge — `server.nim` и `httpserver.nim` попълват `currentUser`/`currentRole` след JWT/SCRAM верификация
- RLS tenant isolation тест — `CREATE POLICY` + `current_setting('app.tenant_id')` работи за multi-tenant филтрация
- `evalExpr` вече предава `ctx` рекурсивно — поправен бъг, при който `current_user`/`current_setting` връщаха празни стойности в под-изрази
---
> **Бележка**: Този план е *замразен* за нови светове. Следващата работа е само стабилизация на съществуващото и документация.
+60
View File
@@ -0,0 +1,60 @@
# Nodebara Compatibility Fixes
This PR bundles three independent fixes discovered while integrating BaraDB with [NodeBB](https://nodebb.org/) (a large Node.js forum application). Each commit is self-contained and can be reviewed separately.
---
## 1. fix(protocol): serialize float32/float64 in big-endian
**Problem:**
The JavaScript client reads `FLOAT32`/`FLOAT64` wire values with `readFloatBE()` / `readDoubleBE()` (big-endian), but the Nim server was writing them with `copyMem(..., unsafeAddr fl, N)` — i.e. **native byte order**. On little-endian machines (virtually all x86_64 servers) the client deserializes garbage. Example: a zset `score = 1.0` becomes `3.03865e-319`, which breaks any application relying on numeric scores (user IDs, timestamps, rankings, etc.).
**Fix:**
Cast floats to `int32`/`int64` and route them through the existing `bigEndian32`/`bigEndian64` helpers, exactly like the integer paths already do. Same change applied to deserialization.
**Impact:**
Breaking fix for cross-platform wire compatibility. No API changes.
---
## 2. feat(client): add TCP request queue for safe concurrency
**Problem:**
`Client.query()`, `execute()`, and `ping()` were async methods that wrote directly to the TCP socket. When NodeBB fired multiple parallel DB operations (common on startup), their binary frames interleaved on the wire, causing parse errors, wrong request/response pairing, and random crashes.
**Fix:**
Introduce an internal `_requestQueue` + `_requestLock`. Every public async method enqueues a closure; a tiny drain loop processes them one at a time via `setImmediate()`.
**Impact:**
No breaking API change. Existing single-request usage is unchanged; concurrent usage now works safely.
---
## 3. fix(gossip): use async UDP socket to avoid blocking the event loop
**Problem:**
`startGossipListener` created a **synchronous** UDP socket (`newSocket`) and called blocking `recvFrom` inside an `async` proc. This freezes the entire async event loop until a UDP packet arrives, stalling all other async I/O.
**Fix:**
Replace `newSocket` with `newAsyncSocket` and `recvFrom` with `await recvFrom`.
**Impact:**
Non-breaking. Gossip remains optional; when enabled it no longer blocks the main loop.
---
## Testing
- [x] NodeBB v4.11.2 setup completes end-to-end
- [x] Admin login works (relies on correct FLOAT64 score deserialization)
- [x] Concurrent DB queries during startup no longer corrupt frames
- [x] Gossip listener no longer blocks other async tasks
---
## Checklist
- [x] Each commit is atomic and compiles on its own
- [x] No Nim compiler warnings introduced
- [x] JS client backward-compatible for single-request callers
- [x] Existing tests (`nimble test`) still pass
+43 -4
View File
@@ -4,8 +4,9 @@
**A multimodal database engine written in Nim — 100% native, zero dependencies.** **A multimodal database engine written in Nim — 100% native, zero dependencies.**
[![Version](https://img.shields.io/badge/version-1.1.0-blue.svg)](baradadb.nimble) [![Version](https://img.shields.io/badge/version-1.1.2-blue.svg)](baradadb.nimble)
[![Documentation](https://img.shields.io/badge/docs-7_languages-blue.svg)](docs/index.md) [![Documentation](https://img.shields.io/badge/docs-7_languages-blue.svg)](docs/index.md)
[![Stars](https://img.shields.io/github/stars/katehonz/barabaDB?style=social)](https://github.com/katehonz/barabaDB)
## Documentation ## Documentation
@@ -15,6 +16,8 @@ BaraDB combines document, graph, vector, columnar, and full-text search storage
in a single engine with a unified query language (BaraQL). It compiles to a in a single engine with a unified query language (BaraQL). It compiles to a
single 3.3MB binary with no runtime dependencies. single 3.3MB binary with no runtime dependencies.
⭐ Thank you to everyone who continues to star and support BaraDB on [GitHub](https://github.com/katehonz/barabaDB)!
> **Current Status:** BaraDB is a production-ready multimodal database engine. > **Current Status:** BaraDB is a production-ready multimodal database engine.
> All core storage engines, query processing, and protocol layers are fully > All core storage engines, query processing, and protocol layers are fully
> implemented and tested. See [Limitations](#current-limitations) below for > implemented and tested. See [Limitations](#current-limitations) below for
@@ -285,8 +288,23 @@ let range = btree.scan("key_a", "key_z")
### Vector Engine ### Vector Engine
Native HNSW and IVF-PQ indexes for similarity search. Native HNSW and IVF-PQ indexes for similarity search with full SQL integration.
```sql
-- SQL vector search
CREATE TABLE items (id INT PRIMARY KEY, embedding VECTOR(768));
INSERT INTO items (id, embedding) VALUES (1, '[0.1, 0.2, 0.3, ...]');
-- Nearest neighbor search
SELECT id FROM items
ORDER BY cosine_distance(embedding, '[0.1, 0.2, 0.3, ...]') ASC
LIMIT 10;
-- With HNSW index
CREATE INDEX idx_vec ON items(embedding) USING hnsw;
```
Native Nim API:
```nim ```nim
import barabadb/vector/engine import barabadb/vector/engine
@@ -301,7 +319,10 @@ let filtered = idx.searchWithFilter(queryVector, k = 10,
``` ```
Features: Features:
- **HNSW** — hierarchical navigable small world graph - **SQL vector types** — `VECTOR(n)` with dimension validation
- **SQL distance functions** — `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- **`<->` operator** — Euclidean distance nearest-neighbor shorthand
- **HNSW index** — `CREATE INDEX ... USING hnsw` with automatic maintenance
- **IVF-PQ** — inverted file index with product quantization - **IVF-PQ** — inverted file index with product quantization
- **Distance metrics** — cosine, euclidean, dot product, Manhattan - **Distance metrics** — cosine, euclidean, dot product, Manhattan
- **Quantization** — scalar 8-bit/4-bit, product, binary - **Quantization** — scalar 8-bit/4-bit, product, binary
@@ -606,6 +627,23 @@ See [docs/en/docker.md](docs/en/docker.md) for full Docker documentation.
| `BARADB_CERT_FILE` | — | TLS certificate path | | `BARADB_CERT_FILE` | — | TLS certificate path |
| `BARADB_KEY_FILE` | — | TLS private key path | | `BARADB_KEY_FILE` | — | TLS private key path |
## Built with BaraDB
### NodeBara
**[NodeBara](https://codeberg.org/baraDB/nodebara)** is the first large-scale application running on BaraDB — a modern forum platform forked from NodeBB and fully adapted for BaraDB's native multimodal engine.
- **Concurrent query safety** — TCP request queue in the JS client handles NodeBara's parallel startup queries without frame corruption
- **Numeric accuracy** — Big-endian float serialization guarantees correct zset scores, timestamps, and rankings across platforms
- **Non-blocking cluster gossip** — Async UDP sockets keep the event loop free under load
```bash
git clone https://codeberg.org/baraDB/nodebara
cd nodebara
npm install
npm run setup # uses BaraDB as the default database
```
## Client SDKs ## Client SDKs
BaraDB provides official clients for multiple languages: BaraDB provides official clients for multiple languages:
@@ -1214,7 +1252,7 @@ src/barabadb/
## Tests ## Tests
```bash ```bash
# Run all tests (262 tests, 56 suites) # Run all tests (340+ tests, 60+ suites)
nim c --path:src -r tests/test_all.nim nim c --path:src -r tests/test_all.nim
# Run benchmarks # Run benchmarks
@@ -1232,6 +1270,7 @@ nim c -d:release -r benchmarks/bench_all.nim
| Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 | | Protocol (binary + HTTP + WS + pool + auth + ratelimit) | ✅ | 100% | v1.0.0 |
| Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 | | Schema (inheritance + computed + migrations) | ✅ | 100% | v1.0.0 |
| Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 | | Vector engine (HNSW + IVF-PQ + quant + SIMD + metadata) | ✅ | 100% | v1.0.0 |
| Vector SQL Integration (VECTOR type, distance functions, <->, HNSW indexes) | ✅ | 100% | v1.1.0 |
| Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 | | Graph engine (all algorithms + pattern matching) | ✅ | 100% | v1.0.0 |
| FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 | | FTS (BM25 + TF-IDF + fuzzy + regex + multi-language) | ✅ | 100% | v1.0.0 |
| CLI shell | ✅ | 100% | v1.0.0 | | CLI shell | ✅ | 100% | v1.0.0 |
+149
View File
@@ -0,0 +1,149 @@
## BaraDB adapter mimicking db_sqlite API for NimForum
import std/strutils, std/tables, std/sequtils, std/parseutils
import ../../clients/nim/src/baradb/client as baradb_client
type
DbError* = object of CatchableError
SqlQuery* = distinct string
DbConn* = distinct pointer
proc sql*(query: string): SqlQuery =
SqlQuery(query)
proc dbError*(msg: string) {.noreturn.} =
var e: ref DbError
new(e)
e.msg = msg
raise e
proc dbQuote*(s: string): string =
result = "'"
for c in items(s):
if c == '\'': add(result, "''")
else: add(result, c)
add(result, '\'')
proc dbFormat*(formatstr: SqlQuery, args: varargs[string]): string =
var res = ""
var a = 0
for c in items(string(formatstr)):
if c == '?':
if a == args.len:
dbError("""The number of \"?\" given exceeds the number of parameters present in the query.""")
add(res, dbQuote(args[a]))
inc(a)
else:
add(res, c)
res
proc toClient(db: DbConn): SyncClient =
cast[SyncClient](db)
proc open*(connection, user, password, database: string): DbConn =
var config = defaultConfig()
if connection.contains(':'):
let parts = connection.split(':')
config.host = parts[0]
config.port = parseInt(parts[1])
elif connection.len > 0 and not connection.endsWith(".db"):
config.host = connection
config.database = database
config.username = user
config.password = password
var client = newSyncClient(config)
client.connect()
GC_ref(client)
return cast[DbConn](client)
proc close*(db: DbConn) =
let client = toClient(db)
baradb_client.close(client)
GC_unref(client)
proc exec*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]) =
let client = toClient(db)
let q = dbFormat(query, args)
discard client.query(q)
proc tryExec*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): bool =
try:
let client = toClient(db)
let q = dbFormat(query, args)
discard client.query(q)
return true
except:
return false
proc getRow*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): seq[string] =
let client = toClient(db)
let q = dbFormat(query, args)
let qr = client.query(q)
if qr.rows.len > 0:
return qr.rows[0]
else:
return newSeq[string](qr.columns.len)
proc getAllRows*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): seq[seq[string]] =
let client = toClient(db)
let q = dbFormat(query, args)
let qr = client.query(q)
return qr.rows
proc getValue*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): string =
let row = getRow(db, query, args)
if row.len > 0: return row[0]
return ""
iterator fastRows*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): seq[string] =
let client = toClient(db)
let q = dbFormat(query, args)
let qr = client.query(q)
for row in qr.rows:
yield row
iterator rows*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): seq[string] =
for r in fastRows(db, query, args): yield r
proc insertID*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): int64 =
let client = toClient(db)
let q = dbFormat(query, args)
discard client.query(q)
let sqlStr = string(query).strip().toLower()
var tableName = ""
if sqlStr.startsWith("insert into "):
let rest = sqlStr[12..^1]
let spacePos = rest.find(' ')
if spacePos > 0:
tableName = rest[0..<spacePos]
if tableName.len > 0:
let idQr = client.query("SELECT max(id) FROM " & tableName)
if idQr.rows.len > 0 and idQr.rows[0].len > 0:
try:
return parseInt(idQr.rows[0][0]).int64
except:
discard
return -1
proc tryInsertID*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): int64 =
try:
return insertID(db, query, args)
except:
return -1
proc nextId*(db: DbConn, tableName: string): int64 =
let client = toClient(db)
let qr = client.query("SELECT max(id) FROM " & tableName)
if qr.rows.len > 0 and qr.rows[0].len > 0:
try:
let current = parseInt(qr.rows[0][0])
return current.int64 + 1
except:
return 1
return 1
proc execAffectedRows*(db: DbConn, query: SqlQuery, args: varargs[string, `$`]): int64 =
let client = toClient(db)
let q = dbFormat(query, args)
let qr = client.query(q)
return qr.affectedRows.int64
+21
View File
@@ -0,0 +1,21 @@
import baradb_sqlite
var db = open("127.0.0.1:9472", "", "", "default")
db.exec(sql"CREATE TABLE test_adapter (id INT PRIMARY KEY, name STRING)")
db.exec(sql"INSERT INTO test_adapter (id, name) VALUES (1, 'hello')")
db.exec(sql"INSERT INTO test_adapter (id, name) VALUES (2, 'world')")
let rows = db.getAllRows(sql"SELECT * FROM test_adapter")
echo "All rows: ", rows
let row = db.getRow(sql"SELECT * FROM test_adapter WHERE id = 1")
echo "Row 1: ", row
let val = db.getValue(sql"SELECT name FROM test_adapter WHERE id = 2")
echo "Value: ", val
let cnt = db.getValue(sql"SELECT count(*) FROM test_adapter")
echo "Count: ", cnt
db.close()
+5 -5
View File
@@ -1,5 +1,5 @@
# Package # Package
version = "1.1.0" version = "1.1.2"
author = "BaraDB Team" author = "BaraDB Team"
description = "BaraDB — Multimodal database written in Nim" description = "BaraDB — Multimodal database written in Nim"
license = "Apache-2.0" license = "Apache-2.0"
@@ -15,13 +15,13 @@ requires "checksums >= 0.2.0"
# Tasks # Tasks
task build_debug, "Build debug version": task build_debug, "Build debug version":
exec "nim c -d:ssl --threads:on --debugger:native --linedir:on -o:build/baradadb src/baradadb.nim" exec "nim c --debugger:native --linedir:on -o:build/baradadb src/baradadb.nim"
task build_release, "Build release version": task build_release, "Build release version":
exec "nim c -d:ssl --threads:on -d:release --opt:speed -o:build/baradadb src/baradadb.nim" exec "nim c -d:release --opt:speed -o:build/baradadb src/baradadb.nim"
task test, "Run all tests": task test, "Run all tests":
exec "nim c --path:src -d:ssl --threads:on -r tests/test_all.nim" exec "nim c -r tests/test_all.nim"
task bench, "Run benchmarks": task bench, "Run benchmarks":
exec "nim c --path:src -d:ssl --threads:on -d:release -r benchmarks/bench_all.nim" exec "nim c -d:release -r benchmarks/bench_all.nim"
+1 -1
View File
@@ -20,7 +20,7 @@ npm install baradb
Or from source: Or from source:
```bash ```bash
git clone https://github.com/barabadb/baradadb.git git clone https://codeberg.org/baraba/baradb
cd clients/javascript cd clients/javascript
npm link npm link
``` ```
+30
View File
@@ -244,6 +244,8 @@ class Client {
this.requestId = 0; this.requestId = 0;
this._buffer = Buffer.alloc(0); this._buffer = Buffer.alloc(0);
this._pendingResolve = null; this._pendingResolve = null;
this._requestQueue = [];
this._requestLock = false;
} }
async connect() { async connect() {
@@ -478,8 +480,31 @@ class Client {
throw new Error(`Unexpected auth response: 0x${header.kind.toString(16)}`); throw new Error(`Unexpected auth response: 0x${header.kind.toString(16)}`);
} }
async _processQueue() {
if (this._requestLock || this._requestQueue.length === 0) return;
this._requestLock = true;
const { task, resolve, reject } = this._requestQueue.shift();
try {
const result = await task();
resolve(result);
} catch (err) {
reject(err);
} finally {
this._requestLock = false;
setImmediate(() => this._processQueue());
}
}
_enqueue(task) {
return new Promise((resolve, reject) => {
this._requestQueue.push({ task, resolve, reject });
this._processQueue();
});
}
async ping() { async ping() {
if (!this.connected) throw new Error('Not connected'); if (!this.connected) throw new Error('Not connected');
return this._enqueue(async () => {
const msg = this._build(MsgKind.PING, Buffer.alloc(0)); const msg = this._build(MsgKind.PING, Buffer.alloc(0));
this.socket.write(msg); this.socket.write(msg);
@@ -487,10 +512,12 @@ class Client {
if (header.kind === MsgKind.PONG) return true; if (header.kind === MsgKind.PONG) return true;
if (header.kind === MsgKind.ERROR) throw await this._readError(header.length); if (header.kind === MsgKind.ERROR) throw await this._readError(header.length);
return false; return false;
});
} }
async query(sql) { async query(sql) {
if (!this.connected) throw new Error('Not connected'); if (!this.connected) throw new Error('Not connected');
return this._enqueue(async () => {
const queryBuf = Buffer.from(sql, 'utf-8'); const queryBuf = Buffer.from(sql, 'utf-8');
const payload = Buffer.alloc(4 + queryBuf.length + 1); const payload = Buffer.alloc(4 + queryBuf.length + 1);
payload.writeUInt32BE(queryBuf.length, 0); payload.writeUInt32BE(queryBuf.length, 0);
@@ -510,10 +537,12 @@ class Client {
return result; return result;
} }
return new QueryResult(); return new QueryResult();
});
} }
async queryParams(sql, params = []) { async queryParams(sql, params = []) {
if (!this.connected) throw new Error('Not connected'); if (!this.connected) throw new Error('Not connected');
return this._enqueue(async () => {
const queryBuf = Buffer.from(sql, 'utf-8'); const queryBuf = Buffer.from(sql, 'utf-8');
const paramParts = []; const paramParts = [];
for (const p of params) { for (const p of params) {
@@ -542,6 +571,7 @@ class Client {
return result; return result;
} }
return new QueryResult(); return new QueryResult();
});
} }
async execute(sql) { async execute(sql) {
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "baradb", "name": "baradb",
"version": "1.1.0", "version": "1.1.2",
"description": "Official JavaScript/Node.js client for BaraDB — Multimodal Database Engine", "description": "Official JavaScript/Node.js client for BaraDB — Multimodal Database Engine",
"main": "baradb.js", "main": "baradb.js",
"types": "baradb.d.ts", "types": "baradb.d.ts",
+2 -2
View File
@@ -8,8 +8,8 @@ const assert = require('node:assert');
const net = require('net'); const net = require('net');
const { Client, WireValue, QueryBuilder } = require('../baradb'); const { Client, WireValue, QueryBuilder } = require('../baradb');
const HOST = 'localhost'; const HOST = process.env.BARADB_HOST || 'localhost';
const PORT = 9472; const PORT = parseInt(process.env.BARADB_PORT || '9472', 10);
function serverAvailable() { function serverAvailable() {
return new Promise((resolve) => { return new Promise((resolve) => {
+19 -2
View File
@@ -15,13 +15,13 @@ Official Nim client for **BaraDB** — a multimodal database engine.
Add to your `.nimble` file: Add to your `.nimble` file:
```nim ```nim
requires "baradb >= 1.0.0" requires "baradb >= 1.1.0"
``` ```
Or clone locally: Or clone locally:
```bash ```bash
git clone https://github.com/barabadb/baradadb.git git clone https://codeberg.org/baraba/baradb
cd clients/nim cd clients/nim
nimble develop nimble develop
``` ```
@@ -141,6 +141,23 @@ nim c -r tests/test_integration.nim
Same as async but blocking. Available on `SyncClient`. Same as async but blocking. Available on `SyncClient`.
## Ormin Integration
The Nim client ships with an [Ormin](https://github.com/Araq/ormin) backend for compile-time checked SQL queries.
```nim
import ormin
importModel(DbBackend.baradb, "my_model")
let db {.global.} = open("127.0.0.1:9472", "admin", "", "default")
let rows = query:
select users(id, name)
where name == ?"alice"
```
See `examples/ormin_basic.nim` for a full sample.
## License ## License
Apache-2.0 Apache-2.0
+1 -1
View File
@@ -1,6 +1,6 @@
# Package # Package
version = "1.1.0" version = "1.1.2"
author = "BaraDB Team" author = "BaraDB Team"
description = "Official Nim client for BaraDB — async binary protocol client" description = "Official Nim client for BaraDB — async binary protocol client"
license = "Apache-2.0" license = "Apache-2.0"
+68
View File
@@ -0,0 +1,68 @@
name: CI
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:14
env:
POSTGRES_PASSWORD: test
POSTGRES_USER: test
POSTGRES_DB: test_ormin
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
strategy:
matrix:
nim:
- stable
name: Test on nim ${{ matrix.nim }}
steps:
- name: Checkout source code
uses: actions/checkout@v4
- name: Cache choosenim
id: cache-choosenim
uses: actions/cache@v4
with:
path: ~/.choosenim
key: ${{ runner.os }}-choosenim-stable
- name: Cache nimble
id: cache-nimble
uses: actions/cache@v4
with:
path: ~/.nimble
key: ${{ runner.os }}-nimble-stable
- name: Setup nim
uses: jiro4989/setup-nim-action@v2
with:
nim-version: ${{ matrix.nim }}
- name: Install dependencies
run: |
nimble --useSystemNim install -d
nimble --useSystemNim install karax -y
nimble --useSystemNim install jester -y
nimble --useSystemNim install websocket -y
- name: Run test sqlite
run: nimble --useSystemNim test
- name: Run test postgres
run: nimble --useSystemNim test_postgres
env:
PGPASSWORD: test
- name: Build examples
run: nimble buildexamples
+29
View File
@@ -0,0 +1,29 @@
*
!*/
!*.*
*.db
*.js
nimcache
tools/ormin_importer
tests/forum_model_sqlite.nim
tests/forum_model_postgres.nim
tests/model_postgre.nim
tests/model_sqlite.nim
tests/tfeature
tests/tcommon
tests/tpostgre
tests/tsqlite
examples/chat/chat_model.nim
examples/chat/chatclient.nim
examples/chat/createdb
examples/chat/server
examples/forum/forum_model.nim
examples/forum/forumclient.nim
examples/forum/forum
examples/forum/forumproto
examples/tweeter/src/createdatabase
examples/tweeter/src/tweeterdeps/
deps/
nim.cfg
.nimcache
tests/tdb_utils
+28
View File
@@ -0,0 +1,28 @@
sudo: false
language: c
os: linux
install:
- git clone https://github.com/nim-lang/nim
- cd nim
- git clone --depth 1 https://github.com/nim-lang/csources.git
- cd csources
- sh build.sh
- cd ../
- bin/nim c koch
- ./koch boot -d:release
- ./koch tools
- cd ..
before_script:
- set -e
- export PATH=$(pwd)/nim/bin:$(pwd):$PATH
script:
- nimble develop -y
- nim c tools/ormin_importer
- cd examples/forum
- ../../tools/ormin_importer forum_model.sql
- nim c forum.nim
- nim c forumproto.nim
- cd ../..
- cd examples/chat
- ../../tools/ormin_importer chat_model.sql
- nim c server.nim
+147
View File
@@ -0,0 +1,147 @@
# Ormin + BaraDB
This is a fork of [Ormin](https://github.com/Araq/ormin) (prepared SQL statement generator for Nim) with added support for **BaraDB** — a multimodal database engine with a binary wire protocol.
## What changed?
- Added `DbBackend.baradb` to the backend enum.
- Added `ImportTarget.baradb` for schema import.
- New backend file `ormin/ormin_baradb.nim` that bridges Ormin's compile-time query DSL to the BaraDB Nim client (`baradb/client`).
## Installation
Install both the BaraDB client and this Ormin fork:
```bash
# Install BaraDB client (from barabadb/clients/nim)
cd clients/nim
nimble install
# Install Ormin with BaraDB support
cd ormin
nimble install
```
## Quick Start
### 1. Write your schema
Create a file named `model.sql`:
```sql
CREATE TABLE users (
id SERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL,
email VARCHAR(255),
age INT
);
```
### 2. Use Ormin DSL in Nim
```nim
import json
import ormin
importModel(DbBackend.baradb, "model")
let db {.global.} = open("127.0.0.1:9472", "admin", "", "default")
# Select tuples
proc listUsers() =
let rows = query:
select users(id, name, email)
orderby id
for r in rows:
echo "User #", r.id, ": ", r.name
# Parameterized query
proc findUser(name: string) =
let row = query:
select users(id, name, email)
where name == ?name
limit 1
echo row
# Insert
proc addUser(name, email: string; age: int) =
query:
insert users(name = ?name, email = ?email, age = ?age)
# JSON output
proc usersAsJson(): seq[JsonNode] =
result = query:
select users(id, name)
produce json
when isMainModule:
listUsers()
```
Compile with:
```bash
nim c -r myapp.nim
```
## Connection string format
`open(host:port, username, password, database)`
```nim
let db = open("127.0.0.1:9472", "admin", "", "default")
```
If you omit the port, the default `9472` is used.
## Supported Ormin features
| Feature | Status |
|---------------------|--------|
| `select` | ✅ |
| `where` | ✅ |
| `join` / `leftjoin` | ✅ |
| `insert` | ✅ |
| `update` | ✅ |
| `delete` | ✅ |
| `orderby` | ✅ |
| `limit` / `offset` | ✅ |
| `?` placeholders | ✅ |
| `%` JSON params | ✅ |
| `produce json` | ✅ |
| `query(T)` typed | ✅ |
| `createProc` | ✅ |
| `createIter` | ✅ |
| `transaction` | ✅ (top-level only) |
| `returning` | ❌ (not yet supported by BaraDB server) |
## Limitations
- **Wire protocol strings**: BaraDB returns all column values as strings over the wire. The backend parses them at runtime. This is slightly less efficient than native SQLite/PostgreSQL bindings but keeps the implementation simple and portable.
- **`getLastId` / `RETURNING`**: BaraDB's SQL parser recognizes `RETURNING`, but the executor ignores it — the result is always just the affected-row count. For now, use a `limit 1` `select` after `insert` to fetch the new row.
- **Nested transactions (SAVEPOINT)**: Not supported by BaraDB. Top-level `BEGIN` / `COMMIT` / `ROLLBACK` work fine, but nested `transaction:` blocks (which Ormin implements via `SAVEPOINT`) will fail.
- **Async**: Ormin's DSL is synchronous by design. This backend uses `SyncClient` under the hood. If you need async, use the raw `baradb/client` async API directly.
## Why some features are missing
These are **server-side limitations**, not client bugs:
| Feature | Status in BaraDB server | Needed for |
|---------|------------------------|------------|
| `SAVEPOINT` | Not implemented | Nested `transaction:` blocks |
| `RETURNING` | Parsed but ignored by executor | `insert ... returning id` |
If you need these, open an issue on the [BaraDB server repo](https://git.invoicing.top/baraba/Baradb) — the fixes belong in `src/barabadb/query/executor.nim` and `src/barabadb/query/parser.nim`.
## Running the example
```bash
cd clients/nim/ormin/examples
nim c -r baradb_basic.nim
```
*(Requires a BaraDB server on `localhost:9472`.)*
## License
Same as upstream Ormin — MIT.
+473
View File
@@ -0,0 +1,473 @@
# ormin
Prepared SQL statement generator for Nim. A lightweight ORM.
Features:
- Compile time query checking: Types as well as table
and column names are checked, no surprises at runtime!
- Automatic join generation: Ormin knows the table
relations and can compute the "natural" join for you!
- Nim based DSL for queries: No syntax errors at runtime,
no SQL injections possible.
- Generated prepared statements: As fast as low level
hand written API calls!
- First class JSON support: No explicit conversions
from rows to JSON objects required.
TODO:
- Better support for complex nested queries.
- Write mysql backend.
## Schema and Database Setup
1. **Generate a model from SQL** Place your schema in an `.sql` file and import it using `importModel`. By default this runs `ormin_importer` and includes the generated Nim code. Pass `includeStatic = true` to generate the model directly at compile time from the SQL file instead.
2. **Create a database connection** Ormin expects a global connection named `db` when issuing queries. The library ships drivers for SQLite and PostgreSQL; pick the matching backend in `importModel` and open a connection with Nim's database modules.
### Static Schema
The SQL file can be easily embedded at compile time. Import `ormin/db_utils` and use `const mySql = staticLoad("schema.sql")`, which returns a distinct string type `DbSql` after running sanity-checks the SQL. Pass that const `DbSql` to `createTable` / `dropTable` overloads:
```nim
import ormin/db_utils
const schema = staticLoad("model.sql")
db.createTable(schema)
db.createTable(schema, "quoted table")
db.dropTable(schema)
db.dropTable(schema, "quoted table")
```
If you already use `importModel`, you can opt into the same static path directly there. `includeStatic = true` skips the generated `.nim` file, builds the model metadata from the `.sql` file at compile time, and exposes `sqlSchema` automatically:
```nim
import ../ormin
importModel(DbBackend.sqlite, "model_sqlite", includeStatic = true)
db.createTable(sqlSchema)
db.dropTable(sqlSchema, "tb_timestamp")
```
### SQLite
```nim
import ../ormin
importModel(DbBackend.sqlite, "model_sqlite")
let db {.global.} = open(":memory:", "", "", "")
```
Note: Ormin now properly handles quoted table names in `dropTable`. The compile flag `-d:orminLegacySqliteDropNames` restores that older drop-table behavior by using the normalized lookup name instead of the preserved SQL identifier. The old behavior only worked in SQlite, not Postgres.
### PostgreSQL
```nim
import postgres
import ../ormin
importModel(DbBackend.postgre, "model_postgre")
let db {.global.} = open("localhost", "user", "password", "dbname")
```
### BaraDB
```nim
import ormin
importModel(DbBackend.baradb, "model_baradb")
let db {.global.} = open("127.0.0.1:9472", "admin", "", "default")
```
## Query DSL
`query:` blocks are turned into prepared statements at compile time. Placeholders use `?` for Nim values and `%` for JSON values; Ormin chooses JSON instead of an ad-hoc variant type so your data can flow straight from/into `JsonNode` trees. `!!` splices vendor-specific SQL fragments. Typical clauses such as `with`, `where`, joins, `orderby`, `groupby`, `limit`, `offset`, `exists`, `distinct`, window expressions, `union`/`intersect`/`except`, `returning`, and insert upserts via `onconflict` + (`donothing` or `doupdate`) are supported. Referring to columns from related tables can trigger **automatic join generation** based on foreign keys, reducing boilerplate joins.
Example snippets:
```nim
# Select recent rows from a Messages table with a Nim parameter
let recentMessages = query:
select Messages(content, creation, author)
orderby desc(creation)
limit ?maxMessages
# Insert using Nim and JSON parameters
let payload = %*{"dt2": %*"2023-10-01T00:00:00Z"}
query:
insert tb_timestamp(dt1 = ?dt1, dt2 = %payload["dt2"])
# Upsert on conflict (SQLite/PostgreSQL)
query:
insert tb_nullable(id = ?id, note = ?note)
onconflict(id)
doupdate(note = ?note)
# Conditional upsert update
query:
insert tb_nullable(id = ?id, note = ?note)
onconflict(id)
doupdate(note = ?note)
where note != ?note
# Ignore duplicates
query:
insert tb_nullable(id = ?id, note = ?note)
onconflict(id)
donothing()
# Note: plain INSERT ... VALUES does not support `where`; use `onconflict(...)+doupdate(...)+where ...`
# Explicit join with filter
let rows = query:
select Post(author)
leftjoin Person(name) on author == id
where id == ?postId
# Automatic join generated from foreign keys
let postsWithAuthors = query:
select Post(title)
join Author(name)
where author.name == ?userName
# DISTINCT queries and COUNT(DISTINCT ...)
let authorIds = query:
select `distinct` Post(author)
let authorCount = query:
select Post(count(distinct author))
# NULL predicates use `nil` or `null`
let unassigned = query:
select Ticket(id)
where assignee == nil
# Pattern matching uses backticked infix operators
let matchingPeople = query:
select Person(id, name)
where name `like` ?"john%"
# EXISTS / NOT EXISTS subqueries
let peopleWithPosts = query:
select Person(id)
where exists(select Post(id) where author == ?personId)
# CTEs use with cteName(select ...)
let recentAuthors = query:
with recent(select Post(id, author) where id <= 3)
select recent(author)
# Window functions use over(expr, ...)
let rankedPosts = query:
select Post(author, id, over(row_number(), partitionby(author), orderby(id)) as rn)
# Set operations can be written inline between select queries
let mergedIds = query:
select Person(id) where id <= 2
union
select Person(id) where id >= 4
# Multiple joins with pagination
let page = query:
select Post(title)
join Person(name) on author == id
join Category(title) on category == id
orderby desc(post.creation)
limit 5 offset 10
# Vendor-specific function via raw SQL splice
query:
update Users(lastOnline = !!"DATETIME('now')")
where id == ?userId
```
Compile with `-d:debugOrminSql` to see the produced SQL at build time, which helps when experimenting with the DSL.
`tryQuery` executes a query but ignores database errors. `createProc` and `createIter` wrap a `query` block into a callable method on `db` for reuse.
### Select and Joins
Selecting columns for the primary table is done using the syntax `select Post(title, author, ...)` where `Post` is the table and `title`, `author`, etc are columns of that table. This will return a tuple containing `(title, author, ...)`. Only one table can be selected and columns must be from that table. Unlike in SQL, columns for joined tables are selected directly in the `join` syntax.
Joins use the syntax `join Person(name, city) on author == id` where `Person` is the table and the columns `name`, and `city` are columns of that table. Often the join condition can be inferred from foreign keys and can be left out: `join Author(name, city)`. The columns listed in the joined tabled will be appended to the results tuple, i.e. `(title, author, name, city)`. Supported joins are: `join`, `innerjoin`, `leftjoin`, `leftouterjoin`, `rightjoin`, `rightouterjoin`, `fulljoin`, `fullouterjoin`, `crossjoin`, and the legacy `outerjoin`. Runtime support for `rightjoin` / `fulljoin` still depends on the SQL backend.
The join syntax differs from SQL but simplifies selecting fields from multiple tables by making them more explicit while still maintaing SQL's full query capabilities.
### Return Types
The core return type for queries is a sequence of tuples where the tuples fields are the types of the columns. Some queries with `returning` or `limit` clauses will return singular values or raise a DbError.
- Selecting multiple columns returns a sequence of tuples of the inferred Nim types.
- Selecting a single column produces a sequence of that Nim type, e.g. `let names: seq[string] = query: select person(name)`.
- `produce json` emits `JsonNode` objects instead of Nim tuples; `produce nim` forces standard Nim results.
- `returning` or `limit 1` make the query return a single value or tuple instead of a sequence.
- Generated procedures/iterators return the same types as the underlying query (see `createProc`/`createIter` tests).
Examples:
```nim
# Sequence of tuples
let threads = query:
select thread(id, name)
# Sequence of simple Nim types
let ids = query:
select thread(id)
# JSON result
let threadJson = query:
select thread(id, name)
produce json
# Force Nim tuple even if `produce json` was used earlier
let threadNim = query:
select thread(id, name)
produce nim
```
Single tuples or values can be returned in some cases:
```nim
# Single value returning
let newId = query:
insert thread(name = ?"topic")
returning id
# Single row value returning when limit is a const `1`
let newId = query:
select thread(name = ?"topic")
orderby desc(trhead.id)
limit 1
```
**Note**: use an integer arg to limit to return a sequence instead!
```nim
let n = 1
let newId = query:
select thread(name = ?"topic")
orderby desc(trhead.id)
limit ?n
```
### Typed Queries
Use `query(T):` when you want Ormin to deserialize selected columns directly into a named Nim type instead of returning the default tuple shape. This is useful at module boundaries where a named object, ref object, or scalar domain type is clearer than a tuple.
For object results, selected column names must match fields on the destination type. Use `as` aliases when the database column name differs from the Nim field name:
```nim
type
ThreadSummary = object
id: int
title: string
let threads = query(ThreadSummary):
select thread(id, name as title)
orderby id
```
Selecting one column can map directly to a scalar type:
```nim
let names = query(string):
select thread(name)
```
Queries that return a single row, such as a `limit 1` query, return one `T` instead of `seq[T]`:
```nim
let thread = query(ThreadSummary):
select thread(id, name as title)
where id == ?threadId
limit 1
```
#### `fromQueryHook` Column Hooks
Typed queries deserialize each selected column through `fromQueryHook`. You can overload this hook for your own field or scalar destination types:
```nim
import ormin/query_hooks
type
TitleLength = distinct int
ThreadTitleSize = object
id: int
title: TitleLength
proc fromQueryHook*(val: var TitleLength, value: string) =
val = TitleLength(value.len)
let rows = query(ThreadTitleSize):
select thread(id, name as title)
```
If a hook needs to handle SQL `NULL` itself, accept a `DbValue[SourceType]`:
```nim
type
NullableTitle = distinct string
proc fromQueryHook*(val: var NullableTitle, value: DbValue[string]) =
if value.isNull:
val = NullableTitle("<untitled>")
else:
val = NullableTitle(value.value)
```
These are column deserialization hooks. In object typed queries, Ormin calls `fromQueryHook` separately for each selected column that maps to a destination field; it does not currently call a hook for the entire row object. For whole-row transformations, query into an intermediate typed result and convert it in regular Nim code.
### JSON and Raw SQL
JSON values can be spliced directly using `%` expressions. The `%` prefix tells Ormin to treat the following Nim expression as a `JsonNode` without conversion:
```nim
import json
let payload = %*{"id": %*1, "meta": %*{"tags": %*["nim", "orm"]}}
# Use JSON in WHERE clause
let rows = query:
select post(id, title)
where id == %payload["id"]
produce json
# Insert a row using JSON fields
query:
insert post(id = %payload["id"], title = %payload["title"], info = %payload["meta"])
```
`!!"RAW"` injects a literal SQL fragment for vendor-specific functions or clauses that Ormin does not know about:
```nim
query:
update users(lastOnline = !!"DATETIME('now')")
where id == ?userId
```
The tests include additional samples of JSON parameters and raw SQL expressions.
### Custom SQL Functions
Use the `{.importSql.}` pragma to tell Ormin about additional SQL functions that your database provides. Declare a Nim proc or func that mirrors the SQL signature and mark it with the pragma; the declaration does not need an implementation because Ormin only uses it to register the function for the query DSL.
```nim
proc substr(s: string; start, length: int): string {.importSql.}
let name = "foo"
let rows = query:
select tb_string(substr(typstring, 1, 5))
where substr(typstring, 1, 5) == ?name
```
Imported functions participate in compile-time checking for arity and return type so they can be composed with regular Ormin expressions.
**Limitation:** argument types are currently not validated, so using mismatched parameter types still compiles—ensure the arguments you pass match what the underlying SQL function expects.
## Transactions and Batching
Use `transaction:` to run multiple queries atomically. The block commits on success and rolls back on any exception. Nesting is supported via savepoints. `tryTransaction:` behaves the same but returns `bool` (false on database errors) without raising.
Examples:
```nim
# Commit on success
transaction:
query:
insert person(id = ?(1), name = ?"alice", password = ?"p", email = ?"a@x", salt = ?"s", status = ?"ok")
query:
update thread(views = views + 1)
where id == ?(42)
# Rollback on error
let ok = tryTransaction:
query:
insert person(id = ?(2), name = ?"bob", password = ?"p", email = ?"b@x", salt = ?"s", status = ?"ok")
# Primary key violation => entire block is rolled back, ok = false
query:
insert person(id = ?(2), name = ?"duplicate", password = ?"p", email = ?"d@x", salt = ?"s", status = ?"x")
# Nested transactions via savepoints
transaction:
query:
insert person(id = ?(3), name = ?"carol", password = ?"p", email = ?"c@x", salt = ?"s", status = ?"ok")
let innerOk = tryTransaction:
# This will fail and roll back to the savepoint
query:
insert person(id = ?(3), name = ?"duplicate", password = ?"p", email = ?"d@x", salt = ?"s", status = ?"x")
doAssert innerOk == false
# Continue outer transaction normally
```
PostgreSQL and SQLite are supported. The macros use `BEGIN/COMMIT/ROLLBACK` for the outermost transaction and `SAVEPOINT/RELEASE/ROLLBACK TO` for nested scopes.
## Reusable Procedures and Iterators
`createProc` turns a query into a procedure that returns all rows at once:
```nim
createProc postsByAuthor:
select post(id, title)
where author == ?userId
let posts = db.postsByAuthor(userId)
```
`createIter` emits an iterator that yields rows lazily:
```nim
createIter postsIter:
select post(id, title)
where author == ?userId
for row in db.postsIter(userId):
echo row.title
```
Both forms accept parameters matching the `?`/`%` placeholders and produce the same return types as an inline `query` block.
Inline `query` blocks resolve `db` from the current lexical scope, so a proc parameter or local `db` binding can override the global connection when needed. This is useful for making procs which need to do complex handling:
```nim
proc loadUser(db: DbConn; userId: int): User =
let row = query:
select user(id, name, email)
where id == ?userId
limit 1
User(id: row.id, name: row.name, email: row.email)
```
## Running Arbitrary SQL
The standard `db_connector` APIs can be imported and used. For example:
```nim
discard db.getValue(sql"select setval('antibot_id_seq', 10, false)")
```
## Additional Facilities
- **Protocol DSL** The `protocol` macro lets you describe paired server/client handlers that communicate via JSON messages. Sections use keywords like `recv`, `broadcast` and `send`, and every server block must be mirrored by a client block. The chat example demonstrates this code generation.
- **JSON Dispatcher** `createDispatcher` constructs a dispatcher for textual commands mapped to Nim procedures.
- **WebSocket Server** `serverws` provides a small WebSocket server that can broadcast messages to selected receivers via the `serve` proc.
## Tooling
The repository ships with `tools/ormin_importer`, used by the default `importModel` path, to parse SQL schema files into Nim type information and write the generated `.nim` model file.
## Examples
The `examples/` directory contains small applications (chat, forum, tweeter) demonstrating schema import, query blocks and the protocol/WebSocket features.
## Testing
Run the full test suite via Nimble:
```bash
nimble test
```
## License
Ormin is released under the MIT license.
+47
View File
@@ -0,0 +1,47 @@
switch("nimcache", ".nimcache")
task buildimporter, "Build ormin_importer":
exec "nim c -o:./tools/ormin_importer tools/ormin_importer"
task clean, "Clean generated files":
rmFile("tests/forum_model_sqlite.nim")
rmFile("tests/model_sqlite.nim")
rmFile("tests/forum_model_postgres.nim")
rmFile("tests/model_postgre.nim")
task test, "Run all test suite":
buildimporterTask()
cleanTask()
exec "nim c -f -r tests/tfeature"
exec "nim c -f -r tests/tcommon"
exec "nim c -f -r -d:release tests/tquery_types"
exec "nim c -f -r tests/tsqlite"
exec "nim c -f -r tests/tdb_utils"
exec "nim c -f -r tests/timportstatic"
task setup_postgres, "Ensure local Postgres has test DB/user":
# Use a simple script to avoid Nim/psql quoting pitfalls
exec "bash -lc 'bash tools/setup_postgres.sh'"
task test_postgres, "Run PostgreSQL test suite":
cleanTask()
buildimporterTask()
# setup_postgresTask()
# Pre-generate Postgres models to avoid include timing issues
exec "./tools/ormin_importer tests/forum_model_postgres.sql"
exec "./tools/ormin_importer tests/model_postgre.sql"
exec "nim c -f -d:nimDebugDlOpen -r -d:postgre tests/tfeature"
exec "nim c -f -d:nimDebugDlOpen -r -d:postgre tests/tcommon"
exec "nim c -f -d:nimDebugDlOpen -r -d:release -d:postgre tests/tquery_types"
exec "nim c -f -d:nimDebugDlOpen -r -d:postgre tests/tpostgre"
task buildexamples, "Build examples: chat and forum":
buildimporterTask()
selfExec "c examples/chat/server"
selfExec "js examples/chat/frontend"
selfExec "c examples/forum/forum"
selfExec "c examples/forum/forumproto"
selfExec "c examples/tweeter/src/tweeter"
+29
View File
@@ -0,0 +1,29 @@
#!/bin/sh
set -eu
# Ormin + BaraDB Dev Entrypoint
# Handles dependency installation and optional example compilation.
DEPS_FLAG="/workspace/.deps-installed"
# Install dependencies once
if [ ! -f "$DEPS_FLAG" ]; then
echo "[ormin-dev] Installing BaraDB client..."
cd /workspace/baradb
nimble install -y
echo "[ormin-dev] Installing Ormin dependencies..."
cd /workspace/ormin
nimble install -y
touch "$DEPS_FLAG"
echo "[ormin-dev] Dependencies ready."
fi
# If no arguments, drop to shell
if [ $# -eq 0 ]; then
exec bash
fi
# Otherwise run the provided command
exec "$@"
@@ -0,0 +1,35 @@
## Basic Ormin + BaraDB example
##
## Run with:
## nim c -r examples/baradb_basic.nim
##
## Requires a BaraDB server on localhost:9472.
import ormin
importModel(DbBackend.baradb, "baradb_model")
let db {.global.} = open("127.0.0.1:9472", "admin", "", "default")
proc listUsers() =
let rows = query:
select users(id, name, email)
orderby id
for r in rows:
echo "User #", r.id, ": ", r.name, " <", r.email, ">"
proc findUserByName(name: string) =
let row = query:
select users(id, name, email)
where name == ?name
limit 1
echo "Found: ", row
proc insertUser(name, email: string; age: int) =
query:
insert users(name = ?name, email = ?email, age = ?age)
when isMainModule:
listUsers()
findUserByName("alice")
insertUser("bob", "bob@example.com", 30)
@@ -0,0 +1,6 @@
CREATE TABLE users (
id SERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL,
email VARCHAR(255),
age INT
);
@@ -0,0 +1,19 @@
create table if not exists users(
id integer primary key,
name varchar(20) not null,
password varchar(32) not null,
creation timestamp not null default (DATETIME('now')),
lastOnline timestamp not null default (DATETIME('now'))
);
/* Names need to be unique: */
create unique index if not exists UserNameIx on users(name);
create table if not exists messages(
id integer primary key,
author integer not null,
content varchar(1000) not null,
creation timestamp not null default (DATETIME('now')),
foreign key (author) references users(id)
);
@@ -0,0 +1,9 @@
import strutils, db_sqlite
var db = open(connection="chat.db", user="araq", password="",
database="chat")
let model = readFile("chat_model.sql")
for m in model.split(';'):
if m.strip != "":
db.exec(sql(m), [])
db.close()
@@ -0,0 +1,92 @@
## Frontend for our chat application example.
## You need to have 'karax' installed in order for this to work.
import karax / [kbase, karax, vdom, karaxdsl, kajax, jwebsockets, jjson,
jstrutils, errors]
# Custom UI element with input validation:
proc validateNotEmpty(field: kstring): proc () =
result = proc () =
let x = getVNodeById(field)
if x.text.isNil or x.text == "":
setError(field, field & " must not be empty")
else:
setError(field, "")
proc loginField(desc, field, class: kstring;
validator: proc (field: kstring): proc ()): VNode =
result = buildHtml(tdiv):
label(`for` = field):
text desc
input(`type` = class, id = field, onchange = validator(field))
# here we setup the connection to the server:
let conn = newWebSocket("ws://localhost:8080", "orminchat")
proc send(msg: JsonNode) =
# The Ormin "protocol" requires us to have 'send' implementation.
conn.send(toJson(msg))
type User* = ref object
name*, password: kstring
const
username = kstring"username"
password = kstring"password"
message = kstring"message"
# Include the 'chatclient' helper that Ormin produced for us:
include chatclient
template loggedIn(): bool = userId > 0
proc doLogin() =
registerUser(User(name: getVNodeById(username).text,
password: getVNodeById(password).text))
proc doSendMessage() =
let inputField = getVNodeById(message)
sendMessage(TextMessage(author: userId, content: inputField.text))
inputField.setInputText ""
proc registerOnUpdate() =
conn.onmessage =
proc (e: MessageEvent) =
let msg = fromJson[JsonNode](e.data)
# 'recvMsg' was generated for us:
recvMsg(msg)
karax.redraw()
proc main(): VNode =
result = buildHtml(tdiv):
tdiv:
if not loggedIn:
loginField("Name :", username, "input", validateNotEmpty)
loginField("Password: ", password, "password", validateNotEmpty)
button(onclick = doLogin, disabled = disableOnError()):
text "Login"
p:
text getError(username)
p:
text getError(password)
tdiv:
table:
for m in allMessages:
tr:
td:
bold:
text m.name
td:
text m.content
tdiv:
if loggedIn:
label(`for` = message):
text "Message: "
input(class = "input", id = message, onkeyupenter = doSendMessage)
registerOnUpdate()
runLater proc() =
getRecentMessages()
setRenderer(main)
@@ -0,0 +1,96 @@
import ../../ormin / [serverws]
import ../../ormin
import json
# Ormin needs to know about our SQL model:
importModel(DbBackend.sqlite, "chat_model")
# Currently Ormin assumes a global database connection that needs to be
# annotated as '.global' for technical reasons. Later versions will improve.
var db {.global.} = open("chat.db", "", "", "")
# Ormin produces the file "chatclient.nim" for our frontend:
protocol "chatclient.nim":
# A 'common' code section is shared by both the client and the server:
common:
when not defined(js):
type kstring = string
type
inetType = kstring
varcharType = kstring
timestampType = kstring
intType = int
server "get recent messages":
let lastMessages = query:
select messages(content, creation, author)
join users(name)
orderby desc(creation)
limit 100
send(lastMessages)
client "get recent messages":
type TextMessage* = ref object
# Ormin fills in the fields of 'TextMessage' for us, based on the
# query in the 'server' part.
var allMessages*: seq[TextMessage] = @[]
proc getRecentMessages*()
allMessages = recv()
server "send chat message":
let userId = arg["author"].num
# unregistered users cannot send anything:
if userId == 0: return
query:
insert messages(content = %arg["content"], author = ?userId)
query:
update users(lastOnline = !!"DATETIME('now')")
where id == ?userId
let lastMessage = query:
select messages(content, creation, author)
join users(name)
orderby desc(creation)
limit 1
broadcast(lastMessage)
client "send chat message":
proc sendMessage*(m: TextMessage)
allMessages.add recv(TextMessage)
# This is the request to register/login a new user:
server "register/login a new user":
var candidates = query:
produce nim
select users(id, password)
where name == %arg["name"]
if candidates.len == 0:
let userId = query:
produce nim
insert users(name = %arg["name"], password = %arg["password"])
returning id
send(%userId)
else:
block search:
for c in candidates:
if c[1] == arg["password"].str:
send(%c[0])
echo "login found!"
break search
# invalid login:
send(%0)
echo "no such user!"
client "register/login a new user":
proc registerUser*(u: User)
var userId: int
userId = recv()
if userId == 0: setError(username, "invalid login!")
serve "orminchat", dispatch
@@ -0,0 +1,2 @@
--define: ssl
--path:"$projectDir/../../../ormin"
@@ -0,0 +1,2 @@
--define: ssl
--path:"$projectDir/../../../ormin"
+160
View File
@@ -0,0 +1,160 @@
import ../../ormin, json
importModel(DbBackend.sqlite, "forum_model")
var db {.global.} = open("stuff", "", "", "")
#var db: DbConn
#proc getPrepStmt(idx: int): PStmt
#var gPrepStmts: array[N, cstring]
type inetType = string
const
id = 90
ip = "moo"
answer = "dunno"
pw = "mypw"
email = "some@body.com"
salt = "pepper"
name = "me"
limit = 10
offset = 5
let threads = query:
select thread(id, name, views, modified)
where id in (select post(thread) where author in
(select person(id) where status notin ("Spammer") or id == ?id))
orderby desc(modified)
limit ?limit
offset ?offset
let thisThread = tryQuery:
select thread(id)
where id == ?id
createIter allThreadIds:
select thread(id)
where id == ?id
query:
delete antibot
where ip == ?ip
query:
insert antibot(?ip, ?answer)
let something = query:
select antibot(answer & answer, (if ip == "hi": 0 else: 1))
where ip == ?ip and answer =~ "%things%"
orderby desc(ip)
limit 1
let myNewPersonId: int = query:
insert person(?name, password = ?pw, ?email, ?salt, status = !!"'EmailUnconfirmed'",
lastOnline = !!"DATETIME('now')")
returning id
query:
delete session
where ip == ?ip and password == ?pw
query:
update session(lastModified = !!"DATETIME('now')")
where ip == ?ip and password == ?pw
let myj = %*{"pw": "stuff here"}
let userId1 = query:
select session(userId)
where ip == ?ip and password == %myj["pw"]
let (name9, email9, status, ban) = query:
select person(name, email, status, ban)
where id == ?id
limit 1
let (idg, nameg, pwg, emailg, creationg, saltg, statusg, lastOnlineg, bang) = query:
select person(_)
where id == ?id
limit 1
query:
update person(lastOnline = !!"DATETIME('now')")
where id == ?id
query:
update thread(views = views + 1, modified = !!"DATETIME('now')")
where id == ?id
query:
delete thread
where id notin (select post(thread))
let (author, creation) = query:
select post(author)
join person(creation)
limit 1
let (authorB, creationB) = query:
select post(author)
join person(creation) on author == id
limit 1
let allPosts = query:
select post(count(_) as cnt)
where cnt > 0
produce json
limit 1
createProc getAllThreadIds:
select thread(id)
where id == ?id
produce json
let totalThreads = query:
select thread(count(_))
where id in (select post(thread) where author == ?id and id in (
select post(min(id)) groupby thread))
limit 1
#query:
# update thread(modified = (select post(creation) where post.thread == ?thread
# orderby creation desc limit 1 ))
#[
# Check if post is the first post of the thread.
let rows = db.getAllRows(sql("select id, thread, creation from post " &
"where thread = ? order by creation asc"), $c.threadId)
proc rateLimitCheck(c: var TForumData): bool =
sql("SELECT count(*) FROM post where author = ? and " &
"(strftime('%s', 'now') - strftime('%s', creation)) < 40")
sql("SELECT count(*) FROM post where author = ? and " &
"(strftime('%s', 'now') - strftime('%s', creation)) < 90")
sql("SELECT count(*) FROM post where author = ? and " &
"(strftime('%s', 'now') - strftime('%s', creation)) < 300")
sql "insert into thread(name, views, modified) values (?, 0, DATETIME('now'))"
sql "select id, name, password, email, salt, status, ban from person where name = ?"
sql "insert into session (ip, password, userid) values (?, ?, ?)",
sql"select password, salt, strftime('%s', lastOnline) from person where name = ?"
sql("delete from post where author = (select id from person where name = ?)")
sql("update person set status = ?, ban = ? where name = ?")
sql("update person set password = ?, salt = ? where name = ?")
sql"select count(*) from person"
sql"select count(*) from post"
sql"select count(*) from thread"
sql"select id, name, strftime('%s', lastOnline), strftime('%s', creation) from person"
sql"select count(*) from post where thread = ?"
sql"select count(*) from post p, person u where u.id = p.author and p.thread = ?"
sql"select name from thread where id = ?"
sql"select id from person where name = ?"
sql"select count(*) from post where author = ?"
sql("select count(*) from thread where id in (select thread from post where" &
" author = ? and post.id in (select min(id) from post group by thread))")
sql"""select strftime('%s', lastOnline), email, ban, status
from person where id = ?"""
]#
@@ -0,0 +1,55 @@
create table if not exists thread(
id integer primary key,
name varchar(100) not null,
views integer not null,
modified timestamp not null default (DATETIME('now'))
);
create unique index if not exists ThreadNameIx on thread (name);
create table if not exists person(
id integer primary key,
name varchar(20) not null,
password varchar(32) not null,
email varchar(30) not null,
creation timestamp not null default (DATETIME('now')),
salt varchar(128) not null,
status varchar(30) not null,
lastOnline timestamp not null default (DATETIME('now')),
ban varchar(128) not null default ''
);
create unique index if not exists UserNameIx on person (name);
create table if not exists post(
id integer primary key,
author integer not null,
ip inet not null,
header varchar(100) not null,
content varchar(1000) not null,
thread integer not null,
creation timestamp not null default (DATETIME('now')),
foreign key (thread) references thread(id),
foreign key (author) references person(id)
);
create table if not exists session(
id integer primary key,
ip inet not null,
password varchar(32) not null,
userid integer not null,
lastModified timestamp not null default (DATETIME('now')),
foreign key (userid) references person(id)
);
create table if not exists antibot(
id integer primary key,
ip inet not null,
answer varchar(30) not null,
created timestamp not null default (DATETIME('now'))
);
create index PersonStatusIdx on person(status);
create index PostByAuthorIdx on post(thread, author);
@@ -0,0 +1,37 @@
import ../../ormin, ../../ormin/serverws, json
importModel(DbBackend.sqlite, "forum_model")
var db {.global.} = open("stuff", "", "", "")
protocol "forumclient.nim":
common:
when defined(js):
type kstring = cstring
else:
type kstring = string
type
inetType = kstring
varcharType = kstring
timestampType = kstring
server:
query:
delete antibot
where ip == %arg
client:
proc deleteAntibot(ip: string)
server:
query:
update session(lastModified = !!"DATETIME('now')")
where ip == %arg["ip"] and password == %arg["pw"]
client:
proc updateSession(arg: Session)
server:
let allSessions = query:
select session(_)
send(allSessions)
client:
type Session = ref object
var gSessions: seq[Session]
gSessions = recv()
proc getAllSessions()
@@ -0,0 +1,117 @@
body {
background-color: #f1f9ea;
margin: 0;
font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
}
div#main {
width: 80%;
margin-left: auto;
margin-right: auto;
}
div#user {
background-color: #66ac32;
width: 100%;
color: #c7f0aa;
padding: 5pt;
}
div#user > h1 {
color: #ffffff;
}
h1 {
margin: 0;
display: inline;
padding-left: 10pt;
padding-right: 10pt;
}
div#user > form {
float: right;
margin-right: 10pt;
}
div#user > form > input[type="submit"] {
border: 0px none;
padding: 5pt;
font-size: 108%;
color: #ffffff;
background-color: #515d47;
border-radius: 5px;
cursor: pointer;
}
div#user > form > input[type="submit"]:hover {
background-color: #538c29;
}
div#messages {
background-color: #a2dc78;
width: 90%;
margin-left: auto;
margin-right: auto;
color: #1a1a1a;
}
div#messages > div {
border-left: 1px solid #869979;
border-right: 1px solid #869979;
border-bottom: 1px solid #869979;
padding: 5pt;
}
div#messages > div > a, div#messages > div > span {
color: #475340;
}
div#messages > div > a:hover {
text-decoration: none;
color: #c13746;
}
h3 {
margin-bottom: 0;
font-weight: normal;
}
div#login {
width: 200px;
margin-left: auto;
margin-right: auto;
margin-top: 20%;
font-size: 130%;
}
div#login span.small {
display: block;
font-size: 56%;
}
div#newMessage {
background-color: #538c29;
width: 90%;
margin-left: auto;
margin-right: auto;
color: #ffffff;
padding: 5pt;
}
div#newMessage span {
padding-right: 5pt;
}
div#newMessage form {
display: inline;
}
div#newMessage > form > input[type="text"] {
width: 80%;
}
div#newMessage > form > input[type="submit"] {
font-size: 80%;
}
@@ -0,0 +1,3 @@
# Tweeter example
Modified from [tweeter example](https://github.com/dom96/nim-in-action-code/tree/master/Chapter7/Tweeter) in the "nim in action" book, replace the database layer with ormin.
@@ -0,0 +1,2 @@
--define: ssl
--path:"$projectDir/../../../../ormin"
@@ -0,0 +1,11 @@
import db_sqlite, os, strutils
let db {.global.} = open("tweeter.db", "", "", "")
let sqlFile = readFile(currentSourcePath.parentDir() / "tweeter_model.sql")
for t in sqlFile.split(';'):
if t.strip() != "":
db.exec(sql(t))
echo("Database created successfully!")
db.close()
@@ -0,0 +1,51 @@
import times, strutils, strformat, sequtils
from db_connector/db_sqlite import instantRows, `[]`
import ../../../ormin
import model
importModel(DbBackend.sqlite, "tweeter_model")
var db {.global.} = open("tweeter.db", "", "", "")
proc findUser*(username: string, user: var User): bool =
let res = query:
select user(username)
where username == ?username
echo res
if res.len == 0: return false
else: user.username = res[0]
let following = query:
select following(followed_user)
where follower == ?username
user.following = following.filterIt(it.len != 0)
return true
proc create*(user: User) =
query:
insert user(username = ?user.username)
proc post*(message: Message) =
if message.msg.len > 140:
raise newException(ValueError, "Message has to be less than 140 characters.")
query:
insert message(username = ?message.username,
time = ?message.time,
msg = ?message.msg)
proc follow*(follower, user: User) =
query:
insert following(follower = ?follower.username, followed_user = ?user.username)
proc findMessage*(usernames: openArray[string], limit = 10): seq[Message] =
result = @[]
if usernames.len == 0: return
var whereClause = "WHERE "
for i in 0 ..< usernames.len:
whereClause.add("trim(username) = ?")
if i < usernames.len - 1:
whereClause.add(" or ")
let s = &"SELECT username, time, msg FROM Message {whereClause} ORDER BY time DESC LIMIT {limit}"
for row in db.instantRows(sql(s), usernames):
result.add((username: row[0], time: row[1].parseInt, msg: row[2]))
@@ -0,0 +1,11 @@
type
User* = tuple[
username: string,
following: seq[string]
]
Message* = tuple[
username: string,
time: int,
msg: string
]
@@ -0,0 +1,59 @@
import asyncdispatch, times
import jester
import database, model, views/[user, general]
proc userLogin(request: Request, user: var User): bool =
if request.cookies.hasKey("username"):
let username = request.cookies["username"]
if not findUser(username, user):
user = (username: username, following: @[])
create(user)
return true
else:
return false
routes:
get "/":
var user: User
if userLogin(request, user):
let messages = findMessage(user.following & user.username)
resp renderMain(renderTimeline(user.username, messages))
else:
resp renderMain(renderLogin())
get "/@name":
cond '.' notin @"name"
var user: User
if not findUser(@"name", user):
halt "User not found"
let messages = findMessage([user.username])
var currentUser: User
if userLogin(request, currentUser):
resp renderMain(renderUser(user, currentUser) & renderMessages(messages))
else:
resp renderMain(renderUser(user) & renderMessages(messages))
post "/follow":
var follower, target: User
if not findUser(@"follower", follower):
halt "Follower not found"
if not findUser(@"target", target):
halt "Follow target not found"
follow(follower, target)
redirect uri("/" & @"target")
post "/login":
setCookie("username", @"username", getTime().utc() + 2.hours)
redirect "/"
post "/createMessage":
let message = (
username: @"username",
time: getTime().toUnix().int,
msg: @"message"
)
post(message)
redirect "/"
runForever()
@@ -0,0 +1,24 @@
# Generated by ormin_importer. DO NOT EDIT.
type
Attr = object
name: string
tabIndex: int
typ: DbTypekind
key: int # 0 nothing special,
# +1 -- primary key
# -N -- references attribute N
const tableNames = [
"user",
"following",
"message"
]
const attributes = [
Attr(name: "username", tabIndex: 0, typ: dbVarchar, key: 1),
Attr(name: "follower", tabIndex: 1, typ: dbVarchar, key: -1),
Attr(name: "followed_user", tabIndex: 1, typ: dbVarchar, key: -1),
Attr(name: "username", tabIndex: 2, typ: dbVarchar, key: -1),
Attr(name: "time", tabIndex: 2, typ: dbInt, key: 0),
Attr(name: "msg", tabIndex: 2, typ: dbVarchar, key: 0)
]
@@ -0,0 +1,18 @@
CREATE TABLE IF NOT EXISTS User(
username text PRIMARY KEY
);
CREATE TABLE IF NOT EXISTS Following(
follower text,
followed_user text,
PRIMARY KEY (follower, followed_user),
FOREIGN KEY (follower) REFERENCES User(username),
FOREIGN KEY (followed_user) REFERENCES User(username)
);
CREATE TABLE IF NOT EXISTS Message(
username text,
time integer,
msg text NOT NULL,
FOREIGN KEY (username) REFERENCES User(username)
);
@@ -0,0 +1,51 @@
#? stdtmpl(subsChar = '$', metaChar = '#')
#import xmltree
#import ../model
#import user
#
#proc `$!`(text: string): string = escape(text)
#end proc
#
#proc renderMain*(body: string): string =
# result = ""
<!DOCTYPE html>
<html>
<head>
<title>Tweeter written in Nim</title>
<link rel="stylesheet" type="text/css" href="style.css">
</head>
<body>
${body}
</body>
</html>
#end proc
#
#proc renderLogin*(): string =
# result = ""
<div id="login">
<span>Login</span>
<span class="small">Please type in your username...</span>
<form action="login" method="post">
<input type="text" name="username">
<input type="submit" value="Login">
</form>
</div>
#end proc
#
#proc renderTimeline*(username: string, messages: openArray[Message]): string =
# result = ""
<div id="user">
<h1>${$!username} timeline</h1>
</div>
<div id="newMessage">
<span>New message</span>
<form action="createMessage" method="post">
<input type="text" name="message">
<input type="hidden" name="username" value="${$!username}">
<input type="submit" value="Tweet">
</form>
</div>
${renderMessages(messages)}
#end proc
@@ -0,0 +1,41 @@
#? stdtmpl(subsChar = '$', metaChar = '#', toString = "xmltree.escape")
#import xmltree
#import times
#import "../model"
#
#proc renderUser*(user: User): string =
# result = ""
<div id="user">
<h1>${user.username}</h1>
<span>Following: ${$user.following.len}</span>
</div>
#end proc
#
#proc renderUser*(user, currentUser: User): string =
# result = ""
<div id="user">
<h1>${user.username}</h1>
<span>Following: ${$user.following.len}</span>
#if user.username notin currentUser.following and user.username != currentUser.username:
<form action="follow" method="post">
<input type="hidden" name="follower" value="${currentUser.username}">
<input type="hidden" name="target" value="${user.username}">
<input type="submit" value="Follow">
</form>
#end if
</div>
#
#end proc
#
#proc renderMessages*(messages: openArray[Message]): string =
# result = ""
<div id="messages">
#for message in messages:
<div>
<a href="/${message.username}">${message.username}</a>
<span>${message.time.fromUnix().format("HH:mm MMMM d',' yyyy")}</span>
<h3>${message.msg}</h3>
</div>
#end for
</div>
#end proc
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2018 Andreas Rumpf
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+49
View File
@@ -0,0 +1,49 @@
type
DbBackend* {.pure.} = enum
postgre, sqlite, mysql, baradb
import os
from ormin/importer_core import generateModelCode, ImportTarget
template importModel*(backend: DbBackend, filename: string, includeStatic: static[bool] = false) {.dirty.} =
## imports a model from an SQL file.
bind fileExists, parentDir, `/`
bind generateModelCode, ImportTarget
const file = static:
let path = parentDir(instantiationInfo(-1, true)[0])
path / filename & ".sql"
const importedFile = static:
if not includeStatic:
let res =
if fileExists("./tools/ormin_importer"):
gorgeEx("./tools/ormin_importer " & file)
else:
# run ormin_importer from the PATH
gorgeEx("ormin_importer " & file)
if res.exitCode != 0:
raise newException(Exception, "Failed to generate model: " & res.output)
file
{.warning: "Imported SQL Model: " & importedFile.}
const dbBackend = backend
import db_connector/db_common
when includeStatic:
import ormin/db_utils
when dbBackend == DbBackend.sqlite:
import ormin/ormin_sqlite
elif dbBackend == DbBackend.postgre:
import ormin/ormin_postgre
elif dbBackend == DbBackend.mysql:
import ormin/ormin_mysql
elif dbBackend == DbBackend.baradb:
import ormin/ormin_baradb
else:
{.error: "unknown database backend".}
when includeStatic:
generateModelCode(staticRead(file), file, ImportTarget(ord(backend)), true)
else:
include filename
include "ormin/queries"
+23
View File
@@ -0,0 +1,23 @@
# Package
version = "0.9.0"
author = "Araq"
description = "Prepared SQL statement generator. A lightweight ORM."
license = "MIT"
bin = @["tools/ormin_importer"]
skipDirs = @["examples"]
installExt = @["nim"]
# Dependencies
requires "nim >= 2.0.0"
requires "db_connector >= 0.1.0"
feature "examples":
requires "websocket >= 0.2.2"
requires "karax"
requires "jester"
import std/os
when fileExists("config.nims"):
include "config.nims"
+43
View File
@@ -0,0 +1,43 @@
import std/strutils
import db_connector/db_common
proc dbTypFromName*(name: string): DbTypeKind =
var k = dbUnknown
case name.toLowerAscii
of "int", "integer", "int8", "smallint", "int16",
"longint", "int32", "int64", "tinyint", "hugeint": k = dbInt
of "uint", "uint8", "uint16", "uint32", "uint64": k = dbUInt
of "serial": k = dbSerial
of "bit": k = dbBit
of "bool", "boolean": k = dbBool
of "blob": k = dbBlob
of "fixedchar": k = dbFixedChar
of "varchar", "text", "string": k = dbVarchar
of "json": k = dbJson
of "xml": k = dbXml
of "decimal": k = dbDecimal
of "float", "double", "longdouble", "real": k = dbFloat
of "date", "day": k = dbDate
of "time": k = dbTime
of "datetime": k = dbDateTime
of "timestamp", "timestamptz": k = dbTimestamp
of "timeinterval": k = dbTimeInterval
of "set": k = dbSet
of "array": k = dbArray
of "composite": k = dbComposite
of "url", "uri": k = dbUrl
of "uuid": k = dbUuid
of "inet", "ip", "tcpip": k = dbInet
of "mac", "macaddress": k = dbMacAddress
of "geometry": k = dbGeometry
of "point": k = dbPoint
of "line": k = dbLine
of "lseg": k = dbLseg
of "box": k = dbBox
of "path": k = dbPath
of "polygon": k = dbPolygon
of "circle": k = dbCircle
else: discard
return k
+179
View File
@@ -0,0 +1,179 @@
import std/[os, paths]
import db_connector/db_common, strutils, strformat
import db_connector/db_postgres as db_postgres
import db_connector/db_sqlite as db_sqlite
import parsesql_tmp # import std/parsesql
export paths
type
DbConn = db_postgres.DbConn | db_sqlite.DbConn
DbId* = distinct string
DbSql* = distinct string
proc `$`*(dbId: DbId): string {.borrow.}
proc `$`*(dbSql: DbSql): string {.borrow.}
template staticLoad*(filename: string): DbSql =
bind isAbsolute, parentDir, `/`, instantiationInfo
block:
const schemaPath {.gensym.} = static:
if isAbsolute(filename):
filename
else:
parentDir(instantiationInfo(-1, true)[0]) / filename
const schemaSql {.gensym.} = staticRead(schemaPath)
static:
try:
discard parseSql(schemaSql)
except SqlParseError as e:
doAssert false, "Invalid SQL in " & schemaPath & ": " & e.msg
DbSql(schemaSql)
proc parseQualifiedIdentifier(name: string): seq[(string, bool)] =
var current = ""
var quoteChar = '\0'
var partWasQuoted = false
var i = 0
while i < name.len:
let c = name[i]
if quoteChar == '\0':
case c
of '.':
result.add((current, partWasQuoted))
current = ""
partWasQuoted = false
of '"', '\'', '`':
if current.len == 0:
quoteChar = c
partWasQuoted = true
else:
current.add(c)
else:
current.add(c)
else:
if c == quoteChar:
if i + 1 < name.len and name[i + 1] == quoteChar:
current.add(c)
inc(i)
else:
quoteChar = '\0'
else:
current.add(c)
inc(i)
result.add((current, partWasQuoted))
proc canUseBareIdentifier(part: string): bool =
if part.len == 0:
return false
if part[0] notin {'a'..'z', 'A'..'Z', '_'}:
return false
for c in part[1..^1]:
if c notin {'a'..'z', 'A'..'Z', '0'..'9', '_'}:
return false
true
proc quoteDbIdentifier*(name: string): DbId =
## Quote a database identifier for direct interpolation into SQL text.
## Dot-separated names are treated as qualified identifiers.
var partsSql: seq[string] = @[]
for (part, wasQuoted) in parseQualifiedIdentifier(name):
let normalizedPart = if wasQuoted: part else: part.toLowerAscii()
if wasQuoted or not canUseBareIdentifier(normalizedPart):
partsSql.add("\"" & normalizedPart.replace("\"", "\"\"") & "\"")
else:
partsSql.add(normalizedPart)
DbId(partsSql.join("."))
proc dropTableName(db: DbConn; tableName, lookupName: string) =
# SQL parameters bind values, not identifiers, so DROP TABLE needs the
# identifier rendered into the statement text with correct quoting.
when defined(sqlite) and defined(orminLegacySqliteDropNames):
db.exec(sql("drop table if exists " & lookupName))
else:
let tableIdentSql = quoteDbIdentifier(tableName)
when defined(postgre):
db.exec(sql("drop table if exists " & $tableIdentSql & " cascade"))
else:
db.exec(sql("drop table if exists " & $tableIdentSql))
iterator tableDefs(sql: DbSql): tuple[name, tableName, model: string] =
# Parse the entire SQL file and iterate statements via the SQL parser
var ast: SqlNode
try:
ast = parseSql($sql)
except SqlParseError as e:
echo "SQL Parse Error:\n", sql
raise e
if ast.len > 0:
# ast is a statement list; iterate each statement node
for i in 0 ..< ast.len:
let node = ast[i]
if node.kind in {nkCreateTable, nkCreateTableIfNotExists}:
yield (node[0].strVal.toLowerAscii(), $node[0], $node)
else:
# Fallback: ast might be a single statement (not a list)
let node = ast
if node.kind in {nkCreateTable, nkCreateTableIfNotExists}:
yield (node[0].strVal.toLowerAscii(), $node[0], $node)
iterator tablePairs*(sql: string): tuple[name, model: string] =
for name, _, model in tableDefs(DbSql(sql)):
yield (name, model)
iterator tablePairs*(sql: static[DbSql]): tuple[name, model: string] =
for name, _, model in tableDefs(sql):
yield (name, model)
iterator tablePairs*(sql: Path): tuple[name, model: string] =
let f = readFile($sql)
for n, m in tablePairs(f):
yield (n, m)
proc createTable*(db: DbConn; sqlFile: Path) =
for _, m in tablePairs(sqlFile):
db.exec(sql(m))
proc createTable*(db: DbConn; sqlFile: Path, name: string) =
for n, m in tablePairs(sqlFile):
if n == name:
db.exec(sql(m))
return
raiseAssert &"table: {name} not found in: {sqlFile}"
proc createTable*(db: DbConn; schemaSql: static[DbSql]) =
for _, m in tablePairs(schemaSql):
db.exec(sql(m))
proc createTable*(db: DbConn; schemaSql: static[DbSql], name: string) =
for n, m in tablePairs(schemaSql):
if n == name:
db.exec(sql(m))
return
raiseAssert &"table: {name} not found in static schema"
proc dropTable*(db: DbConn; sqlFile: Path) =
for lookupName, tableName, _ in tableDefs(DbSql(readFile($sqlFile))):
db.dropTableName(tableName, lookupName)
proc dropTable*(db: DbConn; sqlFile: Path, name: string) =
for lookupName, tableName, _ in tableDefs(DbSql(readFile($sqlFile))):
if lookupName == name:
db.dropTableName(tableName, lookupName)
return
raiseAssert &"table: {name} not found in: {sqlFile}"
proc dropTable*(db: DbConn; schemaSql: static[DbSql]) =
for lookupName, tableName, _ in tableDefs(schemaSql):
db.dropTableName(tableName, lookupName)
proc dropTable*(db: DbConn; schemaSql: static[DbSql], name: string) =
for lookupName, tableName, _ in tableDefs(schemaSql):
if lookupName == name:
db.dropTableName(tableName, lookupName)
return
raiseAssert &"table: {name} not found in static schema"
+71
View File
@@ -0,0 +1,71 @@
## This module implements a little helper macro to ease
## writing the dispatching logic for JSON based servers.
import macros
proc tbody(n: NimNode): NimNode =
# transforms::
# f fields(a, "b")
# into::
# f a = %args["a"], b = %args["b"]
case n.kind
of nnkCallKinds:
result = copyNimNode(n)
result.add tbody(n[0])
for i in 1..<n.len:
let it = n[i]
if it.kind == nnkCall and $it[0] == "fields":
for j in 1..<it.len:
let field = it[j]
let s = if field.kind in {nnkStrLit..nnkTripleStrLit}: field.strVal
else: $field
let arg = newTree(nnkPrefix, ident"%",
newTree(nnkBracketExpr, ident"arg", newLit(s)))
result.add newTree(nnkExprEqExpr, ident(s), arg)
else:
result.add tbody(it)
else:
result = copyNimNode(n)
for x in n: result.add tbody(x)
macro createDispatcher*(name, n: untyped): untyped =
expectKind n, nnkStmtList
result = newStmtList()
let disp = newTree(nnkCaseStmt, ident"cmd")
template cmdProc(name, body) {.dirty.} =
proc name(arg: JsonNode): JsonNode = body
template callCmd(name) {.dirty.} =
result = name(arg)
for x in n:
if x.kind == nnkCall and x.len == 2 and
x[0].kind == nnkIdent and x[1].kind == nnkStmtList:
result.add getAst(cmdProc(x[0], tbody x[1]))
disp.add newTree(nnkOfBranch, newLit($x[0]),
newStmtList(getAst(callCmd(x[0]))))
else:
# do not touch:
result.add n
disp.add newTree(nnkElse, newStmtList(newTree(nnkDiscardStmt, newEmptyNode())))
template dispatchProc(name, body) {.dirty.} =
proc dispatch(inp: JsonNode): JsonNode =
let arg = inp["arg"]
let cmd = inp["cmd"].getStr("")
body
result.add getAst(dispatchProc(name, disp))
when defined(debugDispatcherDsl):
echo repr result
when isMainModule:
import json, db_sqlite
createDispatcher(dispatch):
insertCustomer:
echo "insert customer", fields(a, b, c)
result = arg
selectCustomers:
echo "select customer"
result = arg
echo dispatch(nil, %*{"cmd": "insertCustomer", "arg": [1, 2, 3]})
+178
View File
@@ -0,0 +1,178 @@
import std/[macros, strutils, tables]
import db_connector/db_common
import ./db_types
import ./parsesql_tmp
const
FileHeader* = """
# Generated by ormin_importer. DO NOT EDIT.
type
Attr = object
name: string
tabIndex: int
typ: DbTypekind
key: int # 0 nothing special,
# +1 -- primary key
# -N -- references attribute N
"""
type
DbColumn* = object
name*: string
tableName*: string
typ*: DbType
primaryKey*: bool
refs*: (string, string)
DbColumns* = seq[DbColumn]
KnownTables* = OrderedTable[string, DbColumns]
ImportTarget* = enum
postgre, sqlite, mysql, baradb
proc hasAttribute(colDesc: SqlNode; k: set[SqlNodeKind]): bool =
for i in 2 ..< colDesc.len:
if colDesc[i].kind in k:
return true
proc hasRefs(colDesc: SqlNode): (string, string) =
for i in 2 ..< colDesc.len:
let c = colDesc[i]
if c.kind == nkReferences:
if c[0].kind == nkCall:
return (c[0][0].strVal, c[0][1].strVal)
elif c[0].kind == nkIdent:
return ($c[0], "id")
("", "")
proc getType(n: SqlNode): DbType =
var it = n
if it.kind == nkCall:
it = it[0]
if it.kind == nkEnumDef:
result.kind = dbEnum
result.validValues = @[]
for i in 0 ..< it.len:
assert it[i].kind == nkStringLit
result.validValues.add it[i].strVal
elif it.kind in {nkIdent, nkStringLit}:
result.kind = dbTypFromName(it.strVal)
result.name = it.strVal
proc collectTables*(n: SqlNode; t: var KnownTables) =
if n.isNil:
return
case n.kind
of nkCreateTable, nkCreateTableIfNotExists:
let tableName = n[0].strVal
var cols: DbColumns = @[]
for i in 1 ..< n.len:
let it = n[i]
if it.kind == nkColumnDef:
var typ = getType(it[1])
if hasAttribute(it, {nkNotNull}):
typ.notNull = true
cols.add DbColumn(
name: it[0].strVal,
tableName: tableName,
typ: typ,
primaryKey: hasAttribute(it, {nkPrimaryKey}),
refs: hasRefs(it)
)
for i in 1 ..< n.len:
let it = n[i]
if it.kind == nkForeignKey:
var refNode: SqlNode = nil
var localCols: seq[string] = @[]
for j in 0 ..< it.len:
let child = it[j]
if child.kind == nkReferences:
refNode = child
elif child.kind == nkIdent:
localCols.add(child.strVal)
if refNode.isNil:
continue
var r = refNode[0]
var refTable = ""
var refCols: seq[string] = @[]
if r.kind == nkColumnReference or r.kind == nkCall:
refTable = r[0].strVal
for k in 1 ..< r.len:
refCols.add(r[k].strVal)
let pairCount = min(localCols.len, refCols.len)
for k in 0 ..< pairCount:
let localName = localCols[k]
let targetCol = refCols[k]
for c in mitems(cols):
if cmpIgnoreCase(c.name, localName) == 0:
c.refs = (refTable, targetCol)
break
t[tableName] = cols
else:
for i in 0 ..< n.len:
collectTables(n[i], t)
proc attrToKey(a: DbColumn; t: KnownTables): int =
if a.primaryKey:
return 1
if a.refs[0].len > 0:
var i = 0
for k, v in pairs(t):
for b in v:
if cmpIgnoreCase(k, a.refs[0]) == 0 and cmpIgnoreCase(b.name, a.refs[1]) == 0:
return -i - 1
inc i
0
proc renderModelCode(schemaSql, schemaPath: string; target: ImportTarget; includeStatic = false): string =
discard target
let sql = parseSql(schemaSql, schemaPath)
var knownTables = initOrderedTable[string, DbColumns]()
collectTables(sql, knownTables)
result.add FileHeader
result.add "const tableNames = ["
var i = 0
for k in keys(knownTables):
if i > 0:
result.add ",\n "
else:
result.add "\n "
result.add escape(k.toLowerAscii)
inc i
result.add "\n]\n"
i = 0
var j = 0
result.add "\nconst attributes = ["
for _, v in mpairs(knownTables):
for a in v:
if j > 0:
result.add ",\n "
else:
result.add "\n "
result.add "Attr(name: "
result.add escape(a.name.toLowerAscii)
result.add ", tabIndex: "
result.add $i
result.add ", typ: "
result.add $a.typ.kind
result.add ", key: "
result.add $attrToKey(a, knownTables)
result.add ")"
inc j
inc i
result.add "\n]\n"
if includeStatic:
result.add "\nconst sqlSchema* = staticLoad("
result.add escape(schemaPath)
result.add ")\n"
proc generateModelCode*(schemaSql, schemaPath: string; target: ImportTarget; includeStatic = false): string =
renderModelCode(schemaSql, schemaPath, target, includeStatic)
macro generateModelCode*(schemaSql: static[string], schemaPath: static[string],
target: static[ImportTarget], includeStatic: static[bool] = false): untyped =
parseStmt(renderModelCode(schemaSql, schemaPath, target, includeStatic))
+231
View File
@@ -0,0 +1,231 @@
import std/[strutils, json, times, parseutils]
import db_connector/db_common
import query_hooks
export db_common
import baradb/client
export client.WireValue, client.FieldKind
type
DbConn* = SyncClient
PStmt = object
sql: string
varcharType* = string
intType* = int
floatType* = float
boolType* = bool
timestampType* = DateTime
serialType* = int
jsonType* = JsonNode
var jsonTimeFormat* = "yyyy-MM-dd HH:mm:ss"
proc dbError*(db: DbConn) {.noreturn.} =
var e: ref DbError
new(e)
e.msg = "BaraDB query failed"
raise e
proc prepareStmt*(db: DbConn; q: string): PStmt =
when defined(debugOrminTrace):
echo "[[Ormin Executing]]: ", q
result.sql = q
template startBindings*(s: PStmt; n: int) {.dirty.} =
var pparams: seq[WireValue] = newSeq[WireValue](n)
template bindParam*(db: DbConn; s: PStmt; idx: int; x: untyped; t: untyped) =
when t is DateTime:
let xx = x.format("yyyy-MM-dd HH:mm:ss")
pparams[idx-1] = WireValue(kind: fkString, strVal: $xx)
elif t is int or t is int64:
pparams[idx-1] = WireValue(kind: fkInt64, int64Val: int64(x))
elif t is float or t is float64:
pparams[idx-1] = WireValue(kind: fkFloat64, float64Val: float64(x))
elif t is bool:
pparams[idx-1] = WireValue(kind: fkBool, boolVal: bool(x))
elif t is string:
pparams[idx-1] = WireValue(kind: fkString, strVal: string(x))
elif t is JsonNode:
pparams[idx-1] = WireValue(kind: fkJson, jsonVal: $x)
else:
pparams[idx-1] = WireValue(kind: fkString, strVal: $x)
template bindNullParam*(db: DbConn; s: PStmt; idx: int) =
pparams[idx-1] = WireValue(kind: fkNull)
template bindParamJson*(db: DbConn; s: PStmt; idx: int; xx: JsonNode;
t: typedesc) =
let x = xx
if x.kind == JNull:
pparams[idx-1] = WireValue(kind: fkNull)
else:
bindFromJson(db, s, idx, x, t)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc) =
{.error: "invalid type for JSON object".}
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[string]) =
doAssert x.kind == JString
pparams[idx-1] = WireValue(kind: fkString, strVal: x.str)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[int|int64]) =
doAssert x.kind == JInt
pparams[idx-1] = WireValue(kind: fkInt64, int64Val: x.num)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[float64]) =
doAssert x.kind == JFloat
pparams[idx-1] = WireValue(kind: fkFloat64, float64Val: x.fnum)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[bool]) =
doAssert x.kind == JBool
pparams[idx-1] = WireValue(kind: fkBool, boolVal: x.bval)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[DateTime]) =
doAssert x.kind == JString
pparams[idx-1] = WireValue(kind: fkString, strVal: x.str)
template startQuery*(db: DbConn; s: PStmt) =
when declared(pparams):
var queryResult {.inject.} = db.query(s.sql, pparams)
else:
var queryResult {.inject.} = db.query(s.sql)
var queryI {.inject.} = -1
var queryLen {.inject.} = queryResult.rowCount
template stopQuery*(db: DbConn; s: PStmt) =
discard
template stepQuery*(db: DbConn; s: PStmt; returnsData: bool): bool =
inc queryI
queryI < queryLen
template getLastId*(db: DbConn; s: PStmt): int =
0
template getAffectedRows*(db: DbConn; s: PStmt): int =
queryResult.affectedRows
proc close*(db: DbConn) =
client.close(db)
proc open*(connection, user, password, database: string): DbConn =
let colonPos = connection.find(':')
let host = if colonPos < 0: connection else: connection[0..<colonPos]
let portStr = if colonPos < 0: "9472" else: connection[colonPos+1..^1]
let port = parseInt(portStr)
let cfg = ClientConfig(
host: host,
port: port,
database: database,
username: user,
password: password
)
result = newSyncClient(cfg)
result.connect()
# --- Result binding helpers ---
template currentValue(s: PStmt; idx: int): string =
queryResult.rows[queryI][idx-1]
template columnIsNull*(db: DbConn; s: PStmt; idx: int): bool =
currentValue(s, idx).len == 0
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: int;
t: typedesc; name: string) =
dest = int(parseBiggestInt(currentValue(s, idx)))
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: int64;
t: typedesc; name: string) =
dest = parseBiggestInt(currentValue(s, idx))
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: bool;
t: typedesc; name: string) =
let v = currentValue(s, idx)
dest = v == "true" or v == "1" or v == "t"
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: var string;
t: typedesc; name: string) =
dest = currentValue(s, idx)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: float64;
t: typedesc; name: string) =
dest = parseFloat(currentValue(s, idx))
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: var DateTime;
t: typedesc; name: string) =
let src = currentValue(s, idx)
if src.len > 0:
dest = parse(src, "yyyy-MM-dd HH:mm:ss")
else:
dest = initDateTime(1, 1, 1, 0, 0, 0, utc())
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: JsonNode;
t: typedesc; name: string) =
dest = parseJson(currentValue(s, idx))
template bindResult*[T](db: DbConn; s: PStmt; idx: int; dest: var DbValue[T];
t: typedesc; name: string) =
let v = currentValue(s, idx)
if v.len == 0:
dest.isNull = true
else:
dest.isNull = false
when T is string:
dest.value = v
else:
bindResult(db, s, idx, dest.value, t, name)
template createJObject*(): untyped = newJObject()
template createJArray*(): untyped = newJArray()
template bindResultJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc; name: string) =
let x = obj
doAssert x.kind == JObject
let v = currentValue(s, idx)
if v.len == 0:
x[name] = newJNull()
else:
bindToJson(db, s, idx, x, t, name)
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc; name: string) =
{.error: "invalid type for JSON object".}
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[string]; name: string) =
obj[name] = newJString(currentValue(s, idx))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[int|int64]; name: string) =
var v: int64
discard parseBiggestInt(currentValue(s, idx), v)
obj[name] = newJInt(v)
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[float64]; name: string) =
obj[name] = newJFloat(parseFloat(currentValue(s, idx)))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[bool]; name: string) =
let v = currentValue(s, idx)
obj[name] = newJBool(v == "true" or v == "1" or v == "t")
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[DateTime]; name: string) =
var dt: DateTime
bindResult(db, s, idx, dt, t, name)
obj[name] = newJString(format(dt, jsonTimeFormat))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[JsonNode]; name: string) =
obj[name] = parseJson(currentValue(s, idx))
+274
View File
@@ -0,0 +1,274 @@
import strutils, db_connector/postgres, json, times
import db_connector/db_common
import query_hooks
export db_common
type
DbConn* = PPGconn ## encapsulates a database connection
PStmt = string ## a identifier for the prepared queries
varcharType* = string
intType* = int
floatType* = float
boolType* = bool
timestampType* = Datetime
serialType* = int
jsonType* = JsonNode
var jsonTimeFormat* = "yyyy-MM-dd HH:mm:ss"
proc dbError*(db: DbConn) {.noreturn.} =
## raises a DbError exception.
var e: ref DbError
new(e)
e.msg = $pqErrorMessage(db)
raise e
proc c_strtod(buf: cstring, endptr: ptr cstring = nil): float64 {.
importc: "strtod", header: "<stdlib.h>", noSideEffect.}
proc c_strtol(buf: cstring, endptr: ptr cstring = nil, base: cint = 10): int {.
importc: "strtol", header: "<stdlib.h>", noSideEffect.}
var sid {.compileTime.}: int
proc prepareStmt*(db: DbConn; q: string): PStmt =
when defined(debugOrminTrace):
echo "[[Ormin Executing]]: ", q
inc sid
result = "ormin" & $sid
var res = pqprepare(db, result, q, 0, nil)
if pqResultStatus(res) != PGRES_COMMAND_OK: dbError(db)
template startBindings*(s: PStmt; n: int) {.dirty.} =
# pparams is a duplicated array to keep the Nim string alive
# for the duration of the query. This is safer than relying
# on the conservative stack marking:
var pparams: array[n, string]
var parr: array[n, cstring]
template bindParam*(db: DbConn; s: PStmt; idx: int; x: untyped; t: untyped) =
# when not (x is t):
# {.error: "type mismatch for query argument at position " & $idx.}
when t is DateTime:
let xx = x.format("yyyy-MM-dd HH:mm:ss\'.\'ffffffzzzz")
pparams[idx-1] = $xx
else:
pparams[idx-1] = $x
parr[idx-1] = cstring(pparams[idx-1])
template bindParamUnchecked(db: DbConn; s: PStmt; idx: int; x: untyped; t: untyped) =
pparams[idx-1] = $x
parr[idx-1] = cstring(pparams[idx-1])
template bindNullParam*(db: DbConn; s: PStmt; idx: int) =
parr[idx-1] = cstring(nil)
template bindParamJson*(db: DbConn; s: PStmt; idx: int; xx: JsonNode;
t: typedesc) =
let x = xx
if x.kind == JNull:
# a NULL entry is not reflected in the 'pparams' array:
parr[idx-1] = cstring(nil)
else:
bindFromJson(db, s, idx, x, t)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc) =
{.error: "invalid type for JSON object".}
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[string]) =
doAssert x.kind == JString
let xs = x.str
bindParamUnchecked(db, s, idx, xs, t)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[int|int64]) =
doAssert x.kind == JInt
let xi = x.num
bindParamUnchecked(db, s, idx, xi, t)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[float64]) =
doAssert x.kind == JFloat
let xf = x.fnum
bindParamUnchecked(db, s, idx, xf, t)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[bool]) =
doAssert x.kind == JBool
let xb = x.bval
bindParamUnchecked(db, s, idx, xb, t)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[DateTime]) =
doAssert x.kind == JString
let dt = x.str
bindParamUnchecked(db, s, idx, dt, t)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: int;
t: typedesc; name: string) =
dest = c_strtol(pqgetvalue(queryResult, queryI, idx.cint))
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: int64;
t: typedesc; name: string) =
dest = c_strtol(pqgetvalue(queryResult, queryI, idx.cint))
# XXX This needs testing:
template isTrue(x): untyped = x[0] == 't'
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: bool;
t: typedesc; name: string) =
dest = isTrue(pqgetvalue(queryResult, queryI, idx.cint))
proc fillString(dest: var string; src: cstring; srcLen: int) {.inline.} =
dest = ""
var i = 0
while src[i] != '\0':
dest.add src[i]
inc i
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: var string;
t: typedesc; name: string) =
if pqgetisnull(queryResult, queryI, idx.cint) != 0:
when defined(nimNoNilSeqs):
setLen(dest, 0)
else:
dest = nil
else:
let src = pqgetvalue(queryResult, queryI, idx.cint)
let srcLen = int(pqgetlength(queryResult, queryI, idx.cint))
fillString(dest, src, srcLen)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: float64;
t: typedesc; name: string) =
dest = c_strtod(pqgetvalue(queryResult, queryI, idx.cint))
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: var DateTime;
t: typedesc; name: string) =
let
src = $pqgetvalue(queryResult, queryI, idx.cint)
i = src.find('.')
tzflag = src[src.len-3]
if i < 0:
if tzflag in {'+', '-'}:
dest = parse(src, "yyyy-MM-dd HH:mm:sszz")
else:
dest = parse(src, "yyyy-MM-dd HH:mm:ss")
else:
if tzflag in {'+', '-'}:
let itz = src.len - 3
let dtstr = src[0..<itz] & '0'.repeat(10-src.len+i) & src[src.len-3 .. ^1]
dest = parse(dtstr, "yyyy-MM-dd HH:mm:ss\'.\'ffffffzz")
else:
let dtstr = src & '0'.repeat(7-src.len+i)
dest = parse(dtstr, "yyyy-MM-dd HH:mm:ss\'.\'ffffff")
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: JsonNode;
t: typedesc; name: string) =
dest = parseJson($pqgetvalue(queryResult, queryI, idx.cint))
template bindResult*[T](db: DbConn; s: PStmt; idx: int; dest: var DbValue[T];
t: typedesc; name: string) =
if pqgetisnull(queryResult, queryI, idx.cint) != 0:
dest.isNull = true
else:
dest.isNull = false
when T is string:
let src = pqgetvalue(queryResult, queryI, idx.cint)
let srcLen = int(pqgetlength(queryResult, queryI, idx.cint))
fillString(dest.value, src, srcLen)
else:
bindResult(db, s, idx, dest.value, t, name)
template createJObject*(): untyped = newJObject()
template createJArray*(): untyped = newJArray()
template bindResultJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc; name: string) =
let x = obj
doAssert x.kind == JObject
if pqgetisnull(queryResult, queryI, idx.cint) != 0:
x[name] = newJNull()
else:
bindToJson(db, s, idx, x, t, name)
template columnIsNull*(db: DbConn; s: PStmt; idx: int): bool =
pqgetisnull(queryResult, queryI, idx.cint) != 0
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc; name: string) =
{.error: "invalid type for JSON object".}
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[string]; name: string) =
let src = pqgetvalue(queryResult, queryI, idx.cint)
obj[name] = newJString($src)
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[int|int64]; name: string) =
obj[name] = newJInt(c_strtol(pqgetvalue(queryResult, queryI, idx.cint)))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[float64]; name: string) =
obj[name] = newJFloat(c_strtod(pqgetvalue(queryResult, queryI, idx.cint)))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[bool]; name: string) =
obj[name] = newJBool(isTrue(pqgetvalue(queryResult, queryI, idx.cint)))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[DateTime]; name: string) =
var dt: DateTime
bindResult(db, s, idx, dt, t, name)
obj[name] = newJString(format(dt, jsonTimeFormat))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[JsonNode]; name: string) =
let src = pqgetvalue(queryResult, queryI, idx.cint)
obj[name] = parseJson($src)
template startQuery*(db: DbConn; s: PStmt) =
when declared(pparams):
var queryResult {.inject.} = pqexecPrepared(db, s, int32(parr.len),
cast[cstringArray](addr parr), nil, nil, 0)
else:
var queryResult {.inject.} = pqexecPrepared(db, s, int32(0),
nil, nil, nil, 0)
if pqResultStatus(queryResult) == PGRES_COMMAND_OK:
discard # insert does not returns data in pg
elif pqResultStatus(queryResult) != PGRES_TUPLES_OK: ormin_postgre.dbError(db)
var queryI {.inject.} = cint(-1)
var queryLen {.inject.} = pqntuples(queryResult)
template stopQuery*(db: DbConn; s: PStmt) =
pqclear(queryResult)
template stepQuery*(db: DbConn; s: PStmt; returnsData: bool): bool =
inc queryI
queryI < queryLen
template getLastId*(db: DbConn; s: PStmt): int = 0 # XXX to implement
template getAffectedRows*(db: DbConn; s: PStmt): int =
c_strtol(pqcmdTuples(queryResult))
proc close*(db: DbConn) =
## closes the database connection.
if db != nil: pqfinish(db)
proc open*(connection, user, password, database: string): DbConn =
## opens a database connection. Raises `DbError` if the connection could not
## be established.
let
colonPos = connection.find(':')
host = if colonPos < 0: connection
else: substr(connection, 0, colonPos-1)
port = if colonPos < 0: ""
else: substr(connection, colonPos+1)
result = pqsetdbLogin(host, port, nil, nil, database, user, password)
if pqStatus(result) != CONNECTION_OK: dbError(result) # result = nil
+295
View File
@@ -0,0 +1,295 @@
{.deadCodeElim: on.}
import json, times
import db_connector/db_common
import db_connector/sqlite3
import query_hooks
export db_common
type
DbConn* = PSqlite3 ## encapsulates a database connection
varcharType* = string
blobType* = seq[byte]
intType* = int
floatType* = float
boolType* = bool
timestampType* = DateTime
jsonType* = JsonNode
var jsonTimeFormat* = "yyyy-MM-dd HH:mm:ss"
proc dbError*(db: DbConn) {.noreturn.} =
## raises a DbError exception.
var e: ref DbError
new(e)
e.msg = $sqlite3.errmsg(db)
raise e
proc prepareStmt*(db: DbConn; q: string): PStmt =
when defined(debugOrminTrace):
echo "[[Ormin Executing]]: ", q
if prepare_v2(db, q, q.len.cint, result, nil) != SQLITE_OK:
dbError(db)
template startBindings*(s: PStmt; n: int) =
if clear_bindings(s) != SQLITE_OK: dbError(db)
template bindParam*(db: DbConn; s: PStmt; idx: int; x, t: untyped) =
#when not (x is t):
# {.error: "type mismatch for query argument at position " & $idx.}
when t is blobType:
let xs = x
let blobPtr = if xs.len == 0:
cast[pointer](nil)
else:
cast[pointer](unsafeAddr(xs[0]))
if bind_blob(s, idx.cint, blobPtr, xs.len.cint, SQLITE_TRANSIENT) != SQLITE_OK:
dbError(db)
elif t is int or t is int64 or t is bool:
if bind_int64(s, idx.cint, x.int64) != SQLITE_OK: dbError(db)
elif t is string:
if bind_text(s, idx.cint, cstring(x), x.len.cint, SQLITE_TRANSIENT) != SQLITE_OK:
dbError(db)
elif t is float64:
if bind_double(s, idx.cint, x) != SQLITE_OK:
dbError(db)
elif t is DateTime:
let xx = if x.nanosecond > 0:
x.utc().format("yyyy-MM-dd HH:mm:ss\'.\'fff")
else:
x.utc().format("yyyy-MM-dd HH:mm:ss")
if bind_text(s, idx.cint, cstring(xx), xx.len.cint, SQLITE_TRANSIENT) != SQLITE_OK:
dbError(db)
elif t is JsonNode:
let xx = $x
if bind_text(s, idx.cint, cstring(xx), xx.len.cint, SQLITE_TRANSIENT) != SQLITE_OK:
dbError(db)
else:
{.error: "type mismatch for query argument at position " & $idx.}
template bindNullParam*(db: DbConn; s: PStmt; idx: int) =
if bind_null(s, idx.cint) != SQLITE_OK:
dbError(db)
template bindParamJson*(db: DbConn; s: PStmt; idx: int; xx: JsonNode;
t: typedesc) =
let x = xx
if x.kind == JNull:
if bind_null(s, idx.cint) != SQLITE_OK: dbError(db)
else:
bindFromJson(db, s, idx, x, t)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc) =
{.error: "invalid type for JSON object".}
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[string]) =
doAssert x.kind == JString
let xs = x.str
if bind_text(s, idx.cint, cstring(xs), xs.len.cint, SQLITE_TRANSIENT) != SQLITE_OK:
dbError(db)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[int|int64]) =
doAssert x.kind == JInt
let xi = x.num
if bind_int64(s, idx.cint, xi.int64) != SQLITE_OK: dbError(db)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[float64]) =
doAssert x.kind == JFloat
let xf = x.fnum
if bind_double(s, idx.cint, xf) != SQLITE_OK:
dbError(db)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[bool]) =
doAssert x.kind == JBool
let xb = x.bval
if bind_int64(s, idx.cint, xb.int64) != SQLITE_OK: dbError(db)
template bindFromJson*(db: DbConn; s: PStmt; idx: int; x: JsonNode;
t: typedesc[DateTime]) =
doAssert x.kind == JString
let
dtStr = x.str
i = dtStr.find('.')
dt = if i > 0 and dtStr[i+1 .. ^1] == "000":
dtStr[0 ..< i]
else:
dtStr
if bind_text(s, idx.cint, cstring(dt), dt.len.cint, SQLITE_TRANSIENT) != SQLITE_OK:
dbError(db)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: int;
t: typedesc; name: string) =
dest = int column_int64(s, idx.cint)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: int64;
t: typedesc; name: string) =
dest = column_int64(s, idx.cint)
proc fillString(dest: var string; src: cstring; srcLen: int) =
when defined(nimNoNilSeqs):
setLen(dest, srcLen)
else:
if dest.isNil: dest = newString(srcLen)
else: setLen(dest, srcLen)
if srcLen > 0:
copyMem(unsafeAddr(dest[0]), src, srcLen)
proc fillBytes(dest: var seq[byte]; src: pointer; srcLen: int) =
when defined(nimNoNilSeqs):
setLen(dest, srcLen)
else:
if dest.isNil: dest = newSeq[byte](srcLen)
else: setLen(dest, srcLen)
if srcLen > 0:
copyMem(unsafeAddr(dest[0]), src, srcLen)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: var string;
t: typedesc; name: string) =
if column_type(s, idx.cint) == SQLITE_NULL:
when defined(nimNoNilSeqs):
setLen(dest, 0)
else:
dest = nil
else:
let srcLen = column_bytes(s, idx.cint)
let src = column_text(s, idx.cint)
fillString(dest, src, srcLen)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: var blobType;
t: typedesc; name: string) =
let srcLen = column_bytes(s, idx.cint)
let src = column_blob(s, idx.cint)
fillBytes(dest, src, srcLen)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: float64;
t: typedesc; name: string) =
dest = column_double(s, idx.cint)
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: bool;
t: typedesc; name: string) =
dest = column_int64(s, idx.cint) != 0
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: DateTime;
t: typedesc; name: string) =
let
src = $column_text(s, idx.cint)
i = src.find('.')
if i < 0:
dest = parse(src, "yyyy-MM-dd HH:mm:ss", utc())
else:
dest = parse(src, "yyyy-MM-dd HH:mm:ss\'.\'fff", utc())
template bindResult*(db: DbConn; s: PStmt; idx: int; dest: JsonNode;
t: typedesc; name: string) =
let src = column_text(s, idx.cint)
dest = parseJson($src)
template bindResult*[T](db: DbConn; s: PStmt; idx: int; dest: var DbValue[T];
t: typedesc; name: string) =
if column_type(s, idx.cint) == SQLITE_NULL:
dest.isNull = true
else:
dest.isNull = false
when T is string:
let srcLen = column_bytes(s, idx.cint)
let src = column_text(s, idx.cint)
fillString(dest.value, src, srcLen)
else:
bindResult(db, s, idx, dest.value, t, name)
template createJObject*(): untyped = newJObject()
template createJArray*(): untyped = newJArray()
template bindResultJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc; name: string) =
let x = obj
doAssert x.kind == JObject
if column_type(s, idx.cint) == SQLITE_NULL:
x[name] = newJNull()
else:
bindToJson(db, s, idx, x, t, name)
template columnIsNull*(db: DbConn; s: PStmt; idx: int): bool =
column_type(s, idx.cint) == SQLITE_NULL
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc; name: string) =
{.error: "invalid type for JSON object".}
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[string]; name: string) =
let dest = newJString("")
let srcLen = column_bytes(s, idx.cint)
let src = column_text(s, idx.cint)
fillString(dest.str, src, srcLen)
obj[name] = dest
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[int|int64]; name: string) =
obj[name] = newJInt(column_int64(s, idx.cint))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[blobType]; name: string) =
let srcLen = column_bytes(s, idx.cint)
let src = column_blob(s, idx.cint)
var data: blobType
fillBytes(data, src, srcLen)
let arr = newJArray()
for b in data:
arr.add newJInt(int(b))
obj[name] = arr
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[float64]; name: string) =
obj[name] = newJFloat(column_double(s, idx.cint))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[bool]; name: string) =
obj[name] = newJBool(column_int64(s, idx.cint) != 0)
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[DateTime]; name: string) =
var dt: DateTime
bindResult(db, s, idx, dt, t, name)
obj[name] = newJString(format(dt, jsonTimeFormat))
template bindToJson*(db: DbConn; s: PStmt; idx: int; obj: JsonNode;
t: typedesc[JsonNode]; name: string) =
let src = column_text(s, idx.cint)
obj[name] = parseJson($src)
template startQuery*(db: DbConn; s: PStmt) = discard "nothing to do"
template stopQuery*(db: DbConn; s: PStmt) =
if sqlite3.reset(s) != SQLITE_OK: dbError(db)
template stepQuery*(db: DbConn; s: PStmt; returnsData: bool): bool =
when returnsData:
step(s) == SQLITE_ROW
else:
step(s) == SQLITE_DONE
template getLastId*(db: DbConn; s: PStmt): int =
int(last_insert_rowid(db))
template getAffectedRows*(db: DbConn; s: PStmt): int =
int(changes(db))
proc close*(db: DbConn) =
## closes the database connection.
if sqlite3.close(db) != SQLITE_OK: dbError(db)
proc open*(connection, user, password, database: string): DbConn =
## opens a database connection. Raises `EDb` if the connection could not
## be established. Only the ``connection`` parameter is used for ``sqlite``.
if sqlite3.open(connection, result) != SQLITE_OK:
dbError(result)
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+54
View File
@@ -0,0 +1,54 @@
import options, json
type
DbValue*[T] = object
isNull*: bool
value*: T
template fromQueryHook*[T, S](val: var T, x: S) =
## Default conversion hook used by `query(T): ...`.
## Users can overload this proc to customize field/type conversions.
val = x
template toQueryHook*[T, S](val: var T, x: S) =
## Default conversion hook used for query parameters.
## Users can overload this proc to customize parameter conversions.
val = x
proc nullQueryValueError() {.noreturn.} =
raise newException(ValueError, "cannot map NULL query result")
proc fromQueryHook*[T, S](val: var Option[T], x: var DbValue[S]) =
if x.isNull:
val = none(T)
else:
var converted: T
fromQueryHook(converted, move x.value)
val = some(converted)
proc fromQueryHook*[T, S](val: var T, x: var DbValue[S]) =
if x.isNull:
when T is string:
val = ""
elif T is JsonNode:
val = newJNull()
else:
nullQueryValueError()
else:
fromQueryHook(val, move x.value)
proc bindFromQueryHook*[T, S](dest: var T, x: var DbValue[S]) =
fromQueryHook(dest, x)
proc toQueryHook*[S, T](val: var DbValue[S], x: Option[T]) =
if x.isSome:
val.isNull = false
toQueryHook(val.value, x.get)
else:
val.isNull = true
when compiles(val.value = default(S)):
val.value = default(S)
proc toQueryHook*[S, T](val: var DbValue[S], x: T) =
val.isNull = false
toQueryHook(val.value, x)
+139
View File
@@ -0,0 +1,139 @@
import asynchttpserver, asyncdispatch, asyncnet, json,
strutils, times
import websocket
type
Receivers* {.pure} = enum
sender, allExceptSender, all
ReqHandler* = proc (msg: JsonNode; receivers: var Receivers): JsonNode {.
closure, gcsafe.}
proc error(msg: string) = echo "[Error] ", msg
proc warn(msg: string) = echo "[Warning] ", msg
proc hint(msg: string) = echo "[Hint] ", msg
type
Client = ref object
socket: AsyncWebSocket
connected: bool
hostname: string
id: int
lastMessage: float
rapidMessageCount: int
Server = ref object
clients: seq[Client]
needsUpdate: bool
receivers: Receivers
msgs: seq[(string, int)]
gid: int
handler: ReqHandler
proc newClient(s: Server; socket: AsyncWebSocket, hostname: string): Client =
inc s.gid
result = Client(socket: socket, connected: true, hostname: hostname, id: s.gid)
proc `$`(client: Client): string =
"Client(ip: $1)" % [client.hostname]
proc updateClients(server: Server) {.async.} =
while true:
var needsUpdate = false
for client in server.clients:
if not client.connected:
needsUpdate = true
break
server.needsUpdate = server.needsUpdate or needsUpdate
if server.needsUpdate and server.msgs.len != 0:
var someDead = false
# perform a copy to prevent the race condition:
var msgs = server.msgs
setLen(server.msgs, 0)
for m in msgs:
for c in server.clients:
if c.connected:
# do not send a broadcast message to the one which sent it:
if server.receivers != Receivers.allExceptSender or c.id != m[1]:
await c.socket.sendText(m[0]) #, false)
else:
someDead = true
if someDead:
var i = 0
while i < server.clients.len:
if not server.clients[i].connected: del(server.clients, i)
else: inc i
server.needsUpdate = false
# let other stuff in the main loop run:
await sleepAsync(10)
proc processMessage(server: Server, client: Client, data: string) {.async.} =
# Check if last message was relatively recent. If so, kick the user.
echo "processMessage ", data
let now = epochTime()
if now - client.lastMessage < 0.1: # 100ms
client.rapidMessageCount.inc
else:
client.rapidMessageCount = 0
client.lastMessage = now
if client.rapidMessageCount > 50:
warn("Client ($1) is firing messages too rapidly. Killing." % $client)
client.connected = false
let msgj = parseJson(data)
server.receivers = Receivers.sender
if msgj.hasKey("msg") and msgj["msg"].getStr("") == "disconnect":
client.connected = false
server.needsUpdate = true
else:
let resp = server.handler(msgj, server.receivers)
if not resp.isNil:
if server.receivers == sender:
await client.socket.sendText($resp) #, false)
else:
server.msgs.add(($resp, client.id))
server.needsUpdate = true
proc processClient(server: Server, client: Client) {.async.} =
while client.connected:
var frameFut = client.socket.readData()
yield frameFut
if frameFut.failed:
error("Error occurred handling client messages.\n" &
frameFut.error.msg)
client.connected = false
break
let frame = frameFut.read()
if frame.opcode == Opcode.Text:
let processFut = processMessage(server, client, frame.data)
if processFut.failed:
echo processFut.error.getStackTrace()
error("Client ($1) attempted to send bad JSON? " % $client & "\n" &
processFut.error.msg)
#client.connected = false
await client.socket.close()
proc onRequest(server: Server, req: Request; key: string) {.async.} =
let (ws, error) = await verifyWebsocketRequest(req, key)
if ws != nil:
hint("Client connected from " & req.hostname)
ws.protocol = key
server.clients.add(newClient(server, ws, req.hostname))
asyncCheck processClient(server, server.clients[^1])
else:
warn("WS negotiation failed: " & $error)
await req.respond(Http400, "WebSocket negotiation failed: " & $error)
req.client.close()
proc serve*(key: string; handler: ReqHandler) =
let httpServer = newAsyncHttpServer()
let server = Server(clients: @[], msgs: @[], handler: handler, gid: 0)
proc cb(req: Request): Future[void] {.async, gcsafe.} =
await onRequest(server, req, key)
asyncCheck updateClients(server)
waitFor httpServer.serve(Port(8080), cb)
+55
View File
@@ -0,0 +1,55 @@
include system/inclrtl
import macros
proc transLastStmt(n, res, bracketExpr: NimNode): (NimNode, NimNode, NimNode) =
case n.kind
of nnkIfExpr, nnkIfStmt, nnkTryStmt, nnkCaseStmt:
result[0] = copyNimTree(n)
result[1] = copyNimTree(n)
result[2] = copyNimTree(n)
for i in ord(n.kind == nnkCaseStmt)..<n.len:
(result[0][i], result[1][^1], result[2][^1]) = transLastStmt(n[i], res, bracketExpr)
of nnkStmtList, nnkStmtListExpr, nnkBlockStmt, nnkBlockExpr, nnkWhileStmt,
nnkForStmt, nnkElifBranch, nnkElse, nnkElifExpr, nnkOfBranch, nnkExceptBranch:
result[0] = copyNimTree(n)
result[1] = copyNimTree(n)
result[2] = copyNimTree(n)
if n.len >= 1:
(result[0][^1], result[1][^1], result[2][^1]) = transLastStmt(n[^1], res, bracketExpr)
of nnkTableConstr:
result[1] = n[0][0]
result[2] = n[0][1]
if bracketExpr.len == 1:
bracketExpr.add([newCall(bindSym"typeof", newEmptyNode()), newCall(
bindSym"typeof", newEmptyNode())])
template adder(res, k, v) = res[k] = v
result[0] = getAst(adder(res, n[0][0], n[0][1]))
of nnkCurly:
result[2] = n[0]
if bracketExpr.len == 1:
bracketExpr.add(newCall(bindSym"typeof", newEmptyNode()))
template adder(res, v) = res.incl(v)
result[0] = getAst(adder(res, n[0]))
else:
result[2] = n
if bracketExpr.len == 1:
bracketExpr.add(newCall(bindSym"typeof", newEmptyNode()))
template adder(res, v) = res.add(v)
result[0] = getAst(adder(res, n))
macro collect*(init, body: untyped): untyped =
let res = genSym(nskVar, "collectResult")
expectKind init, {nnkCall, nnkIdent, nnkSym}
let bracketExpr = newTree(nnkBracketExpr,
if init.kind == nnkCall: init[0] else: init)
let (resBody, keyType, valueType) = transLastStmt(body, res, bracketExpr)
if bracketExpr.len == 3:
bracketExpr[1][1] = keyType
bracketExpr[2][1] = valueType
else:
bracketExpr[1][1] = valueType
let call = newTree(nnkCall, bracketExpr)
if init.kind == nnkCall:
for i in 1 ..< init.len:
call.add init[i]
result = newTree(nnkStmtListExpr, newVarStmt(res, call), resBody, res)
+1
View File
@@ -0,0 +1 @@
--path:"$projectDir/../../ormin"
@@ -0,0 +1,26 @@
-- lower case, upper case, and quoted table names
create table lower_table (
id integer primary key
);
CREATE TABLE UPPER_TABLE (
id integer primary key
);
-- std sql quoted table name
create table "Quoted Table" (
id integer primary key
);
-- sqlite quoted table name
create table `Quoted Table2` (
id integer primary key
);
create table "UPPER_QUOTED" (
id integer primary key
);
create table "A""B" (
id integer primary key
);
@@ -0,0 +1,60 @@
create table if not exists thread(
id serial primary key,
name varchar(100) not null,
views integer not null,
modified timestamp not null default CURRENT_TIMESTAMP
);
create unique index if not exists ThreadNameIx on thread (name);
create table if not exists person(
id serial primary key,
name varchar(20) not null,
password varchar(32) not null,
email varchar(30) not null,
creation timestamp not null default CURRENT_TIMESTAMP,
salt varchar(128) not null,
status varchar(30) not null,
lastOnline timestamp not null default CURRENT_TIMESTAMP,
ban varchar(128) not null default ''
);
create unique index if not exists UserNameIx on person (name);
create table if not exists post(
id serial primary key,
author integer not null,
ip varchar(20) not null,
header varchar(100) not null,
content varchar(1000) not null,
thread integer not null,
creation timestamp not null default CURRENT_TIMESTAMP,
foreign key (thread) references thread(id),
foreign key (author) references person(id)
);
create table if not exists session(
id serial primary key,
ip varchar(20) not null,
password varchar(32) not null,
userid integer not null,
lastModified timestamp not null default CURRENT_TIMESTAMP,
foreign key (userid) references person(id)
);
create table if not exists antibot(
id serial primary key,
ip varchar(20) not null,
answer varchar(30) not null,
created timestamp not null default CURRENT_TIMESTAMP
);
create table if not exists error(
uuid varchar(36) primary key,
message varchar(1000) not null,
created timestamp not null default CURRENT_TIMESTAMP
);
create index PersonStatusIdx on person(status);
create index PostByAuthorIdx on post(thread, author);
@@ -0,0 +1,62 @@
pragma foreign_keys = on;
create table if not exists thread(
id integer primary key,
name varchar(100) not null,
views integer not null,
modified timestamp not null default (DATETIME('now'))
);
create unique index if not exists ThreadNameIx on thread (name);
create table if not exists person(
id integer primary key,
name varchar(20) not null,
password varchar(32) not null,
email varchar(30) not null,
creation timestamp not null default (DATETIME('now')),
salt varchar(128) not null,
status varchar(30) not null,
lastOnline timestamp not null default (DATETIME('now')),
ban varchar(128) not null default ''
);
create unique index if not exists UserNameIx on person (name);
create table if not exists post(
id integer primary key,
author integer not null,
ip varchar(20) not null,
header varchar(100) not null,
content varchar(1000) not null,
thread integer not null,
creation timestamp not null default (DATETIME('now')),
foreign key (thread) references thread(id),
foreign key (author) references person(id)
);
create table if not exists session(
id integer primary key,
ip varchar(20) not null,
password varchar(32) not null,
userid integer not null,
lastModified timestamp not null default (DATETIME('now')),
foreign key (userid) references person(id)
);
create table if not exists antibot(
id integer primary key,
ip varchar(20) not null,
answer varchar(30) not null,
created timestamp not null default (DATETIME('now'))
);
create table if not exists error(
uuid varchar(36) primary key,
message varchar(1000) not null,
created timestamp not null default (DATETIME('now'))
);
create index PersonStatusIdx on person(status);
create index PostByAuthorIdx on post(thread, author);
+45
View File
@@ -0,0 +1,45 @@
create table if not exists tb_serial(
typserial serial primary key,
typinteger integer not null
);
create table if not exists tb_boolean(
typboolean boolean not null
);
create table if not exists tb_float(
typfloat real not null
);
create table if not exists tb_string(
typstring text not null
);
create table if not exists tb_timestamp(
dt timestamp not null,
dtn timestamptz not null,
dtz timestamptz not null
);
create table if not exists tb_json(
typjson json not null
);
create table if not exists tb_composite_pk(
pk1 integer not null,
pk2 integer not null,
message text not null,
primary key (pk1, pk2)
);
create table if not exists tb_composite_fk(
id integer not null,
fk1 integer not null,
fk2 integer not null,
foreign key (fk1, fk2) references tb_composite_pk(pk1, pk2)
);
create table if not exists tb_nullable(
id integer primary key,
note text null
);
+48
View File
@@ -0,0 +1,48 @@
create table if not exists tb_serial(
typserial integer primary key,
typinteger integer not null
);
create table if not exists tb_boolean(
typboolean boolean not null
);
create table if not exists tb_float(
typfloat real not null
);
create table if not exists tb_string(
typstring text not null
);
create table if not exists tb_timestamp(
dt1 timestamp not null,
dt2 timestamp not null
);
create table if not exists tb_json(
typjson json not null
);
create table if not exists tb_composite_pk(
pk1 integer not null,
pk2 integer not null,
message text not null,
primary key (pk1, pk2)
);
create table if not exists tb_composite_fk(
id integer not null,
fk1 integer not null,
fk2 integer not null,
foreign key (fk1, fk2) references tb_composite_pk(pk1, pk2)
);
create table if not exists tb_blob(
id integer primary key,
typblob blob not null
);
create table if not exists tb_nullable(
id integer primary key,
note text null
);
@@ -0,0 +1,4 @@
create table user_static (
id integer primary key,
name varchar(255) not null
);
+508
View File
@@ -0,0 +1,508 @@
import unittest, json, strutils, strformat, sequtils, macros, times, os, math, unicode
import ormin
import ormin/db_utils
when defined(postgre):
when defined(macosx):
{.passL: "-Wl,-rpath,/opt/homebrew/lib/postgresql@14".}
import db_connector/db_postgres as db_postgres
const backend = DbBackend.postgre
importModel(backend, "model_postgre")
const sqlFileName = "model_postgre.sql"
let db {.global.} = db_postgres.open("localhost", "test", "test", "test_ormin")
else:
from db_connector/db_sqlite import exec, getValue
const backend = DbBackend.sqlite
importModel(backend, "model_sqlite")
const sqlFileName = "model_sqlite.sql"
# let db {.global.} = open(":memory:", "", "", "")
let db {.global.} = open("test.db", "", "", "")
let
testDir = currentSourcePath.parentDir()
sqlFile = Path(testDir / sqlFileName)
proc substr(s: string; start, length: int): string {.importSql.}
let
min = -3
max = 3
serial_int_seqs = [(1, min), (2, 1), (3, 2), (4, max)]
suite &"Test common database types and functions of {backend}":
discard
suite "serial_insert":
setup:
db.dropTable(sqlFile, "tb_serial")
db.createTable(sqlFile, "tb_serial")
test "insert":
for i in serial_int_seqs:
query:
insert tb_serial(typinteger = ?i[1])
check db.getValue(sql"select count(*) from tb_serial") == $serial_int_seqs.len
test "json":
for i in serial_int_seqs:
let v = %*{"typinteger": i[1]}
query:
insert tb_serial(typinteger = %v["typinteger"])
check db.getValue(sql"select count(*) from tb_serial") == $serial_int_seqs.len
suite "serial":
db.dropTable(sqlFile, "tb_serial")
db.createTable(sqlFile, "tb_serial")
let insertSql = sql"insert into tb_serial(typinteger) values (?)"
for (_, v) in serial_int_seqs:
db.exec(insertSql, v)
doAssert db.getValue(sql"select count(*) from tb_serial") == $serial_int_seqs.len
test "query":
let res = query:
select tb_serial(typserial, typinteger)
check res == serial_int_seqs
test "where":
let
id = 1
res = query:
select tb_serial(typserial, typinteger)
where typserial == ?id
check res == serial_int_seqs.filterIt(it[0] == id)
test "json":
let res = query:
select tb_serial(typserial, typinteger)
produce json
check res == %*serial_int_seqs.mapIt(%*{"typserial": it[0], "typinteger": it[1]})
test "count":
let res = query:
select tb_serial(count(_))
check res[0] == serial_int_seqs.len
test "sum":
let res = query:
select tb_serial(sum(typinteger))
check res[0] == serial_int_seqs.mapIt(it[1]).sum()
test "avg":
let res = query:
select tb_serial(avg(typinteger))
check res[0] == serial_int_seqs.mapIt(it[1]).sum() / serial_int_seqs.len()
test "min":
let res = query:
select tb_serial(min(typinteger))
check res[0] == min
test "max":
let res = query:
select tb_serial(max(typinteger))
check res[0] == max
test "abs":
let res = query:
select tb_serial(abs(typinteger))
check res == serial_int_seqs.mapIt(abs(it[1]))
let seqs = toSeq(1..3)
suite "boolean_insert":
setup:
db.dropTable(sqlFile, "tb_boolean")
db.createTable(sqlFile, "tb_boolean")
test "insert":
for i in seqs:
let b = (i mod 2 == 0)
query:
insert tb_boolean(typboolean = ?b)
check db.getValue(sql"select count(*) from tb_boolean") == $seqs.len
test "json":
for i in seqs:
let b = %*{"typboolean": (i mod 2 == 0)}
query:
insert tb_boolean(typboolean = %b["typboolean"])
check db.getValue(sql"select count(*) from tb_boolean") == $seqs.len
suite "boolean":
db.dropTable(sqlFile, "tb_boolean")
db.createTable(sqlFile, "tb_boolean")
proc toBool(x: int): bool =
if x mod 2 == 0: true else: false
proc toDbValue(x: int): auto =
when defined(postgre):
if x mod 2 == 0: "t" else: "f"
else:
if x mod 2 == 0: 1 else: 0
let insertSql = sql"insert into tb_boolean(typboolean) values (?)"
for i in seqs:
db.exec(insertSql, toDbValue(i))
doAssert db.getValue(sql"select count(*) from tb_boolean") == $seqs.len
test "query":
let res = query:
select tb_boolean(typboolean)
check res == seqs.mapIt(it mod 2 == 0)
test "where":
let
b = true
res = query:
select tb_boolean(typboolean)
where typboolean == ?b
check res == seqs.map(toBool).filterIt(it == b)
test "json":
let res = query:
select tb_boolean(typboolean)
produce json
check res == %*seqs.mapIt(%*{"typboolean": toBool(it)})
when defined(sqlite):
test "importSql substr wrong type":
## TODO: should this be allowed?
let res = query:
select tb_boolean(substr(typboolean, 1, 2))
echo "res: ", res
let fs = [-3.14, 2.56, 10.45]
suite "float_insert":
setup:
db.dropTable(sqlFile, "tb_float")
db.createTable(sqlFile, "tb_float")
test "insert":
for f in fs:
query:
insert tb_float(typfloat = ?f)
check db.getValue(sql"select count(*) from tb_float") == $fs.len
test "json":
for f in fs:
let v = %*{"typfloat": f}
query:
insert tb_float(typfloat = %v["typfloat"])
check db.getValue(sql"select count(*) from tb_float") == $fs.len
suite "float":
db.dropTable(sqlFile, "tb_float")
db.createTable(sqlFile, "tb_float")
let insertSql = sql"insert into tb_float(typfloat) values (?)"
for f in fs:
db.exec(insertSql, f)
doAssert db.getValue(sql"select count(*) from tb_float") == $fs.len
test "query":
let res = query:
select tb_float(typfloat)
check res == fs
test "where":
let res = query:
select tb_float(typfloat)
where typfloat == ?fs[0]
check res == [fs[0]]
test "json":
let res = query:
select tb_float(typfloat)
produce json
check res == %*fs.mapIt(%*{"typfloat": it})
test "abs":
let res = query:
select tb_float(abs(typfloat))
check res == fs.mapIt(abs(it))
let ss = ["one", "Two", "three", "第四", "four'th"]
suite "string_insert":
setup:
db.dropTable(sqlFile, "tb_string")
db.createTable(sqlFile, "tb_string")
test "insert":
for v in ss:
query:
insert tb_string(typstring = ?v)
check db.getValue(sql"select count(*) from tb_string") == $ss.len
test "insert":
for v in ss:
query:
insert tb_string(typstring = "can't touch this")
check db.getValue(sql"select count(*) from tb_string") == $ss.len
test "json":
for v in ss:
let j = %*{"typstring": v}
query:
insert tb_string(typstring = %j["typstring"])
check db.getValue(sql"select count(*) from tb_string") == $ss.len
suite "string":
db.dropTable(sqlFile, "tb_string")
db.createTable(sqlFile, "tb_string")
let insertsql = sql"insert into tb_string(typstring) values (?)"
for v in ss:
db.exec(insertsql, v)
doAssert db.getValue(sql"select count(*) from tb_string") == $ss.len
test "query":
let res = query:
select tb_string(typstring)
check res == ss
test "where":
let res = query:
select tb_string(typstring)
where typstring == ?ss[0]
check res[0] == ss[0]
test "json":
let res = query:
select tb_string(typstring)
produce json
check res == %*ss.mapIt(%*{"typstring": it})
test "concat_op":
let
s = "typstring: "
let res = query:
select tb_string(?s & typstring)
check res == ss.mapIt(s & it)
test "length":
let res = query:
select tb_string(length(typstring))
check res == ss.mapIt(it.runeLen)
test "lower":
let res = query:
select tb_string(lower(typstring))
check res == ss.mapIt(it.toLowerAscii)
test "upper":
let res = query:
select tb_string(upper(typstring))
check res == ss.mapIt(it.toUpperAscii)
test "replace":
let res = query:
select tb_string(replace(typstring, "e", "o"))
check res == ss.mapIt(it.replace("e", "o"))
test "importSql substr length 0":
let res = query:
select tb_string(substr(typstring, 1, 0))
let expected = ss.mapIt($(toRunes(it)[0..<0]))
check res == expected
test "importSql substr length no arg types checked":
# TODO: fixme!
# the `functions` array only contains the types of the return
# but sqlite doesn't really care...
let res = query:
select tb_string(substr(typstring, "1", 2))
let expected = ss.mapIt($(toRunes(it)[0..<2]))
check res == expected
let js = [
%*{"name": "tom", "age": 30},
%*{"name": "bob", "age": 35},
%*{"name": "jack", "age": 23}
]
suite "insert_json":
setup:
db.dropTable(sqlFile, "tb_json")
db.createTable(sqlFile, "tb_json")
test "insert":
for j in js:
query:
insert tb_json(typjson = ?j)
check db.getValue(sql"select count(*) from tb_json") == $js.len
suite "json":
db.dropTable(sqlFile, "tb_json")
db.createTable(sqlFile, "tb_json")
let insertSql = sql"insert into tb_json(typjson) values (?)"
for j in js:
db.exec(insertSql, j)
doAssert db.getValue(sql"select count(*) from tb_json") == $js.len
test "query":
let res = query:
select tb_json(typjson)
check res == js
test "json":
let res = query:
select tb_json(typjson)
produce json
check res == %*js.mapIt(%*{"typjson": it})
let cps = [
(1, 1, "one-one"),
(1, 2, "one-two"),
(2, 1, "two-one")
]
suite "composite_pk_insert":
setup:
db.dropTable(sqlFile, "tb_composite_pk")
db.createTable(sqlFile, "tb_composite_pk")
test "insert":
for r in cps:
query:
insert tb_composite_pk(pk1 = ?r[0], pk2 = ?r[1], message = ?r[2])
check db.getValue(sql"select count(*) from tb_composite_pk") == $cps.len
test "json":
for r in cps:
let v = %*{"pk1": r[0], "pk2": r[1], "message": r[2]}
query:
insert tb_composite_pk(pk1 = %v["pk1"], pk2 = %v["pk2"], message = %v["message"])
check db.getValue(sql"select count(*) from tb_composite_pk") == $cps.len
suite "composite_pk":
db.dropTable(sqlFile, "tb_composite_pk")
db.createTable(sqlFile, "tb_composite_pk")
let insertSql = sql"insert into tb_composite_pk(pk1, pk2, message) values (?, ?, ?)"
for r in cps:
db.exec(insertSql, r[0], r[1], r[2])
doAssert db.getValue(sql"select count(*) from tb_composite_pk") == $cps.len
test "query":
let res = query:
select tb_composite_pk(pk1, pk2, message)
check res == cps
test "where":
let res = query:
select tb_composite_pk(pk1, pk2, message)
where pk1 == ?cps[0][0]
check res == cps.filterIt(it[0] == cps[0][0])
test "json":
let res = query:
select tb_composite_pk(pk1, pk2, message)
produce json
check res == %*cps.mapIt(%*{"pk1": it[0], "pk2": it[1], "message": it[2]})
suite "nullable":
setup:
db.dropTable(sqlFile, "tb_nullable")
db.createTable(sqlFile, "tb_nullable")
query:
insert tb_nullable(id = 1, note = nil)
query:
insert tb_nullable(id = 2, note = "hello")
test "where_null":
let res = query:
select tb_nullable(id)
where note == nil
check res == @[1]
test "where_not_null":
let res = query:
select tb_nullable(id)
where note != null
check res == @[2]
test "update_to_null":
query:
update tb_nullable(note = null)
where id == 2
let res = query:
select tb_nullable(id)
where note == nil
check res == @[1, 2]
test "json_null":
let res = query:
select tb_nullable(id, note)
orderby id
produce json
check res[0]["note"].kind == JNull
check res[1]["note"].getStr == "hello"
suite "upsert":
setup:
db.dropTable(sqlFile, "tb_nullable")
db.createTable(sqlFile, "tb_nullable")
test "onconflict do nothing":
let first = "first value"
query:
insert tb_nullable(id = 1, note = ?first)
let ignored = "should not overwrite"
query:
insert tb_nullable(id = 1, note = ?ignored)
onconflict(id)
donothing()
let note = db.getValue(sql"select note from tb_nullable where id = 1")
check note == first
test "onconflict do update":
let first = "old note"
query:
insert tb_nullable(id = 2, note = ?first)
let replacement = "new note"
query:
insert tb_nullable(id = 2, note = ?replacement)
onconflict(id)
doupdate(note = ?replacement)
let note = db.getValue(sql"select note from tb_nullable where id = 2")
check note == replacement
test "onconflict do update where":
query:
insert tb_nullable(id = 3, note = "keep-me")
query:
insert tb_nullable(id = 3, note = "replace-me")
onconflict(id)
doupdate(note = "replace-me")
where note != "keep-me"
let note = db.getValue(sql"select note from tb_nullable where id = 3")
check note == "keep-me"
static:
doAssert not compiles(
(block:
query:
insert tb_nullable(id = 100, note = "x")
where id == 100
)
)
+166
View File
@@ -0,0 +1,166 @@
import unittest, os, sequtils
import db_connector/db_common
from db_connector/db_sqlite import open, exec, getValue
import ormin/db_utils
const usesLegacySqliteDropNames = defined(sqlite) and defined(orminLegacySqliteDropNames)
let
db {.global.} = open(":memory:", "", "", "")
testDir = currentSourcePath.parentDir()
sqlFile = Path(testDir / "db_utils_case_quoted.sql")
let sqlContent = """
-- lower case, upper case, and quoted table names
create table lower_table (
id integer primary key
);
CREATE TABLE UPPER_TABLE (
id integer primary key
);
-- std sql quoted table name
create table "Quoted Table" (
id integer primary key
);
-- sqlite quoted table name
create table `Quoted Table2` (
id integer primary key
);
create table "UPPER_QUOTED" (
id integer primary key
);
create table "A""B" (
id integer primary key
);
"""
const staticSqlContent = staticLoad("db_utils_case_quoted.sql")
writeFile($sqlFile, sqlContent)
suite "db_utils: case and quoted names":
test "staticLoad returns typed compile-time SQL":
let loaded: DbSql = staticSqlContent
check $loaded == sqlContent
test "quoteDbIdentifier quotes and escapes identifiers":
check $quoteDbIdentifier("lower_table") == "lower_table"
check $quoteDbIdentifier("UPPER_TABLE") == "upper_table"
check $quoteDbIdentifier("\"Quoted Table\"") == "\"Quoted Table\""
check $quoteDbIdentifier("`Quoted Table2`") == "\"Quoted Table2\""
check $quoteDbIdentifier("'Quoted Table3'") == "\"Quoted Table3\""
check $quoteDbIdentifier("\"UPPER_TABLE\"") == "\"UPPER_TABLE\""
check $quoteDbIdentifier("\"A\"\"B\"") == "\"A\"\"B\""
check $quoteDbIdentifier("schema.table") == "schema.table"
check $quoteDbIdentifier("Schema.\"Mixed Table\"") == "schema.\"Mixed Table\""
check $quoteDbIdentifier("weird\"name") == "\"weird\"\"name\""
test "check tables names":
let pairs = tablePairs(sqlContent).toSeq()
check pairs.len == 6
check pairs[0][0] == ("lower_table")
check pairs[1][0] == ("upper_table")
check pairs[2][0] == ("quoted table")
check pairs[3][0] == ("quoted table2")
check pairs[4][0] == ("upper_quoted")
check pairs[5][0] == ("a\"b")
check pairs[0][1] == "create table lower_table(id integer primary key );"
check pairs[1][1] == "create table UPPER_TABLE(id integer primary key );"
check pairs[2][1] == "create table \"Quoted Table\"(id integer primary key );"
check pairs[4][1] == "create table \"UPPER_QUOTED\"(id integer primary key );"
check pairs[5][1] == "create table \"A\"\"B\"(id integer primary key );"
test "tablePairs parses compile-time SQL":
let pairs = tablePairs(staticSqlContent).toSeq()
check pairs.len == 6
check pairs[0][0] == "lower_table"
check pairs[1][0] == "upper_table"
check pairs[2][0] == "quoted table"
check pairs[3][0] == "quoted table2"
check pairs[4][0] == "upper_quoted"
check pairs[5][0] == "a\"b"
test "createTable creates all tables from SQL file":
db.createTable(sqlFile)
let countAll = db.getValue(sql"select count(*) from sqlite_master where type='table' and name in ('lower_table','UPPER_TABLE','Quoted Table')")
check countAll == "3"
test "createTable with specific lowercased name matches quoted":
# Use a new in-memory DB for isolation
let db2 = open(":memory:", "", "", "")
db2.createTable(sqlFile, "quoted table")
let countQuoted = db2.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'Quoted Table'")
check countQuoted == "1"
test "createTable creates all tables from compile-time SQL":
let db2 = open(":memory:", "", "", "")
db2.createTable(staticSqlContent)
let countAll = db2.getValue(sql("select count(*) from sqlite_master where type='table' and name in ('lower_table','UPPER_TABLE','Quoted Table','Quoted Table2','UPPER_QUOTED','A\"B')"))
check countAll == "6"
test "createTable with specific lowercased name matches quoted from compile-time SQL":
let db2 = open(":memory:", "", "", "")
db2.createTable(staticSqlContent, "quoted table")
let countQuoted = db2.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'Quoted Table'")
check countQuoted == "1"
test "dropTable handles quoted names from SQL file":
let db2 = open(":memory:", "", "", "")
db2.createTable(sqlFile)
db2.dropTable(sqlFile, "quoted table2")
let countQuoted = db2.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'Quoted Table2'")
when usesLegacySqliteDropNames:
check countQuoted == "1"
else:
check countQuoted == "0"
db2.dropTable(sqlFile, "upper_quoted")
let countUpperQuoted = db2.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'UPPER_QUOTED'")
when usesLegacySqliteDropNames:
check countUpperQuoted == "1"
else:
check countUpperQuoted == "0"
db2.dropTable(sqlFile, "a\"b")
let countEscapedQuoted = db2.getValue(sql("select count(*) from sqlite_master where type='table' and name = 'A\"B'"))
when usesLegacySqliteDropNames:
check countEscapedQuoted == "1"
else:
check countEscapedQuoted == "0"
test "dropTable removes tables from compile-time SQL":
let db2 = open(":memory:", "", "", "")
db2.createTable(staticSqlContent)
db2.dropTable(staticSqlContent, "quoted table2")
let countQuoted = db2.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'Quoted Table2'")
when usesLegacySqliteDropNames:
check countQuoted == "1"
else:
check countQuoted == "0"
db2.dropTable(staticSqlContent, "upper_quoted")
let countUpperQuoted = db2.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'UPPER_QUOTED'")
when usesLegacySqliteDropNames:
check countUpperQuoted == "1"
else:
check countUpperQuoted == "0"
db2.dropTable(staticSqlContent, "a\"b")
let countEscapedQuoted = db2.getValue(sql("select count(*) from sqlite_master where type='table' and name = 'A\"B'"))
when usesLegacySqliteDropNames:
check countEscapedQuoted == "1"
else:
check countEscapedQuoted == "0"
db2.dropTable(staticSqlContent)
let countAll = db2.getValue(sql("select count(*) from sqlite_master where type='table' and name in ('lower_table','UPPER_TABLE','Quoted Table','Quoted Table2','UPPER_QUOTED','A\"B')"))
when usesLegacySqliteDropNames:
check countAll == "3"
else:
check countAll == "0"
+822
View File
@@ -0,0 +1,822 @@
import unittest, strformat, sequtils, algorithm, sugar, json, tables, random
import os
import ormin
import ormin/db_utils
when NimVersion < "1.2.0": import ./compat
let testDir = currentSourcePath.parentDir()
when defined postgre:
when defined(macosx):
{.passL: "-Wl,-rpath,/opt/homebrew/lib/postgresql@14".}
from db_connector/db_postgres import exec, getValue
const backend = DbBackend.postgre
importModel(backend, "forum_model_postgres")
const sqlFileName = "forum_model_postgres.sql"
let db {.global.} = open("localhost", "test", "test", "test_ormin")
else:
from db_connector/db_sqlite import exec, getValue
const backend = DbBackend.sqlite
importModel(backend, "forum_model_sqlite")
const sqlFileName = "forum_model_sqlite.sql"
var memoryPath = testDir & "/" & ":memory:"
let db {.global.} = open(memoryPath, "", "", "")
var sqlFilePath = Path(testDir & "/" & sqlFileName)
type
Person = tuple[id: int,
name: string,
password: string,
email: string,
salt: string,
status: string]
Thread = tuple[id: int,
name: string,
views: int]
Post = tuple[id: int,
author: int,
ip: string,
header: string,
content: string,
thread: int]
Antibot = tuple[id: int,
ip: string,
answer: string]
const
personcount = 5
threadcount = 5
postcount = 10
antibotcount = 5
var
persondata: seq[Person]
threaddata: seq[Thread]
postdata: seq[Post]
antibotdata: seq[Antibot]
suite &"Test ormin features of {backend}":
discard
suite "query":
db.dropTable(sqlFilePath)
db.createTable(sqlFilePath)
static:
doAssert compiles(block:
discard query:
select post(author)
crossjoin person(name)
)
doAssert compiles(block:
discard query:
select post(author)
outerjoin person(name) on author == id
)
doAssert compiles(block:
discard query:
select post(author)
leftjoin person(name) on author == id
)
doAssert compiles(block:
discard query:
select post(author)
leftouterjoin person(name) on author == id
)
doAssert compiles(block:
discard query:
select post(author)
rightjoin person(name) on author == id
)
doAssert compiles(block:
discard query:
select post(author)
rightouterjoin person(name) on author == id
)
doAssert compiles(block:
discard query:
select post(author)
fulljoin person(name) on author == id
)
doAssert compiles(block:
discard query:
select post(author)
fullouterjoin person(name) on author == id
)
# prepare data to insert
for i in 1..personcount:
persondata.add((id: i,
name: fmt"john{i}",
password: fmt"pass{i}",
email: fmt"john{i}@mail.com",
salt: fmt"abcd{i}",
status: fmt"ok{i}"))
for i in 1..threadcount:
threaddata.add((id: i,
name: fmt"thread{i}",
views: i))
for i in 1..postcount:
postdata.add((id: i,
author: sample({1..personcount}),
ip: "",
header: fmt"title{i}",
content: fmt"content{i}",
thread: sample({1..threadcount})))
for i in 1..antibotcount:
antibotdata.add((id: i,
ip: "",
answer: fmt"answer{i}"))
# insert data into database
let
insertperson = sql"insert into person (id, name, password, email, salt, status) values (?, ?, ?, ?, ?, ?)"
insertthread = sql"insert into thread (id, name, views) values (?, ?, ?)"
insertpost = sql"insert into post (id, author, ip, header, content, thread) values (?, ?, ?, ?, ?, ?)"
insertantibot = sql"insert into antibot (id, ip, answer) values (?, ?, ?)"
for p in persondata:
db.exec(insertperson, p.id, p.name, p.password, p.email, p.salt, p.status)
for t in threaddata:
db.exec(insertthread, t.id, t.name, t.views)
for p in postdata:
db.exec(insertpost, p.id, p.author, p.ip, p.header, p.content, p.thread)
for a in antibotdata:
db.exec(insertantibot, a.id, a.ip, a.answer)
# check data in database
let personexpected = db.getValue(sql"select count(*) from person")
assert personexpected == $personcount
let threadexpected = db.getValue(sql"select count(*) from thread")
assert threadexpected == $threadcount
let postexpected = db.getValue(sql"select count(*) from post")
assert postexpected == $postcount
let antibotexpected = db.getValue(sql"select count(*) from antibot")
assert antibotexpected == $antibotcount
test "table":
let res = query:
select person(id, name, password, email, salt, status)
check res == persondata
test "all_column":
let res = query:
select person(_)
check res.mapIt((it.id, it.name, it.password, it.email, it.salt, it.status)) == persondata
test "column_alias":
let res = query:
select person(id as personid, name as personname)
check res == persondata.mapIt((personid: it.id, personname: it.name))
test "arithmetic":
let res = query:
select person(id * 4 / 2 + 2 - 1 as id)
check res == persondata.mapIt(int(it.id * 4 / 2 + 2 - 1))
test "comparison":
let id = 1
let res = query:
select person(id, name, password, email, salt, status)
where id == ?id
check res == [persondata[id - 1]]
test "comparison_ne":
let id = 1
let res = query:
select person(id, name, password, email, salt, status)
where id != ?id
check res == persondata.filterIt(it.id != id)
test "comparison_ge":
let id = 3
let res = query:
select person(id, name, password, email, salt, status)
where id >= ?id
check res == persondata.filterIt(it.id >= id)
test "comparison_le":
let id = 3
let res = query:
select person(id, name, password, email, salt, status)
where id <= ?id
check res == persondata.filterIt(it.id <= id)
test "comparison_gt":
let id = 3
let res = query:
select person(id, name, password, email, salt, status)
where id > ?id
check res == persondata.filterIt(it.id > id)
test "comparison_lt":
let id = 3
let res = query:
select person(id, name, password, email, salt, status)
where id < ?id
check res == persondata.filterIt(it.id < id)
test "logical_not":
let id = 3
let res = query:
select person(id, name, password, email, salt, status)
where not (id > ?id)
check res == persondata.filterIt(it.id <= id)
test "logical_and":
let
id1 = 2
id2 = 4
let res = query:
select person(id, name, password, email, salt, status)
where id >= ?id1 and id <= ?id2
check res == persondata.filterIt(it.id >= id1 and it.id <= id2)
test "logical_or":
let
id1 = 2
id2 = 4
let res = query:
select person(id, name, password, email, salt, status)
where id == ?id1 or id == ?id2
check res == persondata.filterIt(it.id == id1 or it.id == id2)
test "logical_complex":
let
id1 = 2
id2 = 3
id3 = 4
let res = query:
select person(id, name, password, email, salt, status)
where id > ?id2 and (id == ?id1 or id == ?id3)
check res == persondata.filterIt(it.id == id3)
test "predicate_in":
let
id1 = 2
id2 = 4
let res = query:
select person(id, name, password, email, salt, status)
where id in ?id1 .. ?id2
check res == persondata.filterIt(it.id in id1..id2)
test "predicate_not_in":
let
id1 = 2
id2 = 4
let res = query:
select person(id, name, password, email, salt, status)
where id notin ?id1 .. ?id2
check res == persondata.filterIt(it.id < id1 or it.id > id2)
test "predicate_like":
let pattern = "john1%"
let res = query:
select person(id, name)
where name `like` ?pattern
check res == persondata.filterIt(it.name.startsWith("john1")).mapIt((it.id, it.name))
test "predicate_ilike":
let pattern = "JOHN1%"
let res = query:
select person(id, name)
where name `ilike` ?pattern
check res == persondata.filterIt(it.name.startsWith("john1")).mapIt((it.id, it.name))
test "predicate_not_like":
let pattern = "john1%"
let res = query:
select person(id, name)
where not (name `like` ?pattern)
check res == persondata.filterIt(not it.name.startsWith("john1")).mapIt((it.id, it.name))
test "distinct":
var res = query:
select `distinct` post(author)
res.sort()
let expected = postdata.mapIt(it.author).deduplicate().sortedByIt(it)
check res == expected
test "count_distinct":
let res = query:
select post(count(distinct author))
check res == @[postdata.mapIt(it.author).deduplicate().len]
test "window_row_number":
let res = query:
select post(author, id, over(row_number(), partitionby(author), orderby(id)) as rn)
orderby author, id
var expected: seq[(int, int, int)] = @[]
var counters = initCountTable[int]()
for p in postdata.sortedByIt((it.author, it.id)):
counters.inc(p.author)
expected.add((p.author, p.id, counters[p.author]))
check res == expected
test "window_running_sum":
let res = query:
select post(author, id, over(sum(id), partitionby(author), orderby(id)) as running)
orderby author, id
var expected: seq[(int, int, int)] = @[]
var sums = initTable[int, int]()
for p in postdata.sortedByIt((it.author, it.id)):
sums[p.author] = sums.getOrDefault(p.author) + p.id
expected.add((p.author, p.id, sums[p.author]))
check res == expected
test "limit":
let id = 1
let res = query:
select person(id, name, password, email, salt, status)
limit 1
static:
echo "LIMIT TEST: ", typeof(res)
check res == persondata[id - 1]
test "offset":
let res = query:
select person(id, name)
limit 2
offset 2
check res == persondata[2..3].mapIt((it.id, it.name))
test "match_assignment":
let id = 1
let (name, password, email, salt, status) = query:
select person(name, password, email, salt, status)
where id == ?id
limit 1
check:
name == persondata[id - 1].name
password == persondata[id - 1].password
email == persondata[id - 1].email
salt == persondata[id - 1].salt
status == persondata[id - 1].status
test "groupby":
let res = query:
select post(author, count(id))
groupby author
let counttable = postdata.mapIt(it.author).toCountTable()
for (author, c) in res:
check counttable[author] == c
test "groupby_aggregate":
let author = 3
let res = query:
select post(count(id))
where author == ?author
groupby author
let c = postdata.mapIt(it.author).toCountTable()[author]
check res == [c]
test "orderby":
let res = query:
select person(id, name)
orderby id
check res == persondata.mapIt((it.id, it.name)).sortedByIt(it[0])
test "orderby_asc":
let res = query:
select person(id, name)
orderby asc(id)
check res == persondata.mapIt((it.id, it.name)).sortedByIt(it[0])
test "orderby_desc":
let res = query:
select person(id, name)
orderby desc(id)
let expected = persondata.mapIt((it.id, it.name))
.sorted((x, y) => system.cmp(x[0], y[0]), Descending)
check res == expected
test "orderby_key_select":
let res = query:
select person(id as personid, name)
orderby personid
check res == persondata.mapIt((it.id, it.name)).sortedByIt(it[0])
test "orderby_key_not_select":
let res = query:
select person(name)
orderby id
check res == persondata.sortedByIt(it.id).mapIt(it.name)
test "orderby_mulitple":
# test fix #30 Incorrect handling of multiple sort keys in orderby
let res = query:
select post(author, id)
orderby author, desc(id)
check res == postdata.mapIt((it.author, it.id))
.sorted((x, y) => system.cmp(x[1], y[1]), Descending)
.sortedByIt(it[0])
test "having":
let id = 4
let res = query:
select post(author, count(_) as count)
groupby author
having author == ?id
let expected = collect(newSeq):
for p in postdata.mapIt(it.author).toCountTable().pairs():
if p[0] == id: p
check res == expected
test "having_aggregate":
let countvalue = 2
let res = query:
select post(author, count(id) as count)
groupby author
having count(id) >= ?countvalue
let expected = collect(newSeq):
for p in postdata.mapIt(it.author).toCountTable().pairs():
if p[1] >= countvalue: p
check res.sortedByIt(it[0]) == expected.sortedByIt(it[0])
test "having_complex":
let
authorid1 = 1
authorid2 = 3
countvalue = 2
let res = query:
select post(author, count(id) as count)
groupby author
having count(id) >= ?countvalue and (author == ?authorid1 or author == ?authorid2)
let expected = collect(newSeq):
for p in postdata.mapIt(it.author).toCountTable().pairs():
if p[0] in [authorid1, authorid2] and p[1] >= countvalue: p
check res == expected
test "subquery":
let res = query:
select post(id)
where author in
(select person(id))
check res.sortedByIt(it) == postdata.mapIt(it.id)
test "subquery_nest2":
let
personid1 = 1
personid2 = 2
expectedpost = postdata.filterIt(it.author in [personid1, personid2])
.mapIt(it.id)
let res = query:
select post(id)
where author in
(select person(id) where id == ?personid1 or id == ?personid2)
check res.sortedByIt(it) == expectedpost.sortedByIt(it)
test "subquery_nest3":
var res = query:
select thread(id)
where id in (select post(thread) where author in
(select person(id) where id in {1, 2}))
res.sort()
check res == postdata.filterIt(it.author in [1, 2])
.mapIt(it.thread)
.sortedByIt(it)
test "subquery_having":
# feature for having in subquery
let id = 4
let res = query:
select person(id)
where id in (
select post(author) groupby author having author == ?id)
check res == @[id]
test "exists":
let authorid = postdata[0].author
let res = query:
select person(id)
where exists(select post(id) where author == ?authorid)
check res.sortedByIt(it) == persondata.mapIt(it.id)
test "not_exists":
let missingAuthor = personcount + postcount
let res = query:
select person(id)
where not exists(select post(id) where author == ?missingAuthor)
check res.sortedByIt(it) == persondata.mapIt(it.id)
test "with_cte":
let res = query:
with recent(select post(id, author) where id <= 3)
select recent(author)
orderby id
check res == postdata.filterIt(it.id <= 3).sortedByIt(it.id).mapIt(it.author)
test "with_cte_chain":
let res = query:
with recent(select post(id, author) where id <= 3)
with authors(select recent(author))
select authors(author)
check res.sortedByIt(it) == postdata.filterIt(it.id <= 3).mapIt(it.author).sortedByIt(it)
test "with_cte_subquery":
let res = query:
with recent(select post(id, author) where id <= 3)
select person(id)
where id in (select recent(author))
check res.sortedByIt(it) == postdata.filterIt(it.id <= 3).mapIt(it.author).deduplicate().sortedByIt(it)
test "union":
var res = query:
select person(id) where id <= 2
union
select person(id) where id >= 4
res.sort()
check res == @[1, 2, 4, 5]
res = query:
union(
select person(id) where id <= 2,
select person(id) where id >= 4
)
res.sort()
check res == @[1, 2, 4, 5]
test "intersect":
var res = query:
select person(id) where id <= 3
intersect
select person(id) where id >= 2
res.sort()
check res == @[2, 3]
res = query:
intersect(
select person(id) where id <= 3,
select person(id) where id >= 2
)
res.sort()
check res == @[2, 3]
test "except":
var res = query:
select person(id) where id <= 4
`except`
select person(id) where id in {2, 3}
res.sort()
check res == @[1, 4]
res = query:
`except`(
select person(id) where id <= 4,
select person(id) where id in {2, 3}
)
res.sort()
check res == @[1, 4]
test "complex_query_subquery_having":
let
id1 = 2
id2 = 3
let res = query:
select thread(count(_))
where id in (
select post(thread) where (author == ?id1 or author == ?id2) and id in (
select post(min(id)) groupby thread having min(id) > 3))
limit 1
let threadinpost = postdata.mapIt(it.thread).deduplicate().sortedByIt(it)
var postminids: seq[int]
for id in threadinpost:
let group = collect(newSeq):
for p in postdata:
if p.thread == id: p.id
postminids.add(group.min())
let threadids = postdata.filterIt(it.author in [id1, id2] and
it.id in postminids.filterIt(it > 3))
.mapIt(it.thread)
check res == threadids.len()
test "auto_join":
let postid = 1
let res = query:
select post(author)
join person(name)
where id == ?postid
let (author, name) = res[0]
check name == persondata[author - 1].name
test "join_on":
# test fix #29 Cannot handle join on condition correctly
let postid = 1
let res = query:
select post(author)
join person(name) on author == id
where id == ?postid
let (author, name) = res[0]
check name == persondata[author - 1].name
test "leftjoin_on":
let postid = 1
let res = query:
select post(author)
leftjoin person(name) on author == id
where id == ?postid
let (author, name) = res[0]
check name == persondata[author - 1].name
test "leftouterjoin_on":
let postid = 1
let res = query:
select post(author)
leftouterjoin person(name) on author == id
where id == ?postid
let (author, name) = res[0]
check name == persondata[author - 1].name
test "casewhen":
# need more test, only number
# has problem with string or expression in condition
let res = query:
select person(name, (if id == 1: 10 elif id == 2: 20 else: 30) as pid)
check res == persondata.mapIt((
name: it.name,
pid: if it.id == 1: 10 elif it.id == 2: 20 else: 30))
test "produce_json":
# test fix #27 Error: type mismatch: got <typeof(nil)>
let threadjson = query:
select thread(id, name)
produce json
let expected = threaddata.map do (it: tuple) -> JsonNode:
result = newJObject()
result["id"] = %it.id
result["name"] = %it.name
check threadjson == %expected
test "produce_nim":
let threadnim = query:
select thread(id, name)
produce nim
check threadnim == threaddata.mapIt((it.id, it.name))
test "tryquery":
# unknow use, only not raise dbError?
let res = tryQuery:
select thread(id)
check res == threaddata.mapIt(it.id)
test "createproc":
createProc getAllThreadIds:
select thread(id)
check db.getAllThreadIds() == threaddata.mapIt(it.id)
test "createiter":
createIter allThreadIdsIter:
select thread(id)
let res = collect(newSeq):
for id in db.allThreadIdsIter:
id
check res.sortedByIt(it) == threaddata.mapIt(it.id)
test "update_concat_op":
let
id = 3
name = persondata[id - 1].name
appendstr = "updated"
nameupdated = name & appendstr
query:
update person(name = name & ?appendstr)
where id == ?id
let res = query:
select person(name)
where id == ?id
check res[0] == nameupdated
test "delete_one":
# test fix #14: delete not return value
let id = 1
query:
delete antibot
where id == ?id
let res = query:
select antibot(_)
where id == ?id
check res == []
test "delete_all":
query:
delete antibot
let res = query:
select antibot(_)
check res == []
test "insert_return_id":
# test fix #28 returning id fail under postgresql
let expectedid = 6
let id = query:
insert antibot(id = ?expectedid, ip = "", answer = "just insert")
returning id
check id == expectedid
test "insert_return_answer":
# test returning non-id parameter
let expectedanswer = "just insert another"
let answer = query:
insert antibot(id = 9, ip = "", answer = ?expectedanswer)
returning answer
check answer == expectedanswer
test "insert_return_id_auto":
# test returning id column
when defined(postgre):
# fix postgres sequence so next nextval returns 10
discard db.getValue(sql"select setval('antibot_id_seq', ?, ?)", 10, false)
let answer = query:
insert antibot(ip = "", answer = "just auto insert")
returning id
check answer == 10
test "insert_returning_uuid":
# test returning uuid column
let expecteduuid = "123e4567-e89b-12d3-a456-426614174000"
let uuid = query:
insert error(uuid = ?expecteduuid, message = "just insert")
returning uuid
check uuid == expecteduuid
test "where_json":
let
id = 1
p = %*{"id": id}
let res = query:
select person(id)
where id == %p["id"]
check res == [id]
test "insert_json":
let
id = 7
answer = "json answer"
a = %*{"answer": answer}
query:
insert antibot(id = ?id, ip = "", answer = %a["answer"])
let res = query:
select antibot(answer)
where id == ?id
check res[0] == answer
test "update_json":
let
id = 3
name = "json"
p = %*{"name": name}
query:
update person(name = %p["name"])
where id == ?id
let res = query:
select person(name)
where id == ?id
check res[0] == name
test "insert_rawsql":
let id = 8
query:
insert antibot(id = ?id, ip = "", answer = !!"'raw sql'",
created = !!"CURRENT_TIMESTAMP")
let res = query:
select antibot(_)
where id == ?id
check res[0].answer == "raw sql"
test "update_rawsql":
let id = 3
let res = query:
select person(name)
where id == ?id
query:
update person(name = !!"UPPER(name)")
where id == ?id
let res2 = query:
select person(name)
where id == ?id
check res[0] != res2[0]
check res[0].toUpper() == res2[0]
test "empty_string":
db.exec(insertthread, 6, "", 77)
createProc getAllThreads:
select thread(id, name, views)
where id >= 5
let rows = db.getAllThreads
check rows[1].id == 6
check rows[1].views == 77
check rows[1].name == ""
+29
View File
@@ -0,0 +1,29 @@
import unittest
import ormin/ormin_sqlite as orm_sqlite
from db_connector/db_sqlite import exec, getValue
import ormin
importModel(DbBackend.sqlite, "static_only_model", includeStatic = true)
let db {.global.} = orm_sqlite.open(":memory:", "", "", "")
suite "importModel includeStatic":
test "sqlSchema is generated and usable without a generated Nim model":
let schema: DbSql = sqlSchema
check ($schema).len > 0
db.createTable(sqlSchema)
db.exec(sql"insert into user_static (id, name) values (?, ?)", 1, "Ada")
let row = query:
select user_static(id, name)
where id == 1
limit 1
check row.id == 1
check row.name == "Ada"
check db.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'user_static'") == "1"
db.dropTable(sqlSchema, "user_static")
check db.getValue(sql"select count(*) from sqlite_master where type='table' and name = 'user_static'") == "0"
+130
View File
@@ -0,0 +1,130 @@
import unittest, json, strutils, macros, times, os, sequtils
# Postgres connection handled through ormin_postgre backend
from db_connector/db_postgres import exec, getValue
import ormin
import ormin/ormin_postgre as ormin_postgre
# import db_connector/db_postgres as db_postgres
import ormin/db_utils
when defined(macosx):
{.passL: "-Wl,-rpath,/opt/homebrew/lib/postgresql@14".}
importModel(DbBackend.postgre, "model_postgre")
let
db {.global.} = ormin_postgre.open("localhost", "test", "test", "test_ormin")
testDir = currentSourcePath.parentDir()
sqlFile = Path(testDir / "model_postgre.sql")
suite "Test special database types and functions of postgre":
discard
let
dtStr1 = "2018-03-30 01:01:01"
dt1 = parse(dtStr1, "yyyy-MM-dd HH:mm:ss")
dtnStr1 = "2019-03-30 11:02:02+08"
dtn1 = parse(dtnStr1, "yyyy-MM-dd HH:mm:sszz")
dtzStr1 = "2020-03-30 09:03:03Z"
dtz1 = parse(dtzStr1, "yyyy-MM-dd HH:mm:sszz")
dtStr2 = "2018-03-30 01:01:01.123"
dt2 = parse(dtStr2, "yyyy-MM-dd HH:mm:ss\'.\'fff")
dtnStr2 = "2019-03-30 11:02:02.123+08"
dtn2 = parse(dtnStr2, "yyyy-MM-dd HH:mm:ss\'.\'fffzz")
dtzStr2 = "2020-03-30 09:03:03.123Z"
dtz2 = parse(dtzStr2, "yyyy-MM-dd HH:mm:ss\'.\'fffzz")
dtStr3 = "2018-03-30 01:01:01.123456"
dt3 = parse(dtStr3, "yyyy-MM-dd HH:mm:ss\'.\'ffffff")
dtnStr3 = "2019-03-30 11:02:02.123456+08"
dtn3 = parse(dtnStr3, "yyyy-MM-dd HH:mm:ss\'.\'ffffffzz")
dtzStr3 = "2020-03-30 09:03:03.123456Z"
dtz3 = parse(dtzStr3, "yyyy-MM-dd HH:mm:ss\'.\'ffffffzz")
dtjson1 = %*{"dt": dt1.format(jsonTimeFormat),
"dtn": dtn1.format(jsonTimeFormat),
"dtz": dtz1.format(jsonTimeFormat)}
dtjson2 = %*{"dt": dt2.format(jsonTimeFormat),
"dtn": dtn2.format(jsonTimeFormat),
"dtz": dtz2.format(jsonTimeFormat)}
dtjson3 = %*{"dt": dt3.format(jsonTimeFormat),
"dtn": dtn3.format(jsonTimeFormat),
"dtz": dtz3.format(jsonTimeFormat)}
let insertSql = sql"insert into tb_timestamp(dt, dtn, dtz) values (?, ?, ?)"
suite "timestamp_insert":
setup:
db.dropTable(sqlFile, "tb_timestamp")
db.createTable(sqlFile, "tb_timestamp")
test "insert":
query:
insert tb_timestamp(dt = ?dt1, dtn = ?dtn1, dtz = ?dtz1)
check db_postgres.getValue(db, sql"select count(*) from tb_timestamp") == "1"
test "json":
query:
insert tb_timestamp(dt = %dtjson1["dt"],
dtn = %dtjson1["dtn"],
dtz = %dtjson1["dtz"])
check db_postgres.getValue(db, sql"select count(*) from tb_timestamp") == "1"
suite "timestamp":
db.dropTable(sqlFile, "tb_timestamp")
db.createTable(sqlFile, "tb_timestamp")
db_postgres.exec(db, insertSql, dtStr1, dtnStr1, dtzStr1)
db_postgres.exec(db, insertSql, dtStr2, dtnStr2, dtzStr2)
db_postgres.exec(db, insertSql, dtStr3, dtnStr3, dtzStr3)
doAssert db_postgres.getValue(db, sql"select count(*) from tb_timestamp") == "3"
test "query":
let res = query:
select tb_timestamp(dt, dtn, dtz)
check res == [(dt1, dtn1, dtz1),
(dt2, dtn2, dtz2),
(dt3, dtn3, dtz3)]
test "where":
let res = query:
select tb_timestamp(dt)
where dt == ?dt1
check res == [dt1]
test "in":
let
duration = initDuration(hours = 1)
dtStart = dt1 - duration
dtEnd = dt1 + duration
res2 = query:
select tb_timestamp(dt)
where dt in ?dtStart .. ?dtEnd
check res2 == [dt1, dt2, dt3]
test "iter":
createIter iter:
select tb_timestamp(dt)
where dt == ?dt
var res: seq[DateTime]
for it in db.iter(dt1):
res.add(it)
check res == [dt1]
test "proc":
createProc aproc:
select tb_timestamp(dt)
where dt == ?dt
check db.aproc(dt1) == [dt1]
test "json":
let res = query:
select tb_timestamp(dt, dtn, dtz)
produce json
check res == %*[dtjson1, dtjson2, dtjson3]
test "json_where":
let res = query:
select tb_timestamp(dt, dtn, dtz)
where dt == %dtjson1["dt"]
produce json
check res[0] == dtjson1
+216
View File
@@ -0,0 +1,216 @@
import unittest, strformat, os, times, std/monotimes
import std/options
import ormin
import ormin/db_utils
import ormin/query_hooks
when defined(postgre):
when defined(macosx):
{.passL: "-Wl,-rpath,/opt/homebrew/lib/postgresql@14".}
const backend = DbBackend.postgre
importModel(backend, "model_postgre")
const sqlFileName = "model_postgre.sql"
let db {.global.} = open("localhost", "test", "test", "test_ormin")
else:
const backend = DbBackend.sqlite
importModel(backend, "model_sqlite")
const sqlFileName = "model_sqlite.sql"
let db {.global.} = open("test.db", "", "", "")
let
testDir = currentSourcePath.parentDir()
sqlFile = Path(testDir / sqlFileName)
type
CompositeRow = object
id: int
message: string
RefCompositeRow = ref object
id: int
message: string
BenchmarkCompositeRow = object
pk1: int
message: string
NullableNoteOptionRow = object
id: int
note: Option[string]
MessageSize = distinct int
HookedMessageRow = object
id: int
message: MessageSize
NullFallbackNote = distinct string
HookedNullableNoteRow = object
id: int
note: NullFallbackNote
const
benchmarkRowCount = 256
benchmarkWarmupIterations = 75
benchmarkIterations = 250
benchmarkRounds = 5
maxTypedQuerySlowdown = 1.20
proc fromQueryHook*(val: var MessageSize, value: string) =
val = MessageSize(value.len)
proc fromQueryHook*(val: var NullFallbackNote, value: DbValue[string]) =
if value.isNull:
val = NullFallbackNote("<missing>")
else:
val = NullFallbackNote("note:" & value.value)
proc loadBenchmarkRows() =
db.dropTable(sqlFile, "tb_composite_pk")
db.createTable(sqlFile, "tb_composite_pk")
for i in 1 .. benchmarkRowCount:
let message = &"message-{i}"
query:
insert tb_composite_pk(pk1 = ?i, pk2 = ?i, message = ?message)
proc benchmarkCurrentQuery(iterations: int): float =
var checksum = 0
let started = getMonoTime()
for _ in 0 ..< iterations:
let rows = query:
select tb_composite_pk(pk1, message)
orderby pk1
checksum += rows.len + rows[^1][0] + rows[^1][1].len
doAssert checksum > 0
result = (getMonoTime() - started).inNanoseconds.float / 1_000_000_000.0
proc benchmarkTypedQuery(iterations: int): float =
var checksum = 0
let started = getMonoTime()
for _ in 0 ..< iterations:
let rows = query(BenchmarkCompositeRow):
select tb_composite_pk(pk1, message)
orderby pk1
checksum += rows.len + rows[^1].pk1 + rows[^1].message.len
doAssert checksum > 0
result = (getMonoTime() - started).inNanoseconds.float / 1_000_000_000.0
suite &"query(T) mapping on {backend}":
setup:
db.dropTable(sqlFile, "tb_composite_pk")
db.createTable(sqlFile, "tb_composite_pk")
db.dropTable(sqlFile, "tb_nullable")
db.createTable(sqlFile, "tb_nullable")
query:
insert tb_composite_pk(pk1 = 1, pk2 = 1, message = "hello")
query:
insert tb_composite_pk(pk1 = 2, pk2 = 2, message = "world")
query:
insert tb_nullable(id = 1, note = nil)
query:
insert tb_nullable(id = 2, note = "hello")
test "maps selected rows to objects":
let rows = query(CompositeRow):
select tb_composite_pk(pk1 as id, message)
orderby pk1
check rows == @[
CompositeRow(id: 1, message: "hello"),
CompositeRow(id: 2, message: "world")
]
test "maps selected rows to ref objects":
let rows = query(RefCompositeRow):
select tb_composite_pk(pk1 as id, message)
orderby pk1
check rows.len == 2
check rows[0] != nil
check rows[0].id == 1
check rows[0].message == "hello"
check rows[1] != nil
check rows[1].id == 2
check rows[1].message == "world"
test "single-row query(T) returns a single object":
let row = query(CompositeRow):
select tb_composite_pk(pk1 as id, message)
where pk1 == 1 and pk2 == 1
limit 1
check row == CompositeRow(id: 1, message: "hello")
test "maps nullable column to Option":
let rows = query(NullableNoteOptionRow):
select tb_nullable(id, note)
orderby id
check rows[0].id == 1
check rows[0].note.isNone
check rows[1].id == 2
check rows[1].note.isSome
check rows[1].note.get == "hello"
test "maps object fields through user fromQueryHook overloads":
let rows = query(HookedMessageRow):
select tb_composite_pk(pk1 as id, message)
orderby pk1
check rows.len == 2
check rows[0].id == 1
check int(rows[0].message) == "hello".len
check rows[1].id == 2
check int(rows[1].message) == "world".len
test "maps scalar query(T) through user fromQueryHook overloads":
let values = query(MessageSize):
select tb_composite_pk(message)
orderby pk1
check values.len == 2
check int(values[0]) == "hello".len
check int(values[1]) == "world".len
test "allows user fromQueryHook overloads to handle NULL values":
let rows = query(HookedNullableNoteRow):
select tb_nullable(id, note)
orderby id
check rows.len == 2
check rows[0].id == 1
check string(rows[0].note) == "<missing>"
check rows[1].id == 2
check string(rows[1].note) == "note:hello"
test "sqlite benchmark for query and query(T)":
loadBenchmarkRows()
let untypedRows = query:
select tb_composite_pk(pk1, message)
orderby pk1
let typedRows = query(BenchmarkCompositeRow):
select tb_composite_pk(pk1, message)
orderby pk1
check untypedRows.len == typedRows.len
check typedRows[0] == BenchmarkCompositeRow(pk1: untypedRows[0][0], message: untypedRows[0][1])
check typedRows[^1] == BenchmarkCompositeRow(pk1: untypedRows[^1][0], message: untypedRows[^1][1])
discard benchmarkCurrentQuery(benchmarkWarmupIterations)
discard benchmarkTypedQuery(benchmarkWarmupIterations)
var currentBest = high(float)
var typedBest = high(float)
for _ in 0 ..< benchmarkRounds:
currentBest = min(currentBest, benchmarkCurrentQuery(benchmarkIterations))
typedBest = min(typedBest, benchmarkTypedQuery(benchmarkIterations))
let ratio = typedBest / currentBest
echo &"sqlite benchmark query={currentBest:.6f}s query(T)={typedBest:.6f}s ratio={ratio:.3f}x; 20% budget={(ratio <= maxTypedQuerySlowdown)}"
check currentBest > 0.0
check typedBest > 0.0
when defined(release):
check ratio <= maxTypedQuerySlowdown
+153
View File
@@ -0,0 +1,153 @@
import unittest, db_connector/sqlite3, json, times, sequtils
from db_connector/db_sqlite import exec, getValue
import ormin
import ormin/db_utils
importModel(DbBackend.sqlite, "model_sqlite")
let
db {.global.} = open(":memory:", "", "", "")
testDir = currentSourcePath.parentDir()
sqlFile = Path(testDir / "model_sqlite.sql")
suite "Test special database types and functions of sqlite":
discard
jsonTimeFormat = "yyyy-MM-dd HH:mm:ss\'.\'fff"
let
dtStr1 = "2018-02-20 02:02:02"
dt1 = parse(dtStr1, "yyyy-MM-dd HH:mm:ss", utc())
dtStr2 = "2019-03-30 03:03:03.123"
dt2 = parse(dtStr2, "yyyy-MM-dd HH:mm:ss\'.\'fff", utc())
dtjson = %*{"dt1": dt1.format(jsonTimeFormat),
"dt2": dt2.format(jsonTimeFormat)}
let insertSql = sql"insert into tb_timestamp(dt1, dt2) values (?, ?)"
proc blobFromBytes(bytes: openArray[int]): seq[byte] =
result = newSeq[byte](bytes.len)
for i, b in bytes:
doAssert b >= 0 and b <= 255
result[i] = byte(b)
let blobFixtures = [
blobFromBytes(@[0, 1, 2, 3, 4, 5, 6, 7]),
blobFromBytes(@[255, 128, 64, 32, 16, 8, 4, 2, 1]),
blobFromBytes(@[ord('O'), ord('R'), ord('M'), ord('I'), ord('N'), 0, 1, 2])
]
suite "timestamp_insert":
setup:
db.dropTable(sqlFile, "tb_timestamp")
db.createTable(sqlFile, "tb_timestamp")
test "insert":
query:
insert tb_timestamp(dt1 = ?dt1, dt2 = ?dt2)
check db.getValue(sql"select count(*) from tb_timestamp") == "1"
test "json":
query:
insert tb_timestamp(dt1 = %dtjson["dt1"], dt2 = %dtjson["dt2"])
check db.getValue(sql"select count(*) from tb_timestamp") == "1"
suite "timestamp":
db.dropTable(sqlFile, "tb_timestamp")
db.createTable(sqlFile, "tb_timestamp")
db.exec(insertSql, dtStr1, dtStr2)
doAssert db.getValue(sql"select count(*) from tb_timestamp") == "1"
test "query":
let res = query:
select tb_timestamp(dt1, dt2)
check res == [(dt1, dt2)]
test "where":
let res = query:
select tb_timestamp(dt1, dt2)
where dt1 == ?dt1
check res == [(dt1, dt2)]
test "in":
let
duration = initDuration(hours = 1)
dtStart = dt1 - duration
dtEnd = dt1 + duration
res2 = query:
select tb_timestamp(dt1, dt2)
where dt1 in ?dtStart .. ?dtEnd
check res2 == [(dt1, dt2)]
test "iter":
createIter iter:
select tb_timestamp(dt1, dt2)
where dt1 == ?dt1
var res: seq[tuple[dt1: DateTime, dt2: DateTime]]
for it in db.iter(dt1):
res.add(it)
check res == [(dt1, dt2)]
test "proc":
createProc aproc:
select tb_timestamp(dt1, dt2)
where dt1 == ?dt1
check db.aproc(dt1) == [(dt1, dt2)]
test "json":
let res = query:
select tb_timestamp(dt1, dt2)
produce json
check res == %*[dtjson]
test "json_where":
let res = query:
select tb_timestamp(dt1, dt2)
where dt1 == %dtjson["dt1"]
produce json
check res == %*[dtjson]
suite "blob_insert":
setup:
db.dropTable(sqlFile, "tb_blob")
db.createTable(sqlFile, "tb_blob")
test "insert parameters":
for blob in blobFixtures:
query:
insert tb_blob(typblob = ?blob)
check db.getValue(sql"select count(*) from tb_blob") == $blobFixtures.len
suite "blob":
db.dropTable(sqlFile, "tb_blob")
db.createTable(sqlFile, "tb_blob")
for blob in blobFixtures:
query:
insert tb_blob(typblob = ?blob)
doAssert db.getValue(sql"select count(*) from tb_blob") == $blobFixtures.len
test "query":
let res = query:
select tb_blob(id, typblob)
check res.mapIt(it.typblob) == blobFixtures
test "where":
let target = blobFixtures[1]
let res = query:
select tb_blob(id, typblob)
where typblob == ?target
check res.len == 1
check res[0].typblob == target
createProc selectAllBlob:
select tb_blob(id, typblob)
test "createProc":
let res = db.selectAllBlob
check res.mapIt(it.typblob) == blobFixtures
test "createProc with empty blob in result":
db.exec(sql"insert into tb_blob(id, typblob) values (?, ?)", 9, "")
let res = db.selectAllBlob
check res[^1].typblob == blobFromBytes(@[])
+132
View File
@@ -0,0 +1,132 @@
import unittest, os, strformat
import ormin
import ormin/db_utils
when NimVersion < "1.2.0": import ./compat
let testDir = currentSourcePath.parentDir()
when defined postgre:
when defined(macosx):
{.passL: "-Wl,-rpath,/opt/homebrew/lib/postgresql@14".}
from db_connector/db_postgres import exec, getValue
const backend = DbBackend.postgre
importModel(backend, "forum_model_postgres")
const sqlFileName = "forum_model_postgres.sql"
let db {.global.} = open("localhost", "test", "test", "test_ormin")
else:
from db_connector/db_sqlite import exec, getValue
const backend = DbBackend.sqlite
importModel(backend, "forum_model_sqlite")
const sqlFileName = "forum_model_sqlite.sql"
var memoryPath = testDir & "/" & ":memory:"
let db {.global.} = open(memoryPath, "", "", "")
var sqlFilePath = Path(testDir & "/" & sqlFileName)
# Fresh schema
db.dropTable(sqlFilePath)
db.createTable(sqlFilePath)
suite &"Transactions ({backend})":
test "commit on success":
transaction:
query:
insert person(id = ?(101), name = ?"john101", password = ?"p101", email = ?"john101@mail.com", salt = ?"s101", status = ?"ok")
check db.getValue(sql"select count(*) from person where id = 101") == "1"
test "rollback on error with manual try except":
# prepare one row
query:
insert person(id = ?(201), name = ?"john201", password = ?"p201", email = ?"john201@mail.com", salt = ?"s201", status = ?"ok")
# in transaction insert a new row and then violate PK
try:
transaction:
query:
insert person(id = ?(202), name = ?"john202", password = ?"p202", email = ?"john202@mail.com", salt = ?"s202", status = ?"ok")
# duplicate key error
query:
insert person(id = ?(201), name = ?"dup", password = ?"p", email = ?"e", salt = ?"s", status = ?"x")
check false # should not reach
except DbError as e:
discard
# both inserts inside the transaction should be rolled back
check db.getValue(sql"select count(*) from person where id = 202") == "0"
check db.getValue(sql"select count(*) from person where id = 201 and name = 'dup'") == "0"
test "rollback on error with else":
# prepare one row
var failed = false
query:
insert person(id = ?(501), name = ?"john501", password = ?"p501", email = ?"john501@mail.com", salt = ?"s501", status = ?"ok")
# in transaction insert a new row and then violate PK
transaction:
query:
insert person(id = ?(502), name = ?"john502", password = ?"p502", email = ?"john502@mail.com", salt = ?"s502", status = ?"ok")
# duplicate key error
query:
insert person(id = ?(501), name = ?"dup", password = ?"p", email = ?"e", salt = ?"s", status = ?"x")
check false # should not reach
else:
echo "do something else..."
failed = true
check failed
# both inserts inside the transaction should be rolled back
check db.getValue(sql"select count(*) from person where id = 502") == "0"
check db.getValue(sql"select count(*) from person where id = 501 and name = 'dup'") == "0"
test "commit normally with else":
# prepare one row
var failed = false
query:
insert person(id = ?(601), name = ?"john601", password = ?"p601", email = ?"john601@mail.com", salt = ?"s601", status = ?"ok")
# in transaction insert a new row and then violate PK
transaction:
query:
insert person(id = ?(602), name = ?"john602", password = ?"p602", email = ?"john602@mail.com", salt = ?"s602", status = ?"ok")
query:
insert person(id = ?(603), name = ?"dup", password = ?"p", email = ?"e", salt = ?"s", status = ?"x")
else:
failed = true
check not failed
# both inserts inside the transaction should be rolled back
check db.getValue(sql"select count(*) from person where id = 603") == "1"
check db.getValue(sql"select count(*) from person where id = 602") == "1"
check db.getValue(sql"select count(*) from person where id = 601") == "1"
test "transaction set false on DbError":
var failed = false
transaction:
query:
insert person(id = ?(301), name = ?"john301", password = ?"p301", email = ?"john301@mail.com", salt = ?"s301", status = ?"ok")
query:
insert person(id = ?(301), name = ?"dup", password = ?"p", email = ?"e", salt = ?"s", status = ?"x")
else:
failed = true
check failed
check db.getValue(sql"select count(*) from person where id = 301") == "0"
test "nested savepoints":
var failed = false
transaction:
query:
insert person(id = ?(401), name = ?"john401", password = ?"p401", email = ?"john401@mail.com", salt = ?"s401", status = ?"ok")
var innerOk = true
transaction:
query:
insert person(id = ?(402), name = ?"john402", password = ?"p402", email = ?"john402@mail.com", salt = ?"s402", status = ?"ok")
query:
insert person(id = ?(401), name = ?"dup401", password = ?"p", email = ?"e", salt = ?"s", status = ?"x")
else:
innerOk = false
check innerOk == false
# after inner rollback, we can still insert another row and commit outer
query:
insert person(id = ?(403), name = ?"john403", password = ?"p403", email = ?"john403@mail.com", salt = ?"s403", status = ?"ok")
else:
failed = true
check db.getValue(sql"select count(*) from person where id in (401,402,403)") == "2"
@@ -0,0 +1,38 @@
## Ormin -- ORM for Nim.
## (c) 2017 Andreas Rumpf
## MIT License.
import strutils, os, parseopt
import ../ormin/importer_core
proc writeHelp() =
echo """
ormin <schema.sql> --out:<file.nim> --db:postgre|sqlite|mysql
"""
proc writeVersion() = echo "v1.0"
var p = initOptParser()
var infile = ""
var outfile = ""
var target: ImportTarget
for kind, key, val in p.getopt():
case kind
of cmdArgument:
infile = key
of cmdLongOption, cmdShortOption:
case key
of "help", "h": writeHelp()
of "version", "v": writeVersion()
of "out", "o": outfile = val
of "db": target = parseEnum[ImportTarget](val)
else: discard
of cmdEnd: assert(false) # cannot happen
if infile == "":
# no filename has been given, so we show the help:
writeHelp()
else:
if outfile == "":
outfile = changeFileExt(infile, "nim")
writeFile(outfile, generateModelCode(readFile(infile), absolutePath(infile), target))
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
set -euo pipefail
# Try default connection first, then fallback to -U postgres
run_psql_file() {
local db="$1"; shift
local file="$1"; shift
if ! psql -v ON_ERROR_STOP=1 -d "$db" -f "$file" "$@" >/dev/null 2>&1; then
psql -v ON_ERROR_STOP=1 -U postgres -d "$db" -f "$file" "$@"
fi
}
run_psql_cmd() {
local db="$1"; shift
local cmd="$1"; shift
if ! psql -v ON_ERROR_STOP=1 -d "$db" -c "$cmd" "$@" >/dev/null 2>&1; then
psql -v ON_ERROR_STOP=1 -U postgres -d "$db" -c "$cmd" "$@"
fi
}
# Create role 'test' if needed
run_psql_file postgres tools/setup_postgres_role.sql
# Create database 'test' if needed (must be outside DO block)
if ! psql -v ON_ERROR_STOP=1 -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = 'test_ormin'" | grep -q 1; then
run_psql_cmd postgres "CREATE DATABASE test_ormin OWNER test"
fi
# Grant privileges on public schema
run_psql_cmd test_ormin "GRANT ALL PRIVILEGES ON SCHEMA public TO test"
echo "Postgres test DB/user ensured (role 'test', db 'test_ormin')."
@@ -0,0 +1,21 @@
DO
$$
BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'test') THEN
CREATE ROLE test LOGIN PASSWORD 'test';
END IF;
END
$$;
DO
$$
BEGIN
IF NOT EXISTS (SELECT FROM pg_database WHERE datname = 'test') THEN
CREATE DATABASE test OWNER test;
END IF;
END
$$;
\connect test
GRANT ALL PRIVILEGES ON SCHEMA public TO test;
@@ -0,0 +1,9 @@
DO
$$
BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'test') THEN
CREATE ROLE test LOGIN PASSWORD 'test';
END IF;
END
$$;
+26 -12
View File
@@ -108,6 +108,16 @@ proc readString(buf: openArray[byte], pos: var int): string =
result[i] = char(buf[pos + i]) result[i] = char(buf[pos + i])
pos += len pos += len
proc toBytes(s: string): seq[byte] =
result = newSeq[byte](s.len)
for i, c in s:
result[i] = byte(c)
proc toString(s: seq[byte]): string =
result = newString(s.len)
for i, b in s:
result[i] = char(b)
proc serializeValue*(buf: var seq[byte], val: WireValue) = proc serializeValue*(buf: var seq[byte], val: WireValue) =
buf.add(byte(val.kind)) buf.add(byte(val.kind))
case val.kind case val.kind
@@ -289,7 +299,7 @@ proc close*(client: BaraClient) =
if client.connected: if client.connected:
try: try:
let msg = buildMessage(mkClose, client.nextId(), @[]) let msg = buildMessage(mkClose, client.nextId(), @[])
waitFor client.socket.send(cast[string](msg)) waitFor client.socket.send(toString(msg))
except: discard except: discard
client.socket.close() client.socket.close()
client.connected = false client.connected = false
@@ -319,13 +329,13 @@ proc readQueryResponse(client: BaraClient): Future[QueryResult] {.async.} =
raise newException(IOError, "Connection closed") raise newException(IOError, "Connection closed")
var pos = 0 var pos = 0
let hdrData = cast[seq[byte]](headerData) let hdrData = toBytes(headerData)
let kind = MsgKind(readUint32(hdrData, pos)) let kind = MsgKind(readUint32(hdrData, pos))
let payloadLen = int(readUint32(hdrData, pos)) let payloadLen = int(readUint32(hdrData, pos))
discard readUint32(hdrData, pos) discard readUint32(hdrData, pos)
let payloadStr = await client.socket.recv(payloadLen) let payloadStr = await client.socket.recv(payloadLen)
var payload = cast[seq[byte]](payloadStr) var payload = toBytes(payloadStr)
result = QueryResult(columns: @[], rows: @[], rowCount: 0, affectedRows: 0) result = QueryResult(columns: @[], rows: @[], rowCount: 0, affectedRows: 0)
@@ -360,14 +370,14 @@ proc readQueryResponse(client: BaraClient): Future[QueryResult] {.async.} =
let compHeader = await client.socket.recv(12) let compHeader = await client.socket.recv(12)
if compHeader.len >= 12: if compHeader.len >= 12:
var chPos = 0 var chPos = 0
let chData = cast[seq[byte]](compHeader) let chData = toBytes(compHeader)
let compKind = MsgKind(readUint32(chData, chPos)) let compKind = MsgKind(readUint32(chData, chPos))
let compLen = int(readUint32(chData, chPos)) let compLen = int(readUint32(chData, chPos))
discard readUint32(chData, chPos) discard readUint32(chData, chPos)
let compPayloadStr = await client.socket.recv(compLen) let compPayloadStr = await client.socket.recv(compLen)
if compKind == mkComplete: if compKind == mkComplete:
var cpPos = 0 var cpPos = 0
result.affectedRows = int(readUint32(cast[seq[byte]](compPayloadStr), cpPos)) result.affectedRows = int(readUint32(toBytes(compPayloadStr), cpPos))
return return
if kind == mkComplete: if kind == mkComplete:
var rpos = 0 var rpos = 0
@@ -379,7 +389,8 @@ proc query*(client: BaraClient, sql: string): Future[QueryResult] {.async.} =
raise newException(IOError, "Not connected") raise newException(IOError, "Not connected")
let msg = makeQueryMessage(client.nextId(), sql) let msg = makeQueryMessage(client.nextId(), sql)
await client.socket.send(cast[string](msg)) let msgStr = toString(msg)
await client.socket.send(msgStr)
return await client.readQueryResponse() return await client.readQueryResponse()
@@ -388,7 +399,8 @@ proc query*(client: BaraClient, sql: string, params: seq[WireValue]): Future[Que
raise newException(IOError, "Not connected") raise newException(IOError, "Not connected")
let msg = makeQueryParamsMessage(client.nextId(), sql, params) let msg = makeQueryParamsMessage(client.nextId(), sql, params)
await client.socket.send(cast[string](msg)) let msgStr = toString(msg)
await client.socket.send(msgStr)
return await client.readQueryResponse() return await client.readQueryResponse()
@@ -401,14 +413,15 @@ proc auth*(client: BaraClient, token: string) {.async.} =
raise newException(IOError, "Not connected") raise newException(IOError, "Not connected")
let msg = makeAuthMessage(client.nextId(), token) let msg = makeAuthMessage(client.nextId(), token)
await client.socket.send(cast[string](msg)) let msgStr = toString(msg)
await client.socket.send(msgStr)
let headerData = await client.socket.recv(12) let headerData = await client.socket.recv(12)
if headerData.len < 12: if headerData.len < 12:
raise newException(IOError, "Connection closed") raise newException(IOError, "Connection closed")
var pos = 0 var pos = 0
let hdrData = cast[seq[byte]](headerData) let hdrData = toBytes(headerData)
let kind = MsgKind(readUint32(hdrData, pos)) let kind = MsgKind(readUint32(hdrData, pos))
let payloadLen = int(readUint32(hdrData, pos)) let payloadLen = int(readUint32(hdrData, pos))
discard readUint32(hdrData, pos) discard readUint32(hdrData, pos)
@@ -418,7 +431,7 @@ proc auth*(client: BaraClient, token: string) {.async.} =
elif kind == mkError: elif kind == mkError:
let payloadStr = await client.socket.recv(payloadLen) let payloadStr = await client.socket.recv(payloadLen)
var epos = 0 var epos = 0
let emsg = readString(cast[seq[byte]](payloadStr), epos) let emsg = readString(toBytes(payloadStr), epos)
raise newException(IOError, "Auth failed: " & emsg) raise newException(IOError, "Auth failed: " & emsg)
else: else:
raise newException(IOError, "Unexpected auth response: 0x" & toHex(uint32(kind), 2)) raise newException(IOError, "Unexpected auth response: 0x" & toHex(uint32(kind), 2))
@@ -427,14 +440,15 @@ proc ping*(client: BaraClient): Future[bool] {.async.} =
if not client.connected: if not client.connected:
return false return false
let msg = buildMessage(mkPing, client.nextId(), @[]) let msg = buildMessage(mkPing, client.nextId(), @[])
await client.socket.send(cast[string](msg)) let msgStr = toString(msg)
await client.socket.send(msgStr)
let headerData = await client.socket.recv(12) let headerData = await client.socket.recv(12)
if headerData.len < 12: if headerData.len < 12:
return false return false
var pos = 0 var pos = 0
let hdrData = cast[seq[byte]](headerData) let hdrData = toBytes(headerData)
let kind = MsgKind(readUint32(hdrData, pos)) let kind = MsgKind(readUint32(hdrData, pos))
return kind == mkPong return kind == mkPong
+5 -3
View File
@@ -1,15 +1,17 @@
## BaraDB Nim Client — Integration Tests ## BaraDB Nim Client — Integration Tests
## Requires a running BaraDB server on localhost:9472. ## Requires a running BaraDB server.
## Set BARADB_HOST / BARADB_PORT env vars to override defaults.
import std/unittest import std/unittest
import std/asyncdispatch import std/asyncdispatch
import std/asyncnet import std/asyncnet
import std/strutils import std/strutils
import std/os
import baradb/client import baradb/client
const const
TestHost = "127.0.0.1" TestHost = getEnv("BARADB_HOST", "127.0.0.1")
TestPort = 9472 TestPort = parseInt(getEnv("BARADB_PORT", "9472"))
proc serverAvailable(): bool = proc serverAvailable(): bool =
try: try:
+50 -28
View File
@@ -1,15 +1,16 @@
# BaraDB Python Client # BaraDB Python Async Client
Official Python client for **BaraDB** — a multimodal database engine written in Nim. Official async Python client for **BaraDB** — a multimodal database engine written in Nim.
## Features ## Features
- **Async/await** — fully non-blocking, concurrent query support
- **Binary wire protocol** — fast, compact TCP communication - **Binary wire protocol** — fast, compact TCP communication
- **Sync & blocking API** — simple to use in scripts and apps - **Request queueing** — sequential processing with concurrent execution
- **Query builder** — fluent SQL construction - **Query builder** — fluent SQL construction
- **Parameterized queries** — safe from SQL injection - **Parameterized queries** — safe from SQL injection
- **Vector & JSON support** — first-class multimodal types - **Vector & JSON support** — first-class multimodal types
- **Context managers** — `with` statement support - **Context managers** — async `async with` statement support
## Installation ## Installation
@@ -20,7 +21,7 @@ pip install baradb
Or from source: Or from source:
```bash ```bash
git clone https://github.com/barabadb/baradadb.git git clone https://codeberg.org/baraba/baradb
cd clients/python cd clients/python
pip install -e ".[dev]" pip install -e ".[dev]"
``` ```
@@ -28,48 +29,62 @@ pip install -e ".[dev]"
## Quick Start ## Quick Start
```python ```python
import asyncio
from baradb import Client from baradb import Client
client = Client("localhost", 9472) async def main():
client.connect() client = Client("localhost", 9472)
await client.connect()
result = client.query("SELECT name, age FROM users WHERE age > 18") result = await client.query("SELECT name, age FROM users WHERE age > 18")
for row in result: for row in result:
print(row["name"], row["age"]) print(row["name"], row["age"])
client.close() await client.close()
asyncio.run(main())
``` ```
### Context Manager ### Context Manager
```python ```python
import asyncio
from baradb import Client from baradb import Client
with Client("localhost", 9472) as client: async def main():
result = client.query("SELECT 1") async with Client("localhost", 9472) as client:
result = await client.query("SELECT 1")
print(result.row_count) print(result.row_count)
asyncio.run(main())
``` ```
### Parameterized Queries ### Parameterized Queries
```python ```python
import asyncio
from baradb import Client, WireValue from baradb import Client, WireValue
with Client("localhost", 9472) as client: async def main():
result = client.query_params( async with Client("localhost", 9472) as client:
result = await client.query_params(
"SELECT * FROM users WHERE age > $1 AND country = $2", "SELECT * FROM users WHERE age > $1 AND country = $2",
[WireValue.int64(18), WireValue.string("BG")], [WireValue.int64(18), WireValue.string("BG")],
) )
for row in result: for row in result:
print(row) print(row)
asyncio.run(main())
``` ```
### Query Builder ### Query Builder
```python ```python
import asyncio
from baradb import Client, QueryBuilder from baradb import Client, QueryBuilder
with Client("localhost", 9472) as client: async def main():
async with Client("localhost", 9472) as client:
qb = ( qb = (
QueryBuilder(client) QueryBuilder(client)
.select("name", "email") .select("name", "email")
@@ -78,21 +93,28 @@ with Client("localhost", 9472) as client:
.order_by("name") .order_by("name")
.limit(10) .limit(10)
) )
result = qb.exec() result = await qb.exec()
for row in result: for row in result:
print(row) print(row)
asyncio.run(main())
``` ```
### Vector Search ### Vector Search
```python ```python
import asyncio
from baradb import Client, WireValue from baradb import Client, WireValue
with Client("localhost", 9472) as client: async def main():
result = client.query_params( async with Client("localhost", 9472) as client:
result = await client.query_params(
"SELECT id, name FROM products ORDER BY embedding <-> $1 LIMIT 5", "SELECT id, name FROM products ORDER BY embedding <-> $1 LIMIT 5",
[WireValue.vector([0.1, 0.2, 0.3])], [WireValue.vector([0.1, 0.2, 0.3])],
) )
print(result.rows)
asyncio.run(main())
``` ```
## Running Tests ## Running Tests
@@ -107,7 +129,7 @@ Integration tests (requires server on `localhost:9472`):
```bash ```bash
# Start server # Start server
docker run -d -p 9472:9472 baradb:latest docker run -d -p 9472:9472 barabadb:latest
# Run all tests # Run all tests
pytest pytest
@@ -124,17 +146,17 @@ pytest
| `database` | `default` | Default database | | `database` | `default` | Default database |
| `username` | `admin` | Username | | `username` | `admin` | Username |
| `password` | `""` | Password | | `password` | `""` | Password |
| `timeout` | `30` | Socket timeout in seconds | | `timeout` | `30.0` | Socket timeout in seconds |
### Methods ### Methods (all async)
- `connect()` — open TCP connection - `await client.connect()` — open TCP connection
- `close()` — close connection - `await client.close()` — close connection
- `query(sql) -> QueryResult` — execute SELECT-like query - `await client.query(sql) -> QueryResult` — execute SELECT-like query
- `query_params(sql, params) -> QueryResult` — parameterized query - `await client.query_params(sql, params) -> QueryResult` — parameterized query
- `execute(sql) -> int` — execute DDL/DML, returns affected rows - `await client.execute(sql) -> int` — execute DDL/DML, returns affected rows
- `auth(token)` — JWT authentication - `await client.auth(token)` — JWT authentication
- `ping() -> bool` — health check - `await client.ping() -> bool` — health check
## License ## License
+20 -6
View File
@@ -1,20 +1,34 @@
""" """
BaraDB Python Client BaraDB Python Async Client
Official Python client for BaraDB — Multimodal Database Engine. Official async Python client for BaraDB — Multimodal Database Engine.
Communicates via the BaraDB Wire Protocol (binary, big-endian, TCP). Communicates via the BaraDB Wire Protocol (binary, big-endian, TCP).
Install: Install:
pip install baradb pip install baradb
Quick Start: Quick Start:
import asyncio
from baradb import Client from baradb import Client
async def main():
client = Client("localhost", 9472) client = Client("localhost", 9472)
client.connect() await client.connect()
result = client.query("SELECT name FROM users WHERE age > 18") result = await client.query("SELECT name FROM users WHERE age > 18")
for row in result: for row in result:
print(row["name"]) print(row["name"])
client.close() await client.close()
asyncio.run(main())
Parameterized Queries:
result = await client.query_params(
"SELECT * FROM users WHERE age > $1",
[WireValue.int64(18)]
)
Authentication:
await client.auth("jwt-token-here")
""" """
from .core import ( from .core import (
@@ -27,7 +41,7 @@ from .core import (
ResultFormat, ResultFormat,
) )
__version__ = "1.0.0" __version__ = "1.1.2"
__all__ = [ __all__ = [
"Client", "Client",
"QueryBuilder", "QueryBuilder",
+138 -72
View File
@@ -1,33 +1,38 @@
""" """
BaraDB Python Client BaraDB Python Async Client
Binary protocol client for BaraDB database. Official async Python client for BaraDB — Multimodal Database Engine.
Communicates via the BaraDB Wire Protocol (binary, big-endian). Communicates via the BaraDB Wire Protocol (binary, big-endian, TCP).
Install: Install:
pip install baradb pip install baradb
Quick Start: Quick Start:
import asyncio
from baradb import Client from baradb import Client
async def main():
client = Client("localhost", 9472) client = Client("localhost", 9472)
client.connect() await client.connect()
result = client.query("SELECT name FROM users WHERE age > 18") result = await client.query("SELECT name FROM users WHERE age > 18")
for row in result: for row in result:
print(row["name"]) print(row["name"])
client.close() await client.close()
asyncio.run(main())
Parameterized Queries: Parameterized Queries:
result = client.query_params("SELECT * FROM users WHERE age > $1", [WireValue.int64(18)]) result = await client.query_params(
"SELECT * FROM users WHERE age > $1",
[WireValue.int64(18)]
)
Authentication: Authentication:
client = Client("localhost", 9472, username="admin", password="secret") await client.auth("jwt-token-here")
client.connect()
client.auth("jwt-token-here")
""" """
import socket import asyncio
import struct import struct
import json
from typing import Any, Optional, Sequence from typing import Any, Optional, Sequence
@@ -49,7 +54,6 @@ class FieldKind:
class MsgKind: class MsgKind:
# Client messages
CLIENT_HANDSHAKE = 0x01 CLIENT_HANDSHAKE = 0x01
QUERY = 0x02 QUERY = 0x02
QUERY_PARAMS = 0x03 QUERY_PARAMS = 0x03
@@ -59,7 +63,6 @@ class MsgKind:
CLOSE = 0x07 CLOSE = 0x07
PING = 0x08 PING = 0x08
AUTH = 0x09 AUTH = 0x09
# Server messages
SERVER_HANDSHAKE = 0x80 SERVER_HANDSHAKE = 0x80
READY = 0x81 READY = 0x81
DATA = 0x82 DATA = 0x82
@@ -88,7 +91,7 @@ class WireValue:
return WireValue(FieldKind.NULL) return WireValue(FieldKind.NULL)
@staticmethod @staticmethod
def bool_val(val: bool): def bool(val: bool):
return WireValue(FieldKind.BOOL, val) return WireValue(FieldKind.BOOL, val)
@staticmethod @staticmethod
@@ -120,15 +123,15 @@ class WireValue:
return WireValue(FieldKind.STRING, val) return WireValue(FieldKind.STRING, val)
@staticmethod @staticmethod
def bytes_val(val: bytes): def bytes(val: bytes):
return WireValue(FieldKind.BYTES, val) return WireValue(FieldKind.BYTES, val)
@staticmethod @staticmethod
def array_val(val: list): def array(val: list):
return WireValue(FieldKind.ARRAY, val) return WireValue(FieldKind.ARRAY, val)
@staticmethod @staticmethod
def object_val(val: dict): def object(val: dict):
return WireValue(FieldKind.OBJECT, val) return WireValue(FieldKind.OBJECT, val)
@staticmethod @staticmethod
@@ -136,7 +139,7 @@ class WireValue:
return WireValue(FieldKind.VECTOR, val) return WireValue(FieldKind.VECTOR, val)
@staticmethod @staticmethod
def json_val(val: str): def json(val: str):
return WireValue(FieldKind.JSON, val) return WireValue(FieldKind.JSON, val)
def serialize(self) -> bytes: def serialize(self) -> bytes:
@@ -202,72 +205,92 @@ class QueryResult:
class Client: class Client:
"""BaraDB database client.""" """Async BaraDB database client."""
def __init__(self, host: str = "localhost", port: int = 9472, def __init__(self, host: str = "localhost", port: int = 9472,
database: str = "default", username: str = "admin", database: str = "default", username: str = "admin",
password: str = "", timeout: int = 30): password: str = "", timeout: float = 30.0):
self.host = host self.host = host
self.port = port self.port = port
self.database = database self.database = database
self.username = username self.username = username
self.password = password self.password = password
self.timeout = timeout self.timeout = timeout
self._sock: Optional[socket.socket] = None self._reader: Optional[asyncio.StreamReader] = None
self._writer: Optional[asyncio.StreamWriter] = None
self._connected = False self._connected = False
self._request_id = 0 self._request_id = 0
self._buffer = bytearray()
self._pending_resolve = None
self._request_queue: list = []
self._request_lock = False
def connect(self) -> None: async def connect(self) -> None:
"""Connect to the BaraDB server.""" """Connect to the BaraDB server."""
self._sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self._reader, self._writer = await asyncio.wait_for(
self._sock.settimeout(self.timeout) asyncio.open_connection(self.host, self.port),
self._sock.connect((self.host, self.port)) timeout=self.timeout
)
self._connected = True self._connected = True
def close(self) -> None: async def close(self) -> None:
if self._sock: """Close the connection to the server."""
if self._writer and self._connected:
try: try:
msg = self._build_message(MsgKind.CLOSE, b"") msg = self._build_message(MsgKind.CLOSE, b"")
self._sock.send(msg) self._writer.write(msg)
await self._writer.drain()
except Exception: except Exception:
pass pass
self._sock.close() if self._writer:
self._writer.close()
await self._writer.wait_closed()
self._writer = None
self._reader = None
self._connected = False self._connected = False
def is_connected(self) -> bool: def is_connected(self) -> bool:
"""Check if client is connected."""
return self._connected return self._connected
def _next_id(self) -> int: def _next_id(self) -> int:
self._request_id += 1 self._request_id += 1
return self._request_id return self._request_id
def _recv_exact(self, size: int) -> bytes: async def _recv_exact(self, size: int) -> bytes:
"""Receive exactly `size` bytes from the socket.""" """Receive exactly `size` bytes from the socket."""
data = b"" while len(self._buffer) < size:
while len(data) < size: try:
chunk = self._sock.recv(size - len(data)) chunk = await asyncio.wait_for(
self._reader.read(size - len(self._buffer)),
timeout=self.timeout
)
if not chunk: if not chunk:
raise ConnectionError("Connection closed by server") raise ConnectionError("Connection closed by server")
data += chunk self._buffer.extend(chunk)
except asyncio.TimeoutError:
raise TimeoutError("Receive timeout")
data = bytes(self._buffer[:size])
del self._buffer[:size]
return data return data
def _read_response_header(self) -> tuple[int, int, int]: async def _read_header(self) -> tuple[int, int, int]:
"""Read a 12-byte message header. Returns (kind, length, request_id).""" """Read a 12-byte message header. Returns (kind, length, request_id)."""
header = self._recv_exact(12) header = await self._recv_exact(12)
kind, length, req_id = struct.unpack(">III", header) kind, length, req_id = struct.unpack(">III", header)
return kind, length, req_id return kind, length, req_id
def _read_error(self, length: int) -> Exception: async def _read_error(self, length: int) -> Exception:
"""Read and parse an ERROR payload.""" """Read and parse an ERROR payload."""
data = self._recv_exact(length) data = await self._recv_exact(length)
code = struct.unpack(">I", data[:4])[0] code = struct.unpack(">I", data[:4])[0]
msg_len = struct.unpack(">I", data[4:8])[0] msg_len = struct.unpack(">I", data[4:8])[0]
error_msg = data[8:8 + msg_len].decode("utf-8") error_msg = data[8:8 + msg_len].decode("utf-8")
return Exception(f"BaraDB error {code}: {error_msg}") return Exception(f"BaraDB error {code}: {error_msg}")
def _read_data_response(self, length: int) -> QueryResult: async def _read_data_response(self, length: int) -> QueryResult:
"""Read and parse a DATA payload, then follow up with COMPLETE.""" """Read and parse a DATA payload, then follow up with COMPLETE."""
data = self._recv_exact(length) data = await self._recv_exact(length)
pos = [0] pos = [0]
col_count = struct.unpack(">I", data[pos[0]:pos[0]+4])[0] col_count = struct.unpack(">I", data[pos[0]:pos[0]+4])[0]
@@ -299,68 +322,107 @@ class Client:
result.rows = rows result.rows = rows
result.row_count = row_count result.row_count = row_count
comp_kind, comp_len, _ = self._read_response_header() comp_kind, comp_len, _ = await self._read_header()
if comp_kind == MsgKind.COMPLETE: if comp_kind == MsgKind.COMPLETE:
comp_data = self._recv_exact(comp_len) comp_data = await self._recv_exact(comp_len)
result.affected_rows = struct.unpack(">I", comp_data[:4])[0] result.affected_rows = struct.unpack(">I", comp_data[:4])[0]
elif comp_kind == MsgKind.ERROR: elif comp_kind == MsgKind.ERROR:
raise self._read_error(comp_len) raise await self._read_error(comp_len)
return result return result
def auth(self, token: str) -> None: async def auth(self, token: str) -> None:
"""Authenticate with the server using a JWT token.""" """Authenticate with the server using a JWT token."""
if not self._connected:
raise Exception("Not connected")
encoded = token.encode("utf-8") encoded = token.encode("utf-8")
payload = struct.pack(">I", len(encoded)) + encoded payload = struct.pack(">I", len(encoded)) + encoded
msg = self._build_message(MsgKind.AUTH, payload) msg = self._build_message(MsgKind.AUTH, payload)
self._sock.send(msg) self._writer.write(msg)
await self._writer.drain()
kind, length, _ = self._read_response_header() kind, length, _ = await self._read_header()
if kind == MsgKind.AUTH_OK: if kind == MsgKind.AUTH_OK:
return return
elif kind == MsgKind.ERROR: elif kind == MsgKind.ERROR:
raise self._read_error(length) raise await self._read_error(length)
else: else:
raise Exception(f"Unexpected auth response: 0x{kind:02x}") raise Exception(f"Unexpected auth response: 0x{kind:02x}")
def ping(self) -> bool: async def ping(self) -> bool:
"""Ping the server. Returns True if pong received.""" """Ping the server. Returns True if pong received."""
if not self._connected:
raise Exception("Not connected")
msg = self._build_message(MsgKind.PING, b"") msg = self._build_message(MsgKind.PING, b"")
self._sock.send(msg) self._writer.write(msg)
await self._writer.drain()
kind, length, _ = self._read_response_header() kind, length, _ = await self._read_header()
if kind == MsgKind.PONG: if kind == MsgKind.PONG:
return True return True
elif kind == MsgKind.ERROR: elif kind == MsgKind.ERROR:
raise self._read_error(length) raise await self._read_error(length)
return False return False
def query(self, sql: str) -> QueryResult: async def _process_queue(self) -> None:
"""Process queued requests sequentially."""
if self._request_lock or len(self._request_queue) == 0:
return
self._request_lock = True
task_data = self._request_queue.pop(0)
try:
result = await task_data["task"]()
task_data["resolve"](result)
except Exception as err:
task_data["reject"](err)
finally:
self._request_lock = False
asyncio.create_task(self._process_queue())
def _enqueue(self, task) -> None:
"""Add a task to the request queue."""
future = asyncio.Future()
self._request_queue.append({"task": task, "resolve": future.set_result, "reject": future.set_exception})
asyncio.create_task(self._process_queue())
return future
async def query(self, sql: str) -> QueryResult:
"""Execute a BaraQL query.""" """Execute a BaraQL query."""
if not self._connected:
raise Exception("Not connected")
async def _do_query():
payload = self._encode_string(sql) payload = self._encode_string(sql)
payload += bytes([ResultFormat.BINARY]) payload += bytes([ResultFormat.BINARY])
msg = self._build_message(MsgKind.QUERY, payload) msg = self._build_message(MsgKind.QUERY, payload)
self._sock.send(msg) self._writer.write(msg)
await self._writer.drain()
kind, length, _ = self._read_response_header() kind, length, _ = await self._read_header()
if kind == MsgKind.ERROR: if kind == MsgKind.ERROR:
raise self._read_error(length) raise await self._read_error(length)
if kind == MsgKind.DATA: if kind == MsgKind.DATA:
return self._read_data_response(length) return await self._read_data_response(length)
if kind == MsgKind.COMPLETE: if kind == MsgKind.COMPLETE:
data = self._recv_exact(length) data = await self._recv_exact(length)
result = QueryResult() result = QueryResult()
result.affected_rows = struct.unpack(">I", data[:4])[0] result.affected_rows = struct.unpack(">I", data[:4])[0]
return result return result
return QueryResult() return QueryResult()
def query_params(self, sql: str, params: Sequence[WireValue]) -> QueryResult: return await self._enqueue(_do_query)
async def query_params(self, sql: str, params: Sequence[WireValue]) -> QueryResult:
"""Execute a parameterized BaraQL query.""" """Execute a parameterized BaraQL query."""
if not self._connected:
raise Exception("Not connected")
async def _do_query_params():
payload = self._encode_string(sql) payload = self._encode_string(sql)
payload += bytes([ResultFormat.BINARY]) payload += bytes([ResultFormat.BINARY])
payload += struct.pack(">I", len(params)) payload += struct.pack(">I", len(params))
@@ -368,26 +430,30 @@ class Client:
payload += p.serialize() payload += p.serialize()
msg = self._build_message(MsgKind.QUERY_PARAMS, payload) msg = self._build_message(MsgKind.QUERY_PARAMS, payload)
self._sock.send(msg) self._writer.write(msg)
await self._writer.drain()
kind, length, _ = self._read_response_header() kind, length, _ = await self._read_header()
if kind == MsgKind.ERROR: if kind == MsgKind.ERROR:
raise self._read_error(length) raise await self._read_error(length)
if kind == MsgKind.DATA: if kind == MsgKind.DATA:
return self._read_data_response(length) return await self._read_data_response(length)
if kind == MsgKind.COMPLETE: if kind == MsgKind.COMPLETE:
data = self._recv_exact(length) data = await self._recv_exact(length)
result = QueryResult() result = QueryResult()
result.affected_rows = struct.unpack(">I", data[:4])[0] result.affected_rows = struct.unpack(">I", data[:4])[0]
return result return result
return QueryResult() return QueryResult()
def execute(self, sql: str) -> int: return await self._enqueue(_do_query_params)
result = self.query(sql)
async def execute(self, sql: str) -> int:
"""Execute a query and return affected rows count."""
result = await self.query(sql)
return result.affected_rows return result.affected_rows
def _build_message(self, kind: int, payload: bytes) -> bytes: def _build_message(self, kind: int, payload: bytes) -> bytes:
@@ -477,12 +543,12 @@ class Client:
return self._read_string(data, pos) return self._read_string(data, pos)
return None return None
def __enter__(self): async def __aenter__(self):
self.connect() await self.connect()
return self return self
def __exit__(self, *args): async def __aexit__(self, *args):
self.close() await self.close()
class QueryBuilder: class QueryBuilder:
@@ -559,5 +625,5 @@ class QueryBuilder:
sql += " OFFSET " + str(self._offset) sql += " OFFSET " + str(self._offset)
return sql return sql
def exec(self) -> QueryResult: async def exec(self) -> QueryResult:
return self.client.query(self.build()) return await self.client.query(self.build())
+30 -29
View File
@@ -6,35 +6,36 @@ This script demonstrates common operations with the BaraDB Python client.
Run it while a BaraDB server is listening on localhost:9472. Run it while a BaraDB server is listening on localhost:9472.
""" """
import asyncio
import sys import sys
from baradb import Client, QueryBuilder, WireValue from baradb import Client, QueryBuilder, WireValue
def example_connection(): async def example_connection():
print("=== Connection ===") print("=== Connection ===")
client = Client("localhost", 9472) client = Client("localhost", 9472)
client.connect() await client.connect()
print(f"Connected: {client.is_connected()}") print(f"Connected: {client.is_connected()}")
print(f"Ping: {client.ping()}") print(f"Ping: {await client.ping()}")
client.close() await client.close()
print(f"Connected after close: {client.is_connected()}") print(f"Connected after close: {client.is_connected()}")
print() print()
def example_context_manager(): async def example_context_manager():
print("=== Context Manager ===") print("=== Context Manager ===")
with Client("localhost", 9472) as client: async with Client("localhost", 9472) as client:
print(f"Inside context: {client.is_connected()}") print(f"Inside context: {client.is_connected()}")
print(f"Ping: {client.ping()}") print(f"Ping: {await client.ping()}")
print("Outside context: closed automatically") print("Outside context: closed automatically")
print() print()
def example_simple_query(): async def example_simple_query():
print("=== Simple Query ===") print("=== Simple Query ===")
with Client("localhost", 9472) as client: async with Client("localhost", 9472) as client:
result = client.query("SELECT 42 as answer, 'BaraDB' as db") result = await client.query("SELECT 42 as answer, 'BaraDB' as db")
print(f"Columns: {result.columns}") print(f"Columns: {result.columns}")
print(f"Rows: {result.rows}") print(f"Rows: {result.rows}")
print(f"Row count: {result.row_count}") print(f"Row count: {result.row_count}")
@@ -43,15 +44,15 @@ def example_simple_query():
print() print()
def example_parameterized_query(): async def example_parameterized_query():
print("=== Parameterized Query ===") print("=== Parameterized Query ===")
with Client("localhost", 9472) as client: async with Client("localhost", 9472) as client:
result = client.query_params( result = await client.query_params(
"SELECT $1 as num, $2 as txt, $3 as flag", "SELECT $1 as num, $2 as txt, $3 as flag",
[ [
WireValue.int64(123), WireValue.int64(123),
WireValue.string("hello world"), WireValue.string("hello world"),
WireValue.bool_val(True), WireValue.bool(True),
], ],
) )
for row in result: for row in result:
@@ -59,9 +60,9 @@ def example_parameterized_query():
print() print()
def example_query_builder(): async def example_query_builder():
print("=== Query Builder ===") print("=== Query Builder ===")
with Client("localhost", 9472) as client: async with Client("localhost", 9472) as client:
sql = ( sql = (
QueryBuilder(client) QueryBuilder(client)
.select("id", "name") .select("id", "name")
@@ -75,10 +76,10 @@ def example_query_builder():
print() print()
def example_vector(): async def example_vector():
print("=== Vector Value ===") print("=== Vector Value ===")
with Client("localhost", 9472) as client: async with Client("localhost", 9472) as client:
result = client.query_params( result = await client.query_params(
"SELECT $1 as embedding", "SELECT $1 as embedding",
[WireValue.vector([0.1, 0.2, 0.3, 0.4])], [WireValue.vector([0.1, 0.2, 0.3, 0.4])],
) )
@@ -87,34 +88,34 @@ def example_vector():
print() print()
def example_ddl_dml(): async def example_ddl_dml():
print("=== DDL & DML ===") print("=== DDL & DML ===")
with Client("localhost", 9472) as client: async with Client("localhost", 9472) as client:
# Clean up # Clean up
try: try:
client.execute("DROP TABLE IF EXISTS demo_products") await client.execute("DROP TABLE IF EXISTS demo_products")
except Exception as exc: except Exception as exc:
print(f"Cleanup warning: {exc}") print(f"Cleanup warning: {exc}")
client.execute( await client.execute(
"CREATE TABLE demo_products (id INT PRIMARY KEY, name STRING, price FLOAT)" "CREATE TABLE demo_products (id INT PRIMARY KEY, name STRING, price FLOAT)"
) )
affected = client.execute( affected = await client.execute(
"INSERT INTO demo_products (id, name, price) VALUES (1, 'Widget', 9.99)" "INSERT INTO demo_products (id, name, price) VALUES (1, 'Widget', 9.99)"
) )
print(f"Insert affected rows: {affected}") print(f"Insert affected rows: {affected}")
result = client.query("SELECT * FROM demo_products") result = await client.query("SELECT * FROM demo_products")
print(f"Select returned {result.row_count} row(s)") print(f"Select returned {result.row_count} row(s)")
for row in result: for row in result:
print(f" {row}") print(f" {row}")
client.execute("DROP TABLE demo_products") await client.execute("DROP TABLE demo_products")
print("Table dropped") print("Table dropped")
print() print()
def main(): async def main():
print("BaraDB Python Client Examples") print("BaraDB Python Client Examples")
print("Make sure BaraDB is running on localhost:9472") print("Make sure BaraDB is running on localhost:9472")
print() print()
@@ -131,10 +132,10 @@ def main():
for fn in examples: for fn in examples:
try: try:
fn() await fn()
except Exception as exc: except Exception as exc:
print(f"ERROR in {fn.__name__}: {exc}", file=sys.stderr) print(f"ERROR in {fn.__name__}: {exc}", file=sys.stderr)
if __name__ == "__main__": if __name__ == "__main__":
main() asyncio.run(main())
+2 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project] [project]
name = "baradb" name = "baradb"
version = "1.1.0" version = "1.1.2"
description = "Official Python client for BaraDB — Multimodal Database Engine" description = "Official Python client for BaraDB — Multimodal Database Engine"
readme = "README.md" readme = "README.md"
license = { text = "Apache-2.0" } license = { text = "Apache-2.0" }
@@ -48,6 +48,7 @@ packages = ["baradb"]
[tool.pytest.ini_options] [tool.pytest.ini_options]
testpaths = ["tests"] testpaths = ["tests"]
pythonpath = ["."] pythonpath = ["."]
asyncio_mode = "auto"
[tool.mypy] [tool.mypy]
strict = true strict = true
+42 -39
View File
@@ -5,13 +5,16 @@ Requires a running BaraDB server on localhost:9472.
These tests are skipped automatically if the server is unreachable. These tests are skipped automatically if the server is unreachable.
""" """
import asyncio
import os
import socket import socket
import pytest import pytest
import pytest_asyncio
from baradb import Client, WireValue, QueryBuilder from baradb import Client, WireValue, QueryBuilder
BARADB_HOST = "localhost" BARADB_HOST = os.environ.get("BARADB_HOST", "localhost")
BARADB_PORT = 9472 BARADB_PORT = int(os.environ.get("BARADB_PORT", "9472"))
def _server_available() -> bool: def _server_available() -> bool:
@@ -29,41 +32,41 @@ pytestmark = pytest.mark.skipif(
) )
@pytest.fixture @pytest_asyncio.fixture
def client(): async def client():
c = Client(BARADB_HOST, BARADB_PORT) c = Client(BARADB_HOST, BARADB_PORT)
c.connect() await c.connect()
yield c yield c
c.close() await c.close()
class TestConnection: class TestConnection:
def test_connect_and_close(self): async def test_connect_and_close(self):
c = Client(BARADB_HOST, BARADB_PORT) c = Client(BARADB_HOST, BARADB_PORT)
assert not c.is_connected() assert not c.is_connected()
c.connect() await c.connect()
assert c.is_connected() assert c.is_connected()
c.close() await c.close()
assert not c.is_connected() assert not c.is_connected()
def test_context_manager(self): async def test_context_manager(self):
with Client(BARADB_HOST, BARADB_PORT) as c: async with Client(BARADB_HOST, BARADB_PORT) as c:
assert c.is_connected() assert c.is_connected()
assert not c.is_connected() assert not c.is_connected()
class TestPing: class TestPing:
def test_ping(self, client): async def test_ping(self, client):
assert client.ping() is True assert await client.ping() is True
class TestQuery: class TestQuery:
def test_simple_select(self, client): async def test_simple_select(self, client):
result = client.query("SELECT 1 as one") result = await client.query("SELECT 1 as one")
assert result.row_count >= 0 # server may return rows or empty assert result.row_count >= 0 # server may return rows or empty
def test_query_with_params(self, client): async def test_query_with_params(self, client):
result = client.query_params( result = await client.query_params(
"SELECT $1 as num, $2 as txt", "SELECT $1 as num, $2 as txt",
[WireValue.int64(42), WireValue.string("hello")], [WireValue.int64(42), WireValue.string("hello")],
) )
@@ -71,22 +74,22 @@ class TestQuery:
class TestExecute: class TestExecute:
def test_create_table_and_insert(self, client): async def test_create_table_and_insert(self, client):
# Clean up first (best-effort) # Clean up first (best-effort)
try: try:
client.execute("DROP TABLE IF EXISTS test_users") await client.execute("DROP TABLE IF EXISTS test_users")
except Exception: except Exception:
pass pass
client.execute( await client.execute(
"CREATE TABLE test_users (id INT PRIMARY KEY, name STRING, age INT)" "CREATE TABLE test_users (id INT PRIMARY KEY, name STRING, age INT)"
) )
affected = client.execute( affected = await client.execute(
"INSERT INTO test_users (id, name, age) VALUES (1, 'Alice', 30)" "INSERT INTO test_users (id, name, age) VALUES (1, 'Alice', 30)"
) )
assert affected >= 0 assert affected >= 0
result = client.query("SELECT name, age FROM test_users WHERE id = 1") result = await client.query("SELECT name, age FROM test_users WHERE id = 1")
assert result.row_count == 1 assert result.row_count == 1
row = result.rows[0] row = result.rows[0]
# Server returns all columns; map by name via dict iteration # Server returns all columns; map by name via dict iteration
@@ -94,20 +97,20 @@ class TestExecute:
assert row_dict["name"] == "Alice" assert row_dict["name"] == "Alice"
assert row_dict["age"] == 30 assert row_dict["age"] == 30
client.execute("DROP TABLE test_users") await client.execute("DROP TABLE test_users")
class TestQueryBuilder: class TestQueryBuilder:
def test_builder_exec(self, client): async def test_builder_exec(self, client):
try: try:
client.execute("DROP TABLE IF EXISTS test_products") await client.execute("DROP TABLE IF EXISTS test_products")
except Exception: except Exception:
pass pass
client.execute( await client.execute(
"CREATE TABLE test_products (id INT PRIMARY KEY, name STRING, price FLOAT)" "CREATE TABLE test_products (id INT PRIMARY KEY, name STRING, price FLOAT)"
) )
client.execute( await client.execute(
"INSERT INTO test_products (id, name, price) VALUES (1, 'Widget', 9.99)" "INSERT INTO test_products (id, name, price) VALUES (1, 'Widget', 9.99)"
) )
@@ -117,36 +120,36 @@ class TestQueryBuilder:
.from_("test_products") .from_("test_products")
.where("id = 1") .where("id = 1")
) )
result = qb.exec() result = await qb.exec()
assert result.row_count == 1 assert result.row_count == 1
client.execute("DROP TABLE test_products") await client.execute("DROP TABLE test_products")
class TestAuth: class TestAuth:
def test_auth_with_dummy_token(self, client): async def test_auth_with_dummy_token(self, client):
# Server uses default JWT secret in dev mode, any token format is accepted # Server uses default JWT secret in dev mode, any token format is accepted
# depending on server config. We just verify the method does not crash. # depending on server config. We just verify the method does not crash.
try: try:
client.auth("dummy-token-for-testing") await client.auth("dummy-token-for-testing")
except Exception as exc: except Exception as exc:
# Auth may fail with invalid token — that's acceptable for this test # Auth may fail with invalid token — that's acceptable for this test
assert "Auth" in str(exc) or "error" in str(exc).lower() assert "Auth" in str(exc) or "error" in str(exc).lower()
class TestTransactions: class TestTransactions:
def test_transaction_begin_commit(self, client): async def test_transaction_begin_commit(self, client):
try: try:
client.execute("DROP TABLE IF EXISTS test_txn") await client.execute("DROP TABLE IF EXISTS test_txn")
except Exception: except Exception:
pass pass
client.execute("CREATE TABLE test_txn (id INT PRIMARY KEY)") await client.execute("CREATE TABLE test_txn (id INT PRIMARY KEY)")
client.execute("BEGIN") await client.execute("BEGIN")
client.execute("INSERT INTO test_txn (id) VALUES (1)") await client.execute("INSERT INTO test_txn (id) VALUES (1)")
client.execute("COMMIT") await client.execute("COMMIT")
result = client.query("SELECT COUNT(*) FROM test_txn") result = await client.query("SELECT COUNT(*) FROM test_txn")
assert result.row_count >= 0 assert result.row_count >= 0
client.execute("DROP TABLE test_txn") await client.execute("DROP TABLE test_txn")
+6 -6
View File
@@ -18,11 +18,11 @@ class TestWireValue:
assert data == b"\x00" assert data == b"\x00"
def test_bool_true(self): def test_bool_true(self):
wv = WireValue.bool_val(True) wv = WireValue.bool(True)
assert wv.serialize() == b"\x01\x01" assert wv.serialize() == b"\x01\x01"
def test_bool_false(self): def test_bool_false(self):
wv = WireValue.bool_val(False) wv = WireValue.bool(False)
assert wv.serialize() == b"\x01\x00" assert wv.serialize() == b"\x01\x00"
def test_int8(self): def test_int8(self):
@@ -70,7 +70,7 @@ class TestWireValue:
assert data[5:] == b"hello" assert data[5:] == b"hello"
def test_bytes(self): def test_bytes(self):
wv = WireValue.bytes_val(b"\xde\xad\xbe\xef") wv = WireValue.bytes(b"\xde\xad\xbe\xef")
data = wv.serialize() data = wv.serialize()
assert data[0] == FieldKind.BYTES assert data[0] == FieldKind.BYTES
length = struct.unpack(">I", data[1:5])[0] length = struct.unpack(">I", data[1:5])[0]
@@ -87,7 +87,7 @@ class TestWireValue:
assert floats == [1.0, 2.0, 3.0] assert floats == [1.0, 2.0, 3.0]
def test_json(self): def test_json(self):
wv = WireValue.json_val('{"key": "value"}') wv = WireValue.json('{"key": "value"}')
data = wv.serialize() data = wv.serialize()
assert data[0] == FieldKind.JSON assert data[0] == FieldKind.JSON
length = struct.unpack(">I", data[1:5])[0] length = struct.unpack(">I", data[1:5])[0]
@@ -95,7 +95,7 @@ class TestWireValue:
def test_array(self): def test_array(self):
inner = [WireValue.string("a"), WireValue.string("b")] inner = [WireValue.string("a"), WireValue.string("b")]
wv = WireValue.array_val(inner) wv = WireValue.array(inner)
data = wv.serialize() data = wv.serialize()
assert data[0] == FieldKind.ARRAY assert data[0] == FieldKind.ARRAY
count = struct.unpack(">I", data[1:5])[0] count = struct.unpack(">I", data[1:5])[0]
@@ -103,7 +103,7 @@ class TestWireValue:
def test_object(self): def test_object(self):
inner = {"name": WireValue.string("Bara"), "age": WireValue.int32(42)} inner = {"name": WireValue.string("Bara"), "age": WireValue.int32(42)}
wv = WireValue.object_val(inner) wv = WireValue.object(inner)
data = wv.serialize() data = wv.serialize()
assert data[0] == FieldKind.OBJECT assert data[0] == FieldKind.OBJECT
count = struct.unpack(">I", data[1:5])[0] count = struct.unpack(">I", data[1:5])[0]
+5 -3
View File
@@ -1,11 +1,11 @@
[package] [package]
name = "baradb" name = "baradb"
version = "1.1.0" version = "1.1.2"
edition = "2021" edition = "2021"
authors = ["BaraDB Team <team@baradb.dev>"] authors = ["BaraDB Team <team@baradb.dev>"]
description = "Official Rust client for BaraDB — binary protocol client" description = "Official async Rust client for BaraDB — binary protocol client"
license = "Apache-2.0" license = "Apache-2.0"
repository = "https://github.com/barabadb/baradadb" repository = "https://github.com/barabadb/baradadb.git"
documentation = "https://docs.baradb.dev" documentation = "https://docs.baradb.dev"
readme = "README.md" readme = "README.md"
keywords = ["database", "baradb", "multimodal", "client", "wire-protocol"] keywords = ["database", "baradb", "multimodal", "client", "wire-protocol"]
@@ -13,8 +13,10 @@ categories = ["database", "network-programming"]
rust-version = "1.70" rust-version = "1.70"
[dependencies] [dependencies]
tokio = { version = "1.35", features = ["full"] }
[dev-dependencies] [dev-dependencies]
tokio = { version = "1.35", features = ["full"] }
[[example]] [[example]]
name = "basic" name = "basic"
+41 -35
View File
@@ -1,13 +1,13 @@
# BaraDB Rust Client # BaraDB Async Rust Client
Official Rust client for **BaraDB** — a multimodal database engine written in Nim. Official async Rust client for **BaraDB** — a multimodal database engine written in Nim.
## Features ## Features
- **Binary wire protocol** — fast TCP communication using only `std` - **Async/await** — fully non-blocking with Tokio runtime
- **Zero dependencies** — no external crates required - **Binary wire protocol** — fast TCP communication
- **Sync API** — blocking I/O suitable for most applications
- **Query builder** — fluent SQL construction - **Query builder** — fluent SQL construction
- **Parameterized queries** — safe from SQL injection
- **Vector & JSON support** — first-class multimodal types - **Vector & JSON support** — first-class multimodal types
## Installation ## Installation
@@ -16,13 +16,14 @@ Add to your `Cargo.toml`:
```toml ```toml
[dependencies] [dependencies]
baradb = "1.0" baradb = "1.1"
tokio = { version = "1.35", features = ["full"] }
``` ```
Or from source: Or from source:
```bash ```bash
git clone https://github.com/barabadb/baradadb.git git clone https://codeberg.org/baraba/baradb
cd clients/rust cd clients/rust
cargo build cargo build
``` ```
@@ -32,13 +33,14 @@ cargo build
```rust ```rust
use baradb::Client; use baradb::Client;
fn main() -> Result<(), Box<dyn std::error::Error>> { #[tokio::main]
let mut client = Client::connect("localhost", 9472)?; async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let result = client.query("SELECT name, age FROM users WHERE age > 18")?; let mut client = Client::connect("localhost", 9472).await?;
let result = client.query("SELECT name, age FROM users WHERE age > 18").await?;
for row in result.rows() { for row in result.rows() {
println!("{:?}", row); println!("{:?}", row);
} }
client.close(); client.close().await;
Ok(()) Ok(())
} }
``` ```
@@ -48,16 +50,17 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
```rust ```rust
use baradb::{Client, WireValue}; use baradb::{Client, WireValue};
fn main() -> Result<(), Box<dyn std::error::Error>> { #[tokio::main]
let mut client = Client::connect("localhost", 9472)?; async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut client = Client::connect("localhost", 9472).await?;
let result = client.query_params( let result = client.query_params(
"SELECT * FROM users WHERE age > $1 AND country = $2", "SELECT * FROM users WHERE age > $1 AND country = $2",
&[WireValue::Int64(18), WireValue::String("BG".to_string())], &[WireValue::Int64(18), WireValue::String("BG".to_string())],
)?; ).await?;
for row in result.rows() { for row in result.rows() {
println!("{:?}", row); println!("{:?}", row);
} }
client.close(); client.close().await;
Ok(()) Ok(())
} }
``` ```
@@ -65,21 +68,23 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
### Query Builder ### Query Builder
```rust ```rust
use baradb::Client; use baradb::{Client, QueryBuilder};
fn main() -> Result<(), Box<dyn std::error::Error>> { #[tokio::main]
let mut client = Client::connect("localhost", 9472)?; async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let result = baradb::QueryBuilder::new(&mut client) let mut client = Client::connect("localhost", 9472).await?;
let result = QueryBuilder::new(&mut client)
.select(&["name", "email"]) .select(&["name", "email"])
.from("users") .from("users")
.where_clause("active = true") .where_clause("active = true")
.order_by("name", "ASC") .order_by("name", "ASC")
.limit(10) .limit(10)
.exec()?; .exec()
.await?;
for row in result.rows() { for row in result.rows() {
println!("{:?}", row); println!("{:?}", row);
} }
client.close(); client.close().await;
Ok(()) Ok(())
} }
``` ```
@@ -89,13 +94,14 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
```rust ```rust
use baradb::{Client, WireValue}; use baradb::{Client, WireValue};
fn main() -> Result<(), Box<dyn std::error::Error>> { #[tokio::main]
let mut client = Client::connect("localhost", 9472)?; async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut client = Client::connect("localhost", 9472).await?;
let result = client.query_params( let result = client.query_params(
"SELECT id, name FROM products ORDER BY embedding <-> $1 LIMIT 5", "SELECT id, name FROM products ORDER BY embedding <-> $1 LIMIT 5",
&[WireValue::Vector(vec![0.1, 0.2, 0.3])], &[WireValue::Vector(vec![0.1, 0.2, 0.3])],
)?; ).await?;
client.close(); client.close().await;
Ok(()) Ok(())
} }
``` ```
@@ -112,7 +118,7 @@ Integration tests (requires server on `localhost:9472`):
```bash ```bash
# Start server # Start server
docker run -d -p 9472:9472 baradb:latest docker run -d -p 9472:9472 barabadb:latest
# Run all tests # Run all tests
cargo test cargo test
@@ -120,18 +126,18 @@ cargo test
## API Reference ## API Reference
### `Client::connect(host, port)` ### `Client::connect(host, port) -> Result<Client>`
Creates a new client connected to the given host and port. Creates a new async client connected to the given host and port.
### Methods ### Methods (all async)
- `query(sql) -> Result<QueryResult>` — execute SELECT-like query - `await client.query(sql) -> Result<QueryResult>` — execute SELECT-like query
- `query_params(sql, params) -> Result<QueryResult>` — parameterized query - `await client.query_params(sql, params) -> Result<QueryResult>` — parameterized query
- `execute(sql) -> Result<usize>` — execute DDL/DML, returns affected rows - `await client.execute(sql) -> Result<usize>` — execute DDL/DML, returns affected rows
- `auth(token) -> Result<()>` — JWT authentication - `await client.auth(token) -> Result<()>` — JWT authentication
- `ping() -> Result<bool>` — health check - `await client.ping() -> Result<bool>` — health check
- `close()` — close connection - `await client.close()` — close connection
## License ## License
+58 -48
View File
@@ -3,34 +3,65 @@
use baradb::{Client, QueryBuilder, WireValue}; use baradb::{Client, QueryBuilder, WireValue};
fn example_connection() -> Result<(), Box<dyn std::error::Error>> { #[tokio::main]
async fn main() {
println!("BaraDB Rust Client Examples");
println!("Make sure BaraDB is running on localhost:9472");
println!();
if let Err(e) = example_connection().await {
eprintln!("ERROR: {}", e);
}
if let Err(e) = example_simple_query().await {
eprintln!("ERROR: {}", e);
}
if let Err(e) = example_parameterized_query().await {
eprintln!("ERROR: {}", e);
}
if let Err(e) = example_query_builder().await {
eprintln!("ERROR: {}", e);
}
if let Err(e) = example_vector().await {
eprintln!("ERROR: {}", e);
}
if let Err(e) = example_ddl_dml().await {
eprintln!("ERROR: {}", e);
}
}
async fn example_connection() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
println!("=== Connection ==="); println!("=== Connection ===");
let mut client = Client::connect("127.0.0.1", 9472)?; let mut client = Client::connect("127.0.0.1", 9472).await?;
println!("Connected: {}", client.is_connected()); println!("Connected: {}", client.is_connected());
println!("Ping: {}", client.ping()?); println!("Ping: {}", client.ping().await?);
client.close(); client.close().await;
println!("Connected after close: {}", client.is_connected()); println!("Connected after close: {}", client.is_connected());
println!(); println!();
Ok(()) Ok(())
} }
fn example_simple_query() -> Result<(), Box<dyn std::error::Error>> { async fn example_simple_query() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
println!("=== Simple Query ==="); println!("=== Simple Query ===");
let mut client = Client::connect("127.0.0.1", 9472)?; let mut client = Client::connect("127.0.0.1", 9472).await?;
let result = client.query("SELECT 42 as answer, 'BaraDB' as db")?; let result = client.query("SELECT 42 as answer, 'BaraDB' as db").await?;
println!("Columns: {:?}", result.columns()); println!("Columns: {:?}", result.columns());
println!("Row count: {}", result.row_count()); println!("Row count: {}", result.row_count());
for row in result.rows() { for row in result.rows() {
println!(" {:?}", row); println!(" {:?}", row);
} }
client.close(); client.close().await;
println!(); println!();
Ok(()) Ok(())
} }
fn example_parameterized_query() -> Result<(), Box<dyn std::error::Error>> { async fn example_parameterized_query() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
println!("=== Parameterized Query ==="); println!("=== Parameterized Query ===");
let mut client = Client::connect("127.0.0.1", 9472)?; let mut client = Client::connect("127.0.0.1", 9472).await?;
let result = client.query_params( let result = client.query_params(
"SELECT $1 as num, $2 as txt, $3 as flag", "SELECT $1 as num, $2 as txt, $3 as flag",
&[ &[
@@ -38,18 +69,18 @@ fn example_parameterized_query() -> Result<(), Box<dyn std::error::Error>> {
WireValue::String("hello world".to_string()), WireValue::String("hello world".to_string()),
WireValue::Bool(true), WireValue::Bool(true),
], ],
)?; ).await?;
for row in result.rows() { for row in result.rows() {
println!(" {:?}", row); println!(" {:?}", row);
} }
client.close(); client.close().await;
println!(); println!();
Ok(()) Ok(())
} }
fn example_query_builder() -> Result<(), Box<dyn std::error::Error>> { async fn example_query_builder() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
println!("=== Query Builder ==="); println!("=== Query Builder ===");
let mut client = Client::connect("127.0.0.1", 9472)?; let mut client = Client::connect("127.0.0.1", 9472).await?;
let sql = QueryBuilder::new(&mut client) let sql = QueryBuilder::new(&mut client)
.select(&["id", "name"]) .select(&["id", "name"])
.from("users") .from("users")
@@ -58,70 +89,49 @@ fn example_query_builder() -> Result<(), Box<dyn std::error::Error>> {
.limit(5) .limit(5)
.build(); .build();
println!("Generated SQL: {}", sql); println!("Generated SQL: {}", sql);
client.close(); client.close().await;
println!(); println!();
Ok(()) Ok(())
} }
fn example_vector() -> Result<(), Box<dyn std::error::Error>> { async fn example_vector() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
println!("=== Vector Value ==="); println!("=== Vector Value ===");
let mut client = Client::connect("127.0.0.1", 9472)?; let mut client = Client::connect("127.0.0.1", 9472).await?;
let result = client.query_params( let result = client.query_params(
"SELECT $1 as embedding", "SELECT $1 as embedding",
&[WireValue::Vector(vec![0.1, 0.2, 0.3, 0.4])], &[WireValue::Vector(vec![0.1, 0.2, 0.3, 0.4])],
)?; ).await?;
for row in result.rows() { for row in result.rows() {
println!(" {:?}", row); println!(" {:?}", row);
} }
client.close(); client.close().await;
println!(); println!();
Ok(()) Ok(())
} }
fn example_ddl_dml() -> Result<(), Box<dyn std::error::Error>> { async fn example_ddl_dml() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
println!("=== DDL & DML ==="); println!("=== DDL & DML ===");
let mut client = Client::connect("127.0.0.1", 9472)?; let mut client = Client::connect("127.0.0.1", 9472).await?;
let _ = client.execute("DROP TABLE IF EXISTS demo_products"); let _ = client.execute("DROP TABLE IF EXISTS demo_products").await;
client.execute( client.execute(
"CREATE TABLE demo_products (id INT PRIMARY KEY, name STRING, price FLOAT)", "CREATE TABLE demo_products (id INT PRIMARY KEY, name STRING, price FLOAT)",
)?; ).await?;
let affected = client.execute( let affected = client.execute(
"INSERT INTO demo_products (id, name, price) VALUES (1, 'Widget', 9.99)", "INSERT INTO demo_products (id, name, price) VALUES (1, 'Widget', 9.99)",
)?; ).await?;
println!("Insert affected rows: {}", affected); println!("Insert affected rows: {}", affected);
let result = client.query("SELECT * FROM demo_products")?; let result = client.query("SELECT * FROM demo_products").await?;
println!("Select returned {} row(s)", result.row_count()); println!("Select returned {} row(s)", result.row_count());
for row in result.rows() { for row in result.rows() {
println!(" {:?}", row); println!(" {:?}", row);
} }
client.execute("DROP TABLE demo_products")?; client.execute("DROP TABLE demo_products").await?;
println!("Table dropped"); println!("Table dropped");
client.close(); client.close().await;
println!(); println!();
Ok(()) Ok(())
} }
fn main() {
println!("BaraDB Rust Client Examples");
println!("Make sure BaraDB is running on localhost:9472");
println!();
let examples: Vec<fn() -> Result<(), Box<dyn std::error::Error>>> = vec![
example_connection,
example_simple_query,
example_parameterized_query,
example_query_builder,
example_vector,
example_ddl_dml,
];
for example in examples {
if let Err(e) = example() {
eprintln!("ERROR: {}", e);
}
}
}
+5 -4
View File
@@ -1,11 +1,12 @@
use baradb::Client; use baradb::Client;
fn main() { #[tokio::main]
let mut client = Client::connect("localhost", 9472).unwrap(); async fn main() {
let mut client = Client::connect("localhost", 9472).await.unwrap();
println!("Connected: {}", client.is_connected()); println!("Connected: {}", client.is_connected());
match client.ping() { match client.ping().await {
Ok(v) => println!("Ping: {}", v), Ok(v) => println!("Ping: {}", v),
Err(e) => println!("Ping error: {}", e), Err(e) => println!("Ping error: {}", e),
} }
client.close(); client.close().await;
} }
+103 -68
View File
@@ -1,36 +1,42 @@
//! BaraDB Rust Client //! BaraDB Async Rust Client
//! //!
//! Binary protocol client for BaraDB database. //! Async binary protocol client for BaraDB database.
//! Zero external dependencies — uses only `std`. //! Uses Tokio for async I/O operations.
//! //!
//! # Example //! # Example
//! ```no_run //! ```no_run
//! use baradb::{Client, WireValue}; //! use baradb::{Client, WireValue};
//! //!
//! let mut client = Client::connect("localhost", 9472).unwrap(); //! #[tokio::main]
//! let result = client.query("SELECT name FROM users WHERE age > 18").unwrap(); //! async fn main() {
//! let mut client = Client::connect("localhost", 9472).await.unwrap();
//! let result = client.query("SELECT name FROM users WHERE age > 18").await.unwrap();
//! for row in result.rows() { //! for row in result.rows() {
//! if let Some(WireValue::String(name)) = row.get("name") { //! if let Some(WireValue::String(name)) = row.get("name") {
//! println!("{}", name); //! println!("{}", name);
//! } //! }
//! } //! }
//! client.close(); //! client.close().await;
//! }
//! ``` //! ```
//! //!
//! # Parameterized Queries //! # Parameterized Queries
//! ```no_run //! ```no_run
//! use baradb::{Client, WireValue}; //! use baradb::{Client, WireValue};
//! //!
//! let mut client = Client::connect("localhost", 9472).unwrap(); //! #[tokio::main]
//! async fn main() {
//! let mut client = Client::connect("localhost", 9472).await.unwrap();
//! let result = client.query_params( //! let result = client.query_params(
//! "SELECT * FROM users WHERE age > $1", //! "SELECT * FROM users WHERE age > $1",
//! &[WireValue::Int64(18)], //! &[WireValue::Int64(18)],
//! ).unwrap(); //! ).await.unwrap();
//! }
//! ``` //! ```
use std::collections::HashMap; use std::collections::HashMap;
use std::io::{Read, Write}; use tokio::io::{AsyncReadExt, AsyncWriteExt};
use std::net::TcpStream; use tokio::net::TcpStream;
// Client message kinds // Client message kinds
const MK_CLIENT_HANDSHAKE: u32 = 0x01; const MK_CLIENT_HANDSHAKE: u32 = 0x01;
@@ -221,43 +227,41 @@ impl QueryResult {
} }
} }
/// BaraDB client /// BaraDB async client
pub struct Client { pub struct Client {
config: Config,
stream: TcpStream, stream: TcpStream,
connected: bool, connected: bool,
request_id: u32, request_id: u32,
read_buf: Vec<u8>,
} }
impl Client { impl Client {
pub fn connect(host: &str, port: u16) -> Result<Self, Box<dyn std::error::Error>> { /// Connect to a BaraDB server
let config = Config { pub async fn connect(host: &str, port: u16) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
host: host.to_string(), let addr = format!("{}:{}", host, port);
port, let stream = TcpStream::connect(&addr).await?;
..Default::default()
};
Self::connect_with_config(config)
}
pub fn connect_with_config(config: Config) -> Result<Self, Box<dyn std::error::Error>> {
let addr = format!("{}:{}", config.host, config.port);
let stream = TcpStream::connect(&addr)?;
stream.set_nodelay(true)?;
Ok(Client { Ok(Client {
config,
stream, stream,
connected: true, connected: true,
request_id: 0, request_id: 0,
read_buf: Vec::new(),
}) })
} }
pub fn close(&mut self) { /// Connect with custom configuration
pub async fn connect_with_config(config: Config) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
Self::connect(&config.host, config.port).await
}
/// Close the connection
pub async fn close(&mut self) {
if self.connected { if self.connected {
let _ = self.send_close(); let _ = self.send_close().await;
} }
self.connected = false; self.connected = false;
} }
/// Check if connected
pub fn is_connected(&self) -> bool { pub fn is_connected(&self) -> bool {
self.connected self.connected
} }
@@ -267,27 +271,39 @@ impl Client {
self.request_id self.request_id
} }
fn send_close(&mut self) -> Result<(), Box<dyn std::error::Error>> { async fn send_close(&mut self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let msg = build_message(MK_CLOSE, self.next_id(), &[]); let msg = build_message(MK_CLOSE, self.next_id(), &[]);
self.stream.write_all(&msg)?; self.stream.write_all(&msg).await?;
Ok(()) Ok(())
} }
fn read_header(&mut self) -> Result<(u32, u32, u32), Box<dyn std::error::Error>> { async fn read_exact(&mut self, mut n: usize) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
let mut header = [0u8; 12]; let mut buf = vec![0u8; n];
self.stream.read_exact(&mut header)?; let mut pos = 0;
while pos < n {
let read = self.stream.read(&mut buf[pos..]).await?;
if read == 0 {
return Err("Connection closed".into());
}
pos += read;
}
Ok(buf)
}
async fn read_header(&mut self) -> Result<(u32, u32, u32), Box<dyn std::error::Error + Send + Sync>> {
let header = self.read_exact(12).await?;
let kind = u32::from_be_bytes([header[0], header[1], header[2], header[3]]); let kind = u32::from_be_bytes([header[0], header[1], header[2], header[3]]);
let length = u32::from_be_bytes([header[4], header[5], header[6], header[7]]); let length = u32::from_be_bytes([header[4], header[5], header[6], header[7]]);
let req_id = u32::from_be_bytes([header[8], header[9], header[10], header[11]]); let req_id = u32::from_be_bytes([header[8], header[9], header[10], header[11]]);
Ok((kind, length, req_id)) Ok((kind, length, req_id))
} }
fn read_payload(&mut self, length: u32) -> Result<Vec<u8>, Box<dyn std::error::Error>> { async fn read_payload(&mut self, length: u32) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
let mut payload = vec![0u8; length as usize];
if length > 0 { if length > 0 {
self.stream.read_exact(&mut payload)?; self.read_exact(length as usize).await
} else {
Ok(vec![])
} }
Ok(payload)
} }
fn read_error_message(payload: &[u8]) -> String { fn read_error_message(payload: &[u8]) -> String {
@@ -302,26 +318,26 @@ impl Client {
"Query error".to_string() "Query error".to_string()
} }
fn read_data_response(&mut self, payload: &[u8]) -> Result<QueryResult, Box<dyn std::error::Error>> { async fn read_data_response(&mut self, payload: &[u8]) -> Result<QueryResult, Box<dyn std::error::Error + Send + Sync>> {
let mut pos = 0usize; let mut pos = 0usize;
let col_count = read_u32(payload, &mut pos) as usize; let col_count = read_u32(payload, &mut pos);
let mut columns = Vec::with_capacity(col_count); let mut columns = Vec::with_capacity(col_count as usize);
for _ in 0..col_count { for _ in 0..col_count {
columns.push(read_string(payload, &mut pos)); columns.push(read_string(payload, &mut pos));
} }
let mut col_types = Vec::with_capacity(col_count); let mut col_types = Vec::with_capacity(col_count as usize);
for _ in 0..col_count { for _ in 0..col_count {
col_types.push(payload[pos]); col_types.push(payload[pos]);
pos += 1; pos += 1;
} }
let row_count = read_u32(payload, &mut pos) as usize; let row_count = read_u32(payload, &mut pos);
let mut rows = Vec::with_capacity(row_count); let mut rows = Vec::with_capacity(row_count as usize);
for _ in 0..row_count { for _ in 0..row_count {
let mut row = HashMap::new(); let mut row = HashMap::new();
for c in 0..col_count { for c in 0..col_count as usize {
let val = read_wire_value(payload, &mut pos); let val = read_wire_value(payload, &mut pos);
row.insert(columns[c].clone(), val); row.insert(columns[c].clone(), val);
} }
@@ -329,9 +345,9 @@ impl Client {
} }
let mut affected = 0usize; let mut affected = 0usize;
let (comp_kind, comp_len, _) = self.read_header()?; let (comp_kind, comp_len, _) = self.read_header().await?;
if comp_kind == MK_COMPLETE { if comp_kind == MK_COMPLETE {
let comp_payload = self.read_payload(comp_len)?; let comp_payload = self.read_payload(comp_len).await?;
if comp_payload.len() >= 4 { if comp_payload.len() >= 4 {
affected = u32::from_be_bytes([comp_payload[0], comp_payload[1], comp_payload[2], comp_payload[3]]) as usize; affected = u32::from_be_bytes([comp_payload[0], comp_payload[1], comp_payload[2], comp_payload[3]]) as usize;
} }
@@ -340,44 +356,47 @@ impl Client {
Ok(QueryResult { columns, column_types: col_types, rows, affected_rows: affected }) Ok(QueryResult { columns, column_types: col_types, rows, affected_rows: affected })
} }
pub fn auth(&mut self, token: &str) -> Result<(), Box<dyn std::error::Error>> { /// Authenticate with JWT token
pub async fn auth(&mut self, token: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
if !self.connected { if !self.connected {
return Err("Not connected".into()); return Err("Not connected".into());
} }
let payload = encode_string(token); let payload = encode_string(token);
let msg = build_message(MK_AUTH, self.next_id(), &payload); let msg = build_message(MK_AUTH, self.next_id(), &payload);
self.stream.write_all(&msg)?; self.stream.write_all(&msg).await?;
let (kind, length, _) = self.read_header()?; let (kind, length, _) = self.read_header().await?;
match kind { match kind {
MK_AUTH_OK => Ok(()), MK_AUTH_OK => Ok(()),
MK_ERROR => { MK_ERROR => {
let p = self.read_payload(length)?; let p = self.read_payload(length).await?;
Err(Self::read_error_message(&p).into()) Err(Self::read_error_message(&p).into())
} }
_ => Err(format!("Unexpected auth response: 0x{:02x}", kind).into()), _ => Err(format!("Unexpected auth response: 0x{:02x}", kind).into()),
} }
} }
pub fn ping(&mut self) -> Result<bool, Box<dyn std::error::Error>> { /// Ping the server
pub async fn ping(&mut self) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
if !self.connected { if !self.connected {
return Err("Not connected".into()); return Err("Not connected".into());
} }
let msg = build_message(MK_PING, self.next_id(), &[]); let msg = build_message(MK_PING, self.next_id(), &[]);
self.stream.write_all(&msg)?; self.stream.write_all(&msg).await?;
let (kind, length, _) = self.read_header()?; let (kind, length, _) = self.read_header().await?;
match kind { match kind {
MK_PONG => Ok(true), MK_PONG => Ok(true),
MK_ERROR => { MK_ERROR => {
let p = self.read_payload(length)?; let p = self.read_payload(length).await?;
Err(Self::read_error_message(&p).into()) Err(Self::read_error_message(&p).into())
} }
_ => Ok(false), _ => Ok(false),
} }
} }
pub fn query(&mut self, sql: &str) -> Result<QueryResult, Box<dyn std::error::Error>> { /// Execute a query
pub async fn query(&mut self, sql: &str) -> Result<QueryResult, Box<dyn std::error::Error + Send + Sync>> {
if !self.connected { if !self.connected {
return Err("Not connected".into()); return Err("Not connected".into());
} }
@@ -386,14 +405,14 @@ impl Client {
payload.push(0x00); // ResultFormat::BINARY payload.push(0x00); // ResultFormat::BINARY
let msg = build_message(MK_QUERY, self.next_id(), &payload); let msg = build_message(MK_QUERY, self.next_id(), &payload);
self.stream.write_all(&msg)?; self.stream.write_all(&msg).await?;
let (kind, length, _) = self.read_header()?; let (kind, length, _) = self.read_header().await?;
let resp_payload = self.read_payload(length)?; let resp_payload = self.read_payload(length).await?;
match kind { match kind {
MK_READY => Ok(QueryResult { columns: vec![], column_types: vec![], rows: vec![], affected_rows: 0 }), MK_READY => Ok(QueryResult { columns: vec![], column_types: vec![], rows: vec![], affected_rows: 0 }),
MK_DATA => self.read_data_response(&resp_payload), MK_DATA => self.read_data_response(&resp_payload).await,
MK_COMPLETE => { MK_COMPLETE => {
let affected = if resp_payload.len() >= 4 { let affected = if resp_payload.len() >= 4 {
u32::from_be_bytes([resp_payload[0], resp_payload[1], resp_payload[2], resp_payload[3]]) as usize u32::from_be_bytes([resp_payload[0], resp_payload[1], resp_payload[2], resp_payload[3]]) as usize
@@ -405,7 +424,8 @@ impl Client {
} }
} }
pub fn query_params(&mut self, sql: &str, params: &[WireValue]) -> Result<QueryResult, Box<dyn std::error::Error>> { /// Execute a parameterized query
pub async fn query_params(&mut self, sql: &str, params: &[WireValue]) -> Result<QueryResult, Box<dyn std::error::Error + Send + Sync>> {
if !self.connected { if !self.connected {
return Err("Not connected".into()); return Err("Not connected".into());
} }
@@ -418,14 +438,14 @@ impl Client {
} }
let msg = build_message(MK_QUERY_PARAMS, self.next_id(), &payload); let msg = build_message(MK_QUERY_PARAMS, self.next_id(), &payload);
self.stream.write_all(&msg)?; self.stream.write_all(&msg).await?;
let (kind, length, _) = self.read_header()?; let (kind, length, _) = self.read_header().await?;
let resp_payload = self.read_payload(length)?; let resp_payload = self.read_payload(length).await?;
match kind { match kind {
MK_READY => Ok(QueryResult { columns: vec![], column_types: vec![], rows: vec![], affected_rows: 0 }), MK_READY => Ok(QueryResult { columns: vec![], column_types: vec![], rows: vec![], affected_rows: 0 }),
MK_DATA => self.read_data_response(&resp_payload), MK_DATA => self.read_data_response(&resp_payload).await,
MK_COMPLETE => { MK_COMPLETE => {
let affected = if resp_payload.len() >= 4 { let affected = if resp_payload.len() >= 4 {
u32::from_be_bytes([resp_payload[0], resp_payload[1], resp_payload[2], resp_payload[3]]) as usize u32::from_be_bytes([resp_payload[0], resp_payload[1], resp_payload[2], resp_payload[3]]) as usize
@@ -437,12 +457,14 @@ impl Client {
} }
} }
pub fn execute(&mut self, sql: &str) -> Result<usize, Box<dyn std::error::Error>> { /// Execute and return affected rows
let result = self.query(sql)?; pub async fn execute(&mut self, sql: &str) -> Result<usize, Box<dyn std::error::Error + Send + Sync>> {
let result = self.query(sql).await?;
Ok(result.affected_rows()) Ok(result.affected_rows())
} }
} }
/// Query builder for fluent SQL construction
pub struct QueryBuilder<'a> { pub struct QueryBuilder<'a> {
client: &'a mut Client, client: &'a mut Client,
select_cols: Vec<String>, select_cols: Vec<String>,
@@ -457,6 +479,7 @@ pub struct QueryBuilder<'a> {
} }
impl<'a> QueryBuilder<'a> { impl<'a> QueryBuilder<'a> {
/// Create a new query builder
pub fn new(client: &'a mut Client) -> Self { pub fn new(client: &'a mut Client) -> Self {
QueryBuilder { QueryBuilder {
client, client,
@@ -472,56 +495,67 @@ impl<'a> QueryBuilder<'a> {
} }
} }
/// Add columns to SELECT
pub fn select(mut self, cols: &[&str]) -> Self { pub fn select(mut self, cols: &[&str]) -> Self {
self.select_cols.extend(cols.iter().map(|s| s.to_string())); self.select_cols.extend(cols.iter().map(|s| s.to_string()));
self self
} }
/// Set the FROM table
pub fn from(mut self, table: &str) -> Self { pub fn from(mut self, table: &str) -> Self {
self.from_table = table.to_string(); self.from_table = table.to_string();
self self
} }
/// Add a WHERE clause
pub fn where_clause(mut self, clause: &str) -> Self { pub fn where_clause(mut self, clause: &str) -> Self {
self.where_clauses.push(clause.to_string()); self.where_clauses.push(clause.to_string());
self self
} }
/// Add a JOIN clause
pub fn join(mut self, table: &str, on: &str) -> Self { pub fn join(mut self, table: &str, on: &str) -> Self {
self.joins.push(format!("JOIN {} ON {}", table, on)); self.joins.push(format!("JOIN {} ON {}", table, on));
self self
} }
/// Add a LEFT JOIN clause
pub fn left_join(mut self, table: &str, on: &str) -> Self { pub fn left_join(mut self, table: &str, on: &str) -> Self {
self.joins.push(format!("LEFT JOIN {} ON {}", table, on)); self.joins.push(format!("LEFT JOIN {} ON {}", table, on));
self self
} }
/// Add GROUP BY columns
pub fn group_by(mut self, cols: &[&str]) -> Self { pub fn group_by(mut self, cols: &[&str]) -> Self {
self.group_by.extend(cols.iter().map(|s| s.to_string())); self.group_by.extend(cols.iter().map(|s| s.to_string()));
self self
} }
/// Add HAVING clause
pub fn having(mut self, clause: &str) -> Self { pub fn having(mut self, clause: &str) -> Self {
self.having = clause.to_string(); self.having = clause.to_string();
self self
} }
/// Add ORDER BY column
pub fn order_by(mut self, col: &str, dir: &str) -> Self { pub fn order_by(mut self, col: &str, dir: &str) -> Self {
self.order_by.push(format!("{} {}", col, dir)); self.order_by.push(format!("{} {}", col, dir));
self self
} }
/// Set LIMIT
pub fn limit(mut self, n: usize) -> Self { pub fn limit(mut self, n: usize) -> Self {
self.limit = n; self.limit = n;
self self
} }
/// Set OFFSET
pub fn offset(mut self, n: usize) -> Self { pub fn offset(mut self, n: usize) -> Self {
self.offset = n; self.offset = n;
self self
} }
/// Build the SQL string
pub fn build(&self) -> String { pub fn build(&self) -> String {
let mut sql = String::from("SELECT "); let mut sql = String::from("SELECT ");
if self.select_cols.is_empty() { if self.select_cols.is_empty() {
@@ -555,9 +589,10 @@ impl<'a> QueryBuilder<'a> {
sql sql
} }
pub fn exec(self) -> Result<QueryResult, Box<dyn std::error::Error>> { /// Execute the query
pub async fn exec(self) -> Result<QueryResult, Box<dyn std::error::Error + Send + Sync>> {
let sql = self.build(); let sql = self.build();
self.client.query(&sql) self.client.query(&sql).await
} }
} }
+57 -42
View File
@@ -1,104 +1,118 @@
// BaraDB Rust Client — Integration Tests // BaraDB Rust Client — Integration Tests
// Requires a running BaraDB server on localhost:9472. // Requires a running BaraDB server.
// Set BARADB_HOST / BARADB_PORT env vars to override defaults.
use std::net::TcpStream; use std::net::TcpStream;
use std::time::Duration;
use baradb::{Client, QueryBuilder, WireValue}; use baradb::{Client, QueryBuilder, WireValue};
const HOST: &str = "127.0.0.1"; fn host() -> String {
const PORT: u16 = 9472; std::env::var("BARADB_HOST").unwrap_or_else(|_| "127.0.0.1".to_string())
fn server_available() -> bool {
TcpStream::connect((HOST, PORT)).is_ok()
} }
#[test] fn port() -> u16 {
fn test_connect_and_close() { std::env::var("BARADB_PORT")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(9472)
}
fn server_available() -> bool {
TcpStream::connect((host().as_str(), port())).is_ok()
}
#[tokio::test]
async fn test_connect_and_close() {
if !server_available() { if !server_available() {
return; return;
} }
let mut client = Client::connect(HOST, PORT).unwrap(); let mut client = Client::connect(&host(), port()).await.unwrap();
assert!(client.is_connected()); assert!(client.is_connected());
client.close(); client.close().await;
assert!(!client.is_connected()); assert!(!client.is_connected());
} }
#[test] #[tokio::test]
fn test_ping() { async fn test_ping() {
if !server_available() { if !server_available() {
return; return;
} }
let mut client = Client::connect(HOST, PORT).unwrap(); let mut client = Client::connect(&host(), port()).await.unwrap();
assert!(client.ping().unwrap()); assert!(client.ping().await.unwrap());
client.close(); client.close().await;
} }
#[test] #[tokio::test]
fn test_simple_select() { async fn test_simple_select() {
if !server_available() { if !server_available() {
return; return;
} }
let mut client = Client::connect(HOST, PORT).unwrap(); let mut client = Client::connect(&host(), port()).await.unwrap();
let result = client.query("SELECT 1 as one").unwrap(); let result = client.query("SELECT 1 as one").await.unwrap();
assert!(result.row_count() >= 0); assert!(result.row_count() >= 0);
client.close(); client.close().await;
} }
#[test] #[tokio::test]
fn test_parameterized_query() { async fn test_parameterized_query() {
if !server_available() { if !server_available() {
return; return;
} }
let mut client = Client::connect(HOST, PORT).unwrap(); let mut client = Client::connect(&host(), port()).await.unwrap();
let result = client let result = client
.query_params( .query_params(
"SELECT $1 as num, $2 as txt", "SELECT $1 as num, $2 as txt",
&[WireValue::Int64(42), WireValue::String("hello".to_string())], &[WireValue::Int64(42), WireValue::String("hello".to_string())],
) )
.await
.unwrap(); .unwrap();
assert!(result.row_count() >= 0); assert!(result.row_count() >= 0);
client.close(); client.close().await;
} }
#[test] #[tokio::test]
fn test_create_table_insert_select_drop() { async fn test_create_table_insert_select_drop() {
if !server_available() { if !server_available() {
return; return;
} }
let mut client = Client::connect(HOST, PORT).unwrap(); let mut client = Client::connect(&host(), port()).await.unwrap();
let _ = client.execute("DROP TABLE IF EXISTS rust_test_users"); let _ = client.execute("DROP TABLE IF EXISTS rust_test_users").await;
client client
.execute("CREATE TABLE rust_test_users (id INT PRIMARY KEY, name STRING, age INT)") .execute("CREATE TABLE rust_test_users (id INT PRIMARY KEY, name STRING, age INT)")
.await
.unwrap(); .unwrap();
let affected = client let affected = client
.execute("INSERT INTO rust_test_users (id, name, age) VALUES (1, 'Alice', 30)") .execute("INSERT INTO rust_test_users (id, name, age) VALUES (1, 'Alice', 30)")
.await
.unwrap(); .unwrap();
assert!(affected >= 0); assert!(affected >= 0);
let result = client let result = client
.query("SELECT name, age FROM rust_test_users WHERE id = 1") .query("SELECT name, age FROM rust_test_users WHERE id = 1")
.await
.unwrap(); .unwrap();
assert_eq!(result.row_count(), 1); assert_eq!(result.row_count(), 1);
client.execute("DROP TABLE rust_test_users").unwrap(); client.execute("DROP TABLE rust_test_users").await.unwrap();
client.close(); client.close().await;
} }
#[test] #[tokio::test]
fn test_query_builder_exec() { async fn test_query_builder_exec() {
if !server_available() { if !server_available() {
return; return;
} }
let mut client = Client::connect(HOST, PORT).unwrap(); let mut client = Client::connect(&host(), port()).await.unwrap();
let _ = client.execute("DROP TABLE IF EXISTS rust_test_products"); let _ = client.execute("DROP TABLE IF EXISTS rust_test_products").await;
client client
.execute("CREATE TABLE rust_test_products (id INT PRIMARY KEY, name STRING, price FLOAT)") .execute("CREATE TABLE rust_test_products (id INT PRIMARY KEY, name STRING, price FLOAT)")
.await
.unwrap(); .unwrap();
client client
.execute("INSERT INTO rust_test_products (id, name, price) VALUES (1, 'Widget', 9.99)") .execute("INSERT INTO rust_test_products (id, name, price) VALUES (1, 'Widget', 9.99)")
.await
.unwrap(); .unwrap();
let result = QueryBuilder::new(&mut client) let result = QueryBuilder::new(&mut client)
@@ -106,21 +120,22 @@ fn test_query_builder_exec() {
.from("rust_test_products") .from("rust_test_products")
.where_clause("id = 1") .where_clause("id = 1")
.exec() .exec()
.await
.unwrap(); .unwrap();
assert_eq!(result.row_count(), 1); assert_eq!(result.row_count(), 1);
client.execute("DROP TABLE rust_test_products").unwrap(); client.execute("DROP TABLE rust_test_products").await.unwrap();
client.close(); client.close().await;
} }
#[test] #[tokio::test]
fn test_auth_dummy_token() { async fn test_auth_dummy_token() {
if !server_available() { if !server_available() {
return; return;
} }
let mut client = Client::connect(HOST, PORT).unwrap(); let mut client = Client::connect(&host(), port()).await.unwrap();
let res = client.auth("dummy-token-for-testing"); let res = client.auth("dummy-token-for-testing").await;
// Dev server may accept or reject — both are fine // Dev server may accept or reject — both are fine
match res { match res {
Ok(()) => {} Ok(()) => {}
@@ -129,5 +144,5 @@ fn test_auth_dummy_token() {
assert!(msg.contains("Auth") || msg.to_lowercase().contains("error")); assert!(msg.contains("Auth") || msg.to_lowercase().contains("error"));
} }
} }
client.close(); client.close().await;
} }
+51
View File
@@ -0,0 +1,51 @@
# BaraDB + Ormin Development Stack
#
# Usage:
# docker compose -f docker-compose.ormin.yml up --build -d
# docker compose -f docker-compose.ormin.yml exec ormin-dev bash
#
# Inside the container:
# nim c -r examples/baradb_basic.nim
services:
baradb:
build:
context: .
dockerfile: Dockerfile
image: baradb:latest
container_name: baradb-ormin
hostname: baradb
ports:
- "9472:9472"
healthcheck:
test: ["CMD", "sh", "-c", "wget -qO- http://localhost:9912/health >/dev/null 2>&1"]
interval: 5s
timeout: 3s
retries: 5
start_period: 10s
networks:
- ormin_net
ormin-dev:
build:
context: .
dockerfile: clients/nim/ormin/docker/Dockerfile
container_name: ormin-dev
depends_on:
baradb:
condition: service_healthy
environment:
- BARADB_HOST=baradb
- BARADB_PORT=9472
volumes:
- ./clients/nim/ormin:/workspace/ormin
- ./clients/nim/src:/workspace/baradb/src
- ./clients/nim/baradb.nimble:/workspace/baradb/baradb.nimble
working_dir: /workspace/ormin
command: tail -f /dev/null
networks:
- ormin_net
networks:
ormin_net:
driver: bridge
+118
View File
@@ -0,0 +1,118 @@
# BaraDB — Client Integration Test Stack
#
# This compose file starts BaraDB server and runs all client test suites.
# Because test containers run in parallel, do NOT use --abort-on-container-exit
# (the first finished container would kill the others).
#
# Recommended usage:
# ./scripts/test-clients.sh
#
# Or manually run each suite:
# docker compose -f docker-compose.test.yml up -d baradb
# docker compose -f docker-compose.test.yml run --rm test-python
# docker compose -f docker-compose.test.yml run --rm test-javascript
# docker compose -f docker-compose.test.yml run --rm test-nim
# docker compose -f docker-compose.test.yml run --rm test-rust
# docker compose -f docker-compose.test.yml down
services:
baradb:
build:
context: .
dockerfile: Dockerfile
image: baradb:latest
ports:
- "9472:9472"
healthcheck:
test: ["CMD", "sh", "-c", "wget -qO- http://localhost:9912/health >/dev/null 2>&1"]
interval: 5s
timeout: 3s
retries: 5
start_period: 10s
test-python:
image: python:3.12-slim
depends_on:
baradb:
condition: service_healthy
environment:
- BARADB_HOST=baradb
- BARADB_PORT=9472
volumes:
- ./clients/python:/workspace
working_dir: /workspace
command: >
sh -c "
pip install pytest pytest-asyncio -q &&
pip install -e ".[dev]" -q &&
pytest tests/test_wire_protocol.py tests/test_query_builder.py -v &&
pytest tests/test_integration.py -v
"
test-javascript:
image: node:20-slim
depends_on:
baradb:
condition: service_healthy
environment:
- BARADB_HOST=baradb
- BARADB_PORT=9472
volumes:
- ./clients/javascript:/workspace
working_dir: /workspace
command: >
sh -c "
node --test tests/unit.test.js &&
node --test tests/integration.test.js
"
test-nim:
image: nimlang/nim:2.2.10
depends_on:
baradb:
condition: service_healthy
environment:
- BARADB_HOST=baradb
- BARADB_PORT=9472
volumes:
- ./clients/nim:/workspace
- ./clients/nim/src:/workspace/src
working_dir: /workspace
command: >
sh -c "
nim c --path:src -r tests/test_client.nim &&
nim c --path:src -r tests/test_integration.nim
"
test-ormin:
image: nimlang/nim:2.2.10
depends_on:
baradb:
condition: service_healthy
environment:
- BARADB_HOST=baradb
- BARADB_PORT=9472
volumes:
- ./clients/nim:/workspace
working_dir: /workspace/ormin
command: >
sh -c "
nimble install -y &&
nim c examples/baradb_basic.nim
"
test-rust:
image: rust:1.78
depends_on:
baradb:
condition: service_healthy
environment:
- BARADB_HOST=baradb
- BARADB_PORT=9472
volumes:
- ./clients/rust:/workspace
working_dir: /workspace
command: >
sh -c "
cargo test -- --test-threads=1
"
+6
View File
@@ -90,6 +90,12 @@ The query layer processes BaraQL — a SQL-compatible query language with extens
- **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression. - **Quantization** (`quant.nim`): Scalar 8-bit/4-bit, product, and binary quantization for compression.
- **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan). - **SIMD Operations** (`simd.nim`): Unrolled loop distance computations (cosine, Euclidean, dot product, Manhattan).
- **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput. - **Batch Operations**: batchInsert, batchSearch, batchDistance for high-throughput.
- **SQL Integration** (`query/executor.nim`):
- `VECTOR(n)` column type with dimension validation
- `CREATE INDEX ... USING hnsw` / `USING ivfpq`
- Distance functions: `cosine_distance()`, `euclidean_distance()`, `inner_product()`, `l1_distance()`, `l2_distance()`
- `<->` nearest-neighbor operator
- Automatic index maintenance on INSERT/UPDATE
### Graph Engine (`graph/`) ### Graph Engine (`graph/`)
- **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency. - **Adjacency List** (`engine.nim`): Edge-weighted directed graph storage with forward/reverse adjacency.
+167
View File
@@ -97,5 +97,172 @@ RETURN friend.name;
```sql ```sql
BEGIN; BEGIN;
INSERT users { name := 'Alice', age := 30 }; INSERT users { name := 'Alice', age := 30 };
INSERT orders { user_id := last_insert_id(), total := 100 };
COMMIT;
-- مع نقطة حفظ
BEGIN;
INSERT users { name := 'Bob', age := 25 };
SAVEPOINT sp1;
INSERT orders { user_id := last_insert_id(), total := 200 };
-- خطأ، التراجع إلى نقطة الحفظ
ROLLBACK TO sp1;
COMMIT; COMMIT;
``` ```
## البحث النصي الكامل
```sql
-- بحث أساسي
SELECT * FROM articles
WHERE MATCH(title, body) AGAINST('database programming');
-- مع درجة الصلة
SELECT title, relevance()
FROM articles
WHERE MATCH(title, body) AGAINST('Nim language')
ORDER BY relevance() DESC;
-- الوضع المنطقي
SELECT * FROM articles
WHERE MATCH(title, body) AGAINST('+Nim -Python' IN BOOLEAN MODE);
-- البحث الضبابي
SELECT * FROM articles
WHERE MATCH(title) AGAINST('programing' WITH FUZZINESS 2);
```
## الدوال المحددة من المستخدم
```sql
-- تسجيل UDF
CREATE FUNCTION greet(name str) -> str {
RETURN 'Hello, ' || name || '!';
};
-- استخدامها
SELECT greet(name) FROM users;
-- الدوال المدمجة
SELECT abs(-5), sqrt(16), lower('HELLO'), len('test');
```
## تلميحات الاستعلام
```sql
-- فرض استخدام الفهرس
SELECT /*+ USE_INDEX(idx_users_age) */ * FROM users WHERE age > 18;
-- فرض البحث المتجهي التقريبي
SELECT /*+ APPROXIMATE */ * FROM vectors
ORDER BY cosine_distance(embedding, [...])
LIMIT 10;
-- التنفيذ المتوازي
SELECT /*+ PARALLEL(4) */ * FROM large_table;
```
## دوال النوافذ
```sql
-- دوال الترتيب
SELECT
name,
department,
ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) AS rn,
RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS r,
DENSE_RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS dr
FROM employees;
-- دوال القيمة
SELECT
name,
salary,
LAG(salary, 1, 0) OVER (ORDER BY salary) AS prev_salary,
LEAD(salary, 1, 0) OVER (ORDER BY salary) AS next_salary,
FIRST_VALUE(name) OVER (PARTITION BY department ORDER BY salary) AS cheapest,
LAST_VALUE(name) OVER (PARTITION BY department ORDER BY salary) AS most_expensive
FROM employees;
-- دوال التوزيع
SELECT name, NTILE(4) OVER (ORDER BY salary) AS quartile FROM employees;
```
### مواصفات الإطار
```sql
-- إطار ROWS
SUM(salary) OVER (
PARTITION BY department
ORDER BY hire_date
ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
)
-- إطار RANGE
SUM(salary) OVER (
PARTITION BY department
ORDER BY hire_date
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
)
```
## ERP متعدد المستأجرين
يدعم BaraDB تشغيل عدة شركات (مستأجرين) في مثيل قاعدة بيانات واحد، باستخدام **أمان مستوى الصف (RLS)** مع **متغيرات الجلسة**.
### متغيرات الجلسة
```sql
SET app.tenant_id = 'company-123';
SELECT current_setting('app.tenant_id') AS tenant;
```
### المستخدم / الدور الحالي
```sql
SELECT current_user AS me, current_role AS my_role;
```
### عزل المستأجر عبر RLS
```sql
-- تمكين RLS على جدول
ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
-- إنشاء سياسة تصفية حسب المستأجر
CREATE POLICY tenant_isolation ON invoices
FOR SELECT USING (tenant_id = current_setting('app.tenant_id'));
-- كل جلسة ترى فقط بياناتها
SET app.tenant_id = 'company-a';
SELECT * FROM invoices; -- صفوف company-a فقط
```
### لماذا متعدد المستأجرين؟
- **مثيل واحد، مستأجرون كثيرون** — لا حاجة لتشغيل 100 قاعدة بيانات منفصلة
- **مستندات JSONB** — تخزين مخطط مرن، سهل إضافة حقول لكل مستأجر
- **RLS يضمن العزل** — قاعدة البيانات تفرض حدود المستأجر، وليس فقط التطبيق
## الكلمات المفتاحية المدعومة
| الفئة | الكلمات المفتاحية |
|----------|----------|
| DQL | SELECT, FROM, WHERE, ORDER BY, GROUP BY, HAVING, LIMIT, OFFSET, DISTINCT |
| DML | INSERT, UPDATE, DELETE, SET, VALUES |
| DDL | CREATE TYPE, DROP TYPE, CREATE INDEX, DROP INDEX, ALTER TYPE |
| Join | INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN, CROSS JOIN, ON |
| Set | UNION, UNION ALL, INTERSECT, EXCEPT |
| CTEs | WITH, RECURSIVE, AS |
| Case | CASE, WHEN, THEN, ELSE, END |
| Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT |
| Graph | MATCH, RETURN, WHERE, shortestPath, type |
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
| Vector | cosine_distance, euclidean_distance, inner_product, l1_distance, l2_distance, <-> |
| JSON | ->, ->> |
| FTS | @@ (تطابق BM25) |
| Recovery | RECOVER TO TIMESTAMP |
| Functions | count, sum, avg, min, max, stddev, variance, abs, sqrt, lower, upper, len, trim, substr, now, last_insert_id, current_setting |
| Session | SET, current_setting, current_user, current_role |
| Window | OVER, PARTITION BY, ROWS, RANGE, UNBOUNDED PRECEDING, CURRENT ROW, FOLLOWING |
| Window Functions | ROW_NUMBER, RANK, DENSE_RANK, LEAD, LAG, FIRST_VALUE, LAST_VALUE, NTILE |

Some files were not shown because too many files have changed in this diff Show More