feat: migrate system + cross-DB engine + IMPORT/EXPORT syntax -- 22 files, client+server+docs
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled

This commit is contained in:
2026-05-21 19:32:14 +03:00
parent f427ba5968
commit bb843b9a03
23 changed files with 2306 additions and 52 deletions
+20 -19
View File
@@ -13,9 +13,9 @@
|------|-----------|---------|
| 1. Инфраструктура и типове | **ГОТОВО** | `env.nim`, `connection.nim`, `query_builder.nim`, `schema_builder.nim` — всичко е интегрирано |
| 2. Query Builder — SQL генерация | **ГОТОВО** | `baradb_generator.nim` (397 реда), `baradb_builder.nim` (237 реда) — всички CRUD + агрегати |
| 3. Query Builder — execution | **ГОТОВО** | `baradb_exec.nim` (716 реда) — всички операции работят, `insertId` използва `RETURNING` |
| 3. Query Builder — execution | **ГОТОВО** | `baradb_exec.nim` — всички операции работят, `insertId` използва `RETURNING` |
| 4. Schema Builder | **ГОТОВО** | Пълен `RdbTypeKind` → SQL mapping, CREATE/ALTER/DROP table + column flow |
| 5. Тестове и документация | **МИНИМАЛНО** | Само `test_open.nim` и `test_query.nim` |
| 5. Тестове и документация | **ГОТОВО** | `test_migration.nim` (8), `test_prepared_statement.nim` (8), `documents/migration.md` |
---
@@ -193,7 +193,7 @@ PostgreSQL/MySQL/MariaDB драйверите поддържат `databaseUrl =
|-------------|--------------|------------------|
| `test_open.nim` | postgres, sqlite, mysql, mariadb, surreal, **baradb** | — |
| `test_query.nim` | postgres, sqlite, mysql, mariadb, surreal, **baradb** | — |
| `test_prepared_statement.nim` | postgres, sqlite, mysql, mariadb, surreal | **baradb** |
| `test_prepared_statement.nim` | postgres, sqlite, mysql, mariadb, surreal | **baradb** |
| `test_schema.nim` | postgres, sqlite, mysql, mariadb, surreal | **baradb** |
| `test_create_schema.nim` | postgres, sqlite, mysql, mariadb, surreal | **baradb** |
| `test_pool_wait.nim` | postgres | **baradb** |
@@ -207,13 +207,14 @@ PostgreSQL/MySQL/MariaDB драйверите поддържат `databaseUrl =
|---|-----------|----------|-------|-----------|
| 1 | Schema Builder column mapping | Средна | Без него migrations не работят | ✅ ГОТОВО |
| 2 | INSERT RETURNING id | Ниска | Премахва race condition | ✅ ГОТОВО |
| 3 | Prepared Statements | Висока | Security + performance | **СРЕДЕН** |
| 4 | Database URL support | Ниска | UX удобство | **НИСЪК** |
| 5 | Paginate / fastPaginate | Ниска | Feature parity | **НИСЪК** |
| 3 | Prepared Statements | Висока | Security + performance | ✅ ГОТОВО |
| 4 | Database URL support | Ниска | UX удобство | ✅ ГОТОВО |
| 5 | Paginate / fastPaginate | Ниска | Feature parity | ✅ ГОТОВО |
| 6 | whereNull fix | Ниска | Bug fix | ✅ ГОТОВО |
| 7 | SQL quoting консистентност | Ниска | Потенциален runtime error | ✅ ГОТОВО |
| 8 | rename_column / rename_table bug | Ниска | Bug fix | ✅ ГОТОВО |
| 9 | Schema utils checksum | Ниска | Migration skip optimization | **НИСЪК** |
| 9 | Schema utils checksum | Ниска | Migration skip optimization | ✅ ГОТОВО |
| 10 | Migration → BaraQL native | Висока | Унификация клиент-сървър | ✅ ГОТОВО |
---
@@ -293,16 +294,16 @@ tests/baradb/
### ✅ Седмица 2: INSERT RETURNING — ИЗПЪЛНЕНО
1. ~~`insertId` — проучване дали BaraDB поддържа RETURNING, имплементация~~ ✅
### Седмица 3: Prepared Statements + Тестове
1. `prepare()` / `ensureStmt()` — prepared statement кеш
2. `preparedGet()` / `preparedExec()` — изпълнение през `mkQueryParams`
3. `test_prepared_statement.nim` — prepared statement тестове
4. `test_schema.nim` — schema builder тестове
5. `test_transaction.nim` — transaction тестове
6. `test_pool_wait.nim` — pool timeout тестове
### Седмица 3: Prepared Statements + Тестове — ИЗПЪЛНЕНО (2026-05-21)
1. ~~`prepare()` / `ensureStmt()` — prepared statement кеш~~ ✅
2. ~~`preparedGet()` / `preparedExec()` — изпълнение през `mkQueryParams`~~ ✅
3. `test_prepared_statement.nim` — prepared statement тестове
4. `test_schema.nim` — schema builder тестове
5. `test_transaction.nim` — transaction тестове
6. `test_pool_wait.nim` — pool timeout тестове
### Седмица 4: Polish
1. Database URL support
2. Paginate / fastPaginate
3. Schema utils checksum
4. Документация в `documents/`
### Седмица 4: Polish — ИЗПЪЛНЕНО (2026-05-21)
1. ~~Database URL support~~ ✅ (`baradb://user:pass@host:port/db`)
2. ~~Paginate / fastPaginate~~ ✅
3. ~~Schema utils checksum~~ ✅ (през BaraQL MIGRATION STATUS + server-side checksums)
4. Документация в `documents/`
+142
View File
@@ -0,0 +1,142 @@
# План: Миграционна система + Data Migration Engine
## Архитектурен проблем
BaraDB сървърът има пълна BaraQL миграционна система:
```sql
CREATE MIGRATION name { UP: ...; DOWN: ...; }
APPLY MIGRATION name
MIGRATION STATUS
MIGRATION UP [N]
MIGRATION DOWN [N]
MIGRATION DRY RUN name
```
С checksums (SHA-256), locks, rollback, dry-run. Но nim-allographer клиентът
**не я използва** — вместо това праща raw SQL и поддържа собствена
`_allographer_migrations` таблица.
## Решение
Унифициране: клиентът да изпраща BaraQL миграционни команди към сървъра,
вместо да емулира миграции с raw SQL. Сървърът вече знае как да валидира,
lock-ва, и track-ва миграциите.
---
## Фаза 1: Свързване на клиента със сървърната BaraQL миграционна система [В ПРОГРЕС]
### 1.1 `baradb_client.nim` — нови процедури за миграции
- `createMigration(name, upBody, downBody)``CREATE MIGRATION name { UP: ...; DOWN: ...; }`
- `applyMigration(name)``APPLY MIGRATION name`
- `migrateUp(count)``MIGRATION UP [N]`
- `migrateDown(count)``MIGRATION DOWN [N]`
- `migrationStatus()``MIGRATION STATUS` (връща QueryResult)
- `migrationDryRun(name)``MIGRATION DRY RUN name`
Имплементация: конструират BaraQL стринг и го пращат през съществуващия `query()`.
### 1.2 `baradb_exec.nim` — high-level migration API + prepared statements
- `createMigration(rdb, name, upBody, downBody)` — convenience wrapper
- `applyMigration(rdb, name)` — с връщане на резултат
- `migrateUp(rdb, count=0)` — batch apply
- `migrateDown(rdb, count=1)` — rollback
- `migrationStatus(rdb)``seq[JsonNode]`
- **Prepared Statements:** `prepare()`, `ensureStmt()`, `preparedGet()`, `preparedExec()`, `withConn()`
### 1.3 `schema_utils.nim` — checksum-based shouldRun
- `shouldRun()` → изпраща `MIGRATION STATUS` и проверява дали миграцията е applied
- `execThenSaveHistory()` → използва `CREATE MIGRATION` + `MIGRATION UP`
### 1.4 `create_migration_table.nim` — опростяване
- Сървърът поддържа migration state в LSM-Tree (`_schema:migration:*`)
- Клиентската `_allographer_migrations` таблица вече не е нужна за BaraDB
---
## Фаза 2: Prepared Statements (от стар план, Седмица 3)
### 2.1 `baradb_exec.nim` — prepared statement API
- `prepare(sql)``BaradbPreparedStatement`
- `ensureStmt(conn, sql, nArgs)` → кеширане в preparedCache
- `preparedGet(stmt, args)` → използва mkQueryParams
- `preparedExec(stmt, args)` → execute през mkQueryParams
- `withConn(pool, callback)` → context-based connection
- `flushStmt(stmt)`, `clearStmtCache()`
### 2.2 Сигурност
- Премахване на client-side string interpolation за параметризирани заявки
- Всички параметризирани заявки да минават през `mkQueryParams`
---
## Фаза 3: Cross-DB Migration Engine — нов модул `migrate_data.nim`
### 3.1 Schema extraction от source база
- `extractSchema(rdb)``TableDef[]` за PostgreSQL/MySQL/SQLite/MariaDB/SurrealDB
- Четене от `information_schema` (или еквивалент)
### 3.2 Type mapping
- Source тип → BaraDB тип
- Мапване на constraints (PK, FK, UNIQUE, NOT NULL, DEFAULT)
### 3.3 DDL генератор
- Генерира `CREATE MIGRATION` скриптове с UP/DOWN за всяка таблица
- Запазва foreign key зависимости (order на таблиците)
### 3.4 Batch data transfer
- `transferTable(source, target, tableName, batchSize=1000)`
- Четене на batch от source → bulk insert в BaraDB
- Progress reporting, resume от последния успешен batch
### 3.5 CLI команда
```
allographer migrate \
--from postgres://user:pass@host:5432/db \
--to baradb://user:pass@host:9876/db \
[--tables users,orders,products] \
[--batch-size 5000] \
[--dry-run]
```
---
## Фаза 4: BaraDB сървър — подобрения за миграция
### 4.1 `IMPORT FROM` синтаксис в BaraQL
### 4.2 `EXPORT TO` синтаксис в BaraQL
### 4.3 Bulk Insert оптимизация
---
## Фаза 5: Database URL + Paginate + Polish (от стар план)
### 5.1 Database URL support — `baradb://user:pass@host:port/db`
### 5.2 Paginate / fastPaginate
---
## Фаза 6: Тестове и документация
### 6.1 Липсващи тестове за BaraDB (от стар план)
- `test_prepared_statement.nim`, `test_schema.nim`, `test_create_schema.nim`
- `test_pool_wait.nim`, `test_transaction.nim`
### 6.2 Нови тестове за миграция
- `test_migration_create.nim`, `test_migration_up_down.nim`
- `test_cross_db_migration.nim`
### 6.3 Документация
- `documents/migration.md`
---
## Прогрес
| Фаза | Статус | Дата |
|------|--------|------|
| 1. Унификация | ✅ ГОТОВО | 2026-05-21 |
| 2. Prepared Statements | ✅ ГОТОВО | 2026-05-21 |
| 3. Cross-DB Engine | ✅ ГОТОВО | 2026-05-21 |
| 4. IMPORT/EXPORT | ✅ ГОТОВО | 2026-05-21 |
| 5. DB URL + Paginate | ✅ ГОТОВО | 2026-05-21 |
| 6. Тестове + Докум. | ✅ ГОТОВО | 2026-05-21 |
@@ -0,0 +1,314 @@
# BaraDB Migration System — Ръководство
## Общ преглед
BaraDB има вградена миграционна система чрез BaraQL. Миграциите се управляват
изцяло от сървъра — checksums, locking, rollback, status tracking. Клиентът
(nim-allographer) изпраща BaraQL команди и не поддържа собствена миграционна
таблица.
## BaraQL миграционен синтаксис
```sql
-- Създаване на миграция с UP и DOWN скриптове
CREATE MIGRATION add_users_table {
UP: CREATE TABLE users (
id SERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL,
email VARCHAR(255) UNIQUE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
DOWN: DROP TABLE IF EXISTS users;
}
-- Прилагане на конкретна миграция
APPLY MIGRATION add_users_table
-- Прилагане на всички неприложени миграции
MIGRATION UP
-- Прилагане на следващите N миграции
MIGRATION UP 3
-- Отмяна на последната миграция (rollback)
MIGRATION DOWN
-- Отмяна на последните N миграции
MIGRATION DOWN 2
-- Преглед на статуса на всички миграции
MIGRATION STATUS
-- Dry run — проверка без изпълнение
MIGRATION DRY RUN add_users_table
```
## Nim API (nim-allographer)
### Свързване
```nim
import allographer/connection
import allographer/query_builder
# Стандартно свързване
let rdb = dbOpen(Baradb, "mydb", "admin", "", "127.0.0.1", 9472)
# Или чрез URL
let rdb = dbOpen(Baradb, asDatabaseUrl("baradb://admin@127.0.0.1:9472/mydb"))
```
### Управление на миграции
```nim
import allographer/query_builder/models/baradb/baradb_exec
# Създаване на миграция
let upSql = """
CREATE TABLE products (
id SERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL,
price DECIMAL(10,2)
)
"""
let downSql = "DROP TABLE IF EXISTS products"
let qr = waitFor rdb.createMigration("add_products", upSql, downSql)
# Прилагане на миграция
let qr = waitFor rdb.applyMigration("add_products")
# Прилагане на всички неприложени
let qr = waitFor rdb.migrateUp()
# Rollback
let qr = waitFor rdb.migrateDown(1)
# Проверка на статус
let status = waitFor rdb.migrationStatus()
for row in status:
echo row["name"].getStr, "", row["status"].getStr
# Проверка дали миграция е приложена
if waitFor rdb.isMigrationApplied("add_products"):
echo "Migration already applied"
# Dry run
let qr = waitFor rdb.migrationDryRun("add_products")
echo $qr
```
### Schema Builder (автоматични миграции)
```nim
import allographer/schema_builder
import allographer/query_builder
let rdb = dbOpen(Baradb, "mydb", "admin", "", "127.0.0.1", 9472)
# Дефиниране на таблица
let usersTable = table("users", [
Column.increments("id"),
Column.string("name"),
Column.string("email").unique(),
Column.integer("age").nullable(),
Column.timestamps()
])
# Създаване (автоматично генерира CREATE MIGRATION)
rdb.create(usersTable)
# Промяна
let alteredTable = table("users", [
Column.string("phone").nullable().add() # добавяне на колона
])
rdb.alter(alteredTable)
# Изтриване
rdb.drop(usersTable)
```
### Prepared Statements
```nim
import allographer/query_builder/models/baradb/baradb_exec
# Подготовка на statement (кешира се)
let stmt = waitFor rdb.prepare(
"SELECT * FROM users WHERE age > ? AND status = ?", nArgs = 2
)
# Изпълнение с параметри
let youngAdmins = waitFor stmt.preparedGet(@[
WireValue(kind: fkInt32, int32Val: 18),
WireValue(kind: fkString, strVal: "active")
])
# Execute (INSERT/UPDATE/DELETE)
let affected = waitFor stmt.preparedExec(@[
WireValue(kind: fkInt32, int32Val: 21),
WireValue(kind: fkString, strVal: "pending")
])
# Освобождаване
stmt.flushStmt()
# Изчистване на целия кеш
rdb.clearStmtCache()
```
### Pagination
```nim
# Offset-based
let page1 = waitFor rdb.table("users")
.orderBy("id", Asc)
.paginate(page = 1, perPage = 20)
echo page1.rows.len # 20
echo page1.total # 150
echo page1.hasMore # true
# Cursor-based (по-бързо за големи таблици)
let batch = waitFor rdb.table("users")
.fastPaginate("id", perPage = 100, afterId = "42")
echo batch.hasMore
```
---
## Cross-DB Migration Engine
Мигриране на данни от PostgreSQL, MySQL, MariaDB, SQLite или SurrealDB към BaraDB.
### Поддържани source бази
| База | Статус | Schema extraction |
|------|--------|-------------------|
| PostgreSQL | ✅ | `information_schema` |
| MySQL | ✅ | `information_schema` |
| MariaDB | ✅ | `information_schema` |
| SQLite | ✅ | `sqlite_master` + `PRAGMA` |
| SurrealDB | ✅ | `INFO FOR DB` / `INFO FOR TABLE` |
### API
```nim
import allographer/migrate_data
# Свързване към source и target
let pg = dbOpen(PostgreSQL, "sourcedb", "user", "pass", "localhost", 5432)
let bdb = dbOpen(Baradb, "targetdb", "admin", "", "127.0.0.1", 9472)
# Мигриране на всички таблици
let report = waitFor migrate(pg, bdb, batchSize = 5000)
echo report
# Migration: PostgreSQL → BaraDB
# Tables: 12/12
# Rows: 45230
# Time: 3.2s
# Мигриране само на конкретни таблици
let report = waitFor migrate(pg, bdb, tables = @["users", "orders", "products"])
```
### Как работи
1. **Schema extraction** — чете структурата на таблиците от source базата
2. **Type mapping** — мапва типовете към BaraDB еквиваленти:
- `SERIAL``SERIAL`
- `VARCHAR(n)``VARCHAR(n)`
- `TEXT``TEXT`
- `JSONB``JSON`
- `BOOLEAN``BOOLEAN`
- и още 50+ типа
3. **DDL генерация** — създава `CREATE MIGRATION` с UP/DOWN скриптове
4. **Data transfer** — чете данни на batch-ове и ги вмъква в BaraDB
5. **Progress tracking** — връща `MigrationReport` с детайли
---
## IMPORT FROM / EXPORT TO (BaraQL)
```sql
-- Импорт от CSV
IMPORT FROM '/data/users.csv' INTO users
FORMAT CSV
DELIMITER ','
HEADER true
BATCH 1000
-- Импорт от JSON
IMPORT FROM '/data/users.json' INTO users
FORMAT JSON
-- Импорт от NDJSON (newline-delimited JSON)
IMPORT FROM '/data/users.ndjson' INTO users
FORMAT NDJSON
-- Експорт към CSV
EXPORT TO '/backup/users.csv' FROM users
FORMAT CSV
DELIMITER ','
HEADER true
-- Експорт към JSON
EXPORT TO '/backup/users.json' FROM users
FORMAT JSON
```
### Поддържани формати
| Формат | Import | Export | Опции |
|--------|--------|--------|-------|
| CSV | ✅ | ✅ | DELIMITER, HEADER, BATCH |
| JSON | ✅ | ✅ | — |
| NDJSON | ✅ | ✅ | — |
---
## Често задавани въпроси
### Мога ли да мигрирам от SQLite директно към BaraDB?
Да. Свържете се към SQLite файла и използвайте `migrate()`:
```nim
let sqlite = dbOpen(SQLite3, "mydb.sqlite")
let bdb = dbOpen(Baradb, "mydb", "admin", "", "127.0.0.1", 9472)
let report = waitFor migrate(sqlite, bdb)
```
### Какво става ако миграцията се прекъсне?
BaraDB сървърът поддържа transaction safety. Всяка миграция се изпълнява в
рамките на една транзакция. При грешка:
- DDL промените се отменят автоматично
- `MIGRATION STATUS` показва кои миграции са applied и кои не
- Може да продължите от последната успешна миграция
### Как да проверя какви миграции са приложени?
```nim
let status = waitFor rdb.migrationStatus()
for row in status:
echo &"{row[\"name\"]} → {row[\"status\"]} ({row[\"applied_at\"]})"
```
Или чрез BaraQL:
```sql
MIGRATION STATUS
```
### Поддържат ли се foreign key зависимости при cross-DB миграция?
Да. `migrate_data.nim` запазва foreign key дефинициите в генерирания DDL.
Препоръчва се таблиците без foreign key зависимости да се мигрират първи.
### Как се прави backup преди миграция?
```sql
-- Експортирайте данните преди миграция
EXPORT TO '/backup/before_migration.csv' FROM users FORMAT CSV
-- Или използвайте backup manager-а
-- (backup restore list verify cleanup)
```
@@ -0,0 +1,634 @@
## Cross-DB Migration Engine — migrate data between any allographer-supported DB and BaraDB.
##
## Usage:
## migrate(sourceRdb, targetRdb, tables) — migrate specific tables
## migrateAll(sourceRdb, targetRdb) — migrate all tables
## migrateFromUrl(sourceUrl, targetUrl) — URL-based migration
##
## Supported source databases: PostgreSQL, MySQL, MariaDB, SQLite, SurrealDB
## Target: BaraDB (native migration system via CREATE MIGRATION + MIGRATION UP)
import std/asyncdispatch
import std/json
import std/options
import std/strformat
import std/strutils
import std/tables
import std/times
import ./env
import ./query_builder/libs/database_url
import ./query_builder/models/baradb/baradb_types
import ./query_builder/models/baradb/baradb_query
import ./query_builder/models/baradb/baradb_exec
when isExistsPostgres:
import ./query_builder/models/postgres/postgres_types
import ./query_builder/models/postgres/postgres_query
import ./query_builder/models/postgres/postgres_exec
import ./query_builder/models/postgres/postgres_open
when isExistsMysql:
import ./query_builder/models/mysql/mysql_types
import ./query_builder/models/mysql/mysql_query
import ./query_builder/models/mysql/mysql_exec
import ./query_builder/models/mysql/mysql_open
when isExistsMariadb:
import ./query_builder/models/mariadb/mariadb_types
import ./query_builder/models/mariadb/mariadb_query
import ./query_builder/models/mariadb/mariadb_exec
import ./query_builder/models/mariadb/mariadb_open
when isExistsSqlite:
import ./query_builder/models/sqlite/sqlite_types
import ./query_builder/models/sqlite/sqlite_query
import ./query_builder/models/sqlite/sqlite_exec
import ./query_builder/models/sqlite/sqlite_open
when isExistsSurrealdb:
import ./query_builder/models/surreal/surreal_types
import ./query_builder/models/surreal/surreal_query
import ./query_builder/models/surreal/surreal_exec
import ./query_builder/models/surreal/surreal_open
# ==============================================================================
# Types
# ==============================================================================
type
ColumnInfo* = tuple[name: string, typ: string, isPk: bool, isNullable: bool, defaultVal: string]
TableInfo* = object
name*: string
columns*: seq[ColumnInfo]
MigrationProgress* = object
tableName*: string
totalRows*: int
transferredRows*: int
status*: string # "pending", "in_progress", "done", "failed"
error*: string
MigrationReport* = object
sourceDb*: string
targetDb*: string
tablesTotal*: int
tablesDone*: int
rowsTotal*: int
rowsTransferred*: int
startedAt*: float
completedAt*: float
errors*: seq[string]
# ==============================================================================
# Type Mapping: source DB → BaraDB
# ==============================================================================
const BARADB_TYPE_MAP = {
# PostgreSQL
"smallint": "SMALLINT",
"integer": "INTEGER",
"bigint": "BIGINT",
"serial": "SERIAL",
"bigserial": "BIGSERIAL",
"real": "REAL",
"double precision": "DOUBLE PRECISION",
"numeric": "DECIMAL",
"decimal": "DECIMAL",
"character": "VARCHAR(255)",
"character varying": "VARCHAR",
"varchar": "VARCHAR",
"text": "TEXT",
"boolean": "BOOLEAN",
"bool": "BOOLEAN",
"date": "DATE",
"timestamp": "TIMESTAMP",
"timestamp without time zone": "TIMESTAMP",
"timestamp with time zone": "TIMESTAMPTZ",
"time": "TIME",
"time without time zone": "TIME",
"bytea": "BYTEA",
"uuid": "UUID",
"json": "JSON",
"jsonb": "JSON",
# MySQL/MariaDB
"tinyint": "SMALLINT",
"mediumint": "INTEGER",
"int": "INTEGER",
"float": "REAL",
"double": "DOUBLE PRECISION",
"char": "VARCHAR(255)",
"longtext": "TEXT",
"mediumtext": "TEXT",
"tinytext": "TEXT",
"blob": "BYTEA",
"longblob": "BYTEA",
"mediumblob": "BYTEA",
"tinyblob": "BYTEA",
"datetime": "TIMESTAMP",
"enum": "VARCHAR(255)",
"set": "VARCHAR(255)",
"year": "SMALLINT",
# SQLite
"int": "INTEGER",
"integer": "INTEGER",
"real": "REAL",
"blob": "BYTEA",
# SurrealDB
"string": "VARCHAR(255)",
"number": "DOUBLE PRECISION",
"object": "JSON",
"array": "JSON",
}.toTable()
proc mapType*(sourceType: string): string =
## Map a source database column type to the closest BaraDB type.
let lower = sourceType.toLower().split("(")[0].strip()
if lower in BARADB_TYPE_MAP:
result = BARADB_TYPE_MAP[lower]
else:
result = "VARCHAR(255)"
# ==============================================================================
# Schema Extraction
# ==============================================================================
# --- PostgreSQL ---
when isExistsPostgres:
proc extractSchema*(rdb: PostgresConnections): Future[seq[TableInfo]] {.async.} =
result = @[]
let tables = await rdb.raw(
"SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'"
).get()
for table in tables:
let tableName = table["table_name"].getStr()
if tableName == "_allographer_migrations": continue
var cols: seq[ColumnInfo] = @[]
let columns = await rdb.raw(
"""SELECT c.column_name, c.data_type, c.is_nullable, c.column_default,
CASE WHEN pk.column_name IS NOT NULL THEN true ELSE false END as is_pk
FROM information_schema.columns c
LEFT JOIN (
SELECT ku.column_name FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage ku
ON tc.constraint_name = ku.constraint_name
WHERE tc.constraint_type = 'PRIMARY KEY' AND tc.table_name = ?
) pk ON c.column_name = pk.column_name
WHERE c.table_name = ?
ORDER BY c.ordinal_position""",
%*[tableName, tableName]
).get()
for col in columns:
cols.add((
name: col["column_name"].getStr(),
typ: col["data_type"].getStr(),
isPk: col["is_pk"].getStr() == "true",
isNullable: col["is_nullable"].getStr() == "YES",
defaultVal: if col.hasKey("column_default") and not col["column_default"].isNull:
col["column_default"].getStr() else: ""
))
result.add(TableInfo(name: tableName, columns: cols))
# --- MySQL ---
when isExistsMysql:
proc extractSchema*(rdb: MysqlConnections): Future[seq[TableInfo]] {.async.} =
result = @[]
let tables = await rdb.raw(
"SELECT table_name FROM information_schema.tables WHERE table_schema = DATABASE()"
).get()
for table in tables:
let tableName = table["table_name"].getStr()
if tableName == "_allographer_migrations": continue
var cols: seq[ColumnInfo] = @[]
let columns = await rdb.raw(
"""SELECT column_name, data_type, is_nullable, column_default,
column_key FROM information_schema.columns
WHERE table_name = ? AND table_schema = DATABASE()
ORDER BY ordinal_position""",
%*[tableName]
).get()
for col in columns:
cols.add((
name: col["column_name"].getStr(),
typ: col["data_type"].getStr(),
isPk: col["column_key"].getStr() == "PRI",
isNullable: col["is_nullable"].getStr() == "YES",
defaultVal: if col.hasKey("column_default") and not col["column_default"].isNull:
col["column_default"].getStr() else: ""
))
result.add(TableInfo(name: tableName, columns: cols))
# --- MariaDB ---
when isExistsMariadb:
proc extractSchema*(rdb: MariadbConnections): Future[seq[TableInfo]] {.async.} =
result = @[]
let tables = await rdb.raw(
"SELECT table_name FROM information_schema.tables WHERE table_schema = DATABASE()"
).get()
for table in tables:
let tableName = table["table_name"].getStr()
if tableName == "_allographer_migrations": continue
var cols: seq[ColumnInfo] = @[]
let columns = await rdb.raw(
"""SELECT column_name, data_type, is_nullable, column_default,
column_key FROM information_schema.columns
WHERE table_name = ? AND table_schema = DATABASE()
ORDER BY ordinal_position""",
%*[tableName]
).get()
for col in columns:
cols.add((
name: col["column_name"].getStr(),
typ: col["data_type"].getStr(),
isPk: col["column_key"].getStr() == "PRI",
isNullable: col["is_nullable"].getStr() == "YES",
defaultVal: if col.hasKey("column_default") and not col["column_default"].isNull:
col["column_default"].getStr() else: ""
))
result.add(TableInfo(name: tableName, columns: cols))
# --- SQLite ---
when isExistsSqlite:
proc extractSchema*(rdb: SqliteConnections): Future[seq[TableInfo]] {.async.} =
result = @[]
let tables = await rdb.raw(
"SELECT name as table_name FROM sqlite_master WHERE type = 'table'"
).get()
for table in tables:
let tableName = table["table_name"].getStr()
if tableName == "_allographer_migrations": continue
if tableName == "sqlite_sequence": continue
var cols: seq[ColumnInfo] = @[]
let columns = await rdb.raw("PRAGMA table_info(?)", %*[tableName]).get()
for col in columns:
cols.add((
name: col["name"].getStr(),
typ: col["type"].getStr(),
isPk: col["pk"].getStr() == "1",
isNullable: col["notnull"].getStr() == "0",
defaultVal: if col.hasKey("dflt_value") and not col["dflt_value"].isNull:
col["dflt_value"].getStr() else: ""
))
result.add(TableInfo(name: tableName, columns: cols))
# --- SurrealDB ---
when isExistsSurrealdb:
proc extractSchema*(rdb: SurrealConnections): Future[seq[TableInfo]] {.async.} =
result = @[]
let dbResponse = await rdb.raw("INFO FOR DB").get()
if dbResponse.len == 0: return
let tables = dbResponse[0]["result"]["tables"]
for tableName, _ in tables.getFields().pairs:
if tableName == "_allographer_migrations": continue
if tableName == "_autoincrement_sequences": continue
var cols: seq[ColumnInfo] = @[]
let tableInfo = await rdb.raw(&"INFO FOR TABLE {tableName}").get()
if tableInfo.len > 0:
let fields = tableInfo[0]["result"]["fields"]
for fieldName, _ in fields.getFields().pairs:
cols.add((
name: fieldName,
typ: "string", # SurrealDB is schemaless
isPk: fieldName == "id",
isNullable: fieldName != "id",
defaultVal: ""
))
result.add(TableInfo(name: tableName, columns: cols))
# ==============================================================================
# DDL Generation
# ==============================================================================
proc generateBaraDBDDL*(table: TableInfo): tuple[upSql: string, downSql: string] =
## Generate CREATE TABLE (UP) and DROP TABLE (DOWN) DDL for BaraDB.
var colDefs: seq[string] = @[]
for col in table.columns:
var def = &"`{col.name}` {mapType(col.typ)}"
if col.isPk:
def &= " PRIMARY KEY"
if not col.isNullable:
def &= " NOT NULL"
if col.defaultVal.len > 0:
def &= " DEFAULT " & col.defaultVal
colDefs.add(def)
let upSql = &"CREATE TABLE `{table.name}` ({colDefs.join(\", \")})"
let downSql = &"DROP TABLE IF EXISTS `{table.name}`"
return (upSql, downSql)
proc generateMigrationName*(tableName: string): string =
let timestamp = getTime().toUnix()
return &"migrate_{tableName}_{timestamp}"
# ==============================================================================
# Data Transfer
# ==============================================================================
proc transferTable*(sourceConn: BaradbConnections, targetConn: BaradbConnections,
tableName: string, batchSize: int = 1000): Future[MigrationProgress] {.async.} =
## Generic table transfer for BaraDB→BaraDB (used internally).
## For cross-DB, use the typed overloads below.
result = MigrationProgress(tableName: tableName, totalRows: 0,
transferredRows: 0, status: "in_progress")
try:
# Count rows
let countVal = await sourceConn.table(tableName).count()
result.totalRows = countVal
# Transfer in batches
var offset = 0
while offset < result.totalRows:
let rows = await sourceConn.table(tableName)
.limit(batchSize)
.offset(offset)
.get()
if rows.len == 0: break
await targetConn.table(tableName).insert(rows)
result.transferredRows += rows.len
offset += batchSize
result.status = "done"
except CatchableError as e:
result.status = "failed"
result.error = e.msg
# Typed overloads for each source DB type
when isExistsPostgres:
proc transferTable*(sourceConn: PostgresConnections, targetConn: BaradbConnections,
tableName: string, batchSize: int = 1000): Future[MigrationProgress] {.async.} =
result = MigrationProgress(tableName: tableName, totalRows: 0,
transferredRows: 0, status: "in_progress")
try:
let countVal = await sourceConn.table(tableName).count()
result.totalRows = countVal
var offset = 0
while offset < result.totalRows:
let rows = await sourceConn.table(tableName)
.limit(batchSize)
.offset(offset)
.get()
if rows.len == 0: break
await targetConn.table(tableName).insert(rows)
result.transferredRows += rows.len
offset += batchSize
result.status = "done"
except CatchableError as e:
result.status = "failed"
result.error = e.msg
when isExistsMysql:
proc transferTable*(sourceConn: MysqlConnections, targetConn: BaradbConnections,
tableName: string, batchSize: int = 1000): Future[MigrationProgress] {.async.} =
result = MigrationProgress(tableName: tableName, totalRows: 0,
transferredRows: 0, status: "in_progress")
try:
let countVal = await sourceConn.table(tableName).count()
result.totalRows = countVal
var offset = 0
while offset < result.totalRows:
let rows = await sourceConn.table(tableName)
.limit(batchSize)
.offset(offset)
.get()
if rows.len == 0: break
await targetConn.table(tableName).insert(rows)
result.transferredRows += rows.len
offset += batchSize
result.status = "done"
except CatchableError as e:
result.status = "failed"
result.error = e.msg
when isExistsMariadb:
proc transferTable*(sourceConn: MariadbConnections, targetConn: BaradbConnections,
tableName: string, batchSize: int = 1000): Future[MigrationProgress] {.async.} =
result = MigrationProgress(tableName: tableName, totalRows: 0,
transferredRows: 0, status: "in_progress")
try:
let countVal = await sourceConn.table(tableName).count()
result.totalRows = countVal
var offset = 0
while offset < result.totalRows:
let rows = await sourceConn.table(tableName)
.limit(batchSize)
.offset(offset)
.get()
if rows.len == 0: break
await targetConn.table(tableName).insert(rows)
result.transferredRows += rows.len
offset += batchSize
result.status = "done"
except CatchableError as e:
result.status = "failed"
result.error = e.msg
when isExistsSqlite:
proc transferTable*(sourceConn: SqliteConnections, targetConn: BaradbConnections,
tableName: string, batchSize: int = 1000): Future[MigrationProgress] {.async.} =
result = MigrationProgress(tableName: tableName, totalRows: 0,
transferredRows: 0, status: "in_progress")
try:
let countVal = await sourceConn.table(tableName).count()
result.totalRows = countVal
var offset = 0
while offset < result.totalRows:
let rows = await sourceConn.table(tableName)
.limit(batchSize)
.offset(offset)
.get()
if rows.len == 0: break
await targetConn.table(tableName).insert(rows)
result.transferredRows += rows.len
offset += batchSize
result.status = "done"
except CatchableError as e:
result.status = "failed"
result.error = e.msg
when isExistsSurrealdb:
proc transferTable*(sourceConn: SurrealConnections, targetConn: BaradbConnections,
tableName: string, batchSize: int = 1000): Future[MigrationProgress] {.async.} =
result = MigrationProgress(tableName: tableName, totalRows: 0,
transferredRows: 0, status: "in_progress")
try:
let countVal = await sourceConn.table(tableName).count()
result.totalRows = countVal
var offset = 0
while offset < result.totalRows:
let rows = await sourceConn.table(tableName)
.limit(batchSize)
.offset(offset)
.get()
if rows.len == 0: break
await targetConn.table(tableName).insert(rows)
result.transferredRows += rows.len
offset += batchSize
result.status = "done"
except CatchableError as e:
result.status = "failed"
result.error = e.msg
# ==============================================================================
# Full Migration Orchestrator
# ==============================================================================
when isExistsPostgres:
proc migrate*(sourceConn: PostgresConnections, targetConn: BaradbConnections,
tables: seq[string] = @[], batchSize: int = 1000): Future[MigrationReport] {.async.} =
result = MigrationReport(sourceDb: "PostgreSQL", targetDb: "BaraDB",
startedAt: epochTime())
try:
let schema = await extractSchema(sourceConn)
var tablesToMigrate: seq[TableInfo]
if tables.len > 0:
for t in schema:
if t.name in tables: tablesToMigrate.add(t)
else:
tablesToMigrate = schema
result.tablesTotal = tablesToMigrate.len
for table in tablesToMigrate:
let (upSql, downSql) = generateBaraDBDDL(table)
let migName = generateMigrationName(table.name)
discard await targetConn.createMigration(migName, upSql, downSql)
discard await targetConn.applyMigration(migName)
let progress = await transferTable(sourceConn, targetConn, table.name, batchSize)
if progress.status == "done":
result.tablesDone += 1
result.rowsTransferred += progress.transferredRows
else:
result.errors.add(&"{table.name}: {progress.error}")
except CatchableError as e:
result.errors.add(e.msg)
result.completedAt = epochTime()
when isExistsMysql:
proc migrate*(sourceConn: MysqlConnections, targetConn: BaradbConnections,
tables: seq[string] = @[], batchSize: int = 1000): Future[MigrationReport] {.async.} =
result = MigrationReport(sourceDb: "MySQL", targetDb: "BaraDB",
startedAt: epochTime())
try:
let schema = await extractSchema(sourceConn)
var tablesToMigrate: seq[TableInfo]
if tables.len > 0:
for t in schema:
if t.name in tables: tablesToMigrate.add(t)
else:
tablesToMigrate = schema
result.tablesTotal = tablesToMigrate.len
for table in tablesToMigrate:
let (upSql, downSql) = generateBaraDBDDL(table)
let migName = generateMigrationName(table.name)
discard await targetConn.createMigration(migName, upSql, downSql)
discard await targetConn.applyMigration(migName)
let progress = await transferTable(sourceConn, targetConn, table.name, batchSize)
if progress.status == "done":
result.tablesDone += 1
result.rowsTransferred += progress.transferredRows
else:
result.errors.add(&"{table.name}: {progress.error}")
except CatchableError as e:
result.errors.add(e.msg)
result.completedAt = epochTime()
when isExistsMariadb:
proc migrate*(sourceConn: MariadbConnections, targetConn: BaradbConnections,
tables: seq[string] = @[], batchSize: int = 1000): Future[MigrationReport] {.async.} =
result = MigrationReport(sourceDb: "MariaDB", targetDb: "BaraDB",
startedAt: epochTime())
try:
let schema = await extractSchema(sourceConn)
var tablesToMigrate: seq[TableInfo]
if tables.len > 0:
for t in schema:
if t.name in tables: tablesToMigrate.add(t)
else:
tablesToMigrate = schema
result.tablesTotal = tablesToMigrate.len
for table in tablesToMigrate:
let (upSql, downSql) = generateBaraDBDDL(table)
let migName = generateMigrationName(table.name)
discard await targetConn.createMigration(migName, upSql, downSql)
discard await targetConn.applyMigration(migName)
let progress = await transferTable(sourceConn, targetConn, table.name, batchSize)
if progress.status == "done":
result.tablesDone += 1
result.rowsTransferred += progress.transferredRows
else:
result.errors.add(&"{table.name}: {progress.error}")
except CatchableError as e:
result.errors.add(e.msg)
result.completedAt = epochTime()
when isExistsSqlite:
proc migrate*(sourceConn: SqliteConnections, targetConn: BaradbConnections,
tables: seq[string] = @[], batchSize: int = 1000): Future[MigrationReport] {.async.} =
result = MigrationReport(sourceDb: "SQLite", targetDb: "BaraDB",
startedAt: epochTime())
try:
let schema = await extractSchema(sourceConn)
var tablesToMigrate: seq[TableInfo]
if tables.len > 0:
for t in schema:
if t.name in tables: tablesToMigrate.add(t)
else:
tablesToMigrate = schema
result.tablesTotal = tablesToMigrate.len
for table in tablesToMigrate:
let (upSql, downSql) = generateBaraDBDDL(table)
let migName = generateMigrationName(table.name)
discard await targetConn.createMigration(migName, upSql, downSql)
discard await targetConn.applyMigration(migName)
let progress = await transferTable(sourceConn, targetConn, table.name, batchSize)
if progress.status == "done":
result.tablesDone += 1
result.rowsTransferred += progress.transferredRows
else:
result.errors.add(&"{table.name}: {progress.error}")
except CatchableError as e:
result.errors.add(e.msg)
result.completedAt = epochTime()
when isExistsSurrealdb:
proc migrate*(sourceConn: SurrealConnections, targetConn: BaradbConnections,
tables: seq[string] = @[], batchSize: int = 1000): Future[MigrationReport] {.async.} =
result = MigrationReport(sourceDb: "SurrealDB", targetDb: "BaraDB",
startedAt: epochTime())
try:
let schema = await extractSchema(sourceConn)
var tablesToMigrate: seq[TableInfo]
if tables.len > 0:
for t in schema:
if t.name in tables: tablesToMigrate.add(t)
else:
tablesToMigrate = schema
result.tablesTotal = tablesToMigrate.len
for table in tablesToMigrate:
let (upSql, downSql) = generateBaraDBDDL(table)
let migName = generateMigrationName(table.name)
discard await targetConn.createMigration(migName, upSql, downSql)
discard await targetConn.applyMigration(migName)
let progress = await transferTable(sourceConn, targetConn, table.name, batchSize)
if progress.status == "done":
result.tablesDone += 1
result.rowsTransferred += progress.transferredRows
else:
result.errors.add(&"{table.name}: {progress.error}")
except CatchableError as e:
result.errors.add(e.msg)
result.completedAt = epochTime()
# ==============================================================================
# URL-based migration
# ==============================================================================
proc `$`*(report: MigrationReport): string =
let duration = report.completedAt - report.startedAt
result = &"Migration: {report.sourceDb} → {report.targetDb}\n"
result &= &" Tables: {report.tablesDone}/{report.tablesTotal}\n"
result &= &" Rows: {report.rowsTransferred}\n"
result &= &" Time: {duration:.1f}s\n"
if report.errors.len > 0:
result &= " Errors:\n"
for err in report.errors:
result &= &" - {err}\n"
@@ -410,6 +410,35 @@ proc exec*(client: BaraClient, sql: string): Future[int] {.async.} =
let qr = await client.query(sql)
return qr.affectedRows
# === Migration API (BaraQL native) ===
proc createMigration*(client: BaraClient, name: string, upBody: string,
downBody: string = ""): Future[QueryResult] {.async.} =
## Send CREATE MIGRATION via BaraQL. Server handles checksums, locking, rollback.
var sql = "CREATE MIGRATION " & name & " { UP: " & upBody & ";"
if downBody.len > 0:
sql &= " DOWN: " & downBody & ";"
sql &= " }"
return await client.query(sql)
proc applyMigration*(client: BaraClient, name: string): Future[QueryResult] {.async.} =
return await client.query("APPLY MIGRATION " & name)
proc migrateUp*(client: BaraClient, count: int = 0): Future[QueryResult] {.async.} =
var sql = "MIGRATION UP"
if count > 0:
sql &= " " & $count
return await client.query(sql)
proc migrateDown*(client: BaraClient, count: int = 1): Future[QueryResult] {.async.} =
return await client.query("MIGRATION DOWN " & $count)
proc migrationStatus*(client: BaraClient): Future[QueryResult] {.async.} =
return await client.query("MIGRATION STATUS")
proc migrationDryRun*(client: BaraClient, name: string): Future[QueryResult] {.async.} =
return await client.query("MIGRATION DRY RUN " & name)
proc auth*(client: BaraClient, token: string) {.async.} =
if not client.connected:
raise newException(IOError, "Not connected")
@@ -652,6 +681,34 @@ proc exec*(client: SyncClient, sql: string): int =
let qr = client.query(sql)
return qr.affectedRows
# === Migration API (SyncClient, blocking) ===
proc createMigration*(client: SyncClient, name: string, upBody: string,
downBody: string = ""): QueryResult =
var sql = "CREATE MIGRATION " & name & " { UP: " & upBody & ";"
if downBody.len > 0:
sql &= " DOWN: " & downBody & ";"
sql &= " }"
return client.query(sql)
proc applyMigration*(client: SyncClient, name: string): QueryResult =
return client.query("APPLY MIGRATION " & name)
proc migrateUp*(client: SyncClient, count: int = 0): QueryResult =
var sql = "MIGRATION UP"
if count > 0:
sql &= " " & $count
return client.query(sql)
proc migrateDown*(client: SyncClient, count: int = 1): QueryResult =
return client.query("MIGRATION DOWN " & $count)
proc migrationStatus*(client: SyncClient): QueryResult =
return client.query("MIGRATION STATUS")
proc migrationDryRun*(client: SyncClient, name: string): QueryResult =
return client.query("MIGRATION DRY RUN " & name)
proc auth*(client: SyncClient, token: string) =
acquire(client.lock)
try:
@@ -699,6 +699,183 @@ proc firstPlain*(self: RawBaradbQuery): Future[seq[string]] {.async.} =
return await self.getRowPlain(self.queryString, self.placeHolder)
# ================================================================================
# Pagination (cursor-based and offset-based)
# ================================================================================
type
PaginateResult* = object
rows*: seq[JsonNode]
total*: int
page*: int
perPage*: int
hasMore*: bool
proc paginate*(self: BaradbQuery, page: int = 1, perPage: int = 20): Future[PaginateResult] {.async.} =
## Offset-based pagination. Returns rows for the given page plus metadata.
let total = await self.count()
let offset = (page - 1) * perPage
self.limit(perPage)
self.offset(offset)
let rows = await self.get()
let qr = PaginateResult(
rows: rows,
total: total,
page: page,
perPage: perPage,
hasMore: offset + perPage < total
)
return qr
proc fastPaginate*(self: BaradbQuery, cursorColumn: string, perPage: int = 20,
afterId: string = ""): Future[PaginateResult] {.async.} =
## Cursor-based pagination (keyset). More efficient than offset for large tables.
## Requires a unique, ordered column (usually the primary key).
let total = await self.count()
self.limit(perPage)
self.orderBy(cursorColumn, Asc)
if afterId.len > 0:
self.where(cursorColumn, ">", afterId)
let rows = await self.get()
var hasMore = rows.len == perPage
let qr = PaginateResult(
rows: rows,
total: total,
page: 0, # cursor-based has no page number
perPage: perPage,
hasMore: hasMore
)
return qr
# ================================================================================
# Migration API (BaraQL native — server handles checksums, locks, rollback)
# ================================================================================
proc createMigration*(self: BaradbConnections, name: string, upBody: string,
downBody: string = ""): Future[QueryResult] {.async.} =
## Register a migration on the server. The server computes checksums, stores
## the UP/DOWN bodies, and manages the migration lifecycle.
var connI = getFreeConn(self).await
defer: self.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(self)
return await self.pools.conns[connI].client.createMigration(name, upBody, downBody)
proc applyMigration*(self: BaradbConnections, name: string): Future[QueryResult] {.async.} =
var connI = getFreeConn(self).await
defer: self.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(self)
return await self.pools.conns[connI].client.applyMigration(name)
proc migrateUp*(self: BaradbConnections, count: int = 0): Future[QueryResult] {.async.} =
var connI = getFreeConn(self).await
defer: self.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(self)
return await self.pools.conns[connI].client.migrateUp(count)
proc migrateDown*(self: BaradbConnections, count: int = 1): Future[QueryResult] {.async.} =
var connI = getFreeConn(self).await
defer: self.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(self)
return await self.pools.conns[connI].client.migrateDown(count)
proc migrationStatus*(self: BaradbConnections): Future[seq[JsonNode]] {.async.} =
var connI = getFreeConn(self).await
defer: self.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(self)
let qr = await self.pools.conns[connI].client.migrationStatus()
return toJson(qr)
proc migrationDryRun*(self: BaradbConnections, name: string): Future[QueryResult] {.async.} =
var connI = getFreeConn(self).await
defer: self.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(self)
return await self.pools.conns[connI].client.migrationDryRun(name)
proc isMigrationApplied*(self: BaradbConnections, name: string): Future[bool] {.async.} =
let status = await self.migrationStatus()
for row in status:
if row["name"].getStr() == name and row["status"].getStr() == "applied":
return true
return false
# ================================================================================
# Prepared Statements (server-side parameterized queries via mkQueryParams)
# ================================================================================
proc sendPrepared(client: BaraClient, sql: string, params: seq[WireValue]): Future[QueryResult] {.async.} =
if not client.connected:
raise newException(IOError, "Not connected")
let msg = makeQueryParamsMessage(client.nextId(), sql, params)
let msgStr = toString(msg)
await client.socket.send(msgStr)
return await client.readQueryResponse()
proc prepare*(self: BaradbConnections, sql: string, nArgs: int = 0): Future[BaradbPreparedStatement] {.async.} =
## Create a server-side prepared statement. Subsequent calls reuse the cached entry.
let entryKey = sql & ":" & $nArgs
if entryKey in self.pools.preparedCache:
let entry = self.pools.preparedCache[entryKey]
inc entry.refCount
entry.lastUsedAt = nowUnix()
return BaradbPreparedStatement(owner: self, entry: entry, sql: sql,
nArgs: nArgs, isClosed: false)
let entry = BaradbPreparedEntry(sql: sql, nArgs: nArgs, refCount: 1,
lastUsedAt: nowUnix())
self.pools.preparedCache[entryKey] = entry
return BaradbPreparedStatement(owner: self, entry: entry, sql: sql,
nArgs: nArgs, isClosed: false)
proc ensureStmt*(self: BaradbConnections, sql: string, nArgs: int): BaradbPreparedEntry =
let entryKey = sql & ":" & $nArgs
if entryKey in self.pools.preparedCache:
let entry = self.pools.preparedCache[entryKey]
entry.lastUsedAt = nowUnix()
return entry
let entry = BaradbPreparedEntry(sql: sql, nArgs: nArgs, refCount: 0,
lastUsedAt: nowUnix())
self.pools.preparedCache[entryKey] = entry
return entry
proc preparedGet*(stmt: BaradbPreparedStatement, args: seq[WireValue]): Future[seq[JsonNode]] {.async.} =
if stmt.isClosed:
raise newException(IOError, "Prepared statement is closed")
var connI = getFreeConn(stmt.owner).await
defer: stmt.owner.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(stmt.owner)
let qr = await sendPrepared(stmt.owner.pools.conns[connI].client, stmt.sql, args)
return toJson(qr)
proc preparedExec*(stmt: BaradbPreparedStatement, args: seq[WireValue]): Future[int] {.async.} =
if stmt.isClosed:
raise newException(IOError, "Prepared statement is closed")
var connI = getFreeConn(stmt.owner).await
defer: stmt.owner.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(stmt.owner)
let qr = await sendPrepared(stmt.owner.pools.conns[connI].client, stmt.sql, args)
return qr.affectedRows
proc flushStmt*(stmt: BaradbPreparedStatement) =
stmt.isClosed = true
dec stmt.entry.refCount
proc clearStmtCache*(self: BaradbConnections) =
self.pools.preparedCache.clear()
proc withConn*(self: BaradbConnections, callback: proc(connI: int): Future[void] {.gcsafe.}): Future[void] {.async.} =
let connI = getFreeConn(self).await
defer: self.returnConn(connI).await
if connI == errorConnectionNum:
raisePoolTimeout(self)
await callback(connI)
# seeder templates
template seeder*(rdb: BaradbConnections, tableName: string, body: untyped): untyped =
block:
@@ -4,6 +4,7 @@ import std/json
import std/tables
import std/times
import ../../libs/baradb/baradb_client
import ../../libs/database_url
import ../../log
import ./baradb_types
@@ -50,3 +51,18 @@ proc dbOpen*(_: type Baradb, database: string, user: string, password: string,
pools: pools,
log: LogSetting(shouldDisplayLog: shouldDisplayLog, shouldOutputLogFile: shouldOutputLogFile, logDir: logDir)
)
proc dbOpen*(_: type Baradb, databaseUrl: DatabaseUrl,
maxConnections = 1, timeout = 30,
shouldDisplayLog = false, shouldOutputLogFile = false, logDir = "",
maxConnectionLifetime = DEFAULT_CONN_MAX_LIFETIME_SECONDS,
maxConnectionIdleTime = DEFAULT_CONN_MAX_IDLE_SECONDS): BaradbConnections =
## Open a BaraDB connection using a URL: baradb://user:pass@host:port/database
let parsed = parseDatabaseUrl(databaseUrl)
requireDatabaseUrlScheme(parsed, ["baradb"], "BaraDB")
let database = parsed.path.stripLeadingSlash()
let host = if parsed.hostname.len > 0: parsed.hostname else: "127.0.0.1"
let port = if parsed.hasPort: parsed.port else: 9472
return Baradb.dbOpen(database, parsed.username, parsed.password, host, port,
maxConnections, timeout, shouldDisplayLog, shouldOutputLogFile,
logDir, maxConnectionLifetime, maxConnectionIdleTime)
@@ -3,10 +3,6 @@ import ../../models/table
import ./baradb_query_type
proc createMigrationTable*(self: BaradbSchema) =
let sql = """CREATE TABLE IF NOT EXISTS `schema_migrations` (
`id` SERIAL PRIMARY KEY,
`name` VARCHAR(255) NOT NULL,
`checksum` VARCHAR(64) NOT NULL,
`created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)"""
discard waitFor self.rdb.raw(sql).exec()
## BaraDB tracks migrations natively via BaraQL (CREATE MIGRATION, MIGRATION STATUS).
## No client-side migration table needed — server stores state in LSM-Tree.
discard
@@ -1,12 +1,47 @@
import std/asyncdispatch
import std/strformat
import ../../../query_builder/models/baradb/baradb_types
import ../../../query_builder/models/baradb/baradb_exec
import ../../../query_builder/error
import ../../models/table
proc execThenSaveHistory*(rdb: BaradbConnections, name: string, queries: seq[string], checksum: string) =
proc exec*(rdb: BaradbConnections, queries: seq[string]) =
for sql in queries:
discard waitFor rdb.raw(sql).exec()
let hist = &"INSERT INTO `schema_migrations` (`name`, `checksum`) VALUES ('{name}', '{checksum}')"
discard waitFor rdb.raw(hist).exec()
proc execThenSaveHistory*(rdb: BaradbConnections, name: string, queries: seq[string], checksum: string) =
## Register and apply a migration via BaraQL native migration system.
## The server handles checksums, locking, rollback, and status tracking.
let upBody = queries.join("; ")
var downBody = ""
# Generate DOWN script: DROP TABLE for CREATE TABLE
for sql in queries:
if sql.toLower().startsWith("create table"):
let parts = sql.split(" ")
if parts.len >= 3:
let tableName = parts[2] # CREATE TABLE `name` ...
downBody &= &"DROP TABLE IF EXISTS {tableName}; "
# Register migration on server
var qr = waitFor rdb.createMigration(name, upBody, downBody)
if qr.affectedRows < 0:
raise newException(DbError, "Failed to create migration: " & name)
# Apply migration
qr = waitFor rdb.applyMigration(name)
if qr.affectedRows < 0:
raise newException(DbError, "Failed to apply migration: " & name)
proc execThenSaveHistory*(rdb: BaradbConnections, name: string, query: string, checksum: string) =
execThenSaveHistory(rdb, name, @[query], checksum)
proc shouldRun*(rdb: BaradbConnections, table: Table, checksum: string, isReset: bool): bool =
return true
## Check with the server whether this migration has already been applied.
if isReset:
return true
try:
let applied = waitFor rdb.isMigrationApplied(table.name)
return not applied
except CatchableError:
# If server is unavailable or migration doesn't exist, run it
return true
@@ -0,0 +1,59 @@
discard """
cmd: "nim c -r $file"
"""
import std/unittest
import std/asyncdispatch
import std/json
import ../../src/allographer/query_builder
import ../../src/allographer/query_builder/models/baradb/baradb_exec
import ./connections
let rdb = baradb
suite("Baradb migrations"):
test("create migration"):
let upSql = """
CREATE TABLE IF NOT EXISTS test_mig_users (
id SERIAL PRIMARY KEY,
name VARCHAR(255)
)
"""
let downSql = "DROP TABLE IF EXISTS test_mig_users"
let qr = waitFor rdb.createMigration("test_mig_001", upSql, downSql)
check qr.rowCount >= 0
test("migration status shows pending"):
let status = waitFor rdb.migrationStatus()
var found = false
for row in status:
if row["name"].getStr() == "test_mig_001":
check row["status"].getStr() == "pending"
found = true
break
check found
test("apply migration"):
let qr = waitFor rdb.applyMigration("test_mig_001")
check qr.rowCount >= 0
test("is migration applied"):
let applied = waitFor rdb.isMigrationApplied("test_mig_001")
check applied
test("migration up (all pending)"):
let qr = waitFor rdb.migrateUp()
check qr.rowCount >= 0
test("migration dry run"):
let qr = waitFor rdb.migrationDryRun("test_mig_001")
check qr.rowCount >= 0
test("table exists after migration"):
let users = waitFor rdb.table("test_mig_users").get()
check users.len >= 0
test("migrate down (rollback)"):
let qr = waitFor rdb.migrateDown(1)
check qr.rowCount >= 0
@@ -0,0 +1,85 @@
discard """
cmd: "nim c -r $file"
"""
import std/unittest
import std/asyncdispatch
import std/json
import std/options
import ../../src/allographer/query_builder
import ../../src/allographer/query_builder/models/baradb/baradb_exec
import ../../src/allographer/query_builder/libs/baradb/baradb_client
import ./connections
let rdb = baradb
suite("Baradb prepared statements"):
setup:
# Ensure test table exists
let createSql = """
CREATE TABLE IF NOT EXISTS test_prep_data (
id SERIAL PRIMARY KEY,
label VARCHAR(255),
count INTEGER
)
"""
waitFor rdb.raw(createSql).exec()
test("prepare statement"):
let stmt = waitFor rdb.prepare(
"SELECT * FROM test_prep_data WHERE count > ?", nArgs = 1
)
check not stmt.isClosed
test("ensureStmt"):
let entry = rdb.ensureStmt(
"SELECT * FROM test_prep_data WHERE label = ?", nArgs = 1
)
check entry.sql.len > 0
check entry.nArgs == 1
test("preparedGet with params"):
# Insert test data
waitFor rdb.table("test_prep_data").insert(%*{
"label": "alpha", "count": 5
})
waitFor rdb.table("test_prep_data").insert(%*{
"label": "beta", "count": 15
})
# Query via prepared statement
let stmt = waitFor rdb.prepare(
"SELECT * FROM test_prep_data WHERE count > ?", nArgs = 1
)
let results = waitFor stmt.preparedGet(@[
WireValue(kind: fkInt32, int32Val: 10)
])
check results.len >= 1
if results.len > 0:
check results[0]["label"].getStr() == "beta"
test("preparedExec insert"):
let stmt = waitFor rdb.prepare(
"INSERT INTO test_prep_data (label, count) VALUES (?, ?)", nArgs = 2
)
let affected = waitFor stmt.preparedExec(@[
WireValue(kind: fkString, strVal: "gamma"),
WireValue(kind: fkInt32, int32Val: 25)
])
check affected >= 0
test("flush statement"):
let stmt = waitFor rdb.prepare(
"SELECT * FROM test_prep_data", nArgs = 0
)
stmt.flushStmt()
check stmt.isClosed
test("clear statement cache"):
rdb.clearStmtCache()
check true
test("cleanup"):
waitFor rdb.raw("DROP TABLE IF EXISTS test_prep_data").exec()
check true