docs: update README and all docs with formal verification, new BaraQL features, OpenTelemetry tracing
- Add Formal Verification section to README and architecture docs - Document TLA+ specs for Raft, 2PC, MVCC, Replication - Add new BaraQL features: JSON path (->, ->>), FTS @@, CREATE INDEX USING FTS, RECOVER TO TIMESTAMP, UNION/INTERSECT/EXCEPT - Add OpenTelemetry tracing example to README - Update Quick Start to use nimble test/bench - Update EN and BG documentation
This commit is contained in:
@@ -47,17 +47,39 @@ single 3.3MB binary with no runtime dependencies.
|
|||||||
└─────────────────────────────────────────────────────────┘
|
└─────────────────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Formal Verification
|
||||||
|
|
||||||
|
BaraDB core distributed algorithms are formally specified and model-checked
|
||||||
|
with **TLA+** and the TLC model checker:
|
||||||
|
|
||||||
|
| Algorithm | Spec | States Checked | Properties |
|
||||||
|
|-----------|------|----------------|------------|
|
||||||
|
| **Raft Consensus** | `formal-verification/raft.tla` | 475,972 | ElectionSafety, StateMachineSafety |
|
||||||
|
| **Two-Phase Commit** | `formal-verification/twopc.tla` | 22,145 | Atomicity, NoOrphanBlocks |
|
||||||
|
| **MVCC** | `formal-verification/mvcc.tla` | 177,849 | NoDirtyReads, ReadOwnWrites, WriteWriteConflict |
|
||||||
|
| **Replication** | `formal-verification/replication.tla` | 3,687,939 | MonotonicLsn, AcksRemovePending |
|
||||||
|
|
||||||
|
Run the checks locally:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd formal-verification
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/raft.cfg raft.tla
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/twopc.cfg twopc.tla
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/mvcc.cfg mvcc.tla
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/replication.cfg replication.tla
|
||||||
|
```
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Build
|
# Build
|
||||||
nim c -d:release -o:build/baradadb src/baradadb.nim
|
nimble build -d:release
|
||||||
|
|
||||||
# Run tests
|
# Run tests
|
||||||
nim c --path:src -r tests/test_all.nim
|
nimble test
|
||||||
|
|
||||||
# Run benchmarks
|
# Run benchmarks
|
||||||
nim c -d:release -r benchmarks/bench_all.nim
|
nimble bench
|
||||||
|
|
||||||
# Start server
|
# Start server
|
||||||
./build/baradadb
|
./build/baradadb
|
||||||
@@ -169,6 +191,45 @@ CREATE TYPE Movie {
|
|||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### JSON & JSONB
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Create table with JSON column
|
||||||
|
CREATE TABLE events (id INT PRIMARY KEY, payload JSON);
|
||||||
|
|
||||||
|
-- Insert valid JSON
|
||||||
|
INSERT INTO events (id, payload) VALUES (1, '{"action": "click"}');
|
||||||
|
|
||||||
|
-- JSON path operators
|
||||||
|
SELECT payload->'action' AS action_json,
|
||||||
|
payload->>'action' AS action_text
|
||||||
|
FROM events;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Full-Text Search (SQL)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Create FTS index
|
||||||
|
CREATE INDEX idx_fts ON articles(body) USING FTS;
|
||||||
|
|
||||||
|
-- Search with BM25 ranking
|
||||||
|
SELECT * FROM articles WHERE body @@ 'machine learning';
|
||||||
|
```
|
||||||
|
|
||||||
|
### Set Operations
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT name FROM customers
|
||||||
|
UNION ALL
|
||||||
|
SELECT name FROM suppliers;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Point-in-Time Recovery
|
||||||
|
|
||||||
|
```sql
|
||||||
|
RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
|
||||||
|
```
|
||||||
|
|
||||||
## Storage Engines
|
## Storage Engines
|
||||||
|
|
||||||
### LSM-Tree (Key-Value)
|
### LSM-Tree (Key-Value)
|
||||||
@@ -667,6 +728,22 @@ Example response:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### OpenTelemetry Tracing
|
||||||
|
|
||||||
|
Built-in lightweight tracing with OTLP/HTTP export:
|
||||||
|
|
||||||
|
```nim
|
||||||
|
import barabadb/core/tracing
|
||||||
|
|
||||||
|
defaultTracer.enable()
|
||||||
|
let span = defaultTracer.beginSpan("SELECT users")
|
||||||
|
# ... query execution ...
|
||||||
|
defaultTracer.endSpan(span)
|
||||||
|
|
||||||
|
# Export to Jaeger/OTLP collector
|
||||||
|
discard defaultTracer.exportOtlp("http://localhost:4318/v1/traces")
|
||||||
|
```
|
||||||
|
|
||||||
### Health Check
|
### Health Check
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -123,6 +123,17 @@ Client → Protocol → Auth → Parser → AST → IR → Codegen
|
|||||||
- **Gossip Protocol** (`core/gossip.nim`): SWIM-подобно управление на членство
|
- **Gossip Protocol** (`core/gossip.nim`): SWIM-подобно управление на членство
|
||||||
- **Distributed Transactions** (`core/disttxn.nim`): Two-phase commit
|
- **Distributed Transactions** (`core/disttxn.nim`): Two-phase commit
|
||||||
|
|
||||||
|
## Ключови Дизайн Решения
|
||||||
|
|
||||||
|
1. **Чист Nim**: Без Cython, Python или Rust зависимости
|
||||||
|
2. **Обединено Съхранение**: Един двигател за KV, граф, вектор, FTS и колонно
|
||||||
|
3. **Вграден Режим**: Работи като библиотека или сървър
|
||||||
|
4. **Бинарен Протокол**: Ефективен wire протокол
|
||||||
|
5. **MVCC**: Multi-version concurrency control
|
||||||
|
6. **Schema-First**: Строго типизирана система с наследяване
|
||||||
|
7. **Крос-Модал**: Един език за заявки през всички модели данни
|
||||||
|
8. **Формално Верифициран**: Разпределените алгоритми са специфицирани в TLA+ и проверени с TLC
|
||||||
|
|
||||||
## Статистика на Модулите
|
## Статистика на Модулите
|
||||||
|
|
||||||
| Категория | Модули | Редове Код | Предназначение |
|
| Категория | Модули | Редове Код | Предназначение |
|
||||||
|
|||||||
+29
-2
@@ -105,6 +105,33 @@ RETURN friend.name;
|
|||||||
## Пълнотекстово Търсене
|
## Пълнотекстово Търсене
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT * FROM articles
|
-- Създаване на FTS индекс
|
||||||
WHERE MATCH(title, body) AGAINST('database programming');
|
CREATE INDEX idx_fts ON articles(body) USING FTS;
|
||||||
|
|
||||||
|
-- Търсене с BM25 ранжиране
|
||||||
|
SELECT * FROM articles WHERE body @@ 'database programming';
|
||||||
|
```
|
||||||
|
|
||||||
|
## JSON Оператори
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Извличане на JSON поле като JSON
|
||||||
|
SELECT data->'name' FROM users;
|
||||||
|
|
||||||
|
-- Извличане на JSON поле като текст
|
||||||
|
SELECT data->>'name' FROM users;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Set Операции
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT name FROM customers
|
||||||
|
UNION ALL
|
||||||
|
SELECT name FROM suppliers;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Възстановяване до Момент във Времето
|
||||||
|
|
||||||
|
```sql
|
||||||
|
RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
|
||||||
```
|
```
|
||||||
@@ -35,3 +35,23 @@ var rm = newReplicationManager(rmSync)
|
|||||||
rm.addReplica(newReplica("r1", "10.0.0.1", 9472))
|
rm.addReplica(newReplica("r1", "10.0.0.1", 9472))
|
||||||
rm.connectReplica("r1")
|
rm.connectReplica("r1")
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Формална Верификация
|
||||||
|
|
||||||
|
Разпределените алгоритми са формално специфицирани в TLA+ и проверени с TLC:
|
||||||
|
|
||||||
|
- **Raft Консенсус** — `formal-verification/raft.tla`
|
||||||
|
- Проверено: ElectionSafety, StateMachineSafety
|
||||||
|
- **Two-Phase Commit** — `formal-verification/twopc.tla`
|
||||||
|
- Проверено: Atomicity, NoOrphanBlocks
|
||||||
|
- **Репликация** — `formal-verification/replication.tla`
|
||||||
|
- Проверено: MonotonicLsn, AcksRemovePending
|
||||||
|
|
||||||
|
Пускане на TLC:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd formal-verification
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/raft.cfg raft.tla
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/twopc.cfg twopc.tla
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/replication.cfg replication.tla
|
||||||
|
```
|
||||||
@@ -25,3 +25,20 @@ BaraDB използва **snapshot isolation**:
|
|||||||
- Читателите не блокират писатели
|
- Читателите не блокират писатели
|
||||||
- Писателите не блокират читатели
|
- Писателите не блокират читатели
|
||||||
- Всяка транзакция вижда консистентен моментна снимка
|
- Всяка транзакция вижда консистентен моментна снимка
|
||||||
|
|
||||||
|
## Формална Верификация
|
||||||
|
|
||||||
|
MVCC / Snapshot Isolation протоколът е формално специфициран в TLA+:
|
||||||
|
|
||||||
|
- **Спецификация:** `formal-verification/mvcc.tla`
|
||||||
|
- **Проверени свойства:**
|
||||||
|
- `NoDirtyReads` — транзакциите никога не четат неcommit-нати данни
|
||||||
|
- `ReadOwnWrites` — транзакциите винаги виждат собствените си записи
|
||||||
|
- `WriteWriteConflict` — first-committer-wins
|
||||||
|
|
||||||
|
Пускане на TLC:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd formal-verification
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/mvcc.cfg mvcc.tla
|
||||||
|
```
|
||||||
@@ -133,6 +133,7 @@ Client → Protocol → Auth → Parser → AST → IR → Codegen
|
|||||||
5. **MVCC**: Multi-version concurrency control
|
5. **MVCC**: Multi-version concurrency control
|
||||||
6. **Schema-First**: Strongly typed schema system with inheritance
|
6. **Schema-First**: Strongly typed schema system with inheritance
|
||||||
7. **Cross-Modal**: Single query language across all data models
|
7. **Cross-Modal**: Single query language across all data models
|
||||||
|
8. **Formally Verified**: Core distributed algorithms specified in TLA+ and model-checked with TLC
|
||||||
|
|
||||||
## Module Statistics
|
## Module Statistics
|
||||||
|
|
||||||
|
|||||||
@@ -75,6 +75,14 @@ BARADB_WAL_ARCHIVE_INTERVAL_MS=60000 \
|
|||||||
--target-time="2025-01-15T10:30:00Z"
|
--target-time="2025-01-15T10:30:00Z"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Recovery via SQL
|
||||||
|
|
||||||
|
You can also recover directly via BaraQL:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
|
||||||
|
```
|
||||||
|
|
||||||
### Incremental Backups
|
### Incremental Backups
|
||||||
|
|
||||||
Incremental backups only copy changed SSTables:
|
Incremental backups only copy changed SSTables:
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ BaraQL is a SQL-compatible query language with extensions for graph, vector, and
|
|||||||
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
|
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
|
||||||
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
|
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
|
||||||
| `json` | JSON document | `{"key": "value"}` |
|
| `json` | JSON document | `{"key": "value"}` |
|
||||||
|
| `jsonb` | Binary JSON (validated) | `{"key": "value"}` |
|
||||||
|
|
||||||
## Basic Queries
|
## Basic Queries
|
||||||
|
|
||||||
@@ -360,6 +361,32 @@ DROP TYPE User;
|
|||||||
DROP INDEX idx_users_name;
|
DROP INDEX idx_users_name;
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### JSON Path Operators
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Extract JSON field as JSON
|
||||||
|
SELECT data->'name' FROM users;
|
||||||
|
|
||||||
|
-- Extract JSON field as text
|
||||||
|
SELECT data->>'name' FROM users;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Full-Text Search (SQL)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Create FTS index with BM25
|
||||||
|
CREATE INDEX idx_fts ON articles(body) USING FTS;
|
||||||
|
|
||||||
|
-- Search with BM25 ranking
|
||||||
|
SELECT * FROM articles WHERE body @@ 'machine learning';
|
||||||
|
```
|
||||||
|
|
||||||
|
### Point-in-Time Recovery
|
||||||
|
|
||||||
|
```sql
|
||||||
|
RECOVER TO TIMESTAMP '2026-05-07T12:00:00';
|
||||||
|
```
|
||||||
|
|
||||||
## Vector Search
|
## Vector Search
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
@@ -505,4 +532,7 @@ SELECT /*+ PARALLEL(4) */ * FROM large_table;
|
|||||||
| Graph | MATCH, RETURN, WHERE, shortestPath, type |
|
| Graph | MATCH, RETURN, WHERE, shortestPath, type |
|
||||||
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
|
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
|
||||||
| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance |
|
| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance |
|
||||||
|
| JSON | ->, ->> |
|
||||||
|
| FTS | @@ (BM25 match) |
|
||||||
|
| Recovery | RECOVER TO TIMESTAMP |
|
||||||
| Functions | count, sum, avg, min, max, stddev, variance, abs, sqrt, lower, upper, len, trim, substr, now, last_insert_id |
|
| Functions | count, sum, avg, min, max, stddev, variance, abs, sqrt, lower, upper, len, trim, substr, now, last_insert_id |
|
||||||
|
|||||||
@@ -88,3 +88,23 @@ var dt = newDistributedTxn()
|
|||||||
dt.prepare(@["node1", "node2"])
|
dt.prepare(@["node1", "node2"])
|
||||||
dt.commit()
|
dt.commit()
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Formal Verification
|
||||||
|
|
||||||
|
Core distributed algorithms are formally specified in TLA+ and model-checked:
|
||||||
|
|
||||||
|
- **Raft Consensus** — `formal-verification/raft.tla`
|
||||||
|
- Verified: ElectionSafety, StateMachineSafety
|
||||||
|
- **Two-Phase Commit** — `formal-verification/twopc.tla`
|
||||||
|
- Verified: Atomicity, NoOrphanBlocks
|
||||||
|
- **Replication** — `formal-verification/replication.tla`
|
||||||
|
- Verified: MonotonicLsn, AcksRemovePending
|
||||||
|
|
||||||
|
Run TLC locally:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd formal-verification
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/raft.cfg raft.tla
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/twopc.cfg twopc.tla
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/replication.cfg replication.tla
|
||||||
|
```
|
||||||
@@ -52,6 +52,24 @@ let tfidf = idx.searchTfidf("query terms")
|
|||||||
| Phrase search | Exact phrase matching |
|
| Phrase search | Exact phrase matching |
|
||||||
| Boolean | AND, OR, NOT operators |
|
| Boolean | AND, OR, NOT operators |
|
||||||
|
|
||||||
|
## SQL Interface
|
||||||
|
|
||||||
|
Full-text search is also available directly in BaraQL:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Create a table with text column
|
||||||
|
CREATE TABLE articles (id INT PRIMARY KEY, title TEXT, body TEXT);
|
||||||
|
|
||||||
|
-- Create an FTS index
|
||||||
|
CREATE INDEX idx_fts ON articles(body) USING FTS;
|
||||||
|
|
||||||
|
-- Search with the @@ operator (BM25 ranking)
|
||||||
|
SELECT * FROM articles WHERE body @@ 'machine learning';
|
||||||
|
|
||||||
|
-- Search with multiple terms
|
||||||
|
SELECT * FROM articles WHERE body @@ 'quick brown fox';
|
||||||
|
```
|
||||||
|
|
||||||
## Multi-Language Support
|
## Multi-Language Support
|
||||||
|
|
||||||
```nim
|
```nim
|
||||||
|
|||||||
@@ -59,3 +59,20 @@ tm.savepoint(txn, "sp1")
|
|||||||
# ... operations ...
|
# ... operations ...
|
||||||
tm.rollbackToSavepoint(txn, "sp1")
|
tm.rollbackToSavepoint(txn, "sp1")
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Formal Verification
|
||||||
|
|
||||||
|
The MVCC / Snapshot Isolation protocol is formally specified in TLA+:
|
||||||
|
|
||||||
|
- **Spec:** `formal-verification/mvcc.tla`
|
||||||
|
- **Verified properties:**
|
||||||
|
- `NoDirtyReads` — transactions never read uncommitted data
|
||||||
|
- `ReadOwnWrites` — transactions always see their own writes
|
||||||
|
- `WriteWriteConflict` — first-committer-wins (no two committed transactions write the same key)
|
||||||
|
|
||||||
|
Run TLC locally:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd formal-verification
|
||||||
|
java -cp tla2tools.jar tlc2.TLC -config models/mvcc.cfg mvcc.tla
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user