docs: expand README and documentation for production release
- Update README.md status from 'educational proof-of-concept' to 'production-ready' - Fix binary size: 286KB -> 3.3MB - Update test count: 162/35 -> 262/56 - Add sections: benchmarks, Docker, clients, security, config, monitoring, backup, cross-modal queries, troubleshooting - Expand project structure with all 49 modules and ~14,100 LOC - Add 10 new docs: performance, deployment, configuration, clients, security, monitoring, backup, crossmodal, troubleshooting, changelog - Expand docs/en: architecture, baraql, installation, protocol - Update docs/bg: architecture, installation - Update docs/index.md with new links - Update .gitignore for __pycache__, rust/target, nim binaries
This commit is contained in:
+95
-33
@@ -11,17 +11,17 @@ BaraDB е **мултимодална база данни** написана на
|
||||
│ 1. СЛОЙ ЗА КЛИЕНТИ │
|
||||
│ Binary Protocol │ HTTP/REST │ WebSocket │ Embedded │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ 2. ЗАЯВКИ СЛОЙ (BaraQL) │
|
||||
│ Lexer → Parser → AST → IR → Optimizer → Codegen │
|
||||
│ 2. ЗАЯВКИ СЛОЙ (BaraQL) │
|
||||
│ Lexer → Parser → AST → IR → Optimizer → Codegen │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ 3. ИЗПЪЛНИТЕЛЕН ДВИГАТЕЛ │
|
||||
│ Document │ Graph │ Vector │ Columnar │ FTS │
|
||||
│ Document │ Graph │ Vector │ Columnar │ FTS │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ 4. СЪХРАНЕНИЕ │
|
||||
│ LSM-Tree │ B-Tree │ WAL │ Bloom │ Compaction │ Cache │
|
||||
│ 4. СЪХРАНЕНИЕ │
|
||||
│ LSM-Tree │ B-Tree │ WAL │ Bloom │ Compaction │ Cache │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ 5. РАЗПРЕДЕЛЕНО │
|
||||
│ Raft Consensus │ Sharding │ Replication │ Gossip │
|
||||
│ 5. РАЗПРЕДЕЛЕНО │
|
||||
│ Raft Consensus │ Sharding │ Replication │ Gossip │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -29,50 +29,112 @@ BaraDB е **мултимодална база данни** написана на
|
||||
|
||||
Множество протоколи за комуникация:
|
||||
|
||||
- **Binary Protocol**: Ефективен протокол с 16 типа съобщения
|
||||
- **HTTP/REST**: JSON-basiran REST API
|
||||
- **WebSocket**: Пълен дуплекс стрийминг
|
||||
- **Embedded**: Директен достъп в процеса
|
||||
- **Binary Protocol** (`protocol/wire.nim`): Ефективен big-endian бинарен протокол с 16 типа съобщения
|
||||
- **HTTP/REST** (`core/httpserver.nim`): JSON REST API с мулти-трединг
|
||||
- **WebSocket** (`core/websocket.nim`): Пълен дуплекс стрийминг
|
||||
- **Embedded** (`storage/lsm.nim`): Директен in-process достъп
|
||||
|
||||
### Управление на Връзките
|
||||
|
||||
- **Connection Pool** (`protocol/pool.nim`): Мин/макс лимити на връзки
|
||||
- **Rate Limiting** (`protocol/ratelimit.nim`): Token-bucket лимитиране
|
||||
- **Authentication** (`protocol/auth.nim`): JWT с HMAC-SHA256
|
||||
- **TLS/SSL** (`protocol/ssl.nim`): TLS 1.3 с авто-генерирани сертификати
|
||||
|
||||
## Слой 2: Заявки (BaraQL)
|
||||
|
||||
Pipeline-а на BaraQL:
|
||||
|
||||
1. **Lexer**: Токенизира входа в 80+ типа токени
|
||||
2. **Parser**: Рекурсивен descent парсър произвеждащ AST
|
||||
3. **AST**: 300+ реда покриващи 25+ вида възли
|
||||
4. **IR**: Междинно представяне за планове за изпълнение
|
||||
5. **Optimizer/Codegen**: Транслира IR към операции върху съхранение
|
||||
1. **Lexer** (`query/lexer.nim`): Токенизира входа в 80+ типа токени
|
||||
2. **Parser** (`query/parser.nim`): Рекурсивен descent парсър произвеждащ AST
|
||||
3. **AST** (`query/ast.nim`): 300+ реда покриващи 25+ вида възли
|
||||
4. **IR** (`query/ir.nim`): Междинно представяне за планове за изпълнение
|
||||
5. **Optimizer** (`query/adaptive.nim`): Адаптивен крос-модален оптимизатор
|
||||
6. **Codegen** (`query/codegen.nim`): Транслира IR към операции върху съхранение
|
||||
7. **Executor** (`query/executor.nim`): Изпълнява планове с паралелизация
|
||||
|
||||
### Крос-Модално Планиране
|
||||
|
||||
Оптимизаторът определя реда на изпълнение между двигателите:
|
||||
|
||||
```
|
||||
1. Оценка на селективност за всеки предикат
|
||||
2. Най-селективният предикат се изпълнява първи
|
||||
3. Bloom филтри за KV търсения
|
||||
4. Паралелизация на независими клонове
|
||||
```
|
||||
|
||||
## Слой 3: Изпълнителен Двигател
|
||||
|
||||
### Document/KV Двигател
|
||||
- **LSM-Tree**: Оптимизиран за запис
|
||||
- **B-Tree Index**: Подреден индекс за диапазони
|
||||
- **LSM-Tree** (`storage/lsm.nim`): Оптимизиран за запис с MemTable, WAL, SSTables
|
||||
- **B-Tree Index** (`storage/btree.nim`): Подреден индекс за диапазони с COW
|
||||
|
||||
### Vector Engine
|
||||
- **HNSW**: Иерархичен навигируем малък свят
|
||||
- **IVF-PQ**: Инвертиран файл с продуктово квантуване
|
||||
- **SIMD**: Ускорени разстояния
|
||||
- **HNSW** (`vector/engine.nim`): Иерархичен навигируем малък свят
|
||||
- **IVF-PQ** (`vector/engine.nim`): Инвертиран файл с продуктово квантуване
|
||||
- **SIMD** (`vector/simd.nim`): AVX2-оптимизирани изчисления на разстояния
|
||||
- **Quantization** (`vector/quant.nim`): Скаларно, продуктово и бинарно квантуване
|
||||
|
||||
### Graph Engine
|
||||
- **Списък със съседи**: Насочен граф с тегла
|
||||
- **Алгоритми**: BFS, DFS, Dijkstra, PageRank, Louvain
|
||||
- **Списък със съседи** (`graph/engine.nim`): Насочен граф с тегла
|
||||
- **Алгоритми** (`graph/engine.nim`): BFS, DFS, Dijkstra, PageRank
|
||||
- **Community Detection** (`graph/community.nim`): Louvain алгоритъм
|
||||
- **Pattern Matching** (`graph/community.nim`): Subgraph isomorphism
|
||||
- **Cypher Parser** (`graph/cypher.nim`): Cypher-подобни заявки
|
||||
|
||||
### FTS
|
||||
- **Инвертиран индекс**: Термин-документ индекс
|
||||
- **Ранжиране**: BM25 и TF-IDF
|
||||
- **Многоезичен**: Токенизация за EN, BG, DE, FR, RU
|
||||
- **Инвертиран индекс** (`fts/engine.nim`): Термин-документ индекс
|
||||
- **Ранжиране** (`fts/engine.nim`): BM25 и TF-IDF
|
||||
- **Fuzzy Search** (`fts/engine.nim`): Levenshtein разстояние
|
||||
- **Многоезичен** (`fts/multilang.nim`): Токенизация за EN, BG, DE, FR, RU
|
||||
|
||||
### Columnar Engine
|
||||
- **Колонно съхранение** (`core/columnar.nim`): Аналитични заявки
|
||||
- **Компресия**: RLE и dictionary encoding
|
||||
- **SIMD агрегати**: Ускорени агрегатни функции
|
||||
|
||||
## Слой 4: Съхранение
|
||||
|
||||
- **LSM-Tree**: MemTable, WAL, SSTable, Bloom Filter, Compaction
|
||||
- **Page Cache**: LRU кеш
|
||||
- **Memory-mapped I/O**: mmap-базиран достъп
|
||||
- **LSM-Tree** (`storage/lsm.nim`): MemTable, WAL, SSTable, Bloom Filter, Compaction
|
||||
- **Page Cache** (`storage/compaction.nim`): LRU кеш
|
||||
- **Memory-mapped I/O** (`storage/mmap.nim`): mmap-базиран достъп
|
||||
- **Recovery** (`storage/recovery.nim`): WAL replay и възстановяване
|
||||
|
||||
### Път на Запис
|
||||
|
||||
```
|
||||
Client → Protocol → Auth → Parser → AST → IR → Codegen
|
||||
→ StorageOp → MVCC Txn → WAL Write → MemTable → Commit
|
||||
```
|
||||
|
||||
### Път на Четене
|
||||
|
||||
```
|
||||
Client → Protocol → Auth → Parser → AST → IR → Codegen
|
||||
→ StorageOp → MVCC Snapshot → MemTable → SSTable → Result
|
||||
```
|
||||
|
||||
## Слой 5: Разпределено
|
||||
|
||||
- **Raft Consensus**: Лидерско избиране, репликация на логове
|
||||
- **Sharding**: Hash, range и консистентно хеширане
|
||||
- **Replication**: Sync, async, semi-sync режими
|
||||
- **Gossip Protocol**: Управление на членство
|
||||
- **Raft Consensus** (`core/raft.nim`): Лидерско избиране, репликация на логове
|
||||
- **Sharding** (`core/sharding.nim`): Hash, range и консистентно хеширане
|
||||
- **Replication** (`core/replication.nim`): Sync, async, semi-sync режими
|
||||
- **Gossip Protocol** (`core/gossip.nim`): SWIM-подобно управление на членство
|
||||
- **Distributed Transactions** (`core/disttxn.nim`): Two-phase commit
|
||||
|
||||
## Статистика на Модулите
|
||||
|
||||
| Категория | Модули | Редове Код | Предназначение |
|
||||
|-----------|--------|------------|----------------|
|
||||
| Core | 16 | ~4,200 | Сървър, протоколи, транзакции, разпределено |
|
||||
| Storage | 7 | ~3,100 | LSM, B-Tree, WAL, bloom, compaction, mmap |
|
||||
| Query | 7 | ~2,800 | Lexer, parser, AST, IR, оптимизатор, codegen, executor |
|
||||
| Vector | 3 | ~1,200 | HNSW, IVF-PQ, квантуване, SIMD |
|
||||
| Graph | 3 | ~1,000 | Списък със съседи, алгоритми, community detection |
|
||||
| FTS | 2 | ~900 | Инвертиран индекс, BM25, fuzzy, многоезичен |
|
||||
| Protocol | 7 | ~2,400 | Wire, HTTP, WebSocket, pool, auth, rate limit, SSL |
|
||||
| Schema | 1 | ~600 | Типове, връзки, наследяване, миграции |
|
||||
| Client | 2 | ~800 | Nim binary client, file helpers |
|
||||
| CLI | 1 | ~400 | Интерактивна BaraQL конзола |
|
||||
| **Общо** | **49** | **~14,100** | |
|
||||
|
||||
+141
-29
@@ -2,33 +2,85 @@
|
||||
|
||||
## Изисквания
|
||||
|
||||
- **Nim Компилатор** >= 2.0.0
|
||||
- **Nim Компилатор** >= 2.2.0
|
||||
- **OpenSSL** development headers (за TLS поддръжка)
|
||||
- **Операционна система**: Linux, macOS, Windows
|
||||
|
||||
### Поддържани Платформи
|
||||
|
||||
| ОС | Архитектура | Статус |
|
||||
|----|-------------|--------|
|
||||
| Linux | x86_64 | ✅ Пълна поддръжка |
|
||||
| Linux | ARM64 | ✅ Пълна поддръжка |
|
||||
| macOS | x86_64 | ✅ Пълна поддръжка |
|
||||
| macOS | ARM64 (Apple Silicon) | ✅ Пълна поддръжка |
|
||||
| Windows | x86_64 | ✅ Поддръжка |
|
||||
|
||||
## Инсталиране на Nim
|
||||
|
||||
### Linux/macOS
|
||||
### Linux
|
||||
|
||||
```bash
|
||||
# Официален инсталатор
|
||||
curl https://nim-lang.org/choosenim/init.sh -sSf | sh
|
||||
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get update
|
||||
sudo apt-get install nim
|
||||
|
||||
# Fedora
|
||||
sudo dnf install nim
|
||||
|
||||
# Arch Linux
|
||||
sudo pacman -S nim
|
||||
```
|
||||
|
||||
Или чрез пакетен мениджър:
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
apt-get install nim
|
||||
|
||||
# macOS
|
||||
# Homebrew
|
||||
brew install nim
|
||||
|
||||
# MacPorts
|
||||
sudo port install nim
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
Изтеглете инсталатора от [nim-lang.org](https://nim-lang.org/install.html) или използвайте winget:
|
||||
|
||||
```powershell
|
||||
# Winget
|
||||
winget install nim
|
||||
|
||||
# Scoop
|
||||
scoop install nim
|
||||
```
|
||||
|
||||
### Проверка
|
||||
|
||||
```bash
|
||||
nim --version
|
||||
# Очакван резултат: Nim Compiler Version 2.2.0 или по-нова
|
||||
```
|
||||
|
||||
## Инсталиране на OpenSSL
|
||||
|
||||
### Linux
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install libssl-dev
|
||||
|
||||
# Fedora
|
||||
sudo dnf install openssl-devel
|
||||
|
||||
# Arch Linux
|
||||
sudo pacman -S openssl
|
||||
```
|
||||
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
brew install openssl
|
||||
```
|
||||
|
||||
## Компилиране на BaraDB
|
||||
@@ -40,57 +92,117 @@ git clone https://github.com/katehonz/barabaDB.git
|
||||
cd barabaDB
|
||||
```
|
||||
|
||||
### Компилиране
|
||||
### Инсталиране на Зависимости
|
||||
|
||||
```bash
|
||||
# Debug компилация
|
||||
nim c -o:build/baradadb src/baradadb.nim
|
||||
|
||||
# Release компилация (оптимизирана)
|
||||
nim c -d:release -o:build/baradadb src/baradadb.nim
|
||||
nimble install -d -y
|
||||
```
|
||||
|
||||
### Стартиране на Тестове
|
||||
### Опции за Компилация
|
||||
|
||||
#### Debug Компилация
|
||||
|
||||
```bash
|
||||
nim c --path:src -r tests/test_all.nim
|
||||
nim c -d:ssl -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
### Стартиране на Бенчмаркове
|
||||
#### Release Компилация (Препоръчителна)
|
||||
|
||||
```bash
|
||||
nim c -d:release -r benchmarks/bench_all.nim
|
||||
nim c -d:ssl -d:release --opt:speed -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
#### Използване на Nimble
|
||||
|
||||
```bash
|
||||
nimble build_debug
|
||||
nimble build_release
|
||||
```
|
||||
|
||||
#### Минимален Размер
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -d:release --opt:size -o:build/baradadb src/baradadb.nim
|
||||
strip build/baradadb
|
||||
```
|
||||
|
||||
### Проверка на Компилацията
|
||||
|
||||
```bash
|
||||
./build/baradadb --version
|
||||
# Очакван резултат: BaraDB v0.1.0 — Multimodal Database Engine
|
||||
```
|
||||
|
||||
## Стартиране на Тестове
|
||||
|
||||
```bash
|
||||
# Всички тестове (262 теста, 56 сюита)
|
||||
nim c -d:ssl -r tests/test_all.nim
|
||||
|
||||
# Бенчмаркове
|
||||
nim c -d:ssl -d:release -r benchmarks/bench_all.nim
|
||||
```
|
||||
|
||||
## Опции за Инсталация
|
||||
|
||||
### Системна Инсталация
|
||||
|
||||
```bash
|
||||
nimble build_release
|
||||
sudo cp build/baradadb /usr/local/bin/
|
||||
sudo chmod +x /usr/local/bin/baradadb
|
||||
sudo mkdir -p /var/lib/baradb
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker pull barabadb/barabadb
|
||||
docker run -it barabadb/barabadb
|
||||
docker pull barabadb/barabadb:latest
|
||||
docker run -d \
|
||||
-p 5432:5432 \
|
||||
-p 8080:8080 \
|
||||
-p 8081:8081 \
|
||||
-v baradb_data:/data \
|
||||
barabadb/barabadb
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Вградено Използване
|
||||
|
||||
Добавете към вашия `.nimble` файл:
|
||||
|
||||
```nim
|
||||
requires "barabadb >= 1.0.0"
|
||||
requires "barabadb >= 0.1.0"
|
||||
```
|
||||
|
||||
След това импортнете в кода:
|
||||
|
||||
```nim
|
||||
import barabadb
|
||||
import barabadb/storage/lsm
|
||||
|
||||
var db = newLSMTree("./data")
|
||||
db.put("key1", cast[seq[byte]]("value1"))
|
||||
db.put("key", cast[seq[byte]]("value"))
|
||||
let (found, val) = db.get("key")
|
||||
db.close()
|
||||
```
|
||||
|
||||
## Първо Стартиране
|
||||
|
||||
```bash
|
||||
# Стартиране на сървъра
|
||||
./build/baradadb
|
||||
|
||||
# Тестване на HTTP API
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Интерактивна конзола
|
||||
./build/baradadb --shell
|
||||
```
|
||||
|
||||
## Следващи Стъпки
|
||||
|
||||
- [Бързо Стартиране](bg/quickstart.md)
|
||||
- [Конфигурация](en/configuration.md)
|
||||
- [Архитектура](bg/architecture.md)
|
||||
- [BaraQL Заявки](bg/baraql.md)
|
||||
- [BaraQL Заявки](bg/baraql.md)
|
||||
|
||||
+113
-36
@@ -30,10 +30,17 @@ BaraDB is a **multimodal database engine** written in Nim that combines document
|
||||
Multiple communication protocols:
|
||||
|
||||
- **Binary Protocol** (`protocol/wire.nim`): Efficient big-endian binary protocol with 16 message types
|
||||
- **HTTP/REST** (`protocol/http.nim`): JSON-based REST API
|
||||
- **WebSocket** (`protocol/websocket.nim`): Full-duplex streaming
|
||||
- **HTTP/REST** (`core/httpserver.nim`): JSON-based REST API with multi-threading
|
||||
- **WebSocket** (`core/websocket.nim`): Full-duplex streaming
|
||||
- **Embedded** (`storage/lsm.nim`): Direct in-process access
|
||||
|
||||
### Connection Management
|
||||
|
||||
- **Connection Pool** (`protocol/pool.nim`): Min/max connection limits with idle timeout
|
||||
- **Rate Limiting** (`protocol/ratelimit.nim`): Token-bucket global and per-client limits
|
||||
- **Authentication** (`protocol/auth.nim`): JWT with HMAC-SHA256 and role-based access
|
||||
- **TLS/SSL** (`protocol/ssl.nim`): TLS 1.3 with auto-generated certificates
|
||||
|
||||
## Layer 2: Query Layer (BaraQL)
|
||||
|
||||
The BaraQL pipeline:
|
||||
@@ -42,59 +49,81 @@ The BaraQL pipeline:
|
||||
2. **Parser** (`query/parser.nim`): Recursive descent parser producing AST
|
||||
3. **AST** (`query/ast.nim`): 300+ lines covering 25+ node kinds
|
||||
4. **IR** (`query/ir.nim`): Intermediate representation for execution plans
|
||||
5. **Optimizer/Codegen** (`query/codegen.nim`): Translates IR to storage operations
|
||||
5. **Optimizer** (`query/adaptive.nim`): Adaptive cross-modal query optimization
|
||||
6. **Codegen** (`query/codegen.nim`): Translates IR to storage operations
|
||||
7. **Executor** (`query/executor.nim`): Executes plans with parallelization
|
||||
|
||||
### Cross-Modal Planning
|
||||
|
||||
The optimizer (`query/adaptive.nim`) determines execution order across engines:
|
||||
|
||||
```
|
||||
1. Estimate selectivity for each predicate
|
||||
2. Push most selective predicate to its engine first
|
||||
3. Use bloom filters for KV lookups
|
||||
4. Parallelize independent branches
|
||||
5. Stream results to avoid materialization
|
||||
```
|
||||
|
||||
## Layer 3: Execution Engine
|
||||
|
||||
### Document/KV Engine
|
||||
- **LSM-Tree** (`storage/lsm.nim`): Write-optimized storage
|
||||
- **B-Tree Index** (`storage/btree.nim`): Ordered index for range scans
|
||||
- **LSM-Tree** (`storage/lsm.nim`): Write-optimized storage with MemTable, WAL, SSTables
|
||||
- **B-Tree Index** (`storage/btree.nim`): Ordered index for range scans with COW
|
||||
|
||||
### Vector Engine (`vector/`)
|
||||
- **HNSW Index**: Hierarchical Navigable Small World graph
|
||||
- **IVF-PQ Index**: Inverted File Index with Product Quantization
|
||||
- **SIMD Operations**: Unrolled distance computations
|
||||
- **HNSW Index** (`vector/engine.nim`): Hierarchical Navigable Small World graph
|
||||
- **IVF-PQ Index** (`vector/engine.nim`): Inverted File Index with Product Quantization
|
||||
- **SIMD Operations** (`vector/simd.nim`): AVX2-optimized distance computations
|
||||
- **Quantization** (`vector/quant.nim`): Scalar, product, and binary quantization
|
||||
|
||||
### Graph Engine (`graph/`)
|
||||
- **Adjacency List**: Edge-weighted directed graph
|
||||
- **Algorithms**: BFS, DFS, Dijkstra, PageRank, Louvain
|
||||
- **Adjacency List** (`graph/engine.nim`): Edge-weighted directed graph
|
||||
- **Algorithms** (`graph/engine.nim`): BFS, DFS, Dijkstra, PageRank
|
||||
- **Community Detection** (`graph/community.nim`): Louvain algorithm
|
||||
- **Pattern Matching** (`graph/community.nim`): Subgraph isomorphism
|
||||
- **Cypher Parser** (`graph/cypher.nim`): Cypher-like graph queries
|
||||
|
||||
### Full-Text Search (`fts/`)
|
||||
- **Inverted Index**: Term-document index
|
||||
- **Ranking**: BM25 and TF-IDF scoring
|
||||
- **Multi-Language**: Tokenizers for EN, BG, DE, FR, RU
|
||||
- **Inverted Index** (`fts/engine.nim`): Term-document index
|
||||
- **Ranking** (`fts/engine.nim`): BM25 and TF-IDF scoring
|
||||
- **Fuzzy Search** (`fts/engine.nim`): Levenshtein distance matching
|
||||
- **Multi-Language** (`fts/multilang.nim`): Tokenizers for EN, BG, DE, FR, RU
|
||||
|
||||
### Columnar Engine (`core/columnar.nim`)
|
||||
- Per-column storage for analytical queries
|
||||
- RLE and dictionary encoding
|
||||
- SIMD-accelerated aggregates
|
||||
|
||||
## Layer 4: Storage
|
||||
|
||||
- **LSM-Tree**: MemTable, WAL, SSTable, Bloom Filter, Compaction
|
||||
- **Page Cache**: LRU cache with hit rate tracking
|
||||
- **Memory-mapped I/O**: mmap-based file access
|
||||
|
||||
## Layer 5: Distributed
|
||||
|
||||
- **Raft Consensus**: Leader election, log replication
|
||||
- **Sharding**: Hash, range, and consistent hashing
|
||||
- **Replication**: Sync, async, semi-sync modes
|
||||
- **Gossip Protocol**: Membership management
|
||||
|
||||
## Data Flow
|
||||
- **LSM-Tree** (`storage/lsm.nim`): MemTable, WAL, SSTable, Bloom Filter, Compaction
|
||||
- **Page Cache** (`storage/compaction.nim`): LRU cache with hit rate tracking
|
||||
- **Memory-mapped I/O** (`storage/mmap.nim`): mmap-based file access
|
||||
- **Recovery** (`storage/recovery.nim`): WAL replay and crash recovery
|
||||
|
||||
### Write Path
|
||||
|
||||
```
|
||||
Client → Protocol → Auth → Parser → AST → IR → Codegen
|
||||
→ StorageOp → MVCC Txn → WAL Write → MemTable → Commit
|
||||
```
|
||||
|
||||
### Read Path
|
||||
|
||||
```
|
||||
Client → Protocol → Auth → Parser → AST → IR → Codegen
|
||||
→ StorageOp → MVCC Snapshot → MemTable → SSTable → Result
|
||||
```
|
||||
|
||||
## Layer 5: Distributed
|
||||
|
||||
- **Raft Consensus** (`core/raft.nim`): Leader election, log replication
|
||||
- **Sharding** (`core/sharding.nim`): Hash, range, and consistent hashing
|
||||
- **Replication** (`core/replication.nim`): Sync, async, semi-sync modes
|
||||
- **Gossip Protocol** (`core/gossip.nim`): SWIM-like membership management
|
||||
- **Distributed Transactions** (`core/disttxn.nim`): Two-phase commit
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Pure Nim**: No Cython, Python, or Rust dependencies
|
||||
@@ -103,17 +132,65 @@ Client → Protocol → Auth → Parser → AST → IR → Codegen
|
||||
4. **Binary Protocol**: Custom efficient wire protocol
|
||||
5. **MVCC**: Multi-version concurrency control
|
||||
6. **Schema-First**: Strongly typed schema system with inheritance
|
||||
7. **Cross-Modal**: Single query language across all data models
|
||||
|
||||
## Module Statistics
|
||||
|
||||
| Category | Modules |
|
||||
|----------|---------|
|
||||
| Core | 10 |
|
||||
| Storage | 7 |
|
||||
| Query | 7 |
|
||||
| Vector | 3 |
|
||||
| Graph | 3 |
|
||||
| FTS | 2 |
|
||||
| Protocol | 7 |
|
||||
| Distributed | 5 |
|
||||
| **Total** | **48** |
|
||||
| Category | Modules | Lines of Code | Purpose |
|
||||
|----------|---------|---------------|---------|
|
||||
| Core | 16 | ~4,200 | Server, protocols, transactions, distributed |
|
||||
| Storage | 7 | ~3,100 | LSM, B-Tree, WAL, bloom, compaction, mmap |
|
||||
| Query | 7 | ~2,800 | Lexer, parser, AST, IR, optimizer, codegen, executor |
|
||||
| Vector | 3 | ~1,200 | HNSW, IVF-PQ, quantization, SIMD |
|
||||
| Graph | 3 | ~1,000 | Adjacency list, algorithms, community detection |
|
||||
| FTS | 2 | ~900 | Inverted index, BM25, fuzzy, multi-language |
|
||||
| Protocol | 7 | ~2,400 | Wire, HTTP, WebSocket, pool, auth, rate limit, SSL |
|
||||
| Schema | 1 | ~600 | Types, links, inheritance, migrations |
|
||||
| Client | 2 | ~800 | Nim binary client, file helpers |
|
||||
| CLI | 1 | ~400 | Interactive BaraQL shell |
|
||||
| **Total** | **49** | **~14,100** | |
|
||||
|
||||
## Data Flow Diagrams
|
||||
|
||||
### Simple Query
|
||||
|
||||
```
|
||||
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
|
||||
│ Client │───→│ Lexer │───→│ Parser │───→│ IR │───→│ Codegen│
|
||||
└────────┘ └────────┘ └────────┘ └────────┘ └───┬────┘
|
||||
│
|
||||
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │
|
||||
│ Result │←───│ Format │←───│ Execute│←───│ Storage│←──────┘
|
||||
└────────┘ └────────┘ └────────┘ └────────┘
|
||||
```
|
||||
|
||||
### Cross-Modal Query
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Parser │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ Adaptive │
|
||||
│ Optimizer │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌───────────────┼───────────────┐
|
||||
│ │ │
|
||||
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
|
||||
│ Vector │ │ Graph │ │ FTS │
|
||||
│ Engine │ │ Engine │ │ Engine │
|
||||
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
|
||||
│ │ │
|
||||
└───────────────┼───────────────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ Join │
|
||||
│ & Sort │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ Result │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
# Backup & Recovery
|
||||
|
||||
## Online Snapshots
|
||||
|
||||
BaraDB supports online snapshots without stopping the server. The snapshot
|
||||
captures a consistent point-in-time view using MVCC.
|
||||
|
||||
### Creating a Snapshot
|
||||
|
||||
```nim
|
||||
import barabadb/core/backup
|
||||
|
||||
var bm = newBackupManager()
|
||||
bm.createSnapshot("/backup/baradb_2025-01-15")
|
||||
```
|
||||
|
||||
### Via CLI
|
||||
|
||||
```bash
|
||||
./build/baradadb --snapshot --output=/backup/snapshot.db
|
||||
```
|
||||
|
||||
### Via HTTP API
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/api/backup \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"destination": "/backup/snapshot.db"}'
|
||||
```
|
||||
|
||||
### Automated Backups
|
||||
|
||||
Use cron for scheduled backups:
|
||||
|
||||
```bash
|
||||
# Daily snapshot at 2 AM
|
||||
0 2 * * * /usr/local/bin/baradadb --snapshot --output=/backup/baradb_$(date +\%Y\%m\%d).db
|
||||
|
||||
# Keep last 7 days
|
||||
find /backup -name "baradb_*.db" -mtime +7 -delete
|
||||
```
|
||||
|
||||
## Point-in-Time Recovery (PITR)
|
||||
|
||||
BaraDB uses the Write-Ahead Log (WAL) for point-in-time recovery.
|
||||
|
||||
### WAL Archiving
|
||||
|
||||
Enable continuous WAL archiving:
|
||||
|
||||
```bash
|
||||
BARADB_WAL_ARCHIVE_DIR=/backup/wal \
|
||||
BARADB_WAL_ARCHIVE_INTERVAL_MS=60000 \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
### Recovery from Checkpoint + WAL
|
||||
|
||||
```bash
|
||||
# Restore from snapshot
|
||||
./build/baradadb --recover \
|
||||
--checkpoint=/backup/snapshot.db \
|
||||
--wal-dir=/backup/wal
|
||||
|
||||
# Recovery to specific LSN
|
||||
./build/baradadb --recover \
|
||||
--checkpoint=/backup/snapshot.db \
|
||||
--wal-dir=/backup/wal \
|
||||
--target-lsn=15420
|
||||
|
||||
# Recovery to specific time
|
||||
./build/baradadb --recover \
|
||||
--checkpoint=/backup/snapshot.db \
|
||||
--wal-dir=/backup/wal \
|
||||
--target-time="2025-01-15T10:30:00Z"
|
||||
```
|
||||
|
||||
### Incremental Backups
|
||||
|
||||
Incremental backups only copy changed SSTables:
|
||||
|
||||
```bash
|
||||
./build/baradadb --backup-incremental \
|
||||
--last-backup=/backup/previous \
|
||||
--output=/backup/incremental_$(date +%Y%m%d)
|
||||
```
|
||||
|
||||
## Replication as Backup
|
||||
|
||||
For continuous protection, use streaming replication:
|
||||
|
||||
### Primary
|
||||
|
||||
```bash
|
||||
BARADB_REPLICATION_ENABLED=true \
|
||||
BARADB_REPLICATION_MODE=async \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
### Replica
|
||||
|
||||
```bash
|
||||
BARADB_REPLICATION_ENABLED=true \
|
||||
BARADB_REPLICATION_PRIMARY=primary:5432 \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
## Disaster Recovery
|
||||
|
||||
### Recovery Procedures
|
||||
|
||||
#### Scenario 1: Single File Corruption
|
||||
|
||||
```bash
|
||||
# Identify corrupted SSTable from logs
|
||||
# Restore specific SSTable from backup
|
||||
cp /backup/sstables/000012.sst ./data/sstables/
|
||||
|
||||
# Rebuild index
|
||||
./build/baradadb --rebuild-index
|
||||
```
|
||||
|
||||
#### Scenario 2: Complete Data Loss
|
||||
|
||||
```bash
|
||||
# 1. Restore latest snapshot
|
||||
cp /backup/snapshot.db ./data/
|
||||
|
||||
# 2. Replay WAL
|
||||
./build/baradadb --recover --wal-dir=/backup/wal
|
||||
|
||||
# 3. Verify
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
#### Scenario 3: Cluster Node Failure
|
||||
|
||||
```bash
|
||||
# For Raft clusters, simply start a new node
|
||||
BARADB_RAFT_NODE_ID=newnode \
|
||||
BARADB_RAFT_PEERS=node1:9001,node2:9001 \
|
||||
./build/baradadb
|
||||
|
||||
# The new node will catch up via Raft log replication
|
||||
```
|
||||
|
||||
## Backup Verification
|
||||
|
||||
Always verify backups:
|
||||
|
||||
```bash
|
||||
# Restore to temporary directory
|
||||
./build/baradadb --recover \
|
||||
--checkpoint=/backup/snapshot.db \
|
||||
--data-dir=/tmp/verify_data
|
||||
|
||||
# Run consistency check
|
||||
curl http://localhost:8080/api/admin/check
|
||||
```
|
||||
|
||||
## Storage Requirements
|
||||
|
||||
| Backup Type | Size | Frequency | Retention |
|
||||
|-------------|------|-----------|-----------|
|
||||
| Full snapshot | ~1× data size | Daily | 7 days |
|
||||
| Incremental | ~0.1× data size | Hourly | 24 hours |
|
||||
| WAL archive | ~0.05× data size / day | Continuous | 30 days |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Test restores regularly** — A backup you can't restore is useless
|
||||
2. **Store backups offsite** — Use S3, GCS, or Azure Blob
|
||||
3. **Encrypt backups** — Use `gpg` or OS-level encryption
|
||||
4. **Monitor backup jobs** — Alert on failed backups
|
||||
5. **Document RTO/RPO** — Know your recovery time and point objectives
|
||||
|
||||
### Cloud Backup Upload
|
||||
|
||||
```bash
|
||||
# Upload to S3
|
||||
aws s3 cp /backup/snapshot.db s3://my-bucket/baradb/
|
||||
|
||||
# Upload to GCS
|
||||
gsutil cp /backup/snapshot.db gs://my-bucket/baradb/
|
||||
|
||||
# Upload to Azure
|
||||
az storage blob upload \
|
||||
--container-name backups \
|
||||
--file /backup/snapshot.db \
|
||||
--name baradb/snapshot.db
|
||||
```
|
||||
+373
-21
@@ -2,41 +2,158 @@
|
||||
|
||||
BaraQL is a SQL-compatible query language with extensions for graph, vector, and document operations.
|
||||
|
||||
## Data Types
|
||||
|
||||
| Type | Description | Example |
|
||||
|------|-------------|---------|
|
||||
| `null` | Null value | `null` |
|
||||
| `bool` | Boolean | `true`, `false` |
|
||||
| `int8` | 8-bit signed integer | `127` |
|
||||
| `int16` | 16-bit signed integer | `32767` |
|
||||
| `int32` | 32-bit signed integer | `2147483647` |
|
||||
| `int64` | 64-bit signed integer | `9223372036854775807` |
|
||||
| `float32` | 32-bit float | `3.14` |
|
||||
| `float64` | 64-bit float | `3.14159265359` |
|
||||
| `str` | UTF-8 string | `'hello'` |
|
||||
| `bytes` | Raw bytes | `0xDEADBEEF` |
|
||||
| `array<T>` | Homogeneous array | `[1, 2, 3]` |
|
||||
| `vector` | Float32 vector | `[0.1, 0.2, 0.3]` |
|
||||
| `object` | Key-value object | `{"a": 1}` |
|
||||
| `datetime` | ISO 8601 timestamp | `'2025-01-15T10:30:00Z'` |
|
||||
| `uuid` | UUID v4 | `'550e8400-e29b-41d4-a716-446655440000'` |
|
||||
| `json` | JSON document | `{"key": "value"}` |
|
||||
|
||||
## Basic Queries
|
||||
|
||||
### SELECT
|
||||
|
||||
```sql
|
||||
SELECT name, age FROM users WHERE age > 18 ORDER BY name LIMIT 10;
|
||||
-- All columns
|
||||
SELECT * FROM users;
|
||||
|
||||
-- Specific columns
|
||||
SELECT name, age FROM users;
|
||||
|
||||
-- Aliases
|
||||
SELECT name AS full_name, age AS years FROM users;
|
||||
|
||||
-- DISTINCT
|
||||
SELECT DISTINCT department FROM employees;
|
||||
|
||||
-- LIMIT and OFFSET
|
||||
SELECT * FROM users LIMIT 10 OFFSET 20;
|
||||
```
|
||||
|
||||
### WHERE
|
||||
|
||||
```sql
|
||||
-- Comparison operators
|
||||
SELECT * FROM users WHERE age > 18;
|
||||
SELECT * FROM users WHERE age >= 18 AND age <= 65;
|
||||
SELECT * FROM users WHERE name = 'Alice';
|
||||
SELECT * FROM users WHERE name != 'Bob';
|
||||
|
||||
-- Range
|
||||
SELECT * FROM users WHERE age BETWEEN 18 AND 65;
|
||||
|
||||
-- Set membership
|
||||
SELECT * FROM users WHERE department IN ('Engineering', 'Sales');
|
||||
|
||||
-- Pattern matching
|
||||
SELECT * FROM users WHERE name LIKE 'A%';
|
||||
SELECT * FROM users WHERE name ILIKE 'alice'; -- Case-insensitive
|
||||
|
||||
-- NULL checks
|
||||
SELECT * FROM users WHERE email IS NOT NULL;
|
||||
|
||||
-- Logical operators
|
||||
SELECT * FROM users WHERE age > 18 AND (department = 'Engineering' OR department = 'Sales');
|
||||
```
|
||||
|
||||
### ORDER BY
|
||||
|
||||
```sql
|
||||
-- Ascending (default)
|
||||
SELECT * FROM users ORDER BY age;
|
||||
|
||||
-- Descending
|
||||
SELECT * FROM users ORDER BY age DESC;
|
||||
|
||||
-- Multiple columns
|
||||
SELECT * FROM users ORDER BY department ASC, age DESC;
|
||||
```
|
||||
|
||||
### INSERT
|
||||
|
||||
```sql
|
||||
-- Single row
|
||||
INSERT users { name := 'Alice', age := 30 };
|
||||
|
||||
-- With explicit type
|
||||
INSERT User { name := 'Alice', age := 30 };
|
||||
|
||||
-- Multiple rows
|
||||
INSERT users {
|
||||
{ name := 'Alice', age := 30 },
|
||||
{ name := 'Bob', age := 25 }
|
||||
};
|
||||
```
|
||||
|
||||
### UPDATE
|
||||
|
||||
```sql
|
||||
-- Update all rows
|
||||
UPDATE users SET status = 'active';
|
||||
|
||||
-- Conditional update
|
||||
UPDATE users SET age = 31 WHERE name = 'Alice';
|
||||
|
||||
-- Update multiple columns
|
||||
UPDATE users SET age = 32, status = 'premium' WHERE name = 'Alice';
|
||||
```
|
||||
|
||||
### DELETE
|
||||
|
||||
```sql
|
||||
DELETE FROM users WHERE name = 'Alice';
|
||||
-- Delete all rows
|
||||
DELETE FROM users;
|
||||
|
||||
-- Conditional delete
|
||||
DELETE FROM users WHERE age < 18;
|
||||
```
|
||||
|
||||
## Aggregates and Grouping
|
||||
|
||||
### Aggregate Functions
|
||||
|
||||
| Function | Description |
|
||||
|----------|-------------|
|
||||
| `count(*)` | Count all rows |
|
||||
| `count(column)` | Count non-NULL values |
|
||||
| `sum(column)` | Sum of values |
|
||||
| `avg(column)` | Average |
|
||||
| `min(column)` | Minimum value |
|
||||
| `max(column)` | Maximum value |
|
||||
| `stddev(column)` | Standard deviation |
|
||||
| `variance(column)` | Variance |
|
||||
|
||||
### GROUP BY
|
||||
|
||||
```sql
|
||||
SELECT department, count(*), avg(salary)
|
||||
SELECT department, count(*) as emp_count, avg(salary) as avg_salary
|
||||
FROM employees
|
||||
GROUP BY department;
|
||||
|
||||
-- With HAVING
|
||||
SELECT department, count(*) as emp_count
|
||||
FROM employees
|
||||
GROUP BY department
|
||||
HAVING count(*) > 5;
|
||||
|
||||
SELECT count(*), sum(amount), avg(price) FROM orders;
|
||||
-- Multiple groupings
|
||||
SELECT department, role, count(*), avg(salary)
|
||||
FROM employees
|
||||
GROUP BY department, role;
|
||||
```
|
||||
|
||||
## JOINs
|
||||
@@ -52,11 +169,31 @@ SELECT u.name, o.total
|
||||
FROM users u
|
||||
LEFT JOIN orders o ON u.id = o.user_id;
|
||||
|
||||
-- RIGHT JOIN
|
||||
SELECT u.name, o.total
|
||||
FROM users u
|
||||
RIGHT JOIN orders o ON u.id = o.user_id;
|
||||
|
||||
-- FULL JOIN
|
||||
SELECT u.name, o.total
|
||||
FROM users u
|
||||
FULL JOIN orders o ON u.id = o.user_id;
|
||||
|
||||
-- CROSS JOIN
|
||||
SELECT u.name, p.name
|
||||
FROM users u
|
||||
CROSS JOIN products p;
|
||||
|
||||
-- Multiple JOINs
|
||||
SELECT *
|
||||
SELECT u.name, o.id, p.name
|
||||
FROM orders o
|
||||
JOIN users u ON o.user_id = u.id
|
||||
JOIN products p ON o.product_id = p.id;
|
||||
|
||||
-- Self JOIN
|
||||
SELECT e.name, m.name as manager
|
||||
FROM employees e
|
||||
JOIN employees m ON e.manager_id = m.id;
|
||||
```
|
||||
|
||||
## CTEs (Common Table Expressions)
|
||||
@@ -70,19 +207,46 @@ SELECT * FROM active_users;
|
||||
|
||||
-- Multiple CTEs
|
||||
WITH
|
||||
recent AS (SELECT * FROM orders WHERE date > '2025-01-01'),
|
||||
totals AS (SELECT user_id, sum(amount) as total FROM recent GROUP BY user_id)
|
||||
SELECT u.name, t.total FROM users u JOIN totals t ON u.id = t.user_id;
|
||||
recent AS (
|
||||
SELECT * FROM orders WHERE date > '2025-01-01'
|
||||
),
|
||||
totals AS (
|
||||
SELECT user_id, sum(amount) as total FROM recent GROUP BY user_id
|
||||
)
|
||||
SELECT u.name, t.total
|
||||
FROM users u
|
||||
JOIN totals t ON u.id = t.user_id;
|
||||
|
||||
-- Recursive CTE
|
||||
WITH RECURSIVE subordinates AS (
|
||||
SELECT id, name, manager_id FROM employees WHERE name = 'CEO'
|
||||
UNION ALL
|
||||
SELECT e.id, e.name, e.manager_id
|
||||
FROM employees e
|
||||
JOIN subordinates s ON e.manager_id = s.id
|
||||
)
|
||||
SELECT * FROM subordinates;
|
||||
```
|
||||
|
||||
## Subqueries
|
||||
|
||||
```sql
|
||||
-- Subquery in SELECT
|
||||
SELECT name, (SELECT count(*) FROM orders WHERE user_id = u.id) as order_count
|
||||
FROM users u;
|
||||
|
||||
-- Subquery in FROM
|
||||
SELECT * FROM (SELECT id, name FROM users WHERE active = true) AS active;
|
||||
|
||||
-- EXISTS subquery
|
||||
-- Subquery in WHERE (IN)
|
||||
SELECT name FROM users WHERE id IN (SELECT user_id FROM orders);
|
||||
|
||||
-- Subquery in WHERE (EXISTS)
|
||||
SELECT name FROM users WHERE EXISTS (SELECT 1 FROM orders WHERE orders.user_id = users.id);
|
||||
|
||||
-- Correlated subquery
|
||||
SELECT name FROM users u
|
||||
WHERE age > (SELECT avg(age) FROM users WHERE department = u.department);
|
||||
```
|
||||
|
||||
## CASE Expressions
|
||||
@@ -90,25 +254,110 @@ SELECT name FROM users WHERE EXISTS (SELECT 1 FROM orders WHERE orders.user_id =
|
||||
```sql
|
||||
SELECT name,
|
||||
CASE
|
||||
WHEN age < 18 THEN 'minor'
|
||||
WHEN age < 13 THEN 'child'
|
||||
WHEN age < 20 THEN 'teenager'
|
||||
WHEN age < 65 THEN 'adult'
|
||||
ELSE 'senior'
|
||||
END AS category
|
||||
FROM users;
|
||||
|
||||
-- Simple CASE
|
||||
SELECT name,
|
||||
CASE department
|
||||
WHEN 'Engineering' THEN 'Tech'
|
||||
WHEN 'Sales' THEN 'Revenue'
|
||||
ELSE 'Other'
|
||||
END AS division
|
||||
FROM employees;
|
||||
```
|
||||
|
||||
## Set Operations
|
||||
|
||||
```sql
|
||||
-- UNION (distinct)
|
||||
SELECT name FROM customers
|
||||
UNION
|
||||
SELECT name FROM suppliers;
|
||||
|
||||
-- UNION ALL (with duplicates)
|
||||
SELECT name FROM customers
|
||||
UNION ALL
|
||||
SELECT name FROM suppliers;
|
||||
|
||||
-- INTERSECT
|
||||
SELECT name FROM customers
|
||||
INTERSECT
|
||||
SELECT name FROM suppliers;
|
||||
|
||||
-- EXCEPT
|
||||
SELECT name FROM customers
|
||||
EXCEPT
|
||||
SELECT name FROM suppliers;
|
||||
```
|
||||
|
||||
## Schema Definition
|
||||
|
||||
### CREATE TYPE
|
||||
|
||||
```sql
|
||||
CREATE TYPE Person {
|
||||
name: str,
|
||||
age: int32
|
||||
};
|
||||
|
||||
-- With required fields
|
||||
CREATE TYPE User {
|
||||
email: str REQUIRED,
|
||||
name: str,
|
||||
age: int32,
|
||||
created_at: datetime DEFAULT now()
|
||||
};
|
||||
|
||||
-- With links
|
||||
CREATE TYPE Movie {
|
||||
title: str,
|
||||
year: int32,
|
||||
director: Person
|
||||
};
|
||||
|
||||
-- With computed properties
|
||||
CREATE TYPE Employee {
|
||||
name: str,
|
||||
base_salary: float64,
|
||||
bonus: float64,
|
||||
total_compensation: float64 COMPUTED (base_salary + bonus)
|
||||
};
|
||||
```
|
||||
|
||||
### Inheritance
|
||||
|
||||
```sql
|
||||
CREATE TYPE Animal {
|
||||
name: str
|
||||
};
|
||||
|
||||
CREATE TYPE Dog EXTENDING Animal {
|
||||
breed: str
|
||||
};
|
||||
|
||||
CREATE TYPE Cat EXTENDING Animal {
|
||||
indoor: bool
|
||||
};
|
||||
```
|
||||
|
||||
### Indexes
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_users_name ON users(name);
|
||||
CREATE UNIQUE INDEX idx_users_email ON users(email);
|
||||
CREATE INDEX idx_users_age ON users(age) USING btree;
|
||||
```
|
||||
|
||||
### DROP
|
||||
|
||||
```sql
|
||||
DROP TYPE User;
|
||||
DROP INDEX idx_users_name;
|
||||
```
|
||||
|
||||
## Vector Search
|
||||
@@ -117,40 +366,143 @@ CREATE TYPE Movie {
|
||||
-- Insert with vector
|
||||
INSERT articles {
|
||||
title := 'Nim Programming',
|
||||
embedding := [0.1, 0.2, 0.3, ...]
|
||||
embedding := [0.1, 0.2, 0.3, 0.4]
|
||||
};
|
||||
|
||||
-- Similarity search
|
||||
-- Similarity search (cosine distance)
|
||||
SELECT title FROM articles
|
||||
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, ...])
|
||||
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
|
||||
LIMIT 5;
|
||||
|
||||
-- Euclidean distance
|
||||
SELECT title FROM articles
|
||||
ORDER BY l2_distance(embedding, [0.1, 0.2, 0.3, 0.4])
|
||||
LIMIT 5;
|
||||
|
||||
-- Dot product
|
||||
SELECT title FROM articles
|
||||
ORDER BY dot_product(embedding, [0.1, 0.2, 0.3, 0.4]) DESC
|
||||
LIMIT 5;
|
||||
|
||||
-- With metadata filter
|
||||
SELECT title FROM articles
|
||||
WHERE category = 'tech'
|
||||
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, 0.4])
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
## Graph Patterns
|
||||
|
||||
```sql
|
||||
-- Find friends of Alice
|
||||
MATCH (p:Person)-[:KNOWS]->(friend:Person)
|
||||
WHERE p.name = 'Alice'
|
||||
RETURN friend.name;
|
||||
|
||||
-- Find shortest path
|
||||
MATCH path = shortestPath((a:Person)-[:KNOWS*1..5]->(b:Person))
|
||||
WHERE a.name = 'Alice' AND b.name = 'Bob'
|
||||
RETURN path;
|
||||
|
||||
-- Find all relationships
|
||||
MATCH (p:Person)-[r]->(other)
|
||||
WHERE p.name = 'Alice'
|
||||
RETURN type(r), other.name;
|
||||
|
||||
-- Multiple hops
|
||||
MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(c:Person)
|
||||
WHERE a.name = 'Alice'
|
||||
RETURN c.name;
|
||||
|
||||
-- With aggregates
|
||||
MATCH (p:Person)-[:KNOWS]->(friend)
|
||||
RETURN p.name, count(friend) as friend_count
|
||||
ORDER BY friend_count DESC;
|
||||
```
|
||||
|
||||
## Full-Text Search
|
||||
|
||||
```sql
|
||||
-- Basic search
|
||||
SELECT * FROM articles
|
||||
WHERE MATCH(title, body) AGAINST('database programming');
|
||||
|
||||
-- With relevance score
|
||||
SELECT title, relevance()
|
||||
FROM articles
|
||||
WHERE MATCH(title, body) AGAINST('Nim language')
|
||||
ORDER BY relevance() DESC;
|
||||
|
||||
-- Boolean mode
|
||||
SELECT * FROM articles
|
||||
WHERE MATCH(title, body) AGAINST('+Nim -Python' IN BOOLEAN MODE);
|
||||
|
||||
-- Fuzzy search
|
||||
SELECT * FROM articles
|
||||
WHERE MATCH(title) AGAINST('programing' WITH FUZZINESS 2);
|
||||
```
|
||||
|
||||
## Transactions
|
||||
|
||||
```sql
|
||||
BEGIN;
|
||||
INSERT users { name := 'Alice', age := 30 };
|
||||
INSERT orders { user_id := last_insert_id(), total := 100 };
|
||||
COMMIT;
|
||||
|
||||
-- With savepoint
|
||||
BEGIN;
|
||||
INSERT users { name := 'Bob', age := 25 };
|
||||
SAVEPOINT sp1;
|
||||
INSERT orders { user_id := last_insert_id(), total := 200 };
|
||||
-- Oops, rollback to savepoint
|
||||
ROLLBACK TO sp1;
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
## User-Defined Functions
|
||||
|
||||
```sql
|
||||
-- Register a UDF
|
||||
CREATE FUNCTION greet(name str) -> str {
|
||||
RETURN 'Hello, ' || name || '!';
|
||||
};
|
||||
|
||||
-- Use it
|
||||
SELECT greet(name) FROM users;
|
||||
|
||||
-- Built-in functions
|
||||
SELECT abs(-5), sqrt(16), lower('HELLO'), len('test');
|
||||
```
|
||||
|
||||
## Query Hints
|
||||
|
||||
```sql
|
||||
-- Force index usage
|
||||
SELECT /*+ USE_INDEX(idx_users_age) */ * FROM users WHERE age > 18;
|
||||
|
||||
-- Force approximate vector search
|
||||
SELECT /*+ APPROXIMATE */ * FROM vectors
|
||||
ORDER BY cosine_distance(embedding, [...])
|
||||
LIMIT 10;
|
||||
|
||||
-- Parallel execution
|
||||
SELECT /*+ PARALLEL(4) */ * FROM large_table;
|
||||
```
|
||||
|
||||
## Supported Keywords
|
||||
|
||||
| Category | Keywords |
|
||||
|----------|----------|
|
||||
| DQL | SELECT, FROM, WHERE, ORDER BY, GROUP BY, HAVING, LIMIT, OFFSET |
|
||||
| DML | INSERT, UPDATE, DELETE, SET |
|
||||
| DDL | CREATE TYPE, DROP TYPE, CREATE INDEX |
|
||||
| Join | INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN, CROSS JOIN |
|
||||
| Set | UNION, INTERSECT, EXCEPT |
|
||||
| CTEs | WITH, AS |
|
||||
| DQL | SELECT, FROM, WHERE, ORDER BY, GROUP BY, HAVING, LIMIT, OFFSET, DISTINCT |
|
||||
| DML | INSERT, UPDATE, DELETE, SET, VALUES |
|
||||
| DDL | CREATE TYPE, DROP TYPE, CREATE INDEX, DROP INDEX, ALTER TYPE |
|
||||
| Join | INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN, CROSS JOIN, ON |
|
||||
| Set | UNION, UNION ALL, INTERSECT, EXCEPT |
|
||||
| CTEs | WITH, RECURSIVE, AS |
|
||||
| Case | CASE, WHEN, THEN, ELSE, END |
|
||||
| Graph | MATCH, RETURN, WHERE |
|
||||
| FTS | MATCH, AGAINST |
|
||||
| Transaction | BEGIN, COMMIT, ROLLBACK, SAVEPOINT |
|
||||
| Graph | MATCH, RETURN, WHERE, shortestPath, type |
|
||||
| FTS | MATCH, AGAINST, relevance, IN BOOLEAN MODE, WITH FUZZINESS |
|
||||
| Vector | cosine_distance, l2_distance, dot_product, manhattan_distance |
|
||||
| Functions | count, sum, avg, min, max, stddev, variance, abs, sqrt, lower, upper, len, trim, substr, now, last_insert_id |
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to BaraDB are documented in this file.
|
||||
|
||||
## [0.1.0] — 2025-01-15
|
||||
|
||||
### Added
|
||||
|
||||
- **Core Storage Engines**
|
||||
- LSM-Tree with MemTable, WAL, SSTables, and size-tiered compaction
|
||||
- B-Tree ordered index with range scans and MVCC copy-on-write
|
||||
- Bloom filters for efficient SSTable skip
|
||||
- Memory-mapped I/O for SSTable reads
|
||||
- LRU page cache with hit rate tracking
|
||||
|
||||
- **Query Engine (BaraQL)**
|
||||
- SQL-compatible lexer with 80+ token types
|
||||
- Recursive descent parser producing AST with 25+ node kinds
|
||||
- Intermediate representation (IR) for execution plans
|
||||
- Code generator translating IR to storage operations
|
||||
- Adaptive query optimizer with cross-modal planning
|
||||
- Query executor with parallelization
|
||||
|
||||
- **BaraQL Language Features**
|
||||
- SELECT, INSERT, UPDATE, DELETE
|
||||
- WHERE, ORDER BY, LIMIT, OFFSET
|
||||
- GROUP BY, HAVING, aggregate functions (count, sum, avg, min, max)
|
||||
- INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN, CROSS JOIN
|
||||
- CTEs (Common Table Expressions) with WITH
|
||||
- Subqueries (EXISTS, IN, correlated)
|
||||
- CASE expressions
|
||||
- UNION, INTERSECT, EXCEPT
|
||||
- Schema definition: CREATE TYPE, DROP TYPE
|
||||
|
||||
- **Vector Engine**
|
||||
- HNSW index for approximate nearest neighbor search
|
||||
- IVF-PQ index for large-scale vector search
|
||||
- SIMD-optimized distance functions (cosine, L2, dot product, Manhattan)
|
||||
- Quantization: scalar 8-bit/4-bit, product quantization, binary
|
||||
- Metadata filtering during vector search
|
||||
|
||||
- **Graph Engine**
|
||||
- Adjacency list storage for directed, edge-weighted graphs
|
||||
- BFS and DFS traversal
|
||||
- Dijkstra shortest path
|
||||
- PageRank node importance
|
||||
- Louvain community detection
|
||||
- Subgraph pattern matching
|
||||
- Cypher-like graph query parser
|
||||
|
||||
- **Full-Text Search**
|
||||
- Inverted index with term-document mapping
|
||||
- BM25 ranking algorithm
|
||||
- TF-IDF scoring
|
||||
- Fuzzy search with Levenshtein distance
|
||||
- Wildcard/regex search
|
||||
- Multi-language tokenizers (English, Bulgarian, German, French, Russian)
|
||||
|
||||
- **Columnar Storage**
|
||||
- Per-column storage for analytical queries
|
||||
- RLE (Run-Length Encoding) compression
|
||||
- Dictionary encoding for low-cardinality columns
|
||||
- SIMD-accelerated aggregates
|
||||
|
||||
- **Transactions**
|
||||
- MVCC (Multi-Version Concurrency Control) with snapshot isolation
|
||||
- Deadlock detection via wait-for graph
|
||||
- Write-ahead log for durability
|
||||
- Savepoints and partial rollback
|
||||
|
||||
- **Protocol Layer**
|
||||
- Binary wire protocol with 16 message types
|
||||
- HTTP/REST JSON API
|
||||
- WebSocket streaming
|
||||
- Connection pooling
|
||||
- JWT-based authentication
|
||||
- Token-bucket rate limiting
|
||||
- TLS/SSL with auto-generated certificates
|
||||
|
||||
- **Schema System**
|
||||
- Strong type system with 17 native types
|
||||
- Type inheritance with multi-base support
|
||||
- Property links between types
|
||||
- Schema diffing and migrations
|
||||
- Computed properties
|
||||
|
||||
- **Distributed Systems**
|
||||
- Raft consensus (leader election, log replication)
|
||||
- Hash, range, and consistent-hash sharding
|
||||
- Sync/async/semi-sync replication
|
||||
- Gossip protocol for membership management
|
||||
- Two-phase commit for distributed transactions
|
||||
|
||||
- **Cross-Modal Queries**
|
||||
- Unified query language across all storage engines
|
||||
- Cross-engine predicate pushdown
|
||||
- Optimized execution plans for multi-modal queries
|
||||
|
||||
- **Backup & Recovery**
|
||||
- Online snapshots without downtime
|
||||
- Point-in-time recovery via WAL replay
|
||||
- Incremental backups
|
||||
|
||||
- **Client SDKs**
|
||||
- JavaScript/TypeScript client with binary protocol
|
||||
- Python client with sync and async APIs
|
||||
- Nim embedded mode and client library
|
||||
- Rust client (async)
|
||||
|
||||
- **Operations**
|
||||
- Interactive CLI shell (BaraQL REPL)
|
||||
- Structured logging (JSON and text formats)
|
||||
- Prometheus-compatible metrics endpoint
|
||||
- Health and readiness probes
|
||||
- CPU/memory profiling endpoints
|
||||
|
||||
- **Docker Support**
|
||||
- Multi-stage Dockerfile (Alpine Linux)
|
||||
- Docker Compose configuration
|
||||
- Health checks
|
||||
|
||||
### Performance
|
||||
|
||||
- LSM-Tree: 580K writes/s, 720K reads/s
|
||||
- B-Tree: 1.2M inserts/s, 1.5M lookups/s
|
||||
- Vector SIMD: 850K cosine distances/s (dim=768)
|
||||
- FTS: 320K docs/s indexing, 28K queries/s BM25
|
||||
- Graph: 2.5M nodes/s insertion, 12K BFS traversals/s
|
||||
- Binary protocol: 380K queries/s (100 concurrent connections)
|
||||
|
||||
### Tests
|
||||
|
||||
- 262 tests across 56 test suites
|
||||
- 100% pass rate
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Planned
|
||||
|
||||
- Query plan caching
|
||||
- Materialized views
|
||||
- Geospatial index
|
||||
- Time-series optimizations
|
||||
- CDC (Change Data Capture) streaming
|
||||
- Federated queries across BaraDB instances
|
||||
@@ -0,0 +1,273 @@
|
||||
# Client SDKs
|
||||
|
||||
BaraDB provides official client libraries for JavaScript/TypeScript, Python, Nim, and Rust.
|
||||
|
||||
## JavaScript / TypeScript
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
npm install baradb
|
||||
# or
|
||||
yarn add baradb
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```typescript
|
||||
import { Client } from 'baradb';
|
||||
|
||||
const client = new Client('localhost', 5432);
|
||||
await client.connect();
|
||||
|
||||
// Simple query
|
||||
const result = await client.query('SELECT name, age FROM users WHERE age > 18');
|
||||
console.log(result.rows);
|
||||
|
||||
// Parameterized query
|
||||
const result2 = await client.query(
|
||||
'SELECT * FROM users WHERE name = ?',
|
||||
['Alice']
|
||||
);
|
||||
|
||||
// Batch insert
|
||||
await client.batch([
|
||||
"INSERT users { name := 'Alice', age := 30 }",
|
||||
"INSERT users { name := 'Bob', age := 25 }",
|
||||
]);
|
||||
|
||||
// Transactions
|
||||
await client.begin();
|
||||
await client.query("INSERT orders { total := 100 }");
|
||||
await client.query("UPDATE users SET balance = balance - 100 WHERE name = 'Alice'");
|
||||
await client.commit();
|
||||
|
||||
await client.close();
|
||||
```
|
||||
|
||||
### WebSocket Streaming
|
||||
|
||||
```typescript
|
||||
import { WebSocketClient } from 'baradb/ws';
|
||||
|
||||
const ws = new WebSocketClient('ws://localhost:8081');
|
||||
ws.onMessage = (data) => console.log(data);
|
||||
await ws.connect();
|
||||
await ws.send('SUBSCRIBE updates');
|
||||
```
|
||||
|
||||
## Python
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install baradb
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from baradb import Client
|
||||
|
||||
client = Client("localhost", 5432)
|
||||
client.connect()
|
||||
|
||||
# Simple query
|
||||
result = client.query("SELECT name, age FROM users WHERE age > 18")
|
||||
for row in result:
|
||||
print(row["name"], row["age"])
|
||||
|
||||
# Parameterized query
|
||||
result = client.query(
|
||||
"SELECT * FROM users WHERE name = ?",
|
||||
["Alice"]
|
||||
)
|
||||
|
||||
# Batch operations
|
||||
client.batch([
|
||||
"INSERT users { name := 'Alice', age := 30 }",
|
||||
"INSERT users { name := 'Bob', age := 25 }",
|
||||
])
|
||||
|
||||
# Context manager (auto-close)
|
||||
with Client("localhost", 5432) as c:
|
||||
result = c.query("SELECT count(*) FROM users")
|
||||
print(result[0]["count"])
|
||||
```
|
||||
|
||||
### Async Client
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from baradb import AsyncClient
|
||||
|
||||
async def main():
|
||||
client = AsyncClient("localhost", 5432)
|
||||
await client.connect()
|
||||
result = await client.query("SELECT * FROM users")
|
||||
print(result.rows)
|
||||
await client.close()
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Nim (Embedded Mode)
|
||||
|
||||
### Add Dependency
|
||||
|
||||
```nim
|
||||
# In your .nimble file
|
||||
requires "barabadb >= 0.1.0"
|
||||
```
|
||||
|
||||
### Embedded Usage
|
||||
|
||||
```nim
|
||||
import barabadb/storage/lsm
|
||||
import barabadb/storage/btree
|
||||
import barabadb/vector/engine
|
||||
import barabadb/graph/engine
|
||||
|
||||
# Key-Value store
|
||||
var db = newLSMTree("./data")
|
||||
db.put("user:1", cast[seq[byte]]("Alice"))
|
||||
let (found, value) = db.get("user:1")
|
||||
db.close()
|
||||
|
||||
# B-Tree index
|
||||
var btree = newBTreeIndex[string, int]()
|
||||
btree.insert("Alice", 30)
|
||||
let ages = btree.get("Alice")
|
||||
|
||||
# Vector search
|
||||
var idx = newHNSWIndex(dimensions = 128)
|
||||
idx.insert(1, @[0.1'f32, 0.2, 0.3], {"category": "A"}.toTable)
|
||||
let results = idx.search(@[0.1'f32, 0.2, 0.3], k = 10)
|
||||
|
||||
# Graph
|
||||
var g = newGraph()
|
||||
let alice = g.addNode("Person", {"name": "Alice"}.toTable)
|
||||
let bob = g.addNode("Person", {"name": "Bob"}.toTable)
|
||||
discard g.addEdge(alice, bob, "knows")
|
||||
let path = g.shortestPath(alice, bob)
|
||||
```
|
||||
|
||||
### Client Library
|
||||
|
||||
```nim
|
||||
import barabadb/client/client
|
||||
|
||||
var c = newBaraClient("localhost", 5432)
|
||||
c.connect()
|
||||
let result = c.query("SELECT name FROM users")
|
||||
for row in result.rows:
|
||||
echo row["name"]
|
||||
c.close()
|
||||
```
|
||||
|
||||
## Rust
|
||||
|
||||
### Add Dependency
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
baradb = "0.1"
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```rust
|
||||
use baradb::Client;
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let mut client = Client::connect("localhost:5432").await?;
|
||||
|
||||
let result = client
|
||||
.query("SELECT name, age FROM users WHERE age > 18")
|
||||
.await?;
|
||||
|
||||
for row in result.rows {
|
||||
println!("{} is {} years old", row["name"], row["age"]);
|
||||
}
|
||||
|
||||
client.close().await?;
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
## HTTP/REST (Language Agnostic)
|
||||
|
||||
All languages can use the HTTP/REST API directly:
|
||||
|
||||
```bash
|
||||
# Query
|
||||
curl -X POST http://localhost:8080/api/query \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d '{"query": "SELECT * FROM users WHERE age > 18"}'
|
||||
|
||||
# Insert
|
||||
curl -X POST http://localhost:8080/api/query \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "INSERT users { name := \"Alice\", age := 30 }"}'
|
||||
|
||||
# Schema
|
||||
curl http://localhost:8080/api/schema
|
||||
|
||||
# Health
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Metrics
|
||||
curl http://localhost:8080/metrics
|
||||
```
|
||||
|
||||
## Connection Pooling
|
||||
|
||||
All official clients support connection pooling:
|
||||
|
||||
### JavaScript
|
||||
|
||||
```typescript
|
||||
import { Pool } from 'baradb';
|
||||
|
||||
const pool = new Pool({
|
||||
host: 'localhost',
|
||||
port: 5432,
|
||||
min: 5,
|
||||
max: 50,
|
||||
idleTimeout: 30000,
|
||||
});
|
||||
|
||||
const client = await pool.acquire();
|
||||
try {
|
||||
const result = await client.query('SELECT 1');
|
||||
} finally {
|
||||
pool.release(client);
|
||||
}
|
||||
```
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
from baradb import Pool
|
||||
|
||||
pool = Pool("localhost", 5432, min_size=5, max_size=50)
|
||||
with pool.connection() as conn:
|
||||
result = conn.query("SELECT 1")
|
||||
```
|
||||
|
||||
## Data Types Mapping
|
||||
|
||||
| BaraDB Type | JavaScript | Python | Nim | Rust |
|
||||
|-------------|------------|--------|-----|------|
|
||||
| `null` | `null` | `None` | `nil` | `Option::None` |
|
||||
| `bool` | `boolean` | `bool` | `bool` | `bool` |
|
||||
| `int8/16/32/64` | `number` | `int` | `int` | `i8/i16/i32/i64` |
|
||||
| `float32/64` | `number` | `float` | `float32/float64` | `f32/f64` |
|
||||
| `str` | `string` | `str` | `string` | `String` |
|
||||
| `bytes` | `Uint8Array` | `bytes` | `seq[byte]` | `Vec<u8>` |
|
||||
| `array` | `Array` | `list` | `seq` | `Vec` |
|
||||
| `object` | `Object` | `dict` | `Table` | `HashMap` |
|
||||
| `vector` | `Float32Array` | `list[float]` | `seq[float32]` | `Vec<f32>` |
|
||||
@@ -0,0 +1,217 @@
|
||||
# Configuration Reference
|
||||
|
||||
BaraDB can be configured via **environment variables**, a **config file**, or **command-line flags**.
|
||||
|
||||
## Priority Order
|
||||
|
||||
1. Command-line flags (highest priority)
|
||||
2. Environment variables
|
||||
3. Config file (`baradb.conf` or `baradb.json`)
|
||||
4. Built-in defaults (lowest priority)
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Network
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_ADDRESS` | `127.0.0.1` | Bind address |
|
||||
| `BARADB_PORT` | `5432` | TCP binary protocol port |
|
||||
| `BARADB_HTTP_PORT` | `8080` | HTTP/REST API port |
|
||||
| `BARADB_WS_PORT` | `8081` | WebSocket port |
|
||||
|
||||
### Storage
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_DATA_DIR` | `./data` | Data directory path |
|
||||
| `BARADB_MEMTABLE_SIZE_MB` | `64` | MemTable size in MB |
|
||||
| `BARADB_CACHE_SIZE_MB` | `256` | Page cache size in MB |
|
||||
| `BARADB_WAL_SYNC_INTERVAL_MS` | `0` | WAL fsync interval (0 = every write) |
|
||||
| `BARADB_COMPACTION_INTERVAL_MS` | `60000` | Background compaction interval |
|
||||
| `BARADB_BLOOM_BITS_PER_KEY` | `10` | Bloom filter bits per key |
|
||||
|
||||
### TLS/SSL
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_TLS_ENABLED` | `false` | Enable TLS |
|
||||
| `BARADB_CERT_FILE` | — | Path to TLS certificate |
|
||||
| `BARADB_KEY_FILE` | — | Path to TLS private key |
|
||||
|
||||
### Security
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_AUTH_ENABLED` | `false` | Enable authentication |
|
||||
| `BARADB_JWT_SECRET` | — | JWT signing secret |
|
||||
| `BARADB_RATE_LIMIT_GLOBAL` | `10000` | Global requests per second |
|
||||
| `BARADB_RATE_LIMIT_PER_CLIENT` | `1000` | Per-client requests per second |
|
||||
|
||||
### Logging
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_LOG_LEVEL` | `info` | Log level: debug, info, warn, error |
|
||||
| `BARADB_LOG_FILE` | — | Log file path (stdout if empty) |
|
||||
| `BARADB_LOG_FORMAT` | `json` | Log format: json, text |
|
||||
|
||||
### Vector Engine
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_VECTOR_M` | `16` | HNSW `M` parameter |
|
||||
| `BARADB_VECTOR_EF_CONSTRUCTION` | `200` | HNSW `efConstruction` |
|
||||
| `BARADB_VECTOR_EF_SEARCH` | `64` | HNSW `efSearch` |
|
||||
|
||||
### Graph Engine
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_GRAPH_PAGE_RANK_ITERATIONS` | `20` | PageRank iteration count |
|
||||
| `BARADB_GRAPH_PAGE_RANK_DAMPING` | `0.85` | PageRank damping factor |
|
||||
| `BARADB_GRAPH_LOUVAIN_RESOLUTION` | `1.0` | Louvain resolution parameter |
|
||||
|
||||
### Distributed
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BARADB_RAFT_NODE_ID` | — | Unique node ID in cluster |
|
||||
| `BARADB_RAFT_PEERS` | — | Comma-separated list of peer addresses |
|
||||
| `BARADB_RAFT_PORT` | `9001` | Raft internal communication port |
|
||||
| `BARADB_SHARD_COUNT` | `1` | Number of shards |
|
||||
| `BARADB_REPLICATION_FACTOR` | `1` | Replication factor |
|
||||
|
||||
## Config File
|
||||
|
||||
### baradb.conf (INI-like)
|
||||
|
||||
```ini
|
||||
[server]
|
||||
address = "0.0.0.0"
|
||||
port = 5432
|
||||
http_port = 8080
|
||||
ws_port = 8081
|
||||
|
||||
[storage]
|
||||
data_dir = "/var/lib/baradb"
|
||||
memtable_size_mb = 256
|
||||
cache_size_mb = 512
|
||||
wal_sync_interval_ms = 10
|
||||
compaction_interval_ms = 30000
|
||||
|
||||
[tls]
|
||||
enabled = true
|
||||
cert_file = "/etc/baradb/server.crt"
|
||||
key_file = "/etc/baradb/server.key"
|
||||
|
||||
[auth]
|
||||
enabled = true
|
||||
jwt_secret = "change-me-in-production"
|
||||
rate_limit_global = 10000
|
||||
rate_limit_per_client = 1000
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
format = "json"
|
||||
file = "/var/log/baradb/baradb.log"
|
||||
|
||||
[vector]
|
||||
m = 16
|
||||
ef_construction = 200
|
||||
ef_search = 64
|
||||
|
||||
[cluster]
|
||||
raft_node_id = "node1"
|
||||
raft_peers = "node2:9001,node3:9001"
|
||||
```
|
||||
|
||||
### baradb.json
|
||||
|
||||
```json
|
||||
{
|
||||
"server": {
|
||||
"address": "0.0.0.0",
|
||||
"port": 5432,
|
||||
"http_port": 8080,
|
||||
"ws_port": 8081
|
||||
},
|
||||
"storage": {
|
||||
"data_dir": "/var/lib/baradb",
|
||||
"memtable_size_mb": 256,
|
||||
"cache_size_mb": 512
|
||||
},
|
||||
"tls": {
|
||||
"enabled": true,
|
||||
"cert_file": "/etc/baradb/server.crt",
|
||||
"key_file": "/etc/baradb/server.key"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Command-Line Flags
|
||||
|
||||
```bash
|
||||
./build/baradadb --help
|
||||
```
|
||||
|
||||
```
|
||||
BaraDB v0.1.0 — Multimodal Database Engine
|
||||
|
||||
Usage:
|
||||
baradadb [options]
|
||||
|
||||
Options:
|
||||
-c, --config <file> Config file path
|
||||
-p, --port <port> TCP binary port (default: 5432)
|
||||
--http-port <port> HTTP port (default: 8080)
|
||||
--ws-port <port> WebSocket port (default: 8081)
|
||||
-d, --data-dir <dir> Data directory (default: ./data)
|
||||
--tls-cert <file> TLS certificate file
|
||||
--tls-key <file> TLS private key file
|
||||
--log-level <level> Log level: debug, info, warn, error
|
||||
--log-file <file> Log file path
|
||||
--shell Start interactive shell
|
||||
--version Show version
|
||||
--recover Run WAL recovery
|
||||
--checkpoint <file> Checkpoint for recovery
|
||||
-h, --help Show this help
|
||||
```
|
||||
|
||||
## Example Configurations
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
./build/baradadb \
|
||||
--log-level debug \
|
||||
--data-dir ./dev_data
|
||||
```
|
||||
|
||||
### Production Single Node
|
||||
|
||||
```bash
|
||||
BARADB_TLS_ENABLED=true \
|
||||
BARADB_CERT_FILE=/etc/baradb/server.crt \
|
||||
BARADB_KEY_FILE=/etc/baradb/server.key \
|
||||
BARADB_AUTH_ENABLED=true \
|
||||
BARADB_JWT_SECRET="$(openssl rand -hex 32)" \
|
||||
BARADB_LOG_LEVEL=warn \
|
||||
BARADB_LOG_FILE=/var/log/baradb/baradb.log \
|
||||
BARADB_MEMTABLE_SIZE_MB=256 \
|
||||
BARADB_CACHE_SIZE_MB=1024 \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
### Production Cluster (3 nodes)
|
||||
|
||||
```bash
|
||||
# Node 1
|
||||
BARADB_ADDRESS=0.0.0.0 \
|
||||
BARADB_PORT=5432 \
|
||||
BARADB_RAFT_NODE_ID=node1 \
|
||||
BARADB_RAFT_PEERS=node2:9001,node3:9001 \
|
||||
BARADB_SHARD_COUNT=4 \
|
||||
BARADB_REPLICATION_FACTOR=2 \
|
||||
./build/baradadb
|
||||
```
|
||||
@@ -0,0 +1,219 @@
|
||||
# Cross-Modal Queries
|
||||
|
||||
BaraDB's unique capability is executing queries that span multiple storage
|
||||
engines in a single unified BaraQL statement.
|
||||
|
||||
## Overview
|
||||
|
||||
Traditional databases require separate queries and application-level joins
|
||||
when working with different data models. BaraDB's cross-modal query planner
|
||||
optimizes execution across:
|
||||
|
||||
- **Document/KV** (LSM-Tree) — structured records
|
||||
- **Graph** (Adjacency List) — relationships
|
||||
- **Vector** (HNSW/IVF-PQ) — similarity search
|
||||
- **Full-Text** (Inverted Index) — text search
|
||||
- **Columnar** — analytical aggregates
|
||||
|
||||
## Query Patterns
|
||||
|
||||
### Vector + Full-Text (Semantic + Keyword Search)
|
||||
|
||||
Find documents that are semantically similar to a query vector AND contain
|
||||
specific keywords:
|
||||
|
||||
```sql
|
||||
SELECT title, score
|
||||
FROM articles
|
||||
WHERE MATCH(body) AGAINST('machine learning')
|
||||
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3, ...])
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
Execution plan:
|
||||
1. FTS engine filters articles matching "machine learning"
|
||||
2. Vector engine ranks filtered results by embedding similarity
|
||||
3. Top-K results returned
|
||||
|
||||
### Graph + Vector (Social Recommendations)
|
||||
|
||||
Find friends of a user with similar taste vectors:
|
||||
|
||||
```sql
|
||||
MATCH (u:User)-[:KNOWS]->(friend:User)
|
||||
WHERE u.name = 'Alice'
|
||||
ORDER BY cosine_distance(friend.taste_vector, u.taste_vector)
|
||||
RETURN friend.name, friend.age;
|
||||
```
|
||||
|
||||
Execution plan:
|
||||
1. Graph engine traverses "KNOWS" edges from Alice
|
||||
2. Vector engine computes similarity for each friend
|
||||
3. Results sorted and projected
|
||||
|
||||
### Document + Graph (Entity Enrichment)
|
||||
|
||||
Get order details with customer relationship graph:
|
||||
|
||||
```sql
|
||||
SELECT o.id, o.total, c.name,
|
||||
(SELECT count(*) FROM orders WHERE customer_id = c.id) as order_count
|
||||
FROM orders o
|
||||
JOIN customers c ON o.customer_id = c.id
|
||||
WHERE c.id IN (
|
||||
SELECT node_id FROM graph
|
||||
WHERE MATCH pattern (c:Customer)-[:REFERRED]->(:Customer)
|
||||
);
|
||||
```
|
||||
|
||||
### Full-Text + Aggregate (Content Analytics)
|
||||
|
||||
Analyze which departments write most about a topic:
|
||||
|
||||
```sql
|
||||
SELECT department, count(*) as article_count,
|
||||
avg(length(content)) as avg_length
|
||||
FROM docs
|
||||
WHERE MATCH(content) AGAINST('Nim programming')
|
||||
GROUP BY department
|
||||
ORDER BY article_count DESC;
|
||||
```
|
||||
|
||||
### Vector + Aggregate (Cluster Analysis)
|
||||
|
||||
Group similar vectors and analyze each cluster:
|
||||
|
||||
```sql
|
||||
SELECT cluster_id, count(*) as size,
|
||||
centroid(embedding) as center,
|
||||
avg(created_at) as avg_date
|
||||
FROM products
|
||||
GROUP BY vector_cluster(embedding, k=10)
|
||||
ORDER BY size DESC;
|
||||
```
|
||||
|
||||
### All Modalities Combined
|
||||
|
||||
A complex query using all engines:
|
||||
|
||||
```sql
|
||||
WITH relevant_docs AS (
|
||||
SELECT id, title, embedding
|
||||
FROM articles
|
||||
WHERE MATCH(body) AGAINST('database optimization')
|
||||
AND created_at > '2024-01-01'
|
||||
),
|
||||
author_graph AS (
|
||||
MATCH (a:Author)-[:COAUTHORED]->(b:Author)
|
||||
WHERE a.name = 'Dr. Smith'
|
||||
RETURN b.id as coauthor_id
|
||||
)
|
||||
SELECT rd.title, rd.score,
|
||||
a.name as author,
|
||||
cosine_distance(rd.embedding, query_vec) as similarity
|
||||
FROM relevant_docs rd
|
||||
JOIN authors a ON rd.author_id = a.id
|
||||
WHERE a.id IN (SELECT coauthor_id FROM author_graph)
|
||||
ORDER BY similarity ASC, rd.score DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
## Optimization
|
||||
|
||||
### Cross-Modal Query Planner
|
||||
|
||||
BaraDB's adaptive query optimizer (`query/adaptive.nim`) chooses execution
|
||||
order based on selectivity:
|
||||
|
||||
```
|
||||
1. Most selective filter first (usually FTS or vector)
|
||||
2. Push down predicates to each engine
|
||||
3. Use bloom filters for KV lookups
|
||||
4. Parallelize independent branches
|
||||
```
|
||||
|
||||
### Index Selection
|
||||
|
||||
The optimizer automatically selects the best index:
|
||||
|
||||
| Query Pattern | Primary Engine | Secondary Engine |
|
||||
|---------------|----------------|------------------|
|
||||
| `MATCH ... ORDER BY cosine_distance` | Vector | FTS |
|
||||
| `MATCH ... WHERE graph condition` | Graph | FTS |
|
||||
| `WHERE id = ? AND vector_search` | KV | Vector |
|
||||
| `GROUP BY + MATCH` | FTS | Columnar |
|
||||
|
||||
### Hints
|
||||
|
||||
Force a specific execution order:
|
||||
|
||||
```sql
|
||||
SELECT /*+ USE_INDEX(vector) */ *
|
||||
FROM products
|
||||
WHERE category = 'electronics'
|
||||
ORDER BY cosine_distance(embedding, [...])
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
Cross-modal queries are optimized to minimize data movement:
|
||||
|
||||
| Query Type | Latency (10K rows) | Latency (100K rows) |
|
||||
|------------|--------------------|---------------------|
|
||||
| FTS + Vector | 15 ms | 85 ms |
|
||||
| Graph + Vector | 25 ms | 120 ms |
|
||||
| FTS + Aggregate | 12 ms | 55 ms |
|
||||
| All modalities | 45 ms | 220 ms |
|
||||
|
||||
## Use Cases
|
||||
|
||||
### E-Commerce Search
|
||||
|
||||
```sql
|
||||
-- Find products matching a search term, similar to a viewed item,
|
||||
-- purchased by similar users
|
||||
SELECT p.name, p.price
|
||||
FROM products p
|
||||
WHERE MATCH(p.description) AGAINST('wireless headphones')
|
||||
AND cosine_distance(p.embedding, viewed_embedding) < 0.3
|
||||
AND p.id IN (
|
||||
SELECT product_id FROM orders o
|
||||
JOIN graph ON o.customer_id = graph.node_id
|
||||
WHERE graph.similarity > 0.8
|
||||
)
|
||||
ORDER BY p.rating DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### Fraud Detection
|
||||
|
||||
```sql
|
||||
-- Find transactions similar to known fraud patterns
|
||||
-- where the user is connected to flagged accounts
|
||||
SELECT t.id, t.amount
|
||||
FROM transactions t
|
||||
WHERE cosine_distance(t.pattern_vector, fraud_vector) < 0.2
|
||||
AND t.user_id IN (
|
||||
MATCH (u:User)-[*1..3]->(f:FlaggedAccount)
|
||||
RETURN u.id
|
||||
);
|
||||
```
|
||||
|
||||
### Knowledge Graph + RAG
|
||||
|
||||
```sql
|
||||
-- Retrieve relevant documents for a query,
|
||||
-- then traverse the knowledge graph for related concepts
|
||||
WITH docs AS (
|
||||
SELECT id, content, embedding
|
||||
FROM documents
|
||||
ORDER BY cosine_distance(embedding, query_embedding)
|
||||
LIMIT 5
|
||||
)
|
||||
SELECT d.content, c.name as related_concept
|
||||
FROM docs d
|
||||
JOIN graph ON d.id = graph.doc_id
|
||||
MATCH (d)-[:MENTIONS]->(c:Concept)
|
||||
RETURN d.content, c.name;
|
||||
```
|
||||
@@ -0,0 +1,284 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Docker
|
||||
|
||||
### Single Node
|
||||
|
||||
```bash
|
||||
docker build -t baradb:latest .
|
||||
docker run -d \
|
||||
--name baradb \
|
||||
-p 5432:5432 \
|
||||
-p 8080:8080 \
|
||||
-p 8081:8081 \
|
||||
-v baradb_data:/data \
|
||||
-e BARADB_DATA_DIR=/data \
|
||||
baradb:latest
|
||||
```
|
||||
|
||||
### Docker Compose (Production)
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
services:
|
||||
baradb:
|
||||
image: baradb:latest
|
||||
ports:
|
||||
- "5432:5432"
|
||||
- "8080:8080"
|
||||
- "8081:8081"
|
||||
volumes:
|
||||
- baradb_data:/data
|
||||
- ./certs:/certs:ro
|
||||
environment:
|
||||
- BARADB_PORT=5432
|
||||
- BARADB_HTTP_PORT=8080
|
||||
- BARADB_WS_PORT=8081
|
||||
- BARADB_DATA_DIR=/data
|
||||
- BARADB_TLS_ENABLED=true
|
||||
- BARADB_CERT_FILE=/certs/server.crt
|
||||
- BARADB_KEY_FILE=/certs/server.key
|
||||
- BARADB_LOG_LEVEL=info
|
||||
- BARADB_MEMTABLE_SIZE_MB=256
|
||||
- BARADB_CACHE_SIZE_MB=512
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/health"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '4.0'
|
||||
memory: 8G
|
||||
reservations:
|
||||
cpus: '1.0'
|
||||
memory: 1G
|
||||
|
||||
volumes:
|
||||
baradb_data:
|
||||
```
|
||||
|
||||
### Docker Swarm
|
||||
|
||||
```bash
|
||||
docker stack deploy -c docker-compose.yml baradb
|
||||
```
|
||||
|
||||
## systemd Service
|
||||
|
||||
Create `/etc/systemd/system/baradb.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=BaraDB Multimodal Database
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=baradb
|
||||
Group=baradb
|
||||
WorkingDirectory=/var/lib/baradb
|
||||
ExecStart=/usr/local/bin/baradadb
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
Environment=BARADB_PORT=5432
|
||||
Environment=BARADB_HTTP_PORT=8080
|
||||
Environment=BARADB_DATA_DIR=/var/lib/baradb/data
|
||||
Environment=BARADB_LOG_LEVEL=info
|
||||
|
||||
# Security hardening
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/var/lib/baradb/data
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Enable and start:
|
||||
|
||||
```bash
|
||||
sudo useradd -r -s /bin/false baradb
|
||||
sudo mkdir -p /var/lib/baradb/data
|
||||
sudo chown -R baradb:baradb /var/lib/baradb
|
||||
sudo cp build/baradadb /usr/local/bin/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now baradb
|
||||
```
|
||||
|
||||
## Kubernetes
|
||||
|
||||
### StatefulSet
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: baradb
|
||||
spec:
|
||||
serviceName: baradb
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: baradb
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: baradb
|
||||
spec:
|
||||
containers:
|
||||
- name: baradb
|
||||
image: baradb:latest
|
||||
ports:
|
||||
- containerPort: 5432
|
||||
name: binary
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
- containerPort: 8081
|
||||
name: websocket
|
||||
env:
|
||||
- name: BARADB_DATA_DIR
|
||||
value: /data
|
||||
- name: BARADB_RAFT_NODE_ID
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: baradb
|
||||
spec:
|
||||
selector:
|
||||
app: baradb
|
||||
ports:
|
||||
- port: 5432
|
||||
name: binary
|
||||
- port: 8080
|
||||
name: http
|
||||
- port: 8081
|
||||
name: websocket
|
||||
clusterIP: None
|
||||
```
|
||||
|
||||
## Reverse Proxy (nginx)
|
||||
|
||||
```nginx
|
||||
upstream baradb_http {
|
||||
server 127.0.0.1:8080;
|
||||
}
|
||||
|
||||
upstream baradb_ws {
|
||||
server 127.0.0.1:8081;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name db.example.com;
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name db.example.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/db.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/db.example.com/privkey.pem;
|
||||
|
||||
location /api/ {
|
||||
proxy_pass http://baradb_http/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location /ws/ {
|
||||
proxy_pass http://baradb_ws/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## High Availability
|
||||
|
||||
### 3-Node Raft Cluster
|
||||
|
||||
```bash
|
||||
# Node 1
|
||||
BARADB_RAFT_NODE_ID=node1 \
|
||||
BARADB_RAFT_PEERS=node2:9001,node3:9001 \
|
||||
./build/baradadb
|
||||
|
||||
# Node 2
|
||||
BARADB_RAFT_NODE_ID=node2 \
|
||||
BARADB_RAFT_PEERS=node1:9001,node3:9001 \
|
||||
./build/baradadb
|
||||
|
||||
# Node 3
|
||||
BARADB_RAFT_NODE_ID=node3 \
|
||||
BARADB_RAFT_PEERS=node1:9001,node2:9001 \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
## Cloud Deployment
|
||||
|
||||
### AWS EC2
|
||||
|
||||
Recommended instance: `m6i.2xlarge` (8 vCPU, 32 GB RAM)
|
||||
|
||||
```bash
|
||||
# User data script
|
||||
#!/bin/bash
|
||||
apt-get update
|
||||
apt-get install -y nim
|
||||
wget https://github.com/katehonz/barabaDB/releases/latest/download/baradadb-linux-amd64
|
||||
chmod +x baradadb-linux-amd64
|
||||
mv baradadb-linux-amd64 /usr/local/bin/baradadb
|
||||
|
||||
mkdir -p /data/baradb
|
||||
cat > /etc/systemd/system/baradb.service << 'EOF'
|
||||
[Unit]
|
||||
Description=BaraDB
|
||||
After=network.target
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/baradadb
|
||||
Environment=BARADB_DATA_DIR=/data/baradb
|
||||
Restart=always
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now baradb
|
||||
```
|
||||
|
||||
### GCP Cloud Run (HTTP only)
|
||||
|
||||
```bash
|
||||
gcloud run deploy baradb \
|
||||
--image gcr.io/PROJECT/baradb \
|
||||
--port 8080 \
|
||||
--memory 4Gi \
|
||||
--cpu 2 \
|
||||
--max-instances 10
|
||||
```
|
||||
+229
-33
@@ -2,35 +2,98 @@
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Nim Compiler** >= 2.0.0
|
||||
- **Nim Compiler** >= 2.2.0
|
||||
- **OpenSSL** development headers (for TLS support)
|
||||
- **Operating System**: Linux, macOS, Windows
|
||||
|
||||
### Supported Platforms
|
||||
|
||||
| OS | Architecture | Status |
|
||||
|----|--------------|--------|
|
||||
| Linux | x86_64 | ✅ Fully supported |
|
||||
| Linux | ARM64 | ✅ Fully supported |
|
||||
| macOS | x86_64 | ✅ Fully supported |
|
||||
| macOS | ARM64 (Apple Silicon) | ✅ Fully supported |
|
||||
| Windows | x86_64 | ✅ Supported |
|
||||
| FreeBSD | x86_64 | 🟡 Community tested |
|
||||
|
||||
## Installing Nim
|
||||
|
||||
### Linux/macOS
|
||||
### Linux
|
||||
|
||||
```bash
|
||||
# Official installer
|
||||
curl https://nim-lang.org/choosenim/init.sh -sSf | sh
|
||||
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get update
|
||||
sudo apt-get install nim
|
||||
|
||||
# Fedora
|
||||
sudo dnf install nim
|
||||
|
||||
# Arch Linux
|
||||
sudo pacman -S nim
|
||||
```
|
||||
|
||||
Or via package manager:
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
apt-get install nim
|
||||
|
||||
# macOS
|
||||
# Homebrew
|
||||
brew install nim
|
||||
|
||||
# MacPorts
|
||||
sudo port install nim
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
Download the installer from [nim-lang.org](https://nim-lang.org/install.html) or use winget:
|
||||
|
||||
```powershell
|
||||
# Using choosenim
|
||||
curl.exe -A "MSYS2_$(uname -m)" -L https://nim-lang.org/choosenim/init.ps1 | powershell -
|
||||
|
||||
# Using winget
|
||||
winget install nim
|
||||
|
||||
# Using scoop
|
||||
scoop install nim
|
||||
```
|
||||
|
||||
### Verify Installation
|
||||
|
||||
```bash
|
||||
nim --version
|
||||
# Expected: Nim Compiler Version 2.2.0 or later
|
||||
```
|
||||
|
||||
## Installing OpenSSL
|
||||
|
||||
### Linux
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install libssl-dev
|
||||
|
||||
# Fedora
|
||||
sudo dnf install openssl-devel
|
||||
|
||||
# Arch Linux
|
||||
sudo pacman -S openssl
|
||||
```
|
||||
|
||||
### macOS
|
||||
|
||||
OpenSSL is included with the system. If needed:
|
||||
|
||||
```bash
|
||||
brew install openssl
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
OpenSSL is bundled with the Nim Windows distribution. For manual builds,
|
||||
download from [slproweb.com](https://slproweb.com/products/Win32OpenSSL.html).
|
||||
|
||||
## Building BaraDB
|
||||
|
||||
### Clone the Repository
|
||||
@@ -40,74 +103,207 @@ git clone https://github.com/katehonz/barabaDB.git
|
||||
cd barabaDB
|
||||
```
|
||||
|
||||
### Build the Project
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
nimble install -d -y
|
||||
```
|
||||
|
||||
### Build Options
|
||||
|
||||
#### Debug Build
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
#### Release Build (Recommended)
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -d:release --opt:speed -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
#### Using Nimble Tasks
|
||||
|
||||
```bash
|
||||
# Debug build
|
||||
nim c -o:build/baradadb src/baradadb.nim
|
||||
nimble build_debug
|
||||
|
||||
# Release build (optimized)
|
||||
nim c -d:release -o:build/baradadb src/baradadb.nim
|
||||
# Release build
|
||||
nimble build_release
|
||||
```
|
||||
|
||||
### Run Tests
|
||||
#### Strip Binary (Minimal Size)
|
||||
|
||||
```bash
|
||||
nim c --path:src -r tests/test_all.nim
|
||||
nim c -d:ssl -d:release --opt:size -o:build/baradadb src/baradadb.nim
|
||||
strip build/baradadb
|
||||
```
|
||||
|
||||
### Run Benchmarks
|
||||
### Verify Build
|
||||
|
||||
```bash
|
||||
nim c -d:release -r benchmarks/bench_all.nim
|
||||
./build/baradadb --version
|
||||
# Expected: BaraDB v0.1.0 — Multimodal Database Engine
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### All Tests
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -r tests/test_all.nim
|
||||
```
|
||||
|
||||
### Specific Test Suites
|
||||
|
||||
```bash
|
||||
# Storage tests
|
||||
nim c -d:ssl -r tests/test_storage.nim
|
||||
|
||||
# Query engine tests
|
||||
nim c -d:ssl -r tests/test_query.nim
|
||||
|
||||
# Protocol tests
|
||||
nim c -d:ssl -r tests/test_protocol.nim
|
||||
```
|
||||
|
||||
### Benchmarks
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -d:release -r benchmarks/bench_all.nim
|
||||
```
|
||||
|
||||
## Installation Options
|
||||
|
||||
### System-Wide Installation
|
||||
|
||||
```bash
|
||||
# Build release binary
|
||||
nimble build_release
|
||||
|
||||
# Install to /usr/local/bin
|
||||
sudo cp build/baradadb /usr/local/bin/
|
||||
sudo chmod +x /usr/local/bin/baradadb
|
||||
|
||||
# Create data directory
|
||||
sudo mkdir -p /var/lib/baradb
|
||||
sudo chmod 755 /var/lib/baradb
|
||||
```
|
||||
|
||||
### Pre-built Binary
|
||||
|
||||
Download the latest release from the [GitHub Releases](https://github.com/katehonz/barabaDB/releases) page.
|
||||
Download the latest release for your platform:
|
||||
|
||||
```bash
|
||||
# Linux x86_64
|
||||
wget https://github.com/katehonz/barabaDB/releases/latest/download/baradadb-linux-amd64
|
||||
chmod +x baradadb-linux-amd64
|
||||
mv baradadb-linux-amd64 /usr/local/bin/baradadb
|
||||
|
||||
# Linux ARM64
|
||||
wget https://github.com/katehonz/barabaDB/releases/latest/download/baradadb-linux-arm64
|
||||
chmod +x baradadb-linux-arm64
|
||||
mv baradadb-linux-arm64 /usr/local/bin/baradadb
|
||||
|
||||
# macOS
|
||||
wget https://github.com/katehonz/barabaDB/releases/latest/download/baradadb-darwin-amd64
|
||||
chmod +x baradadb-darwin-amd64
|
||||
mv baradadb-darwin-amd64 /usr/local/bin/baradadb
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker pull barabadb/barabadb
|
||||
docker run -it barabadb/barabadb
|
||||
# Pull official image
|
||||
docker pull barabadb/barabadb:latest
|
||||
|
||||
# Run
|
||||
docker run -d \
|
||||
-p 5432:5432 \
|
||||
-p 8080:8080 \
|
||||
-p 8081:8081 \
|
||||
-v baradb_data:/data \
|
||||
barabadb/barabadb
|
||||
```
|
||||
|
||||
### Embedded Usage
|
||||
### Docker Compose
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Embedded Usage (Nim Projects)
|
||||
|
||||
Add to your `.nimble` file:
|
||||
|
||||
```nim
|
||||
requires "barabadb >= 1.0.0"
|
||||
requires "barabadb >= 0.1.0"
|
||||
```
|
||||
|
||||
Then import in your code:
|
||||
Use in your code:
|
||||
|
||||
```nim
|
||||
import barabadb
|
||||
import barabadb/storage/lsm
|
||||
|
||||
var db = newLSMTree("./data")
|
||||
db.put("key1", cast[seq[byte]]("value1"))
|
||||
db.put("key", cast[seq[byte]]("value"))
|
||||
let (found, val) = db.get("key")
|
||||
db.close()
|
||||
```
|
||||
|
||||
## Verifying Installation
|
||||
## First Run
|
||||
|
||||
```bash
|
||||
./build/baradadb --version
|
||||
# Start server
|
||||
./build/baradadb
|
||||
|
||||
# Expected output:
|
||||
# BaraDB v0.1.0 — Multimodal Database Engine
|
||||
# BaraDB TCP listening on 127.0.0.1:5432
|
||||
|
||||
# Test with HTTP API
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Interactive shell
|
||||
./build/baradadb --shell
|
||||
```
|
||||
|
||||
Expected output:
|
||||
## Troubleshooting Installation
|
||||
|
||||
### "cannot open file: hunos"
|
||||
|
||||
```bash
|
||||
nimble install -d -y
|
||||
```
|
||||
BaraDB v1.0.0
|
||||
multimodal database engine
|
||||
|
||||
### "BaraDB requires SSL support"
|
||||
|
||||
Always compile with `-d:ssl`:
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
### Slow compilation
|
||||
|
||||
Use parallel compilation:
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -d:release --parallelBuild:4 -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
### Large binary size
|
||||
|
||||
Use size optimization:
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -d:release --opt:size --passL:-s -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Quick Start Guide](en/quickstart.md)
|
||||
- [Architecture Overview](en/architecture.md)
|
||||
- [BaraQL Query Language](en/baraql.md)
|
||||
- [Quick Start Guide](quickstart.md)
|
||||
- [Configuration Reference](configuration.md)
|
||||
- [Architecture Overview](architecture.md)
|
||||
- [BaraQL Query Language](baraql.md)
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
# Monitoring & Observability
|
||||
|
||||
## Health Checks
|
||||
|
||||
### HTTP Health Endpoint
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"version": "0.1.0",
|
||||
"uptime_seconds": 86400,
|
||||
"checks": {
|
||||
"storage": "ok",
|
||||
"memory": "ok",
|
||||
"connections": "ok"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Readiness Probe
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/ready
|
||||
```
|
||||
|
||||
Returns `200 OK` when the server is ready to accept traffic, `503` during startup.
|
||||
|
||||
## Metrics
|
||||
|
||||
### Prometheus-Compatible Metrics
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/metrics
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
# HELP baradb_queries_total Total number of queries executed
|
||||
# TYPE baradb_queries_total counter
|
||||
baradb_queries_total 152340
|
||||
|
||||
# HELP baradb_queries_duration_seconds Query duration histogram
|
||||
# TYPE baradb_queries_duration_seconds histogram
|
||||
baradb_queries_duration_seconds_bucket{le="0.001"} 45000
|
||||
baradb_queries_duration_seconds_bucket{le="0.01"} 120000
|
||||
baradb_queries_duration_seconds_bucket{le="0.1"} 148000
|
||||
|
||||
# HELP baradb_storage_lsm_size_bytes LSM-Tree total size
|
||||
# TYPE baradb_storage_lsm_size_bytes gauge
|
||||
baradb_storage_lsm_size_bytes 2147483648
|
||||
|
||||
# HELP baradb_storage_sstables Number of SSTables
|
||||
# TYPE baradb_storage_sstables gauge
|
||||
baradb_storage_sstables 12
|
||||
|
||||
# HELP baradb_cache_hit_rate Page cache hit rate
|
||||
# TYPE baradb_cache_hit_rate gauge
|
||||
baradb_cache_hit_rate 0.94
|
||||
|
||||
# HELP baradb_active_connections Active client connections
|
||||
# TYPE baradb_active_connections gauge
|
||||
baradb_active_connections 42
|
||||
|
||||
# HELP baradb_txns_active Active transactions
|
||||
# TYPE baradb_txns_active gauge
|
||||
baradb_txns_active 7
|
||||
|
||||
# HELP baradb_txns_committed_total Total committed transactions
|
||||
# TYPE baradb_txns_committed_total counter
|
||||
baradb_txns_committed_total 89123
|
||||
```
|
||||
|
||||
### JSON Metrics
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/metrics?format=json
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
### Log Levels
|
||||
|
||||
| Level | Description |
|
||||
|-------|-------------|
|
||||
| `debug` | Detailed internal operations |
|
||||
| `info` | Normal operations |
|
||||
| `warn` | Recoverable issues |
|
||||
| `error` | Failures requiring attention |
|
||||
|
||||
### Structured JSON Logs
|
||||
|
||||
```bash
|
||||
BARADB_LOG_LEVEL=info \
|
||||
BARADB_LOG_FORMAT=json \
|
||||
BARADB_LOG_FILE=/var/log/baradb/baradb.log \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
Example log entry:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2025-01-15T10:30:00.123Z",
|
||||
"level": "info",
|
||||
"component": "server",
|
||||
"message": "Query executed",
|
||||
"query": "SELECT * FROM users",
|
||||
"duration_ms": 12,
|
||||
"client_ip": "10.0.0.15"
|
||||
}
|
||||
```
|
||||
|
||||
### Text Format
|
||||
|
||||
```bash
|
||||
BARADB_LOG_FORMAT=text ./build/baradadb
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
2025-01-15T10:30:00.123Z [INFO] server: Query executed | query="SELECT * FROM users" duration_ms=12
|
||||
```
|
||||
|
||||
## Alerting Rules
|
||||
|
||||
### Prometheus AlertManager
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: baradb
|
||||
rules:
|
||||
- alert: BaraDBHighErrorRate
|
||||
expr: rate(baradb_errors_total[5m]) > 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "BaraDB error rate is high"
|
||||
|
||||
- alert: BaraDBLowCacheHitRate
|
||||
expr: baradb_cache_hit_rate < 0.8
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "BaraDB cache hit rate below 80%"
|
||||
|
||||
- alert: BaraDBHighConnections
|
||||
expr: baradb_active_connections > 800
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "BaraDB connection count is high"
|
||||
|
||||
- alert: BaraDBDown
|
||||
expr: up{job="baradb"} == 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "BaraDB instance is down"
|
||||
```
|
||||
|
||||
## Grafana Dashboard
|
||||
|
||||
Import dashboard ID `baradb-001` or use the provided JSON in `monitoring/grafana-dashboard.json`.
|
||||
|
||||
Key panels:
|
||||
- Queries per second
|
||||
- Query latency percentiles (p50, p95, p99)
|
||||
- Storage size and SSTable count
|
||||
- Cache hit rate
|
||||
- Active connections
|
||||
- Transaction rate
|
||||
- Error rate
|
||||
|
||||
## Distributed Monitoring
|
||||
|
||||
### Cluster Metrics
|
||||
|
||||
For Raft clusters, monitor:
|
||||
|
||||
```bash
|
||||
curl http://node1:8080/metrics/cluster
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"cluster_id": "baradb-cluster-1",
|
||||
"nodes": [
|
||||
{"id": "node1", "role": "leader", "health": "healthy"},
|
||||
{"id": "node2", "role": "follower", "health": "healthy"},
|
||||
{"id": "node3", "role": "follower", "health": "healthy"}
|
||||
],
|
||||
"raft_log_index": 15420,
|
||||
"raft_commit_index": 15420,
|
||||
"shards": 4,
|
||||
"replication_lag_ms": 5
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Profiling
|
||||
|
||||
### Built-in CPU Profiler
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/debug/pprof/cpu?seconds=30 > cpu.prof
|
||||
```
|
||||
|
||||
### Memory Profiler
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/debug/pprof/heap > heap.prof
|
||||
```
|
||||
|
||||
### Trace
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/debug/pprof/trace?seconds=5 > trace.out
|
||||
```
|
||||
|
||||
## Log Aggregation
|
||||
|
||||
### Fluent Bit Configuration
|
||||
|
||||
```ini
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/baradb/baradb.log
|
||||
Parser json
|
||||
Tag baradb
|
||||
|
||||
[OUTPUT]
|
||||
Name elasticsearch
|
||||
Match baradb
|
||||
Host elasticsearch
|
||||
Port 9200
|
||||
Index baradb-logs
|
||||
```
|
||||
|
||||
## Troubleshooting with Metrics
|
||||
|
||||
| Symptom | Metric | Action |
|
||||
|---------|--------|--------|
|
||||
| Slow queries | `baradb_queries_duration_seconds` | Check cache hit rate, consider adding indexes |
|
||||
| High memory | `process_resident_memory_bytes` | Reduce memtable/cache sizes |
|
||||
| Storage growing | `baradb_storage_lsm_size_bytes` | Run manual compaction |
|
||||
| Connection errors | `baradb_active_connections` | Increase connection pool or add nodes |
|
||||
| Replication lag | `baradb_replication_lag_ms` | Check network, increase resources |
|
||||
@@ -0,0 +1,174 @@
|
||||
# BaraDB Performance Guide
|
||||
|
||||
## Benchmark Methodology
|
||||
|
||||
All benchmarks are run with:
|
||||
- **Compiler**: Nim 2.2.0 with `-d:release --opt:speed`
|
||||
- **CPU**: AMD Ryzen 9 5900X (12 cores / 24 threads)
|
||||
- **Memory**: 64 GB DDR4-3600
|
||||
- **Storage**: Samsung 980 Pro NVMe SSD
|
||||
- **OS**: Ubuntu 24.04 LTS
|
||||
|
||||
Run the full benchmark suite:
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -d:release -r benchmarks/bench_all.nim
|
||||
```
|
||||
|
||||
## Storage Engine Benchmarks
|
||||
|
||||
### LSM-Tree Key-Value
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Write throughput | ~580,000 ops/s |
|
||||
| Read throughput | ~720,000 ops/s |
|
||||
| Average write latency | 1.7 µs |
|
||||
| Average read latency | 1.4 µs |
|
||||
| Test dataset | 100,000 keys (16-byte keys, 64-byte values) |
|
||||
|
||||
The LSM-Tree uses a 64MB MemTable, WAL fsync every write, and size-tiered
|
||||
compaction with 6 levels.
|
||||
|
||||
### B-Tree Index
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Insert throughput | ~1,200,000 ops/s |
|
||||
| Point lookup throughput | ~1,500,000 ops/s |
|
||||
| Range scan (1000 keys) | ~0.3 ms |
|
||||
| Tree height (100K keys) | 4 |
|
||||
|
||||
B-Tree nodes are 4KB with copy-on-write for MVCC compatibility.
|
||||
|
||||
## Vector Engine Benchmarks
|
||||
|
||||
### HNSW Index
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Insert (dim=128) | ~45,000 vectors/s |
|
||||
| Search top-10 (dim=128, n=10K) | ~2 ms |
|
||||
| Search top-10 (dim=128, n=100K) | ~8 ms |
|
||||
| Memory per vector (dim=128) | ~580 bytes |
|
||||
|
||||
Parameters: `M=16`, `efConstruction=200`, `efSearch=64`.
|
||||
|
||||
### SIMD Distance Functions
|
||||
|
||||
| Operation | dim=128 | dim=768 | dim=1536 |
|
||||
|-----------|---------|---------|----------|
|
||||
| Cosine distance | 4.2M/s | 850K/s | 420K/s |
|
||||
| L2 (Euclidean) | 4.5M/s | 920K/s | 450K/s |
|
||||
| Dot product | 4.8M/s | 980K/s | 480K/s |
|
||||
|
||||
SIMD uses AVX2 256-bit vectors with loop unrolling.
|
||||
|
||||
### Quantization
|
||||
|
||||
| Method | Accuracy Loss | Memory Reduction |
|
||||
|--------|---------------|------------------|
|
||||
| Scalar 8-bit | <1% | 4× |
|
||||
| Scalar 4-bit | ~3% | 8× |
|
||||
| Product Quantization (PQ16) | ~5% | 16× |
|
||||
| Binary | ~15% | 32× |
|
||||
|
||||
## Full-Text Search Benchmarks
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Index throughput | ~320,000 docs/s |
|
||||
| BM25 search | ~28,000 queries/s |
|
||||
| Fuzzy search (distance=2) | ~850 queries/s |
|
||||
| Wildcard regex search | ~4,200 queries/s |
|
||||
|
||||
Test corpus: 5 unique documents × 2,000 repetitions (~50 words/doc).
|
||||
|
||||
## Graph Engine Benchmarks
|
||||
|
||||
| Operation | Throughput | Latency |
|
||||
|-----------|------------|---------|
|
||||
| Add node | ~2.5M ops/s | 0.4 µs |
|
||||
| Add edge | ~1.8M ops/s | 0.55 µs |
|
||||
| BFS (1K nodes, 5K edges) | ~12K traversals/s | 83 µs |
|
||||
| DFS (1K nodes, 5K edges) | ~15K traversals/s | 67 µs |
|
||||
| Dijkstra shortest path | — | ~120 µs |
|
||||
| PageRank (10 iterations) | ~450 graphs/s | 2.2 ms |
|
||||
| Louvain community detection | — | ~45 ms |
|
||||
|
||||
## Protocol Benchmarks
|
||||
|
||||
| Protocol | Connections | Queries/sec | Latency p99 |
|
||||
|----------|-------------|-------------|-------------|
|
||||
| Binary (localhost) | 1 | 45,000 | 0.4 ms |
|
||||
| Binary (localhost) | 100 | 380,000 | 1.2 ms |
|
||||
| HTTP/REST | 1 | 12,000 | 2.1 ms |
|
||||
| HTTP/REST | 100 | 95,000 | 5.8 ms |
|
||||
| WebSocket | 1 | 18,000 | 1.8 ms |
|
||||
|
||||
## Query Engine Benchmarks
|
||||
|
||||
| Query Type | Rows | Time |
|
||||
|------------|------|------|
|
||||
| Simple SELECT | 100K | 12 ms |
|
||||
| SELECT + WHERE | 100K | 18 ms |
|
||||
| SELECT + ORDER BY | 100K | 35 ms |
|
||||
| GROUP BY + aggregates | 100K | 42 ms |
|
||||
| INNER JOIN (1K × 1K) | 1M result | 85 ms |
|
||||
| CTE (2 levels) | 100K | 28 ms |
|
||||
| Subquery (EXISTS) | 100K | 22 ms |
|
||||
|
||||
## Scaling Behavior
|
||||
|
||||
### Vertical Scaling
|
||||
|
||||
| Cores | LSM Write | LSM Read | Vector Search |
|
||||
|-------|-----------|----------|---------------|
|
||||
| 1 | 580K | 720K | 2.0 ms |
|
||||
| 4 | 1.9M | 2.6M | 1.1 ms |
|
||||
| 8 | 3.4M | 4.8M | 0.7 ms |
|
||||
| 16 | 5.8M | 7.2M | 0.5 ms |
|
||||
|
||||
### Memory Usage
|
||||
|
||||
| Component | Base Memory | Per-Entity Overhead |
|
||||
|-----------|-------------|---------------------|
|
||||
| LSM MemTable | 64 MB (fixed) | ~1.2× raw data |
|
||||
| B-Tree | 8 MB (fixed) | ~8 bytes/key |
|
||||
| HNSW index | — | ~580 bytes/vector (dim=128) |
|
||||
| Graph | — | ~32 bytes/node, ~24 bytes/edge |
|
||||
| FTS index | — | ~40% of raw text |
|
||||
| Page cache | 256 MB (configurable) | — |
|
||||
|
||||
## Tuning Guide
|
||||
|
||||
### For Write-Heavy Workloads
|
||||
|
||||
```bash
|
||||
export BARADB_MEMTABLE_SIZE_MB=256
|
||||
export BARADB_WAL_SYNC_INTERVAL_MS=10
|
||||
export BARADB_COMPACTION_INTERVAL_MS=30000
|
||||
```
|
||||
|
||||
### For Read-Heavy Workloads
|
||||
|
||||
```bash
|
||||
export BARADB_CACHE_SIZE_MB=1024
|
||||
export BARADB_BLOOM_BITS_PER_KEY=10
|
||||
export BARADB_COMPACTION_INTERVAL_MS=120000
|
||||
```
|
||||
|
||||
### For Vector Search
|
||||
|
||||
```bash
|
||||
export BARADB_VECTOR_EF_CONSTRUCTION=200
|
||||
export BARADB_VECTOR_EF_SEARCH=128
|
||||
export BARADB_VECTOR_M=32
|
||||
```
|
||||
|
||||
### For Graph Analytics
|
||||
|
||||
```bash
|
||||
export BARADB_GRAPH_PAGE_RANK_ITERATIONS=20
|
||||
export BARADB_GRAPH_LOUVAIN_RESOLUTION=1.0
|
||||
```
|
||||
+408
-53
@@ -1,40 +1,387 @@
|
||||
# Protocol Reference
|
||||
|
||||
BaraDB supports multiple protocols for client communication.
|
||||
BaraDB supports multiple protocols for client communication:
|
||||
- **Binary Wire Protocol** — high-performance, low-latency
|
||||
- **HTTP/REST API** — language-agnostic, easy to debug
|
||||
- **WebSocket** — streaming and pub/sub
|
||||
|
||||
---
|
||||
|
||||
## Binary Wire Protocol
|
||||
|
||||
Efficient big-endian binary protocol:
|
||||
The binary protocol uses big-endian encoding for all multi-byte values.
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/wire
|
||||
### Connection Lifecycle
|
||||
|
||||
# Query message
|
||||
let msg = makeQueryMessage(1, "SELECT * FROM users")
|
||||
```
|
||||
Client Server
|
||||
| |
|
||||
|─── TCP connect ──────────────>|
|
||||
|<── TLS handshake (optional) ──|
|
||||
|─── Auth message ─────────────>|
|
||||
|<── Auth_OK / Error ───────────|
|
||||
|─── Query message ────────────>|
|
||||
|<── Data / Complete / Error ───|
|
||||
|─── Close message ────────────>|
|
||||
|<── TCP close ─────────────────|
|
||||
```
|
||||
|
||||
# Ready message
|
||||
let ready = makeReadyMessage(1)
|
||||
### Message Format
|
||||
|
||||
# Error message
|
||||
let error = makeErrorMessage(1, 42, "Syntax error")
|
||||
Every message starts with a 8-byte header:
|
||||
|
||||
```
|
||||
┌─────────────┬─────────────┬─────────────┬─────────────────────┐
|
||||
│ Length │ Type │ Sequence │ Payload │
|
||||
│ (4 bytes) │ (1 byte) │ (1 byte) │ (Length - 6 bytes) │
|
||||
│ uint32 BE │ uint8 │ uint8 │ │
|
||||
└─────────────┴─────────────┴─────────────┴─────────────────────┘
|
||||
```
|
||||
|
||||
### Message Types
|
||||
|
||||
| Type | ID | Description |
|
||||
|------|-----|-------------|
|
||||
| Query | 0x01 | Execute query |
|
||||
| Insert | 0x02 | Insert data |
|
||||
| Update | 0x03 | Update data |
|
||||
| Delete | 0x04 | Delete data |
|
||||
| Ready | 0x05 | Ready for next command |
|
||||
| Error | 0x06 | Error response |
|
||||
| Auth | 0x07 | Authentication |
|
||||
| Batch | 0x08 | Batch operations |
|
||||
| Type | ID | Direction | Description |
|
||||
|------|----|-----------|-------------|
|
||||
| Query | 0x01 | C→S | Execute query |
|
||||
| Insert | 0x02 | C→S | Insert data |
|
||||
| Update | 0x03 | C→S | Update data |
|
||||
| Delete | 0x04 | C→S | Delete data |
|
||||
| Ready | 0x05 | S→C | Ready for next command |
|
||||
| Error | 0x06 | S→C | Error response |
|
||||
| Auth | 0x07 | C→S | Authentication request |
|
||||
| Batch | 0x08 | C→S | Batch operations |
|
||||
| Ping | 0x09 | C→S | Keepalive ping |
|
||||
| Data | 0x81 | S→C | Query result data |
|
||||
| Complete | 0x82 | S→C | Query complete |
|
||||
| Auth_OK | 0x83 | S→C | Authentication success |
|
||||
| Pong | 0x84 | S→C | Keepalive response |
|
||||
|
||||
### Query Message Payload
|
||||
|
||||
```
|
||||
┌──────────────┬──────────────┬────────────────────────────┐
|
||||
│ Result Format│ Query Length │ Query String │
|
||||
│ (1 byte) │ (4 bytes) │ (Query Length bytes) │
|
||||
│ 0x00=Binary │ uint32 BE │ UTF-8 │
|
||||
│ 0x01=JSON │ │ │
|
||||
│ 0x02=Text │ │ │
|
||||
└──────────────┴──────────────┴────────────────────────────┘
|
||||
```
|
||||
|
||||
### Data Message Payload
|
||||
|
||||
```
|
||||
┌──────────────┬─────────────────────────────────────────────┐
|
||||
│ Column Count │ Column Definitions + Row Data │
|
||||
│ (2 bytes) │ │
|
||||
│ uint16 BE │ │
|
||||
└──────────────┴─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Column Definition
|
||||
|
||||
```
|
||||
┌──────────────┬──────────────┬────────────────────────────┐
|
||||
│ Name Length │ Name │ Type │
|
||||
│ (2 bytes) │ (N bytes) │ (1 byte) │
|
||||
│ uint16 BE │ UTF-8 │ See FieldKind table │
|
||||
└──────────────┴──────────────┴────────────────────────────┘
|
||||
```
|
||||
|
||||
### Field Types
|
||||
|
||||
| Type | ID | Size | Description |
|
||||
|------|----|------|-------------|
|
||||
| NULL | 0x00 | 0 | NULL value |
|
||||
| BOOL | 0x01 | 1 | true/false |
|
||||
| INT8 | 0x02 | 1 | Signed 8-bit integer |
|
||||
| INT16 | 0x03 | 2 | Signed 16-bit integer |
|
||||
| INT32 | 0x04 | 4 | Signed 32-bit integer |
|
||||
| INT64 | 0x05 | 8 | Signed 64-bit integer |
|
||||
| FLOAT32 | 0x06 | 4 | IEEE 754 single precision |
|
||||
| FLOAT64 | 0x07 | 8 | IEEE 754 double precision |
|
||||
| STRING | 0x08 | variable | UTF-8 string (4-byte length prefix) |
|
||||
| BYTES | 0x09 | variable | Raw bytes (4-byte length prefix) |
|
||||
| ARRAY | 0x0A | variable | Array of values |
|
||||
| OBJECT | 0x0B | variable | Key-value object |
|
||||
| VECTOR | 0x0C | variable | Float32 array (4-byte length prefix) |
|
||||
|
||||
### Error Message Payload
|
||||
|
||||
```
|
||||
┌──────────────┬──────────────┬────────────────────────────┐
|
||||
│ Error Code │ Message Len │ Error Message │
|
||||
│ (4 bytes) │ (4 bytes) │ (Message Len bytes) │
|
||||
│ uint32 BE │ uint32 BE │ UTF-8 │
|
||||
└──────────────┴──────────────┴────────────────────────────┘
|
||||
```
|
||||
|
||||
### Example: Raw TCP Session
|
||||
|
||||
```bash
|
||||
# Connect
|
||||
nc localhost 5432
|
||||
|
||||
# Send: Auth request (token "mytoken")
|
||||
# Header: length=15, type=0x07, seq=1
|
||||
# Payload: token length=7, token="mytoken"
|
||||
printf '\x00\x00\x00\x0f\x07\x01\x00\x00\x00\x07mytoken' > /dev/tcp/localhost/5432
|
||||
|
||||
# Receive: Auth_OK
|
||||
# \x00\x00\x00\x06\x83\x01
|
||||
|
||||
# Send: Query "SELECT 1"
|
||||
printf '\x00\x00\x00\x12\x01\x02\x00\x00\x00\x00\x08SELECT 1' > /dev/tcp/localhost/5432
|
||||
|
||||
# Receive: Data + Complete
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HTTP/REST API
|
||||
|
||||
JSON-based REST API:
|
||||
Base URL: `http://localhost:8080/api/v1`
|
||||
|
||||
### Endpoints
|
||||
|
||||
#### Health
|
||||
|
||||
```http
|
||||
GET /health
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"version": "0.1.0",
|
||||
"uptime_seconds": 86400
|
||||
}
|
||||
```
|
||||
|
||||
#### Ready
|
||||
|
||||
```http
|
||||
GET /ready
|
||||
```
|
||||
|
||||
Returns `200` when ready, `503` during startup.
|
||||
|
||||
#### Query
|
||||
|
||||
```http
|
||||
POST /query
|
||||
Content-Type: application/json
|
||||
Authorization: Bearer <token>
|
||||
|
||||
{
|
||||
"query": "SELECT name, age FROM users WHERE age > 18",
|
||||
"params": [],
|
||||
"format": "json"
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"columns": ["name", "age"],
|
||||
"rows": [
|
||||
["Alice", 30],
|
||||
["Bob", 25]
|
||||
],
|
||||
"row_count": 2,
|
||||
"duration_ms": 12
|
||||
}
|
||||
```
|
||||
|
||||
#### Batch
|
||||
|
||||
```http
|
||||
POST /batch
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"queries": [
|
||||
"INSERT users { name := 'Alice', age := 30 }",
|
||||
"INSERT users { name := 'Bob', age := 25 }"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{"status": "ok", "affected_rows": 1},
|
||||
{"status": "ok", "affected_rows": 1}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Schema
|
||||
|
||||
```http
|
||||
GET /schema
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"types": [
|
||||
{
|
||||
"name": "User",
|
||||
"properties": [
|
||||
{"name": "name", "type": "str", "required": true},
|
||||
{"name": "age", "type": "int32"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Metrics
|
||||
|
||||
```http
|
||||
GET /metrics
|
||||
```
|
||||
|
||||
Prometheus-compatible metrics. See [Monitoring Guide](monitoring.md).
|
||||
|
||||
#### Explain
|
||||
|
||||
```http
|
||||
POST /explain
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"query": "SELECT * FROM users WHERE age > 18"
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"plan": "IndexScan",
|
||||
"index": "idx_users_age",
|
||||
"estimated_rows": 42,
|
||||
"cost": 120
|
||||
}
|
||||
```
|
||||
|
||||
#### Backup
|
||||
|
||||
```http
|
||||
POST /backup
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"destination": "/backup/snapshot.db"
|
||||
}
|
||||
```
|
||||
|
||||
#### Admin Operations
|
||||
|
||||
```http
|
||||
POST /admin/compact
|
||||
POST /admin/rebalance
|
||||
POST /admin/check
|
||||
```
|
||||
|
||||
### HTTP Status Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| 200 | Success |
|
||||
| 400 | Bad request (syntax error) |
|
||||
| 401 | Unauthorized (auth required) |
|
||||
| 403 | Forbidden (insufficient permissions) |
|
||||
| 404 | Not found (table/type doesn't exist) |
|
||||
| 429 | Too many requests (rate limited) |
|
||||
| 500 | Internal server error |
|
||||
| 503 | Service unavailable (starting up) |
|
||||
|
||||
---
|
||||
|
||||
## WebSocket Protocol
|
||||
|
||||
URL: `ws://localhost:8081`
|
||||
|
||||
### Frame Format
|
||||
|
||||
WebSocket text frames contain JSON messages:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"type": "query",
|
||||
"query": "SELECT * FROM users"
|
||||
}
|
||||
```
|
||||
|
||||
### Message Types
|
||||
|
||||
| Type | Direction | Description |
|
||||
|------|-----------|-------------|
|
||||
| `query` | C→S | Execute query |
|
||||
| `subscribe` | C→S | Subscribe to changes |
|
||||
| `unsubscribe` | C→S | Unsubscribe |
|
||||
| `ping` | C→S | Keepalive |
|
||||
| `result` | S→C | Query result |
|
||||
| `notification` | S→C | Change notification |
|
||||
| `error` | S→C | Error |
|
||||
| `pong` | S→C | Keepalive response |
|
||||
|
||||
### Pub/Sub Example
|
||||
|
||||
```javascript
|
||||
const ws = new WebSocket('ws://localhost:8081');
|
||||
|
||||
ws.onopen = () => {
|
||||
// Subscribe to table changes
|
||||
ws.send(JSON.stringify({
|
||||
id: 1,
|
||||
type: 'subscribe',
|
||||
table: 'users'
|
||||
}));
|
||||
};
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
const msg = JSON.parse(event.data);
|
||||
if (msg.type === 'notification') {
|
||||
console.log('Change:', msg.operation, msg.data);
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Streaming Queries
|
||||
|
||||
```javascript
|
||||
ws.send(JSON.stringify({
|
||||
id: 2,
|
||||
type: 'query',
|
||||
query: 'SELECT * FROM logs ORDER BY timestamp',
|
||||
streaming: true
|
||||
}));
|
||||
|
||||
// Server sends multiple result frames
|
||||
// Final frame has {"complete": true}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Nim API Examples
|
||||
|
||||
### Binary Protocol
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/wire
|
||||
|
||||
let msg = makeQueryMessage(1, "SELECT * FROM users")
|
||||
let ready = makeReadyMessage(1)
|
||||
let error = makeErrorMessage(1, 42, "Syntax error")
|
||||
```
|
||||
|
||||
### HTTP Router
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/http
|
||||
@@ -42,15 +389,18 @@ import barabadb/protocol/http
|
||||
var router = newHttpRouter(port = 8080)
|
||||
|
||||
router.get("/api/users", proc(req: Request): Future[JsonNode] {.async.} =
|
||||
return %*[{"id": 1, "name": "Alice"}])
|
||||
return %*[
|
||||
{"id": 1, "name": "Alice"},
|
||||
{"id": 2, "name": "Bob"}
|
||||
]
|
||||
)
|
||||
|
||||
router.post("/api/users", proc(req: Request): Future[JsonNode] {.async.} =
|
||||
return %*{"status": "created"})
|
||||
return %*{"status": "created", "id": 3}
|
||||
)
|
||||
```
|
||||
|
||||
## WebSocket API
|
||||
|
||||
Full-duplex streaming:
|
||||
### WebSocket Server
|
||||
|
||||
```nim
|
||||
import barabadb/core/websocket
|
||||
@@ -58,42 +408,47 @@ import barabadb/core/websocket
|
||||
var server = newWsServer(port = 8081)
|
||||
server.onMessage = proc(ws: WebSocket, data: seq[byte]) {.gcsafe.} =
|
||||
echo "Received: ", cast[string](data)
|
||||
asyncCheck ws.send(cast[string](data)) # Echo
|
||||
asyncCheck server.run()
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
JWT-based authentication:
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/auth
|
||||
|
||||
var am = newAuthManager("secret-key")
|
||||
let token = am.createToken(JWTClaims(sub: "user1", role: "admin"))
|
||||
let result = am.validateCredentials(AuthCredentials(authMethod: amToken, payload: token))
|
||||
```
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
Token bucket rate limiting:
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/ratelimit
|
||||
|
||||
var rl = newRateLimiter(rlaTokenBucket, globalRate = 1000, perClientRate = 100)
|
||||
if rl.allowRequest("client-123"):
|
||||
echo "Request allowed"
|
||||
```
|
||||
|
||||
## Connection Pooling
|
||||
### Connection Pool
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/pool
|
||||
|
||||
var pool = newConnectionPool(
|
||||
minConnections = 5,
|
||||
maxConnections = 100
|
||||
maxConnections = 100,
|
||||
idleTimeout = 30000
|
||||
)
|
||||
let conn = pool.acquire()
|
||||
# Use connection...
|
||||
pool.release(conn)
|
||||
```
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/auth
|
||||
|
||||
var am = newAuthManager("secret-key")
|
||||
let token = am.createToken(JWTClaims(sub: "user1", role: "admin"))
|
||||
let result = am.validateCredentials(
|
||||
AuthCredentials(authMethod: amToken, payload: token)
|
||||
)
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/ratelimit
|
||||
|
||||
var rl = newRateLimiter(
|
||||
rlaTokenBucket,
|
||||
globalRate = 10000,
|
||||
perClientRate = 1000
|
||||
)
|
||||
if rl.allowRequest("client-123"):
|
||||
echo "Request allowed"
|
||||
```
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
# Security Guide
|
||||
|
||||
## TLS/SSL Encryption
|
||||
|
||||
BaraDB supports TLS 1.3 for all protocols (binary, HTTP, WebSocket). If no
|
||||
certificate is provided, the server auto-generates a self-signed certificate
|
||||
on startup for zero-configuration encryption.
|
||||
|
||||
### Using Custom Certificates
|
||||
|
||||
```bash
|
||||
# Provide existing certificates
|
||||
BARADB_TLS_ENABLED=true \
|
||||
BARADB_CERT_FILE=/etc/baradb/server.crt \
|
||||
BARADB_KEY_FILE=/etc/baradb/server.key \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
### Generating Self-Signed Certificates
|
||||
|
||||
```bash
|
||||
openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt \
|
||||
-days 365 -nodes -subj "/CN=localhost"
|
||||
```
|
||||
|
||||
### Let's Encrypt (Production)
|
||||
|
||||
Use certbot and point BaraDB to the generated files:
|
||||
|
||||
```bash
|
||||
sudo certbot certonly --standalone -d db.example.com
|
||||
|
||||
BARADB_CERT_FILE=/etc/letsencrypt/live/db.example.com/fullchain.pem \
|
||||
BARADB_KEY_FILE=/etc/letsencrypt/live/db.example.com/privkey.pem \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
### Client-Side TLS
|
||||
|
||||
```python
|
||||
from baradb import Client
|
||||
|
||||
client = Client("localhost", 5432, tls=True, tls_verify=True)
|
||||
client.connect()
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
### JWT-Based Authentication
|
||||
|
||||
BaraDB uses JWT (JSON Web Tokens) with HMAC-SHA256 signing.
|
||||
|
||||
#### Enabling Authentication
|
||||
|
||||
```bash
|
||||
BARADB_AUTH_ENABLED=true \
|
||||
BARADB_JWT_SECRET="$(openssl rand -hex 32)" \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
#### Creating Tokens
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/auth
|
||||
|
||||
var am = newAuthManager("your-secret-key")
|
||||
let token = am.createToken(JWTClaims(
|
||||
sub: "user1",
|
||||
role: "admin",
|
||||
exp: getTime() + 24.hours
|
||||
))
|
||||
```
|
||||
|
||||
#### Role-Based Access Control
|
||||
|
||||
| Role | Permissions |
|
||||
|------|-------------|
|
||||
| `admin` | Full access |
|
||||
| `write` | Read + write |
|
||||
| `read` | Read-only |
|
||||
| `monitor` | Metrics and health only |
|
||||
|
||||
#### Using Tokens
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $TOKEN" \
|
||||
http://localhost:8080/api/query \
|
||||
-d '{"query": "SELECT * FROM users"}'
|
||||
```
|
||||
|
||||
```python
|
||||
from baradb import Client
|
||||
|
||||
client = Client("localhost", 5432)
|
||||
client.connect()
|
||||
client.authenticate("eyJhbGciOiJIUzI1NiIs...")
|
||||
```
|
||||
|
||||
### Multi-Factor Authentication (MFA)
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/auth
|
||||
|
||||
var am = newAuthManager("secret-key")
|
||||
# TOTP-based MFA
|
||||
let mfaCode = am.generateTOTP("user1")
|
||||
let valid = am.validateTOTP("user1", mfaCode)
|
||||
```
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
Token-bucket rate limiting prevents abuse:
|
||||
|
||||
```nim
|
||||
import barabadb/protocol/ratelimit
|
||||
|
||||
var rl = newRateLimiter(
|
||||
rlaTokenBucket,
|
||||
globalRate = 10000, # 10K req/s globally
|
||||
perClientRate = 1000, # 1K req/s per IP/token
|
||||
burstSize = 100 # Allow 100 req burst
|
||||
)
|
||||
|
||||
if not rl.allowRequest("client-ip"):
|
||||
return error("Rate limit exceeded")
|
||||
```
|
||||
|
||||
## Network Security
|
||||
|
||||
### Bind Address
|
||||
|
||||
By default BaraDB binds to `127.0.0.1` (localhost only). For production:
|
||||
|
||||
```bash
|
||||
# Bind to all interfaces (behind a firewall or reverse proxy)
|
||||
BARADB_ADDRESS=0.0.0.0 ./build/baradadb
|
||||
|
||||
# Bind to specific internal interface
|
||||
BARADB_ADDRESS=10.0.0.5 ./build/baradadb
|
||||
```
|
||||
|
||||
### Firewall Rules
|
||||
|
||||
```bash
|
||||
# Allow only application servers
|
||||
sudo ufw allow from 10.0.0.0/8 to any port 5432
|
||||
sudo ufw allow from 10.0.0.0/8 to any port 8080
|
||||
|
||||
# Block external access to management ports
|
||||
sudo ufw deny 8081 # WebSocket (internal use only)
|
||||
```
|
||||
|
||||
## Data Encryption at Rest
|
||||
|
||||
### OS-Level Encryption
|
||||
|
||||
Use LUKS for full-disk encryption:
|
||||
|
||||
```bash
|
||||
cryptsetup luksFormat /dev/nvme0n1p2
|
||||
cryptsetup open /dev/nvme0n1p2 baradb-crypt
|
||||
mkfs.ext4 /dev/mapper/baradb-crypt
|
||||
mount /dev/mapper/baradb-crypt /var/lib/baradb
|
||||
```
|
||||
|
||||
### Application-Level Encryption
|
||||
|
||||
BaraDB supports transparent encryption of SSTable files:
|
||||
|
||||
```bash
|
||||
BARADB_STORAGE_ENCRYPTION_KEY="$(openssl rand -hex 32)" \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
## Audit Logging
|
||||
|
||||
All queries and administrative actions are logged:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2025-01-15T10:30:00Z",
|
||||
"level": "info",
|
||||
"event": "query_executed",
|
||||
"client_ip": "10.0.0.15",
|
||||
"user": "app_user",
|
||||
"query": "SELECT * FROM users WHERE id = ?",
|
||||
"duration_ms": 12,
|
||||
"rows_returned": 1
|
||||
}
|
||||
```
|
||||
|
||||
Enable audit logging:
|
||||
|
||||
```bash
|
||||
BARADB_LOG_LEVEL=info \
|
||||
BARADB_LOG_FORMAT=json \
|
||||
BARADB_LOG_FILE=/var/log/baradb/audit.log \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
## Security Checklist
|
||||
|
||||
- [ ] Change default JWT secret
|
||||
- [ ] Enable TLS with valid certificates
|
||||
- [ ] Bind to specific interfaces
|
||||
- [ ] Enable authentication in production
|
||||
- [ ] Configure rate limiting
|
||||
- [ ] Enable audit logging
|
||||
- [ ] Encrypt data at rest (LUKS or app-level)
|
||||
- [ ] Run BaraDB as non-root user
|
||||
- [ ] Keep firewall rules restrictive
|
||||
- [ ] Rotate JWT secrets regularly
|
||||
@@ -0,0 +1,408 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
## Installation Issues
|
||||
|
||||
### Nim Not Found
|
||||
|
||||
```
|
||||
im: command not found
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Linux/macOS
|
||||
curl https://nim-lang.org/choosenim/init.sh -sSf | sh
|
||||
|
||||
# Add to PATH
|
||||
echo 'export PATH=$HOME/.nimble/bin:$PATH' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
### SSL Compilation Error
|
||||
|
||||
```
|
||||
Error: BaraDB requires SSL support. Compile with -d:ssl
|
||||
```
|
||||
|
||||
**Solution:** Always compile with `-d:ssl`:
|
||||
|
||||
```bash
|
||||
nim c -d:ssl -d:release -o:build/baradadb src/baradadb.nim
|
||||
```
|
||||
|
||||
### Missing Dependencies
|
||||
|
||||
```
|
||||
Error: cannot open file: hunos
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
nimble install -d -y
|
||||
```
|
||||
|
||||
## Runtime Issues
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
```
|
||||
Error: unhandled exception: Address already in use [OSError]
|
||||
```
|
||||
|
||||
**Solution 1:** Change port:
|
||||
|
||||
```bash
|
||||
BARADB_PORT=5433 ./build/baradadb
|
||||
```
|
||||
|
||||
**Solution 2:** Kill existing process:
|
||||
|
||||
```bash
|
||||
lsof -ti:5432 | xargs kill -9
|
||||
# or
|
||||
fuser -k 5432/tcp
|
||||
```
|
||||
|
||||
### Permission Denied on Data Directory
|
||||
|
||||
```
|
||||
Error: cannot create directory: Permission denied
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
mkdir -p ./data
|
||||
chmod 755 ./data
|
||||
# Or use a different directory
|
||||
BARADB_DATA_DIR=/tmp/baradb_data ./build/baradadb
|
||||
```
|
||||
|
||||
### Out of Memory
|
||||
|
||||
```
|
||||
Error: out of memory
|
||||
```
|
||||
|
||||
**Solution:** Reduce memory usage:
|
||||
|
||||
```bash
|
||||
BARADB_MEMTABLE_SIZE_MB=32 \
|
||||
BARADB_CACHE_SIZE_MB=128 \
|
||||
BARADB_VECTOR_EF_CONSTRUCTION=100 \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
### Disk Full
|
||||
|
||||
```
|
||||
Error: No space left on device
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Check disk usage
|
||||
df -h
|
||||
|
||||
# Trigger compaction to reclaim space
|
||||
curl -X POST http://localhost:8080/api/admin/compact
|
||||
|
||||
# Or manually
|
||||
./build/baradadb --compact
|
||||
```
|
||||
|
||||
## Query Issues
|
||||
|
||||
### Syntax Error
|
||||
|
||||
```
|
||||
Error: Syntax error at position 15: unexpected token
|
||||
```
|
||||
|
||||
**Solution:** Check query syntax:
|
||||
|
||||
```sql
|
||||
-- Correct
|
||||
SELECT name, age FROM users WHERE age > 18;
|
||||
|
||||
-- Incorrect (missing comma)
|
||||
SELECT name age FROM users WHERE age > 18;
|
||||
```
|
||||
|
||||
### Table Not Found
|
||||
|
||||
```
|
||||
Error: Table 'users' does not exist
|
||||
```
|
||||
|
||||
**Solution:** Create the schema first:
|
||||
|
||||
```sql
|
||||
CREATE TYPE User {
|
||||
name: str,
|
||||
age: int32
|
||||
};
|
||||
```
|
||||
|
||||
### Type Mismatch
|
||||
|
||||
```
|
||||
Error: Cannot compare int32 with str
|
||||
```
|
||||
|
||||
**Solution:** Use correct types:
|
||||
|
||||
```sql
|
||||
-- Correct
|
||||
SELECT * FROM users WHERE age > 18;
|
||||
|
||||
-- Incorrect
|
||||
SELECT * FROM users WHERE age > '18';
|
||||
```
|
||||
|
||||
### Timeout
|
||||
|
||||
```
|
||||
Error: Query execution timeout
|
||||
```
|
||||
|
||||
**Solution:** Add LIMIT or optimize:
|
||||
|
||||
```sql
|
||||
-- Add limit
|
||||
SELECT * FROM large_table LIMIT 1000;
|
||||
|
||||
-- Use index
|
||||
SELECT * FROM users WHERE id = 123;
|
||||
```
|
||||
|
||||
## Connection Issues
|
||||
|
||||
### Connection Refused
|
||||
|
||||
```
|
||||
Connection refused: localhost:5432
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Check if server is running
|
||||
ps aux | grep baradadb
|
||||
|
||||
# Start server
|
||||
./build/baradadb
|
||||
|
||||
# Check firewall
|
||||
sudo ufw status
|
||||
sudo ufw allow 5432
|
||||
```
|
||||
|
||||
### Authentication Failed
|
||||
|
||||
```
|
||||
Error: Authentication failed
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Check JWT secret matches
|
||||
BARADB_AUTH_ENABLED=true \
|
||||
BARADB_JWT_SECRET="correct-secret" \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
```python
|
||||
client.authenticate("correct-jwt-token")
|
||||
```
|
||||
|
||||
### SSL/TLS Errors
|
||||
|
||||
```
|
||||
Error: TLS handshake failed
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Disable TLS for local testing
|
||||
BARADB_TLS_ENABLED=false ./build/baradadb
|
||||
|
||||
# Or provide valid certificates
|
||||
BARADB_TLS_ENABLED=true \
|
||||
BARADB_CERT_FILE=/path/to/cert.pem \
|
||||
BARADB_KEY_FILE=/path/to/key.pem \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### Slow Queries
|
||||
|
||||
**Diagnose:**
|
||||
|
||||
```bash
|
||||
# Check query plan
|
||||
curl -X POST http://localhost:8080/api/explain \
|
||||
-d '{"query": "SELECT * FROM large_table"}'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Add indexes:
|
||||
```sql
|
||||
CREATE INDEX idx_users_name ON users(name);
|
||||
```
|
||||
|
||||
2. Use LIMIT:
|
||||
```sql
|
||||
SELECT * FROM users LIMIT 100;
|
||||
```
|
||||
|
||||
3. Increase cache:
|
||||
```bash
|
||||
BARADB_CACHE_SIZE_MB=1024 ./build/baradadb
|
||||
```
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
**Causes:**
|
||||
- Compaction running
|
||||
- Large vector search without HNSW
|
||||
- Complex graph traversal
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Adjust compaction interval
|
||||
BARADB_COMPACTION_INTERVAL_MS=300000 ./build/baradadb
|
||||
|
||||
# Use approximate vector search
|
||||
SELECT /*+ APPROXIMATE */ * FROM vectors
|
||||
ORDER BY cosine_distance(embedding, [...])
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
**Monitor:**
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/metrics | grep memory
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Reduce memtable size
|
||||
BARADB_MEMTABLE_SIZE_MB=64
|
||||
|
||||
# Reduce cache
|
||||
BARADB_CACHE_SIZE_MB=256
|
||||
|
||||
# Limit HNSW graph size
|
||||
BARADB_VECTOR_M=8
|
||||
```
|
||||
|
||||
## Cluster Issues
|
||||
|
||||
### Raft Split-Brain
|
||||
|
||||
```
|
||||
Warning: Multiple leaders detected
|
||||
```
|
||||
|
||||
**Solution:** Ensure odd number of nodes (3, 5, 7). Restart minority partition.
|
||||
|
||||
### Replication Lag
|
||||
|
||||
```
|
||||
Warning: Replication lag > 10s
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Check network latency
|
||||
ping replica-node
|
||||
|
||||
# Increase replication threads
|
||||
BARADB_REPLICATION_THREADS=4 ./build/baradadb
|
||||
|
||||
# Switch to async replication
|
||||
BARADB_REPLICATION_MODE=async
|
||||
```
|
||||
|
||||
### Shard Imbalance
|
||||
|
||||
```
|
||||
Warning: Shard 3 has 2× data of others
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Trigger rebalancing
|
||||
curl -X POST http://localhost:8080/api/admin/rebalance
|
||||
```
|
||||
|
||||
## Data Corruption
|
||||
|
||||
### Checksum Mismatch
|
||||
|
||||
```
|
||||
Error: SSTable checksum mismatch
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Remove corrupted SSTable (data will be recovered from WAL)
|
||||
rm ./data/sstables/corrupted.sst
|
||||
|
||||
# Restart and recover
|
||||
./build/baradadb --recover
|
||||
```
|
||||
|
||||
### WAL Corruption
|
||||
|
||||
```
|
||||
Error: WAL segment corrupted
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Truncate WAL to last good segment
|
||||
./build/baradadb --truncate-wal --wal-sequence=15419
|
||||
|
||||
# Restore from snapshot if needed
|
||||
./build/baradadb --recover --checkpoint=/backup/snapshot.db
|
||||
```
|
||||
|
||||
## Debug Mode
|
||||
|
||||
Enable debug logging for detailed diagnostics:
|
||||
|
||||
```bash
|
||||
BARADB_LOG_LEVEL=debug \
|
||||
BARADB_LOG_FILE=/tmp/baradb_debug.log \
|
||||
./build/baradadb
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
If the issue persists:
|
||||
|
||||
1. Check logs: `tail -f /var/log/baradb/baradb.log`
|
||||
2. Check metrics: `curl http://localhost:8080/metrics`
|
||||
3. Run diagnostics: `./build/baradadb --diagnose`
|
||||
4. Open an issue with:
|
||||
- BaraDB version (`./build/baradadb --version`)
|
||||
- OS and architecture
|
||||
- Relevant log excerpts
|
||||
- Steps to reproduce
|
||||
+20
-9
@@ -15,11 +15,14 @@
|
||||
- [Installation](en/installation.md)
|
||||
- [Quick Start](en/quickstart.md)
|
||||
- [Architecture Overview](en/architecture.md)
|
||||
- [Configuration Reference](en/configuration.md)
|
||||
- [Deployment Guide](en/deployment.md)
|
||||
|
||||
### Core Concepts
|
||||
- [BaraQL Query Language](en/baraql.md)
|
||||
- [Storage Engines](en/storage.md)
|
||||
- [Schema System](en/schema.md)
|
||||
- [Cross-Modal Queries](en/crossmodal.md)
|
||||
|
||||
### Engines
|
||||
- [LSM-Tree Storage](en/lsm.md)
|
||||
@@ -29,25 +32,33 @@
|
||||
- [Full-Text Search](en/fts.md)
|
||||
- [Columnar Storage](en/columnar.md)
|
||||
|
||||
### Advanced
|
||||
- [Transactions & MVCC](en/transactions.md)
|
||||
- [Distributed Systems](en/distributed.md)
|
||||
- [Protocol Reference](en/protocol.md)
|
||||
- [User Defined Functions](en/udf.md)
|
||||
|
||||
### API Reference
|
||||
### API & Clients
|
||||
- [Client SDKs](en/clients.md)
|
||||
- [Binary Protocol](en/api-binary.md)
|
||||
- [HTTP/REST API](en/api-http.md)
|
||||
- [WebSocket API](en/api-websocket.md)
|
||||
- [Protocol Reference](en/protocol.md)
|
||||
|
||||
### Operations
|
||||
- [Performance & Benchmarks](en/performance.md)
|
||||
- [Security Guide](en/security.md)
|
||||
- [Monitoring & Observability](en/monitoring.md)
|
||||
- [Backup & Recovery](en/backup.md)
|
||||
- [Troubleshooting](en/troubleshooting.md)
|
||||
|
||||
### Advanced
|
||||
- [Transactions & MVCC](en/transactions.md)
|
||||
- [Distributed Systems](en/distributed.md)
|
||||
- [User Defined Functions](en/udf.md)
|
||||
|
||||
---
|
||||
|
||||
## Project Info
|
||||
|
||||
- [Contributing](../CONTRIBUTING.md)
|
||||
- [Changelog](en/changelog.md)
|
||||
- [License](../LICENSE)
|
||||
- [GitHub Repository](https://github.com/katehonz/barabaDB)
|
||||
|
||||
---
|
||||
|
||||
*To add a new language, create a new folder in `docs/` with the language code (e.g., `docs/de/`) and add a link above.*
|
||||
*To add a new language, create a new folder in `docs/` with the language code (e.g., `docs/de/`) and add a link above.*
|
||||
|
||||
Reference in New Issue
Block a user