Стабилизация на storage слоя — 6 фази
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Фаза 1: SSTable Integrity - SSTable v3 формат с CRC32 footer (data/index/bloom CRC) - Нов модул storage/crc32.nim (zero-dep, IEEE 802.3) - verifySSTable() — проверка на magic, version, CRC - loadSSTable strict mode — отхвърля корумпирани файлове - newLSMTree логва [WARN] при corrupt SSTables Фаза 2: baradb repair - Нов модул tools/repair.nim - Сканира SSTables, проверява CRC, премества битите в corrupt/ - WAL replay (CrashRecovery) за възстановяване на данни - CLI: ./baradadb repair --data-dir=... [--dry-run] Фаза 3: MANIFEST File - Atomic write MANIFEST (tmp + rename) - newLSMTree зарежда от MANIFEST, fallback към walkDir - checkStorageConsistency() — orphan/missing detection - flushUnsafe и compaction записват MANIFEST Фаза 4: WAL Rotation & Incremental Backup - WAL segment rotation при 64MB (wal_archive/wal.000NNN.log) - maybeRotate() на всеки 1000 записа + при flush - backup incremental — архивира MANIFEST + SSTables + WAL - CRC verification на SSTables преди архивиране Фаза 5: Online Consistent Backup - checkpoint() — freeze memtable + flush + WAL rotate - ./baradadb checkpoint — offline consistent snapshot - backup --online — checkpoint + incremental backup Фаза 6: SSTable Version Migration - SSTable.fileVersion поле - listLegacySSTables() — намира v1/v2 файлове - migrateSSTable() — пренаписва към v3 (tmp + rename) - ./baradadb migrate [--dry-run] — offline migration Документация: - Обновени docs/en/backup.md, docs/bg/backup.md - Обновени docs/en/storage.md, docs/bg/storage.md - Добавени тестове в tests/test_all.nim
This commit is contained in:
+102
-8
@@ -21,11 +21,105 @@ db.close()
|
||||
|
||||
- **MemTable**: In-memory sorted buffer
|
||||
- **WAL**: Write-ahead log for durability
|
||||
- **SSTable**: Sorted string tables on disk
|
||||
- **SSTable**: Sorted string tables on disk (v3 with CRC footer)
|
||||
- **Bloom Filter**: Probabilistic set membership
|
||||
- **Compaction**: Size-tiered strategy with level management
|
||||
- **MANIFEST**: Atomic catalog of active SSTables
|
||||
- **Page Cache**: LRU cache with hit rate tracking
|
||||
|
||||
### SSTable Format (v3)
|
||||
|
||||
```
|
||||
[Header] 36 bytes
|
||||
magic: uint32 (0x53535442 = "SSTB")
|
||||
version: uint32 (3 = current)
|
||||
entryCount: uint32
|
||||
level: uint32
|
||||
indexOffset: uint64
|
||||
bloomOffset: uint64
|
||||
footerOffset: uint64
|
||||
|
||||
[Data Block]
|
||||
keyLen: uint32
|
||||
key: bytes[keyLen]
|
||||
valueLen: uint32
|
||||
value: bytes[valueLen]
|
||||
timestamp: uint64
|
||||
deleted: uint8
|
||||
|
||||
[Index Block]
|
||||
keyLen: uint32
|
||||
key: bytes[keyLen]
|
||||
dataOffset: uint64
|
||||
|
||||
[Bloom Filter Block]
|
||||
bloomSize: uint32
|
||||
bloomData: bytes[bloomSize]
|
||||
|
||||
[Footer] 16 bytes
|
||||
dataCrc32: uint32 (CRC32 of Data Block)
|
||||
indexCrc32: uint32 (CRC32 of Index Block)
|
||||
bloomCrc32: uint32 (CRC32 of Bloom Block)
|
||||
reserved: uint32 (must be 0)
|
||||
```
|
||||
|
||||
The CRC footer enables independent verification of each SSTable file via `verifySSTable(path)`.
|
||||
|
||||
### MANIFEST Catalog
|
||||
|
||||
The `MANIFEST` file tracks all active SSTables atomically:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"sequence": 42,
|
||||
"createdAt": 1779103266,
|
||||
"sstables": [
|
||||
{"id": 1, "path": "sstables/1.sst", "level": 0, "minKey": "a", "maxKey": "z", "entryCount": 100}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- Written atomically via `MANIFEST.tmp` + rename
|
||||
- Read at startup for fast loading
|
||||
- Updated after every flush and compaction
|
||||
|
||||
### WAL Rotation
|
||||
|
||||
The Write-Ahead Log rotates when it reaches 64MB:
|
||||
|
||||
```
|
||||
wal/
|
||||
├── wal.log (active segment)
|
||||
└── wal_archive/
|
||||
├── wal.000001.log
|
||||
└── wal.000002.log
|
||||
```
|
||||
|
||||
Rotation triggers:
|
||||
- Every 1000 entries (lightweight check)
|
||||
- On every `flush()` or `checkpoint()`
|
||||
|
||||
### Storage Repair
|
||||
|
||||
Verify and repair storage integrity:
|
||||
|
||||
```bash
|
||||
# Check all SSTables
|
||||
./build/baradadb repair --data-dir=./data --dry-run
|
||||
|
||||
# Full repair
|
||||
./build/baradadb repair --data-dir=./data
|
||||
```
|
||||
|
||||
### SSTable Migration
|
||||
|
||||
Rewrite legacy v1/v2 SSTables to v3:
|
||||
|
||||
```bash
|
||||
./build/baradadb migrate --data-dir=./data
|
||||
```
|
||||
|
||||
## B-Tree Index
|
||||
|
||||
Ordered index for range scans and point lookups.
|
||||
@@ -43,14 +137,14 @@ let range = btree.scan("key_a", "key_z")
|
||||
|
||||
## Write-Ahead Log (WAL)
|
||||
|
||||
Ensures durability of write operations.
|
||||
Ensures durability of write operations with segment rotation.
|
||||
|
||||
```nim
|
||||
import barabadb/storage/wal
|
||||
|
||||
var wal = newWAL("./wal")
|
||||
wal.append("txn1", "SET key1 value1")
|
||||
wal.flush()
|
||||
var wal = newWriteAheadLog("./wal")
|
||||
wal.writePut(key, value, timestamp)
|
||||
wal.sync()
|
||||
```
|
||||
|
||||
## Bloom Filter
|
||||
@@ -73,6 +167,6 @@ Efficient file access using mmap.
|
||||
```nim
|
||||
import barabadb/storage/mmap
|
||||
|
||||
var mapped = mmapFile("./data/file.dat")
|
||||
let data = mapped.read(0, 100)
|
||||
```
|
||||
var mapped = openMmap("./data/file.dat")
|
||||
let val = mapped.readUint32(0)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user