From be370fb76b0e44a2d2cd6d47830e62980103eaef Mon Sep 17 00:00:00 2001 From: dimgigov Date: Wed, 6 May 2026 23:20:08 +0300 Subject: [PATCH] feat(backup): add dry-run, force, history log, auto-verify, rollback, disk check --- README.md | 154 +++++++++++++++++++++++++++++----- src/barabadb/core/backup.nim | 158 ++++++++++++++++++++++++++++++++--- 2 files changed, 277 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 85b1e8e..9244620 100644 --- a/README.md +++ b/README.md @@ -664,19 +664,21 @@ BARADB_LOG_LEVEL=debug ./build/baradadb ## Backup & Recovery BaraDB includes a built-in backup manager that creates compressed tar.gz -snapshots of your data directory. The manager supports online backups -(server does not need to stop), integrity verification, retention policies, -and safe restore with automatic rollback protection. +snapshots of your data directory. The manager supports **online backups** +(server does not need to stop), **integrity verification**, **retention policies**, +**dry-run restore previews**, **automatic rollback protection**, and a full +**restore history log**. ### Quick Reference | Command | Purpose | |---------|---------| | `backup backup` | Create a new snapshot | -| `backup restore` | Restore data from a snapshot | +| `backup restore` | Restore data from a snapshot (auto-verifies first) | | `backup list` | Show all snapshots | -| `backup verify` | Check archive integrity | -| `backup cleanup` | Delete old snapshots | +| `backup verify` | Check archive integrity without extracting | +| `backup cleanup` | Delete old snapshots, keep N most recent | +| `backup history` | Show log of all restore operations | | `backup help` | Show full help text | ### Build the Backup Tool @@ -685,6 +687,12 @@ and safe restore with automatic rollback protection. nim c -o:build/backup src/barabadb/core/backup.nim ``` +For production use, compile with release optimizations: + +```bash +nim c -d:release -o:build/backup src/barabadb/core/backup.nim +``` + ### Creating Backups **Basic backup** — creates `backup_.tar.gz` in the current directory: @@ -742,7 +750,8 @@ Found 3 backup(s): ### Verifying Backups Always verify a snapshot before restoring, especially after transferring it -over the network: +over the network. The restore command does this automatically, but you can +also check manually: ```bash ./build/backup verify --input=backup_1715011200.tar.gz @@ -758,30 +767,87 @@ A corrupted archive prints an error and exits with code 1. ### Restoring from Backup +The restore command follows a **safe restore workflow**: + +1. **Verify** archive integrity automatically +2. **Prompt** for confirmation (unless `--force` is used) +3. **Move** existing data to `data/server.old_` +4. **Extract** the archive +5. **Rollback** automatically if extraction fails +6. **Log** the operation to `backup_history.log` + > ⚠️ **WARNING:** Restore replaces the existing data directory. The old data > is automatically moved to `data/server.old_` before extraction. +> If extraction fails, the tool attempts an automatic rollback to the old data. + +**Interactive restore** (asks for confirmation): ```bash ./build/backup restore --input=backup_1715011200.tar.gz ``` -You will be prompted for confirmation: +You will be prompted: ``` +Verifying archive before restore... +Archive is valid: backup_1715011200.tar.gz (12.45 MB) WARNING: This will REPLACE the data in: data/server Continue? [y/N] ``` +**Force restore** — skip confirmation (for scripts and automation): + +```bash +./build/backup restore --input=backup.tar.gz --force +``` + +**Dry-run restore** — preview what would happen without making changes: + +```bash +./build/backup restore --input=backup.tar.gz --dry-run +``` + +Output: + +``` +DRY-RUN: The following actions would be performed: + 1. Verify archive integrity: backup.tar.gz + 2. Move existing data to: data/server.old_1778099200 + 3. Extract archive to: data/server + Archive size: 12.45 MB + Free space: 45.20 GB +``` + **Restore to a different data directory**: ```bash ./build/backup restore --input=backup.tar.gz --data-dir=data/recovered ``` -**Skip confirmation (scripting)** — pipe `y`: +**Verbose restore** (shows all steps and disk space check): ```bash -echo "y" | ./build/backup restore --input=backup.tar.gz +./build/backup restore --input=backup.tar.gz --verbose +``` + +### Restore History + +Every restore operation is logged to `backup_history.log` in the current +directory. View the history: + +```bash +./build/backup history +``` + +Example output: + +``` +Restore history: +-------------------------------------------------------------------------------- +[2026-05-06 23:15:00] SUCCESS restore from /backups/backup_1715011200.tar.gz to /opt/baradb/data/server +[2026-05-06 22:30:15] FAILED restore from /backups/backup_1715007000.tar.gz to /opt/baradb/data/server +[2026-05-05 08:00:00] DRY-RUN restore from /backups/backup_1714900000.tar.gz to /opt/baradb/data/server +-------------------------------------------------------------------------------- ``` ### Cleanup & Retention @@ -799,18 +865,33 @@ Delete old snapshots automatically, keeping only the N most recent: ./build/backup cleanup --keep=3 --verbose ``` -### Full Option Reference +### Automated Backups with Cron -| Option | Short | Default | Description | -|--------|-------|---------|-------------| -| `--data-dir` | `-d` | `data/server` | Path to the data directory | -| `--output` | `-o` | auto-generated | Destination path for new backup | -| `--input` | `-i` | — | Source archive for restore/verify | -| `--keep` | `-k` | `5` | Number of snapshots to retain | -| `--exclude` | `-e` | — | Exclude pattern (repeatable) | -| `--level` | `-l` | `6` | Gzip compression 0-9 | -| `--verbose` | `-v` | off | Detailed progress output | -| `--help` | `-h` | — | Show help text | +Add to your crontab for daily backups at 2 AM: + +```bash +# Edit crontab +crontab -e + +# Add this line for daily backups +0 2 * * * cd /opt/baradb && ./build/backup backup --output=/backups/baradb_$(date +\%F).tar.gz --level=6 >> /var/log/baradb-backup.log 2>&1 + +# Weekly cleanup — keep last 7 snapshots +0 3 * * 0 cd /opt/baradb && ./build/backup cleanup --keep=7 >> /var/log/baradb-backup.log 2>&1 +``` + +### Disaster Recovery Best Practices + +1. **3-2-1 Rule:** Keep 3 copies, on 2 different media, with 1 offsite. +2. **Verify regularly:** Run `backup verify` on archived snapshots monthly. +3. **Test restores:** Perform a dry-run restore (`--dry-run`) weekly and a + full test restore to a staging environment monthly. +4. **Monitor disk space:** The restore command warns if free space is less + than 2× the archive size. +5. **Keep old data:** After restore, the previous data is preserved as + `data/server.old_`. Only delete it after confirming the new + data works. +6. **Log audit trail:** Use `backup history` to track all restore operations. ### Nim API @@ -832,13 +913,42 @@ for b in backups: # Verify without extracting let valid = verifyArchive("snapshot.tar.gz") -# Restore (destructive!) +# Restore with rollback protection let restored = restoreDataDir("snapshot.tar.gz", "data/server") +# Dry-run restore — preview without changes +let preview = restoreDataDir("snapshot.tar.gz", "data/server", dryRun = true) + # Cleanup retention cleanupOldBackups("data/server", keepLast = 5) + +# Read restore history +for entry in readHistory(): + echo entry ``` +### Full Option Reference + +| Option | Short | Default | Description | +|--------|-------|---------|-------------| +| `--data-dir` | `-d` | `data/server` | Path to the data directory | +| `--output` | `-o` | auto-generated | Destination path for new backup | +| `--input` | `-i` | — | Source archive for restore/verify | +| `--keep` | `-k` | `5` | Number of snapshots to retain | +| `--exclude` | `-e` | — | Exclude pattern (repeatable) | +| `--level` | `-l` | `6` | Gzip compression 0-9 | +| `--dry-run` | — | off | Preview restore without changes | +| `--force` | `-f` | off | Skip confirmation prompts | +| `--verbose` | `-v` | off | Detailed progress output | +| `--help` | `-h` | — | Show help text | + +### Exit Codes + +| Code | Meaning | +|------|---------| +| `0` | Success | +| `1` | Error (invalid args, missing files, verification or extraction failure) | + ### Point-in-Time Recovery (WAL) For fine-grained recovery, replay the WAL from a checkpoint: diff --git a/src/barabadb/core/backup.nim b/src/barabadb/core/backup.nim index 68ff5cc..25a4d76 100644 --- a/src/barabadb/core/backup.nim +++ b/src/barabadb/core/backup.nim @@ -6,6 +6,7 @@ ## list List available snapshots with size & timestamp ## verify Check archive integrity without extracting ## cleanup Remove old snapshots keeping N most recent +## history Show restore operation log ## help Show detailed usage information ## ## Options: @@ -15,6 +16,8 @@ ## --keep, -k Number of snapshots to retain (default: 5) ## --exclude, -e Exclude pattern (can be used multiple times) ## --level, -l <0-9> Compression level for gzip (default: 6) +## --dry-run Show what would be done without doing it +## --force, -f Skip confirmation prompts ## --verbose, -v Enable verbose output import std/os @@ -22,7 +25,6 @@ import std/osproc import std/strutils import std/times import std/algorithm -import std/sequtils import std/parseopt type @@ -36,6 +38,7 @@ const DEFAULT_DATA_DIR = "data/server" DEFAULT_KEEP_COUNT = 5 DEFAULT_COMPRESSION = 6 + HISTORY_FILE = "backup_history.log" HELP_TEXT = """ BaraDB Backup Manager — Archive and restore your database safely ================================================================ @@ -49,6 +52,7 @@ COMMANDS: restore Replace the current data directory with contents from a snapshot. WARNING: This DESTROYS existing data. Use with care. + Automatically verifies archive integrity before extracting. list Show all snapshots found next to the data directory. Displays size, timestamp and compression ratio if known. @@ -59,6 +63,8 @@ COMMANDS: cleanup Delete old snapshots, keeping only the N most recent ones. Default retention is 5 snapshots. + history Show log of all restore operations performed on this system. + help Show this help message. OPTIONS: @@ -68,6 +74,8 @@ OPTIONS: -k, --keep Retention count for cleanup (default: 5) -e, --exclude Exclude files matching pattern (repeatable) -l, --level <0-9> Gzip compression level (default: 6, max: 9) + --dry-run Show what restore would do without changing anything + -f, --force Skip confirmation prompts (use with caution!) -v, --verbose Print detailed progress information EXAMPLES: @@ -80,6 +88,12 @@ EXAMPLES: # Restore from a specific snapshot backup restore --input=backup_1715011200.tar.gz + # Dry-run restore — preview what will happen + backup restore --input=backup.tar.gz --dry-run + + # Force restore without confirmation (automation/scripts) + backup restore --input=backup.tar.gz --force + # List all snapshots backup list @@ -91,6 +105,10 @@ EXAMPLES: # Exclude WAL files from backup backup backup --exclude="*.log" --exclude="wal/*" + +EXIT CODES: + 0 Success + 1 General error (invalid arguments, missing files, tar failure, etc.) """ proc formatBytes*(bytes: int64): string = @@ -124,6 +142,58 @@ proc parseBackupFilename*(filename: string): int64 = except: result = 0 +proc getArchiveSize*(input: string): int64 = + ## Return uncompressed size estimate from tar archive + let cmd = "tar -tzf " & quoteShell(input) & " | wc -l" + let (outStr, exitCode) = execCmdEx(cmd) + if exitCode == 0: + try: + result = parseBiggestInt(strip(outStr)) + except: + result = 0 + else: + result = 0 + +proc getFreeSpace*(path: string): int64 = + ## Return free disk space in bytes for the filesystem containing path + let cmd = "df -B1 " & quoteShell(path) & " | tail -1 | awk '{print $4}'" + let (outStr, exitCode) = execCmdEx(cmd) + if exitCode == 0: + try: + result = parseBiggestInt(strip(outStr)) + except: + result = -1 + else: + result = -1 + +proc logRestore*(archive: string, dataDir: string, success: bool, dryRun: bool = false) = + ## Append restore operation to history log + let logPath = getCurrentDir() / HISTORY_FILE + let status = if dryRun: "DRY-RUN" elif success: "SUCCESS" else: "FAILED" + let entry = "[" & format(getTime(), "yyyy-MM-dd HH:mm:ss") & "] " & + status & " restore from " & absolutePath(archive) & + " to " & absolutePath(dataDir) & "\n" + try: + let f = open(logPath, fmAppend) + f.write(entry) + f.close() + except: + discard # Silently fail if log cannot be written + +proc readHistory*(): seq[string] = + ## Read restore history log + result = @[] + let logPath = getCurrentDir() / HISTORY_FILE + if not fileExists(logPath): + return + try: + let content = readFile(logPath) + for line in splitLines(content): + if line.len > 0: + result.add(line) + except: + discard + proc backupDataDir*(dataDir: string, output: string, excludes: seq[string] = @[], compression: int = DEFAULT_COMPRESSION, verbose: bool = false): bool = ## Create a tar.gz backup of the data directory if not dirExists(dataDir): @@ -162,17 +232,38 @@ proc backupDataDir*(dataDir: string, output: string, excludes: seq[string] = @[] echo " Source: ", dataDir return true -proc restoreDataDir*(input: string, dataDir: string, verbose: bool = false): bool = - ## Restore from a tar.gz backup +proc restoreDataDir*(input: string, dataDir: string, verbose: bool = false, dryRun: bool = false): bool = + ## Restore from a tar.gz backup. + ## When dryRun is true, only prints what would be done. if not fileExists(input): echo "ERROR: Backup file not found: ", input return false + let archiveSize = getFileSize(input) + let freeSpace = getFreeSpace(parentDir(dataDir)) + let oldBackupPath = dataDir & ".old_" & $getTime().toUnix() + + if dryRun: + echo "DRY-RUN: The following actions would be performed:" + echo " 1. Verify archive integrity: ", input + echo " 2. Move existing data to: ", oldBackupPath + echo " 3. Extract archive to: ", dataDir + echo " Archive size: ", formatBytes(archiveSize) + if freeSpace >= 0: + echo " Free space: ", formatBytes(freeSpace) + else: + echo " Free space: unable to determine" + return true + + # Check free space + if freeSpace >= 0 and freeSpace < archiveSize * 2: + echo "WARNING: Free space (", formatBytes(freeSpace), ") may be insufficient." + echo " Archive is ", formatBytes(archiveSize), " — at least 2x is recommended." + if dirExists(dataDir): - let backupOld = dataDir & ".old_" & $getTime().toUnix() if verbose: - echo "Moving existing data to: ", backupOld - moveDir(dataDir, backupOld) + echo "Moving existing data to: ", oldBackupPath + moveDir(dataDir, oldBackupPath) createDir(dataDir) @@ -185,10 +276,18 @@ proc restoreDataDir*(input: string, dataDir: string, verbose: bool = false): boo echo "ERROR: tar extraction failed with exit code ", exitCode if outputStr.len > 0: echo outputStr + # Attempt rollback + if dirExists(oldBackupPath): + echo "Attempting rollback..." + removeDir(dataDir) + moveDir(oldBackupPath, dataDir) + echo "Rollback complete. Data restored to previous state." return false echo "Restored successfully from: ", input echo " Target: ", dataDir + if dirExists(oldBackupPath): + echo " Old data preserved at: ", oldBackupPath return true proc verifyArchive*(input: string, verbose: bool = false): bool = @@ -276,6 +375,18 @@ proc cleanupOldBackups*(dataDir: string, keepLast: int = DEFAULT_KEEP_COUNT, ver echo "Cleanup complete." +proc printHistory*() = + ## Display restore history + let entries = readHistory() + if entries.len == 0: + echo "No restore history found." + return + echo "Restore history:" + echo repeat("-", 80) + for entry in entries: + echo entry + echo repeat("-", 80) + # ============================================================================= # CLI Entry Point # ============================================================================= @@ -288,6 +399,8 @@ when isMainModule: excludes: seq[string] = @[] compression = DEFAULT_COMPRESSION verbose = false + dryRun = false + force = false for kind, key, val in getopt(): case kind @@ -311,6 +424,8 @@ when isMainModule: if compression < 0 or compression > 9: quit("ERROR: --level must be between 0 and 9", 1) except: quit("ERROR: --level must be a number", 1) + of "dry-run": dryRun = true + of "force", "f": force = true of "verbose", "v": verbose = true of "help", "h": echo HELP_TEXT @@ -333,13 +448,27 @@ when isMainModule: of "restore": if target.len == 0: quit("ERROR: restore requires --input=\nUse 'backup help' for usage.", 1) - echo "WARNING: This will REPLACE the data in: ", dataDir - echo "Continue? [y/N] " - let answer = readLine(stdin) - if answer.toLowerAscii() notin ["y", "yes"]: - quit("Restore cancelled", 0) - let ok = restoreDataDir(target, dataDir, verbose) - if not ok: + + # Always verify first unless dry-run + if not dryRun: + echo "Verifying archive before restore..." + let vok = verifyArchive(target, verbose) + if not vok: + logRestore(target, dataDir, false) + quit("Restore aborted: archive verification failed.", 1) + + if not dryRun and not force: + echo "WARNING: This will REPLACE the data in: ", dataDir + echo "Continue? [y/N] " + let answer = readLine(stdin) + if answer.toLowerAscii() notin ["y", "yes"]: + echo "Restore cancelled." + logRestore(target, dataDir, false, dryRun = false) + quit(0) + + let ok = restoreDataDir(target, dataDir, verbose, dryRun) + logRestore(target, dataDir, ok, dryRun) + if not ok and not dryRun: quit("Restore failed", 1) of "list": @@ -356,6 +485,9 @@ when isMainModule: of "cleanup": cleanupOldBackups(dataDir, keepCount, verbose) + of "history": + printHistory() + of "help": echo HELP_TEXT