Backup Verification and Partial Restore

• Type: Runbook
• Status: Current
• Scope: Backrest/restic snapshots, Electra Btrfs snapshots, and staged recovery of individual paths
• Canonical sources: justfiles/maintenance.just; modules/services/backrest.nix; modules/services/btrbk-home.nix; modules/services/postgresql-backup.nix; composed nixosConfig.backup options; Storage and Data Map
• Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); composed backup options and Electra status/snapshot commands checked 2026-09-12
• Review triggers: Backup-path, exclude, schedule, endpoint, credential, restore-recipe, Btrbk, logical-dump, or NAS policy changes; new service state; completed restore exercises

Purpose

Confirm that a required path is in a recent usable snapshot, restore a file or directory into a separate staging location, inspect it, and only then decide whether to replace live data. For a lost host or a full state restore, use Full-Host Recovery. The old whole-root restore recipe remains unsafe pending issue #226.

Prerequisites

• Work in the nixos-config checkout on the host whose snapshot you need. The backup-ls and backup-restore-path recipes select $(hostname).
• Identify the exact original absolute path, snapshot time, and a staging target with sufficient free space. Keep the target away from the live data.
• Have working sops-nix Backrest credentials and a route to the NAS repository for restic operations. Electra's local Btrfs snapshots need neither.
• Check the Storage and Data Map for declared coverage and gaps. A path outside the plan or on a nested NFS filesystem is not rescued merely because an ancestor directory is listed.
• For a service-owned path, know the unit, its writers, and how the service will be stopped or quiesced before any eventual live replacement.

Safety boundary

The listing and status commands here do not modify backups. The restore helpers write files: use a fresh, non-live staging target and inspect its contents, ownership, size, and integrity before any separate promotion. Restoring a database directory or active application state over a running service is outside this procedure.

just backup-restore is not the partial-restore command. It selects the host's latest restic snapshot, targets /, and uses --overwrite always. The current recipe invokes restic as the caller despite root-owned paths in current plans; do not use it for an in-place repair. Forgejo issue #226 tracks that privilege boundary, the CLI/daemon endpoint mismatch, and failed staging cleanup. The full-host runbook must reverify the supported recovery route.

Restored data and logs may contain credentials or personal content. Do not paste raw output into an issue or memory. A successful restic snapshots listing is not evidence that a particular path can be read back.

Procedure

1. Establish declared and actual coverage

Check the owning host and application paths in the Storage and Data Map. The effective Backrest plan selects the primary user's home when enabled, plus declared nixosConfig.backup.dataPaths, then deduplicates child paths beneath a selected parent. It uses --one-file-system and excludes replaceable caches, dependencies, and model data. All six registered hosts currently have a Backrest plan.

Do not infer that a live database directory is safely snapshotted simply because another host backs up /var/lib/docker or /srv/services. Check the application-specific recovery artefact.

2. Check the backup process and snapshot recency

On the affected host:

hostname
date --iso-8601=seconds
just backup-status
just backup-snapshots

backup-status reads backrest.service; backup-snapshots lists restic snapshots filtered to the local hostname. To compare multiple hosts from an authorised machine, use just backup-snapshots-all or just backup-snapshots HOSTNAME. The recipes read SOPS-backed credentials from /run/secrets; they do not print them, but avoid tracing their shell execution or sharing process/environment output.

Compare the newest snapshot time with the last successful scheduled backup and the incident window. Backrest's configured plan runs at most every four hours when reachable, skips unchanged content, and retains daily/weekly/monthly buckets. A running daemon and a configured schedule do not establish that the last snapshot completed or included the requested path.

For bounded context:

just backup-logs

Review logs locally before sharing: they may expose filenames or error text.

3. Inspect the selected restic snapshot

Choose a particular snapshot ID from the list when reproducibility matters. The latest alias may select a newer point if another backup completes while you investigate.

just backup-ls SNAPSHOT_ID /home/nimmo/path/to/item

Substitute the actual absolute path and host-specific snapshot. Confirm the expected file or subtree exists and that its snapshot time precedes the data loss or corruption. An older snapshot may be the correct one for an accidental edit.

The restic CLI helpers currently use a hard-coded NAS LAN endpoint, whereas the Backrest daemon uses mass-storage.mesh:8008. A working daemon does not prove that the CLI works from your current network. If the helper cannot reach the repository, diagnose routing and use a trusted network path; do not change the repository URL or handle credentials ad hoc during an incident. Issue #226 owns the configuration repair.

4. Restore into a fresh staging directory

For a small path whose permissions allow the current user to write the staged content, use:

just backup-restore-path SNAPSHOT_ID /home/nimmo/path/to/item /tmp/recovery-SNAPSHOT_ID

The target must not already exist. Restic recreates the original absolute path under that target: the example item appears below /tmp/recovery-SNAPSHOT_ID/home/nimmo/path/to/item. Use a unique target name and sufficient capacity. The helper creates the directory before restic runs; if it fails, preserve the error and treat that directory as a possibly partial restore. Inspect it before selecting a new target for a retry. Never assume an incomplete target is valid.

Root-owned service data may not restore fully as a normal user. In that case, stop and plan an authorised staged restore with correct privileges and service quiescence; do not turn this command into a blind overwrite of the live path.

Compare the staged file against the expected type, size, owner, mode, and application-specific format. Where appropriate, verify a checksum or open a copy in a disposable environment. Only a verified read-back is evidence of restore usability.

5. Use Electra's local snapshots for recent home loss

Electra alone has an hourly Btrbk timer for /home, with a local snapshot directory and a second-disk replica. They are faster than NAS retrieval but both remain on the same laptop:

just btrfs-snapshots
just btrfs-snapshot-usage

Choose the exact home.YYYYMMDDTHHMM entry and a path beneath /home:

just btrfs-restore SNAPSHOT_NAME /home/nimmo/path/to/item

The helper searches the local snapshot first, then the second-disk replica, and copies into a new path beneath /tmp/restore-SNAPSHOT_NAME/ by default. It refuses an existing target and uses sudo cp -a --reflink=auto, so the staged result can be root-owned. For a named target, choose a new path away from live data. Do not pass untrusted .. components, a /home path outside the intended tree, or a live destination: the helper does not canonicalise paths or perform a dry run.

Confirm btrbk-home.timer and the most recent service result if recency is surprising:

systemctl list-timers btrbk-home.timer --all --no-pager
sudo journalctl -u btrbk-home.service -b -n 50 --no-pager

A local snapshot or second-disk replica does not replace off-host backup.

6. Verify logical database artefacts independently

For PostgreSQL, the dedicated daily postgresql-backup.timer produces a timestamped directory containing globals.sql.zst, a per-database compressed SQL file, and manifest.tsv. A complete set is staged outside its published backup path and moved into place only after dumps complete.

Cosmos publishes completed sets under /var/backups/postgresql and includes that path in Backrest. Vega publishes under /mnt/storage/postgresql-backups, but that directory is not currently selected by Backrest; those local dumps are not an off-host recovery copy. Check timer, journal, directory timestamp, manifest, and the specific database file on the owning host. Use a disposable PostgreSQL instance for a real restore exercise; listing compressed files alone does not prove they import successfully.

BookStack's declared file backup excludes its MariaDB data and has no corresponding logical dump in this repository. Do not report a complete BookStack restore on the strength of the application directory snapshot.

7. Promote only after verification

Record the source snapshot ID and timestamp, original path, staged path, verification performed, and any exclusions or permission failures. For ordinary documents, copy a verified staged item back only after checking whether a newer live version exists. Preserve the current live copy separately when possible.

For service state, databases, or broad directory replacements, prepare an issue/runbook with a maintenance window, service quiescence, explicit owner and mode requirements, application-level validation, and rollback. This page does not authorise replacing a live data directory.

Success checks

• The selected snapshot belongs to the intended host and precedes the loss.
• The exact requested path is present, is not excluded or hidden behind a separate filesystem, and restores completely into a fresh staging target.
• The restored object passes appropriate content/format verification, not just a listing check.
• Original live data was not overwritten during verification.
• Recovery evidence records snapshot ID, timestamp, path, result, and any missing backup layer; sensitive contents are not published.
• Where a live promotion occurred under its owning procedure, the real application/client check and a subsequent backup succeed.

Rollback and recovery

Read-only checks need no rollback. For a failed staged restore, leave the partial target untouched until its state is understood, then use a new target for another attempt. Do not delete an unknown directory by glob or reuse it as if the restore had completed.

If a live path was mistakenly overwritten, stop its writers, retain the current state as evidence, and recover from the known-good snapshot or application-specific artefact. Escalate database and full-host cases to Full-Host Recovery. Nix generations do not roll back mutable application data.

Troubleshooting

• No recent snapshot: check Backrest status/logs, NAS route, credential availability, and plan coverage. skipIfUnchanged can also leave the newest useful snapshot older than a polling cycle.
• Path absent: inspect the actual Backrest path list, excludes, and filesystem boundary before assuming retention deleted it.
• Credentials unavailable: inspect sops-install-secrets.service and the declared secret paths and metadata; never print their contents. Confirm the host is a legitimate recipient under ADR-006.
• Repository locked: determine whether another backup or maintenance job still owns the lock. just backup-unlock changes repository state; use it only after proving the lock is stale.
• Btrfs snapshot missing: check both snapshot directories and the timer. Move to off-host restic for a lost disk or expired local retention.
• Service data staged with wrong permissions: do not promote it; arrange a privileged, isolated restore through the owning service procedure.

Related material

• Data authority and exact declared gaps: Storage and Data Map
• Whole-host recovery: Full-Host Recovery
• Host installation before restore: Host Provisioning
• Service investigation: Monitoring and Service Investigation