nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions
Page: 36 Full Host Recovery
Pages
00 Documentation Map and Scope 00 Documentation Standards 00 Start Here and Governance 10 System Atlas 11 Fleet Overview 12 Host Inventory 13 Service Catalogue 14 Network and Access Topology 15 Storage and Data Map 16 Automation and Control Plane 20 Architecture Decisions 21 ADR 001 Electra Hardware Specialisations 22 ADR 002 Workload Placement 23 ADR 003 Network Exposure 23 Project Tracking 24 ADR 004 Engram Durable Agent Memory 24 Security Baseline 25 ADR 005 Auto Update Control Plane 26 ADR 006 Scoped Secret Files 30 Operations and Runbooks 31 Routine Fleet Operations 32 Safe Testing and Deployment 33 Host Provisioning 34 Monitoring and Service Investigation 35 Backup Verification and Partial Restore 36 Full Host Recovery 37 Electra Boot Trust Recovery 38 Secret Editing and Recipient Rotation 39 Auto Update Incident Response 39a Service Migration and Cutover 39b Wrapper Repository Packaging 40 Engineering Handbook 41 Repository Structure and Import Chain 42 Configuration Ownership Boundaries 43 Adding Packages and Flake Inputs 44 Adding a Module or Profile 45 Adding a Host 46 Adding or Migrating a Service 47 Testing and CI Contracts 48 Secrets Implementation Conventions 49 Documentation Maintenance 50 NixOS Learning Path 51 Guided Repository Tour 52 Nix Language Essentials 53 Declarative Configuration Store and Generations 54 Flakes and Inputs 55 NixOS Modules and Option Merging 56 SpecialArgs and Home Manager Argument Flow 57 Electra Specialisations Worked Example 58 Nix Troubleshooting Primer 90 Historical Material 91 Electra Secure Boot Hardening Record Home
3 36 Full Host Recovery
Nimmo edited this page 2026-09-13 13:14:49 +01:00
Table of contents

Full-Host Recovery

  • Type: Runbook
  • Status: Current
  • Scope: Reconstructing a registered NixOS host after disk or host loss, and restoring its authoritative local state
  • Canonical sources: hosts/*/disko.nix; flake.nix; justfiles/hosts.just; justfiles/maintenance.just; modules/services/backrest.nix; modules/services/postgresql-backup.nix; composed nixosConfig.backup options; Storage and Data Map
  • Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); recovery boundaries reviewed against source 2026-09-12. No end-to-end full-host restore is claimed.
  • Review triggers: Disko, backup plan or endpoint, restore recipe, SOPS recipient, service-state, database dump, boot trust, or recovery-test changes

Purpose

Recover a failed machine without mistaking a rebuilt NixOS configuration for restored data. This is a decision and verification sequence, not a single command that safely writes an entire snapshot over /. The exact files to restore depend on the host, the incident, and the most recent usable backup.

For one lost file or a limited directory, start with Backup Verification and Partial Restore. For a still bootable host, prefer a normal generation rollback or targeted repair under Safe Testing and Deployment before any destructive reinstallation.

Prerequisites and stop conditions

  • Identify the failed host and its currently authoritative services in the Host Inventory, Service Catalogue, and Storage and Data Map. Do not assume every named database or mount is the active authority.
  • Have physical/virtual console or rescue access, a reviewed Git revision for the host, and independent access to the provisioning age key and backup credentials. Keep private material out of commands, issues, and logs.
  • Establish what must survive, which copies are local, off-host, or external, and the selected recovery point. A configured timer or a restic snapshot listing alone does not prove a usable restore.
  • Record the exact target machine and block-device identities before any Disko operation. Repartitioning has no Nix-generation rollback.
  • For Electra, confirm independent LUKS passphrases, recovery media, firmware access, and the encrypted Secure Boot archive before boot-trust work.
  • Open or update the incident's Forgejo issue with scope, recovery point, service owners, missing data, checks, and rollback. Reprovisioning/disk work requires the risk review and checkpoint in Project and Change Governance.

Stop before formatting if the disk identity, backup coverage, or independent credentials are uncertain. Electra's current Disko create/format path has not been validated from scratch; Host Provisioning requires a scratch-disk or VM test before using that path for a real Electra rebuild. Do not turn a working data disk into a test of the recovery plan.

Recovery sequence

1. Stabilise and classify the incident

Stop competing writers where practicable and preserve the failed disk or VM image until a recovery point is chosen. Distinguish configuration/boot failure, loss of a particular path, corruption of application state, and physical disk loss. If the original data remains readable, take a separate copy before a restore; an older snapshot may erase newer valid work.

Check a known-good NixOS generation and the targeted restore path first when the host remains intact. Nix generations recover declarative configuration, not mutable databases or application files. A full rebuild is justified only after the less destructive route is ruled out or the storage is gone.

2. Build a host-specific recovery manifest

For each authoritative data set, record its original path, filesystem and owner, backup mechanism, exact snapshot or dump timestamp, expected gap, and validation method. Use the current Storage and Data Map, then confirm the snapshot actually contains the path by following Backup Verification and Partial Restore. Restic's Backrest plan uses --one-file-system; a listed parent does not include nested NFS mounts.

Current declared coverage is materially different by host:

Host Recovery starting point Important limit
Electra Primary home in off-host Backrest; local Btrbk home snapshots if both laptop disks survive Model/game stores and independent boot keys are not recovered by the home snapshot; Disko fresh format remains untested
Lena Primary user's home in Backrest Claire's separate home is not selected
Vega Home, /srv/services, /srv/stacks, and selected application roots in Backrest PostgreSQL logical dumps at /mnt/storage/postgresql-backups are local-only; BookStack MariaDB and Engram Cloud non-database state lack declared coverage
Lyra Home and selected local Tdarr, Jellyfin, and Redview state in Backrest /srv/stacks, Docker root, models, and NAS-mounted media are not copied by that plan
Cosmos Home and PostgreSQL logical dump sets in Backrest Do not restore the live PostgreSQL cluster as a substitute for a logical import
Chaos Home, Docker root, and /srv/stacks in Backrest A broad Docker-root snapshot still needs service-specific consistency checks

NAS media, Immich libraries, and other NFS data are owned outside this host's restic plan. Verify the external authority and its own recovery mechanism; do not treat an empty mountpoint as a successful restore. If required state has no usable copy, mark the resulting service as not recovered rather than silently declaring success.

3. Rebuild only the declarative base

Use Host Provisioning for the reviewed disko.nix, nixos-anywhere, first boot, SSH host-key/SOPS recipient registration, and retirement of the bootstrap age key. That page owns the destructive commands; retired manual provisioning examples in wiki Git history are not current procedures.

Where a separate data disk or external mount should survive, keep it out of the format target and verify its identity and mount policy before attaching services. On Vega, /mnt/storage and /mnt/partial are managed outside Disko; they need explicit preservation or separate recovery. Electra's Secure Boot signing keys and TPM tokens are not created by Disko. Use the dedicated Electra Boot-Trust and TPM Recovery only after independently validating the intended action and hardware targets.

At first boot, verify hostname, SSH access, storage and network mounts, SOPS decryption, and that the deployed revision matches the reviewed one. Keep application writers stopped until their state has been staged and checked.

4. Restore state in a separate staging area

Choose an explicit snapshot ID and restore selected paths to a fresh non-live target with enough capacity. Inspect content, ownership, mode, timestamps, completeness, and application format before any promotion. The partial-restore runbook documents the existing backup-restore-path helper and its limits; root-owned service state requires an authorised privileged method and a target-specific plan.

Do not run just backup-restore against a live /. The current recipe selects latest, targets /, and uses --overwrite always as the invoking user, even though Backrest includes root-owned service paths. It can produce a partial, mixed-state recovery. Its CLI repository endpoint also differs from the Backrest daemon's mesh endpoint. Forgejo issue #226 tracks these defects. Until a tested full-host method exists, restore each data class under a documented, service-specific change plan; do not improvise a root-target variant of that recipe during an incident.

For PostgreSQL, recover a complete timestamped logical dump set: the manifest.tsv, globals.sql.zst, and per-database .sql.zst files. Cosmos's dump directory is included in Backrest; Vega's is currently only on its storage disk. Check the manifest and restore into a disposable cluster before promoting databases. Resolve roles, database versions, service dependencies, and the owning application before any live import. BookStack files alone do not recover its MariaDB database. Docker state and other live database files must be quiesced and checked for application consistency before copying back.

5. Promote, reconnect, and validate each service

For every staged data set, stop its writers, preserve the current live path where feasible, install the verified copy with correct ownership and modes, and start the owning service. Validate with its actual user-facing read/write path, not only systemctl active. Reattach external mounts only after confirming they point at the intended NAS exports; do not let a service write to an empty local mountpoint while the NAS is absent.

Check systemctl --failed, the relevant service logs, container health, DNS and routing, and the expected application clients. Check whether the restored host's SSH identity changed and that SOPS decrypts through its permanent recipient before deleting provisioning material. If a path or database is unrecoverable, leave its service stopped or isolated and record the gap.

6. Re-establish protection

Confirm the restored host runs Backrest and that a new snapshot completes. Then verify a representative path can be read back from that snapshot. Check PostgreSQL's next logical dump set on hosts that use it. Repeat the normal deployment/test and no-op follow-up checks appropriate to the change. Record the final revision, disk identity, selected backup IDs, restored paths, missing state, application checks, and remaining risks in the incident issue; never attach raw secrets or personal backup contents.

Success checks

  • The host boots its reviewed NixOS revision and can be administered through its permanent identity; no temporary provisioning key remains.
  • Every required filesystem and external mount resolves to the intended device/export, and services cannot silently write to missing mountpoints.
  • Each authoritative data set has a recorded recovery point and verified application result, or is explicitly classified as not recovered.
  • Required services work through real clients; failed units and logs have been reviewed, and a fresh backup and read-back succeed.
  • Any Electra Secure Boot/TPM work has its own verified recovery evidence.

Rollback and unresolved limits

If a staged restore fails, retain its incomplete target for diagnosis and retry into a new target. If promotion fails, stop writers and restore the pre-promotion copy or the selected known-good snapshot; a Nix generation rollback cannot undo mutable-data changes. If new storage was formatted, rollback means reconstructing from independent copies, not reverting Git.

The current repository does not prove an end-to-end full-host restore. Issue #226 blocks recommending the all-in-one root restore helper. The declared coverage gaps above, Electra Disko fresh-format test, and the open Electra boot-trust archive work must be resolved or explicitly accepted for the particular incident before claiming full recovery.

Related material