Disaster Recovery

Status: reference — reviewed 2026-06-29 after electra's Secure Boot and managed TPM hardening. All hosts carry a disko.nix. Electra's data-disk sizing is valid for the format path and TPM enrolment uses stable partlabels. Remaining caveat: electra's disko create/format path has still never been exercised — validate it on a scratch disk/VM before trusting a from-scratch electra reprovision (see step 1).

How to recover a machine from scratch after a wipe, hardware failure, or lost drive. The goal is to reach the state of the last Backrest snapshot, with the full NixOS configuration restored from git.

What is and isn't covered

Recovered automatically:

• NixOS system configuration (from git)
• All files under /home/nimmo at the time of the last snapshot
• Applications, packages, dotfiles, documents, projects
• Any extra paths added to the backup plan via nixosConfig.backrest.extraPaths

Not recovered (intentionally excluded from backups):

• Steam games — re-download from Steam
• LLM model weights (.ollama, ModelFiles, etc.) — re-download
• Build caches, node_modules, cargo registry — regenerated on use
• Browser cache, shader cache — regenerated automatically

Not recovered (outside home, not in git):

• /var/lib/ service state for any locally-hosted services
• /srv/services NixOS-managed service state on server hosts
• /srv/stacks Docker Compose stack state on server hosts
• Any manual changes made outside the NixOS configuration

Server service state is intentionally being moved toward /srv/services as Docker stacks are converted to NixOS services. Treat it as application data: back it up or document it per service before relying on a migrated service.

---

Prerequisites

Before you need to recover, make sure these are in place:

• Provisioning age key exists in your password manager (just setup-provision-key — one-time setup, see README)
• Host configuration is committed to git, including a disko.nix for the disk layout. All hosts now have one (electra, lena, vega, lyra). electra is the only LUKS-encrypted host; its in-place disko adoption was validated end-to-end in a throwaway QEMU VM before deploying. Reprovisioning electra additionally requires re-establishing TPM2 auto-unlock — see step 1a.
• NAS is reachable (backrest runs on 192.168.8.238:8008)
• Electra recovery secrets are held independently of electra: LUKS passphrase, TPM PIN, firmware administrator password, and Secure Boot backup passphrase.
• Electra Secure Boot archive contains the encrypted /var/lib/sbctl key bundle, LUKS headers, pcrlock state, and verification output. Store it off-host and record its location in the password manager.

---

Electra Secure Boot backup

Refresh this backup after changing Secure Boot keys, LUKS keyslots or TPM tokens. The private Secure Boot keys and LUKS headers are sensitive: encrypt the archive before it leaves electra, never commit it, and keep it separate from the archive passphrase.

The backup should contain:

• /var/lib/sbctl, including PK, KEK, db keys and owner GUID
• LUKS2 headers for root, data, and swap
• /var/lib/systemd/pcrlock.json and /var/lib/pcrlock.d
• an ESP archive containing the signed boot artifacts
• status output for Secure Boot, signatures, pcrlock, and all LUKS tokens

The pcrlock files are diagnostic state, not a substitute for the TPM. After a TPM clear or motherboard replacement, regenerate the managed policy in IGPU mode and reenrol both token types.

Record the archive locations and creation date in the password manager. Test decryption and archive listing before deleting temporary LUKS-header files.

Run the following from electra. Set BACKUP_DIR to a mounted off-host destination that is writable by the current user:

cd ~/nixos-config
umask 077
stamp=$(date +%Y%m%d-%H%M%S)
work=$(mktemp -d)
BACKUP_DIR=/path/to/off-host/electra-secure-boot
mkdir -p "$BACKUP_DIR"

sudo cryptsetup luksHeaderBackup \
  /dev/disk/by-partlabel/electra-cryptroot \
  --header-backup-file "$work/cryptroot.luks-header"
sudo cryptsetup luksHeaderBackup \
  /dev/disk/by-partlabel/electra-cryptdata \
  --header-backup-file "$work/cryptdata.luks-header"
sudo cryptsetup luksHeaderBackup \
  /dev/disk/by-partlabel/electra-cryptswap \
  --header-backup-file "$work/cryptswap.luks-header"

{
  date --iso-8601=seconds
  cat /etc/nixos-specialisation
  sudo bootctl status
  sudo sbctl status
  sudo sbctl verify
  systemctl is-enabled systemd-pcrlock-make-policy.service
  sudo jq '{pcrBank, pcrValues, nvIndex}' \
    /var/lib/systemd/pcrlock.json
  for device in \
    /dev/disk/by-partlabel/electra-cryptroot \
    /dev/disk/by-partlabel/electra-cryptdata \
    /dev/disk/by-partlabel/electra-cryptswap
  do
    echo "===== $device ====="
    sudo systemd-cryptenroll "$device"
  done
} > "$work/status.txt" 2>&1

sudo tar -czf - \
  -C / \
  var/lib/sbctl \
  var/lib/systemd/pcrlock.json \
  var/lib/pcrlock.d \
  -C "$work" \
  cryptroot.luks-header \
  cryptdata.luks-header \
  cryptswap.luks-header \
  status.txt \
  | age -p -o "$BACKUP_DIR/electra-secure-state-$stamp.tar.gz.age"

sudo tar -C /boot -czf - . \
  | age -p -o "$BACKUP_DIR/electra-esp-$stamp.tar.gz.age"

Use a strong archive passphrase stored in the password manager. Verify both archives before cleanup:

age -d "$BACKUP_DIR/electra-secure-state-$stamp.tar.gz.age" \
  | tar -tzf - >/dev/null
age -d "$BACKUP_DIR/electra-esp-$stamp.tar.gz.age" \
  | tar -tzf - >/dev/null

sudo rm -r -- "$work"

Copy the encrypted archives to a second independent off-host location if the first destination is not itself backed up.

---

Recovery steps

1. Provision the machine

Boot the target into a NixOS installer or rescue environment with SSH access, then run from another machine in the repo:

just provision HOSTNAME TARGET_IP KEY=AGE-SECRET-KEY-...

Retrieve the provisioning age private key from your password manager and pass it as the KEY argument. nixos-anywhere will partition the disk, install NixOS, and reboot.

electra caveat (untested format path): electra's disko.nix was validated only for in-place adoption — generating the running config against the existing disks. disko's create/format path (what a fresh reprovision runs) has never been exercised for electra. The data-disk sizing is now arranged correctly for it (cryptswap has priority = 1 so the fixed swap is allocated before cryptdata's 100%), but the path itself is still unrun — validate it on a scratch disk / VM before relying on a from-scratch electra reprovision. The other hosts (lena, vega, lyra) were born from disko and are fine.

The provisioning key allows sops-nix to decrypt secrets (including the Backrest credentials) during first activation — before the host SSH key has been generated and registered. The key is kept only in your password manager, independent of the sops trust chain, so it remains available even if all registered machines are lost.

1a. (electra only) Re-establish TPM2 auto-unlock

electra is LUKS-encrypted. disko/nixos-anywhere create the LUKS containers with a passphrase; Secure Boot keys and TPM tokens are not part of the disko spec.

If firmware trust was reset, boot recovery media, unlock with the LUKS passphrase, and restore /var/lib/sbctl from its encrypted archive. Keep Secure Boot disabled or in Setup Mode while restoring, then rebuild the signed artifacts:

sudo nixos-rebuild boot --flake .#electra
sudo sbctl verify
sudo sbctl enroll-keys --microsoft

Re-enable firmware Secure Boot, confirm user mode with bootctl status, and verify firmware administrator access before proceeding.

Boot IGPU and confirm systemd-pcrlock-make-policy.service succeeds with PCRs 4 and 7. If the TPM was cleared or replaced, remove stale pcrlock policy metadata, rebuild in IGPU mode to create a fresh TPM NV policy, then reenrol each volume one at a time:

just tpm2-enrol root
just tpm2-enrol data
just tpm2-enrol swap
just tpm2-enrol-pin root
just tpm2-enrol-pin data
just tpm2-enrol-pin swap

tpm2-enrol references the LUKS containers by their disko partlabels (/dev/disk/by-partlabel/electra-crypt{root,data,swap}), which are stable across a fresh reprovision. It migrates one volume at a time against Lanzaboote's managed pcrlock policy, preserving the passphrase as an independent recovery method. Run the managed-policy rebuild and enrolment in IGPU mode. The tpm2-enrol-pin commands add the PCR 7 + PIN fallback used by DGPU mode.

Restore a LUKS header only when the header or keyslot metadata is damaged. Header restoration overwrites current keyslots and tokens: boot recovery media, verify the exact target device by partlabel and serial, and restore only its matching backup with cryptsetup luksHeaderRestore.

2. Register the new SSH host key

After the machine reboots:

just add-secret-remote HOSTNAME TARGET_IP

This converts the new host's SSH key to an age key, adds it to .sops.yaml, updates the relevant scoped secret files, and commits. The machine can now decrypt its assigned secrets directly via its SSH host key on future boots.

3. Restore home directory from backup

SSH into the recovered machine and run:

just backup-restore

This will:

1. Show the latest snapshot for this host
2. Ask for confirmation
3. Restore all backed-up files to their original paths under /

The restore reads credentials from /run/secrets/ (populated by sops-nix at boot). If sops-nix hasn't activated yet, reboot first.

4. Verify

Check that key files and directories are present:

ls ~/Documents ~/Projects ~/.ssh ~/.config

Trigger a fresh backup to confirm the restored machine is now backing up:

• Open http://localhost:9898 in a browser (Backrest UI)
• Go to Plans → home → Run now

Or from the terminal:

just backup-snapshots   # confirm your host appears with recent snapshots

---

Local btrfs snapshots (electra — fastest for recent loss)

electra keeps hourly local btrbk snapshots of /home on the @snapshots subvol (48h/14d/8w/6m retention). For an accidental deletion or a bad edit, this is faster than the NAS and needs no network:

just btrfs-snapshots                       # list available snapshots
just btrfs-snapshot-usage                  # how much space they use
just btrfs-restore SNAPSHOT PATH [TARGET]  # restore a path (defaults to /tmp)

Use the Backrest restore below when the local snapshots are gone too (full wipe, drive failure) or you need a point further back than local retention.

Partial recovery (single file or directory)

To recover a specific path without touching everything else:

# Preview what's in a snapshot
just backup-ls latest /home/nimmo/Documents

# Restore a path to /tmp for inspection
just backup-restore-path latest /home/nimmo/Documents/important.pdf

# Restore to a specific target directory
just backup-restore-path latest /home/nimmo/Projects/myproject /tmp/myproject-recovery

---

Browsing snapshots

The Backrest UI is the easiest way to browse snapshot contents and trigger targeted restores interactively:

• On the recovered machine: http://localhost:9898
• On the NAS: http://192.168.8.238:9898 (shows snapshots from all hosts)

---

Troubleshooting

just backup-restore fails with "credentials not found"

sops-nix hasn't activated yet. Check:

systemctl status sops-install-secrets
ls /run/secrets/

If /run/secrets/ is empty, the host key may not be registered in .sops.yaml yet. Run just add-secret-remote HOSTNAME TARGET_IP from another machine, then redeploy: sudo nixos-rebuild switch --flake .#HOSTNAME.

Snapshot is older than expected

Backrest backs up on a 4-hour poll schedule when the NAS is reachable. If the machine was away from home for an extended period before the failure, the last snapshot may be up to 4 hours stale (plus however long since you were last home).

NAS unreachable during recovery

The NAS must be on 192.168.8.238 and reachable. If you're not on the home network, connect via the WireGuard VPN (or the Nebula mesh) first — Backrest will route through the same NAS address.