Host Provisioning

• Type: Runbook
• Status: Current
• Scope: New managed NixOS hosts installed with Disko and nixos-anywhere
• Canonical sources: justfiles/hosts.just; justfiles/secrets.just; scripts/new-host-template.nix; flake.nix; modules/common/base.nix; modules/common/sops-host.nix; .sops.yaml; secrets/README.md; hosts/*/disko.nix; potential/
• Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12)
• Review triggers: Changes to host construction, the potential-host lifecycle, Disko layouts, nixos-anywhere, provisioning identities, host SSH identities, or host/secret helper recipes

Purpose

Install a prepared host configuration onto a machine, establish its permanent SSH-derived SOPS identity, verify the first deployment, and remove the temporary provisioning identity when safe.

Prerequisites

• A Forgejo issue, risk review, recovery plan, and annotated pre-change tag for reprovisioning or disk-layout work, as required by Project and Change Governance.
• Verified backups of all state that must survive. A configuration build does not preserve files from a disk that Disko reformats.
• Console or rescue access and working SSH access to the target installer as root.
• A host configuration, hardware configuration, and reviewed disko.nix. Prepare and register the source target through Adding a Host before using this installation procedure.
• The off-host provisioning age private key in the password manager. Never add it to Git, a shell history, an issue, or the wiki.
• A trusted NixOS administration host that can edit the repository's encrypted files and reach the installed machine.

Safety boundary

just provision HOST TARGET invokes nixos-anywhere with the selected flake host. Disko may repartition and format the block devices named in hosts/HOST/disko.nix. A wrong host name, target address, or device path can destroy the wrong machine's data. There is no Nix generation rollback for a reformatted disk.

Keep unrelated data disks detached or otherwise protected where practical. Compare stable device identity, model, serial, size, existing filesystems, and mount use on the target; do not approve a layout from /dev/sdX or /dev/nvmeXnY naming alone.

Electra's current disko.nix was validated for in-place adoption, not for a complete create/format reprovision. Its own source comments require a scratch disk or VM validation before relying on the from-scratch path. TPM enrolment is a separate post-install operation.

Known helper exception

Forgejo issue #224 tracks defects found while verifying this runbook:

• just add-host prints an obsolete inline nixosSystem registration example instead of the current makeNixosSystem form.
• just add-secret and just add-secret-remote do not express deliberate membership across the current scoped recipient rules, and their duplicate error names a recipe that does not exist.

The scaffolded file itself remains useful, but treat all printed follow-up instructions as transitional. Do not accept secret-helper changes without reviewing and correcting the explicit recipient rules.

Procedure

1. Prepare the host composition

Scaffold a starting file if appropriate:

just add-host HOSTNAME

Then review rather than blindly accepting the desktop-oriented template. Select the correct common/server/desktop modules, profiles, system user modules, Home Manager ownership, host metadata, hardware facts, and system.stateVersion.

A configuration under potential/HOSTNAME/ is intentionally not built. When it is ready for deployment, move it to hosts/HOSTNAME/ and register it in flake.nix using the shared builder:

HOSTNAME = makeNixosSystem {
  configName = "HOSTNAME";
};

Do not copy the old inline builder printed by the current add-host helper.

2. Capture and review hardware facts

Boot the target into a current NixOS installer or rescue environment and make it reachable over SSH. From the repository root on the administration host:

ssh root@TARGET nixos-generate-config --show-hardware-config \
  > hosts/HOSTNAME/hardware-configuration.nix
git add hosts/HOSTNAME/hardware-configuration.nix

Review the generated file. Keep reproducible hardware facts there; put the intended filesystem layout in hosts/HOSTNAME/disko.nix and import it from the host. modules/common/base.nix supplies the Disko NixOS module.

3. Review the destructive disk plan

On the target installer, record at least:

lsblk -o NAME,PATH,SIZE,MODEL,SERIAL,TYPE,FSTYPE,MOUNTPOINTS
findmnt

Compare that evidence line by line with every device in the host's disko.nix. Confirm which partitions and external data mounts are preserved, which are rebuilt, and how each required state directory will be restored.

4. Establish the provisioning recovery identity

The one-time fleet setup is:

just setup-provision-key

It creates a keypair, prints the private key once, registers the public recipient, and re-encrypts the affected secrets. Save the private key off-host when instructed and publish the reviewed encrypted changes before depending on it. Do not regenerate it for each host.

Before a new installation, confirm the public provisioning recipient remains present in every encrypted file required during first activation. This is a recipient-policy review, not a reason to add it indiscriminately to new files.

5. Validate and publish the complete host definition

Stage all new Nix files before evaluation:

git add hosts/HOSTNAME flake.nix
just check
nix build .#nixosConfigurations.HOSTNAME.config.system.build.toplevel

Follow the normal issue, branch, review, and merge requirements. Provision from the exact reviewed revision; do not rely on uncommitted differences you cannot reconstruct during recovery.

6. Provision from the installer

Reconfirm the target address and disk evidence immediately before running:

just provision HOSTNAME TARGET

Use the interactive prompt for the provisioning private key. Although the recipe accepts KEY=..., placing private material in a command line risks shell history and process-list exposure.

The recipe creates a private temporary bundle, copies the provisioning age key to /root/.config/sops/age/keys.txt in the installed system, runs nixos-anywhere against .#HOSTNAME, and removes the local temporary bundle on exit. Do not interrupt disk creation merely because the SSH session pauses during reboot.

7. Verify first boot before changing trust

After reboot, confirm console and administrative SSH access, hostname, mounts, and the deployed system:

ssh nimmo@TARGET hostname
ssh nimmo@TARGET findmnt
ssh nimmo@TARGET nixos-version --json

Use the configured primary user if it is not nimmo. Check the host's network path, systemctl --failed, required services, application paths, storage mounts, and backup registration before calling the installation successful.

8. Register the permanent host identity

From the administration host, obtain the installed machine's public SSH host key and derive its age recipient:

ssh nimmo@TARGET 'cat /etc/ssh/ssh_host_ed25519_key.pub' | ssh-to-age

Add that public recipient as a named anchor in .sops.yaml, then add it only to the explicit creation rules for files the host genuinely consumes. Follow ADR-006: host membership is determined by consumer need, not by fleet membership.

Re-encrypt each affected tracked file with an existing authorised identity. On an administration host whose SSH host key is already an authorised recipient, the current helper mechanism is equivalent to:

LOCAL_AGE_KEY=$(sudo ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key)
sudo env SOPS_AGE_KEY="$LOCAL_AGE_KEY" sops updatekeys secrets/REQUIRED-FILE.yaml
unset LOCAL_AGE_KEY

Repeat only for files whose matching rule changed. Review git diff --stat, .sops.yaml, and the encrypted-file metadata without printing plaintext. Commit and publish this trust change through its required review path.

Until issue #224 is resolved, just add-secret-remote HOSTNAME TARGET may be used only as a starting transformation in a clean worktree: inspect and correct its rule placement before accepting any re-encryption.

9. Prove host-key decryption and remove bootstrap material

Ensure the new host has the exact published repository revision (clone/pull its checkout or deliberately stage that tree), then perform a controlled test activation. Confirm required services received their SOPS material without decryption errors. Only then remove the provisioning private key from the new host, using the configured administrative account:

ssh -t nimmo@TARGET 'sudo rm /root/.config/sops/age/keys.txt'

Substitute another configured primary user where necessary. The current provision recipe prints a root@TARGET removal command, but managed hosts that enable SSH prohibit root login. Verify the path is absent and repeat a configuration test using only the host SSH identity. Removing the bootstrap key earlier can leave the machine unable to activate its configuration; leaving it indefinitely broadens the effect of a root compromise.

Success checks

• The host boots without installer media and is reachable through its intended administrative path.
• Block devices, mounts, encryption, swap, and persistent data paths match the reviewed layout.
• just inventory HOSTNAME reports correct role, purpose, location, GPU facts, and tags.
• The running configuration revision matches the reviewed deployment and the expected channel/builder selection.
• No new relevant failed units exist, and every required user-facing service path works.
• Backup jobs include the intended state and complete a first verified run.
• A test activation succeeds after the provisioning private key is removed.
• The issue records disk evidence, validation, deployment, trust transition, restore/backup evidence, and any remaining exceptions.

Rollback and recovery

Before Disko starts, abort on any identity or layout mismatch. After formatting begins, rollback means reprovisioning and restoring verified backups; switching a Nix generation cannot recreate destroyed data.

If nixos-anywhere fails before first boot, preserve its output, return to the installer, re-check network and disks, and correct the declarative definition before retrying. Do not improvise a partly imperative installation that the repository cannot reproduce.

If first boot works only with the provisioning identity, keep that key in place while correcting .sops.yaml, re-encrypting the exact consumer files, and testing host-key decryption. If administrative networking fails, use the prepared console/rescue route.

Boot trust, TPM enrolment, data restore, and service cutovers have independent state and recovery requirements; follow their dedicated runbooks.

Related material

• Current host composition and disk summary: Host Inventory and Storage and Data Map
• Testing and activation: Safe Testing and Deployment
• Secret boundary: Security Baseline and ADR-006
• Full reconstruction and data restoration: Full-Host Recovery
• Electra-specific trust recovery: Electra Boot-Trust and TPM Recovery