Safe Testing and Deployment

• Type: Runbook
• Status: Current
• Scope: NixOS configuration validation and activation on managed hosts
• Canonical sources: AGENTS.md; README.md change lifecycle; justfiles/deploy.just; scripts/nixos-deploy.sh; tests/nixos-deploy-regressions.sh; tests/nixos-deploy-wrapper-regressions.sh; justfiles/maintenance.just
• Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12)
• Review triggers: Changes to validation policy, deploy/test/build recipes, the packaged deploy wrapper, specialisations, update holds, canary requirements, or deployment regression tests

Purpose

Validate and activate a repository change while preserving the target host's current hardware specialisation, preventing automatic-update races, and retaining a clear recovery path.

This runbook owns operator actions. The Project and Change Governance policy owns when an issue, branch, merge request, review, canary, or deployment evidence is required. The Engineering Handbook owns how to select validation for a change.

Prerequisites

• Use a NixOS host whose checkout contains the exact tree to test. Nova is not such a host until its NixOS deployment is confirmed.
• Know the target hostname and review git status --short.
• On Electra, know whether the NVIDIA expansion bay is physically installed. dgpu is valid only when it is installed; igpu is valid when it is absent.
• Stage every new .nix file before flake evaluation. Flakes do not see untracked files.
• For a remote maintenance window, create an automatic-update hold on the target host before testing or deploying.
• Have a reachable console or other recovery route for changes that may affect boot, storage, networking, SSH, authentication, or the active desktop.

Safety boundary

Command	Builds	Changes the running system	Changes the boot profile
just check	Evaluates checks; individual checks may build	No	No
just build	Current host's full top-level system	No	No
just build-diff	Current host's full top-level system	No	No
just test	Current host's full top-level system	Yes	No
just deploy	Selected host's full top-level system	Yes	Yes
just switch-spec MODE	No new build	Yes	Selects an already-built base or specialisation output

just test is a live activation. Services can restart and activation scripts can alter runtime or external state even though reboot selects the prior boot profile. Treat database migrations, storage changes, key rotation, bootloader work, and service cutovers according to their dedicated runbooks.

just deploy --host HOST delegates to that host and builds its own repository checkout. just deploy-remote --host HOST instead copies the local working tree to ~/nixos-staging on the target and performs a full deployment there. The latter is useful for unpublished changes but is still a deployment, and its staging sync deletes files absent from the local tree inside that staging directory.

Procedure

1. Identify the exact target and mode

On the host that will be activated:

hostname
cat /etc/nixos-specialisation 2>/dev/null || true
git status --short
git rev-parse HEAD

An empty specialisation value means the base configuration. Electra may report battery, which the deploy and test wrappers also treat as base.

Pass a specialisation explicitly only when changing hardware mode. Otherwise the wrappers preserve the value recorded in /etc/nixos-specialisation.

2. Prevent an update race on a remote target

Run these commands in the target host's checkout:

just auto-update-hold "testing <issue-or-branch>"
just auto-update-hold-status

Hold creation shares the updater lock and waits for an in-progress detached activation before reporting success. Do not assume that a hold created on one host protects another host.

Local, brief testing may omit a hold when no automatic update can overlap, but remote testing requires one under project policy.

3. Run validation appropriate to the change

For Nix configuration changes, start with:

just check

Also run applicable script, workflow, updater, or policy regression tests when the change crosses those boundaries. Documentation-only changes need focused review rather than a Nix build. Review every warning; do not discard one merely because the command exited successfully.

4. Build and inspect the prospective change

On the target host:

just build-diff

Use just build-diff detailed for dependency changes. On Electra, specify a mode only when intentionally testing a hardware-mode change:

just build-diff igpu detailed

Confirm that package removals, kernel changes, service changes, and the chosen specialisation match the intended scope.

5. Test without writing a boot generation

just test

The recipe builds the full host top-level and activates its detected specialisation with switch-to-configuration test. To test a deliberate Electra mode change, use just test igpu or just test dgpu only when the corresponding hardware state is present.

Exercise both kinds of interface when they exist:

• the documented user-facing command, request, or UI path;
• the corresponding systemd service/socket/timer and its recent journal.

Check systemctl --failed and inspect only relevant failures. For multi-host behavior, validate the canary and its real client path before widening rollout.

6. Deploy the verified tree

For the current host:

just deploy

For a clean, already-published checkout on another host:

just deploy --host vega

For an unpublished local tree that must be staged to another host:

just deploy-remote --host vega

Add detailed to any deploy invocation for the full dependency diff. Add an Electra mode only for an intentional hardware-mode selection. The deploy wrapper builds first, compares the running system with the selected output, sets the system profile to the built top-level, and then activates the selected base or specialisation.

7. Verify and close the maintenance window

Repeat the actual user-facing check and relevant unit checks. Then record:

• host, deployed revision, and active specialisation;
• validation and canary evidence;
• unexpected warnings or failed units;
• rollback or follow-up work still required.

For fleet or auto-update changes, perform and record one genuine deployment and an immediate no-op follow-up before closing the issue.

If a hold was created, release it on the same target host only after validation:

just auto-update-release
just auto-update-hold-status

Success checks

• just fleet-status reports the expected deployed revision and specialisation for the reachable target.
• cat /etc/nixos-specialisation reports the expected mode where the file is present.
• systemctl --failed contains no new relevant failures.
• The service's user-facing path works, not merely its unit state.
• A remote target no longer has a maintenance hold unless one is deliberately retained and documented.
• Required issue/MR checks, canary evidence, and no-op follow-up evidence are recorded under project policy.

Rollback and recovery

After a failed just test, reboot to return to the boot-profile generation. When reboot is undesirable, reactivate the persisted system profile. Use the base path for an empty/battery mode or the specialisation path matching the installed Electra hardware:

sudo /nix/var/nix/profiles/system/bin/switch-to-configuration switch
SPEC=igpu  # or dgpu, matching the installed hardware
sudo "/nix/var/nix/profiles/system/specialisation/$SPEC/bin/switch-to-configuration" switch

Do not run both recovery forms. Select exactly one valid path.

After a failed deployment, use just generations to identify the prior generation. Prefer selecting it from the boot menu when live activation is unsafe. A command-line generation rollback must switch the system profile and then activate the correct base/specialisation output; follow the recovery instructions printed by the generation tooling and preserve evidence before garbage collection.

If SSH or networking is lost, stop remote retries and use the prepared console route. If storage, boot trust, encryption, or application data changed, use the subsystem recovery runbook rather than assuming a Nix generation rollback will undo external state.

Troubleshooting

• A new file is reported missing: confirm it is staged with git add, then rerun evaluation.
• The built specialisation is absent: stop; confirm the host target and available outputs rather than falling back to a different hardware mode.
• Remote deployment used stale code: just deploy --host uses the remote checkout. Publish/synchronise that checkout, or deliberately use just deploy-remote for the local tree.
• The hold command waits: an updater or detached activation may still own the lock. Let the control command determine a safe boundary; do not delete lock or hold files manually.
• Activation succeeds but a feature fails: capture the relevant unit status, journal interval, user-facing failure, revision, and specialisation before restarting or rolling back.

Related material

• Policy and evidence requirements: Project and Change Governance
• Hardware-mode rationale: ADR-001
• Update concurrency and recovery rationale: ADR-005
• Current host and automation reference: Host Inventory and Automation and Control Plane