nimmo/nixos-config

Watch

Fork 0

06 – Generalise Remote Deployment Across the NixOS Fleet #237

New issue

Closed

opened 2026-09-19 20:32:44 +01:00 by nimmo · 4 comments

nimmo commented

2026-09-19 20:32:44 +01:00

Owner

Completed

The controller-to-target remote deployment workflow from #236 has been generalised into a fleet-capable mechanism.

Delivered

Renamed and extended scripts/nixos-controller-test.sh → scripts/nixos-remote-deploy.sh.
Supports test (default), boot, and switch activation actions.
just test-remote now uses a positional target and optional action, e.g. just test-remote vega switch.
Updated README.md and docs/nova-remote-builder.md with the new interface, fleet eligibility, exclusions, and recovery procedures.

Validated

Vega — proven in #236.
Lyra — proven with the generalised workflow.
Lena — proven with the generalised workflow.
Nova unavailable — forced remote build fails cleanly; just deploy local / explicit local override recovers without changing declarative policy.

Controller SSH unavailable after switch — target-side rollback confirmed:

sudo nixos-rebuild --flake .#<hostname> switch --rollback --specialisation <name>

Exclusions recorded

Electra — controller only; hardware-presence specialisations and measured/secure boot make it unsuitable as a routine target.
Nova — dedicated remote builder, not a NixOS flake target.
Chaos — eligible in principle but not yet exercised with the controller workflow.

MR !248 merged.

## Completed The controller-to-target remote deployment workflow from #236 has been generalised into a fleet-capable mechanism. ### Delivered - Renamed and extended `scripts/nixos-controller-test.sh` → `scripts/nixos-remote-deploy.sh`. - Supports `test` (default), `boot`, and `switch` activation actions. - `just test-remote` now uses a positional target and optional action, e.g. `just test-remote vega switch`. - Updated `README.md` and `docs/nova-remote-builder.md` with the new interface, fleet eligibility, exclusions, and recovery procedures. ### Validated - **Vega** — proven in #236. - **Lyra** — proven with the generalised workflow. - **Lena** — proven with the generalised workflow. - **Nova unavailable** — forced remote build fails cleanly; `just deploy local` / explicit local override recovers without changing declarative policy. - **Controller SSH unavailable after switch** — target-side rollback confirmed: ```bash sudo nixos-rebuild --flake .#<hostname> switch --rollback --specialisation <name> ``` ### Exclusions recorded - **Electra** — controller only; hardware-presence specialisations and measured/secure boot make it unsuitable as a routine target. - **Nova** — dedicated remote builder, not a NixOS flake target. - **Chaos** — eligible in principle but not yet exercised with the controller workflow. MR !248 merged.

nimmo added this to the NixOS Build & Deployment Pipeline milestone

2026-09-19 20:32:44 +01:00

nimmo added the

host:fleet

host:nova

phase:deploy

project:nixos-build-deployment-pipeline

service:nix

labels

2026-09-19 21:01:52 +01:00

nimmo referenced this issue

2026-09-19 21:02:37 +01:00

09 – Document Build, Deployment and Recovery Workflows #240

nimmo referenced this issue

2026-09-20 09:07:26 +01:00

05 – Deploy Vega from Electra via Nova #236

nimmo commented

2026-09-21 17:46:03 +01:00

Author

Owner

Evidence update — target hosts exercised

The following hosts have now been proven with the controller-to-target workflow (captured working-tree snapshot → Nova remote build → signed closure copy → target activation):

Vega — proven during #236.
Lyra — proven with the generalised workflow.
Lena — proven with the generalised workflow.

This satisfies the validation requirement to use the same implementation for Vega, Lyra, and at least one additional suitable host.

**Evidence update — target hosts exercised** The following hosts have now been proven with the controller-to-target workflow (captured working-tree snapshot → Nova remote build → signed closure copy → target activation): - **Vega** — proven during #236. - **Lyra** — proven with the generalised workflow. - **Lena** — proven with the generalised workflow. This satisfies the validation requirement to use the same implementation for Vega, Lyra, and at least one additional suitable host.

nimmo commented

2026-09-21 17:46:05 +01:00

Author

Owner

Fleet eligibility and exclusions

Documented in docs/nova-remote-builder.md on the feature branch:

Electra — excluded as a routine target. It runs hardware-presence specialisations (igpu/dgpu) and measured/secure-boot policy, so it acts as the controller, not a deployment target.
Nova — excluded. It is the dedicated remote builder (Fedora 44, not a NixOS flake target), not a deployment target.
Chaos — eligible in principle but not yet exercised. It runs the stable 26.05 package set, so the controller must evaluate its captured snapshot with the same nixpkgs-stable input.

Target-side recovery procedures (Nova unavailable, controller SSH unavailable after switch) are also documented in the same file.

**Fleet eligibility and exclusions** Documented in `docs/nova-remote-builder.md` on the feature branch: - **Electra** — excluded as a routine target. It runs hardware-presence specialisations (`igpu`/`dgpu`) and measured/secure-boot policy, so it acts as the controller, not a deployment target. - **Nova** — excluded. It is the dedicated remote builder (Fedora 44, not a NixOS flake target), not a deployment target. - **Chaos** — eligible in principle but not yet exercised. It runs the stable 26.05 package set, so the controller must evaluate its captured snapshot with the same `nixpkgs-stable` input. Target-side recovery procedures (Nova unavailable, controller SSH unavailable after switch) are also documented in the same file.

nimmo referenced this issue

2026-09-21 17:46:48 +01:00

feat(nix): generalise remote deployment across the fleet (#237) #248

nimmo referenced this issue from a commit

2026-09-21 20:25:00 +01:00

Merge pull request 'feat(nix): generalise remote deployment across the fleet (#237)' (#248) from feat/issue-237-generalise-remote-deployment into main

nimmo commented

2026-09-21 20:36:18 +01:00

Author

Owner

Evidence — Nova unavailable recovery (Electra local fallback)

Test setup:

Electra switched to a mobile hotspot and Nebula mesh disabled so Nova was unreachable.
Ran just deploy remote to force a Nova remote build.

Result:

Build failed cleanly as expected: Failed to find a machine for remote build! / local builds are disabled (max-jobs = 0).
No activation occurred; the failure was safe.

Recovery:

Ran just deploy local (raw equivalent: sudo nixos-rebuild test --flake .#electra --option builders '' --option max-jobs auto).
Build succeeded locally and the configuration activated successfully.

This confirms the documented recovery procedure works without changing the host's declarative builder policy.

**Evidence — Nova unavailable recovery (Electra local fallback)** Test setup: - Electra switched to a mobile hotspot and Nebula mesh disabled so Nova was unreachable. - Ran `just deploy remote` to force a Nova remote build. Result: - Build failed cleanly as expected: `Failed to find a machine for remote build!` / `local builds are disabled (max-jobs = 0)`. - No activation occurred; the failure was safe. Recovery: - Ran `just deploy local` (raw equivalent: `sudo nixos-rebuild test --flake .#electra --option builders '' --option max-jobs auto`). - Build succeeded locally and the configuration activated successfully. This confirms the documented recovery procedure works without changing the host's declarative builder policy.

nimmo commented

2026-09-21 20:48:08 +01:00

Author

Owner

Evidence — target-side rollback after controller SSH loss

Tested on Electra (igpu specialisation):

sudo nixos-rebuild --flake .#electra switch --rollback --specialisation igpu

Result:

Rolled back to the previous generation without rebooting.
Stayed on the correct igpu specialisation.
Did not require NIX_PATH or the legacy nixos-config lookup.

For non-flake or no-repo recovery, the direct path also works:

sudo /nix/var/nix/profiles/system-<N>-link/bin/switch-to-configuration switch

This confirms the documented target-side recovery procedure.

**Evidence — target-side rollback after controller SSH loss** Tested on Electra (igpu specialisation): ```bash sudo nixos-rebuild --flake .#electra switch --rollback --specialisation igpu ``` Result: - Rolled back to the previous generation without rebooting. - Stayed on the correct `igpu` specialisation. - Did not require `NIX_PATH` or the legacy `nixos-config` lookup. For non-flake or no-repo recovery, the direct path also works: ```bash sudo /nix/var/nix/profiles/system-<N>-link/bin/switch-to-configuration switch ``` This confirms the documented target-side recovery procedure.

nimmo closed this issue

2026-09-21 20:49:09 +01:00

nimmo referenced this issue

2026-09-21 20:56:22 +01:00

feat(deploy): add rollback and switch-generation just recipes #249