Critical: repair detached auto-update activation and recover Vega/Lyra

nimmo commented

2026-07-13 07:56:59 +01:00

Owner

Context

The 2026-07-13 review found that every detached switch attempt on both server consumers failed before activation:

nixos-auto-update.sh: line 66: mkdir: command not found

systemd-run started the helper without the parent service's Nix-built PATH. The parent updater returned after handoff, so the helper needed its own durable state, journal, and failure reporting. Live testing also showed that inline activation on Electra can terminate the updater that is performing the switch.

A later review found a remaining handoff race: the parent released run.lock after launching the detached helper, allowing the hold controller to create a maintenance hold while activation was still active. Commit 07ba22f makes hold creation retain the shared run lock while it observes the detached-switch marker, waits for verified active helpers, recovers inactive stale state, handles collected-unit completion, and fails closed when helper state cannot be verified.

Scope

Give the transient helper a complete, deterministic runtime environment.
Write failure state and notify even when failure occurs before metadata is loaded.
Keep hold/marker semantics valid until detached activation concludes.
Clean runtime state after success or failure.
Include transient helper journals in recovery diagnostics.
Use detached activation on every host so updater-unit replacement cannot abort its own switch.
Verify specialisation reactivation and desktop notifications on Electra.

Completion criteria

A detached helper starts in a deliberately minimal environment.
Early helper failures clean runtime metadata and are recoverable from helper journals.
All hosts evaluate to detached activation by default.
No stale marker remains after a genuine successful switch.
A hold cannot claim the update is finished while detached activation is active.
Vega and Lyra complete genuine automatic detached switches.
Electra completes a detached switch while retaining its active specialisation.
Success and failure notifications are verified.
Core services remain healthy after activation.

Final live-validation plan

After deploying commit 07ba22f:

Confirm the installed nixos-auto-update-control contains the wait message for an active detached helper.
During a genuine detached activation, invoke a hold from a second session.
Confirm it reports that it is waiting and does not create /run/nixos-auto-update/hold while switch-in-progress exists.
Confirm the hold is created only after the helper removes its marker.
Release the test hold.
Confirm deployed configurationRevision, active specialisation, notifications, updater/helper results, and zero unexpected failed units.
Confirm the next updater run takes the no-build path.

## Context The 2026-07-13 review found that every detached switch attempt on both server consumers failed before activation: ``` nixos-auto-update.sh: line 66: mkdir: command not found ``` `systemd-run` started the helper without the parent service's Nix-built `PATH`. The parent updater returned after handoff, so the helper needed its own durable state, journal, and failure reporting. Live testing also showed that inline activation on Electra can terminate the updater that is performing the switch. A later review found a remaining handoff race: the parent released `run.lock` after launching the detached helper, allowing the hold controller to create a maintenance hold while activation was still active. Commit `07ba22f` makes hold creation retain the shared run lock while it observes the detached-switch marker, waits for verified active helpers, recovers inactive stale state, handles collected-unit completion, and fails closed when helper state cannot be verified. ## Scope - Give the transient helper a complete, deterministic runtime environment. - Write failure state and notify even when failure occurs before metadata is loaded. - Keep hold/marker semantics valid until detached activation concludes. - Clean runtime state after success or failure. - Include transient helper journals in recovery diagnostics. - Use detached activation on every host so updater-unit replacement cannot abort its own switch. - Verify specialisation reactivation and desktop notifications on Electra. ## Completion criteria - [x] A detached helper starts in a deliberately minimal environment. - [x] Early helper failures clean runtime metadata and are recoverable from helper journals. - [x] All hosts evaluate to detached activation by default. - [x] No stale marker remains after a genuine successful switch. - [x] A hold cannot claim the update is finished while detached activation is active. - [x] Vega and Lyra complete genuine automatic detached switches. - [x] Electra completes a detached switch while retaining its active specialisation. - [x] Success and failure notifications are verified. - [x] Core services remain healthy after activation. ## Final live-validation plan After deploying commit `07ba22f`: 1. Confirm the installed `nixos-auto-update-control` contains the wait message for an active detached helper. 2. During a genuine detached activation, invoke a hold from a second session. 3. Confirm it reports that it is waiting and does not create `/run/nixos-auto-update/hold` while `switch-in-progress` exists. 4. Confirm the hold is created only after the helper removes its marker. 5. Release the test hold. 6. Confirm deployed `configurationRevision`, active specialisation, notifications, updater/helper results, and zero unexpected failed units. 7. Confirm the next updater run takes the no-build path.

nimmo added this to the Automated Update Reliability Review milestone

2026-07-13 07:56:59 +01:00

nimmo added the

host:fleet

project:auto-update-reliability

service:auto-update

phase:prep

labels

2026-07-13 07:56:59 +01:00

nimmo commented

2026-07-13 08:56:15 +01:00

Author

Owner

Implemented the first critical repair locally (not yet committed or deployed):

Changed the updater entry point to a writeShellApplication with explicit runtime inputs.
Detached systemd-run helpers now invoke that wrapper, reconstructing the same deterministic tool PATH instead of running the raw script in an empty manager environment.
Installed cleanup before metadata loading so early helper failures remove stale marker/metadata files safely.
Extended stopped-service recovery journal lookup to include nixos-auto-update-switch-* transient units.

Verification completed:

Bash syntax and git diff --check pass.
Generated Vega ExecStart resolves to the new wrapper.
Built the wrapper derivation and executed --finish-detached-switch under env -i with an isolated runtime directory. It reached the expected missing-metadata error (rather than mkdir: command not found) and removed the stale marker.
nix flake check passed for Electra, Lena, Vega, and Lyra.

The issue remains open because the fix still needs commit/deploy and a real detached switch on both Vega and Lyra. Aggregate parent/helper status and lock lifetime also need review before all completion criteria can be checked.

Implemented the first critical repair locally (not yet committed or deployed): - Changed the updater entry point to a `writeShellApplication` with explicit runtime inputs. - Detached `systemd-run` helpers now invoke that wrapper, reconstructing the same deterministic tool PATH instead of running the raw script in an empty manager environment. - Installed cleanup before metadata loading so early helper failures remove stale marker/metadata files safely. - Extended stopped-service recovery journal lookup to include `nixos-auto-update-switch-*` transient units. Verification completed: - Bash syntax and `git diff --check` pass. - Generated Vega `ExecStart` resolves to the new wrapper. - Built the wrapper derivation and executed `--finish-detached-switch` under `env -i` with an isolated runtime directory. It reached the expected missing-metadata error (rather than `mkdir: command not found`) and removed the stale marker. - `nix flake check` passed for Electra, Lena, Vega, and Lyra. The issue remains open because the fix still needs commit/deploy and a real detached switch on both Vega and Lyra. Aggregate parent/helper status and lock lifetime also need review before all completion criteria can be checked.

nimmo commented

2026-07-13 09:31:30 +01:00

Author

Owner

Committed the first repair as ec42e29 (fix: restore detached auto-update helper runtime). It has not been pushed or deployed.

Follow-up confirmation after deployment:

Deploy/rebuild Vega and Lyra from this commit or later.
Trigger nixos-auto-update.service on each host.
Confirm nixos-auto-update-switch-<host>.service reaches the switch rather than failing before startup.
Confirm /run/nixos-auto-update/{switch-in-progress,detached-switch.env} is removed after completion.
Confirm stopped services are restarted and the recovery preview sees detached-helper journal entries.
Confirm the intended generation is active.

Revision equality remains tracked separately in #54 because the current path: flake build behaviour is a distinct defect.

Committed the first repair as `ec42e29` (`fix: restore detached auto-update helper runtime`). It has **not** been pushed or deployed. Follow-up confirmation after deployment: - [ ] Deploy/rebuild Vega and Lyra from this commit or later. - [ ] Trigger `nixos-auto-update.service` on each host. - [ ] Confirm `nixos-auto-update-switch-<host>.service` reaches the switch rather than failing before startup. - [ ] Confirm `/run/nixos-auto-update/{switch-in-progress,detached-switch.env}` is removed after completion. - [ ] Confirm stopped services are restarted and the recovery preview sees detached-helper journal entries. - [ ] Confirm the intended generation is active. Revision equality remains tracked separately in #54 because the current `path:` flake build behaviour is a distinct defect.

nimmo commented

2026-07-13 10:39:37 +01:00

Author

Owner

Added and pushed manual Forgejo freshness override in ce7b67e (feat: allow forced Forgejo flake updates). Manual Build and Push Attic Cache dispatches can now set both update_inputs=true and force_update=true to bypass only the six-hour flake.lock freshness check. Scheduled runs retain the cooldown, and input-group build acceptance/safety checks remain active. This provides an immediate path to create a candidate revision for the outstanding repaired detached-helper activation test.

Added and pushed manual Forgejo freshness override in `ce7b67e` (`feat: allow forced Forgejo flake updates`). Manual **Build and Push Attic Cache** dispatches can now set both `update_inputs=true` and `force_update=true` to bypass only the six-hour `flake.lock` freshness check. Scheduled runs retain the cooldown, and input-group build acceptance/safety checks remain active. This provides an immediate path to create a candidate revision for the outstanding repaired detached-helper activation test.

nimmo commented

2026-07-13 11:33:55 +01:00

Author

Owner

Committed f5e5144 (fix: detach auto-update activation on all hosts). Fleet default is now detached; inline remains only as an explicit diagnostic override. This addresses the live Electra failure where inline activation terminated nixos-auto-update.service before revision verification. Electra, Lena, and Vega evaluate to switchExecution=detached; full nix flake check passed for all hosts. Commit is local and awaiting push/manual Electra bootstrap deployment.

Committed `f5e5144` (`fix: detach auto-update activation on all hosts`). Fleet default is now detached; `inline` remains only as an explicit diagnostic override. This addresses the live Electra failure where inline activation terminated `nixos-auto-update.service` before revision verification. Electra, Lena, and Vega evaluate to `switchExecution=detached`; full `nix flake check` passed for all hosts. Commit is local and awaiting push/manual Electra bootstrap deployment.

nimmo commented

2026-07-13 18:02:37 +01:00

Author

Owner

✅ Lyra genuine detached activation passed at 18:00 BST for candidate f3d3bd547c094ab947ebf3867c66cf7538482228.

Evidence:

parent updater built as nimmo, handed off successfully, and exited 0;
transient nixos-auto-update-switch-lyra ran from the generated wrapper and exited 0;
activation stopped/restarted Paseo and Polkit without interrupting the helper;
helper verified the full deployed revision equals checkout HEAD;
success ntfy notification was emitted with package/closure summary;
switch-in-progress and detached metadata were removed (only run.lock remains);
no failure stamp;
SSH, Docker, Nebula, and timer active; zero failed units.

This completes Lyra's real-switch evidence. Vega and desktop/Electra evidence remain pending.

✅ **Lyra genuine detached activation passed** at 18:00 BST for candidate `f3d3bd547c094ab947ebf3867c66cf7538482228`. Evidence: - parent updater built as `nimmo`, handed off successfully, and exited 0; - transient `nixos-auto-update-switch-lyra` ran from the generated wrapper and exited 0; - activation stopped/restarted Paseo and Polkit without interrupting the helper; - helper verified the full deployed revision equals checkout HEAD; - success ntfy notification was emitted with package/closure summary; - `switch-in-progress` and detached metadata were removed (only `run.lock` remains); - no failure stamp; - SSH, Docker, Nebula, and timer active; zero failed units. This completes Lyra's real-switch evidence. Vega and desktop/Electra evidence remain pending.

nimmo commented

2026-07-16 12:28:14 +01:00

Author

Owner

Vega and Lyra both completed another genuine detached deployment to b1efea8 on 2026-07-16. Verified checkout HEAD equals active configurationRevision on both, with no failure stamps and only run.lock remaining. This reinforces that detached activation remains stable across subsequent unrelated configuration changes.

Vega and Lyra both completed another genuine detached deployment to `b1efea8` on 2026-07-16. Verified checkout HEAD equals active `configurationRevision` on both, with no failure stamps and only `run.lock` remaining. This reinforces that detached activation remains stable across subsequent unrelated configuration changes.

nimmo commented

2026-07-16 13:54:23 +01:00

Author

Owner

Local commit 07ba22f fixes the remaining hold/detached-helper race.

Validated before commit:

ShellCheck and actionlint pass.
nix flake check passes for Electra, Lena, Vega, and Lyra.
The generated controller wrapper builds successfully.
Hermetic tests prove: active helpers block hold creation; inactive stale state is recovered; collected-unit completion is handled; unverifiable state fails closed.

The commit is intentionally not pushed automatically. Final closure requires deployment and the live hold-during-activation check in the updated issue body.

Local commit `07ba22f` fixes the remaining hold/detached-helper race. Validated before commit: - ShellCheck and actionlint pass. - `nix flake check` passes for Electra, Lena, Vega, and Lyra. - The generated controller wrapper builds successfully. - Hermetic tests prove: active helpers block hold creation; inactive stale state is recovered; collected-unit completion is handled; unverifiable state fails closed. The commit is intentionally not pushed automatically. Final closure requires deployment and the live hold-during-activation check in the updated issue body.

nimmo commented

2026-07-16 14:00:04 +01:00

Author

Owner

Live hold/activation serialization passed on Electra at revision 07ba22fb75588ef3714123242e0b2cb52b477abd.

Evidence:

detached helper started at 13:58:32 and remained active for 5.287 seconds
the new controller logged that it was waiting for nixos-auto-update-switch-electra
no hold was created during activation
helper verified the deployed revision and completed successfully at 13:58:37
helper cleanup removed switch-in-progress and detached-switch.env
the hold was then created at 13:58:37 with the requested reason
Electra retained igpu
success notification was delivered
parent and helper both have Result=success, ExecMainStatus=0
zero failed system units were reported

The critical final criterion is now checked. Release the deliberate validation hold and run one immediate no-op check before closing.

Live hold/activation serialization passed on Electra at revision `07ba22fb75588ef3714123242e0b2cb52b477abd`. Evidence: - detached helper started at 13:58:32 and remained active for 5.287 seconds - the new controller logged that it was waiting for `nixos-auto-update-switch-electra` - no hold was created during activation - helper verified the deployed revision and completed successfully at 13:58:37 - helper cleanup removed `switch-in-progress` and `detached-switch.env` - the hold was then created at 13:58:37 with the requested reason - Electra retained `igpu` - success notification was delivered - parent and helper both have `Result=success`, `ExecMainStatus=0` - zero failed system units were reported The critical final criterion is now checked. Release the deliberate validation hold and run one immediate no-op check before closing.

nimmo commented

2026-07-16 14:06:35 +01:00

Author

Owner

Final follow-up passed on Electra at 14:06.

validation hold released successfully
checkout HEAD and deployed revision both equal 07ba22fb75588ef3714123242e0b2cb52b477abd
pull reported Already up to date
updater logged Nothing changed in pull, system is current
no build or detached helper was launched
no-update desktop notification was delivered
service exited successfully in 691 ms
only run.lock remains in runtime state

All implementation, hermetic regression, fleet activation, hold serialization, cleanup, health, notification, revision, specialisation, and no-op criteria are complete. Closing.

Final follow-up passed on Electra at 14:06. - validation hold released successfully - checkout HEAD and deployed revision both equal `07ba22fb75588ef3714123242e0b2cb52b477abd` - pull reported `Already up to date` - updater logged `Nothing changed in pull, system is current` - no build or detached helper was launched - no-update desktop notification was delivered - service exited successfully in 691 ms - only `run.lock` remains in runtime state All implementation, hermetic regression, fleet activation, hold serialization, cleanup, health, notification, revision, specialisation, and no-op criteria are complete. Closing.

nimmo closed this issue

2026-07-16 14:06:40 +01:00

Rows
Columns

Critical: repair detached auto-update activation and recover Vega/Lyra #52

Context

Scope

Completion criteria

Final live-validation plan