Clarify host auto-update force and failure-stamp recovery controls #227

New issue

Open

opened 2026-09-13 00:04:39 +01:00 by nimmo · 0 comments

nimmo commented

2026-09-13 00:04:39 +01:00

Owner

Problem

The current Justfile recipe auto-update-run-now full=true calls sudo nixos-auto-update --force, while its comment says it also updates flake inputs. All active NixOS hosts are pull-only consumers. Input updates and the six-hour freshness bypass belong to the Forgejo Build and Push Attic Cache manual dispatch (update_inputs=true, force_update=true), not to the host command. README.md also describes the normal host run as respecting a six-hour freshness window, which is producer-side behavior.

The pinned nixos-auto-update source at a32a0ba shows that --force bypasses the prior-failure stamp and detached-switch-in-progress skip, while a deployment hold still wins. Force alone does not clear /var/lib/nixos-auto-update/last-failure; the next normal run may remain suspended. This is an operationally important distinction during incident response.

Scope

Correct Justfile recipe comment/help and README language without silently breaking the existing interface.
Distinguish host retry from producer input-update dispatch and state the force/hold/helper consequences.
Document or implement a guarded, explicit failure-stamp reset after the root cause is repaired and the helper has ended.
Add focused regression coverage if behavior changes.

The wiki Auto-Update Incident Response runbook at commit 91f1677 documents the current behavior. Discovered during wiki rebuild issue #222; source content baseline 556df88494.

## Problem The current Justfile recipe `auto-update-run-now full=true` calls `sudo nixos-auto-update --force`, while its comment says it also updates flake inputs. All active NixOS hosts are pull-only consumers. Input updates and the six-hour freshness bypass belong to the Forgejo Build and Push Attic Cache manual dispatch (`update_inputs=true`, `force_update=true`), not to the host command. README.md also describes the normal host run as respecting a six-hour freshness window, which is producer-side behavior. The pinned nixos-auto-update source at a32a0ba shows that `--force` bypasses the prior-failure stamp and detached-switch-in-progress skip, while a deployment hold still wins. Force alone does not clear `/var/lib/nixos-auto-update/last-failure`; the next normal run may remain suspended. This is an operationally important distinction during incident response. ## Scope - Correct Justfile recipe comment/help and README language without silently breaking the existing interface. - Distinguish host retry from producer input-update dispatch and state the force/hold/helper consequences. - Document or implement a guarded, explicit failure-stamp reset after the root cause is repaired and the helper has ended. - Add focused regression coverage if behavior changes. The wiki Auto-Update Incident Response runbook at commit 91f1677 documents the current behavior. Discovered during wiki rebuild issue #222; source content baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e.

nimmo referenced this issue

2026-09-13 00:04:54 +01:00

[Wiki 4/5] Rebuild operational runbooks and the engineering handbook #222

nimmo referenced this issue

2026-09-14 07:36:50 +01:00

[Wiki 5/5] Condense the learning path and complete the wiki cutover #223

nimmo referenced this issue

2026-09-14 07:44:11 +01:00

[Wiki 5/5] Condense the learning path and complete the wiki cutover #223