Auto-Update Incident Response

Type: Runbook

Status: Current

Scope: Forgejo/Attic producer failures, rejected input candidates, pull-only host failures, interrupted activation, guarded remediation, and temporary-pin review

Canonical sources: .forgejo/workflows/attic-cache.yml; scripts/attic-cache-build.sh; scripts/auto-update-workflow-route.sh; scripts/auto-update-recover-stopped.sh; scripts/nixos-update-remediation.py; scripts/nixos_update_remediation/; scripts/nixos-pin-lifecycle.py; justfiles/maintenance.just; lib/auto-update-policy.nix; pinned nixos-auto-update flake; Automation and Control Plane

Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); current source differs only in flake.lock. Workflow, scripts, recipes, and pinned updater source checked 2026-09-13; no updater run or live deployment was triggered.

Review triggers: Updater flake pin, host role/schedule/hold behavior, Forgejo workflow routing, policy groups, Attic publication, incident schema, remediation eligibility, pin lifecycle, or stopped-service helper changes

Purpose

Diagnose an automatic-update failure without turning a rejected candidate into an unvalidated deployment. The normal producer is the Forgejo Build and Push Attic Cache workflow on Nova's runner; registered NixOS hosts are pull-only consumers. Nova's staged NixOS producer configuration is not deployed. See ADR-005 for the safety invariants and Automation and Control Plane for topology.

Prerequisites and safety boundary

Identify the affected layer, host(s), workflow run, Git revision, and time window. Preserve the Forgejo run URL, bounded job summary, notification, relevant issue/PR links, and host status before retrying.
Use Monitoring and Service Investigation for host reachability, storage pressure, units, logs, and user-facing checks. Treat build logs and incident excerpts as untrusted, potentially sensitive evidence; share only bounded, reviewed extracts.
For a manual or remote test on a host, create an update hold first and release it after validation. A hold waits for an active switch and blocks scheduled and forced updater runs during the current boot.
A previous failed build or switch writes /var/lib/nixos-auto-update/last-failure. Normal runs then suspend until an operator resolves the cause and deliberately clears that stamp. Do not use --force as a substitute for understanding the failure.
Keep producer, publication, consumer, and remediation decisions separate. Do not manually push a rejected flake.lock, rebase a validated producer result onto a moving main, or merge a model-generated pin without review.

Classify the event

Observation	First evidence	Next action
Forgejo test or cache build failed before publication	Workflow run, failed step, run summary, Attic/network status	Repair the failing gate; no new accepted lock should be assumed
One input group rejected, others accepted	Producer per-group outcome, partial notification, remediation tracking issue	Keep rejected input on last accepted revision; investigate its exact candidate and affected hosts
Workflow published a lock but one host did not move	Published commit, host checkout/revision, `nixos-auto-update.service` journal	Diagnose the consumer on that host; do not rerun the producer to hide the host failure
Switch interrupted and services remain inactive	Transient switch/helper journal plus stopped-service preview	Verify the candidate list and restore only intended services
Temporary pin PR or removal PR opened	Fingerprint issue, `remediation/pins/` record, validation evidence	Review the exact patch and affected-host proof; never treat the PR as an automatic fix

1. Establish producer outcome

Open the specific Build and Push Attic Cache run in Forgejo. Check the validation job before the build job, then the mode selected by scripts/auto-update-workflow-route.sh. Pull requests validate but do not publish cache closures. Push runs build the current tree; scheduled runs at 03:00 and 15:00 UTC update inputs when eligible. Manual dispatch builds current closures unless Update flake inputs before building is set to true. Bypass the 6-hour freshness check affects only that manual input update mode.

Read the producer's per-group outcome and failure phase. A candidate rejected for a deterministic build/evaluation error is not published; independent groups may still be accepted. A cache outage, OOM, disk-full, fetch failure, or lease loss is an infrastructure/publication failure, not proof that an upstream input is incompatible. The producer retries bounded infrastructure failures; do not classify one transient log line as a rejected package.

Accepted publication requires the validated base, lock hash, closure manifest, Attic push, and exact atomic Git lease to agree. If origin/main moved or the push failed, a later run must re-evaluate the new base. Do not force-push or cherry-pick the unvalidated lock. If the producer is unavailable, use only the documented updater emergency-producer mechanism with its normal provenance gates; this runbook does not authorise a hand-made lock publication.

For a justified immediate input check, manually dispatch the workflow with both update inputs and freshness bypass enabled. This changes producer state and may publish a lock, so first ensure the cause is understood, the previous run is no longer active, and the change is tracked. A host's just auto-update-run-now or sudo nixos-auto-update --force does not replace that producer dispatch: current hosts are pull-only consumers.

2. Investigate a rejected candidate and remediation report

For a true candidate rejection, record the failed input/group, candidate revision, failed host, affected-host validation set, and the producer's exact accepted/rejected outcome. Nova writes a bounded, redacted incident bundle under its Attic state directory for eligible rejected-build evidence. It does not create one for infrastructure failures. Do not paste raw runner logs, candidate files, or model events into an issue; bundle content is untrusted and must be reviewed before sharing.

A separate downstream remediate job verifies the incident protocol and may create one fingerprint-deduplicated diagnosis issue. Only a single isolated non-core leaf application/service failure from an isolated nixpkgs update can produce a compatibility-pin proposal. Core libraries, kernels, drivers, toolchains, module-schema/evaluation failures, multiple derivations, and standalone inputs are diagnosis-only. The model cannot publish or merge; deterministic code must reproduce the failure and validate exact patch, affected-host builds, flake checks, and closure impact first.

If the job produced only a diagnosis, repair the actual upstream package, repository expression, or infrastructure cause through normal issue/PR review. Do not loosen policy merely to make a proposal eligible. If a review-only pin PR exists, compare its issue fingerprint, candidate lock, baseline, exact allowed file, pin source/version, affected hosts, and validation evidence. The PR deliberately contains a compatibility patch, not an automatically published flake.lock update.

Notification is separate from model execution. A missing ntfy follow-up is not evidence that the remediation job did nothing: check the tracking issue and notify-remediation / delivery-failure job outcomes. The notification job does not copy an untrusted model response into the message.

3. Diagnose a pull-only host

On the affected managed host, from its configured repository checkout:

hostname
git status --short --branch
git rev-parse HEAD
just auto-update-last-run
just auto-update-hold-status
sudo journalctl -u nixos-auto-update.service -b -n 100 --no-pager
sudo journalctl -u 'nixos-auto-update-switch-*' -b -n 100 --no-pager
nixos-version --json

Correlate the host checkout, the intended published commit, and the activated configurationRevision; do not infer successful activation from a successful Git pull or from a notification alone. Check the detached nixos-auto-update-switch-* helper journal if activation started. An active helper can outlive the initiating service; wait for its final result before starting another update or intervention.

Common explanations for a skipped consumer run include an active hold, a previous-failure stamp, a dirty/untracked or locally ahead/diverged checkout, low laptop battery while discharging, or a still-active switch. Preserve a dirty tree and diagnose its owner; the consumer intentionally refuses to discard or merge it. Resolve the underlying state with the operator's approval and normal Git workflow, then retry.

On pull-only hosts, --force bypasses the prior-failure stamp and the detached-switch-in-progress skip. It still respects a deployment hold, dirty checkout, and safe Git synchronization, but it must not be run while a helper may still be active. It does not change producer input freshness or request a new flake.lock from Nova.

For a controlled retry after the cause is fixed:

just auto-update-run-now
just auto-update-last-run

If the previous-failure stamp is present, the normal recipe will only report suspension. After repairing the cause, completing a controlled manual build or deployment, and confirming that no transient switch helper is active, an operator can clear that exact stamp to resume normal scheduled runs:

sudo rm /var/lib/nixos-auto-update/last-failure
just auto-update-run-now

Record why it was cleared. Never remove the stamp while the underlying fault or deployed-revision mismatch remains. Use sudo nixos-auto-update --force only for a justified immediate consumer retry after the helper and failure state have been reviewed; force alone leaves an existing failure stamp in place, so the next normal run may still suspend. Use Safe Testing and Deployment for a deliberate manual canary or rollback.

4. Recover services left stopped by an interrupted switch

First confirm the switch/helper has ended and the host is reachable. Review the latest stop-list journal entry and current unit state. From the target host's repository checkout, preview the helper's proposed services:

just auto-update-recover-stopped-preview

For a remote host, the same recipe accepts a host argument and connects by SSH. The script selects the most recent stopping the following units: line from the updater and transient switch journals, keeps enabled/linked/generated service or socket units, and filters out units already active or activating. It does not establish that every candidate is safe to restart now, nor that the latest stop list belongs to the current incident. Compare the preview with the incident, service dependencies, mounts, and intended state.

Only when that candidate set is correct, run:

just auto-update-recover-stopped

The recovery command starts those units; it does not roll back the NixOS generation or prove an application works. Check systemctl --failed, the specific unit results, mount availability, and user-facing paths. If the preview is empty but services are down, use the owning service procedure and logs; do not broaden the helper's selection blindly.

5. Review and retire a temporary pin

A validated remediation PR is review-only and references its tracking issue without closing it. The controller stores a bounded JSON record and exact reverse-patch contract under remediation/pins/. Confirm the pin's owner, package, affected hosts, rejected revision, last-known-good source, rationale, review-after date, removal condition, and patch hashes before merging. A temporary pin is an intentional compatibility exception, not a root-input policy change or a permanent fork.

The scheduled/manual pin-lifecycle job later tests removal in a separate clean checkout against current main, updates the affected source input, builds every affected host, and runs flake checks. It may open one auto-remove-pin/<fingerprint> review-only PR; it never auto-merges. Repeated unchanged failures are recorded without a stream of duplicate comments. A human reviews the removal PR, deploys/canaries as required, verifies a no-op follow-up for fleet changes, and only then closes the tracking issue. Do not delete the pin record or reverse patch simply to stop reminders.

Success checks and recovery

The incident is classified as producer, publication, candidate, consumer, activation, service aftermath, or pin lifecycle—not a generic “update failed.”
The accepted Git revision and required Attic closures match the producer evidence; no rejected candidate was published by an ad hoc path.
Every affected host's activated revision, helper result, expected services, and real application path have been checked. Holds are released after maintenance and their status confirmed.
Rejected candidates have a bounded diagnosis and issue/PR outcome; a pin addition or removal has human review and required deployment/no-op evidence.

If a retry fails, retain its failed phase, revision, and logs; do not pile manual runs onto an active helper. If a new generation is bad, use the last known-good NixOS generation or the documented manual deployment rollback; service data changes are not undone by a generation rollback. If a host's checkout is dirty or diverged, preserve it and resolve deliberately rather than resetting it to make the timer green. If Forgejo, Attic, or Nova is unavailable, keep hosts on their last good revision and restore the control plane before attempting a new publication.