ADR-005: Treat automatic updates as a fleet deployment control plane

• Status: Accepted
• Date: 2026-07-17
• Scope: Flake-input updates, host closure validation, cache and Git publication, consumer activation, recovery, and remediation
• Supersedes: None
• Superseded by: None

Context

An unattended update can change privileged configuration on every managed host. The original timer-shaped implementation grew in response to concrete failures and review findings: omitted inputs or hosts, wrong-target builds, activation outliving its parent service, races with maintenance holds, hidden partial failures, ambiguous deployed revisions, and a producer validating one Git base before publishing against another.

These are distributed deployment problems, not merely package-refresh problems. A successful nix flake update proves only that a lock file was written; even nix flake check does not prove that every affected host closure was built, cached, published from the validated tree, and activated without racing another state transition.

The July 2026 reliability review established fail-closed input classification, serialized detached activation, explicit partial outcomes, exact lease-aware publication, fast-forward-only consumption, and required regression/KVM state machine tests. The implementation was later extracted into the standalone nixos-auto-update package/module flake while this repository retained fleet policy, producer workflow integration, and remediation controls.

Decision

Automatic updates are a small fleet deployment control plane with distinct producer, publication, consumer, and advisory-remediation stages.

Producer

The normal producer is the Forgejo Actions workflow running on Nova's Fedora runner. It must:

1. derive the complete root-input inventory and reject any input that is neither classified into a host-impact group nor explicitly pinned;
2. update groups in isolation so one incompatible candidate does not hide or unnecessarily reject independent successful groups;
3. run updater regression tests and the NixOS state-machine check before expensive closure publication;
4. build every host closure required by the accepted input groups;
5. push those exact closures to Attic; and
6. publish a traceable flake.lock commit only when the validated base revision, lock hash, and closure manifest still match.

Publication uses an exact atomic Git lease. The producer must not rebase a validated result onto a moving main, because the combined tree was never the tree that passed validation.

Consumers

Managed NixOS hosts are pull-only consumers. A consumer must fast-forward to a published revision, build and activate that exact revision, preserve the active Electra specialisation where applicable, and record a result that can be correlated with repository provenance.

Update, activation, and maintenance transitions share serialized state. A detached activation helper may outlive the initiating systemd process, so locks, progress markers, holds, and cleanup must cover the helper's real lifetime. Missing or ambiguous state fails closed and remains visible through status and notifications.

Freshness windows, randomized schedules, manual force controls, deployment holds, and an emergency producer path are part of this control plane. They must obey the same provenance and serialization invariants rather than bypassing them for convenience.

Remediation

Rejected input updates produce bounded evidence. A guarded Nova worker may diagnose the failure and prepare a narrowly validated temporary-pin proposal, but model output is advisory. It cannot auto-merge, alter the validated evidence after review, or receive unrelated notification credentials. Pin addition and removal remain review-only changes with explicit lifecycle tracking.

Core invariants

1. Every root input is classified or deliberately pinned.
2. Every accepted lock change has all required consumer closures built.
3. The validated base, lock hash, closure manifest, cached closures, and published commit refer to the same artefacts.
4. Publication is exact and atomic; loss of the lease fails closed.
5. Consumers fast-forward and activate the exact published revision.
6. Only one update, activation, hold, or maintenance transition owns the state machine at a time.
7. Missing, partial, or contradictory state is reported rather than collapsed into success.
8. Deterministic tests gate builds, cache publication, and Git publication.

Consequences

• Hosts consume one reviewed and cache-warmed fleet result instead of producing divergent lock files independently.
• A package candidate can be rejected for only its affected group while independent groups continue, with the partial outcome remaining visible.
• Publication and activation are slower and more complex than a simple timer, but every dangerous transition has provenance and a recovery boundary.
• Attic is an optimisation and a publication artefact store; consumers still verify and build the exact system closure rather than trusting a notification alone.
• The system requires shell regression coverage, a real systemd/Git state machine test, workflow routing tests, and live deployment/no-op evidence for material changes.
• External availability of Forgejo, the runner, Attic, Git remotes, and notifications remains a failure domain. The pipeline can fail safely and visibly; it cannot guarantee those services.
• Closure success does not prove every peripheral or application workflow on physical hardware. Risky changes still require the canary and deployment evidence defined by project governance.
• Nova's staged NixOS producer configuration and the live Fedora Forgejo runner are intentionally not conflated. A future Nova deployment must explicitly hand over producer ownership.
• The implementation is large enough to justify its standalone package/module boundary; extraction must preserve the invariants rather than simplifying away safety properties.

Alternatives considered

Let each host run nix flake update

Rejected. Hosts could select different dependency sets, repeat expensive builds, publish no shared proof, and fail independently after an upstream update.

Have one timer update and push flake.lock without closure builds

Rejected. Evaluation or lock generation is not deployment validation. It would publish candidates before proving that every affected system can build.

Build first, then rebase and push

Rejected. Rebasing creates an unvalidated tree from a validated lock change and a newer base. Exact lease failure must cause a fresh producer run.

Treat partial group failure as total failure or total success

Rejected. Total rejection delays independent safe updates; total success hides the incompatible candidate. Per-group outcomes preserve both progress and diagnostic truth.

Let the remediation model repair and merge automatically

Rejected. A model may propose a bounded pin after deterministic reproduction, but it does not hold merge or deployment authority. Human review remains the boundary for changes that intentionally diverge from upstream updates.

Remove emergency or manual controls

Rejected. Recovery needs a path when the normal producer is unavailable, and operators need holds and forced checks. Those controls are safe only when they participate in the same locking, provenance, and publication rules.

References

• Current topology and boundaries: Automation and Control Plane
• Incident procedures: Auto-Update Incident Response
• Change and rollout requirements: Project and Change Governance
• Current source: lib/auto-update-policy.nix, .forgejo/workflows/attic-cache.yml, scripts/attic-cache-build.sh, scripts/nixos_update_remediation/, tests/nixos-update-remediation-regressions.sh, and the pinned nixos-auto-update flake input and its exported state-machine check
• Reliability review evidence: Forgejo issues #51, #52, #53, #55, and #56
• Input classification: repository commit a3fb725fbe54f333fd159a1ee2b782c9381911ec
• Lease-aware publication: repository commit 9eb45a94e63f7f98437be0f02f0a10e0cb9b0fdf
• State-machine tests and required workflow gate: repository commits 769e8c36edc79401afc6717d626d25624d88f91c and 19363f83a8740b7bcfa7ff350513060fa48049ed
• Standalone module adoption: repository commits c70b92fb4455fac61a0d069c5f555a1d9d3a34a4 and 569d02ef9fc31db63c77194f0c249cd602a7057d
• Guarded remediation worker: repository commit c25788f9e6b94d9d8da821ae95285993429da34c