nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions
Page: 16 Automation and Control Plane
Pages
00 Documentation Map and Scope 00 Documentation Standards 00 Start Here and Governance 10 System Atlas 11 Fleet Overview 12 Host Inventory 13 Service Catalogue 14 Network and Access Topology 15 Storage and Data Map 16 Automation and Control Plane 20 Architecture Decisions 21 ADR 001 Electra Hardware Specialisations 22 ADR 002 Workload Placement 23 ADR 003 Network Exposure 23 Project Tracking 24 ADR 004 Engram Durable Agent Memory 24 Security Baseline 25 ADR 005 Auto Update Control Plane 26 ADR 006 Scoped Secret Files 30 Operations and Runbooks 31 Routine Fleet Operations 32 Safe Testing and Deployment 33 Host Provisioning 34 Monitoring and Service Investigation 35 Backup Verification and Partial Restore 36 Full Host Recovery 37 Electra Boot Trust Recovery 38 Secret Editing and Recipient Rotation 39 Auto Update Incident Response 39a Service Migration and Cutover 39b Wrapper Repository Packaging 40 Engineering Handbook 41 Repository Structure and Import Chain 42 Configuration Ownership Boundaries 43 Adding Packages and Flake Inputs 44 Adding a Module or Profile 45 Adding a Host 46 Adding or Migrating a Service 47 Testing and CI Contracts 48 Secrets Implementation Conventions 49 Documentation Maintenance 50 NixOS Learning Path 51 Guided Repository Tour 52 Nix Language Essentials 53 Declarative Configuration Store and Generations 54 Flakes and Inputs 55 NixOS Modules and Option Merging 56 SpecialArgs and Home Manager Argument Flow 57 Electra Specialisations Worked Example 58 Nix Troubleshooting Primer 90 Historical Material 91 Electra Secure Boot Hardening Record Home
6 16 Automation and Control Plane
Nimmo edited this page 2026-09-14 07:30:17 +01:00
Table of contents

Automation and Control Plane

  • Type: Reference
  • Status: Current
  • Scope: Validation, cache production, fleet updates, remediation, monitoring, notification, and agent control planes
  • Canonical sources: .forgejo/workflows/, scripts/attic-cache-build.sh, scripts/auto-update-*, scripts/nixos_update_remediation/, lib/auto-update-policy.nix, the pinned nixos-auto-update flake, composed host options, and AI service modules
  • Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); updater options evaluated on 2026-09-12
  • Review triggers: Workflow or runner changes, update-policy changes, updater input changes, cache topology changes, timer/hold behavior, remediation policy, monitoring/notification changes, or Paseo/Engram ownership changes

The control plane turns repository changes into reviewed, cached, and deployed NixOS generations. It also gathers operational evidence and supports agent work. No monitoring result, model proposal, or memory record supersedes Git as the configuration source of truth.

ADR-005 records why the update system has producer/publication/consumer boundaries and which invariants its complexity protects.

Configuration delivery flow

                     push / PR / schedule / manual dispatch
                                      |
                                      v
                            Forgejo Actions on Nova
                                      |
                         lint + regressions + flake check
                                      |
                    +-----------------+-----------------+
                    |                                   |
              pull request                         trusted event
             validation only                            |
                                                        v
                                      build current or candidate closures
                                                        |
                                            push closures to Attic
                                                        |
                                      publish accepted flake.lock update
                                                        |
                                      six pull-only NixOS consumers
                                                        |
                                       dry build -> detached activation
                                                        |
                                  revision check + Plasma/ntfy notification

The order is deliberate: an accepted lock update is not published until its required host closures have built and been pushed to Attic. Consumers therefore pull a repository revision whose evaluated closures have already passed the producer-side gate.

Forgejo Actions producer

The Build and Push Attic Cache workflow runs on the Nova Forgejo runner. Nova runs Fedora as the observed runner role; this must not be confused with the undeployed NixOS configuration under potential/nova/.

Trigger Route
Pull request affecting the validation surface Run lint, updater/remediation regressions, workflow validation, and nix flake check; do not build or push Attic closures
Ordinary push Validate, then build and cache current closures
Scheduled run at 03:00 and 15:00 UTC Validate, attempt input updates, isolate failures by policy group, build accepted closures, push cache paths, then publish an accepted lock update
Manual dispatch Build current closures, or explicitly request an input update; an additional option can bypass only the six-hour input freshness gate

Workflow concurrency is serialized under the attic-cache group without cancelling a run already in progress. The test job gates the build job. The build routing script also suppresses a recursive cache build for the workflow's own lock-only auto-update commit.

lib/auto-update-policy.nix is the auditable map from each flake input to the hosts that must validate it. Broad shared inputs rebuild the relevant fleet; host-specific inputs can rebuild only their consumers. Deliberately pinned root inputs are listed separately and are not silently advanced by normal producer runs.

Attic cache publication

scripts/attic-cache-build.sh is the repository-side producer implementation. It uses the pinned updater package's shared update and incident protocols, builds the required host closures, records the resulting paths, pushes them to the nixos cache, and only then completes the accepted lock publication.

Attic has native backends on Vega and Lyra behind the public attic.nimmog.uk route. Consumers trust the declared cache signing key and can fall back to the other configured binary caches. Cache objects are acceleration artefacts, not the source of configuration truth.

Accepted lock publication uses the update protocol's validated base and atomic lease rules. A concurrent repository change causes publication to stop rather than rebasing an unreviewed combination and deploying it.

Pull-only host consumers

Every registered NixOS host currently evaluates to consumer mode. The composed settings are:

Host class Schedule and activation Notifications
Electra and Lena Every two hours, 15-minute randomized delay, pull-only, detached activation Plasma status/success/failure plus ntfy failure
Vega, Lyra, Cosmos, and Chaos Same pull-only schedule and detached activation ntfy success and failure

A consumer accepts only a fast-forward of its clean checkout, performs a dry build, builds the exact Git revision as the repository owner, and gives the privileged switch to a transient helper. The helper avoids an activation interrupting the updater unit that launched it. Completion verifies that the activated configurationRevision matches the intended commit.

Laptop updates defer below the configured battery threshold while discharging. A dirty, locally ahead, or diverged checkout is not automatically merged or discarded.

Deployment holds and recovery

A deployment hold coordinates manual or remote test activation with the update timer. Hold creation shares the updater lock and waits for any activation helper already in progress. While held, scheduled and forced runs stop before pulling, building, or activating and send a reminder. Hold state lives under /run, so it requires explicit release during the boot but is cleared by reboot.

The repository provides a bounded recovery helper for services that remain inactive after an updater switch. Preview mode derives candidates from the most recent updater stop list and filters them to enabled service/socket units; recovery mode starts only that filtered set, locally or over SSH. Procedural commands belong in Auto-Update Incident Response rather than this architecture page.

Guarded update remediation

Rejected input candidates can produce bounded, redacted incident bundles on the Nova runner. Infrastructure failures do not produce candidate incidents, and incident generation cannot change whether an accepted lock is published.

The downstream remediation job has two sharply separated parts:

  1. A sandboxed model may inspect bounded untrusted evidence and propose a small compatibility change for an eligible leaf package.
  2. Deterministic controller code decides eligibility, reproduces the failure, validates allowed paths and sizes, builds the baseline and proposal, checks closure impact, and controls all Forgejo publication.

The model receives neither repository publication credentials nor ntfy credentials. It cannot commit, merge, deploy, or publish flake.lock. Ineligible, unsafe, ambiguous, or failed proposals remain diagnosis-only.

An accepted proposal becomes a fingerprint-deduplicated issue and a review-only temporary-pin pull request. Human review remains mandatory. Scheduled or manually dispatched pin-lifecycle automation later retests tracked pins and may prepare a separate review-only removal pull request; it never auto-merges the removal.

Notification is a separate job so model-capable execution does not inherit ntfy credentials. If delivery fails, another credentialed step can record a bounded status on the tracking issue without copying an untrusted response.

Review and deployment gates

Decision Automated evidence Human authority retained
Merge an ordinary change Required workflow checks and optional AI pull-request review Reviewer decides whether to merge
Publish an input update Policy-scoped builds, cache push, and atomic lease validation Policy is human-authored; failed or partial candidates remain visible
Accept a remediation pin Reproduction, bounded deterministic validation, and affected-host builds Review-only PR must be approved and merged by a person
Remove a temporary pin Scheduled clean retest and review-only removal PR Human reviews removal and closes the lifecycle issue after evidence
Activate a manual deployment Local validation and an appropriate canary Operator chooses test/deployment timing and manages any hold
Close deployment-affecting work CI plus required live deployment and no-op evidence Operator records and judges the evidence

The separate AI pull-request review workflow runs for internal pull requests and manual requests on Nova. It uses the maintained Forgejo review action to post advisory review feedback. Fork pull requests are excluded from the credentialed review job. Its output is review input, not merge authority.

Monitoring and notification plane

Component Role Boundary
Beszel agents Report host health and inventory from every registered host Live evidence only; does not define desired configuration or deployment state
Beszel dashboard Aggregates monitoring outside this repository Availability and retention are external dependencies
Service failure monitor Adds OnFailure notification handlers to explicitly selected server units Sends bounded recent journal context to host-specific ntfy topics
Updater notifications Reports update status, package differences, holds, and failures Desktop Plasma messages and/or authenticated ntfy according to host class
Workflow notifications Reports producer gate, cache, partial-update, and remediation outcomes Credentials remain outside the sandboxed model boundary
Docker report Daily Lyra job documenting mutable stacks into Trilium Script body and stack definitions are restored/mutable state, not Nix-managed source

Monitoring can show that a declared service is absent or unhealthy. It cannot prove why without logs, nor can it prove that a repository change has been deployed unless revision evidence agrees.

Agent control plane

Paseo and Engram support interactive work but occupy different roles:

  • Paseo brokers user-authorized agent sessions. Vega and Lyra run daemons, the public relay route connects remote clients, and Forgejo access is provided through tea or the mesh-hosted Forgejo MCP service. The Network and Access Topology owns its listener and relay paths.
  • Engram stores durable cross-session agent context. Clients launch a pinned local MCP binary; Vega hosts the replication service backed by PostgreSQL. Repository documentation and current code override a stale memory. ADR-004 records why memory is local-first and explicitly non-canonical.
  • Forgejo issues and milestones are the durable execution ledger for planned work. The wiki holds durable operating knowledge after the work closes.

Neither an agent transcript nor Engram memory is deployment evidence. Material changes still follow the repository's issue, branch, validation, review, and deployment rules.

Current-state boundary

Nova's staged NixOS configuration selects producer, but it is not a registered flake target and has not been deployed. Today, the active producer is the Forgejo Actions workflow running on Nova's Fedora runner; all six managed NixOS hosts remain pull-only consumers. A future Nova NixOS deployment must reconcile those two producer mechanisms explicitly rather than assuming the staged timer has taken ownership.

Return to the System Atlas, the Service Catalogue, or the Storage and Data Map. Execution procedures belong in Operations and Runbooks, change rules in the Engineering Handbook, and durable rationale in Architecture Decisions.