nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions
Page: 47 Testing and CI Contracts
Pages
00 Documentation Map and Scope 00 Documentation Standards 00 Start Here and Governance 10 System Atlas 11 Fleet Overview 12 Host Inventory 13 Service Catalogue 14 Network and Access Topology 15 Storage and Data Map 16 Automation and Control Plane 20 Architecture Decisions 21 ADR 001 Electra Hardware Specialisations 22 ADR 002 Workload Placement 23 ADR 003 Network Exposure 23 Project Tracking 24 ADR 004 Engram Durable Agent Memory 24 Security Baseline 25 ADR 005 Auto Update Control Plane 26 ADR 006 Scoped Secret Files 30 Operations and Runbooks 31 Routine Fleet Operations 32 Safe Testing and Deployment 33 Host Provisioning 34 Monitoring and Service Investigation 35 Backup Verification and Partial Restore 36 Full Host Recovery 37 Electra Boot Trust Recovery 38 Secret Editing and Recipient Rotation 39 Auto Update Incident Response 39a Service Migration and Cutover 39b Wrapper Repository Packaging 40 Engineering Handbook 41 Repository Structure and Import Chain 42 Configuration Ownership Boundaries 43 Adding Packages and Flake Inputs 44 Adding a Module or Profile 45 Adding a Host 46 Adding or Migrating a Service 47 Testing and CI Contracts 48 Secrets Implementation Conventions 49 Documentation Maintenance 50 NixOS Learning Path 51 Guided Repository Tour 52 Nix Language Essentials 53 Declarative Configuration Store and Generations 54 Flakes and Inputs 55 NixOS Modules and Option Merging 56 SpecialArgs and Home Manager Argument Flow 57 Electra Specialisations Worked Example 58 Nix Troubleshooting Primer 90 Historical Material 91 Electra Secure Boot Hardening Record Home
1 47 Testing and CI Contracts
Nimmo edited this page 2026-09-13 12:40:41 +01:00
Table of contents

Testing and CI Contracts

  • Type: Reference
  • Status: Current
  • Scope: Validation selection and the repository's current local and Forgejo CI gates
  • Canonical sources: AGENTS.md; README.md change lifecycle; flake.nix; treefmt.nix; justfiles/deploy.just; .forgejo/workflows/attic-cache.yml; .forgejo/workflows/ai-review.yml; scripts/attic-cache-build.sh; tests/; Safe Testing and Deployment
  • Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff changes only flake.lock. Local recipes, flake outputs, and Forgejo workflow gates checked 2026-09-13; no CI run or host deployment was performed.
  • Review triggers: Changes to AGENTS.md validation policy, flake checks/formatter, test scripts, CI path filters/jobs, cache-build script, deploy recipes, or canary and evidence requirements

Purpose and boundary

Choose validation by the behaviour a change affects and know what each gate actually proves. This page describes the engineering contract; Safe Testing and Deployment owns activation, auto-update holds, Electra specialisations, and operational rollback. Passing CI is not proof that a service works for its clients or that the intended host has deployed the change.

Select the gates

Change surface Local evidence before review or commit Additional evidence before calling it live
Wiki, README, or other documentation only Focused source/prose/link review; no Nix build merely for prose Published page/commit and accurate provenance
Nix modules, hosts, packages, flake inputs, or options Stage new .nix files; just check; build affected host closure(s); inspect intended diff Appropriate canary with just test, real user/service check, then deployment evidence
Shell scripts, Just recipes, workflow YAML, updater or policy logic Relevant lint and regression tests, including cross-boundary tests; just check too when Nix outputs or policies are affected Exercise documented command and systemd path; multi-host/update changes need genuine deployment and no-op follow-up
Stateful, security, network, boot, or cross-host changes All applicable gates plus explicit rollback/recovery and target-specific checks Required MR checks, controlled canary, data/client/restore evidence as applicable

These are minimums from AGENTS.md, not a claim that a file extension proves safety. A .nix module can change a service's shell launcher or security boundary; a shell script can change updater policy. Review diagnostic warnings rather than suppressing them. A cross-cutting, service, CI, security, or multi-host change normally uses an issue, feature branch, and MR with the same milestone; a bounded one-application or module toggle can go directly to main after targeted validation. Close an issue only after any required live evidence is recorded.

Local commands and what they prove

just check
nix fmt -- --ci
nix build .#nixosConfigurations.HOST.config.system.build.toplevel --no-link

just check calls nix flake check. The flake currently exposes formatting and an auto-update-state-machine check on x86_64-linux; the latter comes from the pinned standalone updater. The formatting check uses treefmt.nix (nixfmt and shfmt). nix fmt -- --ci checks formatting without rewriting files. New untracked .nix files must be staged first because a Git-backed flake omits them. The explicit nix build constructs one registered host top-level; replace HOST with the affected target. It does not activate it, prove mount/secret availability, or exercise client traffic.

For scripts and workflows, run the relevant repository regressions, not an undifferentiated test folder. Current examples include tests/nixos-update-remediation-regressions.sh, tests/nixos-pin-lifecycle-regressions.sh, and the deployment wrapper tests tests/nixos-deploy-regressions.sh and tests/nixos-deploy-wrapper-regressions.sh. Verify their prerequisites and the precise affected contract. A test existing under tests/ does not mean the Forgejo workflow runs it automatically.

For the activation path, inspect just build-diff, create a remote auto-update hold where required, then use just test on an affected NixOS host. This test activation does not add a boot-menu generation and preserves Electra's current specialisation unless explicitly changed. Check the documented user-facing command/UI and the service/socket/timer plus journal when both are interfaces. Safe Testing and Deployment has the ordered commands and recovery checks. Nova is currently a Fedora CI runner, not a registered NixOS target; CI running there is not a Nova NixOS deployment.

What Forgejo CI currently does

The Build and Push Attic Cache workflow runs on main pushes and pull requests only when paths in .forgejo/workflows/attic-cache.yml match. It also has scheduled and manually dispatched runs. Its test job on Nova runs ShellCheck for scripts/*.sh and tests/*.sh, Actionlint for Forgejo workflows, forgejo-runner validate, the remediation publisher bootstrap check, the two updater/pin-lifecycle regression scripts named above, and nix flake check. A failed test job blocks the dependent build job.

The cache build job is deliberately skipped on pull requests. On eligible non-PR runs, scripts/attic-cache-build.sh builds current host closures or update candidates and pushes cache outputs. Therefore a green PR test job does not establish that every affected host top-level built; perform targeted local builds and record their revision. The workflow's path filters also mean a change outside them may have no cache-workflow PR check at all. Do not infer coverage merely from a green or absent badge.

The separate .forgejo/workflows/ai-review.yml runs an advisory PR review on same-repository pull requests and manual dispatch. It is not a substitute for lint, Nix checks, targeted builds, or live evidence. The MR must identify which checks were required, which ran, and what remains for deployment.

Evidence, failures, and maintenance

  • Record the exact source revision, commands, affected hosts, outcomes, warnings, CI run/MR, canary, and any remaining live gate in the issue/MR. A test of a different commit or stale target checkout is not evidence for the proposed revision.
  • If just check fails, first verify Git staging, formatting, option errors, and the named flake check. If a host build fails after flake check passes, investigate the exact host closure, input/policy and package failure. If CI is absent, inspect path filters and runner state before declaring success.
  • If a test passes but the client path fails, the acceptance gate is still open. Check the activated generation, service logs, network path, secrets, mounts, and relevant external dependencies.
  • Before activation, revert the reviewed source if needed. After activation, use the rollback and recovery procedure appropriate to the change; generation rollback may not reverse data or schema migrations. Do not close a multi-host or auto-update issue until a genuine deployment and no-op follow-up are recorded.
  • When adding a test, decide whether it is a local-only diagnostic, a flake check, or a Forgejo job. Update CI path filters and commands deliberately when automatic coverage is intended. Keep this page aligned with the actual workflow rather than treating its present test list as timeless policy.

Canonical source map

Concern Source
Gate selection, MR and live-evidence policy AGENTS.md, README.md Change Lifecycle
Flake and formatter checks flake.nix, treefmt.nix
Local build, diff, test, and deployment semantics justfiles/deploy.just, Safe Testing and Deployment
CI triggers, lint, regressions, Nix gate, cache build .forgejo/workflows/attic-cache.yml, scripts/attic-cache-build.sh
Advisory review trigger and trust boundary .forgejo/workflows/ai-review.yml
Regression implementation tests/