Add automated tests and linting for update scripts and Forgejo workflow

nimmo commented

2026-07-13 07:57:25 +01:00

Owner

Context

The emergency local path is explicitly documented as untested and able to bitrot. The detached helper regression and null-revision regression both crossed shell/Nix/systemd boundaries that syntax checks alone do not cover. No repository test currently exercises the full update state machine.

The live revision-provenance work in #54 is complete. Its remaining historical migration case—an active legacy generation with a missing revision—belongs here because it can be tested safely and deterministically without deliberately degrading a live host.

Scope

Add shell linting and workflow validation (for example ShellCheck and an Actions-compatible YAML linter).
Create hermetic tests with fake git, nixos-rebuild, systemd-run, notification, and power/network commands.
Test shared-policy rollback and partial acceptance.
Test detached helper with a minimal PATH, startup failure, switch failure, success, cleanup, locking, and hold races.
Test missing/null/matching/diverged configuration revisions.
Test that a legacy generation with no recorded revision triggers one successful activation, records the candidate revision, then converges to a no-build follow-up.
Test workflow event conditions for push, schedule, manual update, manual current build, and auto-update commit loop suppression.
Exercise local emergency producer mode in CI without network mutation.

Completion criteria

Tests reproduce the two critical live failures found by this review before fixes.
Tests run in nix flake check or an equivalent required Forgejo check.
Workflow conditions have table-driven coverage.
Shell and workflow lint failures block merging.
A hermetic legacy missing-revision test proves one activation followed by a no-build run.
The emergency path is no longer labelled untested.

## Context The emergency local path is explicitly documented as untested and able to bitrot. The detached helper regression and null-revision regression both crossed shell/Nix/systemd boundaries that syntax checks alone do not cover. No repository test currently exercises the full update state machine. The live revision-provenance work in #54 is complete. Its remaining historical migration case—an active legacy generation with a missing revision—belongs here because it can be tested safely and deterministically without deliberately degrading a live host. ## Scope - Add shell linting and workflow validation (for example ShellCheck and an Actions-compatible YAML linter). - Create hermetic tests with fake `git`, `nixos-rebuild`, `systemd-run`, notification, and power/network commands. - Test shared-policy rollback and partial acceptance. - Test detached helper with a minimal PATH, startup failure, switch failure, success, cleanup, locking, and hold races. - Test missing/null/matching/diverged configuration revisions. - Test that a legacy generation with no recorded revision triggers one successful activation, records the candidate revision, then converges to a no-build follow-up. - Test workflow event conditions for push, schedule, manual update, manual current build, and auto-update commit loop suppression. - Exercise local emergency producer mode in CI without network mutation. ## Completion criteria - [ ] Tests reproduce the two critical live failures found by this review before fixes. - [ ] Tests run in `nix flake check` or an equivalent required Forgejo check. - [ ] Workflow conditions have table-driven coverage. - [ ] Shell and workflow lint failures block merging. - [ ] A hermetic legacy missing-revision test proves one activation followed by a no-build run. - [ ] The emergency path is no longer labelled untested.

nimmo added this to the Automated Update Reliability Review milestone

2026-07-13 07:57:25 +01:00

nimmo added the

phase:validation

host:fleet

project:auto-update-reliability

service:auto-update

labels

2026-07-13 07:57:25 +01:00

nimmo commented

2026-07-13 12:44:54 +01:00

Author

Owner

Phase-one regression coverage is complete in local isolated branch test/auto-update-regressions, commit 1720a85 (test: add auto-update regression coverage); intentionally not pushed before the evening live updater test.

Coverage added:

generated Nix updater wrapper is independently buildable;
all four hosts must evaluate to detached activation;
exact-revision build call must retain the repository-owner boundary;
real generated wrapper is launched under env -i;
early missing-metadata failure must be reported, avoid exit 127, and remove stale progress state;
ShellCheck and actionlint run in a dedicated Forgejo workflow;
just test-auto-update provides the local entry point;
README warns that explicit .#host selectors can activate another host's configuration.

Validation: regression suite passed, ShellCheck passed, actionlint passed with only the configured custom Nova runner-label exception, and full nix flake check passed all hosts. Remaining issue scope: switch success/failure fixtures, locking/hold races, revision state table, workflow-event table, shared-policy rollback/partial acceptance, and emergency producer mode.

Phase-one regression coverage is complete in local isolated branch `test/auto-update-regressions`, commit `1720a85` (`test: add auto-update regression coverage`); intentionally not pushed before the evening live updater test. Coverage added: - generated Nix updater wrapper is independently buildable; - all four hosts must evaluate to detached activation; - exact-revision build call must retain the repository-owner boundary; - real generated wrapper is launched under `env -i`; - early missing-metadata failure must be reported, avoid exit 127, and remove stale progress state; - ShellCheck and actionlint run in a dedicated Forgejo workflow; - `just test-auto-update` provides the local entry point; - README warns that explicit `.#host` selectors can activate another host's configuration. Validation: regression suite passed, ShellCheck passed, actionlint passed with only the configured custom Nova runner-label exception, and full `nix flake check` passed all hosts. Remaining issue scope: switch success/failure fixtures, locking/hold races, revision state table, workflow-event table, shared-policy rollback/partial acceptance, and emergency producer mode.

nimmo commented

2026-07-16 12:28:08 +01:00

Author

Owner

Phase-one regression coverage is now merged and pushed on main as efdfc41 (test: add auto-update regression coverage). Pre-push validation passed: ShellCheck, actionlint, just test-auto-update, git diff --check, and full nix flake check for all hosts. Forgejo run #311 is queued to exercise the new required workflow; Attic run #310 is building the current closures. Issue remains open for the deeper phase-two state-machine coverage listed above.

Phase-one regression coverage is now merged and pushed on `main` as `efdfc41` (`test: add auto-update regression coverage`). Pre-push validation passed: ShellCheck, actionlint, `just test-auto-update`, `git diff --check`, and full `nix flake check` for all hosts. Forgejo run [#311](https://git.nimmog.uk/nimmo/nixos-config/actions/runs/311) is queued to exercise the new required workflow; Attic run [#310](https://git.nimmog.uk/nimmo/nixos-config/actions/runs/310) is building the current closures. Issue remains open for the deeper phase-two state-machine coverage listed above.

nimmo referenced this issue

2026-07-16 13:28:18 +01:00

High: restore reliable deployed-revision detection and Git flake provenance #54

nimmo commented

2026-07-16 13:30:14 +01:00

Author

Owner

Tracking moved from #54: add a hermetic regression for a legacy active generation whose configurationRevision is missing. The test must prove exactly one activation records the candidate revision and that the immediate follow-up exits before build.

Tracking moved from #54: add a hermetic regression for a legacy active generation whose `configurationRevision` is missing. The test must prove exactly one activation records the candidate revision and that the immediate follow-up exits before build.

nimmo referenced this issue

2026-07-16 13:30:17 +01:00

High: restore reliable deployed-revision detection and Git flake provenance #54

nimmo referenced this issue

2026-07-16 13:30:21 +01:00

High: restore reliable deployed-revision detection and Git flake provenance #54

nimmo commented

2026-07-17 06:32:12 +01:00

Author

Owner

Completed and validated.

Implemented in:

769e8c3 — added table-driven workflow routing, shell regression coverage, and the required NixOS KVM state-machine check.
19363f8 — fixed Forgejo CI environment isolation and made the regression/lint job a prerequisite gate for the Attic build, with ntfy notification on gate failure.

Coverage now includes:

input policy and affected-host inventory validation;
partial acceptance and failure classification;
fast-forward-only consumer Git handling and exact lease-aware producer publication;
workflow routing for push, schedule, manual current/update, force, and updater-loop suppression;
detached activation, hold races, startup/switch failures, cleanup, and minimal environments;
legacy missing revision causing one activation followed by no-build convergence;
hermetic emergency producer update, validation, local bare-remote publication, and activation.

Validation:

local ShellCheck and actionlint passed;
the full shell regression suite passed under Forgejo-style inherited environment variables;
full nix flake check passed across all four host configurations and the KVM state-machine test;
Forgejo run #324 passed the regression gate and the dependent Attic build.

The emergency path is no longer documented as untested. The regression gate now blocks closure builds and publication when validation fails.

Completed and validated. Implemented in: - `769e8c3` — added table-driven workflow routing, shell regression coverage, and the required NixOS KVM state-machine check. - `19363f8` — fixed Forgejo CI environment isolation and made the regression/lint job a prerequisite gate for the Attic build, with ntfy notification on gate failure. Coverage now includes: - input policy and affected-host inventory validation; - partial acceptance and failure classification; - fast-forward-only consumer Git handling and exact lease-aware producer publication; - workflow routing for push, schedule, manual current/update, force, and updater-loop suppression; - detached activation, hold races, startup/switch failures, cleanup, and minimal environments; - legacy missing revision causing one activation followed by no-build convergence; - hermetic emergency producer update, validation, local bare-remote publication, and activation. Validation: - local ShellCheck and actionlint passed; - the full shell regression suite passed under Forgejo-style inherited environment variables; - full `nix flake check` passed across all four host configurations and the KVM state-machine test; - Forgejo run #324 passed the regression gate and the dependent Attic build. The emergency path is no longer documented as untested. The regression gate now blocks closure builds and publication when validation fails.

nimmo closed this issue

2026-07-17 06:32:21 +01:00

Rows
Columns

Add automated tests and linting for update scripts and Forgejo workflow #56

Context

Scope

Completion criteria