Isolate failing flake input groups after combined candidate validation fails

nimmo commented

2026-07-24 17:17:03 +01:00

Owner

Context

The scheduled update on 2026-07-24 found a fixed-output paseo-0.2.0-npm-deps hash mismatch while building Electra. The updater correctly restored the original flake.lock, but because all changed input groups were validated as one combined candidate, the Paseo failure rejected every group. This leaves otherwise valid updates unpublishable until the unrelated upstream issue is resolved.

The current combined-candidate design was deliberately introduced for efficient normal updates: all groups update one lock candidate, affected hosts are deduplicated, and each host is built at most once. Keep that fast path.

Desired behaviour

Use a combined-first, split-on-failure validation strategy:

Update all eligible groups into one candidate lock and build the deduplicated affected-host set once.
If it validates, accept and publish exactly as today.
If it fails, restore the original lock and isolate the failure by retrying smaller candidates, beginning with individual groups or another deterministic split strategy.
Build a final candidate containing every independently accepted group, validate the relevant deduplicated hosts, then publish only that validated lock.
Record rejected groups with the failure context; accepted groups must remain publishable even when another group is broken upstream.

The 2026-07-24 Paseo update is the reference case: reject shared-ai-tooling while allowing unrelated groups to advance if their final combined candidate validates.

Scope

Refactor scripts/nixos-update-common.sh so candidate construction, host derivation, validation, rollback, and outcome recording can be reused during isolation.
Preserve atomic publication guarantees: never publish a lock that has not been validated as its final combined state.
Keep the happy path to one combined validation build.
Make fallback ordering deterministic from lib/auto-update-policy.nix.
Distinguish initial combined validation failure, isolated group rejection, and final-candidate validation failure in logs, Forgejo summaries, commit messages, and ntfy output.
Ensure groups that do not change retain the existing unchanged outcome.
Do not silently accept a group merely because it passes alone if it conflicts with already accepted groups; final validation remains authoritative.

Non-goals

No change to which inputs or hosts are covered by policy.
No automatic upstream pinning or source patching.
No parallel candidate builds in the initial implementation; optimise only after correctness and clear observability are established.

Acceptance criteria

A successful combined candidate still performs one deduplicated host build and publishes normally.
With one failing group and one or more valid groups, valid groups are committed/published and the failing group is rejected with a useful reason.
The final published lock is validated after all accepted groups are assembled.
A group that passes alone but fails in the final combination is not published; reporting identifies it as unresolved/rejected.
Any infrastructure failure during input resolution or required validation fails safely and does not publish an unvalidated lock.
Regression coverage exercises:
- all-groups combined success;
- combined failure followed by one rejected and one accepted group;
- multiple rejected groups;
- final re-combination failure;
- no-change behaviour and existing summary/exit semantics.
ShellCheck, updater regression tests, actionlint, and nix flake check pass.
Before closing, run one genuine Forgejo update/deployment and a no-op follow-up, per repository policy.

Implementation outline

Extract candidate-lock snapshot/restore and candidate host computation from nixos_update_run_input_groups.
Represent proposed group metadata once, then apply it to a scratch candidate in deterministic policy order.
First validate the all-group candidate.
On failure, restore the baseline; trial each proposed group in policy order against the accumulating accepted candidate, recording rejection immediately on failed trial validation.
Rebuild and validate the final accepted candidate before commit/push, because accepted groups may interact.
Reuse existing outcome and notification plumbing, extending detail strings to expose the isolation phase and failed host.

Trade-off

Failures become more expensive: a normal successful run stays at one validation pass, but a failed combined candidate triggers extra builds to isolate it. This is intentional: cheap in the common case, more diagnostic and resilient only when something goes wrong.

## Context The scheduled update on 2026-07-24 found a fixed-output `paseo-0.2.0-npm-deps` hash mismatch while building Electra. The updater correctly restored the original `flake.lock`, but because all changed input groups were validated as one combined candidate, the Paseo failure rejected every group. This leaves otherwise valid updates unpublishable until the unrelated upstream issue is resolved. The current combined-candidate design was deliberately introduced for efficient normal updates: all groups update one lock candidate, affected hosts are deduplicated, and each host is built at most once. Keep that fast path. ## Desired behaviour Use a combined-first, split-on-failure validation strategy: 1. Update all eligible groups into one candidate lock and build the deduplicated affected-host set once. 2. If it validates, accept and publish exactly as today. 3. If it fails, restore the original lock and isolate the failure by retrying smaller candidates, beginning with individual groups or another deterministic split strategy. 4. Build a final candidate containing every independently accepted group, validate the relevant deduplicated hosts, then publish only that validated lock. 5. Record rejected groups with the failure context; accepted groups must remain publishable even when another group is broken upstream. The 2026-07-24 Paseo update is the reference case: reject `shared-ai-tooling` while allowing unrelated groups to advance if their final combined candidate validates. ## Scope - Refactor `scripts/nixos-update-common.sh` so candidate construction, host derivation, validation, rollback, and outcome recording can be reused during isolation. - Preserve atomic publication guarantees: never publish a lock that has not been validated as its final combined state. - Keep the happy path to one combined validation build. - Make fallback ordering deterministic from `lib/auto-update-policy.nix`. - Distinguish initial combined validation failure, isolated group rejection, and final-candidate validation failure in logs, Forgejo summaries, commit messages, and ntfy output. - Ensure groups that do not change retain the existing `unchanged` outcome. - Do not silently accept a group merely because it passes alone if it conflicts with already accepted groups; final validation remains authoritative. ## Non-goals - No change to which inputs or hosts are covered by policy. - No automatic upstream pinning or source patching. - No parallel candidate builds in the initial implementation; optimise only after correctness and clear observability are established. ## Acceptance criteria - A successful combined candidate still performs one deduplicated host build and publishes normally. - With one failing group and one or more valid groups, valid groups are committed/published and the failing group is rejected with a useful reason. - The final published lock is validated after all accepted groups are assembled. - A group that passes alone but fails in the final combination is not published; reporting identifies it as unresolved/rejected. - Any infrastructure failure during input resolution or required validation fails safely and does not publish an unvalidated lock. - Regression coverage exercises: - all-groups combined success; - combined failure followed by one rejected and one accepted group; - multiple rejected groups; - final re-combination failure; - no-change behaviour and existing summary/exit semantics. - ShellCheck, updater regression tests, actionlint, and `nix flake check` pass. - Before closing, run one genuine Forgejo update/deployment and a no-op follow-up, per repository policy. ## Implementation outline - Extract candidate-lock snapshot/restore and candidate host computation from `nixos_update_run_input_groups`. - Represent proposed group metadata once, then apply it to a scratch candidate in deterministic policy order. - First validate the all-group candidate. - On failure, restore the baseline; trial each proposed group in policy order against the accumulating accepted candidate, recording rejection immediately on failed trial validation. - Rebuild and validate the final accepted candidate before commit/push, because accepted groups may interact. - Reuse existing outcome and notification plumbing, extending detail strings to expose the isolation phase and failed host. ## Trade-off Failures become more expensive: a normal successful run stays at one validation pass, but a failed combined candidate triggers extra builds to isolate it. This is intentional: cheap in the common case, more diagnostic and resilient only when something goes wrong.

nimmo added this to the Automated Update Reliability Review milestone

2026-07-24 17:17:03 +01:00

nimmo added the

project:auto-update-reliability

service:auto-update

host:fleet

labels

2026-07-24 17:17:03 +01:00

nimmo referenced this issue from a commit

2026-07-24 18:46:45 +01:00

feat(auto-update): isolate failing flake input groups after combined failure

nimmo referenced this issue