nimmo/nixos-config

Watch

Fork 0

Preserve evaluation evidence and distinguish verified remediation findings #205

New issue

Closed

opened 2026-09-08 21:07:13 +01:00 by nimmo · 1 comment

nimmo commented

2026-09-08 21:07:13 +01:00

Owner

Context

Follow-up to #204 and the result-output fix in #201/#202. The model now returns a valid diagnosis, but evidence omits evaluation traces and the report confuses affected hosts with observed failures.

Scope

Enable host build evaluation traces.
Improve the shared nixos-auto-update excerpt selection to retain the final error and nearby evaluation context within existing redaction and size limits.
Require the worker to separate observations, hypotheses, and unknowns, preserving exact attribute names.
Publish recorded failure and affected-host scope separately.
Keep automatic repair eligibility unchanged.

Validation and completion

Regression tests for long traces, final-error retention, redaction, limits, and report host semantics.
Relevant shell lint, remediation regressions, and Nix checks.
Replay the retained #204 candidate without publishing or deploying; document what the richer trace proves.
Publish coordinated reviewable changes in both repositories only when requested; merge/deployment evidence remains outstanding until integration.

Prepared changes and evidence

nixos-config: 7bc9a856b2e2f7ecc15e507db0e75e93c22d44d7 on fix/remediation-evaluation-evidence (local).
nixos-auto-update: cd61d13c0ad13532504ab3f73d91e144fdc3409e on fix/evaluation-evidence (local).
Passed: both regression suites against the modified updater package, ShellCheck, formatting, and final combined nix flake check including the auto-update VM test. Validation used a local input override; flake.lock is unchanged.
Exact #204 offline evaluation replay reproduced the failure. The new excerpt retains pi-coding-agent-0.84.4, nativeBuildInputs, and the typescript-go rename error within 24 lines/8192 bytes. This verifies improved evidence capture; no new model run or live deployment was performed.
Remaining: publish coordinated branches/MRs, update the nixos-auto-update input pin after the revision is reachable, complete review/CI, and collect post-integration live evidence. Existing upstream deprecation warnings are outside this reporting change.

Real-model replay evidence (2026-09-09)

Two separate disposable, non-publishing replays used real OpenCode/Bubblewrap/Ollama and qwen3.6:35b-a3b-q8_0 under the unchanged 600-second model deadline. The original #204 incident and state hashes were unchanged.

Improved excerpt in JSON only: completed in 468.666 seconds, return code 0, diagnosis-ready, but FAILED cause identification. The model reported a 2000-character cutoff and could not see the final error. OpenCode 1.18.10 truncates each read-tool line at 2000 characters; JSON encodes the excerpt on one line. The package name and final error lay beyond that cutoff.
Same evidence with the new read-only build-output.txt and prompt instruction: completed in 239.353 seconds, return code 0, no timeout/resource-limit breach. PASSED core cause identification: pi-coding-agent-0.84.4, nativeBuildInputs, and the typescript-go rename/replacement by typescript. It named Electra as failed and treated the other hosts as affected scope. It correctly remained diagnosis-only with no patch or publication.

The detailed dependency-chain reconstruction and model confidence/classification remain advisory; this test does not independently validate every causal edge claimed by the model or demonstrate repair eligibility.

The controller now supplies real line breaks and wraps diagnostic lines at 512 characters without dropping content. Regression coverage emulates OpenCode's per-line clipping and verifies the final error survives, text remains read-only, and the original evidence characters are preserved. The remediation regression/integration suite passed after this fix. This Python/prompt follow-up changes no Nix configuration; the prior combined flake/VM checks remain recorded above.

No branches were pushed, no MR was opened, no configuration was deployed, and no scheduled production incident was reprocessed. Coordinated publication and input-pin integration remain pending.

Historical package replay: Moonlight #172 (2026-09-09)

User accepted keeping evaluation failures diagnosis-only and requested a replay of a historical package failure. Selected #172 (928eda57...), the Moonlight 6.1.0 / FFmpeg 9 compilation failure from before the manually approved #170 compatibility pin. Used the unchanged retained incident, exact baseline 999961b155, and rejected candidate lock in a disposable Nova repository with the updated controller/prompt. No policy or eligibility data was altered.

Real qwen3.6:35b-a3b-q8_0 replay completed diagnosis-ready in 229.356 seconds, exit code 0, no timeout or resource-limit breach. It identified Moonlight's use of the removed FFmpeg AVCodec.pix_fmts API, but classified the incident as core-system-regression (culprit ffmpeg), returned diagnosis-only, and proposed no patch. Controller reason: exactly one failing derivation is required. No baseline/patched build validation or temporary-pin generation occurred.

Confirmed blockers:

The incident lists 20 derivations, including build-plan/system dependencies, while retained compiler errors identify Moonlight. Shared nixos_update_incident_derivations_json extracts every .drv occurrence, sorts, and takes the first 20; it does not identify actual failed builders. A deterministic local probe with one failed builder and three planned derivations reproduced the false multi-failure result. These entries must not simply be presumed to be cascade failures.
At the historical base Moonlight is listed in modules/profiles/gaming.nix, outside the permitted automatic patch paths. There is no existing dedicated leaf package file available to patch.
The model's core-library classification overlooks the possibility of scoping an older dependency to the leaf application. #170 demonstrates that a Moonlight-only ffmpeg_8 override was viable without pinning FFmpeg globally. Its automated lifecycle comment on 2026-08-22 reports successful unpinned validation and preparation of a removal PR; that proves historical removal automation, not automatic creation of the original pin.

Original incident hashes unchanged. Assessment retained locally at /tmp/nixos-172-real-model-assessment.json; disposable remote replay cleaned up. No MR publication, deployment, eligibility relaxation, or implementation changes in this replay. Next work should correct actual-failed-builder attribution (preserving rejection of genuinely multiple independent failures) and assess a narrowly permitted leaf compatibility-override path before expecting an end-to-end automatic pin proposal.

## Context Follow-up to #204 and the result-output fix in #201/#202. The model now returns a valid diagnosis, but evidence omits evaluation traces and the report confuses affected hosts with observed failures. ## Scope - Enable host build evaluation traces. - Improve the shared nixos-auto-update excerpt selection to retain the final error and nearby evaluation context within existing redaction and size limits. - Require the worker to separate observations, hypotheses, and unknowns, preserving exact attribute names. - Publish recorded failure and affected-host scope separately. - Keep automatic repair eligibility unchanged. ## Validation and completion - Regression tests for long traces, final-error retention, redaction, limits, and report host semantics. - Relevant shell lint, remediation regressions, and Nix checks. - Replay the retained #204 candidate without publishing or deploying; document what the richer trace proves. - Publish coordinated reviewable changes in both repositories only when requested; merge/deployment evidence remains outstanding until integration. ## Prepared changes and evidence - nixos-config: 7bc9a856b2e2f7ecc15e507db0e75e93c22d44d7 on fix/remediation-evaluation-evidence (local). - nixos-auto-update: cd61d13c0ad13532504ab3f73d91e144fdc3409e on fix/evaluation-evidence (local). - Passed: both regression suites against the modified updater package, ShellCheck, formatting, and final combined nix flake check including the auto-update VM test. Validation used a local input override; flake.lock is unchanged. - Exact #204 offline evaluation replay reproduced the failure. The new excerpt retains pi-coding-agent-0.84.4, nativeBuildInputs, and the typescript-go rename error within 24 lines/8192 bytes. This verifies improved evidence capture; no new model run or live deployment was performed. - Remaining: publish coordinated branches/MRs, update the nixos-auto-update input pin after the revision is reachable, complete review/CI, and collect post-integration live evidence. Existing upstream deprecation warnings are outside this reporting change. ## Real-model replay evidence (2026-09-09) Two separate disposable, non-publishing replays used real OpenCode/Bubblewrap/Ollama and qwen3.6:35b-a3b-q8_0 under the unchanged 600-second model deadline. The original #204 incident and state hashes were unchanged. 1. Improved excerpt in JSON only: completed in 468.666 seconds, return code 0, diagnosis-ready, but FAILED cause identification. The model reported a 2000-character cutoff and could not see the final error. OpenCode 1.18.10 truncates each read-tool line at 2000 characters; JSON encodes the excerpt on one line. The package name and final error lay beyond that cutoff. 2. Same evidence with the new read-only build-output.txt and prompt instruction: completed in 239.353 seconds, return code 0, no timeout/resource-limit breach. PASSED core cause identification: pi-coding-agent-0.84.4, nativeBuildInputs, and the typescript-go rename/replacement by typescript. It named Electra as failed and treated the other hosts as affected scope. It correctly remained diagnosis-only with no patch or publication. The detailed dependency-chain reconstruction and model confidence/classification remain advisory; this test does not independently validate every causal edge claimed by the model or demonstrate repair eligibility. The controller now supplies real line breaks and wraps diagnostic lines at 512 characters without dropping content. Regression coverage emulates OpenCode's per-line clipping and verifies the final error survives, text remains read-only, and the original evidence characters are preserved. The remediation regression/integration suite passed after this fix. This Python/prompt follow-up changes no Nix configuration; the prior combined flake/VM checks remain recorded above. No branches were pushed, no MR was opened, no configuration was deployed, and no scheduled production incident was reprocessed. Coordinated publication and input-pin integration remain pending. ## Historical package replay: Moonlight #172 (2026-09-09) User accepted keeping evaluation failures diagnosis-only and requested a replay of a historical package failure. Selected #172 (928eda57...), the Moonlight 6.1.0 / FFmpeg 9 compilation failure from before the manually approved #170 compatibility pin. Used the unchanged retained incident, exact baseline 999961b1558e771eae152b3653c6c8799da7d033, and rejected candidate lock in a disposable Nova repository with the updated controller/prompt. No policy or eligibility data was altered. Real qwen3.6:35b-a3b-q8_0 replay completed diagnosis-ready in 229.356 seconds, exit code 0, no timeout or resource-limit breach. It identified Moonlight's use of the removed FFmpeg AVCodec.pix_fmts API, but classified the incident as core-system-regression (culprit ffmpeg), returned diagnosis-only, and proposed no patch. Controller reason: exactly one failing derivation is required. No baseline/patched build validation or temporary-pin generation occurred. Confirmed blockers: - The incident lists 20 derivations, including build-plan/system dependencies, while retained compiler errors identify Moonlight. Shared nixos_update_incident_derivations_json extracts every .drv occurrence, sorts, and takes the first 20; it does not identify actual failed builders. A deterministic local probe with one failed builder and three planned derivations reproduced the false multi-failure result. These entries must not simply be presumed to be cascade failures. - At the historical base Moonlight is listed in modules/profiles/gaming.nix, outside the permitted automatic patch paths. There is no existing dedicated leaf package file available to patch. - The model's core-library classification overlooks the possibility of scoping an older dependency to the leaf application. #170 demonstrates that a Moonlight-only ffmpeg_8 override was viable without pinning FFmpeg globally. Its automated lifecycle comment on 2026-08-22 reports successful unpinned validation and preparation of a removal PR; that proves historical removal automation, not automatic creation of the original pin. Original incident hashes unchanged. Assessment retained locally at /tmp/nixos-172-real-model-assessment.json; disposable remote replay cleaned up. No MR publication, deployment, eligibility relaxation, or implementation changes in this replay. Next work should correct actual-failed-builder attribution (preserving rejection of genuinely multiple independent failures) and assess a narrowly permitted leaf compatibility-override path before expecting an end-to-end automatic pin proposal.

nimmo added this to the Automated Update Remediation milestone

2026-09-08 21:07:13 +01:00

nimmo added the

project:auto-update-remediation

service:auto-update

labels

2026-09-08 21:13:21 +01:00

nimmo referenced this issue from a commit

2026-09-09 05:07:54 +01:00

fix: retain named evaluation context and final errors

nimmo referenced this issue from nimmo/nixos-auto-update

2026-09-09 05:09:26 +01:00

Preserve useful evaluation context in rejected-update evidence #1

nimmo referenced this issue

2026-09-09 05:13:33 +01:00

Improve rejected-update evidence and verify real model diagnosis #207

nimmo referenced this issue from a commit

2026-09-09 05:13:35 +01:00

fix: improve rejected-update evidence and diagnosis boundaries

nimmo referenced this issue from a commit

2026-09-09 05:13:36 +01:00

fix: expose diagnostic text without read-tool truncation

nimmo referenced this issue from a commit

2026-09-09 05:13:36 +01:00

chore: pin published remediation evidence helper

nimmo referenced this issue from a commit

2026-09-09 17:04:55 +01:00

Address automated review of remediation evidence MR

nimmo commented

2026-09-09 22:16:55 +01:00

Author

Owner

Closure evidence: Electra deployed revision 4ba715cb38, containing the merged evaluation-evidence integration, at 20:11 on 2026-09-09. Its 22:15 normal follow-up confirmed origin/main and the running system were current at 4a03f7ebc9, with no build or activation and a successful exit.

Closure evidence: Electra deployed revision 4ba715cb3812b7e86021e2a4b47054903ff66ee7, containing the merged evaluation-evidence integration, at 20:11 on 2026-09-09. Its 22:15 normal follow-up confirmed origin/main and the running system were current at 4a03f7ebc909779e6290d436868d5509418604b8, with no build or activation and a successful exit.

nimmo closed this issue

2026-09-09 22:17:08 +01:00