nimmo/nixos-config

Watch

Fork 0

feat: capture rejected update incident evidence #133

Merged

nimmo merged 2 commits from issue-129-structured-update-evidence into main

2026-08-01 04:50:31 +01:00

nimmo commented

2026-08-01 03:12:59 +01:00

Owner

Summary

emit a deterministic, atomically published Nova producer incident bundle only when an isolated flake-input candidate is rejected
retain baseline/rejected lock identities, input revisions, group and host/build metadata, bounded sanitised excerpts, classification, derivation metadata, and a stable Forgejo deduplication fingerprint
retain the exact rejected flake.lock only when it is valid, bounded, and free of credential-like content; otherwise preserve explicit reproducibility metadata without raw secrets
preserve existing infrastructure treatment, accepted-input isolation, atomic lock publication, and partial-success behaviour

Validation

./tests/auto-update-regressions.sh
./tests/nova-remediation-benchmark.sh
nix fmt -- --ci
ShellCheck and actionlint via nix shell nixpkgs#shellcheck nixpkgs#actionlint
nix shell nixpkgs#forgejo-runner -c forgejo-runner validate --directory .
nix flake check
git diff --check

Live/deployment status

No deployment was forced. This changes only Nova producer evidence handling; normal even-hour auto-update observation remains the required post-merge deployment evidence.

Closes #129

## Summary - emit a deterministic, atomically published Nova producer incident bundle only when an isolated flake-input candidate is rejected - retain baseline/rejected lock identities, input revisions, group and host/build metadata, bounded sanitised excerpts, classification, derivation metadata, and a stable Forgejo deduplication fingerprint - retain the exact rejected `flake.lock` only when it is valid, bounded, and free of credential-like content; otherwise preserve explicit reproducibility metadata without raw secrets - preserve existing infrastructure treatment, accepted-input isolation, atomic lock publication, and partial-success behaviour ## Validation - `./tests/auto-update-regressions.sh` - `./tests/nova-remediation-benchmark.sh` - `nix fmt -- --ci` - ShellCheck and actionlint via `nix shell nixpkgs#shellcheck nixpkgs#actionlint` - `nix shell nixpkgs#forgejo-runner -c forgejo-runner validate --directory .` - `nix flake check` - `git diff --check` ## Live/deployment status No deployment was forced. This changes only Nova producer evidence handling; normal even-hour auto-update observation remains the required post-merge deployment evidence. Closes #129

nimmo added this to the Automated Update Remediation milestone

2026-08-01 03:12:59 +01:00

nimmo added the

phase:prep

host:nova

project:auto-update-remediation

service:auto-update

labels

2026-08-01 03:12:59 +01:00

nimmo added 1 commit

2026-08-01 03:13:03 +01:00

feat: capture rejected update incident evidence

Build and Push Attic Cache / test (pull_request) Successful in 1m8s

Details

Build and Push Attic Cache / build (pull_request) Has been skipped

Details

AI pull-request review / review (pull_request) Successful in 5m35s

Details

a1bf587112

nimmo commented

2026-08-01 03:18:39 +01:00

Author

Owner

Automated code review

Reviewed commit: 560fe9aeb4b98c2371af776fbbeacef8f0ad3853

Verdict: Ready to merge

No blocking or important findings were retained, and deterministic checks passed. Suggestions remain advisory.

Overall assessment

Introduces a new incident bundle script to capture deterministic, sanitized evidence when flake-input candidates are rejected during auto-update isolation. Adds regression tests and updates workflow triggers. The implementation correctly bounds output size, sanitizes untrusted build logs, deduplicates via fingerprint, and handles missing or malformed artifacts gracefully.

Blocking findings

None.

Important findings

None.

Suggestions

Hardcoded derivation limit in evidence extraction (scripts/nixos-update-incident-bundle.sh:83; medium confidence)
grep -oE '/nix/store/[a-z0-9]{32}-[^'[:space:]]+.drv' "$log" 2>/dev/null | sort -u | head -20)
Impact: Inconsistent bounding policy; may retain too many or too few derivation paths depending on log volume, unlike the configurable excerpt limits.
Suggested fix: Introduce NIXOS_UPDATE_INCIDENT_MAX_DERIVATIONS and apply it consistently alongside other size bounds.
GNU-specific sed flags in sanitization function (scripts/nixos-update-incident-bundle.sh:13; medium confidence)
sed -E \n -e 's/\x1B0-9;]*[[:alpha://g' ... \n -e 's#(token|password|secret)[=:][^[:space:]]+#\1=#Ig'
Impact: Potential failure if sourced in non-GNU environments (e.g., macOS or BSD CI runners) due to -E and I flag usage.
Suggested fix: Replace with POSIX-compatible regex syntax or add an environment guard to ensure GNU sed is available.

Tests and validation

No configured deterministic checks.

Questions

Does the Forgejo runner environment guarantee GNU coreutils and sed, or should POSIX compatibility be enforced for cross-platform CI runners?
Should the incident bundle emission be wrapped in a set -e trap or explicitly logged if it fails, to ensure operators are aware when evidence capture is skipped due to transient I/O or permission issues?

Review limitations

Review is based on provided diff and repository documentation; runtime behavior depends on NixOS environment variables and Forgejo runner state not fully visible here.
ShellCheck and nix flake check validation results were not supplied, so lint compliance and Nix evaluation impacts are assumed correct per PR description.

Diff coverage

.forgejo/workflows/attic-cache.yml: reviewed — included in a context-limited batch
README.md: reviewed — included in a context-limited batch
scripts/attic-cache-build.sh: reviewed — included in a context-limited batch
scripts/nixos-update-common.sh: reviewed — included in a context-limited batch
scripts/nixos-update-incident-bundle.sh: reviewed — included in a context-limited batch
tests/auto-update-regressions.sh: reviewed — included in a context-limited batch

Review metadata

Reviewer: forgejo-ai-review v1
Model: qwen3.6:35b-a3b-q8_0
Guidance loaded: .forgejo/ai-review.md, .forgejo/ai-review.yml, AGENTS.md, README.md
Commits considered: 2
Previous automated review: updated
Findings: 2

## Automated code review Reviewed commit: `560fe9aeb4b98c2371af776fbbeacef8f0ad3853` ## Verdict: Ready to merge No blocking or important findings were retained, and deterministic checks passed. Suggestions remain advisory. ### Overall assessment Introduces a new incident bundle script to capture deterministic, sanitized evidence when flake-input candidates are rejected during auto-update isolation. Adds regression tests and updates workflow triggers. The implementation correctly bounds output size, sanitizes untrusted build logs, deduplicates via fingerprint, and handles missing or malformed artifacts gracefully. ### Blocking findings None. ### Important findings None. ### Suggestions - **Hardcoded derivation limit in evidence extraction** (scripts/nixos-update-incident-bundle.sh:83; medium confidence) grep -oE '/nix/store/[a-z0-9]{32}-[^'[:space:]]+\.drv' "$log" 2>/dev/null | sort -u | head -20) Impact: Inconsistent bounding policy; may retain too many or too few derivation paths depending on log volume, unlike the configurable excerpt limits. Suggested fix: Introduce NIXOS_UPDATE_INCIDENT_MAX_DERIVATIONS and apply it consistently alongside other size bounds. - **GNU-specific sed flags in sanitization function** (scripts/nixos-update-incident-bundle.sh:13; medium confidence) sed -E \n -e 's/\x1B\[[0-9;]*[[:alpha:]]//g' ... \n -e 's#(token|password|secret)[=:][^[:space:]]+#\1=<redacted>#Ig' Impact: Potential failure if sourced in non-GNU environments (e.g., macOS or BSD CI runners) due to -E and I flag usage. Suggested fix: Replace with POSIX-compatible regex syntax or add an environment guard to ensure GNU sed is available. ### Tests and validation - No configured deterministic checks. ### Questions - Does the Forgejo runner environment guarantee GNU coreutils and sed, or should POSIX compatibility be enforced for cross-platform CI runners? - Should the incident bundle emission be wrapped in a set -e trap or explicitly logged if it fails, to ensure operators are aware when evidence capture is skipped due to transient I/O or permission issues? ### Review limitations - Review is based on provided diff and repository documentation; runtime behavior depends on NixOS environment variables and Forgejo runner state not fully visible here. - ShellCheck and nix flake check validation results were not supplied, so lint compliance and Nix evaluation impacts are assumed correct per PR description. ### Diff coverage - `.forgejo/workflows/attic-cache.yml`: **reviewed** — included in a context-limited batch - `README.md`: **reviewed** — included in a context-limited batch - `scripts/attic-cache-build.sh`: **reviewed** — included in a context-limited batch - `scripts/nixos-update-common.sh`: **reviewed** — included in a context-limited batch - `scripts/nixos-update-incident-bundle.sh`: **reviewed** — included in a context-limited batch - `tests/auto-update-regressions.sh`: **reviewed** — included in a context-limited batch <details><summary>Review metadata</summary> - Reviewer: forgejo-ai-review v1 - Model: qwen3.6:35b-a3b-q8_0 - Guidance loaded: .forgejo/ai-review.md, .forgejo/ai-review.yml, AGENTS.md, README.md - Commits considered: 2 - Previous automated review: updated - Findings: 2 </details>

nimmo force-pushed issue-129-structured-update-evidence from a1bf587112

Build and Push Attic Cache / test (pull_request) Successful in 1m8s

Details

Build and Push Attic Cache / build (pull_request) Has been skipped

Details

AI pull-request review / review (pull_request) Successful in 5m35s

Details

to 560fe9aeb4

Build and Push Attic Cache / test (pull_request) Successful in 44s

Details

Build and Push Attic Cache / build (pull_request) Has been skipped

Details

AI pull-request review / review (pull_request) Successful in 5m10s

Details

2026-08-01 04:43:22 +01:00

Compare

nimmo commented

2026-08-01 04:43:34 +01:00

Author

Owner

Resolved the automated-review findings in 560fe9a.

build_host now sets NIXOS_UPDATE_FAILED_HOST="$host" before the retry wrapper; the regression asserts the wrapper observes that value, so a post-failure-only assignment would fail.
Replaced the test-only rg use with grep, already required by the updater runtime and used throughout the regression suite.
Kept the conservative sanitizer boundary: any token=, password=, or secret= assignment is redacted even if benign. The rationale is documented beside the sanitizer, and a regression proves a benign-looking token=nonsecret-fixture-value is withheld.

Revalidated: formatter, ShellCheck, actionlint, Forgejo workflow validation, auto-update and Nova benchmark regressions, nix flake check, and git diff --check.

Resolved the automated-review findings in 560fe9a. - `build_host` now sets `NIXOS_UPDATE_FAILED_HOST="$host"` before the retry wrapper; the regression asserts the wrapper observes that value, so a post-failure-only assignment would fail. - Replaced the test-only `rg` use with `grep`, already required by the updater runtime and used throughout the regression suite. - Kept the conservative sanitizer boundary: any `token=`, `password=`, or `secret=` assignment is redacted even if benign. The rationale is documented beside the sanitizer, and a regression proves a benign-looking `token=nonsecret-fixture-value` is withheld. Revalidated: formatter, ShellCheck, actionlint, Forgejo workflow validation, auto-update and Nova benchmark regressions, `nix flake check`, and `git diff --check`.

nimmo merged commit 4fcb27611c into main

2026-08-01 04:50:31 +01:00

nimmo referenced this pull request from a commit

2026-08-01 04:50:32 +01:00

Merge pull request 'feat: capture rejected update incident evidence' (#133) from issue-129-structured-update-evidence into main

nimmo referenced this pull request

2026-08-01 04:51:09 +01:00

Capture structured evidence for rejected flake updates #129

nimmo referenced this pull request

2026-08-01 06:11:22 +01:00

Capture structured evidence for rejected flake updates #129

nimmo referenced this pull request