nimmo/nixos-config

Watch

Fork 0

Benchmark Nova update-remediation models #132

Merged

nimmo merged 1 commit from issue-128-nova-remediation-benchmark into main

2026-08-01 02:53:59 +01:00

nimmo commented

2026-08-01 02:47:07 +01:00

Owner

Summary

add a reproducible, isolated OpenCode/Ollama benchmark controller with seven prompt-injection-bearing fixtures and hidden deterministic outcomes
enforce bounded tools/worktrees, clean environment, serial model trials, hard safety disqualifiers, sanitized artifacts, and mandatory Ollama unload verification
compare quality-first single-model and conservative cascade policies using 63 live Nova runs
select qwen3.6:35b-a3b-q8_0 as the single advisory model; reject a cascade
add the focused regression to the Nova Attic workflow path

Live benchmark evidence

OpenCode 1.18.10 on Nova completed 3 models x 7 cases x 3 repetitions in 14,726 seconds. Raw events/model prose were not retained, all nine trial boundaries released the model, and a final ollama ps was empty.

Model	Hard eligible	Quality / 85	Total / 100	Exact oracle	Median	Peak VRAM
Qwen 3.6 35B A3B Q8	yes	77.2	89.3	17/21	100.1s	40.6 GiB
North Mini Code Q8	yes	56.2	70.2	9/21	132.1s	30.9 GiB
Gemma 4 31B	no: unbounded-loop	75.8	85.1	17/21	350.3s	19.1 GiB

Qwen had no hard disqualifier. Its four misses were rejected by the deterministic controller: three correct infrastructure refusals omitted the expected builder culprit, and one source override never passed exact validation or claimed validated success. The best eligible cascade (North then Qwen) was worse at 72.9/85 quality, 66.7% exact pass, and 185.9s median latency. The LLM remains advisory; deterministic code retains validation, credentials, publication, and deployment authority.

qwen3.5:122b remains explicitly excluded because its prior approximately 87.3 GB allocation exceeds Nova hardware.

Validation

./tests/nova-remediation-benchmark.sh
./tests/auto-update-regressions.sh
nix fmt -- --ci
ShellCheck and actionlint using the workflow command
forgejo-runner validate --directory .
ruff check scripts/nova-remediation-benchmark.py
nix flake check (all checks and host evaluations passed)
git diff --check

Live/deployment status

No deployment is required or performed: this adds an offline benchmark, frozen evidence, tests, and CI routing only; no NixOS module imports the harness. No model received Forgejo credentials or access to main.

Closes #128

## Summary - add a reproducible, isolated OpenCode/Ollama benchmark controller with seven prompt-injection-bearing fixtures and hidden deterministic outcomes - enforce bounded tools/worktrees, clean environment, serial model trials, hard safety disqualifiers, sanitized artifacts, and mandatory Ollama unload verification - compare quality-first single-model and conservative cascade policies using 63 live Nova runs - select `qwen3.6:35b-a3b-q8_0` as the single advisory model; reject a cascade - add the focused regression to the Nova Attic workflow path ## Live benchmark evidence OpenCode 1.18.10 on Nova completed 3 models x 7 cases x 3 repetitions in 14,726 seconds. Raw events/model prose were not retained, all nine trial boundaries released the model, and a final `ollama ps` was empty. | Model | Hard eligible | Quality / 85 | Total / 100 | Exact oracle | Median | Peak VRAM | |---|---:|---:|---:|---:|---:|---:| | Qwen 3.6 35B A3B Q8 | yes | 77.2 | 89.3 | 17/21 | 100.1s | 40.6 GiB | | North Mini Code Q8 | yes | 56.2 | 70.2 | 9/21 | 132.1s | 30.9 GiB | | Gemma 4 31B | no: unbounded-loop | 75.8 | 85.1 | 17/21 | 350.3s | 19.1 GiB | Qwen had no hard disqualifier. Its four misses were rejected by the deterministic controller: three correct infrastructure refusals omitted the expected `builder` culprit, and one source override never passed exact validation or claimed validated success. The best eligible cascade (North then Qwen) was worse at 72.9/85 quality, 66.7% exact pass, and 185.9s median latency. The LLM remains advisory; deterministic code retains validation, credentials, publication, and deployment authority. `qwen3.5:122b` remains explicitly excluded because its prior approximately 87.3 GB allocation exceeds Nova hardware. ## Validation - `./tests/nova-remediation-benchmark.sh` - `./tests/auto-update-regressions.sh` - `nix fmt -- --ci` - ShellCheck and actionlint using the workflow command - `forgejo-runner validate --directory .` - `ruff check scripts/nova-remediation-benchmark.py` - `nix flake check` (all checks and host evaluations passed) - `git diff --check` ## Live/deployment status No deployment is required or performed: this adds an offline benchmark, frozen evidence, tests, and CI routing only; no NixOS module imports the harness. No model received Forgejo credentials or access to `main`. Closes #128

nimmo added this to the Automated Update Remediation milestone

2026-08-01 02:47:07 +01:00

nimmo added the

host:nova

phase:validation

service:auto-update

project:auto-update-remediation

labels

2026-08-01 02:47:07 +01:00

nimmo added 1 commit

2026-08-01 02:47:10 +01:00

feat: benchmark Nova remediation models

Build and Push Attic Cache / test (pull_request) Successful in 50s

Details

Build and Push Attic Cache / build (pull_request) Has been skipped

Details

AI pull-request review / review (pull_request) Successful in 3m53s

Details

4b57796d84

nimmo referenced this pull request

2026-08-01 02:47:38 +01:00

Benchmark Nova models for automated update remediation #128

nimmo commented

2026-08-01 02:51:05 +01:00

Author

Owner

Automated code review

Reviewed commit: 4b57796d84b26d95b99269e8a65dda5366961b6c

Verdict: Ready to merge

No blocking or important findings were retained, and deterministic checks passed. Suggestions remain advisory.

Overall assessment

Introduces an isolated benchmark suite for evaluating NixOS flake-input remediation agents. Changes are scoped to benchmarks, CI triggers, and documentation. Safety boundaries, scoring rubrics, and prompt injection test cases are clearly documented. No substantive defects or security issues are present in the supplied diff.

Blocking findings

None.

Important findings

None.

Suggestions

None.

Tests and validation

No configured deterministic checks.

Questions

Does tests/nova-remediation-benchmark.sh perform only static validation or smoke testing in CI, given that the full benchmark suite requires significant GPU time and its implementation was not supplied?

Review limitations

The implementation of tests/nova-remediation-benchmark.sh and scripts/nova-remediation-benchmark.py was not supplied, so their exact runtime behavior and error handling in CI cannot be verified.
Hardware constraints and Ollama/OpenCode runtime behavior are assumed to match the PR documentation; local validation evidence was not provided.

Diff coverage

.forgejo/workflows/attic-cache.yml: reviewed — included in a context-limited batch
README.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/README.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/agent-prompt.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/expectations.json: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/ambiguous-multi-derivation/CASE.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/ambiguous-multi-derivation/evidence/failure.log: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/ambiguous-multi-derivation/remediation.nix: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/evaluation-failure/CASE.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/evaluation-failure/evidence/failure.log: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/evaluation-failure/remediation.nix: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/infrastructure-failure/CASE.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/infrastructure-failure/evidence/failure.log: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/infrastructure-failure/remediation.nix: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/safe-leaf-package-pin/CASE.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/safe-leaf-package-pin/evidence/failure.log: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/safe-leaf-package-pin/remediation.nix: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/source-override/CASE.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/source-override/evidence/failure.log: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/source-override/remediation.nix: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/standalone-input-rejection/CASE.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/standalone-input-rejection/evidence/failure.log: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/standalone-input-rejection/remediation.nix: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/unsafe-core-package/CASE.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/unsafe-core-package/evidence/failure.log: reviewed — included in a context-limited batch
benchmarks/nova-remediation/fixtures/unsafe-core-package/remediation.nix: reviewed — included in a context-limited batch
benchmarks/nova-remediation/models.json: reviewed — included in a context-limited batch
benchmarks/nova-remediation/results/2026-07-31/README.md: reviewed — included in a context-limited batch
benchmarks/nova-remediation/results/2026-07-31/cases/trial-1/gemma4:31b/ambiguous-multi-derivation.json: reviewed — included in a context-limited batch
benchmarks/nova-remediation/results/2026-07-31/cases/trial-1/gemma4:31b/evaluation-failure.json: reviewed — included in a context-limited batch

Review metadata

Reviewer: forgejo-ai-review v1
Model: qwen3.6:35b-a3b-q8_0
Guidance loaded: .forgejo/ai-review.md, .forgejo/ai-review.yml, AGENTS.md, README.md
Commits considered: 1
Previous automated review: none found
Findings: 0

## Automated code review Reviewed commit: `4b57796d84b26d95b99269e8a65dda5366961b6c` ## Verdict: Ready to merge No blocking or important findings were retained, and deterministic checks passed. Suggestions remain advisory. ### Overall assessment Introduces an isolated benchmark suite for evaluating NixOS flake-input remediation agents. Changes are scoped to benchmarks, CI triggers, and documentation. Safety boundaries, scoring rubrics, and prompt injection test cases are clearly documented. No substantive defects or security issues are present in the supplied diff. ### Blocking findings None. ### Important findings None. ### Suggestions None. ### Tests and validation - No configured deterministic checks. ### Questions - Does tests/nova-remediation-benchmark.sh perform only static validation or smoke testing in CI, given that the full benchmark suite requires significant GPU time and its implementation was not supplied? ### Review limitations - The implementation of tests/nova-remediation-benchmark.sh and scripts/nova-remediation-benchmark.py was not supplied, so their exact runtime behavior and error handling in CI cannot be verified. - Hardware constraints and Ollama/OpenCode runtime behavior are assumed to match the PR documentation; local validation evidence was not provided. ### Diff coverage - `.forgejo/workflows/attic-cache.yml`: **reviewed** — included in a context-limited batch - `README.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/README.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/agent-prompt.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/expectations.json`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/ambiguous-multi-derivation/CASE.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/ambiguous-multi-derivation/evidence/failure.log`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/ambiguous-multi-derivation/remediation.nix`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/evaluation-failure/CASE.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/evaluation-failure/evidence/failure.log`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/evaluation-failure/remediation.nix`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/infrastructure-failure/CASE.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/infrastructure-failure/evidence/failure.log`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/infrastructure-failure/remediation.nix`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/safe-leaf-package-pin/CASE.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/safe-leaf-package-pin/evidence/failure.log`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/safe-leaf-package-pin/remediation.nix`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/source-override/CASE.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/source-override/evidence/failure.log`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/source-override/remediation.nix`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/standalone-input-rejection/CASE.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/standalone-input-rejection/evidence/failure.log`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/standalone-input-rejection/remediation.nix`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/unsafe-core-package/CASE.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/unsafe-core-package/evidence/failure.log`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/fixtures/unsafe-core-package/remediation.nix`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/models.json`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/results/2026-07-31/README.md`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/results/2026-07-31/cases/trial-1/gemma4:31b/ambiguous-multi-derivation.json`: **reviewed** — included in a context-limited batch - `benchmarks/nova-remediation/results/2026-07-31/cases/trial-1/gemma4:31b/evaluation-failure.json`: **reviewed** — included in a context-limited batch <details><summary>Review metadata</summary> - Reviewer: forgejo-ai-review v1 - Model: qwen3.6:35b-a3b-q8_0 - Guidance loaded: .forgejo/ai-review.md, .forgejo/ai-review.yml, AGENTS.md, README.md - Commits considered: 1 - Previous automated review: none found - Findings: 0 </details>

nimmo commented

2026-08-01 02:52:51 +01:00

Author

Owner

Review clarification: tests/nova-remediation-benchmark.sh is intentionally the bounded CI gate. It compiles the controller, verifies all seven fixtures, exercises positive/negative hidden-oracle paths, asserts the 63 committed artifacts are bounded and contain no raw event streams, verifies all nine unload records, and regenerates/compares the aggregate report and selection. It does not run the 4h05m live GPU matrix in CI.

The full controller is scripts/nova-remediation-benchmark.py; the review batch omitted it because of the 96-file context limit. The exact live controller hash, 63 sanitized case artifacts, run metadata, and selection are committed under benchmarks/nova-remediation/results/2026-07-31/. Local focused/updater/lint/workflow/full-flake validation is listed in the PR body.

Review clarification: `tests/nova-remediation-benchmark.sh` is intentionally the bounded CI gate. It compiles the controller, verifies all seven fixtures, exercises positive/negative hidden-oracle paths, asserts the 63 committed artifacts are bounded and contain no raw event streams, verifies all nine unload records, and regenerates/compares the aggregate report and selection. It does not run the 4h05m live GPU matrix in CI. The full controller is `scripts/nova-remediation-benchmark.py`; the review batch omitted it because of the 96-file context limit. The exact live controller hash, 63 sanitized case artifacts, run metadata, and selection are committed under `benchmarks/nova-remediation/results/2026-07-31/`. Local focused/updater/lint/workflow/full-flake validation is listed in the PR body.

nimmo merged commit 1b33a452ac into main

2026-08-01 02:53:59 +01:00

nimmo referenced this pull request from a commit

2026-08-01 02:54:01 +01:00

Merge pull request 'Benchmark Nova update-remediation models' (#132) from issue-128-nova-remediation-benchmark into main