nimmo/nixos-config

Watch

Fork 0

Benchmark Nova models for automated update remediation #128

New issue

Closed

opened 2026-07-31 21:59:46 +01:00 by nimmo · 2 comments

nimmo commented

2026-07-31 21:59:46 +01:00

Owner

Context

Nova now has OpenCode 1.18.10 configured against its local Ollama endpoint. The viable catalog is qwen3.6:35b-a3b-q8_0, north-mini-code-1.0:q8_0, and gemma4:31b. Qwen 3.5 122B is deliberately excluded: its roughly 87.3 GB runtime allocation oversubscribed the 64 GiB GPU allocation and stalled at 99 percent VRAM with negligible GPU work.

Scope

Build a reproducible end-to-end benchmark for selecting the model or cascade used by the failed-update remediation worker. Exercise the real OpenCode agent harness and tools, not raw chat completions.

Checklist

Freeze representative historical and synthetic failure fixtures.
Cover a valid leaf-package pin, source override, standalone input rejection, evaluation failure, infrastructure failure, unsafe core-package pin, and ambiguous multi-derivation failure.
Run each candidate with identical prompts, tools, context and fresh isolated worktrees.
Repeat trials enough to expose stochastic tool-use failures.
Score diagnosis, patch validity, deterministic validation, safe refusal, tool reliability, latency and Nova resource use.
Apply hard disqualifiers for unsafe writes, secret access, unvalidated success claims, infrastructure misclassification and unbounded repair loops.
Compare a single champion with a fast-triage plus stronger-remediation cascade.
Record the selected policy and evidence in this issue.

Completion criteria

A repeatable benchmark produces an evidence-backed model policy suitable for the guarded remediation worker. No benchmark run writes to main, publishes a branch, accesses secrets or deploys a host.

## Context Nova now has OpenCode 1.18.10 configured against its local Ollama endpoint. The viable catalog is qwen3.6:35b-a3b-q8_0, north-mini-code-1.0:q8_0, and gemma4:31b. Qwen 3.5 122B is deliberately excluded: its roughly 87.3 GB runtime allocation oversubscribed the 64 GiB GPU allocation and stalled at 99 percent VRAM with negligible GPU work. ## Scope Build a reproducible end-to-end benchmark for selecting the model or cascade used by the failed-update remediation worker. Exercise the real OpenCode agent harness and tools, not raw chat completions. ## Checklist - [x] Freeze representative historical and synthetic failure fixtures. - [x] Cover a valid leaf-package pin, source override, standalone input rejection, evaluation failure, infrastructure failure, unsafe core-package pin, and ambiguous multi-derivation failure. - [x] Run each candidate with identical prompts, tools, context and fresh isolated worktrees. - [x] Repeat trials enough to expose stochastic tool-use failures. - [x] Score diagnosis, patch validity, deterministic validation, safe refusal, tool reliability, latency and Nova resource use. - [x] Apply hard disqualifiers for unsafe writes, secret access, unvalidated success claims, infrastructure misclassification and unbounded repair loops. - [x] Compare a single champion with a fast-triage plus stronger-remediation cascade. - [x] Record the selected policy and evidence in this issue. ## Completion criteria A repeatable benchmark produces an evidence-backed model policy suitable for the guarded remediation worker. No benchmark run writes to main, publishes a branch, accesses secrets or deploys a host.

nimmo added this to the Automated Update Remediation milestone

2026-07-31 21:59:46 +01:00

nimmo added the

phase:validation

service:auto-update

host:nova

project:auto-update-remediation

labels

2026-07-31 21:59:46 +01:00

nimmo referenced this issue

2026-07-31 22:00:03 +01:00

Capture structured evidence for rejected flake updates #129

nimmo referenced this issue

2026-07-31 22:00:21 +01:00

Implement guarded Nova remediation worker and Forgejo proposals #130

nimmo commented

2026-07-31 22:03:12 +01:00

Author

Owner

Recommended Codex model

Use codex/gpt-5.6-sol with xhigh reasoning.

This phase needs the strongest experimental design, cross-case judgment, and evaluation calibration. Sol is the frontier GPT-5.6 model for complex reasoning and coding; xhigh should provide enough exploration without using max on every repeated benchmark run. Once the harness is stable, Terra can execute high-volume repetitions if Sol establishes that results remain equivalent.

Model guidance: https://developers.openai.com/api/docs/guides/latest-model

## Recommended Codex model Use `codex/gpt-5.6-sol` with `xhigh` reasoning. This phase needs the strongest experimental design, cross-case judgment, and evaluation calibration. Sol is the frontier GPT-5.6 model for complex reasoning and coding; xhigh should provide enough exploration without using max on every repeated benchmark run. Once the harness is stable, Terra can execute high-volume repetitions if Sol establishes that results remain equivalent. Model guidance: https://developers.openai.com/api/docs/guides/latest-model

nimmo referenced this issue from a pull request that will close it,

2026-08-01 02:47:07 +01:00

Benchmark Nova update-remediation models #132

nimmo commented

2026-08-01 02:47:38 +01:00

Author

Owner

Benchmark complete; implementation and bounded evidence are in PR #132.

Selected policy: single qwen3.6:35b-a3b-q8_0, with deterministic validation and human review mandatory. Do not use a cascade.

63 live OpenCode cases: 3 models x 7 fixtures x 3 repetitions, identical settings, 14,726 seconds total.
Qwen: 77.2/85 mean quality, 89.3/100 total, 17/21 exact oracle passes, 100.1s median, 40.6 GiB peak VRAM, no hard disqualifier.
North: 56.2/85, 9/21 exact, 132.1s median; it was both less accurate and slower at the median.
Gemma: 75.8/85 and 17/21 exact, but repeated 600s infrastructure/leaf timeouts triggered unbounded-loop; disqualified.
Best eligible cascade (North -> Qwen): 72.9/85, 14/21 exact, 185.9s median; worse than Qwen alone.
Qwen misses: all three infrastructure runs correctly classified/refused but omitted the expected builder culprit; one source run failed exact validation and did not claim success. The deterministic controller rejected all four.
All nine model/trial unloads released memory; final ollama ps was empty. No raw agent prose, credentials, Forgejo authority, main access, publication, or deployment was provided.
qwen3.5:122b remains excluded for its previously measured approximately 87.3 GB allocation.

Local gates passed: focused benchmark regression/evidence regeneration, updater regressions, formatter, ShellCheck, actionlint, Forgejo workflow validation, Ruff, and full nix flake check including all host evaluations.

This change is offline benchmark/CI tooling only, so there is no deployment-affecting live evidence gate. Awaiting PR #132 Forgejo checks and review before merge.

Benchmark complete; implementation and bounded evidence are in PR #132. Selected policy: **single `qwen3.6:35b-a3b-q8_0`**, with deterministic validation and human review mandatory. Do not use a cascade. - 63 live OpenCode cases: 3 models x 7 fixtures x 3 repetitions, identical settings, 14,726 seconds total. - Qwen: 77.2/85 mean quality, 89.3/100 total, 17/21 exact oracle passes, 100.1s median, 40.6 GiB peak VRAM, no hard disqualifier. - North: 56.2/85, 9/21 exact, 132.1s median; it was both less accurate and slower at the median. - Gemma: 75.8/85 and 17/21 exact, but repeated 600s infrastructure/leaf timeouts triggered `unbounded-loop`; disqualified. - Best eligible cascade (North -> Qwen): 72.9/85, 14/21 exact, 185.9s median; worse than Qwen alone. - Qwen misses: all three infrastructure runs correctly classified/refused but omitted the expected `builder` culprit; one source run failed exact validation and did not claim success. The deterministic controller rejected all four. - All nine model/trial unloads released memory; final `ollama ps` was empty. No raw agent prose, credentials, Forgejo authority, `main` access, publication, or deployment was provided. - `qwen3.5:122b` remains excluded for its previously measured approximately 87.3 GB allocation. Local gates passed: focused benchmark regression/evidence regeneration, updater regressions, formatter, ShellCheck, actionlint, Forgejo workflow validation, Ruff, and full `nix flake check` including all host evaluations. This change is offline benchmark/CI tooling only, so there is no deployment-affecting live evidence gate. Awaiting PR #132 Forgejo checks and review before merge.

nimmo closed this issue

2026-08-01 02:54:00 +01:00

nimmo referenced this issue

2026-08-01 06:19:08 +01:00

feat(auto-update): add guarded remediation worker #134