nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions

Capture structured evidence for rejected flake updates #129

New issue
Closed
opened 2026-07-31 22:00:03 +01:00 by nimmo · 4 comments
nimmo commented 2026-07-31 22:00:03 +01:00
Owner
Copy link

Context

The producer already isolates failures to individual flake inputs and affected hosts, but its rejected build log is temporary and is removed after classification. A remediation agent needs deterministic, sanitised evidence rather than scraping a Forgejo console transcript.

Scope

Extend the Nova producer to emit a bounded incident bundle whenever an input candidate is rejected for a non-infrastructure failure.

Checklist

  • Preserve the baseline and rejected candidate lock identities and input revisions.
  • Record the input group, failed host, build target, command and exit status.
  • Classify infrastructure, evaluation, assertion or test, and derivation-build failures.
  • Identify candidate failing derivations and extract pname, version, source position and relevant log excerpts where available.
  • Retain the exact failed candidate flake.lock or an equivalent reproducible lock artifact.
  • Sanitise logs and bound artifact sizes before exposing them to an LLM.
  • Treat all upstream build output as untrusted prompt-injection-capable data.
  • Generate a stable incident fingerprint for Forgejo deduplication.
  • Add regression fixtures for every classification and malformed or missing evidence.
  • Preserve current atomic publication and partial-success behaviour.

Completion criteria

A rejected update leaves a reproducible, sanitised, machine-readable incident bundle while infrastructure failures and unrelated accepted inputs retain their current behaviour.

Depends on the existing per-input isolation and producer publication invariants. Refs #128 for the consumer benchmark contract.

## Context The producer already isolates failures to individual flake inputs and affected hosts, but its rejected build log is temporary and is removed after classification. A remediation agent needs deterministic, sanitised evidence rather than scraping a Forgejo console transcript. ## Scope Extend the Nova producer to emit a bounded incident bundle whenever an input candidate is rejected for a non-infrastructure failure. ## Checklist - [ ] Preserve the baseline and rejected candidate lock identities and input revisions. - [ ] Record the input group, failed host, build target, command and exit status. - [ ] Classify infrastructure, evaluation, assertion or test, and derivation-build failures. - [ ] Identify candidate failing derivations and extract pname, version, source position and relevant log excerpts where available. - [ ] Retain the exact failed candidate flake.lock or an equivalent reproducible lock artifact. - [ ] Sanitise logs and bound artifact sizes before exposing them to an LLM. - [ ] Treat all upstream build output as untrusted prompt-injection-capable data. - [ ] Generate a stable incident fingerprint for Forgejo deduplication. - [ ] Add regression fixtures for every classification and malformed or missing evidence. - [ ] Preserve current atomic publication and partial-success behaviour. ## Completion criteria A rejected update leaves a reproducible, sanitised, machine-readable incident bundle while infrastructure failures and unrelated accepted inputs retain their current behaviour. Depends on the existing per-input isolation and producer publication invariants. Refs #128 for the consumer benchmark contract.
nimmo commented 2026-07-31 22:03:21 +01:00
Author
Owner
Copy link

Recommended Codex model

Use codex/gpt-5.6-terra with high reasoning.

This is a bounded, regression-heavy shell and Nix instrumentation task with clear schemas and invariants. Terra offers the best intelligence and cost balance for sustained implementation and fixture work; high reasoning is appropriate for correctly separating derivation, evaluation, test, and infrastructure failures. Escalate a disputed parser or publication-invariant design to Sol before merging.

Model guidance: https://developers.openai.com/api/docs/guides/latest-model

## Recommended Codex model Use `codex/gpt-5.6-terra` with `high` reasoning. This is a bounded, regression-heavy shell and Nix instrumentation task with clear schemas and invariants. Terra offers the best intelligence and cost balance for sustained implementation and fixture work; high reasoning is appropriate for correctly separating derivation, evaluation, test, and infrastructure failures. Escalate a disputed parser or publication-invariant design to Sol before merging. Model guidance: https://developers.openai.com/api/docs/guides/latest-model
nimmo commented 2026-08-01 04:51:09 +01:00
Author
Owner
Copy link

PR #133 merged after all validation, CI, and automated-review gates passed. Reopened pending required live evidence from a normal even-hour fleet auto-update and the subsequent no-op follow-up. No deployment will be forced.

PR #133 merged after all validation, CI, and automated-review gates passed. Reopened pending required live evidence from a normal even-hour fleet auto-update and the subsequent no-op follow-up. No deployment will be forced.
nimmo commented 2026-08-01 06:11:22 +01:00
Author
Owner
Copy link

Passive rollout evidence: Lyra ran its normal nixos-auto-update timer at 2026-08-01 06:05:54 BST. The build completed, the detached switch finished successfully at 06:07:30, and the updater verified deployed configurationRevision 4fcb27611c (the PR #133 merge). No deployment was forced. Package selection/version changes: none; closure delta: 0 paths net, -12.5 MiB. Keeping this issue open for the required subsequent scheduled no-op follow-up.

Passive rollout evidence: Lyra ran its normal nixos-auto-update timer at 2026-08-01 06:05:54 BST. The build completed, the detached switch finished successfully at 06:07:30, and the updater verified deployed configurationRevision 4fcb27611cf7f05165cad0cbaea4bd2e9066c064 (the PR #133 merge). No deployment was forced. Package selection/version changes: none; closure delta: 0 paths net, -12.5 MiB. Keeping this issue open for the required subsequent scheduled no-op follow-up.
nimmo commented 2026-08-01 08:20:28 +01:00
Author
Owner
Copy link

Scheduled no-op follow-up verified on 2026-08-01 without forcing an update. Vega ran at 08:11:39 BST and Lyra at 08:13:05 BST; both fetched origin/main, reported Repository already matches origin/main (4fcb27611c), logged Nothing changed in pull, system is current, and exited status 0/SUCCESS. Both hosts still report configurationRevision 4fcb27611c. This completes the genuine deployment plus subsequent no-op evidence required for PR #133.

Scheduled no-op follow-up verified on 2026-08-01 without forcing an update. Vega ran at 08:11:39 BST and Lyra at 08:13:05 BST; both fetched origin/main, reported Repository already matches origin/main (4fcb27611cf7f05165cad0cbaea4bd2e9066c064), logged Nothing changed in pull, system is current, and exited status 0/SUCCESS. Both hosts still report configurationRevision 4fcb27611cf7f05165cad0cbaea4bd2e9066c064. This completes the genuine deployment plus subsequent no-op evidence required for PR #133.
nimmo closed this issue 2026-08-01 08:20:28 +01:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/issue-236-electra-nova-vega
feat/issue-233-nova-builder-credentials
docs/wiki-final-cutover
fix/remediation-evaluation-evidence
fix/remediation-evidence-review
fix/chaos-synthseek-queue
pre-electra-disko-migration
Labels
Clear labels   
area:authentication
Issues related to authentication area

  
area:flake-utilities
Issues related to flake utilities area

  
area:performance
Issues related to performance area

  
area:tbd
Area to be determined

  
host:chaos
Issues specific to the chaos host.

  
host:electra
Issues specific to the electra host.

  
host:fleet
Issues that span multiple hosts or the fleet as a whole.

  
host:lyra
Issues specific to the lyra host.

  
host:nova
Issues specific to the nova host.

  
host:vega
Issues specific to the vega host.

  
investigation
Issue is an investigation task

  
phase:cutover
The service cutover execution phase.

  
phase:deploy

   
phase:mcp

   
phase:module

   
phase:packaging

   
phase:prep
Preparation and preflight work before service cutover.

  
phase:validation
Post-cutover validation and rollback-window work.

  
priority:high
High priority issue

  
priority:medium
Medium priority issue

  
project:attic-postgres-lyra-rollout
Issues in the Attic PostgreSQL migration and Lyra rollout workstream.

  
project:auto-update-reliability
Work arising from the automated update reliability review.

  
project:auto-update-remediation
Guarded Nova-assisted diagnosis and remediation of failed automated updates.

  
project:declarative-purity-cleanup
Issues in the declarative purity cleanup workstream.

  
project:external-review
Review of external configurations for ideas and improvements

  
project:fleet-boundary-cleanup
Maintainability work that clarifies fleet configuration and automation boundaries.

  
project:host-facts-refactor
Issues in the host facts refactor workstream.

  
project:lyra-nixos-deploy
Issues in the Lyra NixOS deployment workstream.

  
project:lyra-service-stack-migration
Issues in the lyra service stack migration workstream.

  
project:nebula-mesh-network
Issues in the Nebula mesh network workstream.

  
project:nixos-build-deployment-pipeline
Issues in the NixOS Build & Deployment Pipeline workstream.

  
project:security-hardening
Work belonging to the Security Hardening milestone

  
project:service-stack-migration
Issues in the service stack migration workstream.

  
project:vega-sillytavern-cutover
Issues in the Vega SillyTavern cutover workstream.

  
project:wiki-rebuild
Re-found the repository wiki as the fleet's curated knowledge architecture.

  
repo:numtide/flake-utils
Issues related to numtide/flake-utils repository

  
repo:numtide/nix-auth
Issues related to numtide/nix-auth repository

  
repo:numtide/nixos-passthru-cache
Issues related to numtide/nixos-passthru-cache repository

  
repo:numtide/nix-relay
Issues related to numtide/nix-relay repository

  
service:auto-update
NixOS flake update, cache build, and consumer deployment automation.

  
service:mem0

   
service:nix
Issues concerning Nix build, store, cache, or deployment infrastructure.

  
service:sillytavern
Issues related to the SillyTavern service.

  
service:slskd
Soulseek daemon (slskd) work.

  
service:synthseek
Synthseek music discovery/library manager work.

No labels
area:authentication
area:flake-utilities
area:performance
area:tbd
host:chaos
host:electra
host:fleet
host:lyra
host:nova
host:vega
investigation
phase:cutover
phase:deploy
phase:mcp
phase:module
phase:packaging
phase:prep
phase:validation
priority:high
priority:medium
project:attic-postgres-lyra-rollout
project:auto-update-reliability
project:auto-update-remediation
project:declarative-purity-cleanup
project:external-review
project:fleet-boundary-cleanup
project:host-facts-refactor
project:lyra-nixos-deploy
project:lyra-service-stack-migration
project:nebula-mesh-network
project:nixos-build-deployment-pipeline
project:security-hardening
project:service-stack-migration
project:vega-sillytavern-cutover
project:wiki-rebuild
repo:numtide/flake-utils
repo:numtide/nix-auth
repo:numtide/nixos-passthru-cache
repo:numtide/nix-relay
service:auto-update
service:mem0
service:nix
service:sillytavern
service:slskd
service:synthseek
Milestone
Clear milestone
No items
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
nimmo/nixos-config#129
No description provided.