feat(auto-update): add guarded remediation worker #134
No reviewers
Labels
No labels
area:authentication
area:flake-utilities
area:performance
area:tbd
host:chaos
host:electra
host:fleet
host:lyra
host:nova
host:vega
investigation
phase:cutover
phase:deploy
phase:mcp
phase:module
phase:packaging
phase:prep
phase:validation
priority:high
priority:medium
project:attic-postgres-lyra-rollout
project:auto-update-reliability
project:auto-update-remediation
project:declarative-purity-cleanup
project:external-review
project:fleet-boundary-cleanup
project:host-facts-refactor
project:lyra-nixos-deploy
project:lyra-service-stack-migration
project:nebula-mesh-network
project:nixos-build-deployment-pipeline
project:security-hardening
project:service-stack-migration
project:vega-sillytavern-cutover
project:wiki-rebuild
repo:numtide/flake-utils
repo:numtide/nix-auth
repo:numtide/nixos-passthru-cache
repo:numtide/nix-relay
service:auto-update
service:mem0
service:nix
service:sillytavern
service:slskd
service:synthseek
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
nimmo/nixos-config!134
Loading…
Reference in a new issue
No description provided.
Delete branch "issue-130-guarded-remediation-worker"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Refs #130 (kept open pending genuine normal-rollout and no-op evidence)
Outcome
Implements the isolated OpenCode/Qwen remediation worker and deterministic Forgejo controller for structured rejected-update incidents from #129, using the single
qwen3.6:35b-a3b-q8_0policy selected by #128..git, no credentials, no shell/web/MCP/plugins, a read-only candidate tree, and only two writable proposal outputs.main, merge, deploy, or publishflake.lock.Regression coverage
Hermetic integration uses temporary Git/Forgejo state and fake OpenCode, Bubblewrap, Nix, and Tea; it creates no real incident issue or PR. It covers:
mainadvancement, credential-free model environment, and stale/orphan cleanup.Validation
./tests/nixos-update-remediation-regressions.sh— passed./tests/auto-update-regressions.sh— passed./tests/nova-remediation-benchmark.sh— passedforgejo-runner validate --directory .— passed for both workflowsnix fmt— 124 files checked, no changes requirednix flake check— passed; evaluated electra, lena, vega, lyra, and cosmos and built the formatting checkgit diff --check— passedRollout note
No manual updater run or deployment was performed. This does not interfere with #129 passive rollout observation. After merge, the new Nova workflow should be observed on a normal even-hour run; any required genuine deployment/no-op evidence will be recorded before the issue is finally closed.
CI failure follow-up (
ddcc542)Root cause: Linux
RLIMIT_NPROCcounts all tasks/threads for the real UID. The controller set an absolute ceiling of 128, while the shared Nova runner UID had 434 tasks during reproduction. The fake Bubblewrap and a real Bubblewrap smoke both failed before OpenCode could start withEAGAIN/Resource temporarily unavailable; the eligible case therefore correctly fell back to diagnosis-only, but violated the integration expectation.Fix: reserve the reviewed model allowance above the measured ambient UID task baseline for the immediate kernel limit, while independently measuring the launched subprocess tree and terminating it above the policy ceiling of 128 tasks. This removes the host-load assumption without weakening the worker-specific cap.
Regression/evidence:
published-pr;nix flake check, andgit diff --checkall passed locally.No manual updater run or deployment was performed.
Gate status after rerun monitoring
ddcc54255cbcb56515367f8b992126c62defc9bb.Expecting value,Expecting property name enclosed in double quotes, andExtra data). No automated-review comment or actionable finding was produced.PR #134 remains open and unmerged. The review gate must become genuinely green before merge.
Automated code review
Reviewed commit:
2717aadeada26979b04c7c82f0812b2bf67c12a6Verdict: Ready to merge
No blocking or important findings were retained, and deterministic checks passed. Suggestions remain advisory.
Overall assessment
The pull request implements a guarded remediation worker and deterministic Forgejo controller to handle rejected NixOS flake-input updates. It enforces strict security boundaries via Bubblewrap, fail-closed resource limits, narrow eligibility policies, and deterministic validation (baseline reproduction, patch application, host builds, closure comparison, and package/source proofs). Hermetic integration tests cover eligible, unsafe, blocked-safety, boundary-write, and infrastructure-failure scenarios. The implementation aligns with the stated objectives and passes all provided validation gates.
Blocking findings
None.
Important findings
None.
Suggestions
The file spans 3075 lines and consolidates policy validation, incident state management, Bubblewrap/OpenCode execution, deterministic Nix validation, Git worktree operations, and Forgejo publication via the Tea client into one module.
Impact: High cognitive load for future reviewers and maintainers; increases risk of regression when modifying isolated subsystems like the publisher or sandbox launcher.
Suggested fix: Extract distinct responsibilities (e.g.,
TeaPublisher,ValidationRunner,process_tree_task_count,deterministic_validation) into separate modules or a small package structure to improve test isolation and long-term maintainability.Tests and validation
Questions
Review limitations
Diff coverage
.forgejo/workflows/attic-cache.yml: reviewed — included in a context-limited batchREADME.md: reviewed — included in a context-limited batchflake.nix: reviewed — included in a context-limited batchjustfiles/maintenance.just: reviewed — included in a context-limited batchremediation/nova-policy.json: reviewed — included in a context-limited batchremediation/worker-prompt.md: reviewed — included in a context-limited batchscripts/nixos-update-incident-bundle.sh: reviewed — included in a context-limited batchscripts/nixos-update-remediation.py: reviewed — included in a context-limited batchtests/auto-update-regressions.sh: reviewed — included in a context-limited batchtests/fixtures/nixos-update-remediation/blocked-safety/proposal.patch: reviewed — included in a context-limited batchtests/fixtures/nixos-update-remediation/blocked-safety/result-template.json: reviewed — included in a context-limited batchtests/fixtures/nixos-update-remediation/eligible-leaf/proposal.patch: reviewed — included in a context-limited batchtests/fixtures/nixos-update-remediation/eligible-leaf/result-template.json: reviewed — included in a context-limited batchtests/fixtures/nixos-update-remediation/module-schema/result-template.json: reviewed — included in a context-limited batchtests/fixtures/nixos-update-remediation/unsafe-core/proposal.patch: reviewed — included in a context-limited batchtests/fixtures/nixos-update-remediation/unsafe-core/result-template.json: reviewed — included in a context-limited batchtests/nixos-update-remediation-integration.py: reviewed — included in a context-limited batchtests/nixos-update-remediation-regressions.sh: reviewed — included in a context-limited batchReview metadata
Automated review follow-up for
ddcc542The published review completed successfully at 128k, but its retained important finding is not valid:
Markdown clarification: the two verified commands used the literal revision 47b9c09846a52612727d51e0557a7edf1f790f35^{commit}; angle-bracket placeholders in the preceding comment were rendered as HTML and disappeared.
Final Nova publisher follow-up (
2717aad)The exact target-environment audit found that Nova intentionally has neither an ambient
teabinary nor a persistent Tea login. Installing a mutable host package would have fixed only half of that contract.The revised workflow now:
nixos-update-remediation-tea);forgejo-actionsidentity, which is repository-scoped and destroyed after the run;/tmp, deletes it on step exit, and removesFORGEJO_TOKEN/GITHUB_TOKENbefore starting the deterministic controller;/tmphidden behind the model Bubblewrap tmpfs, while the existing clean-environment assertions continue to prove no Forgejo or deployment token reaches Bubblewrap/OpenCode;Exact-head validation:
2717aadeada26979b04c7c82f0812b2bf67c12a6, built locked Tea, logged in asforgejo-actions, and passed both publisher contract probes plus all updater/remediation/benchmark/Nix gates.truncated=0).nix flake check, real read-only Tea queries, andgit diff --checkall passed.No Tea package was installed on Nova. No updater run or deployment was forced. The PR now uses a non-closing reference to #130 so the issue remains open until genuine normal even-hour rollout and required no-op evidence are recorded.