feat(nix): offload workload-host builds to nova #245
No reviewers
Labels
No labels
area:authentication
area:flake-utilities
area:performance
area:tbd
host:chaos
host:electra
host:fleet
host:lyra
host:nova
host:vega
investigation
phase:cutover
phase:deploy
phase:mcp
phase:module
phase:packaging
phase:prep
phase:validation
priority:high
priority:medium
project:attic-postgres-lyra-rollout
project:auto-update-reliability
project:auto-update-remediation
project:declarative-purity-cleanup
project:external-review
project:fleet-boundary-cleanup
project:host-facts-refactor
project:lyra-nixos-deploy
project:lyra-service-stack-migration
project:nebula-mesh-network
project:nixos-build-deployment-pipeline
project:security-hardening
project:service-stack-migration
project:vega-sillytavern-cutover
project:wiki-rebuild
repo:numtide/flake-utils
repo:numtide/nix-auth
repo:numtide/nixos-passthru-cache
repo:numtide/nix-relay
service:auto-update
service:mem0
service:nix
service:sillytavern
service:slskd
service:synthseek
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
nimmo/nixos-config!245
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/issue-235-service-host-builds"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Refs #235
Moves Vega, Lyra, and Chaos to Nova remote-only builds; returns Electra to local builds; adds one-shot
local/remotecontrols tojust testandjust deploy.Validation completed locally:
nix flake check, Nova policy check, and deploy wrapper/regression suites. Live targetjust testevidence remains required before merge.Automated code review
Reviewed commit:
69f61a0aa734dfbaecbf373d47bb5e251415b4e7Verdict: Requires further work
Resolve the blocking or important findings and investigate failed deterministic checks before merging.
Overall assessment
The PR implements a fleet-wide shift to offload builds for Vega, Lyra, and Chaos to the Nova remote builder, while returning Electra to local-only builds. It introduces one-shot
local/remoteoverrides injust testandnixos-deploy.sh, adds SOPS-managed SSH keys for the new hosts, and updates documentation and checks.Blocking findings
The diff shows
hosts/lyra/default.nixreferencingconfig.sops.secrets."nix-remote-builder/lyra-private-key".path, but the PR only addssecrets/nix-remote-builders/chaos.jsonandsecrets/nix-remote-builders/vega.json. There is nosecrets/nix-remote-builders/lyra.jsonin the diff.Impact: Evaluation of the Lyra host will fail because the SOPS secret file does not exist, preventing deployment or testing of this host.
Suggested fix: Add
secrets/nix-remote-builders/lyra.jsonto the repository with the appropriate encrypted key and SOPS configuration.Important findings
The script uses the variable
remote_builder_key="${NIXOS_DEPLOY_REMOTE_BUILDER_KEY:-/run/secrets/nix-remote-builder/${current_host}-private-key}". The host-specific secret files are named after the host (e.g.,chaos.json,vega.json), but the default path assumes a file named${current_host}-private-key. If the SOPS decryption produces a file with a different name or if the key is not installed at that specific path, the remote build will fail.Impact: Remote builds triggered via the deploy script may fail to find the SSH key if the default path does not match the actual secret location or name.
Suggested fix: Verify that SOPS installs the secret exactly at
/run/secrets/nix-remote-builder/${current_host}-private-keyand matches the expected filename. If the secret file is named differently (e.g.,lyra.json), ensure the path in the script or the SOPS configuration aligns.Suggestions
The
testrecipe parsesSPECIALISATIONandBUILD_MODE. It checks ifSPEC_ARGis 'local' or 'remote' to setMODE. However, the usage comment saysjust test dgpu remote. If a user runsjust test remote, it setsMODE=remote. If they runjust test local, it setsMODE=local. This logic seems correct, but the error message "build mode specified more than once" might be confusing if a user accidentally passes a specialisation that matches 'local' or 'remote' (unlikely but possible).Impact: Low risk of user confusion.
Suggested fix: Consider adding explicit validation or clearer usage examples in the Justfile comments to prevent misuse.
Tests and validation
Questions
secrets/nix-remote-builders/lyra.jsonfile? It is referenced inhosts/lyra/default.nixbut not included in the PR diff..sops.yaml?Review limitations
lyra.jsonoutside the provided diff.nix flake checkto validate the configuration changes.just testornixos-deploy.shscripts in a live environment.Diff coverage
.sops.yaml: reviewed — included in a context-limited batchREADME.md: reviewed — included in a context-limited batchdocs/nova-remote-builder.md: reviewed — included in a context-limited batchflake.nix: reviewed — included in a context-limited batchhosts/chaos/default.nix: reviewed — included in a context-limited batchhosts/electra/default.nix: reviewed — included in a context-limited batchhosts/lyra/default.nix: reviewed — included in a context-limited batchhosts/vega/default.nix: reviewed — included in a context-limited batchjustfiles/deploy.just: reviewed — included in a context-limited batchmodules/common/nova-remote-builder.nix: reviewed — included in a context-limited batchpackages/nixos-deploy/default.nix: reviewed — included in a context-limited batchscripts/nixos-deploy.sh: reviewed — included in a context-limited batchsecrets/nix-remote-builders/chaos.json: reviewed — included in a context-limited batchsecrets/nix-remote-builders/vega.json: reviewed — included in a context-limited batchtests/nixos-deploy-regressions.sh: reviewed — included in a context-limited batchReview metadata
Automated-review follow-up
No code change is required for the reported findings:
secrets/nix-remote-builders/lyra.jsonalready exists on the target branch from #233, and the existing Lyra SOPS creation rule remains in.sops.yaml; it is not newly added by this MR, so it is absent from the diff./run/secrets/nix-remote-builder/{lyra,vega,chaos}-private-key.just testaccepts only the documented hardware specialisations (battery,igpu,dgpu); none conflict withlocalorremote. The deploy regression suite covers both override modes.The deterministic Attic test check and AI-review workflow both succeeded. Local validation also completed
nix flake check, the focused Nova policy check, and both deploy regression suites. Livejust testevidence remains outstanding before merge.