04 – Move Workload-Host Builds Off the Service Hosts #235

New issue

Closed

opened 2026-09-19 20:31:13 +01:00 by nimmo · 1 comment

nimmo commented

2026-09-19 20:31:13 +01:00

Owner

Goal

Move eligible expensive Vega, Lyra, and Chaos builds to Nova without competing with their service workloads, while retaining a documented one-shot local recovery path.

Dependencies

Blocked by #234.

Placement policy

Electra is the fleet local-build workstation for now.
Vega, Lyra, and Chaos use Nova in remote-only mode for ordinary eligible builds.
Remote building changes build placement only: evaluation, closure receipt, and activation remain on the target.
Local compilation remains available only through the documented per-invocation emergency override when Nova is unavailable.

Requirements

Review the capacity and failure impact on each workload host before enabling the reusable client configuration.
Give Vega and Chaos their own SOPS-managed restricted builder identities; retain Lyra’s existing independent identity.
Pin Nova’s SSH host key declaratively and preserve the shared signed-cache policy.
Validate manual rebuilds and the consumer auto-update path. A cache hit alone is not evidence of a remote compilation.
Use just test for target activation tests; do not switch generations as part of this issue.

Validation

For Vega, Lyra, and Chaos: prove a deliberately non-substituted, identifiable build runs on Nova; prove its closure reaches the target; and complete a just test activation. Confirm relevant services remain healthy, record the one-shot local recovery procedure, and collect a no-op consumer auto-update follow-up before closing.

## Goal Move eligible expensive Vega, Lyra, and Chaos builds to Nova without competing with their service workloads, while retaining a documented one-shot local recovery path. ## Dependencies Blocked by #234. ## Placement policy - Electra is the fleet local-build workstation for now. - Vega, Lyra, and Chaos use Nova in `remote-only` mode for ordinary eligible builds. - Remote building changes build placement only: evaluation, closure receipt, and activation remain on the target. - Local compilation remains available only through the documented per-invocation emergency override when Nova is unavailable. ## Requirements - Review the capacity and failure impact on each workload host before enabling the reusable client configuration. - Give Vega and Chaos their own SOPS-managed restricted builder identities; retain Lyra’s existing independent identity. - Pin Nova’s SSH host key declaratively and preserve the shared signed-cache policy. - Validate manual rebuilds and the consumer auto-update path. A cache hit alone is not evidence of a remote compilation. - Use `just test` for target activation tests; do not switch generations as part of this issue. ## Validation For Vega, Lyra, and Chaos: prove a deliberately non-substituted, identifiable build runs on Nova; prove its closure reaches the target; and complete a `just test` activation. Confirm relevant services remain healthy, record the one-shot local recovery procedure, and collect a no-op consumer auto-update follow-up before closing.

nimmo added this to the NixOS Build & Deployment Pipeline milestone

2026-09-19 20:31:13 +01:00

nimmo added the

project:nixos-build-deployment-pipeline

service:nix

labels

2026-09-19 21:01:30 +01:00

nimmo referenced this issue