nimmo/nixos-config

Watch

Fork 0

feat(nix): opt Electra into Nova remote builds #244

Merged

nimmo merged 4 commits from feat/issue-234-nova-builder-opt-in into main

2026-09-20 08:24:06 +01:00

nimmo commented

2026-09-20 07:59:22 +01:00

Owner

Closes #234.

Scope:

Adds an inert reusable Nova distributed-builder client module, with Nova scheduling details isolated from client wiring.
Opts Electra in as the initial remote-only client using its root-only, per-host SOPS credential.
Advertises only x86_64-linux, three jobs, speed factor 1, and no special features.
Documents the initial daemon-policy bootstrap, cache policy, remote-only behavior, and one-shot local recovery.
Adds a focused flake check covering Electra opt-in and Lyra opt-out invariants.

Validation:

nix flake check and nix build .#checks.x86_64-linux.nova-remote-builder-config passed.
Electra completed just test from its battery specialisation, followed by a 2-second no-op just test; no switch was used.
The activated daemon reports max-jobs = 0 and builders-use-substitutes = true; a deliberately non-substituted derivation built on ssh://nix-remote-builder@nova and returned to Electra.
A confirmed-absent package substituted from https://cache.nixos.org.
An unreachable test builder failed without local fallback; the documented command with empty builders and max-jobs = 1 recovered by building locally.
Electra was healthy after validation, and its temporary auto-update hold was released.

Closes #234. Scope: - Adds an inert reusable Nova distributed-builder client module, with Nova scheduling details isolated from client wiring. - Opts Electra in as the initial remote-only client using its root-only, per-host SOPS credential. - Advertises only `x86_64-linux`, three jobs, speed factor 1, and no special features. - Documents the initial daemon-policy bootstrap, cache policy, remote-only behavior, and one-shot local recovery. - Adds a focused flake check covering Electra opt-in and Lyra opt-out invariants. Validation: - `nix flake check` and `nix build .#checks.x86_64-linux.nova-remote-builder-config` passed. - Electra completed `just test` from its `battery` specialisation, followed by a 2-second no-op `just test`; no `switch` was used. - The activated daemon reports `max-jobs = 0` and `builders-use-substitutes = true`; a deliberately non-substituted derivation built on `ssh://nix-remote-builder@nova` and returned to Electra. - A confirmed-absent package substituted from `https://cache.nixos.org`. - An unreachable test builder failed without local fallback; the documented command with empty builders and `max-jobs = 1` recovered by building locally. - Electra was healthy after validation, and its temporary auto-update hold was released.

nimmo added this to the NixOS Build & Deployment Pipeline milestone

2026-09-20 07:59:22 +01:00

nimmo added the

host:nova

phase:module

host:electra

project:nixos-build-deployment-pipeline

service:nix

labels

2026-09-20 07:59:22 +01:00

nimmo added 2 commits

2026-09-20 07:59:22 +01:00

feat(nix): add Nova remote builder client opt-in ca0a9380ff

docs(nova): describe remote builder bootstrap

AI pull-request review / review (pull_request) Successful in 1m4s

Details

Build and Push Attic Cache / test (pull_request) Successful in 1m58s

Details

Build and Push Attic Cache / build (pull_request) Has been skipped

Details

Build and Push Attic Cache / pin-lifecycle (pull_request) Has been skipped

Details

Build and Push Attic Cache / remediate (pull_request) Has been skipped

Details

Build and Push Attic Cache / notify-remediation (pull_request) Has been skipped

Details

Build and Push Attic Cache / record-remediation-notify-failure (pull_request) Has been skipped

Details

771da14a67

nimmo commented

2026-09-20 08:00:27 +01:00

Author

Owner

Automated code review

Reviewed commit: e77c38ffe81faac889ff1204001718a70e6d7eba

Verdict: Requires further work

Resolve the blocking or important findings and investigate failed deterministic checks before merging.

Overall assessment

The PR introduces a reusable NixOS module for configuring Electra as a remote-only client for the 'Nova' builder. It enforces strict build isolation by setting local jobs to zero, pins Nova's SSH host key, and includes flake assertions to verify configuration invariants. The import order dependency on nix-settings.nix has been resolved by importing it directly within the new module.

Blocking findings

None.

Important findings

Hardcoded SSH Host Key in Source Code (modules/common/nova-remote-builder.nix:12; high confidence)
The publicHostKey is hardcoded as a string literal: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILtJEHXw4DrJywYvhG3tXqk6W3kq+MtjFFzbDlLXMH8N".
Impact: If Nova's host key rotates, this module will cause build failures until the code is updated and redeployed. While pinned for security (preventing MITM), it reduces operational flexibility compared to dynamic fetching or external secret management.
Suggested fix: Consider sourcing the host key from a SOPS secret or a dedicated configuration file if key rotation frequency is expected to be high, otherwise document the manual update procedure clearly in docs/nova-remote-builder.md.

Suggestions

Recovery Command Documentation Clarity (docs/nova-remote-builder.md:205; medium confidence)
The documentation provides a specific sudo nixos-rebuild test ... --option builders '' command for recovery.
Impact: Operators might confuse this one-shot override with the declarative configuration. If not clearly distinguished, they might attempt to use it in scripts or CI pipelines where persistence is not desired.
Suggested fix: Ensure the documentation explicitly states that this command overrides the declarative buildMachines for a single invocation only and does not modify the NixOS configuration.

Tests and validation

No configured deterministic checks.

Questions

Is there an automated process to detect if Nova's host key changes, or is this monitored manually via alerts?

Review limitations

I cannot verify the actual connectivity to Nova or the validity of the pinned SSH key against the live service.
I have not executed nix flake check myself, relying on the provided diff and human confirmation.

Diff coverage

docs/nova-remote-builder.md: reviewed — included in a context-limited batch
flake.nix: reviewed — included in a context-limited batch
hosts/electra/default.nix: reviewed — included in a context-limited batch
modules/common/base.nix: reviewed — included in a context-limited batch
modules/common/nova-remote-builder.nix: reviewed — included in a context-limited batch

Review metadata

Reviewer: forgejo-ai-review v1
Model: qwen3.6:35b-a3b-q8_0
Guidance loaded: .forgejo/ai-review.md, .forgejo/ai-review.yml, AGENTS.md, docs/auto-update-remediation.md, docs/nova-remote-builder.md, README.md
Commits considered: 4
Previous automated review: updated
Findings: 2

## Automated code review Reviewed commit: `e77c38ffe81faac889ff1204001718a70e6d7eba` ## Verdict: Requires further work Resolve the blocking or important findings and investigate failed deterministic checks before merging. ### Overall assessment The PR introduces a reusable NixOS module for configuring Electra as a remote-only client for the 'Nova' builder. It enforces strict build isolation by setting local jobs to zero, pins Nova's SSH host key, and includes flake assertions to verify configuration invariants. The import order dependency on `nix-settings.nix` has been resolved by importing it directly within the new module. ### Blocking findings None. ### Important findings - **Hardcoded SSH Host Key in Source Code** (modules/common/nova-remote-builder.nix:12; high confidence) The `publicHostKey` is hardcoded as a string literal: `"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILtJEHXw4DrJywYvhG3tXqk6W3kq+MtjFFzbDlLXMH8N"`. Impact: If Nova's host key rotates, this module will cause build failures until the code is updated and redeployed. While pinned for security (preventing MITM), it reduces operational flexibility compared to dynamic fetching or external secret management. Suggested fix: Consider sourcing the host key from a SOPS secret or a dedicated configuration file if key rotation frequency is expected to be high, otherwise document the manual update procedure clearly in `docs/nova-remote-builder.md`. ### Suggestions - **Recovery Command Documentation Clarity** (docs/nova-remote-builder.md:205; medium confidence) The documentation provides a specific `sudo nixos-rebuild test ... --option builders ''` command for recovery. Impact: Operators might confuse this one-shot override with the declarative configuration. If not clearly distinguished, they might attempt to use it in scripts or CI pipelines where persistence is not desired. Suggested fix: Ensure the documentation explicitly states that this command overrides the declarative `buildMachines` for a single invocation only and does not modify the NixOS configuration. ### Tests and validation - No configured deterministic checks. ### Questions - Is there an automated process to detect if Nova's host key changes, or is this monitored manually via alerts? ### Review limitations - I cannot verify the actual connectivity to Nova or the validity of the pinned SSH key against the live service. - I have not executed `nix flake check` myself, relying on the provided diff and human confirmation. ### Diff coverage - `docs/nova-remote-builder.md`: **reviewed** — included in a context-limited batch - `flake.nix`: **reviewed** — included in a context-limited batch - `hosts/electra/default.nix`: **reviewed** — included in a context-limited batch - `modules/common/base.nix`: **reviewed** — included in a context-limited batch - `modules/common/nova-remote-builder.nix`: **reviewed** — included in a context-limited batch <details><summary>Review metadata</summary> - Reviewer: forgejo-ai-review v1 - Model: qwen3.6:35b-a3b-q8_0 - Guidance loaded: .forgejo/ai-review.md, .forgejo/ai-review.yml, AGENTS.md, docs/auto-update-remediation.md, docs/nova-remote-builder.md, README.md - Commits considered: 4 - Previous automated review: updated - Findings: 2 </details>

nimmo added 1 commit

2026-09-20 08:07:58 +01:00

fix(nix): pin Nova builder host key

AI pull-request review / review (pull_request) Successful in 1m1s

Details

Build and Push Attic Cache / test (pull_request) Successful in 1m25s

Details

Build and Push Attic Cache / build (pull_request) Has been skipped

Details

Build and Push Attic Cache / pin-lifecycle (pull_request) Has been skipped

Details

Build and Push Attic Cache / remediate (pull_request) Has been skipped

Details

Build and Push Attic Cache / notify-remediation (pull_request) Has been skipped

Details

Build and Push Attic Cache / record-remediation-notify-failure (pull_request) Has been skipped

Details

c9655bc1f1

nimmo commented

2026-09-20 08:12:15 +01:00

Author

Owner

Review follow-up:

Addressed the important host-key finding in c9655bc: NixOS uses buildMachines.publicHostKey for this, not an sshKnownHosts field. Nova’s live Ed25519 host key was verified against Electra’s existing known_hosts entry (fingerprint SHA256:igJd4f0dHQKGFXdBK4eNPxVhXdmZFFc0i8tNsa6vk9g) and is now pinned in the module. The focused flake assertion covers it.
Live validation after the pin: Electra just test completed and all three specialisation closures built on Nova; the rendered /etc/nix/machines entry includes the exact pinned key. No switch was used. Electra was healthy afterward and its temporary update hold was released.
supportedFeatures = [ ] is intentional for this initial opt-in. Nova locally enables benchmark, big-parallel, kvm, nixos-test, and uid-range, but #234 deliberately advertises none until a future issue independently verifies that remote jobs may rely on each capability. Ordinary x86_64-linux builds are the only scheduling contract here.
nix flake check passed after the change.

Review follow-up: - Addressed the important host-key finding in `c9655bc`: NixOS uses `buildMachines.publicHostKey` for this, not an `sshKnownHosts` field. Nova’s live Ed25519 host key was verified against Electra’s existing `known_hosts` entry (fingerprint `SHA256:igJd4f0dHQKGFXdBK4eNPxVhXdmZFFc0i8tNsa6vk9g`) and is now pinned in the module. The focused flake assertion covers it. - Live validation after the pin: Electra `just test` completed and all three specialisation closures built on Nova; the rendered `/etc/nix/machines` entry includes the exact pinned key. No switch was used. Electra was healthy afterward and its temporary update hold was released. - `supportedFeatures = [ ]` is intentional for this initial opt-in. Nova locally enables `benchmark`, `big-parallel`, `kvm`, `nixos-test`, and `uid-range`, but #234 deliberately advertises none until a future issue independently verifies that remote jobs may rely on each capability. Ordinary `x86_64-linux` builds are the only scheduling contract here. - `nix flake check` passed after the change.

nimmo added 1 commit

2026-09-20 08:14:23 +01:00

refactor(nix): make Nova cache policy dependency explicit

AI pull-request review / review (pull_request) Successful in 1m0s

Details

Build and Push Attic Cache / test (pull_request) Successful in 1m24s

Details

Build and Push Attic Cache / build (pull_request) Has been skipped

Details

Build and Push Attic Cache / pin-lifecycle (pull_request) Has been skipped

Details

Build and Push Attic Cache / remediate (pull_request) Has been skipped

Details

Build and Push Attic Cache / notify-remediation (pull_request) Has been skipped

Details

Build and Push Attic Cache / record-remediation-notify-failure (pull_request) Has been skipped

Details

e77c38ffe8

nimmo commented

2026-09-20 08:14:32 +01:00

Author

Owner

Second review follow-up:

Addressed the cache-policy import-order finding in e77c38f. nova-remote-builder.nix now imports nix-settings.nix directly, so its builders-use-substitutes = true setting has an explicit dependency on the shared substituters and trusted signing keys. common/base.nix no longer owns that indirect ordering.
The opt-in remains fail-closed: nixosConfig.novaRemoteBuilder.enable defaults to false, only Electra sets it true, and the focused flake check asserts Lyra remains disabled with no build machines.
nix flake check and nix build .#checks.x86_64-linux.nova-remote-builder-config --no-link pass after the change. The earlier live Electra canary also demonstrated both cache substitution and Nova remote builds.

Second review follow-up: - Addressed the cache-policy import-order finding in `e77c38f`. `nova-remote-builder.nix` now imports `nix-settings.nix` directly, so its `builders-use-substitutes = true` setting has an explicit dependency on the shared substituters and trusted signing keys. `common/base.nix` no longer owns that indirect ordering. - The opt-in remains fail-closed: `nixosConfig.novaRemoteBuilder.enable` defaults to false, only Electra sets it true, and the focused flake check asserts Lyra remains disabled with no build machines. - `nix flake check` and `nix build .#checks.x86_64-linux.nova-remote-builder-config --no-link` pass after the change. The earlier live Electra canary also demonstrated both cache substitution and Nova remote builds.

nimmo merged commit aa22c9bc46 into main

2026-09-20 08:24:06 +01:00

nimmo deleted branch feat/issue-234-nova-builder-opt-in

2026-09-20 08:24:07 +01:00

nimmo referenced this pull request from a commit

2026-09-20 08:24:08 +01:00

Merge pull request 'feat(nix): opt Electra into Nova remote builds' (#244) from feat/issue-234-nova-builder-opt-in into main