[INVESTIGATION] Evaluate numtide/nix-relay for potential NixOS configuration integration #150

New issue

Closed

opened 2026-08-06 23:24:31 +01:00 by nimmo · 1 comment

nimmo commented

2026-08-06 23:24:31 +01:00

Owner

Investigation: numtide/nix-relay

Overview

Investigate the numtide/nix-relay repository to determine what it provides and if it offers valuable enhancements, simplifications, or performance improvements for the NixOS configuration at https://git.nimmog.uk/nimmo/nixos-config.

Repository Information

URL: https://github.com/numtide/nix-relay
Description: [To be determined - name suggests relay/proxy functionality for Nix operations]

Investigation Goals

Determine the core functionality and purpose of nix-relay
Evaluate relevance to current NixOS configuration architecture and operations
Identify potential integration points for improving Nix operations, binary caching, or distribution
Assess maintenance burden and compatibility with existing NixOS setup
Determine if proof-of-concept implementation is warranted

Binary cache strategy (likely using default cache.nixos.org)
Attic cache implementation for built artifacts (mentioned in README)
Auto-update system that builds and pushes flake inputs
NixDA or similar binary cache sharing mechanism?
Remote build capabilities?
Paseo system for agent orchestration (might relate to relaying agent operations?)
Various services that might benefit from optimized Nix operations

Specific Areas to Examine

Primary Purpose: What exactly does nix-relay do? (This is the first question to answer)
If it's a binary cache relay/proxy:
- How it compares to current Attic cache setup
- Potential benefits for the existing binary cache strategy
- Integration with current auto-update and build processes
If it's related to Nix daemon relaying:
- Potential benefits for remote builds or distributed Nix operations
- Integration with current build farms or builder systems
If it's related to substitution or transfer optimization:
- How it could improve nix-daemon operations
- Compatibility with current NixOS setup

Suggested Evaluation Approach

Review documentation and README thoroughly to understand purpose
Examine example usage patterns and architecture
Check compatibility with NixOS unstable channel
Determine if it solves any known pain points in current setup
Evaluate complexity of integration vs. potential benefits
Review maintenance history and community adoption

Decision Criteria

Clear understanding of what nix-relay provides
Identifiable benefit to current NixOS configuration operations
Reasonable integration complexity
Active maintenance and community support
Clear use case that aligns with current or planned infrastructure

Notes for Investigating Agent

Feel free to adjust investigation focus based on initial findings
If a repository shows little promise, document reasons for closing investigation
If additional related repositories are discovered during investigation, feel free to create issues for those as well
Consider creating a milestone like "NumTide Repository Evaluation" to group these investigations
Initial investigation should be time-boxed (e.g., 2-4 hours per repository max) unless significant promise is found

# Investigation: numtide/nix-relay ## Overview Investigate the numtide/nix-relay repository to determine what it provides and if it offers valuable enhancements, simplifications, or performance improvements for the NixOS configuration at https://git.nimmog.uk/nimmo/nixos-config. ## Repository Information - **URL**: https://github.com/numtide/nix-relay - **Description**: [To be determined - name suggests relay/proxy functionality for Nix operations] ## Investigation Goals 1. Determine the core functionality and purpose of nix-relay 2. Evaluate relevance to current NixOS configuration architecture and operations 3. Identify potential integration points for improving Nix operations, binary caching, or distribution 4. Assess maintenance burden and compatibility with existing NixOS setup 5. Determine if proof-of-concept implementation is warranted ## Current Related Infrastructure - Binary cache strategy (likely using default cache.nixos.org) - Attic cache implementation for built artifacts (mentioned in README) - Auto-update system that builds and pushes flake inputs - NixDA or similar binary cache sharing mechanism? - Remote build capabilities? - Paseo system for agent orchestration (might relate to relaying agent operations?) - Various services that might benefit from optimized Nix operations ## Specific Areas to Examine 1. **Primary Purpose**: What exactly does nix-relay do? (This is the first question to answer) 2. If it's a binary cache relay/proxy: - How it compares to current Attic cache setup - Potential benefits for the existing binary cache strategy - Integration with current auto-update and build processes 3. If it's related to Nix daemon relaying: - Potential benefits for remote builds or distributed Nix operations - Integration with current build farms or builder systems 4. If it's related to substitution or transfer optimization: - How it could improve nix-daemon operations - Compatibility with current NixOS setup ## Suggested Evaluation Approach - Review documentation and README thoroughly to understand purpose - Examine example usage patterns and architecture - Check compatibility with NixOS unstable channel - Determine if it solves any known pain points in current setup - Evaluate complexity of integration vs. potential benefits - Review maintenance history and community adoption ## Decision Criteria - Clear understanding of what nix-relay provides - Identifiable benefit to current NixOS configuration operations - Reasonable integration complexity - Active maintenance and community support - Clear use case that aligns with current or planned infrastructure ## Notes for Investigating Agent - Feel free to adjust investigation focus based on initial findings - If a repository shows little promise, document reasons for closing investigation - If additional related repositories are discovered during investigation, feel free to create issues for those as well - Consider creating a milestone like "NumTide Repository Evaluation" to group these investigations - Initial investigation should be time-boxed (e.g., 2-4 hours per repository max) unless significant promise is found

nimmo added this to the NumTide Repository Evaluation milestone

2026-08-06 23:24:31 +01:00

nimmo added the

area:tbd

repo:numtide/nix-relay

priority:medium

investigation

labels

2026-08-06 23:24:31 +01:00

nimmo commented

2026-08-09 23:03:44 +01:00

Author

Owner

Evaluation conclusion: do not integrate with the current runner topology

Reviewed upstream numtide/nix-relay at commit 03be587.

What it actually provides

nix-relay is an OIDC-authenticated remote Nix build relay, not a binary-cache relay. A CI client bridges Nix's ssh-ng store protocol over WebSocket; the server validates a token and launches nix-daemon --stdio on the builder.

Its primary zero-static-key workflow targets GitHub Actions OIDC. The current client source also handles GitLab and CircleCI tokens or an explicit NIX_RELAY_TOKEN, but it does not automatically acquire a Forgejo Actions identity token. Local signed JWTs are supported, at the cost of provisioning and rotating another credential.

Fit with this repository

All Forgejo jobs that build or review this repository already use the self-hosted nova runner. The expensive Nix build therefore runs directly on the strongest designated builder with its persistent local Nix store.
No ephemeral hosted runner currently needs secure access to a separate build farm, and no distributed-builder bottleneck is documented.
Adding the relay would introduce a WebSocket service, TLS/ingress, token issuance or Forgejo OIDC compatibility work, client packaging, and remote-store failure modes without moving work to a better machine.
The upstream security notes state that the spawned nix-daemon --stdio is trusted by default and recommend considering the experimental --force-untrusted path. That trust boundary deserves a dedicated threat model before any deployment.
The project is young, with no release or tag at the reviewed point. The implementation is promising, but the current environment does not supply its intended use case.

Validation performed

nix flake check --no-build passed against the reviewed upstream commit.
The NixOS module and package evaluated successfully against this repository's pinned nixpkgs (26.11 at the evaluated revision) using a container test configuration.
Source review confirmed OIDC/local-JWT validation, the client token paths, and the trusted-daemon warning.

Decision

Do not integrate or deploy nix-relay now, and do not create a live proof of concept. Nova already performs the builds locally, so a relay would add a privileged network boundary without increasing useful build capacity.

Revisit if the project adopts ephemeral/off-site runners, creates a real multi-builder pool, or needs to expose remote builds without SSH keys. At that point, first verify Forgejo's issuer and claim compatibility, require least-privilege daemon behavior, and benchmark against ordinary Nix remote builders over the existing Nebula mesh.

## Evaluation conclusion: do not integrate with the current runner topology Reviewed upstream `numtide/nix-relay` at commit [`03be587`](https://github.com/numtide/nix-relay/tree/03be5875c3da85743d887c5fd3923dcbf007a41c). ### What it actually provides `nix-relay` is an [OIDC-authenticated remote Nix build relay](https://github.com/numtide/nix-relay/blob/03be5875c3da85743d887c5fd3923dcbf007a41c/README.md#nix-relay), not a binary-cache relay. A CI client bridges Nix's `ssh-ng` store protocol over WebSocket; the server validates a token and launches `nix-daemon --stdio` on the builder. Its primary zero-static-key workflow targets GitHub Actions OIDC. The current client source also handles GitLab and CircleCI tokens or an explicit `NIX_RELAY_TOKEN`, but it does not automatically acquire a Forgejo Actions identity token. Local signed JWTs are supported, at the cost of provisioning and rotating another credential. ### Fit with this repository - All Forgejo jobs that build or review this repository already use the self-hosted `nova` runner. The expensive Nix build therefore runs directly on the strongest designated builder with its persistent local Nix store. - No ephemeral hosted runner currently needs secure access to a separate build farm, and no distributed-builder bottleneck is documented. - Adding the relay would introduce a WebSocket service, TLS/ingress, token issuance or Forgejo OIDC compatibility work, client packaging, and remote-store failure modes without moving work to a better machine. - The upstream security notes state that the spawned `nix-daemon --stdio` is trusted by default and recommend considering the experimental `--force-untrusted` path. That trust boundary deserves a dedicated threat model before any deployment. - The project is young, with no release or tag at the reviewed point. The implementation is promising, but the current environment does not supply its intended use case. ### Validation performed - `nix flake check --no-build` passed against the reviewed upstream commit. - The NixOS module and package evaluated successfully against this repository's pinned nixpkgs (`26.11` at the evaluated revision) using a container test configuration. - Source review confirmed OIDC/local-JWT validation, the client token paths, and the trusted-daemon warning. ### Decision Do not integrate or deploy `nix-relay` now, and do not create a live proof of concept. Nova already performs the builds locally, so a relay would add a privileged network boundary without increasing useful build capacity. Revisit if the project adopts ephemeral/off-site runners, creates a real multi-builder pool, or needs to expose remote builds without SSH keys. At that point, first verify Forgejo's issuer and claim compatibility, require least-privilege daemon behavior, and benchmark against ordinary Nix remote builders over the existing Nebula mesh.

nimmo closed this issue

2026-08-09 23:03:54 +01:00