nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions
Page: 46 Adding or Migrating a Service
Pages
00 Documentation Map and Scope 00 Documentation Standards 00 Start Here and Governance 10 System Atlas 11 Fleet Overview 12 Host Inventory 13 Service Catalogue 14 Network and Access Topology 15 Storage and Data Map 16 Automation and Control Plane 20 Architecture Decisions 21 ADR 001 Electra Hardware Specialisations 22 ADR 002 Workload Placement 23 ADR 003 Network Exposure 23 Project Tracking 24 ADR 004 Engram Durable Agent Memory 24 Security Baseline 25 ADR 005 Auto Update Control Plane 26 ADR 006 Scoped Secret Files 30 Operations and Runbooks 31 Routine Fleet Operations 32 Safe Testing and Deployment 33 Host Provisioning 34 Monitoring and Service Investigation 35 Backup Verification and Partial Restore 36 Full Host Recovery 37 Electra Boot Trust Recovery 38 Secret Editing and Recipient Rotation 39 Auto Update Incident Response 39a Service Migration and Cutover 39b Wrapper Repository Packaging 40 Engineering Handbook 41 Repository Structure and Import Chain 42 Configuration Ownership Boundaries 43 Adding Packages and Flake Inputs 44 Adding a Module or Profile 45 Adding a Host 46 Adding or Migrating a Service 47 Testing and CI Contracts 48 Secrets Implementation Conventions 49 Documentation Maintenance 50 NixOS Learning Path 51 Guided Repository Tour 52 Nix Language Essentials 53 Declarative Configuration Store and Generations 54 Flakes and Inputs 55 NixOS Modules and Option Merging 56 SpecialArgs and Home Manager Argument Flow 57 Electra Specialisations Worked Example 58 Nix Troubleshooting Primer 90 Historical Material 91 Electra Secure Boot Hardening Record Home
1 46 Adding or Migrating a Service
Nimmo edited this page 2026-09-13 11:20:25 +01:00
Table of contents

Adding or Migrating a Service

  • Type: Runbook
  • Status: Current
  • Scope: Engineering the declared runtime and fleet integration for a new or migrated service; live cutover is separate
  • Canonical sources: AGENTS.md; flake.nix; hosts/*/default.nix; modules/services/; modules/containers/; modules/services/backrest.nix; modules/services/service-failure-monitor.nix; lib/auto-update-policy.nix; .sops.yaml; justfiles/deploy.just; ADR-002; Service Migration and Cutover
  • Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff changes only flake.lock. Service examples and validation contract checked 2026-09-13; no service was deployed or migrated.
  • Review triggers: Workload-placement policy, service/container modules, host imports, SOPS delivery, storage/backup, ingress, updater input policy, validation recipes, or service migration procedure

Purpose and boundary

Make a service's runtime, dependencies, state, access, and operational ownership reviewable in Nix. This page covers implementation and pre-deployment evidence. It does not authorise a data move, route change, shutdown of an old instance, or production deployment. For an existing service, Service Migration and Cutover owns the one-writer handover, verified recovery point, external ingress work, and rollback window.

Service changes normally need a Forgejo issue, feature branch, and merge request under AGENTS.md. Record placement, affected hosts and clients, acceptance checks, data authority, security exposure, deployment order, and rollback in the issue. Use ADR-002 to choose native NixOS, an independent wrapper flake, or OCI by operational fit—not by a blanket migration target. Consult the Service Catalogue for declared placement; configuration is not proof of a healthy live service.

Prerequisites and safety boundary

  • Identify the owning host, service user, exact persistent paths and databases, mount origins, readers and writers, devices/GPU, upstream version, and update owner. For a migration, identify the old runtime and schema compatibility before building a target against copied state.
  • Map listeners, firewall interfaces, reverse-proxy routes, DNS/TLS, and authentication against the Network and Access Topology. A loopback or mesh bind is not a public-ingress declaration; external Pangolin/Newt and DNS changes need their own owner and evidence.
  • List every secret, its consuming unit, SOPS file/recipient scope, and runtime delivery path. Never place plaintext credentials in Nix expressions, the Nix store, Git, issue bodies, or documentation.
  • Establish backup scope and an actual usable recovery route before a stateful live cutover. Adding nixosConfig.backup.dataPaths declares backup intent; it does not prove a successful snapshot or include a nested NAS mount. Keep old data and volumes recoverable through the agreed window.

Ordered procedure

1. Choose and document the ownership boundary

Use an upstream/nixpkgs NixOS service when its module expresses the required runtime. Put thin fleet integration in modules/services/, and import and enable it only from intended hosts/<host>/default.nix entries. An option-gated module is useful when several hosts need different values; a direct import can be enough for one owner. Follow Adding a Module or Profile for the module interface.

If substantial reusable packaging, its own CI/cache build, or independent release cadence is needed, follow Wrapper-Repository Packaging. The external repository owns its package and reusable module; this repository pins the flake input and owns host integration. modules/services/redview.nix demonstrates the boundary: it imports the external module and package, then declares Lyra's GPU assertion, media mount, SOPS inputs, backup path, and failure monitor.

If an official image and container topology are the right boundary, declare the container in modules/containers/ or the relevant host module with explicit image/update behaviour, ports, volumes, networks, and systemd relationships. modules/containers/paseo-webui.nix is an OCI example imported by Chaos: it publishes a loopback-bound host port and restarts on configuration revision. Do not assume an image tag or mutable application settings are reproducible Nix state.

2. Declare the runtime contract

Specify the service user/group, package or image, persistent state directory, database, environment, startup dependencies, mounts, resource/device needs, and update behaviour. Assert prerequisites that must hold. For network or NAS state, order the unit after the mount and fail before startup when the expected filesystem is absent; modules/services/jellyfin.nix and modules/services/redview.nix show explicit mount dependencies/guards. Avoid writing application data onto the underlying mountpoint if a remote mount is missing.

Keep secrets in service-scoped SOPS files with only required recipients. Deliver them at runtime through sops.secrets paths, SOPS templates, LoadCredential, or an environment file appropriate to the service; the BookStack, Redview, and Navidrome MCP modules show different current patterns. Review file ownership and modes as well as the process that reads them. Follow Secret Editing and Recipient Rotation for key/recipient work.

Bind to the narrowest required interface, open only the matching firewall path, and name the ingress component separately. Declare health/failure monitoring and add host-owned persistent paths to nixosConfig.backup.dataPaths; verify the resulting backup plan and a readable restore, especially across filesystems. The Security Baseline and Storage and Data Map own the wider review gates.

If a new root flake input is introduced, classify its actual consuming hosts in lib/auto-update-policy.nix and review cache/build cost. Do not leave a new update source unguarded or assign every host without evidence. Update README.md and the relevant service atlas pages when the implementation changes their described contract.

3. Validate without claiming a cutover

Stage new .nix files before flake evaluation. From the source checkout, run just check and a non-activating build for each affected target:

just check
nix build .#nixosConfigurations.HOST.config.system.build.toplevel --no-link

Replace HOST with a registered target. Review the evaluated unit/container, bind/firewall, state paths, mount ordering, SOPS references, update-policy coverage, and build diff. Check required MR/CI results. On an affected NixOS host, use just test for a canary; on a remote target first create an auto-update hold. just test preserves Electra's active specialisation by default. Exercise the real client workflow and inspect logs and backup read-back, not just unit activation. Record exact validation and deployment evidence in the issue. A local build cannot prove the target is deployed.

For a migration, now hand off to Service Migration and Cutover. Keep one authoritative writer, perform a consistent export/import, move routes deliberately, and verify the new service and restore path before retiring old state. Do not prune old container volumes merely because the new unit starts.

Success checks, rollback, and troubleshooting

  • The intended host imports and enables exactly the owned service; placement and any external package/image responsibility are documented.
  • Runtime identity, state, mount guard, secret delivery, listener/firewall, ingress owner, failure monitoring, backup intent, and update policy are explicit. Real backup and client checks are recorded before declaring live success.
  • just check, each affected target build, required MR checks, and an appropriate canary pass. For multi-host or auto-update changes, record a genuine deployment and no-op follow-up before closing the issue.
  • Before live deployment, revert the reviewed source change if necessary. After data migration or schema upgrade, a Nix generation rollback alone may be unsafe; use the service-specific recovery point and route rollback in the cutover issue/runbook.
  • If evaluation fails, check Git staging, host imports, module options, and flake-input policy. If startup fails, check journalctl for the unit, required mount presence, SOPS file permissions, and service dependencies. If local health works but clients fail, inspect listener, interface firewall, ingress route, DNS/TLS, and authentication separately. If backup verification fails, stop before cutover rather than treating the declared path as protection.

Canonical source map

Concern Source
Workflow, review, canary, and publication gates AGENTS.md, justfiles/deploy.just, Safe Testing and Deployment
Placement and external packaging ADR-002, Wrapper-Repository Packaging
Native and external-module examples modules/services/bookstack.nix, modules/services/redview.nix, modules/services/jellyfin.nix, modules/services/navidrome-mcp.nix
OCI and host ownership examples modules/containers/paseo-webui.nix, hosts/chaos/default.nix, hosts/vega/default.nix, hosts/lyra/default.nix
Backup, monitoring, update, and secrets modules/services/backrest.nix, modules/services/service-failure-monitor.nix, lib/auto-update-policy.nix, .sops.yaml
Live data and traffic handover Service Migration and Cutover