ADR-002: Place workloads by operational fit, with native NixOS as the default

• Status: Accepted
• Date: 2026-06-30
• Scope: Application packaging, service modules, OCI containers, and wrapper repositories
• Supersedes: None
• Superseded by: None

Context

The fleet has inherited Docker Compose applications, first-class NixOS services, applications available only as upstream images, and locally packaged software. Treating all of these identically either leaves important behavior in mutable /srv/stacks state or creates expensive custom packaging for software whose supported operational interface is an image.

The 2026 service-migration work established that native placement materially improves integration when the application fits it. SABnzbd, SillyTavern, Whisper/Piper, Redview, Karakeep, Jellyfin, Tdarr, FreshRSS, Paperless, and Immich moved wholly or partly into NixOS-owned services. Their configuration can now compose declarative users, state paths, mounts, secrets, ordering, monitoring, backup intent, and host-specific acceleration.

The migrations also demonstrated three limits:

• An upstream NixOS module may need a narrow fleet-local wrapper or package override, as with CUDA-enabled Wyoming faster-whisper and Karakeep's scoped dependency workaround.
• Some applications need a separately maintained package/module and update lifecycle. Redview is consumed as an external flake whose own repository builds the NVIDIA runtime closure; nixos-config owns only fleet integration.
• Some applications are still best represented by their official OCI images and private container relationships. SynthSeek, slskd, and Plexdrome on Chaos are current examples.

The Librarry trial proved that a dedicated wrapper repository could package a Go/API and web application reproducibly and participate in guarded flake-input updates. It was removed the same day because the application itself did not fit the fleet. This separates two decisions that must not be conflated: “can this be packaged?” and “should this workload be operated?”

Decision

Choose the least complex placement that gives the fleet adequate declarative ownership, recovery, security, and update control. Apply the following order of preference, but stop when the next step would add more ownership than value.

1. Use an upstream or nixpkgs NixOS service when it fits

Prefer an existing NixOS module and package when it can express the required runtime, state, listener, and dependency behavior. Put fleet-specific option values and composition in modules/services/ or the consuming host.

A small repository-local wrapper is appropriate for host integration, assertions, secrets, mounts, backup contribution, failure monitoring, or a narrow package override. It must not silently become a second general-purpose upstream module.

2. Give reusable packaging an independent repository boundary

Use a dedicated wrapper or application flake when the software is absent from nixpkgs, requires substantial reproducible packaging, or needs its own tests, binary-cache build, release cadence, or upstream pin. That repository owns the package and, where reusable, its NixOS module. nixos-config consumes a pinned flake output and owns only host/service integration.

Before creating that repository, inspect the real upstream build and runtime contract and prove the package locally. A successful build is necessary but not sufficient: confirm that the application is worth operating and that ongoing pin/hash maintenance has an owner.

3. Retain an OCI container when it is the pragmatic supported boundary

Use a container when an official image is the reliable distribution artefact, the build is exotic or impractical to own, the workload depends naturally on a private container topology, or the expected lifetime/value does not justify a new packaging project.

Container placement remains declarative where practical: use virtualisation.oci-containers, SOPS-rendered environment files, explicit mounts and networks, systemd ordering/restart triggers, failure monitoring, and host backup paths. Mutable in-application configuration and image update policy must be acknowledged rather than described as fully reproducible Nix state.

Placement is reviewed per workload

Native migration is not an end in itself. Compare at least:

• availability and quality of an upstream NixOS module/package;
• build complexity, binary artefacts, GPU/runtime requirements, and patch load;
• authoritative state, database, migration, backup, and rollback behavior;
• listener, ingress, secret, runtime-user, and sandboxing requirements;
• coupling to other containers, host devices, NAS mounts, or mutable UI state;
• update source, versioning, validation, cache cost, and maintenance owner;
• operational value and expected lifespan.

Consequences

• Native services are the normal target when NixOS can express the workload without disproportionate packaging ownership.
• Fleet integration stays visible in this repository even when packages or images are produced elsewhere.
• Dedicated wrapper repositories prevent large packaging implementations and independent update automation from becoming host-configuration internals.
• Containers remain a supported architecture, especially on Chaos and in Lyra's transitional estate; their presence is not automatically technical debt.
• Container state, networks, image tags/digests, secrets, and backup coverage still require the same review gates as native services.
• Cross-boundary dependencies are explicit. A native service reached from a container may require a dedicated bridge listener/firewall path, which can outweigh the benefit of a partial migration.
• The fleet may carry both native and container implementations during a tested cutover or rollback window. That transitional duplication must be labelled and eventually resolved.
• Packaging can be retired without deleting its repository when an application fails the operational-fit test, as the Librarry trial demonstrated.

Alternatives considered

Require every service to be native NixOS

Rejected. It would make the fleet responsible for maintaining difficult or short-lived upstream builds and could replace a supported image with a fragile local package merely for architectural uniformity.

Run all third-party applications through Docker Compose

Rejected. It leaves service definitions and important relationships in mutable stack directories, weakens composition with NixOS users/secrets/mounts/backups, and makes evaluated fleet state incomplete when good native modules already exist.

Put all custom packaging directly in nixos-config

Rejected for substantial or reusable packaging. It couples independent source updates, hashes, CI, and binary artefacts to fleet policy and makes reuse or testing outside the host configuration unnecessarily difficult. Small scoped overrides and launchers remain acceptable locally.

Always create a wrapper repository before using a container

Rejected. The Librarry experiment proved the mechanism, not a universal need. Buildability, operational value, and long-term ownership must all justify the extra repository.

References

• Engineering procedure: Adding or Migrating a Service
• Current placement: Service Catalogue and Host Inventory
• Migration procedure: Service Migration and Cutover
• Current container boundaries: Network and Access Topology and Storage and Data Map
• Packaging procedure: Wrapper-Repository Packaging
• Current implementation: modules/services/, modules/containers/, modules/virtualization/, hosts/chaos/default.nix, and the redview input in flake.nix
• Completed migration evidence: Forgejo issues #50, #185, #193, and #198
• Native-wrapper examples: repository commits a78ddbda67670c9355b7a580b1866255c80807a1 (Wyoming), d09390a57e4263e33f35fec03a55efd8bcaa8720 (Redview), and f90a85ca505d9a57143035dab05fc10427a2f256 (Karakeep)
• Container example: repository commit 68159fc3097a9a06bafc4e6f5eff211626a00fdd (slskd)
• Wrapper-repository experiment and retirement: repository commits 8de26d756e905d92fa08ea9bc1b2443d88ef1c93 and 3484086395801ff31f558dca2a63c2207a747430