Service Migration and Cutover

Type: Runbook

Status: Current

Scope: Moving an existing service between hosts, runtimes, databases, storage locations, or ingress routes

Canonical sources: AGENTS.md; modules/services/; modules/containers/; relevant hosts/*/default.nix; justfiles/deploy.just; modules/services/backrest.nix; modules/services/karakeep.nix; Service Catalogue; Network and Access Topology; Storage and Data Map

Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); later source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff changes only flake.lock. Service, deployment, and backup contracts checked 2026-09-13; no live cutover was performed.

Review triggers: Service or container modules, storage/database ownership, backup intent, SOPS or ingress changes, deployment wrappers, migration tooling, or cutover/rollback evidence

Purpose

Move one workload without losing its authoritative state, stranding clients, or silently leaving two writers active. This is a repeatable shape for a service-specific Forgejo issue, not permission to cut over any particular service. ADR-002 owns the native NixOS versus wrapper-repository versus OCI placement decision. The Service Catalogue owns declared service placement; it is not proof of live health.

Prerequisites

Open a service-migration issue and milestone under Project and Change Governance. Put the actual source/target inventory, phase checklist, cutover window, acceptance criteria, and rollback decision in the issue. Use a branch and merge request for the service change; keep the issue open until live evidence is recorded.
Identify the authoritative data owner and all writers: app data, database, object/media store, queue, files, mutable container volumes, user uploads, and any state outside this repository. Inspect the current Storage and Data Map and verify actual mounts and backups. A declared Backrest path does not prove a recent usable snapshot, and its --one-file-system plan does not copy nested NAS mounts.
Map every client and ingress edge in the Network and Access Topology: host/mesh listeners, firewall, Docker bridge peers, Pangolin/Newt or retained SWAG routes, DNS, TLS, and authentication. Pangolin, DNS, NAS, and Proxmox changes may be external to the Nix repository and need named owners.
Confirm the target runtime and data format are compatible. Check the source/target app and database versions, schema migration behavior, credential scope, service user/modes, GPU/device needs, mount ordering, and capacity. Do not assume an older service can read state after the new version has migrated its schema.
Establish a verified recovery point and an annotated pre-change tag for externally stateful service moves that cannot be undone by a normal NixOS generation rollback. Keep the source data, container volumes, and old route recoverable through the agreed rollback window.

Safety boundary

At cutover, one authoritative writer must be named. Running old and new instances against the same writable SQLite file, database, queue, or media path can corrupt or split state. A side-by-side target is safe only when it is read-only, isolated on a different copy, or the application explicitly supports replicated writers. Preserve the source until the destination and its backup have been tested; stopping an old Compose stack is not a reason to prune its volumes.

Do not use an unverified whole-root restore or a Nix generation rollback as the data rollback plan. just backup-restore currently has unsafe root-write and privilege behavior tracked in issue #226. For a stateful service, choose a service-specific staged restore or database import and test it before the window. Never copy a live PostgreSQL data directory as a portable replacement for a verified logical dump.

Ordered procedure

1. Record the service contract and go/no-go gates

In the issue, record the current and proposed answers to these questions:

Gate	Evidence required before a cutover
Data	Exact authority, writers, paths/mounts, database, snapshot or dump ID, read-back test, rollback compatibility
Runtime	Source and target versions, Nix/OCI ownership, service user, state directory, startup ordering, devices, resource capacity
Access	Listener/bind, firewall, route owner, DNS/TLS, client endpoints, authentication and secret recipients
Operations	Monitoring, failure notification, backup plan and actual snapshot, logs, health check, upgrade policy
Rollback	Source preserved, old route recoverable, maximum allowed divergence, decision point and owner

Do not mark a gate complete merely because a module evaluates or a process is active. The check should exercise the actual user/client workflow. If an external dependency or a backup is missing, stop or explicitly narrow the claimed recovery outcome in the issue before proceeding.

2. Build the target without making it authoritative

Choose the runtime under ADR-002. For a native service, put reusable integration in modules/services/ and host ownership in the consuming host; for an OCI service, declare container mounts, networks, credentials, and restart/monitoring relationships where practical. Do not create a wrapper repository merely to make the service look native; use the wrapper packaging procedure when independent packaging is justified.

Declare target state paths, ownership/modes, SOPS files and runtime delivery, required mounts, listeners/firewall, service ordering, failure monitoring, and nixosConfig.backup.dataPaths where the host owns data. For example, the Karakeep native module gates its one-time import on a staged source and empty target, orders import before service startup, and contributes /var/lib/karakeep to backup intent. Jellyfin and Tdarr explicitly require their NAS mounts before starting. These are implementation examples, not a promise that another service has the same guard.

Keep public ingress pointed at the old service and avoid two live writers. Validate the target with Safe Testing and Deployment: stage new Nix files before just check, inspect the build diff, run just test on an affected host, and exercise a private target endpoint. On a remote host, create an auto-update hold before test activation. just deploy --host HOST uses the target's own checkout; an unpublished local tree needs a deliberately staged just deploy-remote --host HOST path. Do not confuse a successful local evaluation with target deployment.

3. Quiesce source writers and capture a consistent point

Announce the write freeze and stop or drain every relevant writer, including background workers, scheduled importers, media scanners, and old containers. Record exactly which units or containers were stopped and how they will be restarted. Check in-flight queues and database transactions. Capture the agreed final snapshot/export/dump after quiescence, verify its identity and readability, and copy to a separate staging location. Leave the original state and backup untouched.

Where an application has a dedicated import contract, follow it. Karakeep's repository module, for instance, refuses a non-empty native state directory and imports a stopped copy before first service start; it does not delete the old source. Do not bypass such a guard to force a rerun. PostgreSQL and MariaDB require their own logical or application-consistent migration steps; file-copy guidance for one service does not generalise to another.

4. Activate and route one authority

Install the staged state at the target with verified owner, mode, and service version. Start only the target's intended writers and verify local health, database connection, mount identity, and a private read/write request. When that passes, move the external route, DNS, proxy, or client endpoint as the issue specifies. Confirm the actual public/mesh route reaches the target and the old path no longer accepts writes. Be alert for stale clients or DNS caches that may continue to send writes to the source.

Cutover may cross boundaries not declared here: a Newt route lives in Pangolin, a SWAG proxy may remain in Lyra's mutable Docker stack, and NAS exports are external. Coordinate and record those changes rather than assuming a Nix switch moved them automatically. Keep credentials scoped to the actual consumers; rotate or retire obsolete identities only after the new path is proved and rollback implications are understood.

5. Verify before retiring the source

Exercise the real client path, authentication/OIDC, representative existing data, a new write and read-back, background jobs, and dependent integrations. Compare relevant counts or checksums when the service has a meaningful measure; do not treat a matching process status as data parity. Check systemctl --failed, the specific unit/container journal, route/TLS behavior, and monitoring/notification delivery. Confirm the intended revision and host deployment, not merely a merged PR.

Trigger or observe a fresh target backup and verify a representative path or logical dump can be restored into staging using Backup Verification and Partial Restore. The old host's backup does not automatically protect new target state. For a multi-host change, record a genuine deployment and the required no-op follow-up before closing the issue. Release each auto-update hold only after live validation.

Keep the old instance stopped and its data intact through the rollback window. Retire old ingress, credentials, modules, volumes, and monitoring only in an explicit follow-up after the new authority and backup are proven. Update the Service Catalogue, Storage and Data Map, and Network and Access Topology when declared ownership changes.

Success checks

Exactly one intended writable authority serves the workload; clients no longer write to an old or empty fallback path.
Required data and database state are present, readable, and writable at the target; the service and dependent clients pass their real workflows.
The public/mesh route, authentication, secrets, mounts, monitoring, and backup read-back match the target contract.
The source remains recoverable for the agreed window, or its retirement is separately authorised and evidenced.
The issue contains revision, source/target, final recovery point, cutover time, checks, rollback status, and any unresolved external dependencies.

Rollback and troubleshooting

Before target writes begin, rollback can usually stop the target and return the route to the preserved source. After target writes or schema migration, do not blindly restart the source. First stop both sides, identify the authoritative write set, assess version/schema compatibility, and plan a reverse transfer or restore from the last known-good point. A Git revert or NixOS generation rollback changes service configuration, not mutable data.

If the target starts with an empty library, check mount identity and ordering before any writes. If authentication fails, check the exact SOPS scope and client redirect/issuer configuration without printing secrets. If the route still reaches the old host, inspect Pangolin/Newt, SWAG, DNS, and client caches as appropriate. If a backup is missing or fails read-back, keep the source intact and leave the migration issue open. Use Monitoring and Service Investigation for bounded evidence collection and Full-Host Recovery for a lost host.