Secret Editing and Recipient Rotation

Type: Runbook

Status: Current

Scope: Repository-managed SOPS files, host and recovery age recipients, application credential changes, and decryption recovery

Canonical sources: .sops.yaml; justfiles/secrets.just; modules/common/sops-host.nix; secrets/README.md; service sops.secrets, sops.templates, and LoadCredential declarations; ADR-006

Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); current source differs only in flake.lock. Commands and creation-rule shape checked 2026-09-12. No secret was decrypted or changed during this review.

Review triggers: SOPS/age tooling, host SSH identities, recovery keys, .sops.yaml rules, secret-file boundaries, service consumers, runtime credential delivery, or rotation-helper changes

Purpose

Edit an encrypted value, grant or remove a decryption identity, and rotate a service credential without confusing those three distinct operations. Policy and rationale belong to ADR-006 and the Security Baseline; this page owns the operating steps.

SOPS encryption is at the file boundary. An age recipient can decrypt every value in that file, not only the key its host or service needs. Ordinary managed hosts decrypt with /etc/ssh/ssh_host_ed25519_key; sops-nix places plaintext at runtime under /run/secrets. The two YubiKey identities and the off-host provisioning identity are deliberate recovery paths.

Prerequisites and safety boundary

Know the exact secrets/<scope>.yaml file, nested key, actual consuming hosts/services, and intended recovery recipients. Verify the corresponding sopsFile and key declarations in source; do not infer consumers from a filename alone.
Work from a reviewed repository revision on an authorised administration host with a working decryption identity. For a host-key or recovery-key rotation, first confirm an independent authorised identity can read every affected file.
Use a dedicated issue, review path, recovery plan, and pre-change annotated checkpoint for key/recipient rotation, as required by Project and Change Governance. Coordinate service-credential changes with their upstream issuer and consumer services.
Keep shell tracing off. Never pass private keys, tokens, passwords, or decrypted YAML as command arguments; do not put them in Git, Nix strings, wiki text, Forgejo comments, agent memory, screenshots, or logs.

The checked-in .sops.yaml is not perfectly least-privilege: its catch-all rule is broad, several legacy split files share a consumer group, and the nebula-ca.yaml rule includes Vega/Lyra despite a comment claiming servers are excluded. Treat these as current exceptions, not approved patterns for new files. Explicit rules must precede the catch-all. Check the actual matched rule and recipients, not a prose comment.

Choose the operation

Need	Change required	What it does not do
Change an application's password/token	Edit the encrypted value and rotate it at its issuer/consumers	Does not narrow SOPS recipients
Allow a new host or recovery identity to decrypt	Add its public age key to the file's creation rule, then `sops updatekeys`	Does not change the plaintext value
Retire a decryption identity	Remove it from every affected rule and re-encrypt every affected file	Does not revoke ciphertext already copied or exposed values
Recover when the ordinary host identity is unavailable	Use an authorised YubiKey or provisioning identity	Does not make an unauthorised host a consumer

Edit an existing encrypted value

Locate the correct scoped file and confirm its current recipients and consumer declarations. For example, server-only Trilium OIDC values live in secrets/trilium-server.yaml, while AI-client ETAPI credentials live in secrets/trilium.yaml. Do not move or duplicate a value across that boundary merely to make editing convenient.
On a host already authorised for that file, run:
```
just edit-secrets backrest
```
Replace the scope; nested files use names such as nebula/vega. This recipe derives the host's age identity from its root-owned SSH private key and opens the selected YAML in SOPS. It does not edit all secrets at once. Avoid editor swap/backup files that persist plaintext.
If the host is not a recipient, use a deliberately authorised recovery identity instead of broadening the rule to include it:
```
just edit-secrets-yubikey trilium-server
# or, with the off-host provisioning identity entered interactively:
just edit-secrets-provisioning trilium-server
```
Check that the YubiKey plugin is available or the provisioning key is retrieved from the password manager privately. Do not paste a private key into a command line or issue. The provisioning helper passes it to SOPS through process environment for this edit; it is not a permanent host key.
Save and exit SOPS, then inspect only encrypted-file and metadata changes:
```
git diff --stat
git diff --check
```
Do not print decrypted content to prove an edit. Have an authorised identity perform a local read/check and confirm the expected YAML path exists without sharing its value. Validate the affected Nix configuration and deploy through Safe Testing and Deployment. Confirm the service's real authentication path after the issuer and all consumers have been coordinated.

Add a new scoped file or key

Decide the consumer set and recovery recipients using ADR-006. A file for one server should not inherit AI-client or whole-fleet recipients; values with different privilege boundaries belong in different files.
Add an explicit, narrowly matched path_regex rule in .sops.yaml before the broad fallback. Check rule order, regular-expression quoting, and that no earlier rule accidentally matches the new path. Include the required consumer hosts and deliberate recovery identities only.
Create secrets/<scope>.yaml with SOPS under an authorised identity; edit plaintext only inside that session. Add the matching sops.secrets declaration and runtime use in the owning Nix module. Prefer a private systemd credential or secret file rather than interpolating plaintext into a Nix string or command line.
Stage the new encrypted YAML and any new Nix file before flake evaluation; Git-based flakes cannot see untracked inputs. Verify encrypted metadata, the selected rule, and decryption using an intended consumer and a recovery identity before deploying. Then run the appropriate Nix and service checks.

The old monolithic secrets/secrets.yaml has been removed; do not recreate it.

Rotate age recipients or a host SSH identity

Inventory every Git-tracked encrypted file that the old identity can decrypt, plus each file the replacement must consume. Check the current .sops.yaml rules and each file's encrypted metadata. The helper recipes enumerate only Git-tracked secrets/*.yaml and nested YAML files; stage a new file first if it must participate.
Ensure independent recovery access works. Add the new public age key as a named anchor and include it in only the explicit creation rules for its actual consumers. For a host SSH key change, keep the old identity available temporarily while the new one is tested. Do not put a private key in .sops.yaml.
From an identity already authorised for each affected file, run sops updatekeys secrets/backrest.yaml (substituting the real file) for that file. Ordinary host SSH keys are not automatically available to the CLI: the repository helpers derive an age identity from the root-owned key and provide it to SOPS. Use the YubiKey or provisioning route for files the current host cannot decrypt. Do not assume one administration host can update every file.

On an authorised host, the current helper mechanism for one file is:
```
LOCAL_AGE_KEY=$(sudo ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key)
sudo env SOPS_AGE_KEY="$LOCAL_AGE_KEY" sops updatekeys secrets/backrest.yaml
unset LOCAL_AGE_KEY
```
Substitute a real filename. Do not use shell tracing, and clear the local variable even if SOPS fails. This gives SOPS the identity for the current process; it does not grant that host permanent access to unrelated files.
Inspect the encrypted metadata to confirm the intended new recipient was added and unrelated recipients were not. Test a real decrypt with the new host or recovery identity; for a managed host, validate sops-nix activation and the consuming service. Do this before retiring the old key.
Remove the old anchor and all references in affected rules, re-run sops updatekeys for every affected tracked file, and verify its recipient metadata and decryption again. Only then remove the old host private key or retire the old recovery identity. For a host replacement, follow Host Provisioning for first-boot SOPS handoff and bootstrap-key removal.

The current just add-secret, just add-secret-remote, and just remove-secret-key helpers transform .sops.yaml and attempt bulk updatekeys; they are not a substitute for choosing per-file consumer rules and testing decryption. The add helpers insert a reference at a single syntactic location and point duplicate-key cases to a nonexistent migrate-host-key recipe. Issue #224 tracks the repair. Do not use these bulk helpers as the routine rotation procedure until they are repaired and tested. The secrets/README.md migration section also mentions the missing recipe; use the manual staged procedure here until #224 is fixed.

Rotate a compromised application credential

Recipient re-encryption alone does not revoke the old plaintext, an already decrypted copy, a leaked process environment, or ciphertext retained in Git history. If exposure is suspected, rotate the credential at its issuer (service, database, OAuth provider, API account, NAS, or certificate authority), then update its SOPS value and deploy all consuming services in a coordinated order. Confirm the new credential works and the old one fails. Revoke tokens, sessions, certificates, or grants as appropriate to that issuer. A Nebula CA exposure requires a separate fleet identity/certificate response, not merely editing its recipient list.

For a shared value, identify every consumer before revocation so one host is not stranded on the old value. Preserve a rollback route where the issuer supports overlap; do not write the previous plaintext into an issue or Git.

Success checks and recovery

The intended consumer and an independent recovery identity can decrypt the exact file; a removed identity is absent from its new ciphertext.
The affected Nix configurations evaluate, sops-nix activation succeeds on required hosts, and real services authenticate with the intended value.
Recipient scope matches actual consumers, including explicit recovery access. Newly added files did not silently fall through to the broad rule.
The issue records which files and identities changed, validation and deployment evidence, and any unresolved revocation risk—never values.

If updatekeys or activation fails, keep the old authorised identity and the last working encrypted revision available. Repair the rule or file using that known-good access, then retest before removing anything. If an identity has already been retired and no current host can decrypt a required file, use a designated YubiKey or the off-host provisioning identity. If no authorised private identity remains, SOPS cannot recover the plaintext from Git; obtain a new credential from its issuer and rebuild the encrypted file. For a suspected compromise, restoring old ciphertext is not a security rollback: revoke and rotate the exposed underlying credential.