3 06 Secrets Management
Nimmo edited this page 2026-05-07 06:53:41 +01:00

Chapter 6: Secrets Management

This chapter explains how secrets (passwords, API keys, repository URLs) are handled in a NixOS configuration where the entire system definition is stored in a public git repository.

The Problem: The Nix Store Is World-Readable

Everything in /nix/store is readable by all users on the system. If you put a password directly in a .nix file:

# DON'T do this
environment.etc."restic-password".text = "hunter2";

That string ends up in /nix/store/abc123-etc-restic-password where any user (or process) can read it. Even if the file is not committed to git, the Nix store is not a safe place for secrets.

NixOS needs a mechanism to deliver secrets to services outside the Nix store, at activation time, into a location that only the intended service can read.

The Solution: sops-nix

sops-nix bridges the gap between encrypted-at-rest secrets and runtime access. It works in three stages:

  1. At rest: Secrets live in secrets/secrets.yaml, encrypted with age. This file is safe to commit to git.

  2. At activation: When the system activates (during nixos-rebuild switch), sops-nix decrypts secrets.yaml using a key available on the host and places each secret as a file under /run/secrets/.

  3. At runtime: Services read their secrets from /run/secrets/ (a tmpfs -- never written to disk).

Why host SSH keys?

sops-nix needs a private key to decrypt secrets. The obvious choice is a user SSH key (~/.ssh/id_ed25519), but this creates a problem: /home may not be mounted during early system activation. If secrets are needed before /home is available, decryption fails.

This configuration uses the host SSH key (/etc/ssh/ssh_host_ed25519_key) instead. This key:

  • Lives on the root filesystem, available from the earliest stages of activation
  • Is generated automatically by sshd on first boot
  • Is unique per machine, so each host can only decrypt secrets intended for it
  • Is root-readable only (mode 0600), protecting it from unprivileged users

How It Is Configured

The sops-nix module (modules/common/sops-host.nix)

This module is imported by hosts/common/default.nix, so it applies to every host:

{ ... }:

{
  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];

  systemd.tmpfiles.rules = [
    "z /etc/ssh/ssh_host_ed25519_key 0600 root root - -"
  ];
}

sops.age.sshKeyPaths tells sops-nix which private key to use for decryption. The tmpfiles rule enforces 0600 root root permissions on the host private key, which sshd requires.

The host SSH key

The host SSH key (/etc/ssh/ssh_host_ed25519_key) is generated automatically by NixOS on first boot. It persists across rebuilds and is available from the earliest stages of system activation, which is why it is used for sops-nix decryption rather than a user SSH key.

The .sops.yaml file

This file in the repository root tells sops which keys can decrypt which files:

keys:
  # electra: host SSH key
  - &electra age1qkndq2jf4ezw3gp2mrq76xhh8u5aczkrux23wxgw35y9pmejj99s90s46l
  # lena: host SSH key
  - &lena age1een3f9hpldw6dcfylznhu6zy0mcqtnzmx0hn7wscxmtd9wv6nv0s4d0sfs
  # vega: host SSH key
  - &vega age1206h9mlmk7anqs7vqpqdzs7xrln20c9wgj95wdw4nuqw2uprt3ds2dnjel
  # YubiKey Nano 5C (backup)
  - &yubikey-nano5c age1yubikey1qf8kmphcjvftsvmp8nrp7xkywhu5qa26f2c6s2ta6t2c85uen2fc2v72slp
  # YubiKey 5 NFC (backup)
  - &yubikey-5-nfc age1yubikey1q0srxtgqt6pryp0gra8a6vq8uyksqh5d60gye8hlqk498gpwqdj8yj67jwj
  # Provisioning age key (private key in password manager)
  - &provision-key age1vlc2eans4eykdev7zrmhuph7w8ytfn50z9uc8gm2jxaccu5dluvqufzxzt

creation_rules:
  - path_regex: secrets/.*\.yaml$
    key_groups:
      - age:
          - *electra
          - *lena
          - *vega
          - *yubikey-nano5c
          - *yubikey-5-nfc
          - *provision-key

Each host's age public key is derived from its SSH host public key using ssh-to-age. The creation_rules section says: any YAML file under secrets/ should be encrypted for all registered keys. The YubiKey and provisioning keys are backup keys — they allow decrypting secrets without a host machine, useful for key rotation or emergency access. When you add a new host, its key is added here so it can decrypt the shared secrets.

The secrets/secrets.yaml file

This is the encrypted secrets file. When viewed raw, it looks like noise:

backrest:
    rest_password: ENC[AES256_GCM,data:...,type:str]
    repo_password: ENC[AES256_GCM,data:...,type:str]
beszel:
    electra:
        agent_env: ENC[AES256_GCM,data:...,type:str]
    lena:
        agent_env: ENC[AES256_GCM,data:...,type:str]
    vega:
        agent_env: ENC[AES256_GCM,data:...,type:str]
opencode-server-password: ENC[AES256_GCM,data:...,type:str]
opencode-openrouter-api-key: ENC[AES256_GCM,data:...,type:str]
sops:
    age:
        - recipient: age1qkndq2jf4...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            ...

The current secrets and which modules use them:

Secret key Module Purpose
backrest/rest_password modules/services/backrest.nix HTTP auth for Backrest REST API
backrest/repo_password modules/services/backrest.nix Restic repository encryption key
beszel/<hostname>/agent_env modules/services/beszel-agent.nix Beszel agent authentication token (per-host)
opencode-server-password modules/profiles/ai-desktop.nix OpenCode server authentication
opencode-openrouter-api-key modules/profiles/ai-desktop.nix OpenRouter API key for opencode

When opened with sops secrets/secrets.yaml (or just edit-secrets), sops decrypts it in-place in your editor. You see the plaintext YAML, make changes, and when you save, sops re-encrypts automatically.

How Modules Consume Secrets

A module that needs a secret declares it in sops.secrets and then references the decrypted path. Here is how modules/services/backrest.nix does it:

Step 1: Declare the secrets

sops.secrets = {
  "backrest_rest_password" = {
    sopsFile = ../../secrets/secrets.yaml;
    key = "backrest/rest_password";   # maps to YAML path: backrest.rest_password
  };
  "backrest_repo_password" = {
    sopsFile = ../../secrets/secrets.yaml;
    key = "backrest/repo_password";   # maps to YAML path: backrest.repo_password
  };
};

The key parameter maps to a path in the YAML structure. backrest/rest_password means the rest_password key under the backrest key. The secret name ("backrest_rest_password") determines the filename under /run/secrets/.

Step 2: Reference the decrypted paths

serviceConfig = {
  LoadCredential = [
    "rest_password:${config.sops.secrets."backrest_rest_password".path}"
    "repo_password:${config.sops.secrets."backrest_repo_password".path}"
  ];
};

config.sops.secrets."backrest_rest_password".path evaluates to /run/secrets/backrest_rest_password at build time. LoadCredential is a systemd mechanism that copies the secret into a private $CREDENTIALS_DIRECTORY for the service, so the secret is never exposed in the process environment or /proc.

Step 3: Read the credentials at runtime

export BACKREST_REST_PASSWORD=$(cat "$CREDENTIALS_DIRECTORY/rest_password")
export BACKREST_REPO_PASSWORD=$(cat "$CREDENTIALS_DIRECTORY/repo_password")

The service script reads from $CREDENTIALS_DIRECTORY (set by systemd) rather than from /run/secrets/ directly. This is defence in depth: even if another process could read /run/secrets/, the credentials directory is private to the service.

The Decryption Flow

Here is the complete path from encrypted YAML to a running service:

1. secrets/secrets.yaml              Encrypted at rest in git (age + sops)
         │
2. nixos-rebuild switch              Triggers system activation
         │
3. sops-nix activation script        Reads /etc/ssh/ssh_host_ed25519_key
         │                           Converts it to age format internally
         │                           Decrypts secrets.yaml
         │
4. /run/secrets/backrest_rest_password  Plaintext on tmpfs (lost on reboot)
   /run/secrets/backrest_repo_password
         │
5. systemd LoadCredential            Copies into private $CREDENTIALS_DIRECTORY
         │
6. backrest service                  Reads credentials, runs backup

Every step after step 1 happens automatically during nixos-rebuild switch. No manual decryption is needed.

Managing Secrets

Editing existing secrets

just edit-secrets

This opens secrets/secrets.yaml in your editor, decrypted. Edit the values, save, and sops re-encrypts on exit. Commit the result:

git add secrets/secrets.yaml
git commit -m "chore: Update restic credentials"

Adding a new secret

  1. Add the value to secrets/secrets.yaml:
just edit-secrets
# Add your new key/value, save and exit
  1. Declare it in a module:
sops.secrets."my_new_secret" = {
  sopsFile = ../../secrets/secrets.yaml;
  key = "path/to/key";   # maps to YAML path: path.to.key
};
  1. Reference the path in your module:
# In a service module:
serviceConfig.LoadCredential = [
  "my_new_secret:${config.sops.secrets."my_new_secret".path}"
];

# In a home-manager activation (for user-facing credentials):
home.activation.myConfig = hmLib.hm.dag.entryAfter [ "writeBoundary" ] ''
  MY_KEY=$(cat "${config.sops.secrets."my_new_secret".path}")
  # ... use the key
'';

Registering a new host

When you add a new machine to this configuration, it needs its own age key so it can decrypt secrets. After the first deployment (which generates the host SSH key):

just add-secret HOSTNAME

This derives the age public key from the host's SSH key, adds it to .sops.yaml, and re-encrypts secrets.yaml for all registered keys. See secrets/README.md for the full procedure including key rotation.

Security Properties

Property How it is achieved
Secrets never in Nix store sops-nix decrypts to /run/secrets/ (tmpfs), bypassing the store entirely
Secrets never on disk /run/secrets/ is tmpfs -- contents exist only in memory, lost on reboot
Per-host isolation Each host has its own age key; .sops.yaml controls which hosts can decrypt which files
Service isolation systemd LoadCredential gives each service a private copy; secrets do not appear in process environment
Safe to commit secrets.yaml is age-encrypted; the plaintext never enters git
Early availability Host SSH keys live on the root filesystem, available before /home is mounted
No manual steps Decryption is automatic during nixos-rebuild switch