nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions
Page: 58 Nix Troubleshooting Primer
Pages
00 Documentation Map and Scope 00 Documentation Standards 00 Start Here and Governance 10 System Atlas 11 Fleet Overview 12 Host Inventory 13 Service Catalogue 14 Network and Access Topology 15 Storage and Data Map 16 Automation and Control Plane 20 Architecture Decisions 21 ADR 001 Electra Hardware Specialisations 22 ADR 002 Workload Placement 23 ADR 003 Network Exposure 23 Project Tracking 24 ADR 004 Engram Durable Agent Memory 24 Security Baseline 25 ADR 005 Auto Update Control Plane 26 ADR 006 Scoped Secret Files 30 Operations and Runbooks 31 Routine Fleet Operations 32 Safe Testing and Deployment 33 Host Provisioning 34 Monitoring and Service Investigation 35 Backup Verification and Partial Restore 36 Full Host Recovery 37 Electra Boot Trust Recovery 38 Secret Editing and Recipient Rotation 39 Auto Update Incident Response 39a Service Migration and Cutover 39b Wrapper Repository Packaging 40 Engineering Handbook 41 Repository Structure and Import Chain 42 Configuration Ownership Boundaries 43 Adding Packages and Flake Inputs 44 Adding a Module or Profile 45 Adding a Host 46 Adding or Migrating a Service 47 Testing and CI Contracts 48 Secrets Implementation Conventions 49 Documentation Maintenance 50 NixOS Learning Path 51 Guided Repository Tour 52 Nix Language Essentials 53 Declarative Configuration Store and Generations 54 Flakes and Inputs 55 NixOS Modules and Option Merging 56 SpecialArgs and Home Manager Argument Flow 57 Electra Specialisations Worked Example 58 Nix Troubleshooting Primer 90 Historical Material 91 Electra Secure Boot Hardening Record Home
1 58 Nix Troubleshooting Primer
Nimmo edited this page 2026-09-13 13:09:51 +01:00
Table of contents

Nix Troubleshooting Primer

  • Type: Tutorial
  • Status: Current
  • Scope: Learning to classify Nix/configuration failures in nixos-config before taking operational action
  • Canonical sources: flake.nix; justfiles/deploy.just; AGENTS.md; Testing and CI Contracts; Safe Testing and Deployment; Monitoring and Service Investigation; NixOS manual
  • Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff changes only flake.lock. Current check/build/test recipes and source layout checked 2026-09-13; no failure was induced or host changed.
  • Review triggers: Validation commands, flake outputs, module layout, deployment wrappers, incident runbooks, or upstream diagnostic guidance

Learning goal and prerequisites

Read an error as evidence about which stage failed, not as an invitation to try random overrides. Work through NixOS Modules and Option Merging first. The NixOS manual has the general recovery reference; this is a safe classification exercise, not a live incident runbook.

Place the failure on the lifecycle

Stage What a failure usually means First safe check
Parse/evaluation or flake check Syntax, missing file/attribute, undeclared or conflicting option, assertion, or a flake check Read the first meaningful error and its file/line; run just check after staging new files
A specific host build The selected closure or package cannot be constructed, even if flake checks pass Build only the affected nixosConfigurations.<host>.config.system.build.toplevel with --no-link and inspect the failed derivation
Test activation or deployment The built system did not activate cleanly, or the wrong target/mode was chosen Confirm exact revision, target, active specialisation, hold, and wrapper output; use Safe Testing and Deployment
Runtime/client path A unit starts or fails, but the real user path, mount, secret, route, or external dependency is wrong Collect unit and client evidence using Monitoring and Service Investigation

Do not equate a green nix flake check with an all-host closure build or a successful deployment. Do not use a NixOS generation rollback as a data restore. Testing and CI Contracts defines what each gate actually covers.

Read the first useful error

Nix often prints a long stack of evaluations. Find the first concrete error: and the nearest path and line in this repository, then trace the import/option chain. Typical interpretations:

  • syntax error or unexpected token: inspect braces, semicolons, list spacing, strings, and the cited line. Format/lint only after understanding the parse failure.
  • attribute ... missing or undefined variable: inspect the exact attribute path and function argument header. A flake input with no packages.<system>.default is not fixed by inventing that output; inspect its real packages, legacyPackages, or nixosModules shape.
  • option ... does not exist: check spelling and whether the module that declares the option is in the host import graph. See Repository Structure and Import Chain.
  • conflicting definitions: find both definitions and determine ownership. mkDefault and mkForce are deliberate priority tools, not a generic way to suppress a duplicate mistake.
  • infinite recursion: find which option value depends on itself through another definition. Consider whether an import or default is using config too early; follow the dependency rather than adding mkForce.
  • missing source path after creating a file: Git-backed flakes do not see untracked .nix or encrypted YAML files. Stage the exact new file, verify its path, then reevaluate.

Warnings also matter. Ignore only a warning already understood and documented as benign, such as the expected dirty-tree notice in a local working tree; review every other diagnostic under AGENTS.md.

Guided exercise

In a clean, read-only checkout, trace the declared Jellyfin service from hosts/lyra/default.nix into modules/services/jellyfin.nix. Imagine the error The option nixosConfig.jellyfin.enable does not exist. Identify the host import and option declaration you would inspect; do not add a second declaration or activate a build. Next imagine the option evaluates but Jellyfin cannot read /mnt/media: why does that belong to mount/startup/runtime investigation rather than Nix syntax debugging? Use the module and Service Migration and Cutover to check the answer.

For real work, reproduce only the narrowest safe gate, record the exact source revision and error class, and escalate to the appropriate engineering or operations page. Never paste secret-bearing logs into a public issue. If a change affects boot, storage, networking, or keys, stop before speculative activation and use its recovery runbook.