nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions
Page: 31 Routine Fleet Operations
Pages
00 Documentation Map and Scope 00 Documentation Standards 00 Start Here and Governance 10 System Atlas 11 Fleet Overview 12 Host Inventory 13 Service Catalogue 14 Network and Access Topology 15 Storage and Data Map 16 Automation and Control Plane 20 Architecture Decisions 21 ADR 001 Electra Hardware Specialisations 22 ADR 002 Workload Placement 23 ADR 003 Network Exposure 23 Project Tracking 24 ADR 004 Engram Durable Agent Memory 24 Security Baseline 25 ADR 005 Auto Update Control Plane 26 ADR 006 Scoped Secret Files 30 Operations and Runbooks 31 Routine Fleet Operations 32 Safe Testing and Deployment 33 Host Provisioning 34 Monitoring and Service Investigation 35 Backup Verification and Partial Restore 36 Full Host Recovery 37 Electra Boot Trust Recovery 38 Secret Editing and Recipient Rotation 39 Auto Update Incident Response 39a Service Migration and Cutover 39b Wrapper Repository Packaging 40 Engineering Handbook 41 Repository Structure and Import Chain 42 Configuration Ownership Boundaries 43 Adding Packages and Flake Inputs 44 Adding a Module or Profile 45 Adding a Host 46 Adding or Migrating a Service 47 Testing and CI Contracts 48 Secrets Implementation Conventions 49 Documentation Maintenance 50 NixOS Learning Path 51 Guided Repository Tour 52 Nix Language Essentials 53 Declarative Configuration Store and Generations 54 Flakes and Inputs 55 NixOS Modules and Option Merging 56 SpecialArgs and Home Manager Argument Flow 57 Electra Specialisations Worked Example 58 Nix Troubleshooting Primer 90 Historical Material 91 Electra Secure Boot Hardening Record Home
8 31 Routine Fleet Operations
Nimmo edited this page 2026-09-13 07:09:32 +01:00
Table of contents

Routine Fleet Operations

  • Type: Index
  • Status: Current
  • Scope: Managed NixOS fleet
  • Canonical sources: Justfile; justfiles/deploy.just; justfiles/hosts.just; justfiles/maintenance.just; service-specific runbooks linked below
  • Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12)
  • Review triggers: Changes to public just recipes, fleet inventory, deployment wrappers, monitoring, backups, maintenance services, or incident runbooks

Use this page to choose a current procedure and to begin routine inspection without changing fleet state. Run commands from the nixos-config checkout unless a runbook says otherwise.

Before changing anything

  1. Confirm the machine and, on Electra, its active hardware specialisation:

    hostname
cat /etc/nixos-specialisation 2>/dev/null || true

  2. Check the repository and fleet view:

    git status --short
just fleet-status

  3. Use the relevant runbook before a command that activates configuration, changes credentials, writes storage, updates firmware, or repairs a service.

just fleet-status fetches origin, evaluates the configured host set, and probes each host over SSH. An unreachable row is an observation, not proof that the machine or its services are down. Use the monitoring runbook when that distinction matters.

Read-only starting points

Question Command Notes
What hosts and declared capabilities exist? just inventory Use just inventory me or a hostname to filter.
Which generation and revision is each reachable host running? just fleet-status Covers Nix metadata; Beszel owns live resource and health metrics.
What would a local build change? just build-diff Builds without activation and preserves the detected specialisation for comparison.
What changed in the current generation? just diff-last-update Add detailed to include dependency changes.
Which rollback generations exist? just generations Shows the most recent ten and Electra specialisations where present.
Did automatic update or garbage collection last succeed? just auto-update-last-run; just nixos-gc-last-run Reads systemd result metadata.
Is an automatic-update hold active? just auto-update-hold-status Check on the host being maintained.
Are Backrest and its snapshots visible? just backup-status; just backup-snapshots Visibility is not restore verification.
Is Ollama active on this host? just ollama-status Reports when the unit is not defined.

just --list is the authoritative command index. A recipe's presence does not mean it applies to every host.

Choose a procedure

The Engineering Handbook tracks the remaining Phase 4 contributor guidance.

Safety boundary

Read-only inspection can still contact the Git remote, evaluate the flake, or contact other hosts. It does not authorise a deployment or repair. In particular:

  • just gc, just firmware-update, backup restore recipes, secret recipes, update recovery recipes, and every deploy/switch recipe change state.
  • just test changes the running system until reboot or explicit recovery even though it does not make the build the boot-profile generation.
  • A service being active is not proof that its user-facing path works; use the success checks in the relevant runbook.

If an inspection fails

Preserve the command, exit status, relevant journal interval, host, current revision, and active specialisation. Do not immediately garbage-collect or restart unrelated services: those actions can remove rollback material or destroy useful evidence. Continue with the monitoring or incident runbook that owns the affected subsystem.