nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions
Page: 30 Operations and Runbooks
Pages
00 Documentation Map and Scope 00 Documentation Standards 00 Start Here and Governance 10 System Atlas 11 Fleet Overview 12 Host Inventory 13 Service Catalogue 14 Network and Access Topology 15 Storage and Data Map 16 Automation and Control Plane 20 Architecture Decisions 21 ADR 001 Electra Hardware Specialisations 22 ADR 002 Workload Placement 23 ADR 003 Network Exposure 23 Project Tracking 24 ADR 004 Engram Durable Agent Memory 24 Security Baseline 25 ADR 005 Auto Update Control Plane 26 ADR 006 Scoped Secret Files 30 Operations and Runbooks 31 Routine Fleet Operations 32 Safe Testing and Deployment 33 Host Provisioning 34 Monitoring and Service Investigation 35 Backup Verification and Partial Restore 36 Full Host Recovery 37 Electra Boot Trust Recovery 38 Secret Editing and Recipient Rotation 39 Auto Update Incident Response 39a Service Migration and Cutover 39b Wrapper Repository Packaging 40 Engineering Handbook 41 Repository Structure and Import Chain 42 Configuration Ownership Boundaries 43 Adding Packages and Flake Inputs 44 Adding a Module or Profile 45 Adding a Host 46 Adding or Migrating a Service 47 Testing and CI Contracts 48 Secrets Implementation Conventions 49 Documentation Maintenance 50 NixOS Learning Path 51 Guided Repository Tour 52 Nix Language Essentials 53 Declarative Configuration Store and Generations 54 Flakes and Inputs 55 NixOS Modules and Option Merging 56 SpecialArgs and Home Manager Argument Flow 57 Electra Specialisations Worked Example 58 Nix Troubleshooting Primer 90 Historical Material 91 Electra Secure Boot Hardening Record Home
12 30 Operations and Runbooks
Nimmo edited this page 2026-09-13 13:14:49 +01:00
Table of contents

Operations and Runbooks

  • Type: Index
  • Status: Current
  • Scope: Fleet operations
  • Canonical sources: Justfile, justfiles/, deployment and maintenance scripts, service modules, host configurations, and backup configuration
  • Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12)
  • Review triggers: Changes to user-facing commands, deployment behavior, recovery mechanisms, service ownership, backups, secrets, or incident response

This section owns executable procedures for operating, investigating, repairing, and recovering the fleet. Every runbook states its prerequisites, safety boundary, ordered steps, success checks, and rollback or recovery path.

Current pages

  • Routine Fleet Operations — read-only starting points and routing to the procedure that owns a state-changing action.
  • Safe Testing and Deployment — validation, specialisation-preserving test activation, update holds, deployment, success checks, and recovery boundaries.
  • Host Provisioning — destructive Disko and nixos-anywhere boundary, first deployment, permanent SSH/SOPS identity, and bootstrap-key retirement.
  • Monitoring and Service Investigation — Beszel triage, deployment correlation, bounded systemd/container evidence, route checks, and diagnosis handoff.
  • Backup Verification and Partial Restore — check coverage and recency, stage and inspect individual restic/Btrfs paths, and distinguish a usable restore from a snapshot listing.
  • Full-Host Recovery — classify host loss, build a host-specific recovery manifest, reconstruct the declarative base, stage authoritative state, and verify each service without an unsafe root overwrite.
  • Electra Boot-Trust and TPM Recovery — separate signed-boot, firmware, managed-PCR, TPM-token, LUKS-header, and whole-disk failure cases while retaining independent passphrase access.
  • Secret Editing and Recipient Rotation — scoped SOPS editing, new-file rules, staged age recipient changes, credential revocation, and recovery when an ordinary host key is unavailable.
  • Auto-Update Incident Response — classify producer, candidate, publication, and consumer failures; inspect guarded remediation; recover stopped services; and review temporary pins.
  • Service Migration and Cutover — plan one writable authority, stage data, switch ingress and clients, verify the target and its backup, and retain a data-aware rollback route.
  • Wrapper-Repository Packaging — prove a separate package flake, classify its update consumers, integrate the fleet service, and validate deployment and rollback boundaries.

The original Electra Secure Boot hardening record is historical evidence, not an operating runbook. Retired manual chapters remain in wiki Git history. The operations rebuild is tracked by issue #222.