nimmo commented

Owner

Objective

Replace the old all-purpose workflow chapters with two distinct bodies of durable documentation: operational runbooks for running/recovering the fleet, and an engineering handbook for changing the repository safely.

Prerequisites and durable context

Milestone: Wiki Knowledge Architecture Rebuild
Complete the foundation issue first. Use stable System Atlas terminology and link to ADRs for rationale.
Project label: project:wiki-rebuild
Harvest still-valid material from the old wiki, but verify every command, recipe, path, service name, specialisation rule, and host fact against current implementation.

Operations and runbooks checklist

Routine Fleet Operations index.
Safe Testing and Deployment, including current-specialisation preservation and update holds.
Host Provisioning, Disko, nixos-anywhere, SSH identity registration, and first deployment.
Monitoring and Service Investigation, including Beszel and systemd evidence collection.
Backup Verification and Partial Restore.
Disaster Recovery and full-host reconstruction.
Electra Secure Boot and TPM/PCR recovery.
Secret Editing and Recipient Rotation.
Auto-Update Incident Response, stopped-service recovery, rejected candidates, remediation evidence, and temporary pins.
Service Migration and Cutover pattern.
Wrapper-Repository Packaging runbook.

Every runbook must state prerequisites, safety boundary, ordered procedure, success checks, rollback/recovery, and canonical sources.

Engineering handbook checklist

Repository Structure and Import Chain.
Boundaries among modules, profiles, hosts, NixOS users, Home Manager, helpers, and packages.
Adding packages and external flake inputs.
Adding a module or profile.
Adding a host.
Adding or migrating a service.
Testing, CI, review, canary, and deployment contracts.
Secret implementation conventions, including runtime credential handling.
Documentation maintenance and page review triggers.

Existing-page disposition

Preserve and revise: Disaster Recovery, Electra Secure Boot, Wrapper-Repo Packaging, Project Tracking, Security Baseline.
Split across reference/ADR/runbook: Configuration Structure, Host Configurations, Specialisations, Secrets Management.
Collapse and rewrite: Workflows, Package Management, Troubleshooting, Quick Reference.
Leave language/theory chapters for the learning-path phase.

Validation

Verify all just recipes against justfiles/ and just --list.
Verify deployment behavior against the current wrapper/scripts, not old prose.
Verify services and recovery paths against current modules and host imports.
Verify new .nix path references exist and use current names.
Run git -C ~/nixos-config.wiki diff --check and validate internal links.
Commit and push in coherent operational/engineering groups; record each commit in issue comments.

Completion criteria

A fleet operator can execute common and emergency procedures without extracting steps from conceptual chapters.
A contributor can place and validate a change without using the README as a second manual.
Procedures, current-state reference, and design rationale link to one another but do not duplicate ownership.

Resume and handoff protocol

At session start, read this issue and its latest comments, pull both repositories, and resume at the first unchecked runbook or handbook page. At every stopping point, comment with source commit, wiki commit, pages completed, commands/facts verified, validation performed, outstanding risks, and the exact next checklist item. A page is not complete until its commands and links are verified and its coherent commit is pushed.

# Objective Replace the old all-purpose workflow chapters with two distinct bodies of durable documentation: operational runbooks for running/recovering the fleet, and an engineering handbook for changing the repository safely. ## Prerequisites and durable context - Milestone: `Wiki Knowledge Architecture Rebuild` - Complete the foundation issue first. Use stable System Atlas terminology and link to ADRs for rationale. - Project label: `project:wiki-rebuild` - Harvest still-valid material from the old wiki, but verify every command, recipe, path, service name, specialisation rule, and host fact against current implementation. ## Operations and runbooks checklist - [ ] Routine Fleet Operations index. - [ ] Safe Testing and Deployment, including current-specialisation preservation and update holds. - [ ] Host Provisioning, Disko, `nixos-anywhere`, SSH identity registration, and first deployment. - [ ] Monitoring and Service Investigation, including Beszel and systemd evidence collection. - [ ] Backup Verification and Partial Restore. - [ ] Disaster Recovery and full-host reconstruction. - [ ] Electra Secure Boot and TPM/PCR recovery. - [ ] Secret Editing and Recipient Rotation. - [ ] Auto-Update Incident Response, stopped-service recovery, rejected candidates, remediation evidence, and temporary pins. - [ ] Service Migration and Cutover pattern. - [ ] Wrapper-Repository Packaging runbook. Every runbook must state prerequisites, safety boundary, ordered procedure, success checks, rollback/recovery, and canonical sources. ## Engineering handbook checklist - [ ] Repository Structure and Import Chain. - [ ] Boundaries among modules, profiles, hosts, NixOS users, Home Manager, helpers, and packages. - [ ] Adding packages and external flake inputs. - [ ] Adding a module or profile. - [ ] Adding a host. - [ ] Adding or migrating a service. - [ ] Testing, CI, review, canary, and deployment contracts. - [ ] Secret implementation conventions, including runtime credential handling. - [ ] Documentation maintenance and page review triggers. ## Existing-page disposition - Preserve and revise: Disaster Recovery, Electra Secure Boot, Wrapper-Repo Packaging, Project Tracking, Security Baseline. - Split across reference/ADR/runbook: Configuration Structure, Host Configurations, Specialisations, Secrets Management. - Collapse and rewrite: Workflows, Package Management, Troubleshooting, Quick Reference. - Leave language/theory chapters for the learning-path phase. ## Validation - [ ] Verify all `just` recipes against `justfiles/` and `just --list`. - [ ] Verify deployment behavior against the current wrapper/scripts, not old prose. - [ ] Verify services and recovery paths against current modules and host imports. - [ ] Verify new `.nix` path references exist and use current names. - [ ] Run `git -C ~/nixos-config.wiki diff --check` and validate internal links. - [ ] Commit and push in coherent operational/engineering groups; record each commit in issue comments. ## Completion criteria - A fleet operator can execute common and emergency procedures without extracting steps from conceptual chapters. - A contributor can place and validate a change without using the README as a second manual. - Procedures, current-state reference, and design rationale link to one another but do not duplicate ownership. ## Resume and handoff protocol At session start, read this issue and its latest comments, pull both repositories, and resume at the first unchecked runbook or handbook page. At every stopping point, comment with source commit, wiki commit, pages completed, commands/facts verified, validation performed, outstanding risks, and the exact next checklist item. A page is not complete until its commands and links are verified and its coherent commit is pushed.

nimmo added this to the Wiki Knowledge Architecture Rebuild milestone

2026-09-12 09:42:46 +01:00

nimmo added the

project:wiki-rebuild

host:fleet

labels

2026-09-12 09:42:46 +01:00

nimmo referenced this issue

2026-09-12 09:43:43 +01:00

[Wiki 1/5] Establish the new information architecture and durable execution contract #219

nimmo referenced this issue

2026-09-12 17:07:21 +01:00

[Wiki 3/5] Establish governance, security policy, and architecture decisions #221

nimmo commented

2026-09-12 17:13:57 +01:00

Author

Owner

Phase 4 started.

Source baseline: 556df88494. Current source HEAD c4c18eb0df differs only by flake.lock.

Published wiki commit 90ae8a9:

31-Routine-Fleet-Operations: current read-only starting points, command applicability and safety boundaries, and routing to state-changing procedures.
32-Safe-Testing-and-Deployment: prerequisites, update holds, validation/build/test/deploy sequence, current-specialisation preservation, remote checkout versus staged-tree behavior, success checks, and rollback boundaries.
30-Operations-and-Runbooks updated to distinguish current pages from remaining Phase 4 work.

Verified against Justfile, justfiles/deploy.just, justfiles/hosts.just, justfiles/maintenance.just, scripts/nixos-deploy.sh, deployment regression tests, AGENTS.md, and README change/deployment contracts. All documented recipe names exist in just --summary. Direct deploy regressions and packaged-wrapper regressions passed. Full-wiki local link validation and git diff --check passed.

Outstanding risk: recovery after storage, boot, credential, or application-state mutation remains deliberately delegated to the subsystem runbooks; a Nix generation rollback is not represented as reversing external state.

Next checklist item: Host Provisioning, including Disko, nixos-anywhere, SSH identity registration, and first deployment.

Phase 4 started. Source baseline: 556df88494686003b1c4f20c8e0b99b5afc16a6e. Current source HEAD c4c18eb0df39d0418dfad2f916a934908c2fb4fb differs only by flake.lock. Published wiki commit 90ae8a9: - 31-Routine-Fleet-Operations: current read-only starting points, command applicability and safety boundaries, and routing to state-changing procedures. - 32-Safe-Testing-and-Deployment: prerequisites, update holds, validation/build/test/deploy sequence, current-specialisation preservation, remote checkout versus staged-tree behavior, success checks, and rollback boundaries. - 30-Operations-and-Runbooks updated to distinguish current pages from remaining Phase 4 work. Verified against Justfile, justfiles/deploy.just, justfiles/hosts.just, justfiles/maintenance.just, scripts/nixos-deploy.sh, deployment regression tests, AGENTS.md, and README change/deployment contracts. All documented recipe names exist in just --summary. Direct deploy regressions and packaged-wrapper regressions passed. Full-wiki local link validation and git diff --check passed. Outstanding risk: recovery after storage, boot, credential, or application-state mutation remains deliberately delegated to the subsystem runbooks; a Nix generation rollback is not represented as reversing external state. Next checklist item: Host Provisioning, including Disko, nixos-anywhere, SSH identity registration, and first deployment.

nimmo referenced this issue

2026-09-12 17:15:15 +01:00

Repair stale new-host and SOPS registration helpers #224

nimmo commented

2026-09-12 17:19:38 +01:00

Author

Owner

Phase 4 checkpoint: Host Provisioning complete.

Published wiki commit 8f5d1b8:

33-Host-Provisioning covers prerequisites, destructive Disko boundary, current makeNixosSystem registration, installer hardware capture, provisioning-key bootstrap, nixos-anywhere execution, first-boot checks, permanent SSH-derived SOPS identity, proof before bootstrap-key removal, success checks, and recovery.
30-Operations-and-Runbooks updated.

Verified source paths and recipes: justfiles/hosts.just, justfiles/secrets.just, scripts/new-host-template.nix, flake.nix, modules/common/base.nix, modules/common/sops-host.nix, .sops.yaml, secrets/README.md, and host Disko imports. All named recipes exist. Full-wiki local links and git diff --check passed.

Verification uncovered source helper defects, now tracked in issue #224:

add-host prints the obsolete inline nixosSystem registration rather than makeNixosSystem;
secret registration does not safely model explicit multi-rule consumer scope and names a nonexistent migration recipe;
provision prints a root-SSH cleanup command even though managed SSH hosts prohibit root login.

The runbook is current because it documents the verified manual/safe path and explicitly gates use of those helper outputs until #224 is fixed. Electra's from-scratch Disko create/format path remains unproven and is called out as a hard safety exception.

Next checklist item: Monitoring and Service Investigation, including Beszel and systemd evidence collection.

Phase 4 checkpoint: Host Provisioning complete. Published wiki commit 8f5d1b8: - 33-Host-Provisioning covers prerequisites, destructive Disko boundary, current makeNixosSystem registration, installer hardware capture, provisioning-key bootstrap, nixos-anywhere execution, first-boot checks, permanent SSH-derived SOPS identity, proof before bootstrap-key removal, success checks, and recovery. - 30-Operations-and-Runbooks updated. Verified source paths and recipes: justfiles/hosts.just, justfiles/secrets.just, scripts/new-host-template.nix, flake.nix, modules/common/base.nix, modules/common/sops-host.nix, .sops.yaml, secrets/README.md, and host Disko imports. All named recipes exist. Full-wiki local links and git diff --check passed. Verification uncovered source helper defects, now tracked in issue #224: - add-host prints the obsolete inline nixosSystem registration rather than makeNixosSystem; - secret registration does not safely model explicit multi-rule consumer scope and names a nonexistent migration recipe; - provision prints a root-SSH cleanup command even though managed SSH hosts prohibit root login. The runbook is current because it documents the verified manual/safe path and explicitly gates use of those helper outputs until #224 is fixed. Electra's from-scratch Disko create/format path remains unproven and is called out as a hard safety exception. Next checklist item: Monitoring and Service Investigation, including Beszel and systemd evidence collection.

nimmo referenced this issue

2026-09-12 19:49:24 +01:00

Remove stale OpenCode service-failure monitor entry on Vega #225

nimmo commented

2026-09-12 19:51:47 +01:00

Author

Owner

Phase 4 checkpoint: Monitoring and Service Investigation complete.

Published wiki commit 9e7061d:

34-Monitoring-and-Service-Investigation provides symptom capture, Beszel triage, independent deployment correlation, bounded systemd/journal evidence, resource/listener/route checks, container evidence, credential timing without secret disclosure, notification coverage, conclusion structure, success criteria, and recovery routing.
Operations and Routine Fleet Operations indexes now link the current runbook.
Legacy Troubleshooting now points service investigation to the current procedure.

Verification performed:

Live Beszel fleet summary and host-detail commands executed successfully.
just fleet-status executed successfully and confirmed that its SSH/revision view is independent of Beszel.
Composed failure-monitor options evaluated for all six NixOS hosts.
Exact OnFailure template and notifier command evaluated from Vega.
Documented local systemd property/list/filter commands checked against electra.
Canonical source paths and fleet-status recipe verified.
Full-wiki local link validation and git diff --check passed.

Live monitoring showed why the runbook avoids overclaiming: Beszel and SSH reachability can disagree, and the host detail can report a failed-service count without naming the unit. Those transient observations were not written into durable wiki state.

Current configuration exception: Vega's composed monitor list still names the retired OpenCode server. Source cleanup and the stale AGENTS.md statement are tracked in issue #225 and linked from the runbook. The runbook also records that ntfy delivery is best-effort and only explicit server-unit lists receive OnFailure handlers.

Next checklist item: Backup Verification and Partial Restore.

Phase 4 checkpoint: Monitoring and Service Investigation complete. Published wiki commit 9e7061d: - 34-Monitoring-and-Service-Investigation provides symptom capture, Beszel triage, independent deployment correlation, bounded systemd/journal evidence, resource/listener/route checks, container evidence, credential timing without secret disclosure, notification coverage, conclusion structure, success criteria, and recovery routing. - Operations and Routine Fleet Operations indexes now link the current runbook. - Legacy Troubleshooting now points service investigation to the current procedure. Verification performed: - Live Beszel fleet summary and host-detail commands executed successfully. - just fleet-status executed successfully and confirmed that its SSH/revision view is independent of Beszel. - Composed failure-monitor options evaluated for all six NixOS hosts. - Exact OnFailure template and notifier command evaluated from Vega. - Documented local systemd property/list/filter commands checked against electra. - Canonical source paths and fleet-status recipe verified. - Full-wiki local link validation and git diff --check passed. Live monitoring showed why the runbook avoids overclaiming: Beszel and SSH reachability can disagree, and the host detail can report a failed-service count without naming the unit. Those transient observations were not written into durable wiki state. Current configuration exception: Vega's composed monitor list still names the retired OpenCode server. Source cleanup and the stale AGENTS.md statement are tracked in issue #225 and linked from the runbook. The runbook also records that ntfy delivery is best-effort and only explicit server-unit lists receive OnFailure handlers. Next checklist item: Backup Verification and Partial Restore.

nimmo referenced this issue

2026-09-12 23:22:14 +01:00

Harden Backrest CLI endpoint and restore recipes #226

nimmo commented

2026-09-12 23:24:46 +01:00

Author

Owner

Phase 4 checkpoint: Backup Verification and Partial Restore complete.

Published wiki commit 62aa379:

35-Backup-Verification-and-Partial-Restore owns coverage checks, snapshot recency, staged restic/Btrfs path recovery, read-back verification, PostgreSQL logical artefact checks, and safe promotion boundaries.
Operations and Routine Fleet Operations indexes updated.
Legacy Disaster Recovery now warns that its full-root instructions are transitional and points partial recovery to the current page.

Verified source commit 556df88494 against justfiles/maintenance.just, Backrest/Btrbk/PostgreSQL backup modules, effective backup options for all six registered hosts, just --list, and Electra's read-only Backrest status, Btrbk timer/journal, and snapshot listings. Current source HEAD c4c18eb0df changes only flake.lock. All documented recipe names and source paths exist; full-wiki local link check and git diff --check passed. No backup data was written or restored as part of documentation verification.

Tracked source defects in #226: interactive restic CLI endpoint differs from Backrest's mesh endpoint; full-root restore runs as the ordinary caller despite root-owned state; partial restore leaves an ambiguous target on failure. The runbook directs recovery to a fresh staging target, refuses to claim that snapshot listing is restore proof, and does not authorise in-place full-root restore.

Known declared gaps remain in Storage and Data Map: Vega PostgreSQL logical dumps are local-only; BookStack MariaDB has no declared recovery artefact; Claire's Lena home is outside the selected primary-user path; other host/service coverage varies. These are not silently described as protected.

Next checklist item: rewrite and verify Disaster Recovery and full-host reconstruction.

Phase 4 checkpoint: Backup Verification and Partial Restore complete. Published wiki commit 62aa379: - 35-Backup-Verification-and-Partial-Restore owns coverage checks, snapshot recency, staged restic/Btrfs path recovery, read-back verification, PostgreSQL logical artefact checks, and safe promotion boundaries. - Operations and Routine Fleet Operations indexes updated. - Legacy Disaster Recovery now warns that its full-root instructions are transitional and points partial recovery to the current page. Verified source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e against justfiles/maintenance.just, Backrest/Btrbk/PostgreSQL backup modules, effective backup options for all six registered hosts, just --list, and Electra's read-only Backrest status, Btrbk timer/journal, and snapshot listings. Current source HEAD c4c18eb0df39d0418dfad2f916a934908c2fb4fb changes only flake.lock. All documented recipe names and source paths exist; full-wiki local link check and git diff --check passed. No backup data was written or restored as part of documentation verification. Tracked source defects in #226: interactive restic CLI endpoint differs from Backrest's mesh endpoint; full-root restore runs as the ordinary caller despite root-owned state; partial restore leaves an ambiguous target on failure. The runbook directs recovery to a fresh staging target, refuses to claim that snapshot listing is restore proof, and does not authorise in-place full-root restore. Known declared gaps remain in Storage and Data Map: Vega PostgreSQL logical dumps are local-only; BookStack MariaDB has no declared recovery artefact; Claire's Lena home is outside the selected primary-user path; other host/service coverage varies. These are not silently described as protected. Next checklist item: rewrite and verify Disaster Recovery and full-host reconstruction.

nimmo commented

2026-09-12 23:34:08 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit 27836cf (Full-Host Recovery and navigation); source commit 556df88494 verified against current source (HEAD c4c18eb, flake.lock-only change). Added current 36-Full-Host-Recovery.md and marked old 20-Disaster-Recovery.md superseded for operations while retaining unique Electra boot-trust history. Verified Backrest plans/--one-file-system, restic helper root-overwrite and endpoint limits, PostgreSQL dump format, host data gaps, and Disko/provisioning boundaries. Validation: wiki git diff --cached --check and internal-link existence check passed; no Nix/config changes. Outstanding: issue #226 prevents recommending just backup-restore for live root; Electra create/format and end-to-end full-host restore remain untested; backup coverage gaps remain explicit. Next checklist item: Electra Secure Boot and TPM/PCR recovery, including consolidation of retained material from legacy page 20.

Phase 4 checkpoint: published wiki commit 27836cf (Full-Host Recovery and navigation); source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e verified against current source (HEAD c4c18eb, flake.lock-only change). Added current 36-Full-Host-Recovery.md and marked old 20-Disaster-Recovery.md superseded for operations while retaining unique Electra boot-trust history. Verified Backrest plans/--one-file-system, restic helper root-overwrite and endpoint limits, PostgreSQL dump format, host data gaps, and Disko/provisioning boundaries. Validation: wiki git diff --cached --check and internal-link existence check passed; no Nix/config changes. Outstanding: issue #226 prevents recommending just backup-restore for live root; Electra create/format and end-to-end full-host restore remain untested; backup coverage gaps remain explicit. Next checklist item: Electra Secure Boot and TPM/PCR recovery, including consolidation of retained material from legacy page 20.

nimmo commented

2026-09-12 23:36:55 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit e5ef86c (37-Electra-Boot-Trust-Recovery.md plus navigation and legacy-page labels). Source baseline 556df88494; current HEAD c4c18eb differs only in flake.lock. Verified current Lanzaboote measured PCR 4+7 policy, two-generation retention, DGPU policy-generation disablement, stable Disko partlabels, tpm2-enrol and tpm2-enrol-pin recipes/just --list. Cross-checked historical pages 20/21 and open issue #42: final post-change encrypted off-host archive is not yet verified. The new page separates signed-boot rollback, TPM/policy repair, firmware trust, LUKS header damage, and disk loss; preserves independent passphrase route and does not claim new live testing. Validation: wiki git diff --cached --check, internal links, source path checks. Outstanding: issue #42 archive; Electra fresh Disko create/format and whole-host recovery remain untested. Next checklist item: Secret Editing and Recipient Rotation.

Phase 4 checkpoint: published wiki commit e5ef86c (37-Electra-Boot-Trust-Recovery.md plus navigation and legacy-page labels). Source baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current HEAD c4c18eb differs only in flake.lock. Verified current Lanzaboote measured PCR 4+7 policy, two-generation retention, DGPU policy-generation disablement, stable Disko partlabels, tpm2-enrol and tpm2-enrol-pin recipes/just --list. Cross-checked historical pages 20/21 and open issue #42: final post-change encrypted off-host archive is not yet verified. The new page separates signed-boot rollback, TPM/policy repair, firmware trust, LUKS header damage, and disk loss; preserves independent passphrase route and does not claim new live testing. Validation: wiki git diff --cached --check, internal links, source path checks. Outstanding: issue #42 archive; Electra fresh Disko create/format and whole-host recovery remain untested. Next checklist item: Secret Editing and Recipient Rotation.

nimmo commented

2026-09-12 23:55:03 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit 8f04c88 (38-Secret-Editing-and-Recipient-Rotation.md, navigation, ADR link, and warning on legacy Secrets Management). Source baseline 556df88494; current source HEAD c4c18eb changes only flake.lock. Verified .sops.yaml rule order/actual recipients, justfiles/secrets.just and just --list, modules/common/sops-host.nix, secrets/README.md, and ADR-006/Security Baseline. Runbook separates plaintext credential change, age recipient change, and issuer-side revocation; requires staged new/old identity testing and independent recovery; does not use broken bulk host-registration helpers as routine rotation. Validation: full-wiki local link check, git diff --cached --check, recipe existence; no secrets decrypted or changed and no Nix/config changes. Outstanding source exceptions: broad fallback/shared rules, nebula-ca comment/recipients mismatch, issue #224 helper defects and stale migrate-host-key README guidance. Next checklist item: Auto-Update Incident Response.

Phase 4 checkpoint: published wiki commit 8f04c88 (38-Secret-Editing-and-Recipient-Rotation.md, navigation, ADR link, and warning on legacy Secrets Management). Source baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD c4c18eb changes only flake.lock. Verified .sops.yaml rule order/actual recipients, justfiles/secrets.just and just --list, modules/common/sops-host.nix, secrets/README.md, and ADR-006/Security Baseline. Runbook separates plaintext credential change, age recipient change, and issuer-side revocation; requires staged new/old identity testing and independent recovery; does not use broken bulk host-registration helpers as routine rotation. Validation: full-wiki local link check, git diff --cached --check, recipe existence; no secrets decrypted or changed and no Nix/config changes. Outstanding source exceptions: broad fallback/shared rules, nebula-ca comment/recipients mismatch, issue #224 helper defects and stale migrate-host-key README guidance. Next checklist item: Auto-Update Incident Response.

nimmo referenced this issue

2026-09-13 00:04:39 +01:00

Clarify host auto-update force and failure-stamp recovery controls #227

nimmo commented

2026-09-13 00:04:54 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit 91f1677 (39-Auto-Update-Incident-Response.md plus Operations, ADR, and Home navigation). Source content baseline 556df88494; current source HEAD c4c18eb differs only in flake.lock, and the pinned updater revision a32a0ba is unchanged. Verified Forgejo workflow routing/dispatch inputs, Attic publication logic, remediation and pin lifecycle scripts, just --list recipes, the pinned upstream updater hold/failure-stamp/force implementation, and the read-only nixos-version configurationRevision output. Validation: full-wiki internal-link check and staged git diff --check passed; no updater, deployment, or recovery command was run live. Runbook separates producer/candidate/publication/consumer incidents, stopped-service preview and controlled restart, review-only remediation pins, and failure-stamp reset after verified repair. The local Justfile/README force wording discrepancy is tracked as #227; force on pull-only hosts bypasses the prior-failure stamp and detached-helper skip but does not update inputs or bypass a hold. Next checklist item: Service Migration and Cutover pattern.

Phase 4 checkpoint: published wiki commit 91f1677 (39-Auto-Update-Incident-Response.md plus Operations, ADR, and Home navigation). Source content baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD c4c18eb differs only in flake.lock, and the pinned updater revision a32a0ba is unchanged. Verified Forgejo workflow routing/dispatch inputs, Attic publication logic, remediation and pin lifecycle scripts, just --list recipes, the pinned upstream updater hold/failure-stamp/force implementation, and the read-only nixos-version configurationRevision output. Validation: full-wiki internal-link check and staged git diff --check passed; no updater, deployment, or recovery command was run live. Runbook separates producer/candidate/publication/consumer incidents, stopped-service preview and controlled restart, review-only remediation pins, and failure-stamp reset after verified repair. The local Justfile/README force wording discrepancy is tracked as #227; force on pull-only hosts bypasses the prior-failure stamp and detached-helper skip but does not update inputs or bypass a hold. Next checklist item: Service Migration and Cutover pattern.

nimmo commented

2026-09-13 07:00:28 +01:00

Author

Owner

Phase 4 checkpoint — Service Migration and Cutover published in wiki commit aab7181 (new 39a-Service-Migration-and-Cutover.md; linked from Home, Operations, Routine Fleet Operations, Service Catalogue, and ADR-002). Source content baseline: 556df88494; current source HEAD during review: b2831d3313 (only flake.lock changed since baseline). Verified Karakeep importSource and guarded one-shot, Jellyfin/Tdarr mount guards, backup.dataPaths, deployment recipe names, local wiki links, and git diff --check. This is a generic cutover procedure with Karakeep as a documented example; no live cutover or configuration change was performed. Follow-up issue #228 tracks retirement of completed one-time import/bootstrap scaffolding after live-state verification, without deleting staged data. Next checklist item: Wrapper-Repository Packaging runbook.

Phase 4 checkpoint — Service Migration and Cutover published in wiki commit aab7181 (new 39a-Service-Migration-and-Cutover.md; linked from Home, Operations, Routine Fleet Operations, Service Catalogue, and ADR-002). Source content baseline: 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD during review: b2831d33132967b4c4fb69e6068b2b05fff8fdff (only flake.lock changed since baseline). Verified Karakeep importSource and guarded one-shot, Jellyfin/Tdarr mount guards, backup.dataPaths, deployment recipe names, local wiki links, and git diff --check. This is a generic cutover procedure with Karakeep as a documented example; no live cutover or configuration change was performed. Follow-up issue #228 tracks retirement of completed one-time import/bootstrap scaffolding after live-state verification, without deleting staged data. Next checklist item: Wrapper-Repository Packaging runbook.

nimmo commented

2026-09-13 07:09:50 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit a5c627a. Added current 39b-Wrapper-Repository-Packaging.md; updated Home, Operations and routine navigation, ADR-002, and the migration runbook; replaced legacy 22-Wrapper-Repo-Service-Packaging.md with a superseded redirect (old text remains in wiki Git history). Source baseline 556df88494; current source HEAD b2831d3313 differs only in flake.lock. Verified flake inputs/specialArgs and Redview/Navidrome MCP consumer patterns, service modules, lib/auto-update-policy.nix, the external updater boundary sourced by scripts/attic-cache-build.sh, just --list and recipe implementations, and local Nix flake update syntax. Corrected two legacy traps: scripts/nixos-update-common.sh is no longer a source-repo file, and just test-auto-update checks remediation regressions rather than input-policy inventory. Validation: all wiki local links resolve, referenced source paths exist, git diff --cached --check passed. No package build, host activation, or live service change was done for this documentation page. Next checklist item: Engineering Handbook, beginning with Repository Structure and Import Chain.

Phase 4 checkpoint: published wiki commit a5c627a. Added current 39b-Wrapper-Repository-Packaging.md; updated Home, Operations and routine navigation, ADR-002, and the migration runbook; replaced legacy 22-Wrapper-Repo-Service-Packaging.md with a superseded redirect (old text remains in wiki Git history). Source baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock. Verified flake inputs/specialArgs and Redview/Navidrome MCP consumer patterns, service modules, lib/auto-update-policy.nix, the external updater boundary sourced by scripts/attic-cache-build.sh, just --list and recipe implementations, and local Nix flake update syntax. Corrected two legacy traps: scripts/nixos-update-common.sh is no longer a source-repo file, and just test-auto-update checks remediation regressions rather than input-policy inventory. Validation: all wiki local links resolve, referenced source paths exist, git diff --cached --check passed. No package build, host activation, or live service change was done for this documentation page. Next checklist item: Engineering Handbook, beginning with Repository Structure and Import Chain.

nimmo commented

2026-09-13 07:37:12 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit 20d023e. Added current 41-Repository-Structure-and-Import-Chain.md, linked it from Home, Engineering Handbook, and System Atlas, and replaced legacy 01-Configuration-Structure.md with a superseded redirect (old text remains in wiki Git history). Source content baseline 556df88494; current source HEAD b2831d3313 differs only in flake.lock. Verified the six flake registrations, stable Cosmos/Chaos input overrides, makeNixosSystem shared modules and configurationRevision, each host entry/import shape, common and laptop/server profile composition, Home Manager home/users/ and extraSpecialArgs, potential/nova and media1 staging, and all referenced source paths. Full-wiki local links and git diff --cached --check pass. This is source-structure documentation only; no Nix check, host activation, or live deployment was needed or performed. Remaining legacy Workflows, Package Management, Quick Reference, and training chapters contain known old paths; keep their transitional warnings until replaced. Next checklist item: Modules, Profiles, Hosts, Users, and Home Manager boundaries.

Phase 4 checkpoint: published wiki commit 20d023e. Added current 41-Repository-Structure-and-Import-Chain.md, linked it from Home, Engineering Handbook, and System Atlas, and replaced legacy 01-Configuration-Structure.md with a superseded redirect (old text remains in wiki Git history). Source content baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock. Verified the six flake registrations, stable Cosmos/Chaos input overrides, makeNixosSystem shared modules and configurationRevision, each host entry/import shape, common and laptop/server profile composition, Home Manager home/users/<primaryUser> and extraSpecialArgs, potential/nova and media1 staging, and all referenced source paths. Full-wiki local links and git diff --cached --check pass. This is source-structure documentation only; no Nix check, host activation, or live deployment was needed or performed. Remaining legacy Workflows, Package Management, Quick Reference, and training chapters contain known old paths; keep their transitional warnings until replaced. Next checklist item: Modules, Profiles, Hosts, Users, and Home Manager boundaries.

nimmo commented

2026-09-13 07:57:21 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit 6546f30. Added current 42-Configuration-Ownership-Boundaries.md and linked it from Home, Engineering Handbook, and 41-Repository-Structure-and-Import-Chain. Source content baseline 556df88494; current source HEAD b2831d3313 differs only in flake.lock. Verified README Architecture Rules and Boundary Decisions against current common/base, backup, GUI/laptop, AI desktop/runtime, Ollama, desktop environment, Paseo daemon, NixOS user, Home Manager, and launcher helper files. Page defines narrowest-owner policy, exact module/profile/host/user/Home Manager/package decision table, intentional mixed-boundary cases, and review gates. Full-wiki local links, referenced source paths, and git diff --cached --check pass. No Nix/config change or live deployment. Next checklist item: Adding Packages and External Flake Inputs.

Phase 4 checkpoint: published wiki commit 6546f30. Added current 42-Configuration-Ownership-Boundaries.md and linked it from Home, Engineering Handbook, and 41-Repository-Structure-and-Import-Chain. Source content baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock. Verified README Architecture Rules and Boundary Decisions against current common/base, backup, GUI/laptop, AI desktop/runtime, Ollama, desktop environment, Paseo daemon, NixOS user, Home Manager, and launcher helper files. Page defines narrowest-owner policy, exact module/profile/host/user/Home Manager/package decision table, intentional mixed-boundary cases, and review gates. Full-wiki local links, referenced source paths, and git diff --cached --check pass. No Nix/config change or live deployment. Next checklist item: Adding Packages and External Flake Inputs.

nimmo commented

2026-09-13 08:45:17 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit a1f933a. Added current 43-Adding-Packages-and-Flake-Inputs.md, linked it from Home/Engineering Handbook/ownership policy, and replaced legacy 05-Package-Management.md with a superseded redirect (old content remains in wiki Git history). Source content baseline 556df88494; current source HEAD b2831d3313 differs only in flake.lock. Verified common, desktop, profile, host, and Home Manager package scopes; real Redview, Navidrome MCP, NUR, TriliumNext, and cross-input Jellyfin output shapes; flake inputs/specialArgs; lib/auto-update-policy.nix; just --list and local nix flake update syntax. Guide requires each new root input to be lock-pinned and classified for actual consumers or explicitly pinned, and distinguishes just test-auto-update remediation regressions from producer input-policy checks. Full-wiki local links, source paths, and git diff --cached --check pass. No lock update, package build, Nix/config change, or live deployment was performed. Next checklist item: Adding a Module or Profile.

Phase 4 checkpoint: published wiki commit a1f933a. Added current 43-Adding-Packages-and-Flake-Inputs.md, linked it from Home/Engineering Handbook/ownership policy, and replaced legacy 05-Package-Management.md with a superseded redirect (old content remains in wiki Git history). Source content baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock. Verified common, desktop, profile, host, and Home Manager package scopes; real Redview, Navidrome MCP, NUR, TriliumNext, and cross-input Jellyfin output shapes; flake inputs/specialArgs; lib/auto-update-policy.nix; just --list and local nix flake update syntax. Guide requires each new root input to be lock-pinned and classified for actual consumers or explicitly pinned, and distinguishes just test-auto-update remediation regressions from producer input-policy checks. Full-wiki local links, source paths, and git diff --cached --check pass. No lock update, package build, Nix/config change, or live deployment was performed. Next checklist item: Adding a Module or Profile.

nimmo commented

2026-09-13 08:54:20 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit 8b3c646. Added current 44-Adding-a-Module-or-Profile.md, linked it from Home/Engineering Handbook/ownership policy, and placed prominent transitional warnings on legacy 02-Workflows.md and 12-Modules.md. Source content baseline 556df88494; current source HEAD b2831d3313 differs only in flake.lock. Verified option-gated Navidrome MCP via ai-desktop -> ai-agents -> service module and Vega enable, import-to-enable gaming in Electra igpu/dgpu shared imports, laptop and backup profile composition, common/base all-host scope, just check/test recipes, and exact source paths. Guide covers when to add a module versus profile, option surface, imports versus activation, staging new Nix files, targeted builds, canary, and rollback. Full-wiki local links and git diff --cached --check pass. No Nix/config change or live deployment. Next checklist item: Adding a Host.

Phase 4 checkpoint: published wiki commit 8b3c646. Added current 44-Adding-a-Module-or-Profile.md, linked it from Home/Engineering Handbook/ownership policy, and placed prominent transitional warnings on legacy 02-Workflows.md and 12-Modules.md. Source content baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock. Verified option-gated Navidrome MCP via ai-desktop -> ai-agents -> service module and Vega enable, import-to-enable gaming in Electra igpu/dgpu shared imports, laptop and backup profile composition, common/base all-host scope, just check/test recipes, and exact source paths. Guide covers when to add a module versus profile, option surface, imports versus activation, staging new Nix files, targeted builds, canary, and rollback. Full-wiki local links and git diff --cached --check pass. No Nix/config change or live deployment. Next checklist item: Adding a Host.

nimmo commented

2026-09-13 11:16:29 +01:00

Author

Owner

Phase 4 checkpoint: published wiki commit 81ca866. Added current 45-Adding-a-Host.md, linked it from Home/Engineering Handbook/structure page/Host Provisioning, and placed a transitional warning on legacy 08-Quick-Reference.md. Source content baseline 556df88494; current source HEAD b2831d3313 differs only in flake.lock. Verified makeNixosSystem and stable channel pairing, host template and add-host behavior, inventory options/recipe, all current host auto-update roles, lib/auto-update-policy.nix host groups, SOPS host key and scoped rules, and issue #224 helper defects. The page separates configuration registration from destructive provisioning and flags that potential/nova is not promotable unchanged: it duplicates the shared updater import and lacks host metadata, which would make the shared Home Manager isServer value false. Full-wiki local links, source paths, and git diff --cached --check pass. No host was registered, built, provisioned, or deployed; no disk or secret was changed. Next checklist item: Adding or Migrating a Service.

Phase 4 checkpoint: published wiki commit 81ca866. Added current 45-Adding-a-Host.md, linked it from Home/Engineering Handbook/structure page/Host Provisioning, and placed a transitional warning on legacy 08-Quick-Reference.md. Source content baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; current source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock. Verified makeNixosSystem and stable channel pairing, host template and add-host behavior, inventory options/recipe, all current host auto-update roles, lib/auto-update-policy.nix host groups, SOPS host key and scoped rules, and issue #224 helper defects. The page separates configuration registration from destructive provisioning and flags that potential/nova is not promotable unchanged: it duplicates the shared updater import and lacks host metadata, which would make the shared Home Manager isServer value false. Full-wiki local links, source paths, and git diff --cached --check pass. No host was registered, built, provisioned, or deployed; no disk or secret was changed. Next checklist item: Adding or Migrating a Service.

nimmo commented

2026-09-13 12:39:09 +01:00

Author

Owner

Phase 4 Engineering Handbook: published Adding or Migrating a Service in wiki commit f34a566 (source baseline 556df88494; source HEAD b2831d3313 differs only in flake.lock). Covers placement, native/wrapper/OCI ownership, runtime state and secret contract, mount guards, backup/monitoring, input policy, validation, and handoff to the live cutover runbook. Updated Home, handbook index, and ADR-002 references. Focused review, staged diff whitespace check, and all internal wiki links passed. No service was deployed or migrated. Next checklist item: Testing and CI Contracts.

Phase 4 Engineering Handbook: published Adding or Migrating a Service in wiki commit f34a566 (source baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock). Covers placement, native/wrapper/OCI ownership, runtime state and secret contract, mount guards, backup/monitoring, input policy, validation, and handoff to the live cutover runbook. Updated Home, handbook index, and ADR-002 references. Focused review, staged diff whitespace check, and all internal wiki links passed. No service was deployed or migrated. Next checklist item: Testing and CI Contracts.

nimmo commented

2026-09-13 12:40:55 +01:00

Author

Owner

Phase 4 Engineering Handbook: published Testing and CI Contracts in wiki commit af9fb11 (source baseline 556df88494; source HEAD b2831d3313 differs only in flake.lock). Documents gate selection, local flake/format/build/test semantics, exact current Forgejo PR test coverage, non-PR cache host builds, advisory AI review, and separate live evidence. Updated Home and handbook index. Focused source review, staged diff whitespace check, and all internal wiki links passed. No CI run or host deployment performed. Next checklist item: Secrets Implementation Conventions.

Phase 4 Engineering Handbook: published Testing and CI Contracts in wiki commit af9fb11 (source baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock). Documents gate selection, local flake/format/build/test semantics, exact current Forgejo PR test coverage, non-PR cache host builds, advisory AI review, and separate live evidence. Updated Home and handbook index. Focused source review, staged diff whitespace check, and all internal wiki links passed. No CI run or host deployment performed. Next checklist item: Secrets Implementation Conventions.

nimmo commented

2026-09-13 12:54:16 +01:00

Author

Owner

Phase 4 Engineering Handbook: published Secrets Implementation Conventions in wiki commit 4f2488e (source baseline 556df88494; source HEAD b2831d3313 differs only in flake.lock). Covers file-level SOPS recipient scope, current rule exceptions, explicit module declarations, runtime file/LoadCredential/template/env delivery, validation and recovery handoff, and issue #224 helper caveat. Updated Home and handbook index. Focused review, staged whitespace check, and internal wiki link check passed; no secret was decrypted or changed. Next item: Documentation Maintenance.

Phase 4 Engineering Handbook: published Secrets Implementation Conventions in wiki commit 4f2488e (source baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock). Covers file-level SOPS recipient scope, current rule exceptions, explicit module declarations, runtime file/LoadCredential/template/env delivery, validation and recovery handoff, and issue #224 helper caveat. Updated Home and handbook index. Focused review, staged whitespace check, and internal wiki link check passed; no secret was decrypted or changed. Next item: Documentation Maintenance.

nimmo commented

2026-09-13 12:56:32 +01:00

Author

Owner

Phase 4 Engineering Handbook: published Documentation Maintenance in wiki commit 2dddedc (source baseline 556df88494; source HEAD b2831d3313 differs only in flake.lock). Covers surface ownership, per-page source review/provenance, status and navigation, link/whitespace review, separate wiki publication, issue evidence, and the outdated legacy skill marker. Home, standards, and handbook index updated. Focused review, staged whitespace check, and internal wiki link check passed. This completes the planned Phase 4 Engineering Handbook pages. Next: audit remaining Phase 4 acceptance criteria and then Phase 5 learning path/final cutover under issue #223.

Phase 4 Engineering Handbook: published Documentation Maintenance in wiki commit 2dddedc (source baseline 556df88494686003b1c4f20c8e0b99b5afc16a6e; source HEAD b2831d33132967b4c4fb69e6068b2b05fff8fdff differs only in flake.lock). Covers surface ownership, per-page source review/provenance, status and navigation, link/whitespace review, separate wiki publication, issue evidence, and the outdated legacy skill marker. Home, standards, and handbook index updated. Focused review, staged whitespace check, and internal wiki link check passed. This completes the planned Phase 4 Engineering Handbook pages. Next: audit remaining Phase 4 acceptance criteria and then Phase 5 learning path/final cutover under issue #223.

nimmo commented

2026-09-13 12:57:42 +01:00

Author

Owner

Phase 4 acceptance audit after wiki 2dddedc: all listed operations and Engineering Handbook pages are published. Compared current 30/40-series just recipe references with just --list; all executable recipe names are present, while references to nonexistent migrate-host-key are explicitly warnings tied to #224. Link and diff checks passed. Legacy Troubleshooting (07), Workflows (02), Quick Reference (08), and conceptual chapters remain Transitional; their durable learning content is assigned to Phase 5 issue #223 before final retirement. Leaving #222 open until that disposition/final cutover is audited; proceeding with #223 Guided Repository Tour.

nimmo commented

2026-09-14 07:35:52 +01:00

Author

Owner

Phase 4 completion: all listed operations/runbook and engineering-handbook pages are published and linked from the current 30/40 indexes. Current wiki commit 3cc52b7 follows the legacy-page retirement in 5d79be5 and current-navigation cutover in e976d14. Focused review checked source modules, host imports, relevant just recipes against just --list, wiki internal links, orphan pages, and git diff --check. The nonexistent migrate-host-key reference is explicitly warned against and tracked by #224; the unsafe whole-root backup-restore behavior is explicitly warned against and tracked by #226. Retired manual chapters remain recoverable in wiki Git history; Electra boot-trust rollout history remains in the 90 section. The remaining final audit and source-doc MR are tracked in #223 and #229.

nimmo closed this issue

2026-09-14 07:35:53 +01:00

Rows
Columns

[Wiki 4/5] Rebuild operational runbooks and the engineering handbook #222

Objective

Prerequisites and durable context

Operations and runbooks checklist

Engineering handbook checklist

Existing-page disposition

Validation

Completion criteria

Resume and handoff protocol