nimmo/nixos-config
nimmo/nixos-config
Watch
1
0
Fork
You've already forked nixos-config
 0
Code Issues 15 Pull requests Wiki Activity Actions

[Wiki 2/5] Build the fleet System Atlas #220

New issue
Closed
opened 2026-09-12 09:42:38 +01:00 by nimmo · 5 comments
nimmo commented 2026-09-12 09:42:38 +01:00
Owner
Copy link

Objective

Build the System Atlas: an accurate, relationship-oriented description of the deployed and staged fleet. This is the factual centre of the rebuilt wiki and must be derived from the live repository rather than the old manual.

Prerequisites and durable context

  • Complete the foundation/navigation issue in milestone Wiki Knowledge Architecture Rebuild first.
  • Project label: project:wiki-rebuild
  • Source repository: ~/nixos-config; wiki: ~/nixos-config.wiki
  • Planning baselines: source 556df88494686003b1c4f20c8e0b99b5afc16a6e; wiki 0740498f335157fd3636ab73083263be3af6e852
  • Always pull and verify against the current source HEAD. The repository, not README, AGENTS.md, memories, or the old wiki, is authoritative for consequential facts.
  • Nova remains staged and runs Fedora 43 until deployment is explicitly confirmed.

Scope and checklist

  • Create a concise Fleet Overview showing hosts, networks, shared storage, public ingress, databases, automation, and major service flows.
  • Create Host Inventory covering Electra, Lena, Vega, Lyra, Cosmos, and Chaos; list Nova separately as staged. Derive role, purpose, location, and tags from nixosConfig.host and verify hardware facts in each host directory.
  • Create a Service Catalogue. For each active service record owner host, native/container implementation, listeners/endpoints, ingress path, persistent state, database dependency, authentication, backup contribution, and relevant module/source path.
  • Create Network and Access Topology covering physical LAN exposure, Nebula addressing and DNS, Pangolin/newt ingress, loopback-only listeners, Docker bridges, and relay boundaries. Verify against modules and host firewall declarations.
  • Create Storage and Data Map covering Disko-managed storage, storage deliberately outside Disko, NAS mounts, application state, PostgreSQL placement, snapshots, Backrest/restic, and logical database dumps.
  • Create Automation and Control Plane covering Forgejo Actions, Attic, update producer/consumer roles, holds and recovery, guarded remediation, Paseo, and Engram.
  • Cross-link service entries to the relevant host, network, storage, operations, and ADR pages without duplicating procedural or rationale text.
  • For every page, add page metadata with exact canonical sources and review triggers.
  • Check all stated ports, addresses, host ownership, service state, and deployment status against current code. Flag genuinely unverified live state rather than guessing.
  • Run git -C ~/nixos-config.wiki diff --check, verify internal links, and inspect the rendered tables/diagrams for readability.
  • Commit and push the System Atlas as one or more coherent commits; record each commit and remaining scope in issue comments.

Completion criteria

  • A reader can answer where a service runs, how it is reached, what state it owns, what it depends on, and how it is protected without reading the full repository.
  • Configured, deployed, staged, transitional, and historical states are not conflated.
  • No copied code block is used where a stable relationship or source link is sufficient.
  • All facts have explicit repository provenance and review triggers.

Resume and handoff protocol

At the start of every work session, read this issue and its latest comments, then pull both repositories and compare the recorded source/wiki commits with current HEAD. Continue from the first unchecked catalogue or topology item. At every stopping point, comment with source commit, wiki commit, pages completed, facts still requiring verification, validation performed, and the exact next checklist item. Commit coherent completed pages before stopping. Never use conversation history as the only record of progress.

# Objective Build the System Atlas: an accurate, relationship-oriented description of the deployed and staged fleet. This is the factual centre of the rebuilt wiki and must be derived from the live repository rather than the old manual. ## Prerequisites and durable context - Complete the foundation/navigation issue in milestone `Wiki Knowledge Architecture Rebuild` first. - Project label: `project:wiki-rebuild` - Source repository: `~/nixos-config`; wiki: `~/nixos-config.wiki` - Planning baselines: source `556df88494686003b1c4f20c8e0b99b5afc16a6e`; wiki `0740498f335157fd3636ab73083263be3af6e852` - Always pull and verify against the current source HEAD. The repository, not README, AGENTS.md, memories, or the old wiki, is authoritative for consequential facts. - Nova remains staged and runs Fedora 43 until deployment is explicitly confirmed. ## Scope and checklist - [ ] Create a concise Fleet Overview showing hosts, networks, shared storage, public ingress, databases, automation, and major service flows. - [ ] Create Host Inventory covering Electra, Lena, Vega, Lyra, Cosmos, and Chaos; list Nova separately as staged. Derive role, purpose, location, and tags from `nixosConfig.host` and verify hardware facts in each host directory. - [ ] Create a Service Catalogue. For each active service record owner host, native/container implementation, listeners/endpoints, ingress path, persistent state, database dependency, authentication, backup contribution, and relevant module/source path. - [ ] Create Network and Access Topology covering physical LAN exposure, Nebula addressing and DNS, Pangolin/newt ingress, loopback-only listeners, Docker bridges, and relay boundaries. Verify against modules and host firewall declarations. - [ ] Create Storage and Data Map covering Disko-managed storage, storage deliberately outside Disko, NAS mounts, application state, PostgreSQL placement, snapshots, Backrest/restic, and logical database dumps. - [ ] Create Automation and Control Plane covering Forgejo Actions, Attic, update producer/consumer roles, holds and recovery, guarded remediation, Paseo, and Engram. - [ ] Cross-link service entries to the relevant host, network, storage, operations, and ADR pages without duplicating procedural or rationale text. - [ ] For every page, add page metadata with exact canonical sources and review triggers. - [ ] Check all stated ports, addresses, host ownership, service state, and deployment status against current code. Flag genuinely unverified live state rather than guessing. - [ ] Run `git -C ~/nixos-config.wiki diff --check`, verify internal links, and inspect the rendered tables/diagrams for readability. - [ ] Commit and push the System Atlas as one or more coherent commits; record each commit and remaining scope in issue comments. ## Completion criteria - A reader can answer where a service runs, how it is reached, what state it owns, what it depends on, and how it is protected without reading the full repository. - Configured, deployed, staged, transitional, and historical states are not conflated. - No copied code block is used where a stable relationship or source link is sufficient. - All facts have explicit repository provenance and review triggers. ## Resume and handoff protocol At the start of every work session, read this issue and its latest comments, then pull both repositories and compare the recorded source/wiki commits with current HEAD. Continue from the first unchecked catalogue or topology item. At every stopping point, comment with source commit, wiki commit, pages completed, facts still requiring verification, validation performed, and the exact next checklist item. Commit coherent completed pages before stopping. Never use conversation history as the only record of progress.
nimmo commented 2026-09-12 10:00:30 +01:00
Author
Owner
Copy link

Phase 2 checkpoint: Fleet Overview and Host Inventory

Source verified: 556df88494 (2026-09-12)
Starting wiki baseline: 5da1a97
Wiki published: 1a6cebe (docs(wiki): add fleet overview and host inventory)

Completed:

  • 11-Fleet-Overview
  • 12-Host-Inventory
  • Updated 10-System-Atlas to link both pages

Validation:

  • Evaluated current host inventory metadata, release selection, state versions, and update roles for all six registered configurations.
  • Verified registration, host imports, hardware declarations, specialisation boundaries, and the staged-host boundary from current source.
  • Used current monitoring evidence only to distinguish deployed operating systems from repository intent; transient availability was deliberately excluded from the reference pages.
  • Corrected the staged Nova description to Fedora 44 while retaining its not-yet-NixOS status.
  • git diff --cached --check passed and all changed-page internal links resolved before commit.

Still to verify:

  • Per-service declared versus live or transitional state.
  • Ingress/listener, persistent-state, database, authentication, and backup relationships.
  • Full network, storage, and automation dependency maps.

Exact next action: build 13-Service-Catalogue from host imports and service/container modules, classify configured/deployed/dormant/transitional state, and cross-reference the Host Inventory.

Phase 2 checkpoint: Fleet Overview and Host Inventory Source verified: 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12) Starting wiki baseline: 5da1a97 Wiki published: 1a6cebe (docs(wiki): add fleet overview and host inventory) Completed: - 11-Fleet-Overview - 12-Host-Inventory - Updated 10-System-Atlas to link both pages Validation: - Evaluated current host inventory metadata, release selection, state versions, and update roles for all six registered configurations. - Verified registration, host imports, hardware declarations, specialisation boundaries, and the staged-host boundary from current source. - Used current monitoring evidence only to distinguish deployed operating systems from repository intent; transient availability was deliberately excluded from the reference pages. - Corrected the staged Nova description to Fedora 44 while retaining its not-yet-NixOS status. - git diff --cached --check passed and all changed-page internal links resolved before commit. Still to verify: - Per-service declared versus live or transitional state. - Ingress/listener, persistent-state, database, authentication, and backup relationships. - Full network, storage, and automation dependency maps. Exact next action: build 13-Service-Catalogue from host imports and service/container modules, classify configured/deployed/dormant/transitional state, and cross-reference the Host Inventory.
nimmo commented 2026-09-12 11:06:43 +01:00
Author
Owner
Copy link

Phase 2 checkpoint — service catalogue published

Completed:

  • Added 13-Service-Catalogue.md and linked it from the System Atlas.
  • Classified repository-declared user applications, internal/control-plane services, and fleet-wide supporting services.
  • Recorded owner/form, reachability, durable state/database, access boundary, backup intent, and canonical source.
  • Explicitly separated declarative configuration from live health and from mutable or externally managed infrastructure.

Published wiki commit: fe3359a (docs(wiki): add fleet service catalogue).
Source baseline: 556df88494686003b1c4f20c8e0b99b5afc16a6e.

Exact next step:

  1. Create 14-Network-and-Access-Topology.md.
  2. Map physical/LAN, Nebula, Docker bridges, Newt/Pangolin, and SWAG paths without repeating the per-service catalogue.
  3. Document listener classes, firewall scope, DNS/lighthouse/exit-node ownership, and external trust boundaries.
  4. Reconcile configuration against older topology prose.
  5. Validate links and git diff --check, then commit/push and post the next checkpoint here.
Phase 2 checkpoint — service catalogue published Completed: - Added `13-Service-Catalogue.md` and linked it from the System Atlas. - Classified repository-declared user applications, internal/control-plane services, and fleet-wide supporting services. - Recorded owner/form, reachability, durable state/database, access boundary, backup intent, and canonical source. - Explicitly separated declarative configuration from live health and from mutable or externally managed infrastructure. Published wiki commit: `fe3359a` (`docs(wiki): add fleet service catalogue`). Source baseline: `556df88494686003b1c4f20c8e0b99b5afc16a6e`. Exact next step: 1. Create `14-Network-and-Access-Topology.md`. 2. Map physical/LAN, Nebula, Docker bridges, Newt/Pangolin, and SWAG paths without repeating the per-service catalogue. 3. Document listener classes, firewall scope, DNS/lighthouse/exit-node ownership, and external trust boundaries. 4. Reconcile configuration against older topology prose. 5. Validate links and `git diff --check`, then commit/push and post the next checkpoint here.
nimmo commented 2026-09-12 11:09:53 +01:00
Author
Owner
Copy link

Phase 2 checkpoint — network/access topology published

Completed:

  • Added 14-Network-and-Access-Topology.md and linked it from the System Atlas.
  • Separated public ingress, relay, Nebula, Docker/SWAG, naming, administrative access, and external trust boundaries.
  • Distinguished facts declared in Nix from Pangolin routes and mutable stack state that require external/live inspection.
  • Carried repository-derived exposure observations forward for the later Security Baseline.

Published wiki commit: 546029f (docs(wiki): map fleet network topology).
Source baseline: 556df88494686003b1c4f20c8e0b99b5afc16a6e.

Exact next step:

  1. Create 15-Storage-and-Data-Map.md.
  2. Inventory local filesystems/subvolumes, persistent service roots, NAS mounts, database authority, snapshots, and Backrest/logical-dump coverage.
  3. Label authoritative, derived, cache, staged-download, and externally-owned data.
  4. Record coverage gaps without changing configuration.
  5. Validate, commit/push, and post the next checkpoint here.
Phase 2 checkpoint — network/access topology published Completed: - Added `14-Network-and-Access-Topology.md` and linked it from the System Atlas. - Separated public ingress, relay, Nebula, Docker/SWAG, naming, administrative access, and external trust boundaries. - Distinguished facts declared in Nix from Pangolin routes and mutable stack state that require external/live inspection. - Carried repository-derived exposure observations forward for the later Security Baseline. Published wiki commit: `546029f` (`docs(wiki): map fleet network topology`). Source baseline: `556df88494686003b1c4f20c8e0b99b5afc16a6e`. Exact next step: 1. Create `15-Storage-and-Data-Map.md`. 2. Inventory local filesystems/subvolumes, persistent service roots, NAS mounts, database authority, snapshots, and Backrest/logical-dump coverage. 3. Label authoritative, derived, cache, staged-download, and externally-owned data. 4. Record coverage gaps without changing configuration. 5. Validate, commit/push, and post the next checkpoint here.
nimmo commented 2026-09-12 11:13:42 +01:00
Author
Owner
Copy link

Phase 2 checkpoint — storage/data map published

Completed:

  • Added 15-Storage-and-Data-Map.md and linked it from the System Atlas.
  • Classified authoritative local, authoritative external, derived/cache, staging, and recovery data.
  • Mapped active Disko layouts, external NAS mounts, database authority, local snapshots, and final composed Backrest coverage.
  • Corrected the Service Catalogue so local logical dumps are not misrepresented as off-host backups.
  • Recorded repository-derived recovery coverage gaps for later policy and implementation work.

Published wiki commit: 362797c (docs(wiki): map fleet storage and data).
Source baseline: 556df88494686003b1c4f20c8e0b99b5afc16a6e.

Exact next step:

  1. Create 16-Automation-and-Control-Plane.md.
  2. Map update producer/consumer flow, Forgejo Actions, Attic warming, deployment holds, failure monitoring, Beszel evidence, and AI-agent control-plane services.
  3. Separate automatic decisions from human review gates and current intent from Nova future state.
  4. Validate, commit/push, and post the next checkpoint here.
  5. Then perform a Phase 2 cross-page consistency review and decide whether issue #220 meets its completion criteria.
Phase 2 checkpoint — storage/data map published Completed: - Added `15-Storage-and-Data-Map.md` and linked it from the System Atlas. - Classified authoritative local, authoritative external, derived/cache, staging, and recovery data. - Mapped active Disko layouts, external NAS mounts, database authority, local snapshots, and final composed Backrest coverage. - Corrected the Service Catalogue so local logical dumps are not misrepresented as off-host backups. - Recorded repository-derived recovery coverage gaps for later policy and implementation work. Published wiki commit: `362797c` (`docs(wiki): map fleet storage and data`). Source baseline: `556df88494686003b1c4f20c8e0b99b5afc16a6e`. Exact next step: 1. Create `16-Automation-and-Control-Plane.md`. 2. Map update producer/consumer flow, Forgejo Actions, Attic warming, deployment holds, failure monitoring, Beszel evidence, and AI-agent control-plane services. 3. Separate automatic decisions from human review gates and current intent from Nova future state. 4. Validate, commit/push, and post the next checkpoint here. 5. Then perform a Phase 2 cross-page consistency review and decide whether issue #220 meets its completion criteria.
nimmo commented 2026-09-12 11:58:50 +01:00
Author
Owner
Copy link

Phase 2 completion candidate

Published:

  • 16-Automation-and-Control-Plane.md at wiki commit 1cbecc0.
  • Cross-page consistency and navigation review at wiki commit 2b9af53.

The complete current Atlas now consists of pages 11 through 16. Metadata, local links, table structure, source provenance, and clean worktrees were verified. Forgejo lists all pages at the expected published commits.

No Phase 2 documentation work remains. Issue #220 is ready to close; the next implementation ledger is issue #221.

Phase 2 completion candidate Published: - `16-Automation-and-Control-Plane.md` at wiki commit `1cbecc0`. - Cross-page consistency and navigation review at wiki commit `2b9af53`. The complete current Atlas now consists of pages 11 through 16. Metadata, local links, table structure, source provenance, and clean worktrees were verified. Forgejo lists all pages at the expected published commits. No Phase 2 documentation work remains. Issue #220 is ready to close; the next implementation ledger is issue #221.
nimmo closed this issue 2026-09-12 12:02:25 +01:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/issue-236-electra-nova-vega
feat/issue-233-nova-builder-credentials
docs/wiki-final-cutover
fix/remediation-evaluation-evidence
fix/remediation-evidence-review
fix/chaos-synthseek-queue
pre-electra-disko-migration
Labels
Clear labels   
area:authentication
Issues related to authentication area

  
area:flake-utilities
Issues related to flake utilities area

  
area:performance
Issues related to performance area

  
area:tbd
Area to be determined

  
host:chaos
Issues specific to the chaos host.

  
host:electra
Issues specific to the electra host.

  
host:fleet
Issues that span multiple hosts or the fleet as a whole.

  
host:lyra
Issues specific to the lyra host.

  
host:nova
Issues specific to the nova host.

  
host:vega
Issues specific to the vega host.

  
investigation
Issue is an investigation task

  
phase:cutover
The service cutover execution phase.

  
phase:deploy

   
phase:mcp

   
phase:module

   
phase:packaging

   
phase:prep
Preparation and preflight work before service cutover.

  
phase:validation
Post-cutover validation and rollback-window work.

  
priority:high
High priority issue

  
priority:medium
Medium priority issue

  
project:attic-postgres-lyra-rollout
Issues in the Attic PostgreSQL migration and Lyra rollout workstream.

  
project:auto-update-reliability
Work arising from the automated update reliability review.

  
project:auto-update-remediation
Guarded Nova-assisted diagnosis and remediation of failed automated updates.

  
project:declarative-purity-cleanup
Issues in the declarative purity cleanup workstream.

  
project:external-review
Review of external configurations for ideas and improvements

  
project:fleet-boundary-cleanup
Maintainability work that clarifies fleet configuration and automation boundaries.

  
project:host-facts-refactor
Issues in the host facts refactor workstream.

  
project:lyra-nixos-deploy
Issues in the Lyra NixOS deployment workstream.

  
project:lyra-service-stack-migration
Issues in the lyra service stack migration workstream.

  
project:nebula-mesh-network
Issues in the Nebula mesh network workstream.

  
project:nixos-build-deployment-pipeline
Issues in the NixOS Build & Deployment Pipeline workstream.

  
project:security-hardening
Work belonging to the Security Hardening milestone

  
project:service-stack-migration
Issues in the service stack migration workstream.

  
project:vega-sillytavern-cutover
Issues in the Vega SillyTavern cutover workstream.

  
project:wiki-rebuild
Re-found the repository wiki as the fleet's curated knowledge architecture.

  
repo:numtide/flake-utils
Issues related to numtide/flake-utils repository

  
repo:numtide/nix-auth
Issues related to numtide/nix-auth repository

  
repo:numtide/nixos-passthru-cache
Issues related to numtide/nixos-passthru-cache repository

  
repo:numtide/nix-relay
Issues related to numtide/nix-relay repository

  
service:auto-update
NixOS flake update, cache build, and consumer deployment automation.

  
service:mem0

   
service:nix
Issues concerning Nix build, store, cache, or deployment infrastructure.

  
service:sillytavern
Issues related to the SillyTavern service.

  
service:slskd
Soulseek daemon (slskd) work.

  
service:synthseek
Synthseek music discovery/library manager work.

No labels
area:authentication
area:flake-utilities
area:performance
area:tbd
host:chaos
host:electra
host:fleet
host:lyra
host:nova
host:vega
investigation
phase:cutover
phase:deploy
phase:mcp
phase:module
phase:packaging
phase:prep
phase:validation
priority:high
priority:medium
project:attic-postgres-lyra-rollout
project:auto-update-reliability
project:auto-update-remediation
project:declarative-purity-cleanup
project:external-review
project:fleet-boundary-cleanup
project:host-facts-refactor
project:lyra-nixos-deploy
project:lyra-service-stack-migration
project:nebula-mesh-network
project:nixos-build-deployment-pipeline
project:security-hardening
project:service-stack-migration
project:vega-sillytavern-cutover
project:wiki-rebuild
repo:numtide/flake-utils
repo:numtide/nix-auth
repo:numtide/nixos-passthru-cache
repo:numtide/nix-relay
service:auto-update
service:mem0
service:nix
service:sillytavern
service:slskd
service:synthseek
Milestone
Clear milestone
No items
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
nimmo/nixos-config#220
No description provided.