Monitoring and Service Investigation

• Type: Runbook
• Status: Current
• Scope: Fleet health triage, NixOS host investigation, systemd services, and container workloads
• Canonical sources: AGENTS.md; justfiles/hosts.just; modules/services/beszel-agent.nix; modules/services/service-failure-monitor.nix; host services.serviceFailureMonitor declarations; service and container modules; Service Catalogue; Automation and Control Plane
• Last verified: Source commit 556df88494686003b1c4f20c8e0b99b5afc16a6e (2026-09-12); composed failure-monitor options and live operator commands checked 2026-09-12
• Review triggers: Changes to Beszel deployment or clients, fleet-status output, systemd failure monitoring, ntfy routing, service ownership, container placement, or logging and retention

Purpose

Turn an alert, failed user request, or suspicious host metric into bounded, timestamped evidence without destroying the state needed to understand the failure. This runbook stops at diagnosis and safe routing: apply a repair only through the owning service, deployment, data, or incident runbook.

Prerequisites

• Record the reported symptom, client path, host or service if known, and the time including timezone.
• Use the nixos-config checkout for just fleet-status and the Beszel helper path declared in AGENTS.md.
• Have SSH access to the suspected host or a console route when SSH is part of the failure.
• Consult the Service Catalogue for declared placement and the Network and Access Topology for the intended client route.
• Know whether the workload is a native systemd service, a NixOS-managed OCI container, or part of the retained mutable Docker estate.

Safety boundary

The inspection commands below are read-only. Do not restart a service, reset its failed state, prune Docker, vacuum journals, garbage-collect, remove a lock, or deploy a configuration until the useful evidence has been captured.

Journals and container logs can contain credentials, URLs, request bodies, personal data, or application content. Review and redact before posting them to Forgejo, chat, notifications, or durable memory. Never collect an entire environment or print files below /run/secrets merely to prove a service has credentials.

Monitoring signals have strict limits:

• Beszel up/down is the hub's view of recent agent contact. “Down” can mean host, agent, route, authentication, or hub trouble; it does not prove power state.
• systemctl is-active describes unit state, not end-to-end application health. An active service may be unable to reach its database or serve its client route.
• A failed-unit count does not identify the unit or establish that it caused the reported symptom.
• No ntfy alert does not prove health. Only explicitly selected server units receive the failure handler, and notification delivery is best-effort.

Procedure

1. Capture the symptom before probing

Write down:

• the exact action or request that failed and the observed response;
• the client location: same host, Docker bridge, LAN, Nebula mesh, relay, or public ingress;
• the first and most recent occurrence with timezone;
• whether the failure is continuous, intermittent, or already recovered;
• recent deployments, restarts, secret rotations, storage events, or network changes that could bound the search window.

Prefer a minimal reproducible user-facing check. Do not repeatedly submit a write, import, migration, or other non-idempotent action.

2. Triage fleet health and deployment state

Start with the Beszel summary:

/home/nimmo/Scripts/beszel-agent/beszel-agent status

It reports the systems known to that Beszel instance, including machines that are not registered NixOS configurations. Treat the Host Inventory as authoritative for fleet membership.

Drill into one host:

/home/nimmo/Scripts/beszel-agent/beszel-agent HOSTNAME

The detail view provides host/resource context and a failed-service count, but may not name the failed unit. Continue with systemd evidence over SSH.

Compare the independent Nix deployment view:

just fleet-status

This fetches origin, evaluates registered hosts, and probes their running kernel, uptime, configuration revision, specialisation, and Git divergence. It does not replace Beszel resource metrics. An unreachable row may be an SSH or name-resolution problem even when Beszel still receives agent data.

Beszel	SSH/fleet-status	First hypothesis to test
Up	Reachable	Service, dependency, storage, or client-route failure
Up	Unreachable	SSH daemon, firewall, name resolution, or path-specific failure
Down	Reachable	Beszel agent, credentials, hub route, or monitoring-only failure
Down	Unreachable	Host or broader network failure; use console and infrastructure evidence

These are starting hypotheses, not conclusions.

3. Establish host and time context

SSH to the host, then capture:

hostname
date --iso-8601=seconds
uptime
nixos-version --json
cat /etc/nixos-specialisation 2>/dev/null || true
systemctl --failed --no-pager

Keep the full configuration revision in the investigation record. On Electra, record the active specialisation and physical GPU-bay state. Use journalctl --list-boots if the failure may cross a reboot boundary.

4. Inspect the exact unit without changing it

Use the full unit name, normally including .service:

systemctl status --no-pager --full UNIT.service
systemctl is-enabled UNIT.service
systemctl show UNIT.service \
  -p LoadState -p ActiveState -p SubState -p Result \
  -p ExecMainStartTimestamp -p ExecMainExitTimestamp \
  -p ExecMainCode -p ExecMainStatus -p NRestarts -p FragmentPath
systemctl cat UNIT.service

The explicit property list avoids dumping the service environment. systemctl cat establishes the effective unit and drop-ins; it does not show the contents of referenced credential or environment files.

If the name is uncertain:

systemctl list-units --all --type=service --no-pager | rg -i 'SEARCH'
systemctl list-unit-files --type=service --no-pager | rg -i 'SEARCH'

5. Read a bounded journal interval

Use the symptom window and host timestamp:

sudo journalctl -u UNIT.service -b \
  --since '2026-09-12 16:00:00' --until '2026-09-12 16:15:00' \
  --no-pager -o short-iso-precise

For a previous boot, use -b -1 after checking journalctl --list-boots. Widen the interval gradually. Preserve the first causal error and preceding context rather than only a repeated downstream failure.

systemctl list-dependencies UNIT.service --no-pager
systemctl list-dependencies --reverse UNIT.service --no-pager

6. Check resource, listener, and route boundaries

free -h
df -h
df -i
sudo ss -ltnup

Inspect the relevant filesystem rather than treating aggregate free space as proof that the service's state path is writable. Compare the listener with the Network and Access Topology: loopback, Docker bridge, LAN, Nebula, and public ingress are different paths.

Test successive boundaries when safe: host-local listener, intended mesh or bridge path, then the real client/ingress path. A local HTTP success does not prove DNS, TLS, identity-aware access, reverse proxying, or the public route.

7. Add container evidence when applicable

For a NixOS OCI container, inspect both its generated docker-NAME.service and Docker state. For retained Compose workloads, first identify the owning stack under /srv/stacks.

docker ps --all --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
docker inspect --format '{{json .State}}' CONTAINER | jq
docker logs --since '2026-09-12T16:00:00' \
  --until '2026-09-12T16:15:00' CONTAINER

Do not run docker compose up, recreate, pull, prune, or remove commands during evidence collection. A container restart can erase the original exit context or trigger application migrations.

8. Check credential and activation timing without reading secrets

When a failure follows secret editing or activation, compare unit start time with metadata for its declared credential/environment paths. sops-nix materialises updated files during activation but does not inherently restart every consumer; a process can continue using an old credential until its unit restarts or declares an appropriate restart trigger.

Use systemctl cat and the owning module to identify delivery through LoadCredential, EnvironmentFile, or another runtime path. Use sudo stat for ownership and timestamps if necessary. Do not use cat, env, systemctl show -p Environment, or /proc/*/environ as a diagnostic shortcut.

9. Understand proactive notification coverage

Every registered host imports the Beszel agent through the common baseline. The service-failure monitor is enabled on server hosts and attaches OnFailure only to their composed explicit service lists. Its handler records unit result/state and the latest 20 journal lines, then attempts a high-priority message to the host-specific ntfy topic.

One current coverage exception is tracked in Forgejo issue #225: Vega's composed list still names the retired OpenCode server. Treat the evaluated list as configuration evidence that must still be reconciled with real owned units.

The send operation tolerates delivery failure so notification trouble does not replace the original unit result. Inspect notification instances with:

systemctl list-units --all 'ntfy-service-failure@*' --no-pager
sudo journalctl -u 'ntfy-service-failure@UNIT.service.service' -b --no-pager

Replace UNIT with the monitored service name. The doubled-looking suffix is intentional: the template instance is the original full UNIT.service name. Client hosts have the option provider but do not enable the notifier, so an empty instance list there is expected.

Evaluate the composed list when exact membership matters:

nix eval --json \
  .#nixosConfigurations.HOSTNAME.config.services.serviceFailureMonitor

10. Record a bounded conclusion

Separate the record into:

1. Observed: timestamps, commands, exact state, exit codes, and redacted log evidence.
2. Correlated: deployment, dependency, resource, route, or credential events in the same time window.
3. Concluded: the narrowest cause supported by evidence.
4. Still unknown: competing explanations not yet disproved.
5. Next action: owning runbook, proposed configuration change, external infrastructure work, or continued observation.

Do not label a restart as a root cause. “Restart restored service” is an observation until the preceding failure mechanism is established.

Success checks

An investigation is complete enough to hand off when:

• the host, component, client path, revision, and time window are explicit;
• monitoring, SSH/deployment, unit, journal, resource, and route signals have been reconciled or their gaps recorded;
• the first relevant failure is preserved with sensitive content removed;
• the scope is known: one client, one process, one host, a dependency, or fleet wide;
• the conclusion distinguishes evidence from inference;
• repair ownership, validation, rollback, and tracking location are identified.

Resolution additionally requires the real client path to work and relevant signals to remain healthy for an appropriate observation period.

Rollback and recovery

This procedure should not change state, so it normally needs no rollback. If a restart, reset, deployment, or container action was performed prematurely, record its exact timestamp because it changes the evidence boundary.

Route repairs as follows:

• configuration or unit definition: Safe Testing and Deployment;
• listener/firewall/ingress mismatch: Network and Access Topology plus the owning service runbook;
• missing/corrupt data: backup or disaster-recovery runbook;
• update failure: auto-update incident-response runbook;
• mutable Docker workload: owning stack/cutover procedure.

Preserve rollback generations and logs until the repair and real client check have succeeded. Do not garbage-collect during an unresolved incident.

Troubleshooting the investigation

• Unit is not found: confirm ownership in the service catalogue and search loaded and installed units. Check for a user service or container rather than inventing a unit name.
• Journal is empty: verify system versus user scope, boot index, time zone, and retention. Missing evidence is not a clean bill of health.
• Beszel data is stale or incomplete: compare last contact with the host clock and SSH evidence; inspect beszel-agent.service separately.
• Failed unit is now active: preserve the earlier journal and result. A retry can restore activity without explaining the original failure.
• Local check succeeds but clients fail: walk outward through listener, firewall, mesh/bridge, proxy, DNS, TLS, and authentication boundaries.

Related material

• Declared placement: Service Catalogue
• Monitoring architecture: Automation and Control Plane
• Route boundaries: Network and Access Topology
• Fleet starting points: Routine Fleet Operations
• Activation boundary: Safe Testing and Deployment