Improve auto-update hold/failure notifications to ntfy #77

New issue

Closed

opened 2026-07-20 06:07:00 +01:00 by nimmo · 0 comments

nimmo commented

2026-07-20 06:07:00 +01:00

Owner

Problem

When nixos-auto-update is paused — either by an explicit deployment hold (/run/nixos-auto-update/hold) or a prior failed switch (failure stamp at /var/lib/nixos-auto-update/last-failure) — the script exits silently or sends a warning-priority ntfy alert. The user has no clear, low-noise signal that auto-updates are paused and that manual action is required to resume.

Observed on vega (2026-07-18): auto-update suspended after a switch failure; just auto-update-run-now exited immediately with "Updates suspended after failure… rebuild manually to resume." No actionable notification was delivered.

Solution

Implemented in scripts/nixos-auto-update.sh (branch feature/hold-improvement-20260719-221710538):

Hold notifications now use notify_status with low/default priority instead of notify_warning, so they do not page the user like a real failure.
Hold message includes: why the update did not run, when it has been held since, and that a manual just auto-update-release is required.
Failure-suspend path keeps its existing warning/urgent alert (genuine failure still pages).

Validation

nix flake check passes (clean tree).
Branch pushed; MR #78 opened against main.

Live hold test on vega: PASSED. With a hold active, just auto-update-run-now sent a low-priority ntfy status:

Automatic update blocked by deployment hold.
Held since: 2026-07-20T06:22:43+01:00
Created by: nimmo
Reason: testing low-priority notification
Release with: just auto-update-release

Fixed hold message to reference just auto-update-release (commit 03c1435).
No-op follow-up note: a second just auto-update-run-now after release refused to deploy because local main is behind origin (expected safety behaviour pre-merge, not a bug).

Acceptance criteria

Hold active → low-priority ntfy status, not warning
Message states reason + since + manual action needed
Failure-suspend path unchanged (still urgent)
Live hold test passes on vega

## Problem When `nixos-auto-update` is paused — either by an explicit deployment hold (`/run/nixos-auto-update/hold`) or a prior failed switch (failure stamp at `/var/lib/nixos-auto-update/last-failure`) — the script exits silently or sends a `warning`-priority ntfy alert. The user has no clear, low-noise signal that auto-updates are paused and that **manual action is required** to resume. Observed on `vega` (2026-07-18): auto-update suspended after a switch failure; `just auto-update-run-now` exited immediately with "Updates suspended after failure… rebuild manually to resume." No actionable notification was delivered. ## Solution Implemented in `scripts/nixos-auto-update.sh` (branch `feature/hold-improvement-20260719-221710538`): - Hold notifications now use `notify_status` with **low/default priority** instead of `notify_warning`, so they do not page the user like a real failure. - Hold message includes: why the update did not run, when it has been held since, and that a manual `just auto-update-release` is required. - Failure-suspend path keeps its existing `warning`/urgent alert (genuine failure still pages). ## Validation - `nix flake check` passes (clean tree). - Branch pushed; MR #78 opened against `main`. - **Live hold test on vega: PASSED.** With a hold active, `just auto-update-run-now` sent a low-priority ntfy status: ``` Automatic update blocked by deployment hold. Held since: 2026-07-20T06:22:43+01:00 Created by: nimmo Reason: testing low-priority notification Release with: just auto-update-release ``` - Fixed hold message to reference `just auto-update-release` (commit `03c1435`). - No-op follow-up note: a second `just auto-update-run-now` after release refused to deploy because local `main` is behind origin (expected safety behaviour pre-merge, not a bug). ## Acceptance criteria - [x] Hold active → low-priority ntfy status, not warning - [x] Message states reason + since + manual action needed - [x] Failure-suspend path unchanged (still urgent) - [x] Live hold test passes on vega

nimmo referenced this issue from a pull request that will close it,

2026-07-20 06:07:10 +01:00

feat(auto-update): default-priority hold notifications with manual-resume guidance #78

nimmo added reference feature/hold-improvement-20260719-221710538

2026-07-20 07:11:05 +01:00

nimmo added this to the Automated Update Reliability Review milestone

2026-07-20 07:11:55 +01:00

nimmo

2026-07-20 07:24:28 +01:00