Prevent tagging from making the service unavailable #9

New issue

Closed

opened 2026-09-13 13:13:02 +01:00 by nimmo · 1 comment

nimmo commented

2026-09-13 13:13:02 +01:00

Owner

Problem

While a large auto-tagging job is active, https://redview.nimmog.uk intermittently shows Pangolin's “No available server” page. Redview remains running; the tunnel marks target 55 unhealthy because GET http://localhost:3210/health exceeds its timeout.

Observed on Lyra

2026-09-13 13:03–13:06 BST: multiple health-check timeouts; the target transitioned unhealthy/healthy twice.
2026-09-13 13:08–13:09 BST: two more timeouts and another unhealthy interval.
A direct local health request took 6.18 s during tagging (normally about 1–2 ms).
redview.service did not restart or crash.

Likely mechanism

routes/tags.js handles each JSON line from tagger.py by synchronously running better-sqlite3 writes, including a transaction containing one insert per generated tag. These writes run on the Node event loop. During active tagging, this can delay otherwise trivial requests such as /health long enough for Newt to remove the target from its available-server pool.

Requested change

Ensure that active tagging cannot make the HTTP service fail health checks. A solution should avoid long synchronous SQLite work on the request/event-loop path—for example, batch/defer tag-result persistence with regular yields, or move persistence to a worker.

Acceptance criteria

/health remains responsive within the Newt health-check timeout throughout a large tag scan.
Public availability is retained while tagging is active.
Tagging result persistence remains correct and cancellation/restart semantics are preserved.
Add a regression test or focused benchmark that exercises sustained tag-result ingestion alongside health requests.

## Problem While a large auto-tagging job is active, `https://redview.nimmog.uk` intermittently shows Pangolin's “No available server” page. Redview remains running; the tunnel marks target 55 unhealthy because `GET http://localhost:3210/health` exceeds its timeout. ## Observed on Lyra - 2026-09-13 13:03–13:06 BST: multiple health-check timeouts; the target transitioned unhealthy/healthy twice. - 2026-09-13 13:08–13:09 BST: two more timeouts and another unhealthy interval. - A direct local health request took 6.18 s during tagging (normally about 1–2 ms). - `redview.service` did not restart or crash. ## Likely mechanism `routes/tags.js` handles each JSON line from `tagger.py` by synchronously running `better-sqlite3` writes, including a transaction containing one insert per generated tag. These writes run on the Node event loop. During active tagging, this can delay otherwise trivial requests such as `/health` long enough for Newt to remove the target from its available-server pool. ## Requested change Ensure that active tagging cannot make the HTTP service fail health checks. A solution should avoid long synchronous SQLite work on the request/event-loop path—for example, batch/defer tag-result persistence with regular yields, or move persistence to a worker. ## Acceptance criteria - `/health` remains responsive within the Newt health-check timeout throughout a large tag scan. - Public availability is retained while tagging is active. - Tagging result persistence remains correct and cancellation/restart semantics are preserved. - Add a regression test or focused benchmark that exercises sustained tag-result ingestion alongside health requests.

nimmo referenced this issue from a commit

2026-09-13 15:41:47 +01:00

fix: keep HTTP service responsive during tagging and purges

nimmo commented

2026-09-13 15:42:22 +01:00

Author

Owner

Fixed in 57c3fa3.

Root cause: each tagger.py stdout line ran its own synchronous better-sqlite3 transaction on the event loop; sustained ingestion starved /health past Newt's timeout.

Change: tag results are queued in memory and flushed in bounded chunks (2000 rows/tx) every 250ms or at 5000 rows, with setImmediate yields between chunks; the job finalizes by draining the queue before status/marker updates, so close/error/cancel/restart semantics are preserved. Also fixed the same class of problem in purge paths: whole-subreddit/poster/tag archives are now chunked (500 rows/tx with yields), and PATCH /api/media/:id uses async unlink.

A/B on identical workload (1500 images x 40 tags): max /health latency 357ms -> 86ms, job wall time 1886ms -> 875ms. Purge of a 5000-media subreddit (200k cascaded tag deletes): 521ms stall -> bounded ~100ms chunks.

Regression tests: tests/tags-ingest-health.test.js (health latency budget during sustained ingestion, cancel flush, error tagging, hung-tagger flush) and tests/blocked-purge-health.test.js (health budget during a 5000-media purge, chunked tag-purge correctness, single-delete archive). Both fail against the pre-fix code. Full suite: 76/76 pass.

Fixed in 57c3fa3. **Root cause**: each tagger.py stdout line ran its own synchronous better-sqlite3 transaction on the event loop; sustained ingestion starved /health past Newt's timeout. **Change**: tag results are queued in memory and flushed in bounded chunks (2000 rows/tx) every 250ms or at 5000 rows, with setImmediate yields between chunks; the job finalizes by draining the queue before status/marker updates, so close/error/cancel/restart semantics are preserved. Also fixed the same class of problem in purge paths: whole-subreddit/poster/tag archives are now chunked (500 rows/tx with yields), and PATCH /api/media/:id uses async unlink. **A/B on identical workload (1500 images x 40 tags)**: max /health latency 357ms -> 86ms, job wall time 1886ms -> 875ms. Purge of a 5000-media subreddit (200k cascaded tag deletes): 521ms stall -> bounded ~100ms chunks. **Regression tests**: tests/tags-ingest-health.test.js (health latency budget during sustained ingestion, cancel flush, error tagging, hung-tagger flush) and tests/blocked-purge-health.test.js (health budget during a 5000-media purge, chunked tag-purge correctness, single-delete archive). Both fail against the pre-fix code. Full suite: 76/76 pass.

nimmo closed this issue

2026-09-13 15:42:22 +01:00

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

nimmo/redview#9

No description provided.

Rows
Columns