nimmo/nixos-config

Watch

Fork 0

Load-balance Attic across vega and lyra #8

New issue

Closed

opened 2026-07-01 16:14:42 +01:00 by nimmo · 3 comments

nimmo commented

2026-07-01 16:14:42 +01:00

Owner

Goal

Switch the public Attic endpoint from a single backend to dual backends only after both nodes are healthy.

Scope

This issue covers the work from projects/attic-postgres-lyra-rollout.md Phase 4.

Acceptance Checklist

Add both vega:8081 and lyra:8081 as upstreams.
Keep health checks enabled if supported.
Prefer passive failure handling and retries on the alternate backend.
Roll out the proxy change separately from the Attic node deployment.
Confirm the public endpoint remains stable under repeated fetches.

Implementation Outline

Update the reverse proxy upstream set.
Enable health checks or passive failover if available.
Roll out only the proxy change.
Re-test public cache info and repeated narinfo fetches.

Verification

curl -I https://attic.nimmog.uk/nixos/nix-cache-info
for i in $(seq 1 10); do
  curl -fsS https://attic.nimmog.uk/nixos/<known-good-hash>.narinfo >/dev/null
done

Rollback

If dual-backend serving is unstable:

Remove lyra from the proxy upstream set.
Leave vega as the sole public backend.
Continue testing lyra only through direct/local access until the fault is understood.

## Goal Switch the public Attic endpoint from a single backend to dual backends only after both nodes are healthy. ## Scope This issue covers the work from `projects/attic-postgres-lyra-rollout.md` Phase 4. ## Acceptance Checklist - Add both `vega:8081` and `lyra:8081` as upstreams. - Keep health checks enabled if supported. - Prefer passive failure handling and retries on the alternate backend. - Roll out the proxy change separately from the Attic node deployment. - Confirm the public endpoint remains stable under repeated fetches. ## Implementation Outline 1. Update the reverse proxy upstream set. 2. Enable health checks or passive failover if available. 3. Roll out only the proxy change. 4. Re-test public cache info and repeated narinfo fetches. ## Verification ```bash curl -I https://attic.nimmog.uk/nixos/nix-cache-info for i in $(seq 1 10); do curl -fsS https://attic.nimmog.uk/nixos/<known-good-hash>.narinfo >/dev/null done ``` ## Rollback If dual-backend serving is unstable: 1. Remove `lyra` from the proxy upstream set. 2. Leave `vega` as the sole public backend. 3. Continue testing `lyra` only through direct/local access until the fault is understood.

nimmo added this to the Attic PostgreSQL Migration and Lyra Rollout milestone

2026-07-01 16:14:42 +01:00

nimmo added the

host:fleet

project:attic-postgres-lyra-rollout

labels

2026-07-01 16:14:42 +01:00

nimmo referenced this issue

2026-07-06 16:25:06 +01:00

Deploy Attic on lyra against shared PostgreSQL and NAS storage #7

nimmo commented

2026-07-06 16:32:43 +01:00

Author

Owner

Diagnosed the current dual-backend proxy failure.

Root cause:

lyra Attic itself is healthy and serves nix-cache-info locally.
However, Attic's allowed-hosts list in the shared module only whitelisted attic.nimmog.uk, loopback, and vega's backend IP host (192.168.8.108:8081).
When Pangolin/newt health-checks or proxies to lyra by backend IP/port (192.168.8.146:8081), Attic rejects the request with 400 Bad Host.

Evidence:

local curl -H 'Host: 192.168.8.146:8081' http://127.0.0.1:8081/nixos/nix-cache-info on lyra returned 400 Bad Host.
curl -H 'Host: attic.nimmog.uk' ... returned 200 OK.
curl http://192.168.8.146:8081/nixos/nix-cache-info from vega also returned 400 Bad Host.

Repo fix in progress:

extend the shared Attic allowed-hosts list to include 192.168.8.146:8081 so Pangolin can treat lyra as healthy over the LAN backend path.

Diagnosed the current dual-backend proxy failure. Root cause: - `lyra` Attic itself is healthy and serves `nix-cache-info` locally. - However, Attic's `allowed-hosts` list in the shared module only whitelisted `attic.nimmog.uk`, loopback, and `vega`'s backend IP host (`192.168.8.108:8081`). - When Pangolin/newt health-checks or proxies to `lyra` by backend IP/port (`192.168.8.146:8081`), Attic rejects the request with `400 Bad Host`. Evidence: - local `curl -H 'Host: 192.168.8.146:8081' http://127.0.0.1:8081/nixos/nix-cache-info` on `lyra` returned `400 Bad Host`. - `curl -H 'Host: attic.nimmog.uk' ...` returned `200 OK`. - `curl http://192.168.8.146:8081/nixos/nix-cache-info` from `vega` also returned `400 Bad Host`. Repo fix in progress: - extend the shared Attic `allowed-hosts` list to include `192.168.8.146:8081` so Pangolin can treat `lyra` as healthy over the LAN backend path.

nimmo commented

2026-07-06 16:38:39 +01:00

Author

Owner

After redeploying the shared Attic allowed-hosts fix, both vega and lyra now report healthy in Pangolin/newt for the public Attic route.

This confirms the earlier lyra health-check failure was caused by Attic rejecting the backend-IP Host header, and that the fix to whitelist 192.168.8.146:8081 resolved it.

Remaining verification for this issue is now just endpoint stability under repeated public fetches, not backend reachability.

After redeploying the shared Attic `allowed-hosts` fix, both `vega` and `lyra` now report healthy in Pangolin/newt for the public Attic route. This confirms the earlier `lyra` health-check failure was caused by Attic rejecting the backend-IP Host header, and that the fix to whitelist `192.168.8.146:8081` resolved it. Remaining verification for this issue is now just endpoint stability under repeated public fetches, not backend reachability.

nimmo commented

2026-07-06 16:39:43 +01:00

Author

Owner

Final public-endpoint verification passed.

Observed:

curl -I https://attic.nimmog.uk/nixos/nix-cache-info returned 200 OK.
Repeated public .narinfo fetches for a known path completed successfully without backend instability.

Issue outcome:

The public Attic endpoint is now serving through both vega and lyra.
The only backend-specific fault encountered during rollout was Attic rejecting lyra's backend-IP Host header; once the shared allowed-hosts list was updated, Pangolin/newt health checks and public serving both stabilized.

This completes the dual-backend serving phase.

Final public-endpoint verification passed. Observed: - `curl -I https://attic.nimmog.uk/nixos/nix-cache-info` returned `200 OK`. - Repeated public `.narinfo` fetches for a known path completed successfully without backend instability. Issue outcome: - The public Attic endpoint is now serving through both `vega` and `lyra`. - The only backend-specific fault encountered during rollout was Attic rejecting `lyra`'s backend-IP Host header; once the shared `allowed-hosts` list was updated, Pangolin/newt health checks and public serving both stabilized. This completes the dual-backend serving phase.

nimmo closed this issue

2026-07-06 16:39:51 +01:00

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

nimmo/nixos-config#8

No description provided.

Rows
Columns