Validate Attic push path after PostgreSQL and lyra rollout

nimmo commented

2026-07-01 16:14:45 +01:00

Owner

Goal

Validate that the Attic write path still works after the metadata migration and dual-node rollout.

Scope

This issue covers the work from projects/attic-postgres-lyra-rollout.md Phase 5.

Acceptance Checklist

Confirm the Forgejo attic-cache.yml workflow still succeeds.
Confirm attic push still succeeds from nova.
Confirm newly pushed paths become fetchable from a third host.
Validate not only read availability but end-to-end publishability.

Validation Targets

Forgejo attic-cache.yml run succeeds.
attic push succeeds from nova.
Newly pushed paths are fetchable from another host.

Verification

ssh -F /dev/null nova 'ps -eo pid,cmd | grep "attic push" | grep -v grep || true'
curl https://attic.nimmog.uk/nixos/<new-hash>.narinfo
ssh -F /dev/null vega 'nix copy --from https://attic.nimmog.uk/nixos /nix/store/<new-path>'

Definition of Done

vega and lyra both serve Attic successfully.
Both share PostgreSQL metadata and NAS storage.
The public endpoint is stable.
Known cache reads still work.
Push/update workflows still complete successfully.

## Goal Validate that the Attic write path still works after the metadata migration and dual-node rollout. ## Scope This issue covers the work from `projects/attic-postgres-lyra-rollout.md` Phase 5. ## Acceptance Checklist - Confirm the Forgejo `attic-cache.yml` workflow still succeeds. - Confirm `attic push` still succeeds from `nova`. - Confirm newly pushed paths become fetchable from a third host. - Validate not only read availability but end-to-end publishability. ## Validation Targets - Forgejo `attic-cache.yml` run succeeds. - `attic push` succeeds from `nova`. - Newly pushed paths are fetchable from another host. ## Verification ```bash ssh -F /dev/null nova 'ps -eo pid,cmd | grep "attic push" | grep -v grep || true' curl https://attic.nimmog.uk/nixos/<new-hash>.narinfo ssh -F /dev/null vega 'nix copy --from https://attic.nimmog.uk/nixos /nix/store/<new-path>' ``` ## Definition of Done - `vega` and `lyra` both serve Attic successfully. - Both share PostgreSQL metadata and NAS storage. - The public endpoint is stable. - Known cache reads still work. - Push/update workflows still complete successfully.

nimmo added this to the Attic PostgreSQL Migration and Lyra Rollout milestone

2026-07-01 16:14:45 +01:00

nimmo added the

host:fleet

project:attic-postgres-lyra-rollout

labels

2026-07-01 16:14:45 +01:00

nimmo commented

2026-07-06 16:46:24 +01:00

Author

Owner

Starting issue #9.

Current write-path status before changes:

the read side is now healthy on both vega and lyra, and the public endpoint is dual-backed and stable,
but the Forgejo attic-cache.yml workflow still points directly at http://192.168.8.108:8081/ and http://192.168.8.108:8081/nixos, which bypasses the new public dual-backend route entirely.

That means the next step is to decide whether to:

keep the write path pinned directly to one backend and merely validate that it still works after the rollout, or
move the workflow/push path to attic.nimmog.uk so issue #9 validates the real public publish path as well.

I'm checking the existing repo wiring now so we can make that choice deliberately rather than inherit stale pre-rollout settings.

Starting issue #9. Current write-path status before changes: - the read side is now healthy on both `vega` and `lyra`, and the public endpoint is dual-backed and stable, - but the Forgejo `attic-cache.yml` workflow still points directly at `http://192.168.8.108:8081/` and `http://192.168.8.108:8081/nixos`, which bypasses the new public dual-backend route entirely. That means the next step is to decide whether to: - keep the write path pinned directly to one backend and merely validate that it still works after the rollout, or - move the workflow/push path to `attic.nimmog.uk` so issue #9 validates the real public publish path as well. I'm checking the existing repo wiring now so we can make that choice deliberately rather than inherit stale pre-rollout settings.

nimmo commented

2026-07-06 16:47:27 +01:00

Author

Owner

Repo changes in progress for the write-path validation phase.

Implemented locally:

modules/common/nix-settings.nix now trusts the Attic public signing key returned by the live public cache-config (nixos:tNn8ouVBQkqLS2qTq86V50TCL1AGn2NgwEhpxReo6jU=), closing the gap where the shared substituter was configured but its signing key was not trusted.
.forgejo/workflows/attic-cache.yml now points ATTIC_SERVER_URL, ATTIC_CACHE_URL, and the workflow substituters entry at https://attic.nimmog.uk/... instead of the old direct vega backend IP.
The workflow trusted-public-keys entry is updated to trust the Attic public key as well.

Why this matters for #9:

the workflow previously bypassed the new dual-backend public route entirely,
and even where the Attic substituter was declared, the signing key trust story was incomplete.

With this change, issue #9 validates the real public publish/read path rather than an obsolete single-backend internal shortcut.

Repo changes in progress for the write-path validation phase. Implemented locally: - `modules/common/nix-settings.nix` now trusts the Attic public signing key returned by the live public cache-config (`nixos:tNn8ouVBQkqLS2qTq86V50TCL1AGn2NgwEhpxReo6jU=`), closing the gap where the shared substituter was configured but its signing key was not trusted. - `.forgejo/workflows/attic-cache.yml` now points `ATTIC_SERVER_URL`, `ATTIC_CACHE_URL`, and the workflow `substituters` entry at `https://attic.nimmog.uk/...` instead of the old direct `vega` backend IP. - The workflow `trusted-public-keys` entry is updated to trust the Attic public key as well. Why this matters for #9: - the workflow previously bypassed the new dual-backend public route entirely, - and even where the Attic substituter was declared, the signing key trust story was incomplete. With this change, issue #9 validates the real public publish/read path rather than an obsolete single-backend internal shortcut.

nimmo commented

2026-07-06 16:55:22 +01:00

Author

Owner

The first real end-to-end validation of the updated write path appears to have succeeded.

Observed outcome:

Forgejo action run 269 completed successfully after the workflow was switched from the old direct vega backend URL to the public https://attic.nimmog.uk/ endpoint and matching cache URL.
This means the cache build workflow can still authenticate, build, and push through the new public Attic route after the PostgreSQL cutover and dual-backend rollout.

Remaining explicit validation from the issue body is now narrow:

confirm that at least one newly pushed path from this successful run is fetchable from another host.

If that fetch check passes, issue #9 can be closed.

The first real end-to-end validation of the updated write path appears to have succeeded. Observed outcome: - Forgejo action run `269` completed successfully after the workflow was switched from the old direct `vega` backend URL to the public `https://attic.nimmog.uk/` endpoint and matching cache URL. - This means the cache build workflow can still authenticate, build, and push through the new public Attic route after the PostgreSQL cutover and dual-backend rollout. Remaining explicit validation from the issue body is now narrow: - confirm that at least one newly pushed path from this successful run is fetchable from another host. If that fetch check passes, issue #9 can be closed.

nimmo commented

2026-07-06 17:04:09 +01:00

Author

Owner

Cross-host fetch validation found one remaining deployment-dependent gap.

Observed:

A newly pushed candidate path (/nix/store/pvv82rdgp6iikgrzsms2ak0763mibh5r-paseo-0.1.104-beta.4) is present in the public Attic cache and serves a valid .narinfo from https://attic.nimmog.uk/nixos.
nix copy --from https://attic.nimmog.uk/nixos ... from lyra initially failed with lacks a signature by a trusted key.

Interpretation:

This is not a cache publish failure. It means the host-side Nix trust configuration on lyra has not yet picked up the newly committed Attic public signing key.
The repo fix for that trust key exists (modules/common/nix-settings.nix), but successful end-to-end host fetch validation depends on deploying that config to the consuming host(s).

Next step:

deploy the new Nix trusted-public-keys configuration to the relevant hosts,
rerun the fetch check,
then close #9 once the host-side trust state matches the repo.

Cross-host fetch validation found one remaining deployment-dependent gap. Observed: - A newly pushed candidate path (`/nix/store/pvv82rdgp6iikgrzsms2ak0763mibh5r-paseo-0.1.104-beta.4`) is present in the public Attic cache and serves a valid `.narinfo` from `https://attic.nimmog.uk/nixos`. - `nix copy --from https://attic.nimmog.uk/nixos ...` from `lyra` initially failed with `lacks a signature by a trusted key`. Interpretation: - This is not a cache publish failure. It means the host-side Nix trust configuration on `lyra` has not yet picked up the newly committed Attic public signing key. - The repo fix for that trust key exists (`modules/common/nix-settings.nix`), but successful end-to-end host fetch validation depends on deploying that config to the consuming host(s). Next step: - deploy the new Nix trusted-public-keys configuration to the relevant hosts, - rerun the fetch check, - then close #9 once the host-side trust state matches the repo.

nimmo commented

2026-07-06 17:19:58 +01:00

Author

Owner

Cross-host Attic read validation is narrower now but still not complete.

Latest test used an Electra-only package (kcalc, /nix/store/9p212z0yhvjizyz2h93bimkj6pb7hmsj-kcalc-26.04.3) and attempted to fetch it from vega via the public Attic endpoint.

What now works:

Electra has kcalc; Vega did not.
https://attic.nimmog.uk/nixos/9p212z0yhvjizyz2h93bimkj6pb7hmsj.narinfo returns valid metadata with a Sig: line.
A dependency that failed during import (mtdev, /nix/store/0jbb8w6zkibl8c4qjybn0ajm9668ginz-mtdev-1.1.7) also returns valid .narinfo with a Sig: line.
On Vega, nix path-info --store https://attic.nimmog.uk/nixos /nix/store/0jbb8w6zkibl8c4qjybn0ajm9668ginz-mtdev-1.1.7 succeeds.
Vega's live /etc/nix/nix.conf includes the public Attic substituter and the Attic public key nixos:tNn8ouVBQkqLS2qTq86V50TCL1AGn2NgwEhpxReo6jU=.

What still fails:

On Vega, nix copy --from https://attic.nimmog.uk/nixos /nix/store/0jbb8w6zkibl8c4qjybn0ajm9668ginz-mtdev-1.1.7 fails deterministically with:
error: cannot add path ... because it lacks a signature by a trusted key
The same happens when fetching the full kcalc closure; the first failing dependency is mtdev.

Important constraint discovered:

nimmo on Vega is not a trusted Nix user, so client-side --option trusted-public-keys=... overrides are ignored. That means this remaining failure is at the running Nix daemon / signature-verification boundary, not just in user-shell config.

So issue #9 is now reduced to: Attic publish and metadata lookup through the public route work, but actual Nix import on Vega still rejects at least some Attic-served paths as untrusted despite the expected key being present in /etc/nix/nix.conf. Further debugging needs either trusted-user/root access on Vega or a fresh consumer host where the daemon can be verified end-to-end.

Cross-host Attic read validation is narrower now but still not complete. Latest test used an Electra-only package (`kcalc`, `/nix/store/9p212z0yhvjizyz2h93bimkj6pb7hmsj-kcalc-26.04.3`) and attempted to fetch it from **vega** via the **public** Attic endpoint. What now works: - Electra has `kcalc`; Vega did not. - `https://attic.nimmog.uk/nixos/9p212z0yhvjizyz2h93bimkj6pb7hmsj.narinfo` returns valid metadata with a `Sig:` line. - A dependency that failed during import (`mtdev`, `/nix/store/0jbb8w6zkibl8c4qjybn0ajm9668ginz-mtdev-1.1.7`) also returns valid `.narinfo` with a `Sig:` line. - On Vega, `nix path-info --store https://attic.nimmog.uk/nixos /nix/store/0jbb8w6zkibl8c4qjybn0ajm9668ginz-mtdev-1.1.7` succeeds. - Vega's live `/etc/nix/nix.conf` includes the public Attic substituter and the Attic public key `nixos:tNn8ouVBQkqLS2qTq86V50TCL1AGn2NgwEhpxReo6jU=`. What still fails: - On Vega, `nix copy --from https://attic.nimmog.uk/nixos /nix/store/0jbb8w6zkibl8c4qjybn0ajm9668ginz-mtdev-1.1.7` fails deterministically with: `error: cannot add path ... because it lacks a signature by a trusted key` - The same happens when fetching the full `kcalc` closure; the first failing dependency is `mtdev`. Important constraint discovered: - `nimmo` on Vega is **not** a trusted Nix user, so client-side `--option trusted-public-keys=...` overrides are ignored. That means this remaining failure is at the running Nix daemon / signature-verification boundary, not just in user-shell config. So issue #9 is now reduced to: Attic publish and metadata lookup through the public route work, but actual Nix import on Vega still rejects at least some Attic-served paths as untrusted despite the expected key being present in `/etc/nix/nix.conf`. Further debugging needs either trusted-user/root access on Vega or a fresh consumer host where the daemon can be verified end-to-end.

nimmo commented

2026-07-06 18:36:11 +01:00

Author

Owner

Resolved.

Final validation after deploying commit ece1674 (which removed the stale nixos:xvnY... Attic trust key and kept only the live advertised nixos:tNn8... key):

On vega, nix copy --from https://attic.nimmog.uk/nixos /nix/store/4da5p6rwzj366cfsq0xslgqi1q7j5nbr-cuda12.9-libcublas-12.9.1.4-lib completed without error and the path is now present locally.
On electra, three genuinely absent server-side store paths were fetched successfully from the public cache:
- /nix/store/5sr8wrylkxi6rkprlhib6yvfgxldl1pv-atticd-atticadm
- /nix/store/pvv82rdgp6iikgrzsms2ak0763mibh5r-paseo-0.1.104-beta.4
- /nix/store/lf6h8d6jy5hck6f7dpmilny48158v1x5-unit-paseo.service

That confirms the remaining Attic read failure was caused by clients trusting two different public keys under the same nixos: key name. Public Attic reads work again after removing the stale trust entry and redeploying consumer config.

Any older cache entries that still prove problematic can be repopulated under the surviving key during normal rebuild/push flow, but the issue-level acceptance criteria for validating the public read path are now satisfied.

Resolved. Final validation after deploying commit `ece1674` (which removed the stale `nixos:xvnY...` Attic trust key and kept only the live advertised `nixos:tNn8...` key): - On vega, `nix copy --from https://attic.nimmog.uk/nixos /nix/store/4da5p6rwzj366cfsq0xslgqi1q7j5nbr-cuda12.9-libcublas-12.9.1.4-lib` completed without error and the path is now present locally. - On electra, three genuinely absent server-side store paths were fetched successfully from the public cache: - `/nix/store/5sr8wrylkxi6rkprlhib6yvfgxldl1pv-atticd-atticadm` - `/nix/store/pvv82rdgp6iikgrzsms2ak0763mibh5r-paseo-0.1.104-beta.4` - `/nix/store/lf6h8d6jy5hck6f7dpmilny48158v1x5-unit-paseo.service` That confirms the remaining Attic read failure was caused by clients trusting two different public keys under the same `nixos:` key name. Public Attic reads work again after removing the stale trust entry and redeploying consumer config. Any older cache entries that still prove problematic can be repopulated under the surviving key during normal rebuild/push flow, but the issue-level acceptance criteria for validating the public read path are now satisfied.

nimmo closed this issue

2026-07-06 18:36:15 +01:00

Rows
Columns

Validate Attic push path after PostgreSQL and lyra rollout #9

Goal

Scope

Acceptance Checklist

Validation Targets

Verification

Definition of Done