Merge pull request 'promote: dev → main (doc accuracy pass, runner unit fix)' (#158) from dev into main

CLAUDE.md

@@ -29,8 +29,10 @@ concurrently. Read both before dispatching parallel work.
 
 **`dev` is a first-class hosted environment, not just an integration branch.**
 It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
-own Postgres container, reachable only on the WireGuard mesh
+own Postgres container, reachable only from vps1 itself (`127.0.0.1:8137` —
-(`10.10.0.2:8137`): no public DNS record, no Caddy site, no TLS. It is deployed
+`infra/docker-compose.yml` binds the port to loopback, so the mesh does not
+reach it either; use an SSH tunnel): no public DNS record, no Caddy site, no
+TLS. It is deployed
 by CI like beta and prod. The desktop hosts no Thermograph environment at all;
 `make dev-up` there is a laptop convenience for running the stack locally, not
 a deployment target.

backend/CLAUDE.md

@@ -45,7 +45,7 @@ for systemd/CI callers. Entry target is `app:app`.
   that disagree with the labels the API returns.
 - **Fahrenheit country set** — `api/content_payloads.py`'s `F_COUNTRIES` is
   canonical; `frontend`'s `server/internal/format::FCountries` and
-  `static/units.js`'s `F_REGIONS` must stay identical to it. Both are asserted
+  `frontend/static/units.js`'s `F_REGIONS` must stay identical to it. Both are asserted
   by tests in `frontend/server/internal/format/format_test.go` — but note the
   backend cross-check **skips in CI** (the frontend image's build context is
   `frontend/`, so this file is unreachable from the builder stage, which is the

backend/README.md

@@ -69,7 +69,7 @@ cities_flavor.json   bundled reference data (generated by gen_cities.py /
   only builds and publishes its own container image
   (`git.thermograph.org/admin_emi/thermograph-backend/app`) and hands infra a tag to
   roll out (`SERVICE=backend` + `BACKEND_IMAGE_TAG` into infra's
-  `deploy/deploy.sh`).
+  `infra/deploy/deploy.sh`).
 - **`thermograph-docs`** holds the cross-repo architecture/runbook docs; this
   README only covers what's local to this service.
 

infra/deploy/db/README.md

@@ -16,7 +16,7 @@ The `db` service runs the stock **`timescale/timescaledb:latest-pg18`** image
   TTL, and the `recent_stamp` token embedded in derived-payload validity.
 
 The schema is created by Alembic (`backend/alembic/versions/0002_climate_hypertables.py`,
-run at app boot via `deploy/entrypoint.sh`). The app reads/writes it through
+run at app boot via `backend/deploy/entrypoint.sh`). The app reads/writes it through
 `backend/data/climate_store.py` (psycopg + polars). See
 `deploy/POSTGRES-MIGRATION.md` for the parquet→hypertable cutover.
 

infra/deploy/forgejo/README.md

@@ -3,7 +3,7 @@
 Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload
 this cluster carries (prod and beta's app stacks are separate `docker stack
 deploy`s that happen to run on the manager node, vps2 — see
-`deploy/stack/README` context in `DEPLOY.md`). Pinned to the **vps1** node
+`DEPLOY.md`). Pinned to the **vps1** node
 (`75.119.132.91`) via the `role=forge` label from
 `deploy/swarm/label-forge-node.sh`.
 
@@ -153,6 +153,28 @@ echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts
 numbering — adjust if you assigned it differently.) The git/web UI keeps
 working normally for everyone else since only `/v2/*` is restricted.
 
+**Pin the canonical name too, not just the image host.** Forgejo derives the
+registry's bearer-token realm from `ROOT_URL`, so a client that already
+resolves `git.thermograph.org` over the mesh is still sent to
+`https://<ROOT_URL host>/v2/token` to collect a token. When `ROOT_URL` became
+`dev.jinemi.com`, that second request took the public route, the `/v2/*`
+matcher above answered 403, and docker fell back to anonymous — every push then
+failed with `unauthorized: reqPackageAccess`, which reads exactly like a
+revoked token or a missing scope. It is neither. Pin both names:
+
+```
+echo "10.10.0.2 git.thermograph.org dev.jinemi.com" | sudo tee -a /etc/hosts
+```
+
+The realm moves whenever `ROOT_URL` does, so read it rather than assuming:
+
+```sh
+curl -sI https://git.thermograph.org/v2/ | grep -i www-authenticate
+```
+
+These `/etc/hosts` pins are host state. Nothing in this repo writes them, so a
+reprovisioned node has to be pinned again before it can pull.
+
 ## Register the Actions runner
 
 Once Forgejo answers at its domain:

infra/deploy/forgejo/register-lan-runner.sh

@@ -93,6 +93,13 @@ cat > "$HOME/.config/systemd/user/forgejo-runner.service" <<EOF
 [Unit]
 Description=Forgejo Actions runner (docker + thermograph-lan)
 After=network-online.target
+# These bound the Restart=always loop below, so a genuinely broken config (bad
+# token, docker unreachable) ends as a stopped unit rather than spinning
+# forever. They belong in [Unit]: systemd accepts them in [Service] without
+# complaint but IGNORES them, leaving the 10s default — \`systemctl --user show
+# forgejo-runner -p StartLimitIntervalUSec\` is how you tell which one you got.
+StartLimitIntervalSec=300
+StartLimitBurst=5
 
 [Service]
 WorkingDirectory=${RUNNER_DIR}
@@ -102,7 +109,16 @@ WorkingDirectory=${RUNNER_DIR}
 # SOPS-configured) fails with "not installed" even though it plainly is.
 Environment=PATH=%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
 ExecStart=${RUNNER_DIR}/forgejo-runner daemon -c config.yaml
-Restart=on-failure
+# \`always\`, NOT \`on-failure\` — the same distinction that took Forgejo down for
+# 27 hours on 2026-07-29 (see docker-stack.yml's restart_policy comment): an
+# always-on daemon that exits **0** is not "finished successfully", it is a
+# daemon that stopped and must come back. \`on-failure\` cannot tell those apart,
+# and forgejo-runner exits 0 on several paths — a lost instance connection it
+# gives up on, or a SIGTERM from a Docker restart it interprets as a clean
+# shutdown. The failure mode is silent: the unit sits \`inactive (dead)\`, the UI
+# shows the runner offline, and every protected-branch merge blocks on a check
+# that will never be produced.
+Restart=always
 RestartSec=5
 
 [Install]

infra/deploy/forgejo/runner-vps2/README.md

@@ -49,13 +49,24 @@ deploys.
 
 So this directory is the source of truth; vps2 gets a copy:
 
+The two files do **not** go to the same place, and getting it wrong fails in a
+confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
+with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
+sitting beside the compose file is invisible to the container. The daemon then
+either refuses to start or silently runs on defaults (no `--add-host`, so every
+registry push fails as if the credential were wrong).
+
 ```bash
-install -d -m 0755 /opt/forgejo-runner
+install -d -m 0755 /opt/forgejo-runner/data
 cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
-   /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
    /opt/forgejo-runner/
+cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
+   /opt/forgejo-runner/data/
 ```
 
+`data/` is also where `register` writes `.runner`, so the runner's whole state —
+config plus registration — lives in the one directory the compose file mounts.
+
 Re-copy after changing either file here, then `docker compose up -d`. There is
 no automation for that step and there should not be: a workflow that redeploys
 the runner runs *on* the runner.

infra/deploy/twa/README.md

@@ -14,7 +14,7 @@ submission ($25 one-time) is deferred.
 
 ## What's in the repo
 - `twa-manifest.json` — Bubblewrap config (reference values; see comment inside).
-- `../../frontend/.well-known/assetlinks.json` — Digital Asset Links, served at
+- `../../../frontend/static/.well-known/assetlinks.json` — Digital Asset Links, served at
   `https://thermograph.org/.well-known/assetlinks.json`. Its fingerprint is a
   **placeholder** until you generate the signing key (step 4).
 
@@ -45,7 +45,7 @@ bubblewrap build
 bubblewrap fingerprint list          # or: keytool -list -v -keystore android-keystore.jks
 ```
 
-Copy the `SHA256` value into `frontend/.well-known/assetlinks.json`, replacing
+Copy the `SHA256` value into `frontend/static/.well-known/assetlinks.json`, replacing
 `REPLACE_WITH_TWA_SIGNING_KEY_SHA256_FINGERPRINT`, then **deploy the site** so the
 new file is live. Confirm:
 
@@ -80,6 +80,7 @@ install for every user. Store it out of the repo, backed up.
   `.aab`; the store review for a well-formed TWA is light. Not part of this pass.
 
 ## Why no backend changes
-Android delegates the TWA's web notifications to the OS, so `backend/push.py`
+Android delegates the TWA's web notifications to the OS, so
-(VAPID) and `backend/notify.py` deliver push to the app unchanged. Nothing in
+`backend/notifications/push.py` (VAPID) and `backend/notifications/notify.py`
+deliver push to the app unchanged. Nothing in
 `backend/` is touched by this track.

infra/terraform/README.md

@@ -183,7 +183,8 @@ Terraform brings the stack up — don't let Terraform recreate containers mid-mi
   `git.thermograph.org` reaches Forgejo's loopback port on vps1.
   `COOKIE_SECURE` is auto-set to `0` when there's no domain (a Secure
   cookie is never sent over plain HTTP) and `1` behind Caddy TLS.
-- The rendered Caddyfile only reverse-proxies the app. The repo's `deploy/Caddyfile`
+- The rendered Caddyfile only reverse-proxies the app. The repo's
+  `deploy/stack/lb/Caddyfile`
   additionally serves the `emigriffith.dev` portfolio and legacy redirects; those are
   host-specific and not templated here.
 - Provisioner-based by design: the hosts already exist, so this is not a

observability/README.md

@@ -147,7 +147,7 @@ LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
   docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
 ```
 
-The mesh must already be up (see the main repo's `deploy/swarm/` / `INFRA.md`);
+The mesh must already be up (see `infra/deploy/swarm/`);
 Swarm is **not** required for the agent itself — it talks plain HTTP over wg0.
 
 <!-- TODO(cutover): desktop no longer runs a Thermograph app stack and ships no