Audited the five CLAUDE.md files and all twenty-one README.md files against the
tree, machine-checking every in-repo path they name and verifying the testable
claims against the live hosts.
The one that matters is in the root file: dev was documented as reachable on
the mesh at 10.10.0.2:8137. It is not, and never was from anywhere but vps1 —
infra/docker-compose.yml binds the port to 127.0.0.1, and the address answers
from neither vps2 nor vps1 itself. Anyone following it gets a connection
refused with nothing to explain it.
The rest are stale paths, several from the reunification:
* assetlinks.json moved under frontend/static/ in the subtree merge; the TWA
README kept the pre-merge path in both places it names it. Following it
would put the file where nothing serves it and Android app-link
verification would fail silently.
* push.py and notify.py now live in backend/notifications/.
* INFRA.md and deploy/stack/README have never existed in this repo, in any
branch.
* the Caddyfile is at deploy/stack/lb/Caddyfile.
* three bare relative paths that resolve for a reader but not from the
directory the file sits in: units.js is the frontend's, deploy.sh is
infra's, entrypoint.sh is the backend's.
Also records why mesh clients must pin the ROOT_URL host and not only the image
host: the registry's bearer-token realm follows ROOT_URL, so pinning
git.thermograph.org alone still sends the token request out the public route,
where the /v2/* matcher returns 403 and docker falls back to anonymous. That
surfaces as `unauthorized: reqPackageAccess`, indistinguishable from a bad
credential.
Verified true and left alone: the four-domain layout, both .claude runbooks,
the absence of any domain-level .forgejo directory, the pinned compose project
name, the deploy contract, prod's eight stack services, beta's five prefixed
ones with no db of its own, dev's five, and every documented make target.
Repos moved to the Jinemi org; the container packages did not follow, since
Forgejo does not transfer packages with a repo. The deploy path still resolved
`emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml
derives its push path from ${github.repository}, now jinemi/thermograph. The
next backend or frontend build would have published somewhere no deploy looks.
Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit
no uppercase, which is why build-push.yml already pipes through tr), the clone
URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account
that actually owns the tokens, rather than the redirect.
The live tags and both ci-runner tags were copied into the Jinemi namespace
first, so the switch has something to pull. thermograph-infra,
thermograph-observability and the retired */app packages stay under admin_emi;
they did not move.
After a cell's archive advances, the first content request recomputes a 45yr payload cold. Add warm_cities.warm_content — idempotent, cache-only, bounded — that pre-populates the content derived-store under the same (kind,key,content_token) the routes read, wired into the leader-gated notifier loop (50 cities/30-min tick) so the ~1000-city set stays warm within hours of each daily advance.
The three content handlers loaded the full ~45yr archive before checking the cache, so even hits paid the load. Restructure to resolve cell + compute content_token cheaply first, check the store, and load history only inside the miss-only build() closure. Cache hits and 304s now return without touching the archive. Keys/kinds/ETag unchanged.
Content pages keyed cache validity on the full-history hist_end, invalidating ~1-2x/day and paying a 45yr load even to compute the token. Add content_token(cell_id) = PAYLOAD_VER:CONTENT_VER:max_date, backed by an indexed MAX(date) (climate_store.history_max_date / climate.history_max_date), so it survives intra-day top-ups and needs no full-history load.
main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:
- compose + Swarm stack: lake AND daemon are sibling services; backend env
carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
daemon), duckdb stays (the lake role's engine).
Merged tree: backend suite green, both compose configs validate.
The gateway bot and APScheduler were long-lived stateful I/O loops running
inside the async web app under a leader election. They move into a single Go
binary that owns ONLY that I/O -- websocket, RESUME, heartbeat, backoff, timers.
It owns no grading logic. Anything needing data calls back over a new
internal-only surface (/internal/discord/grade, /internal/jobs/*). Grading
depends on polars and the parquet cache; reimplementing it in Go would let the
bot's grades drift from the API's. The grade route returns gateway-ready JSON
and Go relays the bytes verbatim.
The binary ships in the backend image and runs as a second compose service off
the same tag, so the two ends of the /internal/* contract can never skew.
deploy.sh rolls daemon alongside backend -- without that the service would never
be created, since a single-service deploy uses --no-deps. It also probes the
image first and skips the daemon when rolling a tag that predates the binary:
infra tracks main while image tags are env-staged, so a host can legitimately be
asked to roll an older backend image, and creating the service anyway would
leave a container crash-looping on a missing binary.
replicas: 1 with order: stop-first replaces the leader election -- Discord
permits one gateway connection per bot token.
THERMOGRAPH_INTERNAL_TOKEN is optional: both ends derive it from
THERMOGRAPH_AUTH_SECRET via HMAC under a domain-separation label, so this needs
no new vault entry. The derivation is pinned to a shared cross-language test
vector asserted on both sides, so drift fails CI instead of 401ing every call.
Fail closed when neither secret is set.
Improvements over the Python: a close intended for RESUME uses 4000 rather than
1000 (Discord invalidates a session closed 1000, so the old default defeated its
own resume); MESSAGE_CREATE runs on a bounded worker pool; and a malformed HELLO
returns an error rather than a clean reconnect, which would otherwise reset
backoff and hot-loop against the gateway.
365 Python tests pass; Go build/vet/test -race clean; shellcheck 0 findings.
Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x,
default severity, fail on any finding, not path-filtered) and drives all 26
scripts to zero findings.
Two defects shellcheck cannot see:
render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt
failed -- the caller's set -e aborted the function before either cleanup ran.
Now removed on every exit path, with `|| return 1` on both sops calls so a
decrypt failure can never write a partial /etc/thermograph.env regardless of the
caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a
sourced function persists into the caller's shell and re-fires when the caller's
next `.`/source completes, where the function-local tmp is unset -- fatal and
silent under deploy.sh's set -u. The file now records that reasoning.
autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk,
so a failed left side was swallowed and the loop autoscaled on empty input.
Promoted to pipefail with a missed sample treated as a skip; verified busybox ash
in docker:27-cli supports it.
Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin,
silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck
source=` paths corrected for the monorepo layout.
Steady-state usage of the Open-Meteo API is removed; it remains only as a dormant fallback. Geocoding -> local GeoNames + Nominatim; wind gusts -> Meteostat; history -> NASA POWER (curated cells seeded with keyless ERA5); recent+forecast -> NASA range + MET Norway. Plus a drift-check tool comparing NASA vs Open-Meteo. All keyless, no new infra. Backend suite green in-image (gate).
One root workflow set replaces the four repos' copies (deleted -- root-only
is where Forgejo reads them, and dead copies are a trap): per-domain
build-push with explicit image paths (emi/thermograph/backend|frontend; the
old github.repository-derived path collides in a monorepo), path-filtered
per-domain beta/prod/dev deploys, a domain-input reusable build check, a
single always-reporting PR gate (path-filtered required checks deadlock
auto-merge), a new infra-sync pipeline (host checkout + secrets render on
infra/** pushes), and ports of secrets-guard / ops-cron /
observability-validate to monorepo paths.
