thermograph/docs/onboarding/11-traps.md

11. Traps and stale docs

Two kinds of thing here: live traps (correct code that will surprise you) and stale docs (files in this repo that currently describe a system that no longer exists). The second list matters as much as the first — this repo has a lot of documentation, most of it excellent, and some of it describing the world before a rewrite.

Everything below was checked against the tree at the time of writing.

---

Stale docs in this repo

frontend/CLAUDE.md and frontend/README.md describe a service that is not deployed

The live frontend is Go (frontend/server/, module thermograph/frontend). Both documents describe the Python FastAPI/Jinja implementation (frontend/app.py, content.py, api_client.py, format.py, templates/*.html.j2), which was superseded by the Go port and is kept in-tree as the reference the port was made from.

What is still correct in them: every contract (API-version pinning, pctOrd, the /cell + ETag relationship, the Fahrenheit set, boot-must-survive- an-unreachable-backend), every environment variable name and default, and all of DESIGN.md.

What is wrong: the language, the file paths, and — notably — frontend/CLAUDE.md's claim that make test-unit is "the tier CI runs." It isn't any more. CI builds the frontend image, and frontend/Dockerfile's builder stage runs gofmt -l + go vet + go test ./... before compiling, so a failing Go test is what fails CI. The Python tiers are local-only now.

Also note api_client.py carries a third copy of the API_VERSION pin. It isn't deployed, but keep it in step or you'll debug a phantom locally.

infra/DEPLOY.md describes the pre-Swarm, pre-monorepo deploy

Three separate layers of drift:

• It says prod is deployed with terraform apply. Prod is deployed by a push to release firing deploy.yml, exactly like beta's main path — see 07.
• It says prod runs a compose stack. Prod runs Swarm (infra/deploy/stack/).
• Its file table points at /opt/thermograph/deploy/deploy.sh and /opt/thermograph/docker-compose.yml. Both moved under infra/ at the monorepo cutover.

The document's own header flags the legacy provision.sh / systemd / venv walkthrough as superseded — but the drift goes further than that note admits. The key-management sections (Part 1) and the /etc/hosts registry note are still accurate and useful.

infra/DEPLOY-DEV.md and infra/ACCESS.md reference deleted workflows

Both talk about .forgejo/workflows/deploy-dev.yml. It does not exist. The two LAN-dev deploy workflows were deleted rather than ported at the CI consolidation — they were documented as inert (they targeted a monorepo layout at ~/thermograph-dev on a box still holding a split-era checkout). LAN dev is a local make dev-up concern, not a CI environment.

ACCESS.md §3c also lists {build,pr-build,deploy-dev,deploy}.yml as the workflow set; there are now nine files and the deploy/build ones were collapsed.

Everything else in ACCESS.md — the host table, the agent-access model, the key rotation procedure, the Swarm/Forgejo tracks — is current and worth reading.

backend/README.md is written for the split-repo era

It describes thermograph-backend as its own repo with sibling repos, uses the retired image path git.thermograph.org/emi/thermograph-backend/app, and says "there is no Makefile in this repo yet" — there is (make test, make smoke, make clean-venv).

The "How it works" section (grid → data → percentiles → caching) is one of the best explanations of the domain model anywhere in the repo. Read it; just mentally translate the topology.

CUTOVER-NOTES.md carries a superseded block, correctly labelled

Its per-domain workflow names (backend-build-push.yml, backend-deploy.yml, …) no longer exist — the six deploy workflows collapsed into deploy.yml and the two build-push workflows into build-push.yml. The file says so in an explicit "Superseded 2026-07-25" note, and everything it describes about behaviour is unchanged.

It also references backend/MONOREPO-CUTOVER-PLAN.md, which is not in the tree.

CUTOVER-NOTES.md remains the source of truth for what is and isn't live.

The domain CLAUDE.md files still use repo-era wording

They say "this repo" and link to sibling repo names. Mentally map thermograph-backend → backend/, and fix the wording as you touch files near it — that's the standing instruction in the root CLAUDE.md.

observability/README.md: merging is not deploying

/opt/observability on beta and prod is a checkout of the archived emi/thermograph-observability repo. It can never git pull again. A previous observability PR merged and had zero live effect for exactly this reason. The README documents the hand-ship procedure; until the checkouts are re-pointed at the monorepo, that's the only path. Flagged again in 09.

---

Live traps

The docker-compose.test.yml files still pull retired image paths

Both backend/docker-compose.test.yml and frontend/docker-compose.test.yml default BACKEND_IMAGE_PATH to emi/thermograph-backend/app — the split-era path. The current image is emi/thermograph/backend.

• backend/make smoke is fine: it builds :local and never pulls.
• frontend/make backend-up does pull, and pins THERMOGRAPH_BACKEND_TEST_TAG to v0.0.2-split-ci. The old packages were deliberately left in the registry for rollback and GC'd only after burn-in, so this may work, may serve a very old backend, or may 404 depending on when you read this. If make backend-up fails or behaves oddly, set the path and tag explicitly:

  BACKEND_IMAGE_PATH=emi/thermograph/backend \
  THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> make backend-up
  

static/units.js's F_REGIONS is guarded by nothing

Source comments in three files claim "a test asserts all three stay identical." The only automated assertion is format_test.go::TestFCountriesMatchesBackend, which re-parses the backend's Python and compares the Go copy. Nothing reads units.js.

All four copies currently are identical (14 codes, checked). But the browser copy is held in step by convention alone. Extending the Go test to parse units.js too is a good first contribution.

The compose project name is load-bearing

infra/docker-compose.yml pins name: thermograph. Without it, running compose from infra/ derives project infra — a silently new stack with empty volumes beside the running one. deploy-dev.sh exports COMPOSE_PROJECT_NAME=thermograph-dev to keep LAN dev separate on purpose. Keep both halves.

deploy.sh hard-resets the host checkout

git reset --hard origin/$BRANCH runs before anything else. Uncommitted edits in /opt/thermograph evaporate. infra-sync.yml does the same reset without deploying. The one deliberate exception is the untracked deploy/.image-tags.env — losing that makes the next single-service deploy roll the sibling onto local.

The deploy scripts eat your rollback image

Both deploy.sh and deploy-stack.sh end by deleting images outside the running pair. On beta that succeeds, so beta typically holds no local rollback target at all; on prod it usually fails only because Swarm's stopped task containers still hold references. docker system prune -a is forbidden on a live box for the same reason — but the deploy script is the thing that actually eats the image, not prune. Verify a tag exists before promising a rollback.

daemon and lake are never deploy targets on their own

They ride with backend. All three run the same image at the same tag — a second binary and a second role inside one image — and the /internal/* contract between daemon and web has no version negotiation. There's also a mechanical reason: a single-service deploy runs up -d --no-deps <targets>, so a service in compose but absent from TARGETS would simply never be created.

docker compose config --images <service> does not filter

Confirmed live on Compose v5.3.1: it prints every service's image, one per line, in file order. A | head -1 therefore grabbed db's image, and deploy.sh's daemon capability probe found no daemon binary in a Postgres image and dropped the daemon from every backend deploy. The script now builds the image reference from the same variables compose interpolates.

Prod's Loki streams carry no service label

Prod runs Swarm; its job="docker" streams have no service label. A raw {host="prod", service="backend"} query matches nothing and reads as "no logs" when it means "wrong selector". Use logs_query's service argument. Prod container names also change on every redeploy (the task suffix) — resolve via docker ps --filter name= at call time, never hardcode.

A Grafana alert rule can provision cleanly and never fire

A rule whose condition names a non-existent refId is valid YAML, provisions without complaint, and pages nobody. Likewise a contact point holding the literal string $DISCORD_ALERT_WEBHOOK_URL looks completely healthy from the API — and the API redacts the URL, so you cannot confirm interpolation by reading it back. The only proof is a message arriving in #ops-alerts.

Terraform has no state anywhere

A casual terraform apply would attempt full re-provisioning of live hosts. It's executable documentation until state is bootstrapped.

The registry is mesh-only

git.thermograph.org resolves publicly to beta's IP, but beta's Caddy rejects /v2/* from outside the WireGuard mesh. Any host that pulls needs 10.10.0.2 git.thermograph.org in /etc/hosts. Beta itself doesn't — it is the box.

Forgejo runners only have the labels they registered with

GitHub implicitly tags every self-hosted runner self-hosted; Forgejo does not. runs-on: [self-hosted, thermograph-lan] is permanently unschedulable there — a real bug, found and fixed once. If a job silently stops appearing in the Actions history, check this first.

Forgejo's needs: doesn't cross workflow files

build-push.yml and deploy.yml fire from the same push with no ordering guarantee, and a deploy racing ahead of its build and failing "not found" is confirmed-live. That's why deploy.sh's pull has a bounded ~5-minute retry.

/api/v2/cell?prefetch=1 must never spend quota

The hard guarantee: a cold cell answers 204 with no body. Anything named "prefetch" or "warm" calls the cache-only loaders. Breaking this gets the whole service rate-limited.

Background timers must go through leader election

singleton.claim_leader() — flock per host, Postgres advisory lock per cluster. A timer-driven sweep running in N uvicorn workers × M Swarm hosts multiplies Open-Meteo quota use N×M. That is the documented cause of overnight 429/503s after prod went to 3 workers. The heartbeat is the deliberate exception — a beat is one local file append, so duplicates are free.

THERMOGRAPH_DATA_DIR exists because of a real collision

The backend's Python package data/ and its runtime data dir <root>/data collide at the same path once the code lives at the image root. Mounting a volume there shadows and erases data/*.py, so import data.climate fails at boot. Point the env var outside the code tree in a container. And never reintroduce __file__-relative paths in a module — paths.py resolves everything from the repo root for this reason.

Alloy can silently stop tailing a container

Observed on prod, 13½ hours of thermograph_worker logs never reaching Loki while the container was healthy and writing. Not root-caused. Workaround: sudo docker restart alloy-alloy-1. ProdWorkerContainerSilent now catches the class.

Nominatim is ~1 req/s, on one thread

All reverse geocoding funnels through a single dedicated worker draining a queue. The earlier design slept inside the caller's thread — which in the server is one of Starlette's shared sync-threadpool threads.

---

If you fix one of these

Docs are cheap to correct and the root CLAUDE.md already asks you to fix repo-era wording as you touch files near it. If you're in frontend/ anyway, bringing frontend/CLAUDE.md in line with the Go service would be the single highest-value documentation change available — it's the file an agent reads before touching that domain.

Back to the index.