# 11. Traps and stale docs Two kinds of thing here: **live traps** (correct code that will surprise you) and **stale docs** (files in this repo that currently describe a system that no longer exists). The second list matters as much as the first — this repo has a lot of documentation, most of it excellent, and some of it describing the world before a rewrite. Everything below was checked against the tree at the time of writing. --- ## Stale docs in this repo ### `frontend/CLAUDE.md` and `frontend/README.md` describe a service that is not deployed **The live frontend is Go** (`frontend/server/`, module `thermograph/frontend`). Both documents describe the Python FastAPI/Jinja implementation (`frontend/app.py`, `content.py`, `api_client.py`, `format.py`, `templates/*.html.j2`), which was superseded by the Go port and is kept in-tree as the reference the port was made from. What is still **correct** in them: every contract (API-version pinning, `pctOrd`, the `/cell` + ETag relationship, the Fahrenheit set, boot-must-survive- an-unreachable-backend), every environment variable name and default, and all of `DESIGN.md`. What is **wrong**: the language, the file paths, and — notably — `frontend/CLAUDE.md`'s claim that `make test-unit` is *"the tier CI runs."* It isn't any more. CI builds the frontend image, and `frontend/Dockerfile`'s builder stage runs `gofmt -l` + `go vet` + `go test ./...` before compiling, so a failing **Go** test is what fails CI. The Python tiers are local-only now. Also note `api_client.py` carries a **third** copy of the `API_VERSION` pin. It isn't deployed, but keep it in step or you'll debug a phantom locally. ### `infra/DEPLOY.md` describes the pre-Swarm, pre-monorepo deploy Three separate layers of drift: - It says **prod is deployed with `terraform apply`**. Prod is deployed by a push to `release` firing `deploy.yml`, exactly like beta's `main` path — see [07](07-ci-and-release.md). - It says prod runs a **compose** stack. Prod runs **Swarm** (`infra/deploy/stack/`). - Its file table points at `/opt/thermograph/deploy/deploy.sh` and `/opt/thermograph/docker-compose.yml`. Both moved under `infra/` at the monorepo cutover. The document's own header flags the legacy `provision.sh` / systemd / venv walkthrough as superseded — but the drift goes further than that note admits. The **key-management** sections (Part 1) and the `/etc/hosts` registry note are still accurate and useful. ### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist. The two LAN-dev deploy workflows were deleted rather than ported at the CI consolidation — they were documented as inert (they targeted a monorepo layout at `~/thermograph-dev` on a box still holding a split-era checkout). **LAN dev is a local `make dev-up` concern, not a CI environment.** `ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the workflow set; there are now nine files and the deploy/build ones were collapsed. Everything else in `ACCESS.md` — the host table, the agent-access model, the key rotation procedure, the Swarm/Forgejo tracks — is current and worth reading. ### `backend/README.md` is written for the split-repo era It describes `thermograph-backend` as its own repo with sibling repos, uses the retired image path `git.thermograph.org/emi/thermograph-backend/app`, and says *"there is no Makefile in this repo yet"* — there is (`make test`, `make smoke`, `make clean-venv`). The **"How it works"** section (grid → data → percentiles → caching) is one of the best explanations of the domain model anywhere in the repo. Read it; just mentally translate the topology. ### `CUTOVER-NOTES.md` carries a superseded block, correctly labelled Its per-domain workflow names (`backend-build-push.yml`, `backend-deploy.yml`, …) no longer exist — the six deploy workflows collapsed into `deploy.yml` and the two build-push workflows into `build-push.yml`. The file says so in an explicit "Superseded 2026-07-25" note, and everything it describes about *behaviour* is unchanged. It also references `backend/MONOREPO-CUTOVER-PLAN.md`, which is not in the tree. `CUTOVER-NOTES.md` remains the source of truth for what is and isn't live. ### The domain `CLAUDE.md` files still use repo-era wording They say "this repo" and link to sibling repo names. Mentally map `thermograph-backend` → `backend/`, and fix the wording as you touch files near it — that's the standing instruction in the root `CLAUDE.md`. ### `observability/README.md`: merging is not deploying `/opt/observability` on beta and prod is a checkout of the **archived** `emi/thermograph-observability` repo. It can never `git pull` again. A previous observability PR merged and had **zero live effect** for exactly this reason. The README documents the hand-ship procedure; until the checkouts are re-pointed at the monorepo, that's the only path. Flagged again in [09](09-observability.md). --- ## Live traps ### The `docker-compose.test.yml` files still pull retired image paths Both `backend/docker-compose.test.yml` and `frontend/docker-compose.test.yml` default `BACKEND_IMAGE_PATH` to **`emi/thermograph-backend/app`** — the split-era path. The current image is `emi/thermograph/backend`. - `backend/make smoke` is fine: it builds `:local` and never pulls. - `frontend/make backend-up` **does** pull, and pins `THERMOGRAPH_BACKEND_TEST_TAG` to `v0.0.2-split-ci`. The old packages were deliberately left in the registry for rollback and GC'd only after burn-in, so this may work, may serve a very old backend, or may 404 depending on when you read this. If `make backend-up` fails or behaves oddly, set the path and tag explicitly: ```bash BACKEND_IMAGE_PATH=emi/thermograph/backend \ THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> make backend-up ``` ### `static/units.js`'s `F_REGIONS` is guarded by nothing Source comments in three files claim *"a test asserts all three stay identical."* The only automated assertion is `format_test.go::TestFCountriesMatchesBackend`, which re-parses the backend's Python and compares the **Go** copy. Nothing reads `units.js`. All four copies currently *are* identical (14 codes, checked). But the browser copy is held in step by convention alone. Extending the Go test to parse `units.js` too is a good first contribution. ### The compose project name is load-bearing `infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose from `infra/` derives project `infra` — a silently **new** stack with empty volumes beside the running one. `deploy-dev.sh` exports `COMPOSE_PROJECT_NAME=thermograph-dev` to keep LAN dev separate on purpose. Keep both halves. ### `deploy.sh` hard-resets the host checkout `git reset --hard origin/$BRANCH` runs before anything else. Uncommitted edits in `/opt/thermograph` evaporate. `infra-sync.yml` does the same reset without deploying. The one deliberate exception is the untracked `deploy/.image-tags.env` — losing *that* makes the next single-service deploy roll the sibling onto `local`. ### The deploy scripts eat your rollback image Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the running pair. On beta that succeeds, so beta typically holds **no local rollback target at all**; on prod it usually fails only because Swarm's stopped task containers still hold references. `docker system prune -a` is forbidden on a live box for the same reason — but the deploy script is the thing that actually eats the image, not prune. Verify a tag exists before promising a rollback. ### `daemon` and `lake` are never deploy targets on their own They ride with `backend`. All three run the same image at the same tag — a second binary and a second role inside one image — and the `/internal/*` contract between daemon and web has no version negotiation. There's also a mechanical reason: a single-service deploy runs `up -d --no-deps `, so a service in compose but absent from `TARGETS` would simply never be created. ### `docker compose config --images ` does not filter Confirmed live on Compose v5.3.1: it prints **every** service's image, one per line, in file order. A `| head -1` therefore grabbed `db`'s image, and `deploy.sh`'s daemon capability probe found no daemon binary in a Postgres image and dropped the daemon from *every* backend deploy. The script now builds the image reference from the same variables compose interpolates. ### Prod's Loki streams carry no `service` label Prod runs Swarm; its `job="docker"` streams have no `service` label. A raw `{host="prod", service="backend"}` query matches nothing and reads as "no logs" when it means "wrong selector". Use `logs_query`'s `service` argument. Prod container names also change on every redeploy (the task suffix) — resolve via `docker ps --filter name=` at call time, never hardcode. ### A Grafana alert rule can provision cleanly and never fire A rule whose `condition` names a non-existent refId is valid YAML, provisions without complaint, and pages nobody. Likewise a contact point holding the literal string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API — and the API **redacts** the URL, so you cannot confirm interpolation by reading it back. The only proof is a message arriving in `#ops-alerts`. ### Terraform has no state anywhere A casual `terraform apply` would attempt full re-provisioning of live hosts. It's executable documentation until state is bootstrapped. ### The registry is mesh-only `git.thermograph.org` resolves publicly to beta's IP, but beta's Caddy rejects `/v2/*` from outside the WireGuard mesh. Any host that pulls needs `10.10.0.2 git.thermograph.org` in `/etc/hosts`. Beta itself doesn't — it *is* the box. ### Forgejo runners only have the labels they registered with GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does not. `runs-on: [self-hosted, thermograph-lan]` is permanently unschedulable there — a real bug, found and fixed once. If a job silently stops appearing in the Actions history, check this first. ### Forgejo's `needs:` doesn't cross workflow files `build-push.yml` and `deploy.yml` fire from the same push with no ordering guarantee, and a deploy racing ahead of its build and failing "not found" is confirmed-live. That's why `deploy.sh`'s pull has a bounded ~5-minute retry. ### `/api/v2/cell?prefetch=1` must never spend quota The hard guarantee: a cold cell answers `204` with no body. Anything named "prefetch" or "warm" calls the cache-only loaders. Breaking this gets the whole service rate-limited. ### Background timers must go through leader election `singleton.claim_leader()` — flock per host, Postgres advisory lock per cluster. A timer-driven sweep running in N uvicorn workers × M Swarm hosts multiplies Open-Meteo quota use N×M. That is the documented cause of overnight 429/503s after prod went to 3 workers. The heartbeat is the deliberate exception — a beat is one local file append, so duplicates are free. ### `THERMOGRAPH_DATA_DIR` exists because of a real collision The backend's Python package `data/` and its runtime data dir `/data` collide at the same path once the code lives at the image root. Mounting a volume there shadows and erases `data/*.py`, so `import data.climate` fails at boot. Point the env var **outside the code tree** in a container. And never reintroduce `__file__`-relative paths in a module — `paths.py` resolves everything from the repo root for this reason. ### Alloy can silently stop tailing a container Observed on prod, 13½ hours of `thermograph_worker` logs never reaching Loki while the container was healthy and writing. Not root-caused. Workaround: `sudo docker restart alloy-alloy-1`. `ProdWorkerContainerSilent` now catches the class. ### Nominatim is ~1 req/s, on one thread All reverse geocoding funnels through a single dedicated worker draining a queue. The earlier design slept inside the caller's thread — which in the server is one of Starlette's shared sync-threadpool threads. --- ## If you fix one of these Docs are cheap to correct and the root `CLAUDE.md` already asks you to fix repo-era wording as you touch files near it. If you're in `frontend/` anyway, bringing `frontend/CLAUDE.md` in line with the Go service would be the single highest-value documentation change available — it's the file an agent reads *before* touching that domain. Back to the [index](README.md).