thermograph/docs/onboarding/11-traps.md

# 11. Traps and stale docs

Two kinds of thing here: **live traps** (correct code that will surprise you)
and **stale docs** (files in this repo that currently describe a system that no
longer exists). The second list matters as much as the first — this repo has a
lot of documentation, most of it excellent, and some of it describing the world
before a rewrite.

Everything below was checked against the tree at the time of writing.

---

## Stale docs in this repo

### `frontend/CLAUDE.md` and `frontend/README.md` describe a service that is not deployed

**The live frontend is Go** (`frontend/server/`, module `thermograph/frontend`).
Both documents describe the Python FastAPI/Jinja implementation
(`frontend/app.py`, `content.py`, `api_client.py`, `format.py`,
`templates/*.html.j2`), which was superseded by the Go port and is kept in-tree
as the reference the port was made from.

What is still **correct** in them: every contract (API-version pinning,
`pctOrd`, the `/cell` + ETag relationship, the Fahrenheit set, boot-must-survive-
an-unreachable-backend), every environment variable name and default, and all of
`DESIGN.md`.

What is **wrong**: the language, the file paths, and — notably —
`frontend/CLAUDE.md`'s claim that `make test-unit` is *"the tier CI runs."* It
isn't any more. CI builds the frontend image, and `frontend/Dockerfile`'s
builder stage runs `gofmt -l` + `go vet` + `go test ./...` before compiling, so
a failing **Go** test is what fails CI. The Python tiers are local-only now.

Also note `api_client.py` carries a **third** copy of the `API_VERSION` pin.
It isn't deployed, but keep it in step or you'll debug a phantom locally.

### `infra/DEPLOY.md` describes the pre-Swarm, pre-monorepo deploy

Three separate layers of drift:

- It says **prod is deployed with `terraform apply`**. Prod is deployed by a
  push to `release` firing `deploy.yml`, exactly like beta's `main` path — see
  [07](07-ci-and-release.md).
- It says prod runs a **compose** stack. Prod runs **Swarm**
  (`infra/deploy/stack/`).
- Its file table points at `/opt/thermograph/deploy/deploy.sh` and
  `/opt/thermograph/docker-compose.yml`. Both moved under `infra/` at the
  monorepo cutover.

The document's own header flags the legacy `provision.sh` / systemd / venv
walkthrough as superseded — but the drift goes further than that note admits.
The **key-management** sections (Part 1) and the `/etc/hosts` registry note are
still accurate and useful.

### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows

Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist. The two
LAN-dev deploy workflows were deleted rather than ported at the CI
consolidation — they were documented as inert (they targeted a monorepo layout
at `~/thermograph-dev` on a box still holding a split-era checkout). **LAN dev
is a local `make dev-up` concern, not a CI environment.**

`ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the
workflow set; there are now nine files and the deploy/build ones were collapsed.

Everything else in `ACCESS.md` — the host table, the agent-access model, the key
rotation procedure, the Swarm/Forgejo tracks — is current and worth reading.

### `backend/README.md` is written for the split-repo era

It describes `thermograph-backend` as its own repo with sibling repos, uses the
retired image path `git.thermograph.org/emi/thermograph-backend/app`, and says
*"there is no Makefile in this repo yet"* — there is (`make test`, `make smoke`,
`make clean-venv`).

The **"How it works"** section (grid → data → percentiles → caching) is one of
the best explanations of the domain model anywhere in the repo. Read it; just
mentally translate the topology.

### `CUTOVER-NOTES.md` carries a superseded block, correctly labelled

Its per-domain workflow names (`backend-build-push.yml`, `backend-deploy.yml`, …)
no longer exist — the six deploy workflows collapsed into `deploy.yml` and the
two build-push workflows into `build-push.yml`. The file says so in an explicit
"Superseded 2026-07-25" note, and everything it describes about *behaviour* is
unchanged.

It also references `backend/MONOREPO-CUTOVER-PLAN.md`, which is not in the tree.

`CUTOVER-NOTES.md` remains the source of truth for what is and isn't live.

### The domain `CLAUDE.md` files still use repo-era wording

They say "this repo" and link to sibling repo names. Mentally map
`thermograph-backend` → `backend/`, and fix the wording as you touch files near
it — that's the standing instruction in the root `CLAUDE.md`.

### `observability/README.md`: merging is not deploying

`/opt/observability` on beta and prod is a checkout of the **archived**
`emi/thermograph-observability` repo. It can never `git pull` again. A previous
observability PR merged and had **zero live effect** for exactly this reason.
The README documents the hand-ship procedure; until the checkouts are re-pointed
at the monorepo, that's the only path. Flagged again in
[09](09-observability.md).

---

## Live traps

### The `docker-compose.test.yml` files still pull retired image paths

Both `backend/docker-compose.test.yml` and `frontend/docker-compose.test.yml`
default `BACKEND_IMAGE_PATH` to **`emi/thermograph-backend/app`** — the
split-era path. The current image is `emi/thermograph/backend`.

- `backend/make smoke` is fine: it builds `:local` and never pulls.
- `frontend/make backend-up` **does** pull, and pins
  `THERMOGRAPH_BACKEND_TEST_TAG` to `v0.0.2-split-ci`. The old packages were
  deliberately left in the registry for rollback and GC'd only after burn-in, so
  this may work, may serve a very old backend, or may 404 depending on when you
  read this. If `make backend-up` fails or behaves oddly, set the path and tag
  explicitly:
  ```bash
  BACKEND_IMAGE_PATH=emi/thermograph/backend \
  THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> make backend-up
  ```

### `static/units.js`'s `F_REGIONS` is guarded by nothing

Source comments in three files claim *"a test asserts all three stay
identical."* The only automated assertion is
`format_test.go::TestFCountriesMatchesBackend`, which re-parses the backend's
Python and compares the **Go** copy. Nothing reads `units.js`.

All four copies currently *are* identical (14 codes, checked). But the browser
copy is held in step by convention alone. Extending the Go test to parse
`units.js` too is a good first contribution.

### The compose project name is load-bearing

`infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose
from `infra/` derives project `infra` — a silently **new** stack with empty
volumes beside the running one. `deploy-dev.sh` exports
`COMPOSE_PROJECT_NAME=thermograph-dev` to keep LAN dev separate on purpose.
Keep both halves.

### `deploy.sh` hard-resets the host checkout

`git reset --hard origin/$BRANCH` runs before anything else. Uncommitted edits
in `/opt/thermograph` evaporate. `infra-sync.yml` does the same reset without
deploying. The one deliberate exception is the untracked
`deploy/.image-tags.env` — losing *that* makes the next single-service deploy
roll the sibling onto `local`.

### The deploy scripts eat your rollback image

Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the
running pair. On beta that succeeds, so beta typically holds **no local rollback
target at all**; on prod it usually fails only because Swarm's stopped task
containers still hold references. `docker system prune -a` is forbidden on a
live box for the same reason — but the deploy script is the thing that actually
eats the image, not prune. Verify a tag exists before promising a rollback.

### `daemon` and `lake` are never deploy targets on their own

They ride with `backend`. All three run the same image at the same tag — a
second binary and a second role inside one image — and the `/internal/*`
contract between daemon and web has no version negotiation. There's also a
mechanical reason: a single-service deploy runs `up -d --no-deps <targets>`, so
a service in compose but absent from `TARGETS` would simply never be created.

### `docker compose config --images <service>` does not filter

Confirmed live on Compose v5.3.1: it prints **every** service's image, one per
line, in file order. A `| head -1` therefore grabbed `db`'s image, and
`deploy.sh`'s daemon capability probe found no daemon binary in a Postgres image
and dropped the daemon from *every* backend deploy. The script now builds the
image reference from the same variables compose interpolates.

### Prod's Loki streams carry no `service` label

Prod runs Swarm; its `job="docker"` streams have no `service` label. A raw
`{host="prod", service="backend"}` query matches nothing and reads as "no logs"
when it means "wrong selector". Use `logs_query`'s `service` argument. Prod
container names also change on every redeploy (the task suffix) — resolve via
`docker ps --filter name=` at call time, never hardcode.

### A Grafana alert rule can provision cleanly and never fire

A rule whose `condition` names a non-existent refId is valid YAML, provisions
without complaint, and pages nobody. Likewise a contact point holding the
literal string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the
API — and the API **redacts** the URL, so you cannot confirm interpolation by
reading it back. The only proof is a message arriving in `#ops-alerts`.

### Terraform has no state anywhere

A casual `terraform apply` would attempt full re-provisioning of live hosts.
It's executable documentation until state is bootstrapped.

### The registry is mesh-only

`git.thermograph.org` resolves publicly to beta's IP, but beta's Caddy rejects
`/v2/*` from outside the WireGuard mesh. Any host that pulls needs
`10.10.0.2 git.thermograph.org` in `/etc/hosts`. Beta itself doesn't — it *is*
the box.

### Forgejo runners only have the labels they registered with

GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does not.
`runs-on: [self-hosted, thermograph-lan]` is permanently unschedulable there —
a real bug, found and fixed once. If a job silently stops appearing in the
Actions history, check this first.

### Forgejo's `needs:` doesn't cross workflow files

`build-push.yml` and `deploy.yml` fire from the same push with no ordering
guarantee, and a deploy racing ahead of its build and failing "not found" is
confirmed-live. That's why `deploy.sh`'s pull has a bounded ~5-minute retry.

### `/api/v2/cell?prefetch=1` must never spend quota

The hard guarantee: a cold cell answers `204` with no body. Anything named
"prefetch" or "warm" calls the cache-only loaders. Breaking this gets the whole
service rate-limited.

### Background timers must go through leader election

`singleton.claim_leader()` — flock per host, Postgres advisory lock per cluster.
A timer-driven sweep running in N uvicorn workers × M Swarm hosts multiplies
Open-Meteo quota use N×M. That is the documented cause of overnight 429/503s
after prod went to 3 workers. The heartbeat is the deliberate exception — a beat
is one local file append, so duplicates are free.

### `THERMOGRAPH_DATA_DIR` exists because of a real collision

The backend's Python package `data/` and its runtime data dir `<root>/data`
collide at the same path once the code lives at the image root. Mounting a
volume there shadows and erases `data/*.py`, so `import data.climate` fails at
boot. Point the env var **outside the code tree** in a container. And never
reintroduce `__file__`-relative paths in a module — `paths.py` resolves
everything from the repo root for this reason.

### Alloy can silently stop tailing a container

Observed on prod, 13½ hours of `thermograph_worker` logs never reaching Loki
while the container was healthy and writing. Not root-caused. Workaround:
`sudo docker restart alloy-alloy-1`. `ProdWorkerContainerSilent` now catches the
class.

### Nominatim is ~1 req/s, on one thread

All reverse geocoding funnels through a single dedicated worker draining a
queue. The earlier design slept inside the caller's thread — which in the server
is one of Starlette's shared sync-threadpool threads.

---

## If you fix one of these

Docs are cheap to correct and the root `CLAUDE.md` already asks you to fix
repo-era wording as you touch files near it. If you're in `frontend/` anyway,
bringing `frontend/CLAUDE.md` in line with the Go service would be the single
highest-value documentation change available — it's the file an agent reads
*before* touching that domain.

Back to the [index](README.md).