# 10. Recipes Task-shaped walkthroughs. Each one names the files you'll touch and the traps specific to that change. --- ## Add a field to a graded API payload 1. **Build it** in `backend/api/payloads.py` (or `content_payloads.py` for the SSR content API). Payload assembly is pure `inputs → dict`; keep HTTP semantics with the caller. 2. **Bump `PAYLOAD_VER`** in `api/payloads.py`. Non-negotiable — the derived store is token-validated with no clock expiry, so without a bump the old shape serves out of cache forever, silently. If the change is content-API only, `CONTENT_VER` is the narrower lever. 3. **Decide whether it's breaking.** Additive → fine on `v2`. Breaking → a new `v3` router mounted alongside `v2` re-registering only the changed handlers, plus `API_CONTRACT_VERSION` bumped in the same PR. 4. **Check `/cell`.** If the field lands in a slice, the slice→view map in `frontend/static/cache.js` has to track it. 5. **Consume it** in `frontend/server/internal/contentapi/types.go` and whichever template or JS renders it. 6. **Test both sides**: `cd backend && make test`, then `cd frontend/server && go test ./...`. 7. **Refresh fixtures if the shape moved**: `cd frontend && make capture-fixtures` against a live backend. Those fixtures feed the **Go** tests too. --- ## Add a new SSR page The backend owns the page's *data and metadata*; the frontend owns its *markup*. 1. `backend/api/content_payloads.py` — a builder returning JSON-safe dicts (raw floats, no `Markup`, no ContextVar unit rendering), including `page_title`, `canonical_path`, `breadcrumb` and `jsonld`. Those live here, **not** in the frontend. 2. `backend/api/content_routes.py` — route wiring plus the same derived-store / ETag pattern the neighbouring routes use. 3. `frontend/server/internal/contentapi/` — a typed client method. 4. `frontend/server/internal/render/templates/.html.tmpl` — embedded in the binary, so a template change needs a rebuild. 5. `frontend/server/internal/content/pages.go` — the handler; register the route. 6. If the page should be crawlable, add it to the sitemap (`backend/api/sitemap.py` and the `/content/sitemap` payload). 7. Copy that isn't per-city data goes in `frontend/content/pages.yaml`, loaded fail-loud at boot. --- ## Change something visual 1. Read [`frontend/DESIGN.md`](../../frontend/DESIGN.md) first. It wins over any general design guidance. 2. Edit `frontend/static/style.css`. **Use `var(--token)`; never paste a hex.** New surfaces take their colour from existing tokens so light mode keeps working for free. 3. If it's a grade colour, keep the scale's order and midpoints intact — they encode meaning, not decoration, and the charts pull from the same tokens as the legend. 4. **Verify at 390 / 800 / 1920 / 2560 / 3840px in both light and dark:** ```bash cd frontend && .venv/bin/python tools/shoot.py ``` or drive a real browser with the Chrome DevTools MCP. Narrow while iterating: `tools/shoot.py index --width 390 --scheme dark`. 5. Watch the hard rules: inputs at `font-size: 16px` minimum (iOS zoom), ~44px touch targets, honour `prefers-reduced-motion`, metric order always `Precip · High · Low`. --- ## Add or rotate a secret ```bash sops edit infra/deploy/secrets/prod.yaml # never a plain editor git commit && push && open a PR ``` Then it reaches the hosts by either route: an `infra/**` push to `main` fires `infra-sync.yml` (re-renders prod's and beta's env files — `/etc/thermograph.env` and `/etc/thermograph-beta.env`, both on vps2 — rolls nothing), or the next app deploy renders it as step 2 of `deploy.sh`. Dev's vault (`dev.yaml`) is never part of that sync — it's rendered only by dev's own deploy on vps1, and it never layers `common.yaml` regardless. Decide **which file** first: - identical on prod *and* beta, and not a "lets one environment act as another" credential → `common.yaml` (never reaches dev, on vps1, at all); - per-environment, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` / `THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`; - dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's. Preview with `infra/deploy/secrets/dry-run-render.sh`. Audit cross-environment drift with the `key-gaps` skill (names only, never values). **Never** hand-edit `/etc/thermograph.env`, paste a plaintext secret onto a box, or run `seed-from-live.sh`. If a rotated value needs containers recreated to take effect, remember `infra-sync` rolls nothing — you need an app deploy or a by-hand `SERVICE=all … deploy.sh`. --- ## Add a recurring background job Ask one question first: **does it need data?** - **No** (a timer, a connection, an external poke) → add a `cron.Job` in `backend/daemon/main.go`. One replica, so no election needed. Give it an env var for its interval, defaulting sensibly and falling back to the default on a malformed value. - **Yes** → add an `/internal/*` route in `backend/api/internal_routes.py` and have the daemon POST to it. Grading depends on polars and the cache; reimplementing it out-of-process would let results drift from the API's. If you genuinely need an in-process background thread in the Python app instead, **read `core/singleton.py` first**. Anything that issues upstream fetches on a timer must run behind `claim_leader()` — N workers × M hosts multiplies quota use N×M, and that has already caused overnight 429s in production. Things that drain a per-worker request-fed queue (the neighbour warmer) correctly run everywhere; so does the heartbeat, because a beat costs one local file append. Then give it a heartbeat (`audit.log_heartbeat()`) so observability can alert on its death — see the "too quiet to alert on" gap in [observability](09-observability.md). --- ## Change a Grafana alert or dashboard **Never through the UI.** Provisioning overwrites it on the next start. 1. Edit `observability/grafana/provisioning/alerting/*.yml` or `observability/grafana/dashboards/*.json`. 2. If you're changing a threshold, **re-derive it against real Loki data** — the working for the current number is in the comments beside the rule. 3. Open a PR. `observability-validate.yml` will check that every condition names a real refId, every policy routes to a real receiver, and no literal webhook URL is committed. 4. **Merging is not deploying.** Ship it by hand per `observability/README.md` — and remember `docker restart` won't pick up a new environment variable. 5. Verify by forcing a real message into `#ops-alerts`. The API redacts the webhook, so arrival is the only proof. --- ## Debug: "the site is serving stale data" Work the token chain, in this order: 1. `GET /api/version` on the affected environment — is `payload_ver` what you expect? If a payload shape changed and this didn't move, that's your answer. 2. Is the derived-store token advancing? It's built from `PAYLOAD_VER` plus the history end date (`history_token`), so a cell whose history stopped updating pins its token. 3. Is the *history* updating? Check `climate_sync` freshness (`infra/ops/dbq.sh prod …`) and whether upstream is in a rate-limit cooldown (the errors log, tagged `retry`/`error`). 4. Is it the **frontend's** TTL cache instead? `THERMOGRAPH_SSR_CACHE_TTL` defaults to 600s, in front of the backend's own caching. 5. Is it the **browser**? `cache.js` seeds IndexedDB from the `/cell` bundle. If ETags stopped being visible cross-origin (`expose_headers`), revalidation silently stops. --- ## Debug: "a deploy went green but nothing changed" Common causes, most likely first: 1. **The tree was already identical.** A promotion between branches whose trees match merges nothing while still firing the deploy workflows. 2. **The domain wasn't touched.** `deploy.yml` re-checks whether *this* push touched *this* domain, and skips the leg if not. 3. **The tag is domain-keyed.** `deployed_version` showing a tag that isn't the branch head is correct behaviour, not drift. 4. **Someone hand-edited `/opt/thermograph`.** `deploy.sh` runs `git reset --hard` first; the edit is gone. 5. **It's an observability change.** `/opt/observability` points at the archived repo and cannot pull. See above. --- ## Promote to beta, and then to prod ``` dev → main your call, batched. Beta is the test environment; not promoting is not testing. main → release ALWAYS the owner's call. Prepare it, state what would ship, the CI evidence and how long it soaked on beta, then recommend — including recommending against. ``` Centralis's `promote(from, to)` does the mechanics and enforces the rule; `forge_pr_merge` refuses `main → release` under any flag. Before promoting, compare the two tips' **trees**, not their commit counts. --- ## Roll back Redeploying the previous image tag: ```bash SERVICE=backend BACKEND_IMAGE_TAG=sha- \ /opt/thermograph/infra/deploy/deploy.sh ``` **Check the tag actually exists first.** Both deploy scripts *end* by deleting images outside the running pair. Beta now runs Swarm exactly like prod (co-resident on vps2), so both usually keep a handful of stopped-task references; dev, the one remaining compose environment (on vps1), is the one that typically holds no local rollback target at all. `rollback_to(dry_run: true)` checks host and registry and will tell you when it's in neither. --- ## Regenerate the bundled city data `backend/cities.json` and `cities_flavor.json` are generated reference data, not runtime state: ```bash cd backend python gen_cities.py # the curated city set python gen_flavor.py # the editorial blurbs ``` Editorial copy has its own voice guide — the **`city-copy`** skill defines what good reads like, and **`backfill-city-copy`** drives generation at scale. Both enforce the relative-grades rule. After a deploy, `warm_cities.py` pre-warms those cells so `/climate` pages serve from cache and a crawl never bursts the archive quota. It's idempotent, paced, and claims a flock — deploys launch it detached with no overlap check, so two overlapping runs would otherwise double-spend quota and race the cache writes. --- ## Work on a live host Prefer Centralis: `run_on_host`, `fleet_status`, `deployed_version`, `logs_query`, `sql_query`. You may, without asking: read anything; restart or roll a wedged service; clear a full disk; kill a runaway process; bounce a crash-looping container; run the blessed deploy path onto an already-published image. You must not: hand-edit the checkout to persist a change (`git reset --hard` eats it); hand-edit `/etc/thermograph.env` (rendered artifact); force-push; deploy prod from an unmerged branch; `docker system prune -a` on a live box (it deletes your rollback image — `docker image prune -f` is the safe form); or restart the `_db` service casually. Postgres/Timescale restarts and volume operations get confirmed with the operator first. Next: [Traps and stale docs](11-traps.md).