thermograph/docs/onboarding/10-recipes.md

10. Recipes

Task-shaped walkthroughs. Each one names the files you'll touch and the traps specific to that change.

---

Add a field to a graded API payload

1. Build it in backend/api/payloads.py (or content_payloads.py for the SSR content API). Payload assembly is pure inputs → dict; keep HTTP semantics with the caller.
2. Bump PAYLOAD_VER in api/payloads.py. Non-negotiable — the derived store is token-validated with no clock expiry, so without a bump the old shape serves out of cache forever, silently. If the change is content-API only, CONTENT_VER is the narrower lever.
3. Decide whether it's breaking. Additive → fine on v2. Breaking → a new v3 router mounted alongside v2 re-registering only the changed handlers, plus API_CONTRACT_VERSION bumped in the same PR.
4. Check /cell. If the field lands in a slice, the slice→view map in frontend/static/cache.js has to track it.
5. Consume it in frontend/server/internal/contentapi/types.go and whichever template or JS renders it.
6. Test both sides: cd backend && make test, then cd frontend/server && go test ./....
7. Refresh fixtures if the shape moved: cd frontend && make capture-fixtures against a live backend. Those fixtures feed the Go tests too.

---

Add a new SSR page

The backend owns the page's data and metadata; the frontend owns its markup.

1. backend/api/content_payloads.py — a builder returning JSON-safe dicts (raw floats, no Markup, no ContextVar unit rendering), including page_title, canonical_path, breadcrumb and jsonld. Those live here, not in the frontend.
2. backend/api/content_routes.py — route wiring plus the same derived-store / ETag pattern the neighbouring routes use.
3. frontend/server/internal/contentapi/ — a typed client method.
4. frontend/server/internal/render/templates/<page>.html.tmpl — embedded in the binary, so a template change needs a rebuild.
5. frontend/server/internal/content/pages.go — the handler; register the route.
6. If the page should be crawlable, add it to the sitemap (backend/api/sitemap.py and the /content/sitemap payload).
7. Copy that isn't per-city data goes in frontend/content/pages.yaml, loaded fail-loud at boot.

---

Change something visual

1. Read frontend/DESIGN.md first. It wins over any general design guidance.
2. Edit frontend/static/style.css. Use var(--token); never paste a hex. New surfaces take their colour from existing tokens so light mode keeps working for free.
3. If it's a grade colour, keep the scale's order and midpoints intact — they encode meaning, not decoration, and the charts pull from the same tokens as the legend.
4. Verify at 390 / 800 / 1920 / 2560 / 3840px in both light and dark:

   cd frontend && .venv/bin/python tools/shoot.py
   

   or drive a real browser with the Chrome DevTools MCP. Narrow while iterating: tools/shoot.py index --width 390 --scheme dark.
5. Watch the hard rules: inputs at font-size: 16px minimum (iOS zoom), ~44px touch targets, honour prefers-reduced-motion, metric order always Precip · High · Low.

---

Add or rotate a secret

sops edit infra/deploy/secrets/prod.yaml    # never a plain editor
git commit && push && open a PR

Then it reaches the hosts by either route: an infra/** push to main fires infra-sync.yml (re-renders prod's and beta's env files — /etc/thermograph.env and /etc/thermograph-beta.env, both on vps2 — rolls nothing), or the next app deploy renders it as step 2 of deploy.sh. Dev's vault (dev.yaml) is never part of that sync — it's rendered only by dev's own deploy on vps1, and it never layers common.yaml regardless.

Decide which file first:

• identical on prod and beta, and not a "lets one environment act as another" credential → common.yaml (never reaches dev, on vps1, at all);
• per-environment, or one of POSTGRES_PASSWORD / THERMOGRAPH_AUTH_SECRET / THERMOGRAPH_DATABASE_URL → prod.yaml / beta.yaml;
• dev → dev.yaml, and give it its own value. Never a copy of prod's.

Preview with infra/deploy/secrets/dry-run-render.sh. Audit cross-environment drift with the key-gaps skill (names only, never values).

Never hand-edit /etc/thermograph.env, paste a plaintext secret onto a box, or run seed-from-live.sh.

If a rotated value needs containers recreated to take effect, remember infra-sync rolls nothing — you need an app deploy or a by-hand SERVICE=all … deploy.sh.

---

Add a recurring background job

Ask one question first: does it need data?

• No (a timer, a connection, an external poke) → add a cron.Job in backend/daemon/main.go. One replica, so no election needed. Give it an env var for its interval, defaulting sensibly and falling back to the default on a malformed value.
• Yes → add an /internal/* route in backend/api/internal_routes.py and have the daemon POST to it. Grading depends on polars and the cache; reimplementing it out-of-process would let results drift from the API's.

If you genuinely need an in-process background thread in the Python app instead, read core/singleton.py first. Anything that issues upstream fetches on a timer must run behind claim_leader() — N workers × M hosts multiplies quota use N×M, and that has already caused overnight 429s in production. Things that drain a per-worker request-fed queue (the neighbour warmer) correctly run everywhere; so does the heartbeat, because a beat costs one local file append.

Then give it a heartbeat (audit.log_heartbeat()) so observability can alert on its death — see the "too quiet to alert on" gap in observability.

---

Change a Grafana alert or dashboard

Never through the UI. Provisioning overwrites it on the next start.

1. Edit observability/grafana/provisioning/alerting/*.yml or observability/grafana/dashboards/*.json.
2. If you're changing a threshold, re-derive it against real Loki data — the working for the current number is in the comments beside the rule.
3. Open a PR. observability-validate.yml will check that every condition names a real refId, every policy routes to a real receiver, and no literal webhook URL is committed.
4. Merging is not deploying. Ship it by hand per observability/README.md — and remember docker restart won't pick up a new environment variable.
5. Verify by forcing a real message into #ops-alerts. The API redacts the webhook, so arrival is the only proof.

---

Debug: "the site is serving stale data"

Work the token chain, in this order:

1. GET /api/version on the affected environment — is payload_ver what you expect? If a payload shape changed and this didn't move, that's your answer.
2. Is the derived-store token advancing? It's built from PAYLOAD_VER plus the history end date (history_token), so a cell whose history stopped updating pins its token.
3. Is the history updating? Check climate_sync freshness (infra/ops/dbq.sh prod …) and whether upstream is in a rate-limit cooldown (the errors log, tagged retry/error).
4. Is it the frontend's TTL cache instead? THERMOGRAPH_SSR_CACHE_TTL defaults to 600s, in front of the backend's own caching.
5. Is it the browser? cache.js seeds IndexedDB from the /cell bundle. If ETags stopped being visible cross-origin (expose_headers), revalidation silently stops.

---

Debug: "a deploy went green but nothing changed"

Common causes, most likely first:

1. The tree was already identical. A promotion between branches whose trees match merges nothing while still firing the deploy workflows.
2. The domain wasn't touched. deploy.yml re-checks whether this push touched this domain, and skips the leg if not.
3. The tag is domain-keyed. deployed_version showing a tag that isn't the branch head is correct behaviour, not drift.
4. Someone hand-edited /opt/thermograph. deploy.sh runs git reset --hard first; the edit is gone.
5. It's an observability change. /opt/observability points at the archived repo and cannot pull. See above.

---

Promote to beta, and then to prod

dev → main   your call, batched. Beta is the test environment;
             not promoting is not testing.
main → release   ALWAYS the owner's call. Prepare it, state what would ship,
             the CI evidence and how long it soaked on beta, then recommend —
             including recommending against.

Centralis's promote(from, to) does the mechanics and enforces the rule; forge_pr_merge refuses main → release under any flag. Before promoting, compare the two tips' trees, not their commit counts.

---

Roll back

Redeploying the previous image tag:

SERVICE=backend BACKEND_IMAGE_TAG=sha-<previous12hex> \
  /opt/thermograph/infra/deploy/deploy.sh

Check the tag actually exists first. Both deploy scripts end by deleting images outside the running pair. Beta now runs Swarm exactly like prod (co-resident on vps2), so both usually keep a handful of stopped-task references; dev, the one remaining compose environment (on vps1), is the one that typically holds no local rollback target at all. rollback_to(dry_run: true) checks host and registry and will tell you when it's in neither.

---

Regenerate the bundled city data

backend/cities.json and cities_flavor.json are generated reference data, not runtime state:

cd backend
python gen_cities.py      # the curated city set
python gen_flavor.py      # the editorial blurbs

Editorial copy has its own voice guide — the city-copy skill defines what good reads like, and backfill-city-copy drives generation at scale. Both enforce the relative-grades rule.

After a deploy, warm_cities.py pre-warms those cells so /climate pages serve from cache and a crawl never bursts the archive quota. It's idempotent, paced, and claims a flock — deploys launch it detached with no overlap check, so two overlapping runs would otherwise double-spend quota and race the cache writes.

---

Work on a live host

Prefer Centralis: run_on_host, fleet_status, deployed_version, logs_query, sql_query.

You may, without asking: read anything; restart or roll a wedged service; clear a full disk; kill a runaway process; bounce a crash-looping container; run the blessed deploy path onto an already-published image.

You must not: hand-edit the checkout to persist a change (git reset --hard eats it); hand-edit /etc/thermograph.env (rendered artifact); force-push; deploy prod from an unmerged branch; docker system prune -a on a live box (it deletes your rollback image — docker image prune -f is the safe form); or restart the _db service casually. Postgres/Timescale restarts and volume operations get confirmed with the operator first.

Next: Traps and stale docs.