thermograph/docs/onboarding/10-recipes.md

# 10. Recipes

Task-shaped walkthroughs. Each one names the files you'll touch and the traps
specific to that change.

---

## Add a field to a graded API payload

1. **Build it** in `backend/api/payloads.py` (or `content_payloads.py` for the
   SSR content API). Payload assembly is pure `inputs → dict`; keep HTTP
   semantics with the caller.
2. **Bump `PAYLOAD_VER`** in `api/payloads.py`. Non-negotiable — the derived
   store is token-validated with no clock expiry, so without a bump the old
   shape serves out of cache forever, silently. If the change is content-API
   only, `CONTENT_VER` is the narrower lever.
3. **Decide whether it's breaking.** Additive → fine on `v2`. Breaking → a new
   `v3` router mounted alongside `v2` re-registering only the changed handlers,
   plus `API_CONTRACT_VERSION` bumped in the same PR.
4. **Check `/cell`.** If the field lands in a slice, the slice→view map in
   `frontend/static/cache.js` has to track it.
5. **Consume it** in `frontend/server/internal/contentapi/types.go` and whichever
   template or JS renders it.
6. **Test both sides**: `cd backend && make test`, then
   `cd frontend/server && go test ./...`.
7. **Refresh fixtures if the shape moved**: `cd frontend && make capture-fixtures`
   against a live backend. Those fixtures feed the **Go** tests too.

---

## Add a new SSR page

The backend owns the page's *data and metadata*; the frontend owns its *markup*.

1. `backend/api/content_payloads.py` — a builder returning JSON-safe dicts (raw
   floats, no `Markup`, no ContextVar unit rendering), including `page_title`,
   `canonical_path`, `breadcrumb` and `jsonld`. Those live here, **not** in the
   frontend.
2. `backend/api/content_routes.py` — route wiring plus the same derived-store /
   ETag pattern the neighbouring routes use.
3. `frontend/server/internal/contentapi/` — a typed client method.
4. `frontend/server/internal/render/templates/<page>.html.tmpl` — embedded in
   the binary, so a template change needs a rebuild.
5. `frontend/server/internal/content/pages.go` — the handler; register the route.
6. If the page should be crawlable, add it to the sitemap
   (`backend/api/sitemap.py` and the `/content/sitemap` payload).
7. Copy that isn't per-city data goes in `frontend/content/pages.yaml`, loaded
   fail-loud at boot.

---

## Change something visual

1. Read [`frontend/DESIGN.md`](../../frontend/DESIGN.md) first. It wins over any
   general design guidance.
2. Edit `frontend/static/style.css`. **Use `var(--token)`; never paste a hex.**
   New surfaces take their colour from existing tokens so light mode keeps
   working for free.
3. If it's a grade colour, keep the scale's order and midpoints intact — they
   encode meaning, not decoration, and the charts pull from the same tokens as
   the legend.
4. **Verify at 390 / 800 / 1920 / 2560 / 3840px in both light and dark:**
   ```bash
   cd frontend && .venv/bin/python tools/shoot.py
   ```
   or drive a real browser with the Chrome DevTools MCP. Narrow while
   iterating: `tools/shoot.py index --width 390 --scheme dark`.
5. Watch the hard rules: inputs at `font-size: 16px` minimum (iOS zoom), ~44px
   touch targets, honour `prefers-reduced-motion`, metric order always
   `Precip · High · Low`.

---

## Add or rotate a secret

```bash
sops edit infra/deploy/secrets/prod.yaml    # never a plain editor
git commit && push && open a PR
```

Then it reaches the hosts by either route: an `infra/**` push to `main` fires
`infra-sync.yml` (re-renders prod's and beta's env files — `/etc/thermograph.env`
and `/etc/thermograph-beta.env`, both on vps2 — rolls nothing), or the next app
deploy renders it as step 2 of `deploy.sh`. Dev's vault (`dev.yaml`) is never
part of that sync — it's rendered only by dev's own deploy on vps1, and it
never layers `common.yaml` regardless.

Decide **which file** first:

- identical on prod *and* beta, and not a "lets one environment act as another"
  credential → `common.yaml` (never reaches dev, on vps1, at all);
- per-environment, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` /
  `THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`;
- dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's.

Preview with `infra/deploy/secrets/dry-run-render.sh`. Audit cross-environment
drift with the `key-gaps` skill (names only, never values).

**Never** hand-edit `/etc/thermograph.env`, paste a plaintext secret onto a box,
or run `seed-from-live.sh`.

If a rotated value needs containers recreated to take effect, remember
`infra-sync` rolls nothing — you need an app deploy or a by-hand
`SERVICE=all … deploy.sh`.

---

## Add a recurring background job

Ask one question first: **does it need data?**

- **No** (a timer, a connection, an external poke) → add a `cron.Job` in
  `backend/daemon/main.go`. One replica, so no election needed. Give it an env
  var for its interval, defaulting sensibly and falling back to the default on a
  malformed value.
- **Yes** → add an `/internal/*` route in `backend/api/internal_routes.py` and
  have the daemon POST to it. Grading depends on polars and the cache;
  reimplementing it out-of-process would let results drift from the API's.

If you genuinely need an in-process background thread in the Python app instead,
**read `core/singleton.py` first**. Anything that issues upstream fetches on a
timer must run behind `claim_leader()` — N workers × M hosts multiplies quota
use N×M, and that has already caused overnight 429s in production. Things that
drain a per-worker request-fed queue (the neighbour warmer) correctly run
everywhere; so does the heartbeat, because a beat costs one local file append.

Then give it a heartbeat (`audit.log_heartbeat()`) so observability can alert on
its death — see the "too quiet to alert on" gap in
[observability](09-observability.md).

---

## Change a Grafana alert or dashboard

**Never through the UI.** Provisioning overwrites it on the next start.

1. Edit `observability/grafana/provisioning/alerting/*.yml` or
   `observability/grafana/dashboards/*.json`.
2. If you're changing a threshold, **re-derive it against real Loki data** — the
   working for the current number is in the comments beside the rule.
3. Open a PR. `observability-validate.yml` will check that every condition names
   a real refId, every policy routes to a real receiver, and no literal webhook
   URL is committed.
4. **Merging is not deploying.** Ship it by hand per
   `observability/README.md` — and remember `docker restart` won't pick up a new
   environment variable.
5. Verify by forcing a real message into `#ops-alerts`. The API redacts the
   webhook, so arrival is the only proof.

---

## Debug: "the site is serving stale data"

Work the token chain, in this order:

1. `GET /api/version` on the affected environment — is `payload_ver` what you
   expect? If a payload shape changed and this didn't move, that's your answer.
2. Is the derived-store token advancing? It's built from `PAYLOAD_VER` plus the
   history end date (`history_token`), so a cell whose history stopped updating
   pins its token.
3. Is the *history* updating? Check `climate_sync` freshness
   (`infra/ops/dbq.sh prod …`) and whether upstream is in a rate-limit cooldown
   (the errors log, tagged `retry`/`error`).
4. Is it the **frontend's** TTL cache instead? `THERMOGRAPH_SSR_CACHE_TTL`
   defaults to 600s, in front of the backend's own caching.
5. Is it the **browser**? `cache.js` seeds IndexedDB from the `/cell` bundle. If
   ETags stopped being visible cross-origin (`expose_headers`), revalidation
   silently stops.

---

## Debug: "a deploy went green but nothing changed"

Common causes, most likely first:

1. **The tree was already identical.** A promotion between branches whose trees
   match merges nothing while still firing the deploy workflows.
2. **The domain wasn't touched.** `deploy.yml` re-checks whether *this* push
   touched *this* domain, and skips the leg if not.
3. **The tag is domain-keyed.** `deployed_version` showing a tag that isn't the
   branch head is correct behaviour, not drift.
4. **Someone hand-edited `/opt/thermograph`.** `deploy.sh` runs
   `git reset --hard` first; the edit is gone.
5. **It's an observability change.** `/opt/observability` points at the archived
   repo and cannot pull. See above.

---

## Promote to beta, and then to prod

```
dev → main   your call, batched. Beta is the test environment;
             not promoting is not testing.
main → release   ALWAYS the owner's call. Prepare it, state what would ship,
             the CI evidence and how long it soaked on beta, then recommend —
             including recommending against.
```

Centralis's `promote(from, to)` does the mechanics and enforces the rule;
`forge_pr_merge` refuses `main → release` under any flag. Before promoting,
compare the two tips' **trees**, not their commit counts.

---

## Roll back

Redeploying the previous image tag:

```bash
SERVICE=backend BACKEND_IMAGE_TAG=sha-<previous12hex> \
  /opt/thermograph/infra/deploy/deploy.sh
```

**Check the tag actually exists first.** Both deploy scripts *end* by deleting
images outside the running pair. Beta now runs Swarm exactly like prod
(co-resident on vps2), so both usually keep a handful of stopped-task
references; dev, the one remaining compose environment (on vps1), is the one
that typically holds no local rollback target at all. `rollback_to(dry_run:
true)` checks host and registry and will tell you when it's in neither.

---

## Regenerate the bundled city data

`backend/cities.json` and `cities_flavor.json` are generated reference data, not
runtime state:

```bash
cd backend
python gen_cities.py      # the curated city set
python gen_flavor.py      # the editorial blurbs
```

Editorial copy has its own voice guide — the **`city-copy`** skill defines what
good reads like, and **`backfill-city-copy`** drives generation at scale. Both
enforce the relative-grades rule.

After a deploy, `warm_cities.py` pre-warms those cells so `/climate` pages serve
from cache and a crawl never bursts the archive quota. It's idempotent, paced,
and claims a flock — deploys launch it detached with no overlap check, so two
overlapping runs would otherwise double-spend quota and race the cache writes.

---

## Work on a live host

Prefer Centralis: `run_on_host`, `fleet_status`, `deployed_version`,
`logs_query`, `sql_query`.

You may, without asking: read anything; restart or roll a wedged service; clear
a full disk; kill a runaway process; bounce a crash-looping container; run the
blessed deploy path onto an already-published image.

You must not: hand-edit the checkout to persist a change (`git reset --hard`
eats it); hand-edit `/etc/thermograph.env` (rendered artifact); force-push;
deploy prod from an unmerged branch; `docker system prune -a` on a live box (it
deletes your rollback image — `docker image prune -f` is the safe form); or
restart the `_db` service casually. Postgres/Timescale restarts and volume
operations get confirmed with the operator first.

Next: [Traps and stale docs](11-traps.md).