Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions

docs: add a developer onboarding guide for the monorepo
All checks were successful
secrets-guard / encrypted (push) Successful in 5s
Details
shell-lint / shellcheck (push) Successful in 6s
Details

Browse source 
Twelve documents under docs/onboarding/ covering orientation, local setup,
the repo map, per-domain deep dives, the cross-service contracts, CI and the
release flow, infra and secrets, observability, task recipes, and a list of
which docs in this tree are currently stale.

Every command in the setup guide was run against this checkout: the backend
suite (429 passed, 8 skipped), the frontend Go suite, a venv boot of the
backend, and a build+boot of the Go frontend.

Two findings recorded along the way:

- static/units.js's F_REGIONS is guarded by no test, despite three source
  comments claiming "a test asserts all three stay identical". The Go test
  only cross-checks the Go copy against the backend's Python. All four copies
  are currently identical.
- backend/ and frontend/docker-compose.test.yml still default to the retired
  emi/thermograph-backend/app image path, and the frontend harness pins the
  split-era v0.0.2-split-ci tag.

Claude-Session: https://claude.ai/code/session_01AfXqHrxCJLs2D7hpQkiUiJ
This commit is contained in:
Emi Griffith 2026-07-25 11:25:52 -07:00 committed by emi
parent b32b5841f3
commit 12441be0c1
13 changed files with 2531 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
README.md
View file

@ -16,6 +16,13 @@ for: **per-domain images, per-domain deploys, and an async FE/BE contract**.
`thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no `thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no
build artifacts, different change cadence). build artifacts, different change cadence).
## New here?
[`docs/onboarding/`](docs/onboarding/README.md) is the developer onboarding
path: orientation, verified local-setup recipes, a per-domain deep dive, the
cross-service contracts that break silently, the release flow, and a list of
which docs in this repo are currently stale.
## How CI stays decoupled ## How CI stays decoupled
Every workflow in `.forgejo/workflows/` is **path-filtered to its domain**: a Every workflow in `.forgejo/workflows/` is **path-filtered to its domain**: a

146
docs/onboarding/01-orientation.md Normal file
View file

@ -0,0 +1,146 @@
# 1. Orientation
## What the product actually claims
Thermograph answers one question: **how unusual is the weather here, right now,
compared to this exact place's own history?**
That framing is not decoration — it constrains the code, the copy and the
colour scales:
- A grade is a **percentile against ~45 years of that grid cell's own record**,
not an absolute reading. 60 °F is *Above Normal* on a cool coast and *Below
Normal* inland on the same afternoon.
- Tier names are therefore relative words — *Near Record*, *High*, *Above
Normal*, *Normal*, *Below Normal*, *Low*, *Near Record* — never "hot",
"warm", "cold". This rule binds prose, alt text, commit messages and API
fields alike.
- Precipitation is graded on its **own** ladder, because most days are dry: a
rainy day is ranked only among the rain days in its seasonal window, and dry
days are coloured by dry-streak length instead.
Public since **2026-07-16**. Free. Run by one operator plus automation, which
is why so much of this repo is about making irreversible things hard.
## The domain model, in six steps
1. **Grid** (`backend/data/grid.py`) — an arbitrary lat/lon snaps to a stable
**~4 sq mi cell**. Latitude rows are ~2 miles tall; longitude spacing scales
by `cos(latitude)` so cells stay roughly square at any latitude. The cell id
is deterministic and is the cache key for everything downstream. Worldwide
coverage, no gaps.
2. **History** (`backend/data/climate.py`) — the full **1980 → present daily
record** (max/min temp, precip, wind, gust, humidity, feels-like) for that
cell, fetched once and stored durably. Expensive to obtain, so it is treated
as source-of-truth data, not cache.
3. **Recent + forecast** — a separate bundle of recent observations plus ~8
forward days, refreshed on its own TTL. This is what "today" is graded from.
4. **Grading** (`backend/data/grading.py`) — for a given day of year, the
reference distribution is *every historical day within ±7 days of it*
(a 15-day seasonal window that wraps the year end). The observed value is
placed on that distribution as an **empirical mid-rank percentile**, then
mapped onto `TEMP_BANDS` / `RAIN_BANDS`.
5. **Payloads** (`backend/api/payloads.py`) — pure "inputs → response dict"
assembly. One definition per payload shape, shared by the HTTP routes, the
`/cell` bundle and offline tooling.
6. **Caching** (`backend/data/store.py`) — every derived payload is stored
against a **validity token** that encodes everything the payload depends on.
A token mismatch is simply a miss. Freshness is driven by the token
advancing, *never* by clock expiry — so stale data cannot be served, and one
`PAYLOAD_VER` bump atomically orphans every pre-upgrade row.
If you internalise one thing: **the token is the cache-correctness mechanism,
and `PAYLOAD_VER` is the lever that moves it.** See
[contracts](06-contracts.md).
## The estate
Four machines on a WireGuard mesh (`10.10.0.0/24`):
| Host | Mesh IP | Public | Runs |
|---|---|---|---|
| **prod** | `10.10.0.1` | `169.58.46.181`, thermograph.org | Docker **Swarm** app stack, TimescaleDB, Postfix, backups, Centralis |
| **beta** | `10.10.0.2` | `75.119.132.91`, beta.thermograph.org | **compose** app stack, Forgejo (git + CI + registry), Grafana + Loki |
| **desktop** | `10.10.0.3` | — | LAN dev stack, the Forgejo Actions runner, your editor |
| phone | — | — | alerts |
Two consequences you will hit within the first week:
- **Forgejo is mesh-only.** `git.thermograph.org` resolves publicly to beta's
IP, but beta's Caddy rejects `/v2/*` (the registry API) from outside the mesh.
Any host that pulls images needs `10.10.0.2 git.thermograph.org` in
`/etc/hosts`.
- **Prod and beta run different orchestrators.** Prod is Swarm
(`infra/deploy/stack/`), beta and LAN dev are compose
(`infra/docker-compose.yml`). `deploy.sh` routes between them by reading
`/etc/thermograph/deploy-mode` on the box. Most of the time you don't care —
but when you're reading logs or naming containers, you do
(see [observability](09-observability.md)).
## How a change travels
```
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
│
promotion PR
▼
main ──▶ beta.thermograph.org
│
promotion PR ← the owner's call
▼
release ──▶ thermograph.org
```
`dev`, `main` and `release` are **protected** — no direct pushes, for humans or
agents. Promotion is by pull request, continuously and per-decision; there is
no release calendar. Full detail in [CI and release](07-ci-and-release.md).
## The four rules that are not negotiable
These come from the operator, via Centralis's own onboarding. They override
convenience.
1. **Everything is a PR.** If it didn't go through the pipeline, it didn't
happen. This includes Grafana dashboards (provisioned from repo JSON — a UI
edit is silently overwritten) and secrets (SOPS vault only).
2. **Boring prod.** Anything clever proves itself on `dev` or beta first.
3. **Privacy is a feature.** No tracking cookies, no per-visitor IDs,
allowlisted anonymous counters only (`backend/core/metrics.py`,
`POST /api/v2/event`). Cheap to keep, expensive to retrofit.
4. **Grades are relative.** Never describe a percentile as hot or cold.
### And a fifth, local to the repo
**Commits and PRs are concise and technical, and never mention AI, assistants
or automated authorship.** Every domain `CLAUDE.md` says this; it applies to
you too.
## Quota discipline
Thermograph runs on free upstream data sources. Two hard rules protect them:
- **Anything named "prefetch" or "warm" must never spend the Open-Meteo
quota.** The warming paths call `climate.load_cached_history` /
`load_cached_recent_forecast` — the cache-only loaders — and skip cells that
aren't warm. A cell self-heals on first real request instead.
- **Nominatim gets at most ~1 request/second.** All reverse geocoding funnels
through a single dedicated worker thread
(`climate._revgeo_worker`) that paces itself; never call it from a request
thread.
Breaking either one gets the whole service rate-limited, which is why
`singleton.claim_leader()` exists: a background sweep running in N uvicorn
workers × M hosts multiplies quota use by N×M. That failure has already
happened once, in production, at 3 workers.
## Where the written record lives
| Question | Where |
|---|---|
| How does this repo work? | The domain `CLAUDE.md` files, then this set |
| Why is the architecture like this? | `thermograph-docs` (ADRs) — `docs_search` |
| How do I operate the fleet? | `thermograph-docs` runbooks, plus the `thermograph-ops` Centralis skill |
| What did someone find out last Tuesday? | `notes_search` |
| What is and isn't live yet? | [`CUTOVER-NOTES.md`](../../CUTOVER-NOTES.md) at the repo root |
Next: [Local setup](02-setup.md).

217
docs/onboarding/02-setup.md Normal file
View file

@ -0,0 +1,217 @@
# 2. Local setup
Every command in this document was run against this checkout and produced the
output shown. If one fails for you, the difference is your machine, not the doc.
## Toolchain
| Tool | Why | Notes |
|---|---|---|
| **Python 3.12** | backend runtime and test suite | Must have the `_sqlite3` extension. A pyenv 3.10 without it will fail at `conftest` import. `scripts/test.sh` looks for `python3.12` (or uses `uv`) rather than whatever `python3` is on `PATH` — deliberately. |
| **Go 1.26** | frontend service and the backend daemon | Both are static `CGO_ENABLED=0` builds. |
| **Docker** | images, the compose/Swarm stacks, smoke tests | Docker CLI ≥ 27 is fine. |
| **uv** *(recommended)* | fast venv creation | `scripts/test.sh` uses it when present. |
| **shellcheck v0.11.0** | matches CI's pin exactly | Install to `~/.local/bin/shellcheck`. A *different* version can invent new findings and fail CI on an unrelated push. |
| **sops + age** | reading/editing the secrets vault | Only needed if you touch `infra/deploy/secrets/`. |
| **jq** | the repo's `.claude` hooks use it | Without it the prod-guard hook fails toward *asking*, which is safe but noisy. |
Check what you have:
```bash
which go python3.12 uv docker sops age jq shellcheck
go version && python3.12 --version && docker --version
```
## Backend
### Test suite (hermetic, no network, no Docker)
```bash
cd backend
make test # or: ./scripts/test.sh
make test ARGS='tests/data -q' # pass pytest args through
```
Verified: **429 passed, 8 skipped in 11.57s** on a cold venv build.
The first run builds `.venv-test/` on Python 3.12 and installs
`requirements-dev.txt`. `tests/conftest.py` is what makes the suite hermetic:
- no real Open-Meteo, Nominatim or GeoNames calls (the places index is marked
as already-loaded so no background download starts);
- a throwaway SQLite accounts DB and derived store in `/tmp`, never the repo's
`data/`;
- audit/error/access/activity/heartbeat log dirs redirected to `/tmp`;
- notifier and heartbeat threads disabled;
- `THERMOGRAPH_FRONTEND_BASE_INTERNAL` pointed at an unreachable placeholder
(the app fails loud at import without it).
### Boot it locally, no Docker, no Postgres
The backend falls back to SQLite when `THERMOGRAPH_DATABASE_URL` isn't a
Postgres URL, so this is enough:
```bash
cd backend
THERMOGRAPH_FRONTEND_BASE_INTERNAL=http://127.0.0.1:8080 \
THERMOGRAPH_BASE=/thermograph \
THERMOGRAPH_ENABLE_NOTIFIER=0 \
THERMOGRAPH_ENABLE_HEARTBEAT=0 \
.venv-test/bin/python -m uvicorn app:app --host 127.0.0.1 --port 8137
```
Verified responses:
```
GET /healthz → {"status":"ok","role":"all"}
GET /thermograph/api/version → {"backend_version":"2","min_frontend":"1","payload_ver":"p2"}
```
`app:app` is a one-line re-export shim for `web/app.py`. Keep it — systemd, CI
and the container entrypoint all target that name.
Note the port: **8137** is the backend everywhere in this project (compose,
Caddy, the smoke harness at 18137, the frontend's default internal base).
### Image smoke test
```bash
cd backend
make smoke # builds the image, boots it + a throwaway TimescaleDB (tmpfs),
# asserts /healthz and /api/version
```
Uses `docker-compose.test.yml` on host port 18137 so it can't collide with a
dev server on 8137.
## Frontend
**The live frontend is Go.** `frontend/server/` is what builds, tests, ships
and runs. The Python files one level up (`app.py`, `content.py`,
`api_client.py`, `format.py`) are the superseded original — see
[traps](11-traps.md).
### Test
```bash
cd frontend/server
go build ./... && go vet ./... && go test ./...
```
Verified: all seven packages `ok` (`config`, `content`, `contentapi`,
`contentdata`, `format`, `handlers`, `render`).
These same commands run inside `frontend/Dockerfile`'s builder stage — plus a
`gofmt -l` check that must come back empty — so **a failing Go test fails the
image build**, which is how CI catches it. There is no separate frontend test
step in the workflows.
### Run it
The process resolves `static/` and `content/` **relative to its working
directory**, so build in `server/` and run from `frontend/`:
```bash
cd frontend/server && go build -o thermograph-frontend .
cd ..
THERMOGRAPH_API_BASE_INTERNAL=http://127.0.0.1:8137 \
THERMOGRAPH_BASE=/thermograph \
PORT=8080 \
./server/thermograph-frontend
```
Verified: `GET /healthz` → `{"status":"ok"}`, with a structured JSON log line
per request.
`THERMOGRAPH_API_BASE_INTERNAL` is **required** — boot fails loudly without it,
by design. Optional: `THERMOGRAPH_BASE` (default `/thermograph`; the image sets
`/`), `THERMOGRAPH_API_VERSION` (default `v2` — only ever change it per the
[API-version contract](06-contracts.md)), `THERMOGRAPH_API_BASE_PUBLIC`,
`THERMOGRAPH_SSR_CACHE_TTL` (seconds, default 600), `THERMOGRAPH_GOOGLE_VERIFY`,
`THERMOGRAPH_BING_VERIFY`, `PORT` (default 8080).
### Frontend against a real backend container
```bash
cd frontend
make backend-up # pulls + runs the published backend image + throwaway db
# on 127.0.0.1:18137, waits for /healthz, prints the URL
make backend-down
```
`make test-integration` runs the Python integration tier against that. CI does
**not** run it — it needs a live backend container.
## The daemon
```bash
cd backend/daemon
go build ./... && go vet ./... && go test ./...
THERMOGRAPH_INTERNAL_TOKEN=dev-token \
THERMOGRAPH_API_BASE_INTERNAL=http://localhost:8137 \
go run .
```
It refuses to start without `THERMOGRAPH_INTERNAL_TOKEN` — and the backend
answers `404` on the whole `/internal/*` surface when that token is unset. Both
ends fail closed. With Discord unconfigured it logs once and runs cron-only.
## The full stack, locally
```bash
cd infra
make dev-up # docker-compose.yml + docker-compose.dev.yml overlay:
# uncapped CPU, backend published on 0.0.0.0:8137 for the LAN
make dev-down
```
The dev overlay exports `COMPOSE_PROJECT_NAME=thermograph-dev` so the LAN stack
keeps volumes separate from anything else. **Do not remove either half of the
project-name pinning** — `infra/docker-compose.yml` pins `name: thermograph`,
and without it running compose from `infra/` derives the project name `infra`,
silently creating a *new* stack beside the running one with fresh volumes.
Other `infra/Makefile` targets: `up`/`down` (pull + run the published images),
`db-up`/`db-down` (just Postgres, e.g. to run the app from a venv against it),
`om-up`/`om-down`/`om-backfill` (the self-hosted Open-Meteo overlay — the
backfill writes ~1–1.5 TB and takes hours).
## Connectors (Centralis and friends)
The fleet is not reachable from a laptop off the WireGuard mesh, and Forgejo is
mesh-only. **Centralis** is the control plane that fronts all of it — the app
database, the ERA5 lake, fleet logs, Grafana, Forgejo, docs and notes, and
Discord.
```bash
claude mcp add --transport http centralis https://mcp.thermograph.org/mcp \
--header "Authorization: Bearer $CENTRALIS_TOKEN"
```
Ask the operator for a token; don't share it. Verify with *"what's running on
prod right now?"* — it should call `fleet_status` and list the Swarm services.
Also worth installing locally: **Chrome DevTools MCP** (design verification
needs a real browser on your machine) and **Figma**. Grafana's official MCP
server is optional and read-only — but remember dashboards are provisioned from
repo JSON, so a durable change is still a PR via `dashboard_write`.
Run `mcp__centralis__onboarding` for the current, authoritative connector list;
it will be fresher than this page.
## Repo-local guardrails
`.claude/settings.json` wires three hooks that travel with the checkout:
| Hook | When | What |
|---|---|---|
| `prod-guard.sh` | before Bash / live-host MCP calls | Classifies by **allowlist**: only positively-recognised read-only commands pass; everything else asks. Beta is guarded as strictly as prod — it hosts Forgejo, Grafana *and* beta.thermograph.org. |
| `secrets-guard.sh` | before Write/Edit | Denies any direct write to `infra/deploy/secrets/*.yaml`. Use `sops edit`. |
| `lint-after-edit.sh` | after Write/Edit | shellchecks an edited `*.sh` and feeds findings straight back. Exits quietly if shellcheck is missing. |
They enforce what `CLAUDE.md` can only ask for. If you change `prod-guard.sh`'s
classifier, re-read `.claude/hooks/README.md` first — it documents a real
silent-total-bypass failure mode in the parsing loop.
Next: [Repo map](03-repo-map.md).

189
docs/onboarding/03-repo-map.md Normal file
View file

@ -0,0 +1,189 @@
# 3. Repo map
The monorepo was reunified on **2026-07-22** by subtree-merging four split
repos with full history (380 commits). The split's *properties* were kept
deliberately: per-domain images, per-domain deploys, and an async FE/BE
contract. That's why the layout looks like four projects sharing a root.
```
thermograph/
├── CLAUDE.md root agent instructions — branch model, deploy contract
├── CUTOVER-NOTES.md what is and isn't live; the reunification record
├── README.md domain table + how CI stays decoupled
├── .forgejo/workflows/ ALL CI. Nine files. Never add workflows elsewhere.
├── .claude/ settings.json + enforcement hooks
├── backend/ FastAPI API + Go daemon → emi/thermograph/backend
├── frontend/ Go SSR + static assets → emi/thermograph/frontend
├── infra/ compose, Swarm, deploy, secrets, terraform, ops
├── observability/ Loki + Grafana + Alloy
└── docs/onboarding/ you are here
```
> **CI lives only in root `.forgejo/workflows/`**, path-filtered per domain.
> Forgejo only reads root workflows — a copy under a domain's own `.forgejo/`
> is inert, and becomes a trap when someone edits it expecting an effect.
## `backend/` — the API service
Python 3.12 / FastAPI. Owns climate data, grading, the API, accounts,
notifications, and the SSR-content JSON API the frontend renders from. Serves
**no** HTML/CSS/JS.
```
app.py one-line re-export shim for web/app.py — keeps `app:app`
stable for systemd/CI/entrypoint. Don't change the name.
paths.py every filesystem location, resolved from the repo root.
NEVER reintroduce __file__-relative paths in a module.
web/app.py (1022) the real FastAPI app: routes, CORS, ETag, middleware,
lifespan, role gating, the frontend proxy fallback
api/
payloads.py (342) pure inputs → response dict builders. One definition per
payload shape; PAYLOAD_VER lives here.
content_payloads.py the SSR content API's builders (JSON-safe, no Markup)
content_routes.py route wiring for /content/* + the same ETag pattern
internal_routes.py /internal/* — the Go daemon's control surface. Fails
closed: no token configured ⇒ the whole surface 404s.
homepage.py (367) the precomputed "unusual right now" feed
sitemap.py
data/
climate.py (1248) fetch + cache the daily record. The source ladder lives
here. The single biggest file in the repo.
grading.py (532) day-of-year climatology, percentiles, TEMP/RAIN_BANDS
grid.py (92) lat/lon → stable ~4 sq mi cell id
store.py (329) derived-payload store (SQLite/Postgres). Pure accelerator.
climate_store.py(353) raw climate record on TimescaleDB hypertables
era5lake.py (173) the ERA5 object-storage lake: layout, grid math, clients
scoring.py (293) climate-shift scoring (recent years vs full baseline)
places.py (355) typo-tolerant local place index for /suggest
cities.py, city_events.py, meteostat.py
core/
metrics.py (805) since-start traffic counters, multi-worker via SQLite
audit.py (167) JSONL audit / errors / access / activity / heartbeat
singleton.py (129) leader election: flock (per host) or PG advisory lock
(per cluster). Read this before adding any timer.
accounts/ fastapi-users: models, schemas, db, users, api_accounts
notifications/
notify.py (542) the subscription evaluation engine (background thread)
push.py, mailer.py, digest.py, discord.py, discord_link.py,
discord_interactions.py
daemon/ the Go daemon — Discord gateway + cron timers
alembic/ Postgres migrations (run by the entrypoint on boot)
deploy/entrypoint.sh container entrypoint: alembic migrate, then serve
lake_app.py (204) the prod-only `lake` service (DuckDB over the bucket)
tests/ the hermetic pytest suite
cities.json, cities_flavor.json bundled reference data (generated)
```
Standalone scripts at the root are tooling, not runtime: `warm_cities.py`,
`indexnow.py`, `seed_era5.py`, `gen_era5_lake.py`, `gen_cities.py`,
`gen_flavor.py`, `drift_check.py`, `migrate*.py`.
Deep dive: [04-backend.md](04-backend.md).
## `frontend/` — SSR + static assets
```
server/ ★ THE LIVE SERVICE (Go 1.26, module thermograph/frontend)
main.go config + mux + static + graceful shutdown
internal/config/ every env var, with the same names/defaults as before
internal/contentapi/ backend /content/* client: TTL cache, bounded LRU,
per-key single-flight, origin forwarding
internal/content/ page handlers, funcmap, SEO
internal/contentdata/ glossary.yaml + pages.yaml loader (fail-loud)
internal/format/ unit-aware °C/°F formatting + band names
internal/handlers/ routing, SPA shells, static
internal/render/ html/template over embed.FS + ETag helpers
internal/render/templates/*.tmpl the embedded page templates
static/ every static asset — app.js, cache.js, shared.js,
account.js, calendar/day/score/compare .js + .html
shells, style.css (all design tokens), icons, sw.js
content/ glossary.yaml, pages.yaml — structured SSR copy
templates/*.html.j2 Jinja templates — used by the PYTHON service only
tests/ Python tiers: unit (hermetic, fixtures) + integration
tests/fixtures/*.json golden payloads — ALSO consumed by the Go tests
tools/shoot.py the `make shots` screenshot sweep
DESIGN.md the visual system: tokens, components, breakpoints
app.py, content.py, ⚠ the superseded Python implementation — not deployed
api_client.py, (see traps)
format.py, paths.py,
content_loader.py
```
Deep dive: [05-frontend.md](05-frontend.md).
## `infra/` — how and where images run
```
docker-compose.yml the compose stack (beta + LAN dev):
db, backend, lake, daemon, frontend.
`name: thermograph` is PINNED — see below.
docker-compose.dev.yml LAN overlay: uncapped, backend on 0.0.0.0:8137
docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only)
Makefile compose orchestration only
deploy/
deploy.sh ★ the single entry point for beta and prod
render-secrets.sh renders /etc/thermograph.env from the SOPS vault
secrets/ the vault: common.yaml, prod/beta/dev.yaml,
centralis.prod.yaml, example.yaml
stack/ the SWARM path (live on prod):
thermograph-stack.yml, deploy-stack.sh,
autoscale.sh, the LB
swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning
Caddyfile the reverse proxy (path-splits FE vs BE)
provision-*.sh host bootstrap: agent access, dev LAN, mail, secrets
db/init/ TimescaleDB init + tuning
migrations/ hand-run SQL migrations
ops/
dbq.sh read-only psql into any environment's db container
iceberg.sh duckdb over the ERA5 Iceberg lake
terraform/ host provisioning — NO STATE IS PERSISTED
ACCESS.md, DEPLOY.md, DEPLOY-DEV.md (partly stale — see traps)
```
**Why the project name is pinned.** Compose creates volumes
`thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stack declares
those exact names as `external: true` at the same mount paths. Running compose
from `infra/` without `name: thermograph` derives project `infra`, which makes a
whole new stack with empty volumes next to the running one. `deploy-dev.sh`
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
keep LAN dev separate on purpose. Keep both halves.
Deep dive: [08-infra-secrets.md](08-infra-secrets.md).
## `observability/` — the logging stack
```
docker-compose.yml Loki + Grafana (runs on beta)
loki/config.yml mesh-only, filesystem storage, 30-day retention
grafana/provisioning/ datasource + dashboard provider (auto-loaded)
grafana/provisioning/alerting/ rules, the Discord contact point, the policy
grafana/dashboards/*.json the fleet-logs dashboard
alloy/config.alloy the per-node shipper
alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host)
caddy-grafana.conf beta's Grafana vhost, kept for reference
```
No build, no deploy automation — it ships by hand. **The repo is the only
durable path**: Grafana is provisioned from this directory at startup, so UI
edits are overwritten. Grafana is at **`dashboard.thermograph.org`** (Google
SSO, pre-provisioned users only) — never `grafana.thermograph.org`.
Deep dive: [09-observability.md](09-observability.md).
## The nine CI workflows
| Workflow | Trigger | Does |
|---|---|---|
| `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. |
| `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. |
| `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. |
| `deploy.yml` | push to `main`/`release` touching an app domain | Branch selects environment; matrix covers services; SSHes and runs `deploy.sh`. |
| `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards each host's checkout and re-renders secrets. Rolls **no** service. |
| `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. |
| `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow. Uses `PROD_SSH_*`. |
| `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. |
| `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. |
Deep dive: [07-ci-and-release.md](07-ci-and-release.md).
Next: [Backend deep dive](04-backend.md).

292
docs/onboarding/04-backend.md Normal file
View file

@ -0,0 +1,292 @@
# 4. Backend deep dive
FastAPI on Python 3.12, plus a Go daemon that ships in the same image. Owns
every piece of climate data and every bit of grading logic in the project.
Read [`backend/CLAUDE.md`](../../backend/CLAUDE.md) alongside this — it is the
binding, terse version.
## The one-process-many-roles model
The backend image runs **five different jobs** depending on `THERMOGRAPH_ROLE`
and which binary compose/Swarm selects. One image means version skew between
them is impossible.
| Service | How it's selected | Does |
|---|---|---|
| **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. |
| **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. |
| **all** | default | Both — the single-process dev default. |
| **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Prod only. |
| **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. |
`deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres
volume can still be starting) and then serves. `/healthz` and `/api/version`
are deliberately **I/O-free** so they stay cheap under tight healthcheck
intervals; they're also deliberately **not** under `THERMOGRAPH_BASE`, so a
healthcheck needn't know the base path.
## The data pipeline
### Where history comes from — the source ladder
`data/climate.py::_load_history` walks this ladder on a cache miss, and the
order matters:
1. **The ERA5 lake** (`data/era5lake.py`) — our own parquet extract of the
public ERA5 archive in Contabo object storage. True ERA5 with measured
gusts, no third-party API in the path. Unconfigured or missing-point cells
fall through at zero cost.
2. **The Open-Meteo archive** — the same ERA5 family, so a backup-sourced
record grades consistently with lake-sourced neighbours. Guarded by its own
rate-limit cooldown.
3. **Stale cache**, served without rewriting it (so `recent_stamp` stays old
and the next request retries upstream first).
4. Otherwise `WeatherUnavailable` → a clean 503.
**NASA POWER is retired from every serving path.** Its MERRA-2 record ran
~1.3 °F off ERA5 and carried no gusts. `drift_check.py` still fetches it, and
only that.
Every source must return a plausibly-full span (`MIN_ARCHIVE_DAYS`) before it's
accepted and cached as complete — a short response is rejected, not stored.
### Where "today" comes from
`_load_recent_forecast` is a separate ladder:
1. **Open-Meteo forecast API** — recent past *and* forward days in one call,
with gusts, ERA5-consistent with the history record. Skipped while its own
cooldown is active.
2. **Fallback** — an archive recent-past range plus **MET Norway** forward days.
Costs no Open-Meteo forecast quota.
3. Stale cache, again without rewriting.
Cached per cell for `FORECAST_TTL_HOURS`, so it tracks model updates without
hourly upstream I/O. `RECENT_PAST_DAYS = 25`, `FORECAST_DAYS = 8`.
### Concurrency and quota guards
- A per-cell lock serialises archive fetches, so N concurrent first-requests for
one cell cost one fetch.
- Rate limits set module-level cooldowns (`_archive_cooldown_until`,
`_forecast_cooldown_until`) rather than retrying into the wall.
- Reverse geocoding runs on **one dedicated worker thread** draining a queue at
~1 req/s. The earlier design slept inside the caller's thread — which, in the
server, is one of Starlette's shared sync-threadpool threads.
## Storage: three tiers with three different contracts
| Tier | Module | Backend | Contract |
|---|---|---|---|
| **Raw climate record** | `data/climate_store.py` | TimescaleDB `climate_history` hypertable (durable), `climate_recent` (rewritten each refresh), `climate_sync` (freshness) | **Source of truth.** Expensive to refetch. Fail-soft: a miss degrades to *fetch-from-upstream*. Parquet files instead when `THERMOGRAPH_DATABASE_URL` isn't Postgres (dev, tests, offline tooling). |
| **Derived payloads** | `data/store.py` | SQLite WAL, or **UNLOGGED** Postgres tables | **Pure accelerator.** Every reader falls back to recomputing from the raw record; every helper swallows its own errors. Deleting the store is a safe full reset. |
| **Accounts** | `accounts/db.py` | Postgres (or SQLite) via SQLAlchemy | **Not regenerable.** Foreign keys enforced, errors surface rather than get swallowed, must be backed up. |
That third row is the important distinction: accounts, subscriptions and
notifications have no source to recompute from, so they live under stricter
rules than everything else in the process.
On Postgres, `accounts/db.py` runs **two async engines** over one primary —
read-only (`default_transaction_read_only`) for pure-GET endpoints, read-write
for everything else — plus a separate **sync** psycopg engine for the notifier
thread, which has no event loop.
### The validity token
Derived rows are keyed `(kind, cell_id, request_key)` and guarded by a `token`
that encodes everything the payload depends on:
```python
PAYLOAD_VER = "p2" # api/payloads.py
history_token(history) → f"{PAYLOAD_VER}:{hist_end(history)}"
content_token(...) → f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date}"
```
A stored token that doesn't match the caller's current token is a **miss**.
There is no clock-based expiry anywhere. That's why one `PAYLOAD_VER` bump
atomically orphans every pre-upgrade row — and why forgetting to bump it serves
old-shaped JSON forever. `CONTENT_VER` is kept separate so a content-only shape
change doesn't invalidate the whole grading cache.
The same token doubles as a **weak ETag**. See [contracts](06-contracts.md) for
what that means for CORS.
## Grading
`data/grading.py` is the heart of the product:
- For a target day of year, the reference distribution is every historical day
within **±`HALF_WINDOW` (7) days**, wrapping the year boundary — a 15-day
seasonal window.
- The observed value is placed as an **empirical mid-rank percentile**, which
handles ties correctly (crucial: most days have zero precipitation).
- `TEMP_BANDS` is a symmetric seven-tier ladder; `RAIN_BANDS` is separate, and
dry days are coloured by dry-streak length instead of rank.
- `pct_ordinal()` floors a percentile into `1..99` — never 0, never 100.
`frontend/static/shared.js::pctOrd()` mirrors it. If they diverge, the same
reading says two different things on two surfaces.
`data/scoring.py` is a different question: how far a cell's **recent years**
have drifted from its full multi-decade baseline, per metric and per
percentile category, expressed in unit-free percentile points. Note
`baseline_overlaps_recent` in the payload — the baseline deliberately includes
the recent years, which mildly and uniformly attenuates the divergence.
## The HTTP surface
`web/app.py` mounts routers so that every version stays served simultaneously:
```
/api → v1 (legacy unversioned alias)
/api/v1 → v1 geocode, grade
/api/v2 → v2 geocode, suggest, place, grade, calendar, day,
score, forecast, cell, metrics, event
/api/v2 → content_routes: /content/hub, /content/sitemap,
/content/indexnow-key, /content/home,
/content/city/{slug}[/month/{month}|/records]
/api/v2/auth, /users fastapi-users: login, logout, register, verify, me
/api/v2/subscriptions, /notifications, /push/* accounts/api_accounts.py
/api/v2/discord/* account linking (OAuth2)
/internal/* the Go daemon's control surface — NOT under BASE
/healthz, /api/version I/O-free
/discord/interactions Discord slash-command webhook (Ed25519-verified)
/digest the footer signup form
/{path:path} catch-all proxy to the frontend ← registered LAST
```
Two routes worth understanding properly:
**`GET /api/v2/cell`** — one bundle carrying every view's payload for a cell, so
the client warms four views with one request. Each slice is the *exact* payload
its per-view endpoint returns, built by the same builders under the same
derived-store keys and tokens, paired with the ETag that endpoint would emit.
The client seeds its per-view cache from the slices and later revalidates each
view individually. `prefetch=1` is the warm-only mode with a hard guarantee:
**it never spends weather-API quota** — a cold cell answers `204` with no body.
`neighbors=1` additionally enqueues the 8 surrounding cells for background
warming, also warm-only.
**The catch-all proxy** — the frontend owns every page and asset. In prod and
beta, Caddy path-splits directly, so this proxy is never exercised. It exists as
the fallback for environments with no Caddy in front (LAN dev, bare-metal). It
forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute
URLs instead of resolving to the internal hop's own address. Because it's
registered dead last, every real backend route wins first —
`/internal/*` in particular is registered before it, deliberately.
## Background work: who runs what, and where
This is where the project's hardest-won lessons live.
- **Neighbour warmer** — a per-worker daemon thread draining a request-fed
queue. Correct to run in every worker: each drains its own queue.
- **Subscription notifier** (`notifications/notify.py`) — a timer-driven sweep
that issues upstream fetches independent of any request. Running it in N
workers multiplies Open-Meteo quota use N-fold. That is not hypothetical: it
is the documented cause of overnight 429/503s after prod went to 3 workers.
So it runs behind `singleton.claim_leader()`:
- `claim()` — a non-blocking `flock` on a lockfile, arbitrating workers **on
one machine**;
- `claim_pg()` — a Postgres advisory lock, visible to every host talking to
the same database, arbitrating **cluster-wide** under Swarm;
- unset (dev, tests, single worker) ⇒ always the leader.
- **Heartbeat** — deliberately **unguarded** across every worker and replica. A
beat is one local file append (no quota, no DB, no lock), so duplicates are
free, and only the process actually being dead stops them. Container-log
silence can't distinguish a live web/worker process from a dead one, so this
is the signal observability alerts on.
- **Discord gateway + recurring jobs** — moved *out* of this process entirely
into the Go daemon. See below.
The notifier's own quiet guards: a `UNIQUE(subscription_id, event_date, metric,
direction, kind)` row-level dedup, and a **weekly cap** — a subscription that
notified within 7 days is skipped entirely. Archive reads come from cache; a
freshly-subscribed cell is fetched **once**, budget-capped per pass.
## The Go daemon (`backend/daemon/`)
One binary, built into the backend image, running as exactly one replica.
**Why it exists:** the Discord gateway tolerates one session per token, and
duplicated timers repeat every job. One replica makes that a *deployment fact*
instead of a runtime election.
**Why it's thin:** Go owns no climate or grading logic. Grading needs polars and
the cache, and the slash-command path deliberately shares one grade builder with
the API so bot grades and API grades can never drift. So anything data-shaped is
a POST back into Python:
| Route | Purpose |
|---|---|
| `POST /internal/discord/grade` | `{"query": "phoenix"}` → gateway-ready Discord message JSON, relayed verbatim |
| `POST /internal/jobs/warm-cities` | trigger the warm-cities job |
| `POST /internal/jobs/indexnow` | trigger IndexNow check-and-submit |
Every request carries `X-Thermograph-Internal-Token`. **Both ends fail closed**:
with no `THERMOGRAPH_INTERNAL_TOKEN`, Python answers 404 across the whole
surface and the daemon refuses to start. Caddy never routes `/internal/*`
publicly, so the token is defence in depth, not the only control.
The cron half runs unconditionally (`warm-cities` every 24h, `indexnow` every
6h, both configurable). A fatal gateway error logs loudly and leaves cron
running — a Discord misconfiguration must not become a crash loop that also
stops the schedule. The cron scheduler defers the first run by one interval,
never overlaps a job with itself, and drops ticks that fire mid-run.
## The lake (`lake_app.py` + `data/era5lake.py`)
Prod-only Swarm service, same image, `THERMOGRAPH_ROLE=lake`.
The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`),
extracted once from the public Earthmover ERA5 Icechunk archive by
`gen_era5_lake.py`. Two layouts under `era5/`:
- `daily/tile=<ti>_<tj>/year=<Y>/month=<M>/part.parquet` — the analytical hive
table. The location grain is a 12×12-point tile (~3°×3°) aligned to the
source's chunk grid; a per-point-per-month grain would be ~170M objects.
- `points/lat=<i>/lon=<j>.parquet` — the serving projection: one file per grid
point with its full history, so the hot path is a single GET.
- `manifest.parquet` — an index of every point file; readers prune with it and
the extractor resumes from it.
Endpoints: `/healthz`, `GET /history?lat=&lon=` (the hot path — nearest point's
history as parquet bytes, disk-cached per point), `POST /query` (SELECT-only
DuckDB SQL over `era5_daily` and `manifest`).
Two operational facts: **Contabo is path-style only** with `region=default`
(virtual-host addressing fails outright), and DuckDB's `httpfs`/`iceberg`
extensions are baked into the image **as uid 10001**, because extensions install
to the invoking user's `~/.duckdb` and a root-time bake left the runtime user
unable to `LOAD` them — seen live as prod's lake `/query` returning 500.
Filter on partition columns (`tile`/`year`/`month`). That predicate is the
difference between a pruned scan and reading the whole warehouse.
## Observability hooks inside the app
- `core/audit.py` writes JSONL streams: `audit` (one line per graded run, with
per-phase timings and a `full`/`partial` run type), `errors` (upstream
failures tagged `retry` or `error`), plus access, activity and heartbeat.
Alloy parses these so `level`/`tag`/`phase` become Loki labels.
- `core/metrics.py` keeps since-start counters, shared across uvicorn workers
via `THERMOGRAPH_METRICS_DB` (a SQLite file) — unset keeps the
zero-dependency in-memory store. Read over the token-gated
`GET /api/v2/metrics`.
- Every unhandled 500 gets an explicit exception handler emitting
`tag="http.error"`. Without it the exception propagates past the middleware
and the 500 is invisible in both the access log and metrics.
## Working on it
```bash
cd backend
make test # hermetic pytest, ~12s
make test ARGS='tests/data/test_grading.py -q'
make smoke # build the image, boot it + a throwaway db
```
CI runs the suite **inside the built image**, so the exact interpreter and
dependencies that ship are what gets tested.
Next: [Frontend deep dive](05-frontend.md).

221
docs/onboarding/05-frontend.md Normal file
View file

@ -0,0 +1,221 @@
# 5. Frontend deep dive
The **SSR + static-asset service**: server-rendered crawlable pages, the
interactive tool's SPA shells, and every static asset. No climate data, no
database, no compute — everything comes from the backend's content API over
HTTP.
> **The live service is Go.** `frontend/server/` builds, tests, ships and runs.
> The Python files one directory up (`app.py`, `content.py`, `api_client.py`,
> `format.py`, `content_loader.py`, `paths.py`, `templates/*.html.j2`) are the
> superseded original implementation, kept in-tree as the reference the port
> was made from. `frontend/CLAUDE.md` and `frontend/README.md` still describe
> the Python service — see [traps](11-traps.md). Everything in those docs about
> **contracts, env vars and design** is still correct; only "which language and
> which file" changed.
## Shape of the Go service
```
main.go config → content client → glossary/pages → mux →
listen → graceful shutdown
internal/config/ every env var the service reads, with the same names,
defaults and required/optional split as the Python
internal/contentapi/ the backend /content/* client
internal/content/ the SSR page handlers, template funcmap, SEO helpers
internal/contentdata/ glossary.yaml / pages.yaml loader
internal/format/ unit-aware °C/°F formatting + band names
internal/handlers/ SPA shells + static serving
internal/render/ html/template over an embed.FS + ETag helpers
internal/render/templates/*.tmpl the page templates (embedded in the binary)
```
Dependencies: **stdlib plus `gopkg.in/yaml.v3`** only. The committed SSR copy
in `frontend/content/*.yaml` is shared with the rest of the repo and uses
block/folded scalars, so a real YAML parser is genuinely required.
**Templates are embedded in the binary; `static/` and `content/` are read from
disk relative to the working directory** (`frontend/` locally, `/app` in the
image). That's why you build in `server/` and run from `frontend/`.
### Fail-loud boot
`main.go` exits at startup on bad configuration *or* bad content: a missing
backend URL, a garbage TTL, malformed glossary/pages YAML. The Python raised at
import for the same cases, and the reasoning is identical — a bad content edit
should break the boot, not silently 500 on the first request.
The one deliberate exception is the IndexNow key: boot tries the key fetch once,
catches failure, and falls back to a lazy per-request lookup. **Boot must
survive an unreachable backend** — asynchronous FE/BE deploys depend on it.
Don't make it fatal.
## Routes
Everything below sits under `THERMOGRAPH_BASE` (default `/thermograph`; the
image sets `/`).
| Path | Kind |
|---|---|
| `/` | SSR homepage |
| `/climate` | SSR climate hub |
| `/climate/{slug}` | SSR per-city page |
| `/climate/{slug}/records` | SSR all-time records |
| `/climate/{slug}/{month}` | SSR per-month page |
| `/glossary`, `/glossary/{term}` | SSR |
| `/about`, `/privacy` | SSR |
| `/robots.txt`, `/sitemap.xml` | generated |
| `/{indexnow-key}.txt` | the IndexNow ownership file |
| `/calendar`, `/day`, `/score`, `/compare`, `/legend`, `/alerts` | SPA shells |
| `/healthz` | liveness, I/O-free (does **not** prove the backend is reachable) |
| everything else | static assets |
The static mount is registered last so the explicit routes win.
**The backend owns the page metadata, not this service.** `page_title`,
`canonical_path`, `breadcrumb` and `jsonld` come from
`backend/api/content_payloads.py`. Don't recompute them here.
### SPA shells and `__ORIGIN__`
The shell HTML files in `static/` carry an `__ORIGIN__` placeholder for the
link-preview/Open Graph tags — preview crawlers need absolute URLs and the host
differs between LAN and prod. The substituted HTML and its ETag are **memoized
per origin**, not recomputed per request.
Origin resolution prefers `X-Forwarded-Host` over `Host`. That matters when the
request arrives through the backend's catch-all proxy fallback: without it, the
page would advertise the internal hop's own address as canonical.
## The content client
`internal/contentapi` is a careful piece of code and worth reading in full
before you change caching behaviour. It has four properties, all load-bearing:
1. **TTL cache** (`THERMOGRAPH_SSR_CACHE_TTL`, default 600s). Climate data
changes at most hourly, so without this a page-view burst on one city costs
one backend round trip per request. This is *in addition to* the backend's
derived-store/ETag caching — this one saves the network hop, the backend's
saves the recompute.
2. **Bounded LRU** (2048 entries). `city()` and `city_records()` fold the
browser-facing `origin` into the cache key, and `origin` is client-controlled
via `Host`/`X-Forwarded-Host`. An unbounded map there is a cheap
memory-exhaustion vector.
3. **Per-key single-flight.** Without it, N concurrent requests missing the same
key (cold start, or right after TTL expiry on a hot city) each fire their own
backend call — a thundering herd.
4. **Origin forwarding.** The original browser-facing origin is passed through as
`Host` / `X-Forwarded-Proto` so the backend builds JSON-LD `url` fields
against the public origin rather than the internal `http://backend:8137`
address.
## Static assets
Hand-written, no build step, no bundler, no framework.
| File | Role |
|---|---|
| `app.js` | the main tool: map, search, results, inline SVG chart |
| `calendar.js` / `day.js` / `score.js` / `compare.js` | the SPA views |
| `cache.js` | IndexedDB cache + the `/cell` bundle prefetch + `If-None-Match` revalidation |
| `account.js` | auth; **pins `API_VERSION` and the `uv(path)` helper** |
| `shared.js` | shared helpers; **`pctOrd()` mirrors the backend's `pct_ordinal()`** |
| `units.js` | `F_REGIONS` — must match the backend's `F_COUNTRIES` |
| `chart.js`, `mappicker.js`, `filtersheet.js`, `nav.js`, `push-client.js`, `ios-install.js`, `digest.js`, `climate.js`, `subscriptions.js` | the rest |
| `style.css` | **the single hand-written stylesheet — every design token lives here** |
| `sw.js`, `manifest.webmanifest`, icons | PWA |
`cache.js` is where the ETag contract actually lands: it fetches
`/api/v2/cell` once per view-set, slices it, and later revalidates each view
with `If-None-Match`. The slice→view map must track the backend's payload
shape. See [contracts](06-contracts.md).
Static responses are served with `Cache-Control: public, max-age=300` — short
on purpose, because asset filenames are **not** content-hashed. It's a
revalidation window, not an immutable cache.
## The design system
[`frontend/DESIGN.md`](../../frontend/DESIGN.md) is the source of truth and
wins over any general design guidance. The essentials:
- **Never hardcode a colour.** Every colour is a CSS custom property in
`style.css`'s `:root`, with a `@media (prefers-color-scheme: light)` override
right below. Always `var(--token)`.
- **Grade palettes encode meaning, not decoration.** Temperature is a 9-step
diverging, colourblind-safe scale (`--rec-cold` … `--normal` green midpoint …
`--rec-hot`). Precipitation is `--dry` plus `--wet-1…9`. Keep the order and
midpoints intact.
- **Inter is not self-hosted.** It renders where the OS has it and falls back to
`system-ui`. Adding a webfont link is a deliberate decision, not a tweak — it's
a network dependency on every page.
- **Inputs are `font-size: 16px` minimum.** Smaller text makes iOS zoom on
focus. Hard rule.
- **Mobile-first.** The base stylesheet is the phone layout; wider screens layer
on via `min-width`. Primary breakpoint 640/641px. The content column widens in
real steps at 1680 / 2400 / 3400px — it does *not* stay a centred 1200px strip.
- **Metric order is always `Precip · High · Low`**, everywhere.
- **Dark is the default**, light comes from `prefers-color-scheme`. Every change
must look right in both. Honour `prefers-reduced-motion`.
- Charts are bespoke inline SVG. No canvas, no charting library.
### Verifying a visual change
**Validate at 390 / 800 / 1920 / 2560 / 3840px in both light and dark.** These
straddle every breakpoint above.
```bash
cd frontend
.venv/bin/pip install -q -r tools/requirements.txt
.venv/bin/python -m playwright install chromium
.venv/bin/python tools/shoot.py # the full matrix
.venv/bin/python tools/shoot.py index --width 390 --scheme dark # while iterating
```
PNGs land in `.screenshots/` (gitignored) as `{page}@{width}-{scheme}.png`.
Point it elsewhere with `SHOTS_BASE`. Chrome DevTools MCP is the other good
option here — a real browser on your own machine.
## Tests
```bash
cd frontend/server && go build ./... && go vet ./... && go test ./...
```
The Go tests read two directories via a fixed relative path: the committed
golden fixtures (`frontend/tests/fixtures/*.json` — the same set the Python
golden-diff comparison used) and the SSR copy (`frontend/content/*.yaml`).
`frontend/Dockerfile`'s builder stage copies them to container-root paths to
preserve that relationship, and runs `gofmt -l` + `go vet` + `go test` **before**
building. A failing Go test therefore fails the image build, which is how CI
catches it — there is no separate frontend test step in the workflows.
The Python tiers still exist and still run locally:
```bash
cd frontend
make test-unit # hermetic: SSR rendering fed committed fixtures
make test-integration # pulls + runs the real backend image; local only
make capture-fixtures # refresh tests/fixtures/*.json from a live backend
```
Treat `make capture-fixtures` as the shared operation: those fixtures feed the
**Go** tests too.
## The image
`python:3.12-slim` → gone. The frontend image is:
- a `golang:1.26` builder that runs `gofmt -l` / `go vet` / `go test`, then
builds a static `CGO_ENABLED=0 -trimpath -ldflags="-s -w"` binary;
- an `alpine:3.22` final stage with **bash** (the Swarm stack bind-mounts a
bash entrypoint shim over the image's entrypoint) and **curl** (the
healthcheck);
- **uid 10001**, matching the backend image — that's the uid infra provisions
readable secrets for. Don't change it.
- `static/` and `content/` copied to `/app`, read-only at runtime.
There is no `WORKERS` knob any more: uvicorn needed a process count, the Go
server handles concurrency in one process.
Next: [Cross-service contracts](06-contracts.md).

183
docs/onboarding/06-contracts.md Normal file
View file

@ -0,0 +1,183 @@
# 6. Cross-service contracts
**Read this before any change that touches both domains.**
Backend and frontend build separate images and deploy independently. That
independence is the point of the architecture — and it means there is no build
step, no type checker and no linker that will catch a mismatch between them. The
items below are the seams. Each one fails *silently and in production*, which is
why they're enumerated rather than left to judgement.
## 1. API version — `GET /api/version`
```json
{"backend_version": "2", "min_frontend": "1", "payload_ver": "p2"}
```
Driven by `API_CONTRACT_VERSION` and `MIN_SUPPORTED_FRONTEND` in
`backend/web/app.py`.
**The rule for a breaking change:** ship a new `v3` router mounted **alongside**
`v2`, re-registering only the handlers that changed, and bump the constant in
the same PR. `/api` and `/api/v1` stay mounted as aliases. Every version stays
served simultaneously — that is how FE and BE ship out of lockstep at all.
**The frontend pins the version in exactly two places:**
| Where | What |
|---|---|
| `frontend/server/internal/config` (Go) — `THERMOGRAPH_API_VERSION`, default `v2` | every server-side path builder reads it |
| `frontend/static/account.js` (JS) — the exported `API_VERSION` and the `uv(path)` helper | every browser-side call |
Never hardcode `api/v2/...` anywhere else. Bump both in one PR, and **only
after** the target backend's `/api/version` confirms it serves that version and
its `min_frontend` doesn't exclude the one you're leaving.
> The Python `frontend/api_client.py` has a third pin. It isn't deployed, but if
> you're using it locally, keep it in step or you'll debug a phantom.
## 2. `PAYLOAD_VER` — the cache invalidation token
`backend/api/payloads.py`, currently `"p2"`.
This is **separate from URL versioning** and does a different job: it's the
cache/ETag invalidation token. Every derived row is stored against a validity
token built from it, and a stored token that doesn't match the caller's current
token is simply a miss.
**Bump it whenever a response payload's shape changes.** One bump atomically
orphans every pre-upgrade cached row across every environment. Forget it and
the backend serves old-shaped JSON out of cache indefinitely, with no error
anywhere — the new code is live and the old data keeps flowing.
`CONTENT_VER` is deliberately separate so a content-only shape change doesn't
invalidate the whole grading cache.
## 3. ETag / `If-None-Match` — and `expose_headers`
Every graded payload is cached against a validity token that doubles as a weak
ETag. `frontend/static/cache.js` fetches `/api/v2/cell` once per view-set,
slices it, and revalidates each view individually with `If-None-Match`.
**`expose_headers=["ETag"]` in the CORS middleware matters as much as
`allow_origins`.** Browsers hide every response header except a fixed "simple"
allowlist from cross-origin JS. Without `ETag` exposed, `cache.js`'s
`res.headers.get("ETag")` silently returns `null`, 304 revalidation quietly
stops working, and **there is no console error**. You'd see it only as a traffic
bill.
CORS is off by default (`THERMOGRAPH_CORS_ORIGINS` unset), which is today's real
same-origin topology. Never set it to `*` — Starlette itself refuses a
credentialed wildcard.
Don't change the ETag derivation or that header list without checking
`cache.js`. And when the payload shape changes, the **slice → view map** in
`cache.js` has to track it.
## 4. `pct_ordinal()` — mirrored in four places
Floor a percentile into `1..99`. **Never 0, never 100.**
| Copy | File | Guarded by |
|---|---|---|
| canonical | `backend/data/grading.py::pct_ordinal` | its own unit tests |
| Go SSR | `frontend/server/internal/format/format.go::PctOrdinal` | `format_test.go::TestPctOrdinal` (table-driven, not a cross-check) |
| browser | `frontend/static/shared.js::pctOrd` | nothing automated |
| Python SSR *(not deployed)* | `frontend/format.py::pct_ordinal` | the Python unit tier |
Every percentile on every surface goes through one of these. If they diverge,
the same reading says two different things in two places. The backend copy is
canonical; the others follow it.
`TEMP_BANDS` / `RAIN_BANDS` in `grading.py` are the source of truth for tier
names and thresholds. Changing them without the frontend produces tiers drawn
in colours that disagree with the labels the API returns.
## 5. The Fahrenheit country set
Fourteen codes: `US PR GU VI AS MP UM BS BZ KY PW FM MH LR`.
| Copy | File |
|---|---|
| canonical | `backend/api/content_payloads.py::F_COUNTRIES` |
| Go SSR | `frontend/server/internal/format/format.go::FCountries` |
| browser | `frontend/static/units.js::F_REGIONS` |
| Python SSR *(not deployed)* | `frontend/format.py::F_COUNTRIES` |
`format_test.go::TestFCountriesMatchesBackend` re-parses the backend's Python
source and asserts the Go copy matches — a genuinely good guard.
> ⚠️ **Verified gap.** Several source comments claim "a test asserts all three
> stay identical." As of this writing that is not true: no test reads
> `static/units.js`. All four copies currently *are* identical (checked), but
> the browser copy is held in step by convention alone. If you touch this set,
> update `units.js` by hand and diff it yourself. Extending the Go test to
> parse `units.js` too would close this properly and is a good first
> contribution.
## 6. Backend boot must survive an unreachable frontend, and vice versa
Both directions are load-bearing for asynchronous deploys:
- **Frontend → backend:** boot tries the IndexNow-key fetch once, catches
failure, and falls back to a lazy per-request lookup. Don't make it fatal.
- **Backend → frontend:** `THERMOGRAPH_FRONTEND_BASE_INTERNAL` is required at
import (fails loud), but it's only *used* by the catch-all proxy fallback —
an unreachable value doesn't stop the backend serving its own routes. That's
why the test suite and the smoke harness both point it at
`http://127.0.0.1:1`.
`/healthz` on both services is liveness only. Neither proves the other is
reachable, deliberately — readiness is the ingress health-gate's job.
## 7. `THERMOGRAPH_BASE` must match on both processes
Default `/thermograph`; both images set `/`. Both processes must be configured
**identically** in every real deployment — it's how they agree on the shared URL
prefix. The Go frontend's content client reads it too, so a sub-path deployment
works rather than only ever working at the root.
## 8. Image tags are keyed to the domain, not the branch tip
Both `build-push.yml` and `deploy.yml` compute the tag as
`sha-` + the first 12 hex of `git log -1 --format=%H -- <domain>/`.
Two consequences:
- An infra-only commit at the branch tip cannot send a deploy chasing an image
no build produced.
- `deployed_version` can legitimately show a tag that is **not** the head of the
branch. That's correct, not drift.
`fetch-depth: 0` in the deploy workflow exists for exactly this: in a
path-filtered monorepo the tip is often another domain's commit, and a depth-1
clone can't see past it.
## 9. Cookie auth is same-origin today
The interactive tool's auth is an HttpOnly session cookie. `account.js` uses
`credentials: "include"` (not the fetch default) so the cookie still rides along
if the script is ever loaded cross-origin — but the backend must actually be
configured to accept credentialed cross-origin requests (CORS +
`SameSite`) for that to work. In the default same-origin deployment it's a
non-issue. **Don't assume a fully decoupled, independently-hosted frontend
"just works" for logged-in features** without checking that configuration.
## Quick checklist
Before opening a PR that touches both domains:
- [ ] Did any response payload shape change? → bump `PAYLOAD_VER`.
- [ ] Did any route or payload break compatibility? → new `vN` router
alongside, bump `API_CONTRACT_VERSION`, check `min_frontend`.
- [ ] Did I change the API version pin? → both Go config **and**
`static/account.js`, in one PR, after checking the target backend.
- [ ] Did I touch tier names, thresholds, or percentile rendering? → check all
four `pct_ordinal` copies and the colour tokens.
- [ ] Did I touch the Fahrenheit set? → four copies, and `units.js` is
unguarded.
- [ ] Did I change the `/cell` bundle shape? → the slice→view map in
`cache.js`.
- [ ] Did I change ETag derivation or CORS headers? → re-read `cache.js`.
Next: [CI and release](07-ci-and-release.md).

229
docs/onboarding/07-ci-and-release.md Normal file
View file

@ -0,0 +1,229 @@
# 7. CI and release
## The branch model
```
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
│
promotion PR
▼
main ──▶ beta.thermograph.org
│
promotion PR ← the owner's call, always
▼
release ──▶ thermograph.org
```
`dev`, `main` and `release` are protected — direct pushes are blocked for
everyone. Forgejo has **no auto-merge-on-green**, so merges are explicit: either
click "Auto merge when checks succeed" on a PR, or merge via the API once CI is
green.
Promotion is continuous and per-decision. There is no release calendar. Size a
promotion to whatever has actually soaked on the environment below.
### Who may merge what
| Hop | Whose call | Norm |
|---|---|---|
| anything → `dev` | yours | Merge it. An open PR into `dev` is unfinished work, not caution. |
| `dev` → `main` (beta) | yours, batched | Promote when a batch is coherent enough to be worth *testing*. Beta is the test environment; not promoting is not testing. |
| `main` → `release` (prod) | **the owner's** | Prepare it, recommend for or against, and ask. Never unilaterally, never by another route. |
Centralis's `promote(from, to)` implements exactly this, and `forge_pr_merge`
enforces it with no back door: it merges into `dev` and `main`, refuses
`main → release` under any flag, and requires `confirm_protected_base: true`
for the separate hotfix path.
### Two things about branch divergence
**Commit counts are not deliverable content.** Before promoting, compare the two
tips' *trees*. Identical trees mean the same changes are already on the target
under different SHAs — promoting there merges nothing while still firing the
target's deploy workflows, including `infra-sync` re-rendering
`/etc/thermograph.env` on prod and beta. The ahead/behind numbers will look like
real work. `promote` refuses this outright.
**Expect `dev`, `main` and `release` to be mutually divergent, and expect
`--ff-only` to fail.** Every promotion merges the source into the target,
creating a merge commit *on the target* that the source never receives — so each
promotion widens the gap it just closed. That is accumulated design, not drift.
(Reconciling merges back into `dev` do happen; `be4aa94` in the history is one.)
## The nine workflows
All in root `.forgejo/workflows/`. **Never add a workflow under a domain's own
`.forgejo/`** — Forgejo only reads the root ones, so a copy there is inert and
becomes a trap for whoever edits it next.
### `pr-build.yml` — the required check
One workflow, one stable required-check name (**`gate`**), any number of
domains:
1. a `changes` job diffs the PR against its base (needs `fetch-depth: 0` — the
base commit isn't reachable from a depth-1 clone);
2. per-domain jobs run only for domains the PR touches;
3. `gate` reduces their results into the single status branch protection
requires.
**Deliberately not path-filtered at the workflow level.** A path-filtered
required check that never fires leaves a PR stuck on *"expected — waiting for
status"* forever, so auto-merge never proceeds. An always-running gate whose
domain jobs skip cleanly gives the same compute savings without the deadlock.
This is the one intentionally *coupled* piece of the CI design.
### `build.yml` — the reusable build check
Called via `uses:` with a `domain` input. The build **context is that domain's
subdirectory**, so each Dockerfile sees exactly the tree it saw when its domain
was a repo root.
For **backend** it then runs the hermetic pytest suite *inside the image it just
built* — the exact interpreter and dependencies that ship. Three flags there are
load-bearing: `-u 0` (the image's default user can't write to site-packages),
`--entrypoint sh` (otherwise the test command is swallowed as entrypoint args
and the container just boots the server forever), and `unset THERMOGRAPH_BASE`
(the image bakes `/`, which moves every route off the `/thermograph` prefix the
tests are written against).
For **frontend** there's no separate test step: it's a Go binary with no
interpreter in the runtime image, and `gofmt`/`vet`/`go test` already run in the
Dockerfile's builder stage — so a failing Go test fails the Build step above.
It is deliberately **not** a live boot+healthz check. That was tried in the split
repos and repeatedly hit the runner's own environment quirks (bridge-IP
reachability, non-unique `$$`, squatted host ports) without ever going reliably
green. Image boot is verified by hand instead — `make smoke`, or `docker run` +
alembic + `/healthz` 200 + a real `/api/v2/place` 200.
### `build-push.yml` — images
Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
`frontend/**`, and on `v*.*.*` tags. Publishes
`git.thermograph.org/emi/thermograph/{backend,frontend}:sha-<12hex>`.
- **Serialised** (`max-parallel: 1`): one physical runner, and two whole image
builds racing each other buys nothing.
- **Semver exception:** a `v*.*.*` tag push carries no paths context, so a
release tag builds **both** images. That's intended — a release names the
whole set, not whichever domain moved last.
- The registry is mesh-only, so the runner host needs
`10.10.0.2 git.thermograph.org` in `/etc/hosts`.
### `deploy.yml` — one file, two environments, two services
Formerly six near-identical files. Branch selects the environment
(`main → beta`, `release → prod`); a matrix covers the services.
The style is **deliberately boring**: no dynamic `fromJSON` matrix, no
`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act
runner may evaluate differently, and the failure mode here is "production does
not deploy" or, worse, "deploys with an empty SSH host". So the two environments
get two explicit, mutually exclusive steps and the tag is computed in shell.
Preserved from the originals, all load-bearing:
- `fetch-depth: 0` — the tag is keyed to the last commit that touched *that
domain*, not the branch tip.
- 12-hex truncation matching `build-push.yml` exactly.
- Per-service, per-environment concurrency with **`cancel-in-progress: false`** —
a half-finished deploy must never be cancelled by a newer one.
- **Separate `PROD_SSH_*` credentials**, so a beta credential leak cannot reach
prod.
- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's
default action registry.
There's also a per-leg refinement: the workflow-level `paths` filter only says
*backend or frontend moved*, so each leg re-checks whether **this** push touched
**this** domain. Without it a backend-only push would also roll the frontend,
losing the independent-deploy property the whole split exists for. When there's
no usable base SHA (first push, force push, `workflow_dispatch`) it deploys
rather than skips — rolling onto the tag already running is a no-op, whereas
skipping silently strands a change.
### `infra-sync.yml` — infra's own pipeline
Push to `main` touching `infra/**` → SSH to beta **and** prod, fast-forward
`/opt/thermograph`, re-render `/etc/thermograph.env` from the vault.
**Rolls no service.** Image tags are the app domains' axis, not infra's. A
compose change that must recreate containers takes effect on the next app
deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh`.
Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release`
maps to LAN→beta→prod via image tags); **infra is not** — both hosts track infra
via `main`. Prod's *app images* are staged by `release`, but its checkout
follows `main`.
### `observability-validate.yml`
Parses every artifact that ships to a node: both compose files, all dashboard
JSON, the Loki and provisioning YAML, and the Alloy config — checked with the
**pinned `alloy` binary (v1.9.1, matching what the fleet runs)**, downloaded
from its GitHub release, no Docker CLI needed.
Alerting gets a stricter third step: every rule's `condition` must name a refId
that exists, every policy must route to a receiver that exists, and a literal
Discord webhook URL in the repo is a hard failure. **A rule pointing at a
missing refId is valid YAML, provisions cleanly, and then never fires** — that
silent no-op is exactly what this check exists to prevent.
### `ops-cron.yml` — the prod backup
Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into prod
and run inside the already-running stack.
**It uses `PROD_SSH_*`, not `SSH_*`.** An earlier revision reused `SSH_*` — so
the "prod" backup was silently dumping *beta*, and prod had no backup at all. If
you touch this file, verify a dump actually lands in
`agent@prod:~/thermograph-backups/`.
### `secrets-guard.yml` and `shell-lint.yml` — the backstops
Both run on **every** PR and push, deliberately un-path-filtered: *a backstop
that only runs when you expect it to isn't a backstop.*
- `secrets-guard` fails if any `infra/deploy/secrets/*.yaml` lacks both a
`sops:` block and `ENC[AES256_GCM` values. ~1 second.
- `shell-lint` runs shellcheck over every `*.sh` found by `find` (not a list —
so a new script is covered the moment it lands). Pinned to **v0.11.0 plus a
sha256**, installed from the official static release rather than `apt-get`:
the distro version drifts with the job image, and a drifted shellcheck grows
new warnings that fail CI on an unrelated push. Bump the version and the hash
together, and expect to fix new findings in the same PR.
Keep the tree at **zero shellcheck findings**. These scripts run as root over
SSH against live hosts with no test suite in front of them; this is the only
guard.
## Runner facts
- Jobs run on the `docker` label — the LAN box's Forgejo Actions runner.
- Job containers get the **host's docker.sock** automounted
(Docker-outside-of-Docker, no privileged mode). The job image
(`node:20-bookworm`) ships no docker CLI, which is why several workflows
`apt-get install docker.io` first.
- Forgejo's runner has **only the labels it was explicitly registered with**.
GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does
not. A `runs-on: [self-hosted, thermograph-lan]` array is permanently
unschedulable there — a real bug that was found and fixed once, and worth
recognising if a job ever silently stops appearing in the Actions history.
## Opening a PR
```bash
git switch -c fix/short-description # branch off dev
# ... work ...
cd backend && make test # or: cd frontend/server && go test ./...
git commit # concise, technical, no AI mentions
git push -u origin fix/short-description
```
Then open the PR into `dev` (Centralis's `forge_pr_open`, or the Forgejo web
UI). Wait for `gate`. Merge — squash is the configured style.
If your change touches shell, expect `shell-lint` to have an opinion. If it
touches `infra/deploy/secrets/`, `secrets-guard` will too.
Next: [Infra and secrets](08-infra-secrets.md).

254
docs/onboarding/08-infra-secrets.md Normal file
View file

@ -0,0 +1,254 @@
# 8. Infra and secrets
`infra/` owns **how and where the already-built images run**. It never builds
app source. Hosts' `/opt/thermograph` is a checkout of this whole monorepo.
Read [`infra/CLAUDE.md`](../../infra/CLAUDE.md) alongside this.
## The machines
| Host | Public | Mesh | Orchestrator | Login |
|---|---|---|---|---|
| **prod** | `169.58.46.181` / thermograph.org | `10.10.0.1` | Docker **Swarm** | `agent`, passwordless sudo |
| **beta** | `75.119.132.91` / beta.thermograph.org | `10.10.0.2` | **compose** | `agent`, passwordless sudo |
| **desktop** | — | `10.10.0.3` | compose (LAN dev) | it's your box |
```bash
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta
```
Beta also hosts **Forgejo** (git + CI + registry) and **Grafana + Loki**, which
is why beta is guarded as strictly as prod: a destructive command there takes
out git, CI and the registry at once. The LAN dev box is deliberately
unguarded.
Every root-effective command on the VPSes is logged by `auditd`
(`ausearch -k agentcmd`). That's a feature — fixes are attributable.
Prefer Centralis for routine work (`run_on_host`, `fleet_status`,
`deployed_version`) over raw SSH.
## The two orchestrators
Which path a host takes is decided by **`/etc/thermograph/deploy-mode`**: the
string `stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything
else is compose. The workflows never need to know which mode a host runs.
| | compose (beta, LAN dev) | Swarm (prod) |
|---|---|---|
| File | `infra/docker-compose.yml` | `infra/deploy/stack/thermograph-stack.yml` |
| Services | `db`, `backend`, `lake`, `daemon`, `frontend` | `db`, `web`, `worker`, `lake`, `daemon`, `frontend`, `autoscaler`, `autoscaler-lake` |
| Rolling | `up -d --no-deps <targets>` | start-first, health-gated, auto-rollback |
| Tag file | `deploy/.image-tags.env` | `deploy/.stack-image-tags.env` |
Note prod splits `backend` into **`web`** and **`worker`** (the
`THERMOGRAPH_ROLE` split from [04](04-backend.md)); compose runs one `backend`
service in role `all`.
`STACK_TEST=1` rehearses the entire Swarm deploy under stack name
`thermograph-test` on throwaway volumes and ports `18137`/`18080`. Leftover
`thermograph-test_*` containers in logs are rehearsal residue — ignore them.
### The volume/project-name coupling
Compose creates `thermograph_pgdata` / `_appdata` / `_applogs`; the Swarm stack
declares those exact names as `external: true` at the same mount paths. That
coupling is why `name: thermograph` is pinned in the compose file — running
compose from `infra/` without it derives project `infra`, silently creating a
whole new stack with empty volumes beside the running one. `deploy-dev.sh`
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
keep LAN dev separate on purpose. **Keep both halves.**
## `deploy/deploy.sh` — read this before you touch a deploy
Single entry point for beta and prod. Contract:
```bash
SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
SERVICE=frontend FRONTEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> …/deploy.sh
```
What it does, in order, and why each step is the way it is:
1. **`flock` on `deploy/.deploy.lock`**, with a self re-exec so the lock spans
the whole run. Backend and frontend deploys can fire for the same push
seconds apart and both SSH into one checkout; concurrent runs race the git
reset, the compose project and the tag file. `-w 600` bounds the wait.
2. **Render secrets** from the SOPS vault into `/etc/thermograph.env`, then
source it, so a by-hand run interpolates the same as the systemd unit does.
Guarded on the helper's existence so the very deploy that *introduces*
`render-secrets.sh` is safe.
3. **`git reset --hard origin/$BRANCH`** on the checkout root (`BRANCH` defaults
to `main`). ⚠️ **Uncommitted edits in `/opt/thermograph` evaporate here.**
The one exception is `deploy/.image-tags.env`, untracked on purpose.
4. **Read `.image-tags.env`** so a single-service roll re-renders compose with
*both* services' real tags and never accidentally recreates or downgrades the
sibling.
5. **Registry login** only if `REGISTRY_TOKEN` is set. The CI paths don't pass
one — the host is already `docker login`ed — and an unconditional login with
an empty token would abort under `set -e`.
6. **Pull, with a bounded ~5-minute retry.** `build-push.yml` is a *separate*
workflow triggered by the same push, and Forgejo's `needs:` only orders jobs
within one file. A deploy racing ahead of its build and failing "not found"
is confirmed-live behaviour; the retry covers a normal build and still fails
loudly on a genuine problem.
7. **Daemon capability probe.** Infra tracks `main` while image tags are
env-staged, so a host can legitimately be asked to roll a backend image older
than the compose file — one built before the daemon binary existed. The
script `docker run`s the image to test for `/usr/local/bin/thermograph-daemon`
and drops `daemon` from this run if it's absent, rather than leaving a
container crash-looping on a missing binary.
8. **Roll.** `TARGETS` are `backend → (backend, lake, daemon)`,
`frontend → (frontend)`, `all → everything`. **`daemon` and `lake` ride with
`backend` and are never targets on their own** — all three run the same image
at the same tag, and rolling one without the others is exactly the version
skew the `/internal/*` contract has no negotiation for. Single-service rolls
use `--no-deps`; a full `all` uses `--remove-orphans` (a renamed-away
service's old container would otherwise squat its port — confirmed live).
9. **Write `.image-tags.env`** *after* `up`, so a failed pull never records a
tag that isn't running.
10. **Health check** (backend 8137, frontend 8080), then detached city warming.
### Rollback
Rollback is redeploying the previous image tag. But **do not treat image
retention as a rollback guarantee**: both `deploy.sh` and `deploy-stack.sh`
*end* by deleting images outside the running pair. On beta that succeeds, so
beta typically holds **no local rollback target at all**; on prod it usually
fails only because Swarm's stopped task containers still hold references, which
is why prod keeps a handful. Verify the target tag is actually present before
promising a rollback — Centralis's `rollback_to(dry_run: true)` checks both host
and registry.
`docker system prune -a` on a live box is forbidden for the same reason.
`docker image prune -f` (dangling only) is the safe form.
## Secrets — the SOPS + age vault
**The single source of truth is `infra/deploy/secrets/*.yaml`**, committed
encrypted (values only — keys stay readable so diffs mean something) and
rendered into `/etc/thermograph.env` at deploy time by
`deploy/render-secrets.sh`.
**Cycling a key is: edit → commit → deploy.** No SSH, no hand-edited root file,
no per-host duplication.
| File | Holds |
|---|---|
| `common.yaml` | the 16 values identical on prod **and** beta — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config |
| `prod.yaml` | prod's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL`, Discord + mail credentials that exist nowhere else |
| `beta.yaml` | beta's own — the three held-back credentials, sizing, base URL |
| `dev.yaml` | the LAN box's own, **self-contained**, 12 values, no production credential |
| `centralis.prod.yaml` | Centralis's nine variables → `/etc/centralis.env`, its own renderer |
| `example.yaml` | format reference / CI fixture (fake values) |
| `../../.sops.yaml` | which age recipients files encrypt to (plaintext config) |
The renderer concatenates `common.yaml` then `<env>.yaml`, so a **host value
wins** (last occurrence of a duplicate key). `<env>` comes from
`/etc/thermograph/secrets-env` on the box.
### Two design decisions worth understanding
**Three credentials are deliberately *not* in `common.yaml`** —
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_DATABASE_URL`.
They hold identical values on prod and beta today, so by the mechanical rule
they belong there. They're kept per-host anyway because these are the
credentials that let one environment *act as* another: with them, a foothold on
beta (the more exposed box — public Forgejo and Grafana) is a foothold on prod's
database and prod's session signing. They match because beta was seeded from
prod, not because the two are meant to be one system. Keeping them per-host
costs one extra line and buys the ability to diverge.
**`dev` renders `dev.yaml` alone** — `deploy-dev.sh` exports
`THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of `common.yaml`'s sixteen values
are live production credentials, including a **read-write** object-storage
keypair on the bucket that also holds prod's database backups, and the VAPID
private key that signs push to real subscribers. The render is plaintext
concatenation, so layering it would put all of those into every container in the
dev stack — on the operator's desktop, which is also the CI runner whose
`docker`-labelled jobs get the host socket, running unreviewed `dev` code. An
override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the
production value is still physically a line in the file.
Dev works fine without them: no S3 keys means the `lake` service answers 503 and
history falls through to the Open-Meteo archive (an accelerator, never a
dependency); VAPID and IndexNow both self-generate and persist to the `appdata`
volume; with no metrics token `/api/v2/metrics` is direct-loopback-only, which
is right on a LAN box. One `common.yaml` value is simply *wrong* for dev —
`THERMOGRAPH_COOKIE_SECURE=1` silently breaks login over plain HTTP.
### Working with the vault
```bash
sops edit infra/deploy/secrets/beta.yaml # never a plain editor
infra/deploy/secrets/dry-run-render.sh # preview a render
```
- **Never hand-edit `/etc/thermograph.env`** — it's a rendered artifact,
overwritten on the next deploy.
- **Never paste a plaintext secret onto a box.**
- `secrets-guard` CI rejects a plaintext commit; the repo's `secrets-guard.sh`
hook blocks a direct Write/Edit to those files locally.
- `deploy/secrets/seed-from-live.sh` reads production secrets and is **not**
for an agent to run.
- The **`key-gaps`** skill audits which keys are missing or drifting across
environments. It reads key *names* only, never values.
There's a cautionary tale in `deploy/secrets/README.md` worth reading: Centralis
was hand-configured until a hand-edit wrote `CENTRALIS_TOKENS` as bare JSON into
a file that gets `source`d, bash stripped the quotes, and Centralis — correctly
failing closed on malformed JSON — dropped to a single identity. Nothing logged
an error. It looked exactly like the file had never been written. That's why
`centralis.prod.yaml` exists: the only durable fix was *a human stops writing
the file*.
## Terraform
`infra/terraform/` provisions and configures hosts, SSH-driven.
> **Never run `terraform apply` casually.** No tfstate is persisted anywhere, so
> an apply would attempt full re-provisioning of live hosts. Terraform here is
> **executable documentation** until state is bootstrapped.
## Ops query tooling
Read-only, uniform across environments:
```bash
infra/ops/dbq.sh dev "select count(*) from climate_history"
infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history"
infra/ops/iceberg.sh prod -c "select count(*) from era5_daily"
```
Both exec **into the container**. No database is exposed over TCP — each listens
only on its private docker network, and prod's is a Swarm **overlay the prod
host itself cannot route to**, so `ssh -L` works for beta and is *impossible*
for prod. Exec works identically everywhere with no ports, no tunnels, no infra
changes. The prod container name is a Swarm task name that changes on every
redeploy, so it's resolved at call time via `docker ps --filter name=`, never
hardcoded.
Queries connect as **`thermograph_ro`** — `NOSUPERUSER`, granted only
`pg_read_all_data`. Read-only is enforced by Postgres, not by convention:
```
$ infra/ops/dbq.sh prod -c "create table t(i int)"
ERROR: permission denied for schema public
```
Centralis's `sql_query` does the same thing with the same role. `write: true`
escalates to the app's superuser and takes effect immediately with no
confirmation — on prod, know what you are doing.
## Backups
The nightly `ops-cron.yml` `pg_dump` into `agent@prod:~/thermograph-backups/`
is it. Known gaps, stated plainly: **a single copy on the same box as the
database, no offsite yet**, and restores must handle TimescaleDB's
`continuous_agg` circular-FK warning (`--disable-triggers`). The `backups/`
prefix in the object-storage bucket belongs to those jobs — don't write outside
the lake's own prefix.
Next: [Observability](09-observability.md).

199
docs/onboarding/09-observability.md Normal file
View file

@ -0,0 +1,199 @@
# 9. Observability
Fleet-wide log aggregation: **Loki + Grafana on beta**, fed by a **Grafana
Alloy** agent on every node, all over the WireGuard mesh.
```
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3)
┌────────────┐ ┌──────────────────┐ ┌────────────┐
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│
└────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘
docker+caddy+app └──────────────────┘ docker+caddy+app
```
- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
`10.10.0.2:3100`. Never public.
- **Grafana** — the UI, fronted by beta's Caddy at
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
`allow_sign_up=false`, plus a break-glass local admin). Use that hostname
everywhere; **never** `grafana.thermograph.org`.
- **Alloy** — on each node, shipping three sources: every Docker container's
stdout/stderr, Caddy's host access logs, and the app's structured JSON logs
(`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` become
labels). Every line is tagged `host = prod|beta|dev`.
There is **no build and no deploy automation** for this domain — it ships by
hand. But `observability-validate.yml` parses every artifact that ships to a
node, so a malformed dashboard or an unparseable Alloy config fails in CI
instead of on the host.
## The rule that matters most
**The repo is the only durable path.** Dashboards and alerting are provisioned
from `observability/grafana/provisioning/` at startup, so **edits made through
Grafana's UI or API are overwritten on the next provision**. A change goes
through a PR. Centralis's `dashboard_write` opens one for you.
## Reading the fleet
Start wide, then narrow:
1. `fleet_status` — HTTP reachability plus what's running on each host.
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
3. `logs_query service=… host=…` — the actual lines.
### ⚠ The prod trap
Prod runs a Swarm stack, and its `job="docker"` streams carry **no `service`
label**. A raw `{host="prod", service="backend"}` query matches nothing and
reads as *"no logs"* when it actually means *"wrong selector"*.
Use Centralis's `logs_query` `service` argument, which rewrites it to a
container name-regex with the churning task suffix wildcarded. Prod service
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
`_lake`.
Related: **prod container names change on every redeploy** (the Swarm task
suffix). Never hardcode one; resolve via `docker ps --filter name=` at call
time.
## The dashboard
`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log
volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a
per-node notifier-liveness indicator, recent app errors, and a live
all-container tail.
## Alerting
Provisioned from `grafana/provisioning/alerting/`, mounted read-only into
Grafana.
**Every rule is LogQL** — there is no Prometheus anywhere in the fleet. Loki is
the only datasource, so every signal is derived from log lines. Thresholds were
backtested against real Loki data and the working is written out in the comments
beside each rule; **re-derive before changing a number rather than guessing**.
### Alerts go to Discord `#ops-alerts`, never email
Three reasons, all learned the hard way:
- Grafana's factory default pointed at the literal string
`<example@email.com>` — a paging path that had never delivered a message to
anyone.
- Beta's Grafana relays SMTP through **prod's** Postfix. Email alerts therefore
travel through the box most likely to be on fire, and vanish exactly when they
matter.
- Discord is off-estate, works when prod is dead, and reaches a phone.
`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
subscription output — real deliveries to real subscribers — so routing ops noise
there would corrupt the evidence.
The webhook URL is a secret and lives only in beta's gitignored `.env` as
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
webhook URL committed to the repo is a **hard CI failure**.
### The twelve rules
| Rule | Fires when | Severity |
|---|---|---|
| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the pipeline is broken | critical |
| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
| `ProdHTTP5xxElevated` | ≥25 prod 5xx over 6h (the error-budget alarm) | warning |
| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` from prod in 20m | critical |
| `NotifierHeartbeatMissingBeta` | same, on beta | warning |
| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
| `AlertingWatchdog` | always — one Discord message per day | watchdog |
`AlertingWatchdog` is the dead-man's switch and fires permanently by design.
**Its value is its absence:** nothing can report its own death, so if
`#ops-alerts` has been silent for more than a day, alerting is broken rather
than the estate healthy.
### The strict CI check, and why
A rule whose `condition` names a refId that does not exist is **perfectly valid
YAML, provisions without complaint, and then never fires** — a paging path that
looks healthy and reaches nobody. `observability-validate.yml` checks that every
condition names a real refId and every policy routes to a receiver that exists,
precisely because nothing else would catch it.
The same asymmetry shows up in verification: `GET
/api/v1/provisioning/contact-points` **redacts** the webhook URL, so you cannot
confirm interpolation by reading it back. A contact point holding the literal
string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and
pages nobody. **The only proof is a message actually arriving in
`#ops-alerts`** — force one with the receivers-test endpoint (the command is in
`observability/README.md`).
## Deploying an observability change
> ⚠️ **Merging is not deploying — and this one has already bitten.**
> `/opt/observability` on beta and prod is a checkout of the **archived**
> `emi/thermograph-observability` repo. It can never `git pull` again; the
> content now lives here under `observability/`. A previous observability PR
> merged and had **zero live effect** for exactly this reason.
Until someone re-points those checkouts at the monorepo, every change ships by
hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up,
`sudo cp` it into place, restart the specific container.
Two gotchas in that sequence:
- Use `sudo docker restart <container>` on **prod** — not `docker compose`,
which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
interpolate from. Beta's stack *does* have an `.env`, so compose works there.
- **`docker restart` will not pick up a new environment variable.** Any change
that introduces one needs `docker compose up -d grafana` on beta.
Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana.
Full step-by-step, including rollback, is in
[`observability/README.md`](../../observability/README.md).
## Known gaps (as documented in the repo)
- **Alloy can silently stop tailing a container after Swarm replaces the task.**
Observed on prod 2026-07-24: `thermograph_worker` was `1/1`, healthy, writing
~19 lines per 10 minutes to its own docker log, yet not one line reached Loki
for 13½ hours. What localises it to Alloy: prod's `tag=heartbeat` lines kept
arriving throughout, and those come from a *different* Alloy source
(`loki.source.file` on `/applogs`) than container stdout
(`loki.source.docker`). One source went deaf, the other didn't. Workaround:
`sudo docker restart alloy-alloy-1` on the node. **Not root-caused.**
`ProdWorkerContainerSilent` now catches this class of failure.
- **Some services are too quiet to alert on.** `thermograph_daemon` (23 lines /
24h), `thermograph_autoscaler` (1) and `_autoscaler-lake` (2) have a healthy
floor indistinguishable from dead; `thermograph_lake` is bursty ETL with
legitimate idle gaps. Giving each an `audit.log_heartbeat()` line would make
all four alertable with the pattern the notifier already uses. **The single
highest-value follow-up in this domain.**
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
dead-man's switch — it relies on somebody noticing the daily message stopped.
An external uptime pinger from off-estate would close this properly.
- **Beta runs a `docker-compose.override.yml` that is not in this repo**, wiring
Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked
compose file or deleted; alerting no longer depends on it.
## When something breaks
1. `fleet_status` — is it up at all?
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
3. `deployed_version env=prod` — did something ship just before it started?
4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass?
5. **Write down what you found with `notes_write` *before* fixing it.** The
reconstruction is worth more than the fix, and it will be gone by tomorrow.
You are authorised to fix production roadblocks directly — restart or roll a
wedged service, clear a full disk, kill a runaway process, bounce a crash-looping
container. What you must not do is make the fix *durable* on the box: writes to
code, docs and dashboards go through a PR, and `deploy.sh` hard-resets the
checkout anyway. The on-box action is stabilisation; the PR is the fix.
Next: [Recipes](10-recipes.md).

256
docs/onboarding/10-recipes.md Normal file
View file

@ -0,0 +1,256 @@
# 10. Recipes
Task-shaped walkthroughs. Each one names the files you'll touch and the traps
specific to that change.
---
## Add a field to a graded API payload
1. **Build it** in `backend/api/payloads.py` (or `content_payloads.py` for the
SSR content API). Payload assembly is pure `inputs → dict`; keep HTTP
semantics with the caller.
2. **Bump `PAYLOAD_VER`** in `api/payloads.py`. Non-negotiable — the derived
store is token-validated with no clock expiry, so without a bump the old
shape serves out of cache forever, silently. If the change is content-API
only, `CONTENT_VER` is the narrower lever.
3. **Decide whether it's breaking.** Additive → fine on `v2`. Breaking → a new
`v3` router mounted alongside `v2` re-registering only the changed handlers,
plus `API_CONTRACT_VERSION` bumped in the same PR.
4. **Check `/cell`.** If the field lands in a slice, the slice→view map in
`frontend/static/cache.js` has to track it.
5. **Consume it** in `frontend/server/internal/contentapi/types.go` and whichever
template or JS renders it.
6. **Test both sides**: `cd backend && make test`, then
`cd frontend/server && go test ./...`.
7. **Refresh fixtures if the shape moved**: `cd frontend && make capture-fixtures`
against a live backend. Those fixtures feed the **Go** tests too.
---
## Add a new SSR page
The backend owns the page's *data and metadata*; the frontend owns its *markup*.
1. `backend/api/content_payloads.py` — a builder returning JSON-safe dicts (raw
floats, no `Markup`, no ContextVar unit rendering), including `page_title`,
`canonical_path`, `breadcrumb` and `jsonld`. Those live here, **not** in the
frontend.
2. `backend/api/content_routes.py` — route wiring plus the same derived-store /
ETag pattern the neighbouring routes use.
3. `frontend/server/internal/contentapi/` — a typed client method.
4. `frontend/server/internal/render/templates/<page>.html.tmpl` — embedded in
the binary, so a template change needs a rebuild.
5. `frontend/server/internal/content/pages.go` — the handler; register the route.
6. If the page should be crawlable, add it to the sitemap
(`backend/api/sitemap.py` and the `/content/sitemap` payload).
7. Copy that isn't per-city data goes in `frontend/content/pages.yaml`, loaded
fail-loud at boot.
---
## Change something visual
1. Read [`frontend/DESIGN.md`](../../frontend/DESIGN.md) first. It wins over any
general design guidance.
2. Edit `frontend/static/style.css`. **Use `var(--token)`; never paste a hex.**
New surfaces take their colour from existing tokens so light mode keeps
working for free.
3. If it's a grade colour, keep the scale's order and midpoints intact — they
encode meaning, not decoration, and the charts pull from the same tokens as
the legend.
4. **Verify at 390 / 800 / 1920 / 2560 / 3840px in both light and dark:**
```bash
cd frontend && .venv/bin/python tools/shoot.py
```
or drive a real browser with the Chrome DevTools MCP. Narrow while
iterating: `tools/shoot.py index --width 390 --scheme dark`.
5. Watch the hard rules: inputs at `font-size: 16px` minimum (iOS zoom), ~44px
touch targets, honour `prefers-reduced-motion`, metric order always
`Precip · High · Low`.
---
## Add or rotate a secret
```bash
sops edit infra/deploy/secrets/prod.yaml # never a plain editor
git commit && push && open a PR
```
Then it reaches the hosts by either route: an `infra/**` push to `main` fires
`infra-sync.yml` (re-renders `/etc/thermograph.env` on beta and prod, rolls
nothing), or the next app deploy renders it as step 2 of `deploy.sh`.
Decide **which file** first:
- identical on prod *and* beta, and not a "lets one environment act as another"
credential → `common.yaml`;
- per-host, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` /
`THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`;
- dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's.
Preview with `infra/deploy/secrets/dry-run-render.sh`. Audit cross-environment
drift with the `key-gaps` skill (names only, never values).
**Never** hand-edit `/etc/thermograph.env`, paste a plaintext secret onto a box,
or run `seed-from-live.sh`.
If a rotated value needs containers recreated to take effect, remember
`infra-sync` rolls nothing — you need an app deploy or a by-hand
`SERVICE=all … deploy.sh`.
---
## Add a recurring background job
Ask one question first: **does it need data?**
- **No** (a timer, a connection, an external poke) → add a `cron.Job` in
`backend/daemon/main.go`. One replica, so no election needed. Give it an env
var for its interval, defaulting sensibly and falling back to the default on a
malformed value.
- **Yes** → add an `/internal/*` route in `backend/api/internal_routes.py` and
have the daemon POST to it. Grading depends on polars and the cache;
reimplementing it out-of-process would let results drift from the API's.
If you genuinely need an in-process background thread in the Python app instead,
**read `core/singleton.py` first**. Anything that issues upstream fetches on a
timer must run behind `claim_leader()` — N workers × M hosts multiplies quota
use N×M, and that has already caused overnight 429s in production. Things that
drain a per-worker request-fed queue (the neighbour warmer) correctly run
everywhere; so does the heartbeat, because a beat costs one local file append.
Then give it a heartbeat (`audit.log_heartbeat()`) so observability can alert on
its death — see the "too quiet to alert on" gap in
[observability](09-observability.md).
---
## Change a Grafana alert or dashboard
**Never through the UI.** Provisioning overwrites it on the next start.
1. Edit `observability/grafana/provisioning/alerting/*.yml` or
`observability/grafana/dashboards/*.json`.
2. If you're changing a threshold, **re-derive it against real Loki data** — the
working for the current number is in the comments beside the rule.
3. Open a PR. `observability-validate.yml` will check that every condition names
a real refId, every policy routes to a real receiver, and no literal webhook
URL is committed.
4. **Merging is not deploying.** Ship it by hand per
`observability/README.md` — and remember `docker restart` won't pick up a new
environment variable.
5. Verify by forcing a real message into `#ops-alerts`. The API redacts the
webhook, so arrival is the only proof.
---
## Debug: "the site is serving stale data"
Work the token chain, in this order:
1. `GET /api/version` on the affected environment — is `payload_ver` what you
expect? If a payload shape changed and this didn't move, that's your answer.
2. Is the derived-store token advancing? It's built from `PAYLOAD_VER` plus the
history end date (`history_token`), so a cell whose history stopped updating
pins its token.
3. Is the *history* updating? Check `climate_sync` freshness
(`infra/ops/dbq.sh prod …`) and whether upstream is in a rate-limit cooldown
(the errors log, tagged `retry`/`error`).
4. Is it the **frontend's** TTL cache instead? `THERMOGRAPH_SSR_CACHE_TTL`
defaults to 600s, in front of the backend's own caching.
5. Is it the **browser**? `cache.js` seeds IndexedDB from the `/cell` bundle. If
ETags stopped being visible cross-origin (`expose_headers`), revalidation
silently stops.
---
## Debug: "a deploy went green but nothing changed"
Common causes, most likely first:
1. **The tree was already identical.** A promotion between branches whose trees
match merges nothing while still firing the deploy workflows.
2. **The domain wasn't touched.** `deploy.yml` re-checks whether *this* push
touched *this* domain, and skips the leg if not.
3. **The tag is domain-keyed.** `deployed_version` showing a tag that isn't the
branch head is correct behaviour, not drift.
4. **Someone hand-edited `/opt/thermograph`.** `deploy.sh` runs
`git reset --hard` first; the edit is gone.
5. **It's an observability change.** `/opt/observability` points at the archived
repo and cannot pull. See above.
---
## Promote to beta, and then to prod
```
dev → main your call, batched. Beta is the test environment;
not promoting is not testing.
main → release ALWAYS the owner's call. Prepare it, state what would ship,
the CI evidence and how long it soaked on beta, then recommend —
including recommending against.
```
Centralis's `promote(from, to)` does the mechanics and enforces the rule;
`forge_pr_merge` refuses `main → release` under any flag. Before promoting,
compare the two tips' **trees**, not their commit counts.
---
## Roll back
Redeploying the previous image tag:
```bash
SERVICE=backend BACKEND_IMAGE_TAG=sha-<previous12hex> \
/opt/thermograph/infra/deploy/deploy.sh
```
**Check the tag actually exists first.** Both deploy scripts *end* by deleting
images outside the running pair, so beta typically holds no local rollback
target at all. `rollback_to(dry_run: true)` checks host and registry and will
tell you when it's in neither.
---
## Regenerate the bundled city data
`backend/cities.json` and `cities_flavor.json` are generated reference data, not
runtime state:
```bash
cd backend
python gen_cities.py # the curated city set
python gen_flavor.py # the editorial blurbs
```
Editorial copy has its own voice guide — the **`city-copy`** skill defines what
good reads like, and **`backfill-city-copy`** drives generation at scale. Both
enforce the relative-grades rule.
After a deploy, `warm_cities.py` pre-warms those cells so `/climate` pages serve
from cache and a crawl never bursts the archive quota. It's idempotent, paced,
and claims a flock — deploys launch it detached with no overlap check, so two
overlapping runs would otherwise double-spend quota and race the cache writes.
---
## Work on a live host
Prefer Centralis: `run_on_host`, `fleet_status`, `deployed_version`,
`logs_query`, `sql_query`.
You may, without asking: read anything; restart or roll a wedged service; clear
a full disk; kill a runaway process; bounce a crash-looping container; run the
blessed deploy path onto an already-published image.
You must not: hand-edit the checkout to persist a change (`git reset --hard`
eats it); hand-edit `/etc/thermograph.env` (rendered artifact); force-push;
deploy prod from an unmerged branch; `docker system prune -a` on a live box (it
deletes your rollback image — `docker image prune -f` is the safe form); or
restart the `_db` service casually. Postgres/Timescale restarts and volume
operations get confirmed with the operator first.
Next: [Traps and stale docs](11-traps.md).

268
docs/onboarding/11-traps.md Normal file
View file

@ -0,0 +1,268 @@
# 11. Traps and stale docs
Two kinds of thing here: **live traps** (correct code that will surprise you)
and **stale docs** (files in this repo that currently describe a system that no
longer exists). The second list matters as much as the first — this repo has a
lot of documentation, most of it excellent, and some of it describing the world
before a rewrite.
Everything below was checked against the tree at the time of writing.
---
## Stale docs in this repo
### `frontend/CLAUDE.md` and `frontend/README.md` describe a service that is not deployed
**The live frontend is Go** (`frontend/server/`, module `thermograph/frontend`).
Both documents describe the Python FastAPI/Jinja implementation
(`frontend/app.py`, `content.py`, `api_client.py`, `format.py`,
`templates/*.html.j2`), which was superseded by the Go port and is kept in-tree
as the reference the port was made from.
What is still **correct** in them: every contract (API-version pinning,
`pctOrd`, the `/cell` + ETag relationship, the Fahrenheit set, boot-must-survive-
an-unreachable-backend), every environment variable name and default, and all of
`DESIGN.md`.
What is **wrong**: the language, the file paths, and — notably —
`frontend/CLAUDE.md`'s claim that `make test-unit` is *"the tier CI runs."* It
isn't any more. CI builds the frontend image, and `frontend/Dockerfile`'s
builder stage runs `gofmt -l` + `go vet` + `go test ./...` before compiling, so
a failing **Go** test is what fails CI. The Python tiers are local-only now.
Also note `api_client.py` carries a **third** copy of the `API_VERSION` pin.
It isn't deployed, but keep it in step or you'll debug a phantom locally.
### `infra/DEPLOY.md` describes the pre-Swarm, pre-monorepo deploy
Three separate layers of drift:
- It says **prod is deployed with `terraform apply`**. Prod is deployed by a
push to `release` firing `deploy.yml`, exactly like beta's `main` path — see
[07](07-ci-and-release.md).
- It says prod runs a **compose** stack. Prod runs **Swarm**
(`infra/deploy/stack/`).
- Its file table points at `/opt/thermograph/deploy/deploy.sh` and
`/opt/thermograph/docker-compose.yml`. Both moved under `infra/` at the
monorepo cutover.
The document's own header flags the legacy `provision.sh` / systemd / venv
walkthrough as superseded — but the drift goes further than that note admits.
The **key-management** sections (Part 1) and the `/etc/hosts` registry note are
still accurate and useful.
### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows
Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist. The two
LAN-dev deploy workflows were deleted rather than ported at the CI
consolidation — they were documented as inert (they targeted a monorepo layout
at `~/thermograph-dev` on a box still holding a split-era checkout). **LAN dev
is a local `make dev-up` concern, not a CI environment.**
`ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the
workflow set; there are now nine files and the deploy/build ones were collapsed.
Everything else in `ACCESS.md` — the host table, the agent-access model, the key
rotation procedure, the Swarm/Forgejo tracks — is current and worth reading.
### `backend/README.md` is written for the split-repo era
It describes `thermograph-backend` as its own repo with sibling repos, uses the
retired image path `git.thermograph.org/emi/thermograph-backend/app`, and says
*"there is no Makefile in this repo yet"* — there is (`make test`, `make smoke`,
`make clean-venv`).
The **"How it works"** section (grid → data → percentiles → caching) is one of
the best explanations of the domain model anywhere in the repo. Read it; just
mentally translate the topology.
### `CUTOVER-NOTES.md` carries a superseded block, correctly labelled
Its per-domain workflow names (`backend-build-push.yml`, `backend-deploy.yml`, …)
no longer exist — the six deploy workflows collapsed into `deploy.yml` and the
two build-push workflows into `build-push.yml`. The file says so in an explicit
"Superseded 2026-07-25" note, and everything it describes about *behaviour* is
unchanged.
It also references `backend/MONOREPO-CUTOVER-PLAN.md`, which is not in the tree.
`CUTOVER-NOTES.md` remains the source of truth for what is and isn't live.
### The domain `CLAUDE.md` files still use repo-era wording
They say "this repo" and link to sibling repo names. Mentally map
`thermograph-backend` → `backend/`, and fix the wording as you touch files near
it — that's the standing instruction in the root `CLAUDE.md`.
### `observability/README.md`: merging is not deploying
`/opt/observability` on beta and prod is a checkout of the **archived**
`emi/thermograph-observability` repo. It can never `git pull` again. A previous
observability PR merged and had **zero live effect** for exactly this reason.
The README documents the hand-ship procedure; until the checkouts are re-pointed
at the monorepo, that's the only path. Flagged again in
[09](09-observability.md).
---
## Live traps
### The `docker-compose.test.yml` files still pull retired image paths
Both `backend/docker-compose.test.yml` and `frontend/docker-compose.test.yml`
default `BACKEND_IMAGE_PATH` to **`emi/thermograph-backend/app`** — the
split-era path. The current image is `emi/thermograph/backend`.
- `backend/make smoke` is fine: it builds `:local` and never pulls.
- `frontend/make backend-up` **does** pull, and pins
`THERMOGRAPH_BACKEND_TEST_TAG` to `v0.0.2-split-ci`. The old packages were
deliberately left in the registry for rollback and GC'd only after burn-in, so
this may work, may serve a very old backend, or may 404 depending on when you
read this. If `make backend-up` fails or behaves oddly, set the path and tag
explicitly:
```bash
BACKEND_IMAGE_PATH=emi/thermograph/backend \
THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> make backend-up
```
### `static/units.js`'s `F_REGIONS` is guarded by nothing
Source comments in three files claim *"a test asserts all three stay
identical."* The only automated assertion is
`format_test.go::TestFCountriesMatchesBackend`, which re-parses the backend's
Python and compares the **Go** copy. Nothing reads `units.js`.
All four copies currently *are* identical (14 codes, checked). But the browser
copy is held in step by convention alone. Extending the Go test to parse
`units.js` too is a good first contribution.
### The compose project name is load-bearing
`infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose
from `infra/` derives project `infra` — a silently **new** stack with empty
volumes beside the running one. `deploy-dev.sh` exports
`COMPOSE_PROJECT_NAME=thermograph-dev` to keep LAN dev separate on purpose.
Keep both halves.
### `deploy.sh` hard-resets the host checkout
`git reset --hard origin/$BRANCH` runs before anything else. Uncommitted edits
in `/opt/thermograph` evaporate. `infra-sync.yml` does the same reset without
deploying. The one deliberate exception is the untracked
`deploy/.image-tags.env` — losing *that* makes the next single-service deploy
roll the sibling onto `local`.
### The deploy scripts eat your rollback image
Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the
running pair. On beta that succeeds, so beta typically holds **no local rollback
target at all**; on prod it usually fails only because Swarm's stopped task
containers still hold references. `docker system prune -a` is forbidden on a
live box for the same reason — but the deploy script is the thing that actually
eats the image, not prune. Verify a tag exists before promising a rollback.
### `daemon` and `lake` are never deploy targets on their own
They ride with `backend`. All three run the same image at the same tag — a
second binary and a second role inside one image — and the `/internal/*`
contract between daemon and web has no version negotiation. There's also a
mechanical reason: a single-service deploy runs `up -d --no-deps <targets>`, so
a service in compose but absent from `TARGETS` would simply never be created.
### `docker compose config --images <service>` does not filter
Confirmed live on Compose v5.3.1: it prints **every** service's image, one per
line, in file order. A `| head -1` therefore grabbed `db`'s image, and
`deploy.sh`'s daemon capability probe found no daemon binary in a Postgres image
and dropped the daemon from *every* backend deploy. The script now builds the
image reference from the same variables compose interpolates.
### Prod's Loki streams carry no `service` label
Prod runs Swarm; its `job="docker"` streams have no `service` label. A raw
`{host="prod", service="backend"}` query matches nothing and reads as "no logs"
when it means "wrong selector". Use `logs_query`'s `service` argument. Prod
container names also change on every redeploy (the task suffix) — resolve via
`docker ps --filter name=` at call time, never hardcode.
### A Grafana alert rule can provision cleanly and never fire
A rule whose `condition` names a non-existent refId is valid YAML, provisions
without complaint, and pages nobody. Likewise a contact point holding the
literal string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the
API — and the API **redacts** the URL, so you cannot confirm interpolation by
reading it back. The only proof is a message arriving in `#ops-alerts`.
### Terraform has no state anywhere
A casual `terraform apply` would attempt full re-provisioning of live hosts.
It's executable documentation until state is bootstrapped.
### The registry is mesh-only
`git.thermograph.org` resolves publicly to beta's IP, but beta's Caddy rejects
`/v2/*` from outside the WireGuard mesh. Any host that pulls needs
`10.10.0.2 git.thermograph.org` in `/etc/hosts`. Beta itself doesn't — it *is*
the box.
### Forgejo runners only have the labels they registered with
GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does not.
`runs-on: [self-hosted, thermograph-lan]` is permanently unschedulable there —
a real bug, found and fixed once. If a job silently stops appearing in the
Actions history, check this first.
### Forgejo's `needs:` doesn't cross workflow files
`build-push.yml` and `deploy.yml` fire from the same push with no ordering
guarantee, and a deploy racing ahead of its build and failing "not found" is
confirmed-live. That's why `deploy.sh`'s pull has a bounded ~5-minute retry.
### `/api/v2/cell?prefetch=1` must never spend quota
The hard guarantee: a cold cell answers `204` with no body. Anything named
"prefetch" or "warm" calls the cache-only loaders. Breaking this gets the whole
service rate-limited.
### Background timers must go through leader election
`singleton.claim_leader()` — flock per host, Postgres advisory lock per cluster.
A timer-driven sweep running in N uvicorn workers × M Swarm hosts multiplies
Open-Meteo quota use N×M. That is the documented cause of overnight 429/503s
after prod went to 3 workers. The heartbeat is the deliberate exception — a beat
is one local file append, so duplicates are free.
### `THERMOGRAPH_DATA_DIR` exists because of a real collision
The backend's Python package `data/` and its runtime data dir `<root>/data`
collide at the same path once the code lives at the image root. Mounting a
volume there shadows and erases `data/*.py`, so `import data.climate` fails at
boot. Point the env var **outside the code tree** in a container. And never
reintroduce `__file__`-relative paths in a module — `paths.py` resolves
everything from the repo root for this reason.
### Alloy can silently stop tailing a container
Observed on prod, 13½ hours of `thermograph_worker` logs never reaching Loki
while the container was healthy and writing. Not root-caused. Workaround:
`sudo docker restart alloy-alloy-1`. `ProdWorkerContainerSilent` now catches the
class.
### Nominatim is ~1 req/s, on one thread
All reverse geocoding funnels through a single dedicated worker draining a
queue. The earlier design slept inside the caller's thread — which in the server
is one of Starlette's shared sync-threadpool threads.
---
## If you fix one of these
Docs are cheap to correct and the root `CLAUDE.md` already asks you to fix
repo-era wording as you touch files near it. If you're in `frontend/` anyway,
bringing `frontend/CLAUDE.md` in line with the Go service would be the single
highest-value documentation change available — it's the file an agent reads
*before* touching that domain.
Back to the [index](README.md).

70
docs/onboarding/README.md Normal file
View file

@ -0,0 +1,70 @@
# Onboarding — becoming a full contributor to Thermograph
This is the developer onboarding path for the `emi/thermograph` monorepo: what
the product is, how the code is shaped, how to run it, what will break silently
if you get it wrong, and how a change actually travels from your editor to
`thermograph.org`.
**Scope.** This covers *working in this repo*. Cross-cutting architecture
decision records and operator runbooks live in the separate `thermograph-docs`
repo (reachable through Centralis's `docs_search`) — this set links out rather
than duplicating them. Where a fact is owned by a `CLAUDE.md`, that file stays
the source of truth and this set explains the context around it.
## Reading order
| # | Doc | Read it when |
|---|-----|--------------|
| 1 | [Orientation](01-orientation.md) | First. What the product actually claims, the estate, the four non-negotiable rules. |
| 2 | [Local setup](02-setup.md) | Before you touch anything. Toolchain, verified run/test recipes for every service. |
| 3 | [Repo map](03-repo-map.md) | To find things. Every domain and directory, and which files matter. |
| 4 | [Backend deep dive](04-backend.md) | Before your first backend change. Data pipeline, grading, caching, roles, daemon. |
| 5 | [Frontend deep dive](05-frontend.md) | Before your first frontend change. The Go SSR service, static assets, design system. |
| 6 | [Cross-service contracts](06-contracts.md) | **Before any change that touches both.** These break silently and in production. |
| 7 | [CI and release](07-ci-and-release.md) | Before you open a PR. Nine workflows, three branches, three environments. |
| 8 | [Infra and secrets](08-infra-secrets.md) | Before you touch deploy, compose, or a secret. |
| 9 | [Observability](09-observability.md) | When something is wrong and you need to see it. |
| 10 | [Recipes](10-recipes.md) | Task-shaped walkthroughs for the things you'll actually do. |
| 11 | [Traps and stale docs](11-traps.md) | **Skim early, re-read often.** Which docs in this repo currently lie, and why. |
## The short version
Thermograph grades how unusual today's weather is at any point on Earth against
~45 years of that exact location's own history. Percentiles, never thermometer
readings.
- **`backend/`** — Python 3.12 / FastAPI. Owns all climate data, grading, the
API, accounts, notifications, plus a Go daemon (`backend/daemon/`) that owns
the Discord gateway and recurring timers. Ships as `emi/thermograph/backend`.
- **`frontend/`** — **Go** SSR service (`frontend/server/`) plus every static
asset. No climate data, no database, no compute. Ships as
`emi/thermograph/frontend`. (The Python implementation at `frontend/*.py` is
the superseded original — see [traps](11-traps.md).)
- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets
vault, Terraform, ops query tooling.
- **`observability/`** — Loki + Grafana on beta, an Alloy agent per node.
Branches stage environments: PR → `dev` (LAN dev) → `main` (beta) → `release`
(prod). The two app domains build and deploy **independently** — that
independence is the whole reason the split-then-reunify history exists, and
[contracts](06-contracts.md) is the list of things that keep it safe.
## Your first day
1. Read [Orientation](01-orientation.md) and [Traps](11-traps.md).
2. Work through [Local setup](02-setup.md) until `make test` is green in both
`backend/` and `frontend/server/`.
3. Read [Repo map](03-repo-map.md) with the tree open beside it.
4. Pick something small in the domain you're least afraid of, and follow
[Recipes](10-recipes.md) end to end — including opening the PR.
## Your first week
- Read [Contracts](06-contracts.md) properly. Every item on that list has
already cost someone a production incident somewhere in this project's
history; that's why each one is written down.
- Get Centralis wired up (see [Local setup](02-setup.md)) and ask it
`fleet_status`, `logs_overview`, `deployed_version`. You cannot debug this
estate blind, and the boxes are not reachable from a laptop off the
WireGuard mesh.
- Read the `CLAUDE.md` for each domain. They are terse, current, and binding.