diff --git a/README.md b/README.md index 7db6473..a999ec5 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,13 @@ for: **per-domain images, per-domain deploys, and an async FE/BE contract**. `thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no build artifacts, different change cadence). +## New here? + +[`docs/onboarding/`](docs/onboarding/README.md) is the developer onboarding +path: orientation, verified local-setup recipes, a per-domain deep dive, the +cross-service contracts that break silently, the release flow, and a list of +which docs in this repo are currently stale. + ## How CI stays decoupled Every workflow in `.forgejo/workflows/` is **path-filtered to its domain**: a diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index ccc1b68..02b479a 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -38,13 +38,20 @@ for systemd/CI callers. Entry target is `app:app`. `cache.js` reads `null` cross-origin and never revalidates. Don't change the ETag derivation or that list without checking `frontend/static/cache.js`. - **`data/grading.py::pct_ordinal()`** is mirrored by `frontend`'s - `shared.js::pctOrd()` — floor a percentile into `1..99`, never 0 or 100. + `server/internal/format::PctOrdinal` (SSR) and `static/shared.js::pctOrd()` + (browser) — floor a percentile into `1..99`, never 0 or 100. `TEMP_BANDS`/`RAIN_BANDS` are the source of truth for tier names and thresholds; changing them without the frontend produces tiers drawn in colours that disagree with the labels the API returns. -- **Fahrenheit country set** — `api/content_payloads.py`'s `F_COUNTRIES` must stay - identical to `frontend`'s `format.py::F_COUNTRIES` and `static/units.js`'s - `F_REGIONS`. There is a test asserting this. +- **Fahrenheit country set** — `api/content_payloads.py`'s `F_COUNTRIES` is + canonical; `frontend`'s `server/internal/format::FCountries` and + `static/units.js`'s `F_REGIONS` must stay identical to it. Both are asserted + by tests in `frontend/server/internal/format/format_test.go` — but note the + backend cross-check **skips in CI** (the frontend image's build context is + `frontend/`, so this file is unreachable from the builder stage, which is the + only place CI runs those tests). It fires on a full checkout. The `units.js` + check does run in the image build. `frontend/format.py`'s copy is the + superseded Python service's and is not deployed. - **`/healthz` and `/api/version` are deliberately I/O-free** so they stay cheap under tight healthcheck intervals. diff --git a/backend/docker-compose.test.yml b/backend/docker-compose.test.yml index 2b1addb..d5ff99e 100644 --- a/backend/docker-compose.test.yml +++ b/backend/docker-compose.test.yml @@ -22,7 +22,11 @@ services: backend: # Defaults to a locally-built `:local` image (scripts/smoke.sh builds it). In CI, # set BACKEND_IMAGE_TAG=sha-<12hex> to smoke the exact published image. - image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph-backend/app}:${BACKEND_IMAGE_TAG:-local} + # The path must match what build-push.yml publishes -- emi/thermograph/backend. + # It read emi/thermograph-backend/app (the retired split-era path) until this + # was corrected; harmless for the default `:local` build, wrong the moment + # BACKEND_IMAGE_TAG names a real published tag. + image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local} depends_on: db: condition: service_healthy diff --git a/docs/onboarding/01-orientation.md b/docs/onboarding/01-orientation.md new file mode 100644 index 0000000..0194611 --- /dev/null +++ b/docs/onboarding/01-orientation.md @@ -0,0 +1,146 @@ +# 1. Orientation + +## What the product actually claims + +Thermograph answers one question: **how unusual is the weather here, right now, +compared to this exact place's own history?** + +That framing is not decoration — it constrains the code, the copy and the +colour scales: + +- A grade is a **percentile against ~45 years of that grid cell's own record**, + not an absolute reading. 60 °F is *Above Normal* on a cool coast and *Below + Normal* inland on the same afternoon. +- Tier names are therefore relative words — *Near Record*, *High*, *Above + Normal*, *Normal*, *Below Normal*, *Low*, *Near Record* — never "hot", + "warm", "cold". This rule binds prose, alt text, commit messages and API + fields alike. +- Precipitation is graded on its **own** ladder, because most days are dry: a + rainy day is ranked only among the rain days in its seasonal window, and dry + days are coloured by dry-streak length instead. + +Public since **2026-07-16**. Free. Run by one operator plus automation, which +is why so much of this repo is about making irreversible things hard. + +## The domain model, in six steps + +1. **Grid** (`backend/data/grid.py`) — an arbitrary lat/lon snaps to a stable + **~4 sq mi cell**. Latitude rows are ~2 miles tall; longitude spacing scales + by `cos(latitude)` so cells stay roughly square at any latitude. The cell id + is deterministic and is the cache key for everything downstream. Worldwide + coverage, no gaps. +2. **History** (`backend/data/climate.py`) — the full **1980 → present daily + record** (max/min temp, precip, wind, gust, humidity, feels-like) for that + cell, fetched once and stored durably. Expensive to obtain, so it is treated + as source-of-truth data, not cache. +3. **Recent + forecast** — a separate bundle of recent observations plus ~8 + forward days, refreshed on its own TTL. This is what "today" is graded from. +4. **Grading** (`backend/data/grading.py`) — for a given day of year, the + reference distribution is *every historical day within ±7 days of it* + (a 15-day seasonal window that wraps the year end). The observed value is + placed on that distribution as an **empirical mid-rank percentile**, then + mapped onto `TEMP_BANDS` / `RAIN_BANDS`. +5. **Payloads** (`backend/api/payloads.py`) — pure "inputs → response dict" + assembly. One definition per payload shape, shared by the HTTP routes, the + `/cell` bundle and offline tooling. +6. **Caching** (`backend/data/store.py`) — every derived payload is stored + against a **validity token** that encodes everything the payload depends on. + A token mismatch is simply a miss. Freshness is driven by the token + advancing, *never* by clock expiry — so stale data cannot be served, and one + `PAYLOAD_VER` bump atomically orphans every pre-upgrade row. + +If you internalise one thing: **the token is the cache-correctness mechanism, +and `PAYLOAD_VER` is the lever that moves it.** See +[contracts](06-contracts.md). + +## The estate + +Four machines on a WireGuard mesh (`10.10.0.0/24`): + +| Host | Mesh IP | Public | Runs | +|---|---|---|---| +| **prod** | `10.10.0.1` | `169.58.46.181`, thermograph.org | Docker **Swarm** app stack, TimescaleDB, Postfix, backups, Centralis | +| **beta** | `10.10.0.2` | `75.119.132.91`, beta.thermograph.org | **compose** app stack, Forgejo (git + CI + registry), Grafana + Loki | +| **desktop** | `10.10.0.3` | — | LAN dev stack, the Forgejo Actions runner, your editor | +| phone | — | — | alerts | + +Two consequences you will hit within the first week: + +- **Forgejo is mesh-only.** `git.thermograph.org` resolves publicly to beta's + IP, but beta's Caddy rejects `/v2/*` (the registry API) from outside the mesh. + Any host that pulls images needs `10.10.0.2 git.thermograph.org` in + `/etc/hosts`. +- **Prod and beta run different orchestrators.** Prod is Swarm + (`infra/deploy/stack/`), beta and LAN dev are compose + (`infra/docker-compose.yml`). `deploy.sh` routes between them by reading + `/etc/thermograph/deploy-mode` on the box. Most of the time you don't care — + but when you're reading logs or naming containers, you do + (see [observability](09-observability.md)). + +## How a change travels + +``` +PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box + │ + promotion PR + ▼ + main ──▶ beta.thermograph.org + │ + promotion PR ← the owner's call + ▼ + release ──▶ thermograph.org +``` + +`dev`, `main` and `release` are **protected** — no direct pushes, for humans or +agents. Promotion is by pull request, continuously and per-decision; there is +no release calendar. Full detail in [CI and release](07-ci-and-release.md). + +## The four rules that are not negotiable + +These come from the operator, via Centralis's own onboarding. They override +convenience. + +1. **Everything is a PR.** If it didn't go through the pipeline, it didn't + happen. This includes Grafana dashboards (provisioned from repo JSON — a UI + edit is silently overwritten) and secrets (SOPS vault only). +2. **Boring prod.** Anything clever proves itself on `dev` or beta first. +3. **Privacy is a feature.** No tracking cookies, no per-visitor IDs, + allowlisted anonymous counters only (`backend/core/metrics.py`, + `POST /api/v2/event`). Cheap to keep, expensive to retrofit. +4. **Grades are relative.** Never describe a percentile as hot or cold. + +### And a fifth, local to the repo + +**Commits and PRs are concise and technical, and never mention AI, assistants +or automated authorship.** Every domain `CLAUDE.md` says this; it applies to +you too. + +## Quota discipline + +Thermograph runs on free upstream data sources. Two hard rules protect them: + +- **Anything named "prefetch" or "warm" must never spend the Open-Meteo + quota.** The warming paths call `climate.load_cached_history` / + `load_cached_recent_forecast` — the cache-only loaders — and skip cells that + aren't warm. A cell self-heals on first real request instead. +- **Nominatim gets at most ~1 request/second.** All reverse geocoding funnels + through a single dedicated worker thread + (`climate._revgeo_worker`) that paces itself; never call it from a request + thread. + +Breaking either one gets the whole service rate-limited, which is why +`singleton.claim_leader()` exists: a background sweep running in N uvicorn +workers × M hosts multiplies quota use by N×M. That failure has already +happened once, in production, at 3 workers. + +## Where the written record lives + +| Question | Where | +|---|---| +| How does this repo work? | The domain `CLAUDE.md` files, then this set | +| Why is the architecture like this? | `thermograph-docs` (ADRs) — `docs_search` | +| How do I operate the fleet? | `thermograph-docs` runbooks, plus the `thermograph-ops` Centralis skill | +| What did someone find out last Tuesday? | `notes_search` | +| What is and isn't live yet? | [`CUTOVER-NOTES.md`](../../CUTOVER-NOTES.md) at the repo root | + +Next: [Local setup](02-setup.md). diff --git a/docs/onboarding/02-setup.md b/docs/onboarding/02-setup.md new file mode 100644 index 0000000..52aa099 --- /dev/null +++ b/docs/onboarding/02-setup.md @@ -0,0 +1,227 @@ +# 2. Local setup + +Every command in this document was run against this checkout and produced the +output shown. If one fails for you, the difference is your machine, not the doc. + +## Toolchain + +| Tool | Why | Notes | +|---|---|---| +| **Python 3.12** | backend runtime and test suite | Must have the `_sqlite3` extension. A pyenv 3.10 without it will fail at `conftest` import. `scripts/test.sh` looks for `python3.12` (or uses `uv`) rather than whatever `python3` is on `PATH` — deliberately. | +| **Go 1.26** | frontend service and the backend daemon | Both are static `CGO_ENABLED=0` builds. | +| **Docker** | images, the compose/Swarm stacks, smoke tests | Docker CLI ≥ 27 is fine. | +| **uv** *(recommended)* | fast venv creation | `scripts/test.sh` uses it when present. | +| **shellcheck v0.11.0** | matches CI's pin exactly | Install to `~/.local/bin/shellcheck`. A *different* version can invent new findings and fail CI on an unrelated push. | +| **sops + age** | reading/editing the secrets vault | Only needed if you touch `infra/deploy/secrets/`. | +| **jq** | the repo's `.claude` hooks use it | Without it the prod-guard hook fails toward *asking*, which is safe but noisy. | + +Check what you have: + +```bash +which go python3.12 uv docker sops age jq shellcheck +go version && python3.12 --version && docker --version +``` + +## Backend + +### Test suite (hermetic, no network, no Docker) + +```bash +cd backend +make test # or: ./scripts/test.sh +make test ARGS='tests/data -q' # pass pytest args through +``` + +Verified: **429 passed, 8 skipped in 11.57s** on a cold venv build. + +The first run builds `.venv-test/` on Python 3.12 and installs +`requirements-dev.txt`. `tests/conftest.py` is what makes the suite hermetic: + +- no real Open-Meteo, Nominatim or GeoNames calls (the places index is marked + as already-loaded so no background download starts); +- a throwaway SQLite accounts DB and derived store in `/tmp`, never the repo's + `data/`; +- audit/error/access/activity/heartbeat log dirs redirected to `/tmp`; +- notifier and heartbeat threads disabled; +- `THERMOGRAPH_FRONTEND_BASE_INTERNAL` pointed at an unreachable placeholder + (the app fails loud at import without it). + +### Boot it locally, no Docker, no Postgres + +The backend falls back to SQLite when `THERMOGRAPH_DATABASE_URL` isn't a +Postgres URL, so this is enough: + +```bash +cd backend +THERMOGRAPH_FRONTEND_BASE_INTERNAL=http://127.0.0.1:8080 \ +THERMOGRAPH_BASE=/thermograph \ +THERMOGRAPH_ENABLE_NOTIFIER=0 \ +THERMOGRAPH_ENABLE_HEARTBEAT=0 \ + .venv-test/bin/python -m uvicorn app:app --host 127.0.0.1 --port 8137 +``` + +Verified responses: + +``` +GET /healthz → {"status":"ok","role":"all"} +GET /thermograph/api/version → {"backend_version":"2","min_frontend":"1","payload_ver":"p2"} +``` + +`app:app` is a one-line re-export shim for `web/app.py`. Keep it — systemd, CI +and the container entrypoint all target that name. + +Note the port: **8137** is the backend everywhere in this project (compose, +Caddy, the smoke harness at 18137, the frontend's default internal base). + +### Image smoke test + +```bash +cd backend +make smoke # builds the image, boots it + a throwaway TimescaleDB (tmpfs), + # asserts /healthz and /api/version +``` + +Uses `docker-compose.test.yml` on host port 18137 so it can't collide with a +dev server on 8137. + +## Frontend + +**The live frontend is Go.** `frontend/server/` is what builds, tests, ships +and runs. The Python files one level up (`app.py`, `content.py`, +`api_client.py`, `format.py`) are the superseded original — see +[traps](11-traps.md). + +### Test + +```bash +cd frontend/server +go build ./... && go vet ./... && go test ./... +``` + +Verified: all seven packages `ok` (`config`, `content`, `contentapi`, +`contentdata`, `format`, `handlers`, `render`). + +These same commands run inside `frontend/Dockerfile`'s builder stage — plus a +`gofmt -l` check that must come back empty — so **a failing Go test fails the +image build**, which is how CI catches it. There is no separate frontend test +step in the workflows. + +### Run it + +The process resolves `static/` and `content/` **relative to its working +directory**, so build in `server/` and run from `frontend/`: + +```bash +cd frontend/server && go build -o thermograph-frontend . +cd .. +THERMOGRAPH_API_BASE_INTERNAL=http://127.0.0.1:8137 \ +THERMOGRAPH_BASE=/thermograph \ +PORT=8080 \ + ./server/thermograph-frontend +``` + +Verified: `GET /healthz` → `{"status":"ok"}`, with a structured JSON log line +per request. + +`THERMOGRAPH_API_BASE_INTERNAL` is **required** — boot fails loudly without it, +by design. Optional: `THERMOGRAPH_BASE` (default `/thermograph`; the image sets +`/`), `THERMOGRAPH_API_VERSION` (default `v2` — only ever change it per the +[API-version contract](06-contracts.md)), `THERMOGRAPH_API_BASE_PUBLIC`, +`THERMOGRAPH_SSR_CACHE_TTL` (seconds, default 600), `THERMOGRAPH_GOOGLE_VERIFY`, +`THERMOGRAPH_BING_VERIFY`, `PORT` (default 8080). + +### Frontend against a real backend container + +```bash +cd frontend +make backend-up # pulls + runs the published backend image + throwaway db + # on 127.0.0.1:18137, waits for /healthz, prints the URL +make backend-down +``` + +The image tag is **derived from your checkout** — `sha-<12hex of git log -1 -- +backend/>`, the same domain-keyed rule `build-push.yml` and `deploy.yml` use — +so the harness follows the tree. If those backend commits are still local-only, +no image exists yet and the script says so; pin a published build with +`THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex>`. + +`make test-integration` runs the Python integration tier against that. CI does +**not** run it — it needs a live backend container. + +> Heads-up: against a freshly-booted throwaway backend, that tier currently +> fails 7 of 16 with `503` — the database is empty, so nothing is warm. +> Pre-existing, and unrelated to which image tag you use. + +## The daemon + +```bash +cd backend/daemon +go build ./... && go vet ./... && go test ./... + +THERMOGRAPH_INTERNAL_TOKEN=dev-token \ +THERMOGRAPH_API_BASE_INTERNAL=http://localhost:8137 \ + go run . +``` + +It refuses to start without `THERMOGRAPH_INTERNAL_TOKEN` — and the backend +answers `404` on the whole `/internal/*` surface when that token is unset. Both +ends fail closed. With Discord unconfigured it logs once and runs cron-only. + +## The full stack, locally + +```bash +cd infra +make dev-up # docker-compose.yml + docker-compose.dev.yml overlay: + # uncapped CPU, backend published on 0.0.0.0:8137 for the LAN +make dev-down +``` + +The dev overlay exports `COMPOSE_PROJECT_NAME=thermograph-dev` so the LAN stack +keeps volumes separate from anything else. **Do not remove either half of the +project-name pinning** — `infra/docker-compose.yml` pins `name: thermograph`, +and without it running compose from `infra/` derives the project name `infra`, +silently creating a *new* stack beside the running one with fresh volumes. + +Other `infra/Makefile` targets: `up`/`down` (pull + run the published images), +`db-up`/`db-down` (just Postgres, e.g. to run the app from a venv against it), +`om-up`/`om-down`/`om-backfill` (the self-hosted Open-Meteo overlay — the +backfill writes ~1–1.5 TB and takes hours). + +## Connectors (Centralis and friends) + +The fleet is not reachable from a laptop off the WireGuard mesh, and Forgejo is +mesh-only. **Centralis** is the control plane that fronts all of it — the app +database, the ERA5 lake, fleet logs, Grafana, Forgejo, docs and notes, and +Discord. + +```bash +claude mcp add --transport http centralis https://mcp.thermograph.org/mcp \ + --header "Authorization: Bearer $CENTRALIS_TOKEN" +``` + +Ask the operator for a token; don't share it. Verify with *"what's running on +prod right now?"* — it should call `fleet_status` and list the Swarm services. + +Also worth installing locally: **Chrome DevTools MCP** (design verification +needs a real browser on your machine) and **Figma**. Grafana's official MCP +server is optional and read-only — but remember dashboards are provisioned from +repo JSON, so a durable change is still a PR via `dashboard_write`. + +Run `mcp__centralis__onboarding` for the current, authoritative connector list; +it will be fresher than this page. + +## Repo-local guardrails + +`.claude/settings.json` wires three hooks that travel with the checkout: + +| Hook | When | What | +|---|---|---| +| `prod-guard.sh` | before Bash / live-host MCP calls | Classifies by **allowlist**: only positively-recognised read-only commands pass; everything else asks. Beta is guarded as strictly as prod — it hosts Forgejo, Grafana *and* beta.thermograph.org. | +| `secrets-guard.sh` | before Write/Edit | Denies any direct write to `infra/deploy/secrets/*.yaml`. Use `sops edit`. | +| `lint-after-edit.sh` | after Write/Edit | shellchecks an edited `*.sh` and feeds findings straight back. Exits quietly if shellcheck is missing. | + +They enforce what `CLAUDE.md` can only ask for. If you change `prod-guard.sh`'s +classifier, re-read `.claude/hooks/README.md` first — it documents a real +silent-total-bypass failure mode in the parsing loop. + +Next: [Repo map](03-repo-map.md). diff --git a/docs/onboarding/03-repo-map.md b/docs/onboarding/03-repo-map.md new file mode 100644 index 0000000..65cebc4 --- /dev/null +++ b/docs/onboarding/03-repo-map.md @@ -0,0 +1,189 @@ +# 3. Repo map + +The monorepo was reunified on **2026-07-22** by subtree-merging four split +repos with full history (380 commits). The split's *properties* were kept +deliberately: per-domain images, per-domain deploys, and an async FE/BE +contract. That's why the layout looks like four projects sharing a root. + +``` +thermograph/ +├── CLAUDE.md root agent instructions — branch model, deploy contract +├── CUTOVER-NOTES.md what is and isn't live; the reunification record +├── README.md domain table + how CI stays decoupled +├── .forgejo/workflows/ ALL CI. Nine files. Never add workflows elsewhere. +├── .claude/ settings.json + enforcement hooks +├── backend/ FastAPI API + Go daemon → emi/thermograph/backend +├── frontend/ Go SSR + static assets → emi/thermograph/frontend +├── infra/ compose, Swarm, deploy, secrets, terraform, ops +├── observability/ Loki + Grafana + Alloy +└── docs/onboarding/ you are here +``` + +> **CI lives only in root `.forgejo/workflows/`**, path-filtered per domain. +> Forgejo only reads root workflows — a copy under a domain's own `.forgejo/` +> is inert, and becomes a trap when someone edits it expecting an effect. + +## `backend/` — the API service + +Python 3.12 / FastAPI. Owns climate data, grading, the API, accounts, +notifications, and the SSR-content JSON API the frontend renders from. Serves +**no** HTML/CSS/JS. + +``` +app.py one-line re-export shim for web/app.py — keeps `app:app` + stable for systemd/CI/entrypoint. Don't change the name. +paths.py every filesystem location, resolved from the repo root. + NEVER reintroduce __file__-relative paths in a module. +web/app.py (1022) the real FastAPI app: routes, CORS, ETag, middleware, + lifespan, role gating, the frontend proxy fallback +api/ + payloads.py (342) pure inputs → response dict builders. One definition per + payload shape; PAYLOAD_VER lives here. + content_payloads.py the SSR content API's builders (JSON-safe, no Markup) + content_routes.py route wiring for /content/* + the same ETag pattern + internal_routes.py /internal/* — the Go daemon's control surface. Fails + closed: no token configured ⇒ the whole surface 404s. + homepage.py (367) the precomputed "unusual right now" feed + sitemap.py +data/ + climate.py (1248) fetch + cache the daily record. The source ladder lives + here. The single biggest file in the repo. + grading.py (532) day-of-year climatology, percentiles, TEMP/RAIN_BANDS + grid.py (92) lat/lon → stable ~4 sq mi cell id + store.py (329) derived-payload store (SQLite/Postgres). Pure accelerator. + climate_store.py(353) raw climate record on TimescaleDB hypertables + era5lake.py (173) the ERA5 object-storage lake: layout, grid math, clients + scoring.py (293) climate-shift scoring (recent years vs full baseline) + places.py (355) typo-tolerant local place index for /suggest + cities.py, city_events.py, meteostat.py +core/ + metrics.py (805) since-start traffic counters, multi-worker via SQLite + audit.py (167) JSONL audit / errors / access / activity / heartbeat + singleton.py (129) leader election: flock (per host) or PG advisory lock + (per cluster). Read this before adding any timer. +accounts/ fastapi-users: models, schemas, db, users, api_accounts +notifications/ + notify.py (542) the subscription evaluation engine (background thread) + push.py, mailer.py, digest.py, discord.py, discord_link.py, + discord_interactions.py +daemon/ the Go daemon — Discord gateway + cron timers +alembic/ Postgres migrations (run by the entrypoint on boot) +deploy/entrypoint.sh container entrypoint: alembic migrate, then serve +lake_app.py (204) the prod-only `lake` service (DuckDB over the bucket) +tests/ the hermetic pytest suite +cities.json, cities_flavor.json bundled reference data (generated) +``` + +Standalone scripts at the root are tooling, not runtime: `warm_cities.py`, +`indexnow.py`, `seed_era5.py`, `gen_era5_lake.py`, `gen_cities.py`, +`gen_flavor.py`, `drift_check.py`, `migrate*.py`. + +Deep dive: [04-backend.md](04-backend.md). + +## `frontend/` — SSR + static assets + +``` +server/ ★ THE LIVE SERVICE (Go 1.26, module thermograph/frontend) + main.go config + mux + static + graceful shutdown + internal/config/ every env var, with the same names/defaults as before + internal/contentapi/ backend /content/* client: TTL cache, bounded LRU, + per-key single-flight, origin forwarding + internal/content/ page handlers, funcmap, SEO + internal/contentdata/ glossary.yaml + pages.yaml loader (fail-loud) + internal/format/ unit-aware °C/°F formatting + band names + internal/handlers/ routing, SPA shells, static + internal/render/ html/template over embed.FS + ETag helpers + internal/render/templates/*.tmpl the embedded page templates +static/ every static asset — app.js, cache.js, shared.js, + account.js, calendar/day/score/compare .js + .html + shells, style.css (all design tokens), icons, sw.js +content/ glossary.yaml, pages.yaml — structured SSR copy +templates/*.html.j2 Jinja templates — used by the PYTHON service only +tests/ Python tiers: unit (hermetic, fixtures) + integration +tests/fixtures/*.json golden payloads — ALSO consumed by the Go tests +tools/shoot.py the `make shots` screenshot sweep +DESIGN.md the visual system: tokens, components, breakpoints +app.py, content.py, ⚠ the superseded Python implementation — not deployed +api_client.py, (see traps) +format.py, paths.py, +content_loader.py +``` + +Deep dive: [05-frontend.md](05-frontend.md). + +## `infra/` — how and where images run + +``` +docker-compose.yml the compose stack (beta + LAN dev): + db, backend, lake, daemon, frontend. + `name: thermograph` is PINNED — see below. +docker-compose.dev.yml LAN overlay: uncapped, backend on 0.0.0.0:8137 +docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only) +Makefile compose orchestration only +deploy/ + deploy.sh ★ the single entry point for beta and prod + render-secrets.sh renders /etc/thermograph.env from the SOPS vault + secrets/ the vault: common.yaml, prod/beta/dev.yaml, + centralis.prod.yaml, example.yaml + stack/ the SWARM path (live on prod): + thermograph-stack.yml, deploy-stack.sh, + autoscale.sh, the LB + swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning + Caddyfile the reverse proxy (path-splits FE vs BE) + provision-*.sh host bootstrap: agent access, dev LAN, mail, secrets + db/init/ TimescaleDB init + tuning + migrations/ hand-run SQL migrations +ops/ + dbq.sh read-only psql into any environment's db container + iceberg.sh duckdb over the ERA5 Iceberg lake +terraform/ host provisioning — NO STATE IS PERSISTED +ACCESS.md, DEPLOY.md, DEPLOY-DEV.md (partly stale — see traps) +``` + +**Why the project name is pinned.** Compose creates volumes +`thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stack declares +those exact names as `external: true` at the same mount paths. Running compose +from `infra/` without `name: thermograph` derives project `infra`, which makes a +whole new stack with empty volumes next to the running one. `deploy-dev.sh` +exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to +keep LAN dev separate on purpose. Keep both halves. + +Deep dive: [08-infra-secrets.md](08-infra-secrets.md). + +## `observability/` — the logging stack + +``` +docker-compose.yml Loki + Grafana (runs on beta) +loki/config.yml mesh-only, filesystem storage, 30-day retention +grafana/provisioning/ datasource + dashboard provider (auto-loaded) +grafana/provisioning/alerting/ rules, the Discord contact point, the policy +grafana/dashboards/*.json the fleet-logs dashboard +alloy/config.alloy the per-node shipper +alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host) +caddy-grafana.conf beta's Grafana vhost, kept for reference +``` + +No build, no deploy automation — it ships by hand. **The repo is the only +durable path**: Grafana is provisioned from this directory at startup, so UI +edits are overwritten. Grafana is at **`dashboard.thermograph.org`** (Google +SSO, pre-provisioned users only) — never `grafana.thermograph.org`. + +Deep dive: [09-observability.md](09-observability.md). + +## The nine CI workflows + +| Workflow | Trigger | Does | +|---|---|---| +| `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. | +| `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. | +| `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. | +| `deploy.yml` | push to `main`/`release` touching an app domain | Branch selects environment; matrix covers services; SSHes and runs `deploy.sh`. | +| `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards each host's checkout and re-renders secrets. Rolls **no** service. | +| `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. | +| `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow. Uses `PROD_SSH_*`. | +| `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. | +| `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. | + +Deep dive: [07-ci-and-release.md](07-ci-and-release.md). + +Next: [Backend deep dive](04-backend.md). diff --git a/docs/onboarding/04-backend.md b/docs/onboarding/04-backend.md new file mode 100644 index 0000000..b6e59b4 --- /dev/null +++ b/docs/onboarding/04-backend.md @@ -0,0 +1,292 @@ +# 4. Backend deep dive + +FastAPI on Python 3.12, plus a Go daemon that ships in the same image. Owns +every piece of climate data and every bit of grading logic in the project. + +Read [`backend/CLAUDE.md`](../../backend/CLAUDE.md) alongside this — it is the +binding, terse version. + +## The one-process-many-roles model + +The backend image runs **five different jobs** depending on `THERMOGRAPH_ROLE` +and which binary compose/Swarm selects. One image means version skew between +them is impossible. + +| Service | How it's selected | Does | +|---|---|---| +| **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. | +| **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. | +| **all** | default | Both — the single-process dev default. | +| **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Prod only. | +| **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. | + +`deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres +volume can still be starting) and then serves. `/healthz` and `/api/version` +are deliberately **I/O-free** so they stay cheap under tight healthcheck +intervals; they're also deliberately **not** under `THERMOGRAPH_BASE`, so a +healthcheck needn't know the base path. + +## The data pipeline + +### Where history comes from — the source ladder + +`data/climate.py::_load_history` walks this ladder on a cache miss, and the +order matters: + +1. **The ERA5 lake** (`data/era5lake.py`) — our own parquet extract of the + public ERA5 archive in Contabo object storage. True ERA5 with measured + gusts, no third-party API in the path. Unconfigured or missing-point cells + fall through at zero cost. +2. **The Open-Meteo archive** — the same ERA5 family, so a backup-sourced + record grades consistently with lake-sourced neighbours. Guarded by its own + rate-limit cooldown. +3. **Stale cache**, served without rewriting it (so `recent_stamp` stays old + and the next request retries upstream first). +4. Otherwise `WeatherUnavailable` → a clean 503. + +**NASA POWER is retired from every serving path.** Its MERRA-2 record ran +~1.3 °F off ERA5 and carried no gusts. `drift_check.py` still fetches it, and +only that. + +Every source must return a plausibly-full span (`MIN_ARCHIVE_DAYS`) before it's +accepted and cached as complete — a short response is rejected, not stored. + +### Where "today" comes from + +`_load_recent_forecast` is a separate ladder: + +1. **Open-Meteo forecast API** — recent past *and* forward days in one call, + with gusts, ERA5-consistent with the history record. Skipped while its own + cooldown is active. +2. **Fallback** — an archive recent-past range plus **MET Norway** forward days. + Costs no Open-Meteo forecast quota. +3. Stale cache, again without rewriting. + +Cached per cell for `FORECAST_TTL_HOURS`, so it tracks model updates without +hourly upstream I/O. `RECENT_PAST_DAYS = 25`, `FORECAST_DAYS = 8`. + +### Concurrency and quota guards + +- A per-cell lock serialises archive fetches, so N concurrent first-requests for + one cell cost one fetch. +- Rate limits set module-level cooldowns (`_archive_cooldown_until`, + `_forecast_cooldown_until`) rather than retrying into the wall. +- Reverse geocoding runs on **one dedicated worker thread** draining a queue at + ~1 req/s. The earlier design slept inside the caller's thread — which, in the + server, is one of Starlette's shared sync-threadpool threads. + +## Storage: three tiers with three different contracts + +| Tier | Module | Backend | Contract | +|---|---|---|---| +| **Raw climate record** | `data/climate_store.py` | TimescaleDB `climate_history` hypertable (durable), `climate_recent` (rewritten each refresh), `climate_sync` (freshness) | **Source of truth.** Expensive to refetch. Fail-soft: a miss degrades to *fetch-from-upstream*. Parquet files instead when `THERMOGRAPH_DATABASE_URL` isn't Postgres (dev, tests, offline tooling). | +| **Derived payloads** | `data/store.py` | SQLite WAL, or **UNLOGGED** Postgres tables | **Pure accelerator.** Every reader falls back to recomputing from the raw record; every helper swallows its own errors. Deleting the store is a safe full reset. | +| **Accounts** | `accounts/db.py` | Postgres (or SQLite) via SQLAlchemy | **Not regenerable.** Foreign keys enforced, errors surface rather than get swallowed, must be backed up. | + +That third row is the important distinction: accounts, subscriptions and +notifications have no source to recompute from, so they live under stricter +rules than everything else in the process. + +On Postgres, `accounts/db.py` runs **two async engines** over one primary — +read-only (`default_transaction_read_only`) for pure-GET endpoints, read-write +for everything else — plus a separate **sync** psycopg engine for the notifier +thread, which has no event loop. + +### The validity token + +Derived rows are keyed `(kind, cell_id, request_key)` and guarded by a `token` +that encodes everything the payload depends on: + +```python +PAYLOAD_VER = "p2" # api/payloads.py +history_token(history) → f"{PAYLOAD_VER}:{hist_end(history)}" +content_token(...) → f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date}" +``` + +A stored token that doesn't match the caller's current token is a **miss**. +There is no clock-based expiry anywhere. That's why one `PAYLOAD_VER` bump +atomically orphans every pre-upgrade row — and why forgetting to bump it serves +old-shaped JSON forever. `CONTENT_VER` is kept separate so a content-only shape +change doesn't invalidate the whole grading cache. + +The same token doubles as a **weak ETag**. See [contracts](06-contracts.md) for +what that means for CORS. + +## Grading + +`data/grading.py` is the heart of the product: + +- For a target day of year, the reference distribution is every historical day + within **±`HALF_WINDOW` (7) days**, wrapping the year boundary — a 15-day + seasonal window. +- The observed value is placed as an **empirical mid-rank percentile**, which + handles ties correctly (crucial: most days have zero precipitation). +- `TEMP_BANDS` is a symmetric seven-tier ladder; `RAIN_BANDS` is separate, and + dry days are coloured by dry-streak length instead of rank. +- `pct_ordinal()` floors a percentile into `1..99` — never 0, never 100. + `frontend/static/shared.js::pctOrd()` mirrors it. If they diverge, the same + reading says two different things on two surfaces. + +`data/scoring.py` is a different question: how far a cell's **recent years** +have drifted from its full multi-decade baseline, per metric and per +percentile category, expressed in unit-free percentile points. Note +`baseline_overlaps_recent` in the payload — the baseline deliberately includes +the recent years, which mildly and uniformly attenuates the divergence. + +## The HTTP surface + +`web/app.py` mounts routers so that every version stays served simultaneously: + +``` +/api → v1 (legacy unversioned alias) +/api/v1 → v1 geocode, grade +/api/v2 → v2 geocode, suggest, place, grade, calendar, day, + score, forecast, cell, metrics, event +/api/v2 → content_routes: /content/hub, /content/sitemap, + /content/indexnow-key, /content/home, + /content/city/{slug}[/month/{month}|/records] +/api/v2/auth, /users fastapi-users: login, logout, register, verify, me +/api/v2/subscriptions, /notifications, /push/* accounts/api_accounts.py +/api/v2/discord/* account linking (OAuth2) +/internal/* the Go daemon's control surface — NOT under BASE +/healthz, /api/version I/O-free +/discord/interactions Discord slash-command webhook (Ed25519-verified) +/digest the footer signup form +/{path:path} catch-all proxy to the frontend ← registered LAST +``` + +Two routes worth understanding properly: + +**`GET /api/v2/cell`** — one bundle carrying every view's payload for a cell, so +the client warms four views with one request. Each slice is the *exact* payload +its per-view endpoint returns, built by the same builders under the same +derived-store keys and tokens, paired with the ETag that endpoint would emit. +The client seeds its per-view cache from the slices and later revalidates each +view individually. `prefetch=1` is the warm-only mode with a hard guarantee: +**it never spends weather-API quota** — a cold cell answers `204` with no body. +`neighbors=1` additionally enqueues the 8 surrounding cells for background +warming, also warm-only. + +**The catch-all proxy** — the frontend owns every page and asset. In prod and +beta, Caddy path-splits directly, so this proxy is never exercised. It exists as +the fallback for environments with no Caddy in front (LAN dev, bare-metal). It +forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute +URLs instead of resolving to the internal hop's own address. Because it's +registered dead last, every real backend route wins first — +`/internal/*` in particular is registered before it, deliberately. + +## Background work: who runs what, and where + +This is where the project's hardest-won lessons live. + +- **Neighbour warmer** — a per-worker daemon thread draining a request-fed + queue. Correct to run in every worker: each drains its own queue. +- **Subscription notifier** (`notifications/notify.py`) — a timer-driven sweep + that issues upstream fetches independent of any request. Running it in N + workers multiplies Open-Meteo quota use N-fold. That is not hypothetical: it + is the documented cause of overnight 429/503s after prod went to 3 workers. + So it runs behind `singleton.claim_leader()`: + - `claim()` — a non-blocking `flock` on a lockfile, arbitrating workers **on + one machine**; + - `claim_pg()` — a Postgres advisory lock, visible to every host talking to + the same database, arbitrating **cluster-wide** under Swarm; + - unset (dev, tests, single worker) ⇒ always the leader. +- **Heartbeat** — deliberately **unguarded** across every worker and replica. A + beat is one local file append (no quota, no DB, no lock), so duplicates are + free, and only the process actually being dead stops them. Container-log + silence can't distinguish a live web/worker process from a dead one, so this + is the signal observability alerts on. +- **Discord gateway + recurring jobs** — moved *out* of this process entirely + into the Go daemon. See below. + +The notifier's own quiet guards: a `UNIQUE(subscription_id, event_date, metric, +direction, kind)` row-level dedup, and a **weekly cap** — a subscription that +notified within 7 days is skipped entirely. Archive reads come from cache; a +freshly-subscribed cell is fetched **once**, budget-capped per pass. + +## The Go daemon (`backend/daemon/`) + +One binary, built into the backend image, running as exactly one replica. + +**Why it exists:** the Discord gateway tolerates one session per token, and +duplicated timers repeat every job. One replica makes that a *deployment fact* +instead of a runtime election. + +**Why it's thin:** Go owns no climate or grading logic. Grading needs polars and +the cache, and the slash-command path deliberately shares one grade builder with +the API so bot grades and API grades can never drift. So anything data-shaped is +a POST back into Python: + +| Route | Purpose | +|---|---| +| `POST /internal/discord/grade` | `{"query": "phoenix"}` → gateway-ready Discord message JSON, relayed verbatim | +| `POST /internal/jobs/warm-cities` | trigger the warm-cities job | +| `POST /internal/jobs/indexnow` | trigger IndexNow check-and-submit | + +Every request carries `X-Thermograph-Internal-Token`. **Both ends fail closed**: +with no `THERMOGRAPH_INTERNAL_TOKEN`, Python answers 404 across the whole +surface and the daemon refuses to start. Caddy never routes `/internal/*` +publicly, so the token is defence in depth, not the only control. + +The cron half runs unconditionally (`warm-cities` every 24h, `indexnow` every +6h, both configurable). A fatal gateway error logs loudly and leaves cron +running — a Discord misconfiguration must not become a crash loop that also +stops the schedule. The cron scheduler defers the first run by one interval, +never overlaps a job with itself, and drops ticks that fire mid-run. + +## The lake (`lake_app.py` + `data/era5lake.py`) + +Prod-only Swarm service, same image, `THERMOGRAPH_ROLE=lake`. + +The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`), +extracted once from the public Earthmover ERA5 Icechunk archive by +`gen_era5_lake.py`. Two layouts under `era5/`: + +- `daily/tile=_/year=/month=/part.parquet` — the analytical hive + table. The location grain is a 12×12-point tile (~3°×3°) aligned to the + source's chunk grid; a per-point-per-month grain would be ~170M objects. +- `points/lat=/lon=.parquet` — the serving projection: one file per grid + point with its full history, so the hot path is a single GET. +- `manifest.parquet` — an index of every point file; readers prune with it and + the extractor resumes from it. + +Endpoints: `/healthz`, `GET /history?lat=&lon=` (the hot path — nearest point's +history as parquet bytes, disk-cached per point), `POST /query` (SELECT-only +DuckDB SQL over `era5_daily` and `manifest`). + +Two operational facts: **Contabo is path-style only** with `region=default` +(virtual-host addressing fails outright), and DuckDB's `httpfs`/`iceberg` +extensions are baked into the image **as uid 10001**, because extensions install +to the invoking user's `~/.duckdb` and a root-time bake left the runtime user +unable to `LOAD` them — seen live as prod's lake `/query` returning 500. + +Filter on partition columns (`tile`/`year`/`month`). That predicate is the +difference between a pruned scan and reading the whole warehouse. + +## Observability hooks inside the app + +- `core/audit.py` writes JSONL streams: `audit` (one line per graded run, with + per-phase timings and a `full`/`partial` run type), `errors` (upstream + failures tagged `retry` or `error`), plus access, activity and heartbeat. + Alloy parses these so `level`/`tag`/`phase` become Loki labels. +- `core/metrics.py` keeps since-start counters, shared across uvicorn workers + via `THERMOGRAPH_METRICS_DB` (a SQLite file) — unset keeps the + zero-dependency in-memory store. Read over the token-gated + `GET /api/v2/metrics`. +- Every unhandled 500 gets an explicit exception handler emitting + `tag="http.error"`. Without it the exception propagates past the middleware + and the 500 is invisible in both the access log and metrics. + +## Working on it + +```bash +cd backend +make test # hermetic pytest, ~12s +make test ARGS='tests/data/test_grading.py -q' +make smoke # build the image, boot it + a throwaway db +``` + +CI runs the suite **inside the built image**, so the exact interpreter and +dependencies that ship are what gets tested. + +Next: [Frontend deep dive](05-frontend.md). diff --git a/docs/onboarding/05-frontend.md b/docs/onboarding/05-frontend.md new file mode 100644 index 0000000..c1dd17e --- /dev/null +++ b/docs/onboarding/05-frontend.md @@ -0,0 +1,221 @@ +# 5. Frontend deep dive + +The **SSR + static-asset service**: server-rendered crawlable pages, the +interactive tool's SPA shells, and every static asset. No climate data, no +database, no compute — everything comes from the backend's content API over +HTTP. + +> **The live service is Go.** `frontend/server/` builds, tests, ships and runs. +> The Python files one directory up (`app.py`, `content.py`, `api_client.py`, +> `format.py`, `content_loader.py`, `paths.py`, `templates/*.html.j2`) are the +> superseded original implementation, kept in-tree as the reference the port was +> made from. Nothing deploys them, and they hold duplicate copies of a couple of +> contracts (`api_client.py`'s `API_VERSION` pin, `format.py`'s `F_COUNTRIES`) — +> keep those in step if you run them locally. Whether to delete them is an open +> question; raise it rather than doing it in passing. + +## Shape of the Go service + +``` +main.go config → content client → glossary/pages → mux → + listen → graceful shutdown +internal/config/ every env var the service reads, with the same names, + defaults and required/optional split as the Python +internal/contentapi/ the backend /content/* client +internal/content/ the SSR page handlers, template funcmap, SEO helpers +internal/contentdata/ glossary.yaml / pages.yaml loader +internal/format/ unit-aware °C/°F formatting + band names +internal/handlers/ SPA shells + static serving +internal/render/ html/template over an embed.FS + ETag helpers +internal/render/templates/*.tmpl the page templates (embedded in the binary) +``` + +Dependencies: **stdlib plus `gopkg.in/yaml.v3`** only. The committed SSR copy +in `frontend/content/*.yaml` is shared with the rest of the repo and uses +block/folded scalars, so a real YAML parser is genuinely required. + +**Templates are embedded in the binary; `static/` and `content/` are read from +disk relative to the working directory** (`frontend/` locally, `/app` in the +image). That's why you build in `server/` and run from `frontend/`. + +### Fail-loud boot + +`main.go` exits at startup on bad configuration *or* bad content: a missing +backend URL, a garbage TTL, malformed glossary/pages YAML. The Python raised at +import for the same cases, and the reasoning is identical — a bad content edit +should break the boot, not silently 500 on the first request. + +The one deliberate exception is the IndexNow key: boot tries the key fetch once, +catches failure, and falls back to a lazy per-request lookup. **Boot must +survive an unreachable backend** — asynchronous FE/BE deploys depend on it. +Don't make it fatal. + +## Routes + +Everything below sits under `THERMOGRAPH_BASE` (default `/thermograph`; the +image sets `/`). + +| Path | Kind | +|---|---| +| `/` | SSR homepage | +| `/climate` | SSR climate hub | +| `/climate/{slug}` | SSR per-city page | +| `/climate/{slug}/records` | SSR all-time records | +| `/climate/{slug}/{month}` | SSR per-month page | +| `/glossary`, `/glossary/{term}` | SSR | +| `/about`, `/privacy` | SSR | +| `/robots.txt`, `/sitemap.xml` | generated | +| `/{indexnow-key}.txt` | the IndexNow ownership file | +| `/calendar`, `/day`, `/score`, `/compare`, `/legend`, `/alerts` | SPA shells | +| `/healthz` | liveness, I/O-free (does **not** prove the backend is reachable) | +| everything else | static assets | + +The static mount is registered last so the explicit routes win. + +**The backend owns the page metadata, not this service.** `page_title`, +`canonical_path`, `breadcrumb` and `jsonld` come from +`backend/api/content_payloads.py`. Don't recompute them here. + +### SPA shells and `__ORIGIN__` + +The shell HTML files in `static/` carry an `__ORIGIN__` placeholder for the +link-preview/Open Graph tags — preview crawlers need absolute URLs and the host +differs between LAN and prod. The substituted HTML and its ETag are **memoized +per origin**, not recomputed per request. + +Origin resolution prefers `X-Forwarded-Host` over `Host`. That matters when the +request arrives through the backend's catch-all proxy fallback: without it, the +page would advertise the internal hop's own address as canonical. + +## The content client + +`internal/contentapi` is a careful piece of code and worth reading in full +before you change caching behaviour. It has four properties, all load-bearing: + +1. **TTL cache** (`THERMOGRAPH_SSR_CACHE_TTL`, default 600s). Climate data + changes at most hourly, so without this a page-view burst on one city costs + one backend round trip per request. This is *in addition to* the backend's + derived-store/ETag caching — this one saves the network hop, the backend's + saves the recompute. +2. **Bounded LRU** (2048 entries). `city()` and `city_records()` fold the + browser-facing `origin` into the cache key, and `origin` is client-controlled + via `Host`/`X-Forwarded-Host`. An unbounded map there is a cheap + memory-exhaustion vector. +3. **Per-key single-flight.** Without it, N concurrent requests missing the same + key (cold start, or right after TTL expiry on a hot city) each fire their own + backend call — a thundering herd. +4. **Origin forwarding.** The original browser-facing origin is passed through as + `Host` / `X-Forwarded-Proto` so the backend builds JSON-LD `url` fields + against the public origin rather than the internal `http://backend:8137` + address. + +## Static assets + +Hand-written, no build step, no bundler, no framework. + +| File | Role | +|---|---| +| `app.js` | the main tool: map, search, results, inline SVG chart | +| `calendar.js` / `day.js` / `score.js` / `compare.js` | the SPA views | +| `cache.js` | IndexedDB cache + the `/cell` bundle prefetch + `If-None-Match` revalidation | +| `account.js` | auth; **pins `API_VERSION` and the `uv(path)` helper** | +| `shared.js` | shared helpers; **`pctOrd()` mirrors the backend's `pct_ordinal()`** | +| `units.js` | `F_REGIONS` — must match the backend's `F_COUNTRIES` | +| `chart.js`, `mappicker.js`, `filtersheet.js`, `nav.js`, `push-client.js`, `ios-install.js`, `digest.js`, `climate.js`, `subscriptions.js` | the rest | +| `style.css` | **the single hand-written stylesheet — every design token lives here** | +| `sw.js`, `manifest.webmanifest`, icons | PWA | + +`cache.js` is where the ETag contract actually lands: it fetches +`/api/v2/cell` once per view-set, slices it, and later revalidates each view +with `If-None-Match`. The slice→view map must track the backend's payload +shape. See [contracts](06-contracts.md). + +Static responses are served with `Cache-Control: public, max-age=300` — short +on purpose, because asset filenames are **not** content-hashed. It's a +revalidation window, not an immutable cache. + +## The design system + +[`frontend/DESIGN.md`](../../frontend/DESIGN.md) is the source of truth and +wins over any general design guidance. The essentials: + +- **Never hardcode a colour.** Every colour is a CSS custom property in + `style.css`'s `:root`, with a `@media (prefers-color-scheme: light)` override + right below. Always `var(--token)`. +- **Grade palettes encode meaning, not decoration.** Temperature is a 9-step + diverging, colourblind-safe scale (`--rec-cold` … `--normal` green midpoint … + `--rec-hot`). Precipitation is `--dry` plus `--wet-1…9`. Keep the order and + midpoints intact. +- **Inter is not self-hosted.** It renders where the OS has it and falls back to + `system-ui`. Adding a webfont link is a deliberate decision, not a tweak — it's + a network dependency on every page. +- **Inputs are `font-size: 16px` minimum.** Smaller text makes iOS zoom on + focus. Hard rule. +- **Mobile-first.** The base stylesheet is the phone layout; wider screens layer + on via `min-width`. Primary breakpoint 640/641px. The content column widens in + real steps at 1680 / 2400 / 3400px — it does *not* stay a centred 1200px strip. +- **Metric order is always `Precip · High · Low`**, everywhere. +- **Dark is the default**, light comes from `prefers-color-scheme`. Every change + must look right in both. Honour `prefers-reduced-motion`. +- Charts are bespoke inline SVG. No canvas, no charting library. + +### Verifying a visual change + +**Validate at 390 / 800 / 1920 / 2560 / 3840px in both light and dark.** These +straddle every breakpoint above. + +```bash +cd frontend +.venv/bin/pip install -q -r tools/requirements.txt +.venv/bin/python -m playwright install chromium +.venv/bin/python tools/shoot.py # the full matrix +.venv/bin/python tools/shoot.py index --width 390 --scheme dark # while iterating +``` + +PNGs land in `.screenshots/` (gitignored) as `{page}@{width}-{scheme}.png`. +Point it elsewhere with `SHOTS_BASE`. Chrome DevTools MCP is the other good +option here — a real browser on your own machine. + +## Tests + +```bash +cd frontend/server && go build ./... && go vet ./... && go test ./... +``` + +The Go tests read two directories via a fixed relative path: the committed +golden fixtures (`frontend/tests/fixtures/*.json` — the same set the Python +golden-diff comparison used) and the SSR copy (`frontend/content/*.yaml`). +`frontend/Dockerfile`'s builder stage copies them to container-root paths to +preserve that relationship, and runs `gofmt -l` + `go vet` + `go test` **before** +building. A failing Go test therefore fails the image build, which is how CI +catches it — there is no separate frontend test step in the workflows. + +The Python tiers still exist and still run locally: + +```bash +cd frontend +make test-unit # hermetic: SSR rendering fed committed fixtures +make test-integration # pulls + runs the real backend image; local only +make capture-fixtures # refresh tests/fixtures/*.json from a live backend +``` + +Treat `make capture-fixtures` as the shared operation: those fixtures feed the +**Go** tests too. + +## The image + +`python:3.12-slim` → gone. The frontend image is: + +- a `golang:1.26` builder that runs `gofmt -l` / `go vet` / `go test`, then + builds a static `CGO_ENABLED=0 -trimpath -ldflags="-s -w"` binary; +- an `alpine:3.22` final stage with **bash** (the Swarm stack bind-mounts a + bash entrypoint shim over the image's entrypoint) and **curl** (the + healthcheck); +- **uid 10001**, matching the backend image — that's the uid infra provisions + readable secrets for. Don't change it. +- `static/` and `content/` copied to `/app`, read-only at runtime. + +There is no `WORKERS` knob any more: uvicorn needed a process count, the Go +server handles concurrency in one process. + +Next: [Cross-service contracts](06-contracts.md). diff --git a/docs/onboarding/06-contracts.md b/docs/onboarding/06-contracts.md new file mode 100644 index 0000000..a0020b3 --- /dev/null +++ b/docs/onboarding/06-contracts.md @@ -0,0 +1,190 @@ +# 6. Cross-service contracts + +**Read this before any change that touches both domains.** + +Backend and frontend build separate images and deploy independently. That +independence is the point of the architecture — and it means there is no build +step, no type checker and no linker that will catch a mismatch between them. The +items below are the seams. Each one fails *silently and in production*, which is +why they're enumerated rather than left to judgement. + +## 1. API version — `GET /api/version` + +```json +{"backend_version": "2", "min_frontend": "1", "payload_ver": "p2"} +``` + +Driven by `API_CONTRACT_VERSION` and `MIN_SUPPORTED_FRONTEND` in +`backend/web/app.py`. + +**The rule for a breaking change:** ship a new `v3` router mounted **alongside** +`v2`, re-registering only the handlers that changed, and bump the constant in +the same PR. `/api` and `/api/v1` stay mounted as aliases. Every version stays +served simultaneously — that is how FE and BE ship out of lockstep at all. + +**The frontend pins the version in exactly two places:** + +| Where | What | +|---|---| +| `frontend/server/internal/config` (Go) — `THERMOGRAPH_API_VERSION`, default `v2` | every server-side path builder reads it | +| `frontend/static/account.js` (JS) — the exported `API_VERSION` and the `uv(path)` helper | every browser-side call | + +Never hardcode `api/v2/...` anywhere else. Bump both in one PR, and **only +after** the target backend's `/api/version` confirms it serves that version and +its `min_frontend` doesn't exclude the one you're leaving. + +> The Python `frontend/api_client.py` has a third pin. It isn't deployed, but if +> you're using it locally, keep it in step or you'll debug a phantom. + +## 2. `PAYLOAD_VER` — the cache invalidation token + +`backend/api/payloads.py`, currently `"p2"`. + +This is **separate from URL versioning** and does a different job: it's the +cache/ETag invalidation token. Every derived row is stored against a validity +token built from it, and a stored token that doesn't match the caller's current +token is simply a miss. + +**Bump it whenever a response payload's shape changes.** One bump atomically +orphans every pre-upgrade cached row across every environment. Forget it and +the backend serves old-shaped JSON out of cache indefinitely, with no error +anywhere — the new code is live and the old data keeps flowing. + +`CONTENT_VER` is deliberately separate so a content-only shape change doesn't +invalidate the whole grading cache. + +## 3. ETag / `If-None-Match` — and `expose_headers` + +Every graded payload is cached against a validity token that doubles as a weak +ETag. `frontend/static/cache.js` fetches `/api/v2/cell` once per view-set, +slices it, and revalidates each view individually with `If-None-Match`. + +**`expose_headers=["ETag"]` in the CORS middleware matters as much as +`allow_origins`.** Browsers hide every response header except a fixed "simple" +allowlist from cross-origin JS. Without `ETag` exposed, `cache.js`'s +`res.headers.get("ETag")` silently returns `null`, 304 revalidation quietly +stops working, and **there is no console error**. You'd see it only as a traffic +bill. + +CORS is off by default (`THERMOGRAPH_CORS_ORIGINS` unset), which is today's real +same-origin topology. Never set it to `*` — Starlette itself refuses a +credentialed wildcard. + +Don't change the ETag derivation or that header list without checking +`cache.js`. And when the payload shape changes, the **slice → view map** in +`cache.js` has to track it. + +## 4. `pct_ordinal()` — mirrored in four places + +Floor a percentile into `1..99`. **Never 0, never 100.** + +| Copy | File | Guarded by | +|---|---|---| +| canonical | `backend/data/grading.py::pct_ordinal` | its own unit tests | +| Go SSR | `frontend/server/internal/format/format.go::PctOrdinal` | `format_test.go::TestPctOrdinal` (table-driven, not a cross-check) | +| browser | `frontend/static/shared.js::pctOrd` | nothing automated | +| Python SSR *(not deployed)* | `frontend/format.py::pct_ordinal` | the Python unit tier | + +Every percentile on every surface goes through one of these. If they diverge, +the same reading says two different things in two places. The backend copy is +canonical; the others follow it. + +`TEMP_BANDS` / `RAIN_BANDS` in `grading.py` are the source of truth for tier +names and thresholds. Changing them without the frontend produces tiers drawn +in colours that disagree with the labels the API returns. + +## 5. The Fahrenheit country set + +Fourteen codes: `US PR GU VI AS MP UM BS BZ KY PW FM MH LR`. + +| Copy | File | +|---|---| +| canonical | `backend/api/content_payloads.py::F_COUNTRIES` | +| Go SSR | `frontend/server/internal/format/format.go::FCountries` | +| browser | `frontend/static/units.js::F_REGIONS` | +| Python SSR *(not deployed)* | `frontend/format.py::F_COUNTRIES` | + +Two tests in `frontend/server/internal/format/format_test.go` guard this: + +- `TestFCountriesMatchesBackend` re-parses the backend's Python and asserts the + Go copy matches. **This one skips in CI** — the frontend image's build context + is `frontend/`, so `backend/` is unreachable from the builder stage, and the + builder stage is the only place CI runs these tests. Treat it as a + checkout-only guard: it fires on your machine and in a full-checkout run, not + during the image build. +- `TestFCountriesMatchesUnitsJS` parses `static/units.js` and diffs both + directions. This one *does* run in the image build — `frontend/Dockerfile` + copies that single file into the builder stage for exactly that reason — so a + divergence fails the build. + +The browser copy was unguarded until that second test was added; several source +comments had long claimed "a test asserts all three stay identical" when only +the Go↔backend pair was checked. The Python `frontend/format.py` copy remains +guarded only by the local Python tier, which is fine — it isn't deployed. + +## 6. Backend boot must survive an unreachable frontend, and vice versa + +Both directions are load-bearing for asynchronous deploys: + +- **Frontend → backend:** boot tries the IndexNow-key fetch once, catches + failure, and falls back to a lazy per-request lookup. Don't make it fatal. +- **Backend → frontend:** `THERMOGRAPH_FRONTEND_BASE_INTERNAL` is required at + import (fails loud), but it's only *used* by the catch-all proxy fallback — + an unreachable value doesn't stop the backend serving its own routes. That's + why the test suite and the smoke harness both point it at + `http://127.0.0.1:1`. + +`/healthz` on both services is liveness only. Neither proves the other is +reachable, deliberately — readiness is the ingress health-gate's job. + +## 7. `THERMOGRAPH_BASE` must match on both processes + +Default `/thermograph`; both images set `/`. Both processes must be configured +**identically** in every real deployment — it's how they agree on the shared URL +prefix. The Go frontend's content client reads it too, so a sub-path deployment +works rather than only ever working at the root. + +## 8. Image tags are keyed to the domain, not the branch tip + +Both `build-push.yml` and `deploy.yml` compute the tag as +`sha-` + the first 12 hex of `git log -1 --format=%H -- /`. + +Two consequences: + +- An infra-only commit at the branch tip cannot send a deploy chasing an image + no build produced. +- `deployed_version` can legitimately show a tag that is **not** the head of the + branch. That's correct, not drift. + +`fetch-depth: 0` in the deploy workflow exists for exactly this: in a +path-filtered monorepo the tip is often another domain's commit, and a depth-1 +clone can't see past it. + +## 9. Cookie auth is same-origin today + +The interactive tool's auth is an HttpOnly session cookie. `account.js` uses +`credentials: "include"` (not the fetch default) so the cookie still rides along +if the script is ever loaded cross-origin — but the backend must actually be +configured to accept credentialed cross-origin requests (CORS + +`SameSite`) for that to work. In the default same-origin deployment it's a +non-issue. **Don't assume a fully decoupled, independently-hosted frontend +"just works" for logged-in features** without checking that configuration. + +## Quick checklist + +Before opening a PR that touches both domains: + +- [ ] Did any response payload shape change? → bump `PAYLOAD_VER`. +- [ ] Did any route or payload break compatibility? → new `vN` router + alongside, bump `API_CONTRACT_VERSION`, check `min_frontend`. +- [ ] Did I change the API version pin? → both Go config **and** + `static/account.js`, in one PR, after checking the target backend. +- [ ] Did I touch tier names, thresholds, or percentile rendering? → check all + four `pct_ordinal` copies and the colour tokens. +- [ ] Did I touch the Fahrenheit set? → four copies, and `units.js` is + unguarded. +- [ ] Did I change the `/cell` bundle shape? → the slice→view map in + `cache.js`. +- [ ] Did I change ETag derivation or CORS headers? → re-read `cache.js`. + +Next: [CI and release](07-ci-and-release.md). diff --git a/docs/onboarding/07-ci-and-release.md b/docs/onboarding/07-ci-and-release.md new file mode 100644 index 0000000..5576594 --- /dev/null +++ b/docs/onboarding/07-ci-and-release.md @@ -0,0 +1,229 @@ +# 7. CI and release + +## The branch model + +``` +PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box + │ + promotion PR + ▼ + main ──▶ beta.thermograph.org + │ + promotion PR ← the owner's call, always + ▼ + release ──▶ thermograph.org +``` + +`dev`, `main` and `release` are protected — direct pushes are blocked for +everyone. Forgejo has **no auto-merge-on-green**, so merges are explicit: either +click "Auto merge when checks succeed" on a PR, or merge via the API once CI is +green. + +Promotion is continuous and per-decision. There is no release calendar. Size a +promotion to whatever has actually soaked on the environment below. + +### Who may merge what + +| Hop | Whose call | Norm | +|---|---|---| +| anything → `dev` | yours | Merge it. An open PR into `dev` is unfinished work, not caution. | +| `dev` → `main` (beta) | yours, batched | Promote when a batch is coherent enough to be worth *testing*. Beta is the test environment; not promoting is not testing. | +| `main` → `release` (prod) | **the owner's** | Prepare it, recommend for or against, and ask. Never unilaterally, never by another route. | + +Centralis's `promote(from, to)` implements exactly this, and `forge_pr_merge` +enforces it with no back door: it merges into `dev` and `main`, refuses +`main → release` under any flag, and requires `confirm_protected_base: true` +for the separate hotfix path. + +### Two things about branch divergence + +**Commit counts are not deliverable content.** Before promoting, compare the two +tips' *trees*. Identical trees mean the same changes are already on the target +under different SHAs — promoting there merges nothing while still firing the +target's deploy workflows, including `infra-sync` re-rendering +`/etc/thermograph.env` on prod and beta. The ahead/behind numbers will look like +real work. `promote` refuses this outright. + +**Expect `dev`, `main` and `release` to be mutually divergent, and expect +`--ff-only` to fail.** Every promotion merges the source into the target, +creating a merge commit *on the target* that the source never receives — so each +promotion widens the gap it just closed. That is accumulated design, not drift. +(Reconciling merges back into `dev` do happen; `be4aa94` in the history is one.) + +## The nine workflows + +All in root `.forgejo/workflows/`. **Never add a workflow under a domain's own +`.forgejo/`** — Forgejo only reads the root ones, so a copy there is inert and +becomes a trap for whoever edits it next. + +### `pr-build.yml` — the required check + +One workflow, one stable required-check name (**`gate`**), any number of +domains: + +1. a `changes` job diffs the PR against its base (needs `fetch-depth: 0` — the + base commit isn't reachable from a depth-1 clone); +2. per-domain jobs run only for domains the PR touches; +3. `gate` reduces their results into the single status branch protection + requires. + +**Deliberately not path-filtered at the workflow level.** A path-filtered +required check that never fires leaves a PR stuck on *"expected — waiting for +status"* forever, so auto-merge never proceeds. An always-running gate whose +domain jobs skip cleanly gives the same compute savings without the deadlock. +This is the one intentionally *coupled* piece of the CI design. + +### `build.yml` — the reusable build check + +Called via `uses:` with a `domain` input. The build **context is that domain's +subdirectory**, so each Dockerfile sees exactly the tree it saw when its domain +was a repo root. + +For **backend** it then runs the hermetic pytest suite *inside the image it just +built* — the exact interpreter and dependencies that ship. Three flags there are +load-bearing: `-u 0` (the image's default user can't write to site-packages), +`--entrypoint sh` (otherwise the test command is swallowed as entrypoint args +and the container just boots the server forever), and `unset THERMOGRAPH_BASE` +(the image bakes `/`, which moves every route off the `/thermograph` prefix the +tests are written against). + +For **frontend** there's no separate test step: it's a Go binary with no +interpreter in the runtime image, and `gofmt`/`vet`/`go test` already run in the +Dockerfile's builder stage — so a failing Go test fails the Build step above. + +It is deliberately **not** a live boot+healthz check. That was tried in the split +repos and repeatedly hit the runner's own environment quirks (bridge-IP +reachability, non-unique `$$`, squatted host ports) without ever going reliably +green. Image boot is verified by hand instead — `make smoke`, or `docker run` + +alembic + `/healthz` 200 + a real `/api/v2/place` 200. + +### `build-push.yml` — images + +Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or +`frontend/**`, and on `v*.*.*` tags. Publishes +`git.thermograph.org/emi/thermograph/{backend,frontend}:sha-<12hex>`. + +- **Serialised** (`max-parallel: 1`): one physical runner, and two whole image + builds racing each other buys nothing. +- **Semver exception:** a `v*.*.*` tag push carries no paths context, so a + release tag builds **both** images. That's intended — a release names the + whole set, not whichever domain moved last. +- The registry is mesh-only, so the runner host needs + `10.10.0.2 git.thermograph.org` in `/etc/hosts`. + +### `deploy.yml` — one file, two environments, two services + +Formerly six near-identical files. Branch selects the environment +(`main → beta`, `release → prod`); a matrix covers the services. + +The style is **deliberately boring**: no dynamic `fromJSON` matrix, no +`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act +runner may evaluate differently, and the failure mode here is "production does +not deploy" or, worse, "deploys with an empty SSH host". So the two environments +get two explicit, mutually exclusive steps and the tag is computed in shell. + +Preserved from the originals, all load-bearing: + +- `fetch-depth: 0` — the tag is keyed to the last commit that touched *that + domain*, not the branch tip. +- 12-hex truncation matching `build-push.yml` exactly. +- Per-service, per-environment concurrency with **`cancel-in-progress: false`** — + a half-finished deploy must never be cancelled by a newer one. +- **Separate `PROD_SSH_*` credentials**, so a beta credential leak cannot reach + prod. +- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's + default action registry. + +There's also a per-leg refinement: the workflow-level `paths` filter only says +*backend or frontend moved*, so each leg re-checks whether **this** push touched +**this** domain. Without it a backend-only push would also roll the frontend, +losing the independent-deploy property the whole split exists for. When there's +no usable base SHA (first push, force push, `workflow_dispatch`) it deploys +rather than skips — rolling onto the tag already running is a no-op, whereas +skipping silently strands a change. + +### `infra-sync.yml` — infra's own pipeline + +Push to `main` touching `infra/**` → SSH to beta **and** prod, fast-forward +`/opt/thermograph`, re-render `/etc/thermograph.env` from the vault. + +**Rolls no service.** Image tags are the app domains' axis, not infra's. A +compose change that must recreate containers takes effect on the next app +deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh`. + +Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release` +maps to LAN→beta→prod via image tags); **infra is not** — both hosts track infra +via `main`. Prod's *app images* are staged by `release`, but its checkout +follows `main`. + +### `observability-validate.yml` + +Parses every artifact that ships to a node: both compose files, all dashboard +JSON, the Loki and provisioning YAML, and the Alloy config — checked with the +**pinned `alloy` binary (v1.9.1, matching what the fleet runs)**, downloaded +from its GitHub release, no Docker CLI needed. + +Alerting gets a stricter third step: every rule's `condition` must name a refId +that exists, every policy must route to a receiver that exists, and a literal +Discord webhook URL in the repo is a hard failure. **A rule pointing at a +missing refId is valid YAML, provisions cleanly, and then never fires** — that +silent no-op is exactly what this check exists to prevent. + +### `ops-cron.yml` — the prod backup + +Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into prod +and run inside the already-running stack. + +**It uses `PROD_SSH_*`, not `SSH_*`.** An earlier revision reused `SSH_*` — so +the "prod" backup was silently dumping *beta*, and prod had no backup at all. If +you touch this file, verify a dump actually lands in +`agent@prod:~/thermograph-backups/`. + +### `secrets-guard.yml` and `shell-lint.yml` — the backstops + +Both run on **every** PR and push, deliberately un-path-filtered: *a backstop +that only runs when you expect it to isn't a backstop.* + +- `secrets-guard` fails if any `infra/deploy/secrets/*.yaml` lacks both a + `sops:` block and `ENC[AES256_GCM` values. ~1 second. +- `shell-lint` runs shellcheck over every `*.sh` found by `find` (not a list — + so a new script is covered the moment it lands). Pinned to **v0.11.0 plus a + sha256**, installed from the official static release rather than `apt-get`: + the distro version drifts with the job image, and a drifted shellcheck grows + new warnings that fail CI on an unrelated push. Bump the version and the hash + together, and expect to fix new findings in the same PR. + +Keep the tree at **zero shellcheck findings**. These scripts run as root over +SSH against live hosts with no test suite in front of them; this is the only +guard. + +## Runner facts + +- Jobs run on the `docker` label — the LAN box's Forgejo Actions runner. +- Job containers get the **host's docker.sock** automounted + (Docker-outside-of-Docker, no privileged mode). The job image + (`node:20-bookworm`) ships no docker CLI, which is why several workflows + `apt-get install docker.io` first. +- Forgejo's runner has **only the labels it was explicitly registered with**. + GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does + not. A `runs-on: [self-hosted, thermograph-lan]` array is permanently + unschedulable there — a real bug that was found and fixed once, and worth + recognising if a job ever silently stops appearing in the Actions history. + +## Opening a PR + +```bash +git switch -c fix/short-description # branch off dev +# ... work ... +cd backend && make test # or: cd frontend/server && go test ./... +git commit # concise, technical, no AI mentions +git push -u origin fix/short-description +``` + +Then open the PR into `dev` (Centralis's `forge_pr_open`, or the Forgejo web +UI). Wait for `gate`. Merge — squash is the configured style. + +If your change touches shell, expect `shell-lint` to have an opinion. If it +touches `infra/deploy/secrets/`, `secrets-guard` will too. + +Next: [Infra and secrets](08-infra-secrets.md). diff --git a/docs/onboarding/08-infra-secrets.md b/docs/onboarding/08-infra-secrets.md new file mode 100644 index 0000000..ed8eec5 --- /dev/null +++ b/docs/onboarding/08-infra-secrets.md @@ -0,0 +1,254 @@ +# 8. Infra and secrets + +`infra/` owns **how and where the already-built images run**. It never builds +app source. Hosts' `/opt/thermograph` is a checkout of this whole monorepo. + +Read [`infra/CLAUDE.md`](../../infra/CLAUDE.md) alongside this. + +## The machines + +| Host | Public | Mesh | Orchestrator | Login | +|---|---|---|---|---| +| **prod** | `169.58.46.181` / thermograph.org | `10.10.0.1` | Docker **Swarm** | `agent`, passwordless sudo | +| **beta** | `75.119.132.91` / beta.thermograph.org | `10.10.0.2` | **compose** | `agent`, passwordless sudo | +| **desktop** | — | `10.10.0.3` | compose (LAN dev) | it's your box | + +```bash +ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod +ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta +``` + +Beta also hosts **Forgejo** (git + CI + registry) and **Grafana + Loki**, which +is why beta is guarded as strictly as prod: a destructive command there takes +out git, CI and the registry at once. The LAN dev box is deliberately +unguarded. + +Every root-effective command on the VPSes is logged by `auditd` +(`ausearch -k agentcmd`). That's a feature — fixes are attributable. + +Prefer Centralis for routine work (`run_on_host`, `fleet_status`, +`deployed_version`) over raw SSH. + +## The two orchestrators + +Which path a host takes is decided by **`/etc/thermograph/deploy-mode`**: the +string `stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything +else is compose. The workflows never need to know which mode a host runs. + +| | compose (beta, LAN dev) | Swarm (prod) | +|---|---|---| +| File | `infra/docker-compose.yml` | `infra/deploy/stack/thermograph-stack.yml` | +| Services | `db`, `backend`, `lake`, `daemon`, `frontend` | `db`, `web`, `worker`, `lake`, `daemon`, `frontend`, `autoscaler`, `autoscaler-lake` | +| Rolling | `up -d --no-deps ` | start-first, health-gated, auto-rollback | +| Tag file | `deploy/.image-tags.env` | `deploy/.stack-image-tags.env` | + +Note prod splits `backend` into **`web`** and **`worker`** (the +`THERMOGRAPH_ROLE` split from [04](04-backend.md)); compose runs one `backend` +service in role `all`. + +`STACK_TEST=1` rehearses the entire Swarm deploy under stack name +`thermograph-test` on throwaway volumes and ports `18137`/`18080`. Leftover +`thermograph-test_*` containers in logs are rehearsal residue — ignore them. + +### The volume/project-name coupling + +Compose creates `thermograph_pgdata` / `_appdata` / `_applogs`; the Swarm stack +declares those exact names as `external: true` at the same mount paths. That +coupling is why `name: thermograph` is pinned in the compose file — running +compose from `infra/` without it derives project `infra`, silently creating a +whole new stack with empty volumes beside the running one. `deploy-dev.sh` +exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to +keep LAN dev separate on purpose. **Keep both halves.** + +## `deploy/deploy.sh` — read this before you touch a deploy + +Single entry point for beta and prod. Contract: + +```bash +SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh +SERVICE=frontend FRONTEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh +SERVICE=all BACKEND_IMAGE_TAG=sha- FRONTEND_IMAGE_TAG=sha- …/deploy.sh +``` + +What it does, in order, and why each step is the way it is: + +1. **`flock` on `deploy/.deploy.lock`**, with a self re-exec so the lock spans + the whole run. Backend and frontend deploys can fire for the same push + seconds apart and both SSH into one checkout; concurrent runs race the git + reset, the compose project and the tag file. `-w 600` bounds the wait. +2. **Render secrets** from the SOPS vault into `/etc/thermograph.env`, then + source it, so a by-hand run interpolates the same as the systemd unit does. + Guarded on the helper's existence so the very deploy that *introduces* + `render-secrets.sh` is safe. +3. **`git reset --hard origin/$BRANCH`** on the checkout root (`BRANCH` defaults + to `main`). ⚠️ **Uncommitted edits in `/opt/thermograph` evaporate here.** + The one exception is `deploy/.image-tags.env`, untracked on purpose. +4. **Read `.image-tags.env`** so a single-service roll re-renders compose with + *both* services' real tags and never accidentally recreates or downgrades the + sibling. +5. **Registry login** only if `REGISTRY_TOKEN` is set. The CI paths don't pass + one — the host is already `docker login`ed — and an unconditional login with + an empty token would abort under `set -e`. +6. **Pull, with a bounded ~5-minute retry.** `build-push.yml` is a *separate* + workflow triggered by the same push, and Forgejo's `needs:` only orders jobs + within one file. A deploy racing ahead of its build and failing "not found" + is confirmed-live behaviour; the retry covers a normal build and still fails + loudly on a genuine problem. +7. **Daemon capability probe.** Infra tracks `main` while image tags are + env-staged, so a host can legitimately be asked to roll a backend image older + than the compose file — one built before the daemon binary existed. The + script `docker run`s the image to test for `/usr/local/bin/thermograph-daemon` + and drops `daemon` from this run if it's absent, rather than leaving a + container crash-looping on a missing binary. +8. **Roll.** `TARGETS` are `backend → (backend, lake, daemon)`, + `frontend → (frontend)`, `all → everything`. **`daemon` and `lake` ride with + `backend` and are never targets on their own** — all three run the same image + at the same tag, and rolling one without the others is exactly the version + skew the `/internal/*` contract has no negotiation for. Single-service rolls + use `--no-deps`; a full `all` uses `--remove-orphans` (a renamed-away + service's old container would otherwise squat its port — confirmed live). +9. **Write `.image-tags.env`** *after* `up`, so a failed pull never records a + tag that isn't running. +10. **Health check** (backend 8137, frontend 8080), then detached city warming. + +### Rollback + +Rollback is redeploying the previous image tag. But **do not treat image +retention as a rollback guarantee**: both `deploy.sh` and `deploy-stack.sh` +*end* by deleting images outside the running pair. On beta that succeeds, so +beta typically holds **no local rollback target at all**; on prod it usually +fails only because Swarm's stopped task containers still hold references, which +is why prod keeps a handful. Verify the target tag is actually present before +promising a rollback — Centralis's `rollback_to(dry_run: true)` checks both host +and registry. + +`docker system prune -a` on a live box is forbidden for the same reason. +`docker image prune -f` (dangling only) is the safe form. + +## Secrets — the SOPS + age vault + +**The single source of truth is `infra/deploy/secrets/*.yaml`**, committed +encrypted (values only — keys stay readable so diffs mean something) and +rendered into `/etc/thermograph.env` at deploy time by +`deploy/render-secrets.sh`. + +**Cycling a key is: edit → commit → deploy.** No SSH, no hand-edited root file, +no per-host duplication. + +| File | Holds | +|---|---| +| `common.yaml` | the 16 values identical on prod **and** beta — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config | +| `prod.yaml` | prod's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL`, Discord + mail credentials that exist nowhere else | +| `beta.yaml` | beta's own — the three held-back credentials, sizing, base URL | +| `dev.yaml` | the LAN box's own, **self-contained**, 12 values, no production credential | +| `centralis.prod.yaml` | Centralis's nine variables → `/etc/centralis.env`, its own renderer | +| `example.yaml` | format reference / CI fixture (fake values) | +| `../../.sops.yaml` | which age recipients files encrypt to (plaintext config) | + +The renderer concatenates `common.yaml` then `.yaml`, so a **host value +wins** (last occurrence of a duplicate key). `` comes from +`/etc/thermograph/secrets-env` on the box. + +### Two design decisions worth understanding + +**Three credentials are deliberately *not* in `common.yaml`** — +`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_DATABASE_URL`. +They hold identical values on prod and beta today, so by the mechanical rule +they belong there. They're kept per-host anyway because these are the +credentials that let one environment *act as* another: with them, a foothold on +beta (the more exposed box — public Forgejo and Grafana) is a foothold on prod's +database and prod's session signing. They match because beta was seeded from +prod, not because the two are meant to be one system. Keeping them per-host +costs one extra line and buys the ability to diverge. + +**`dev` renders `dev.yaml` alone** — `deploy-dev.sh` exports +`THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of `common.yaml`'s sixteen values +are live production credentials, including a **read-write** object-storage +keypair on the bucket that also holds prod's database backups, and the VAPID +private key that signs push to real subscribers. The render is plaintext +concatenation, so layering it would put all of those into every container in the +dev stack — on the operator's desktop, which is also the CI runner whose +`docker`-labelled jobs get the host socket, running unreviewed `dev` code. An +override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the +production value is still physically a line in the file. + +Dev works fine without them: no S3 keys means the `lake` service answers 503 and +history falls through to the Open-Meteo archive (an accelerator, never a +dependency); VAPID and IndexNow both self-generate and persist to the `appdata` +volume; with no metrics token `/api/v2/metrics` is direct-loopback-only, which +is right on a LAN box. One `common.yaml` value is simply *wrong* for dev — +`THERMOGRAPH_COOKIE_SECURE=1` silently breaks login over plain HTTP. + +### Working with the vault + +```bash +sops edit infra/deploy/secrets/beta.yaml # never a plain editor +infra/deploy/secrets/dry-run-render.sh # preview a render +``` + +- **Never hand-edit `/etc/thermograph.env`** — it's a rendered artifact, + overwritten on the next deploy. +- **Never paste a plaintext secret onto a box.** +- `secrets-guard` CI rejects a plaintext commit; the repo's `secrets-guard.sh` + hook blocks a direct Write/Edit to those files locally. +- `deploy/secrets/seed-from-live.sh` reads production secrets and is **not** + for an agent to run. +- The **`key-gaps`** skill audits which keys are missing or drifting across + environments. It reads key *names* only, never values. + +There's a cautionary tale in `deploy/secrets/README.md` worth reading: Centralis +was hand-configured until a hand-edit wrote `CENTRALIS_TOKENS` as bare JSON into +a file that gets `source`d, bash stripped the quotes, and Centralis — correctly +failing closed on malformed JSON — dropped to a single identity. Nothing logged +an error. It looked exactly like the file had never been written. That's why +`centralis.prod.yaml` exists: the only durable fix was *a human stops writing +the file*. + +## Terraform + +`infra/terraform/` provisions and configures hosts, SSH-driven. + +> **Never run `terraform apply` casually.** No tfstate is persisted anywhere, so +> an apply would attempt full re-provisioning of live hosts. Terraform here is +> **executable documentation** until state is bootstrapped. + +## Ops query tooling + +Read-only, uniform across environments: + +```bash +infra/ops/dbq.sh dev "select count(*) from climate_history" +infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history" +infra/ops/iceberg.sh prod -c "select count(*) from era5_daily" +``` + +Both exec **into the container**. No database is exposed over TCP — each listens +only on its private docker network, and prod's is a Swarm **overlay the prod +host itself cannot route to**, so `ssh -L` works for beta and is *impossible* +for prod. Exec works identically everywhere with no ports, no tunnels, no infra +changes. The prod container name is a Swarm task name that changes on every +redeploy, so it's resolved at call time via `docker ps --filter name=`, never +hardcoded. + +Queries connect as **`thermograph_ro`** — `NOSUPERUSER`, granted only +`pg_read_all_data`. Read-only is enforced by Postgres, not by convention: + +``` +$ infra/ops/dbq.sh prod -c "create table t(i int)" +ERROR: permission denied for schema public +``` + +Centralis's `sql_query` does the same thing with the same role. `write: true` +escalates to the app's superuser and takes effect immediately with no +confirmation — on prod, know what you are doing. + +## Backups + +The nightly `ops-cron.yml` `pg_dump` into `agent@prod:~/thermograph-backups/` +is it. Known gaps, stated plainly: **a single copy on the same box as the +database, no offsite yet**, and restores must handle TimescaleDB's +`continuous_agg` circular-FK warning (`--disable-triggers`). The `backups/` +prefix in the object-storage bucket belongs to those jobs — don't write outside +the lake's own prefix. + +Next: [Observability](09-observability.md). diff --git a/docs/onboarding/09-observability.md b/docs/onboarding/09-observability.md new file mode 100644 index 0000000..8605fba --- /dev/null +++ b/docs/onboarding/09-observability.md @@ -0,0 +1,199 @@ +# 9. Observability + +Fleet-wide log aggregation: **Loki + Grafana on beta**, fed by a **Grafana +Alloy** agent on every node, all over the WireGuard mesh. + +``` + prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3) + ┌────────────┐ ┌──────────────────┐ ┌────────────┐ + │ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│ + └────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘ + docker+caddy+app └──────────────────┘ docker+caddy+app +``` + +- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP + `10.10.0.2:3100`. Never public. +- **Grafana** — the UI, fronted by beta's Caddy at + **`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only, + `allow_sign_up=false`, plus a break-glass local admin). Use that hostname + everywhere; **never** `grafana.thermograph.org`. +- **Alloy** — on each node, shipping three sources: every Docker container's + stdout/stderr, Caddy's host access logs, and the app's structured JSON logs + (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` become + labels). Every line is tagged `host = prod|beta|dev`. + +There is **no build and no deploy automation** for this domain — it ships by +hand. But `observability-validate.yml` parses every artifact that ships to a +node, so a malformed dashboard or an unparseable Alloy config fails in CI +instead of on the host. + +## The rule that matters most + +**The repo is the only durable path.** Dashboards and alerting are provisioned +from `observability/grafana/provisioning/` at startup, so **edits made through +Grafana's UI or API are overwritten on the next provision**. A change goes +through a PR. Centralis's `dashboard_write` opens one for you. + +## Reading the fleet + +Start wide, then narrow: + +1. `fleet_status` — HTTP reachability plus what's running on each host. +2. `logs_overview since="1h"` — where the volume is, and what has gone quiet. +3. `logs_query service=… host=…` — the actual lines. + +### ⚠ The prod trap + +Prod runs a Swarm stack, and its `job="docker"` streams carry **no `service` +label**. A raw `{host="prod", service="backend"}` query matches nothing and +reads as *"no logs"* when it actually means *"wrong selector"*. + +Use Centralis's `logs_query` `service` argument, which rewrites it to a +container name-regex with the churning task suffix wildcarded. Prod service +names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`, +`_lake`. + +Related: **prod container names change on every redeploy** (the Swarm task +suffix). Never hardcode one; resolve via `docker ps --filter name=` at call +time. + +## The dashboard + +`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log +volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a +per-node notifier-liveness indicator, recent app errors, and a live +all-container tail. + +## Alerting + +Provisioned from `grafana/provisioning/alerting/`, mounted read-only into +Grafana. + +**Every rule is LogQL** — there is no Prometheus anywhere in the fleet. Loki is +the only datasource, so every signal is derived from log lines. Thresholds were +backtested against real Loki data and the working is written out in the comments +beside each rule; **re-derive before changing a number rather than guessing**. + +### Alerts go to Discord `#ops-alerts`, never email + +Three reasons, all learned the hard way: + +- Grafana's factory default pointed at the literal string + `` — a paging path that had never delivered a message to + anyone. +- Beta's Grafana relays SMTP through **prod's** Postfix. Email alerts therefore + travel through the box most likely to be on fire, and vanish exactly when they + matter. +- Discord is off-estate, works when prod is dead, and reaches a phone. + +`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert +subscription output — real deliveries to real subscribers — so routing ops noise +there would corrupt the evidence. + +The webhook URL is a secret and lives only in beta's gitignored `.env` as +`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal +webhook URL committed to the repo is a **hard CI failure**. + +### The twelve rules + +| Rule | Fires when | Severity | +|---|---|---| +| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the pipeline is broken | critical | +| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical | +| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical | +| `ProdHTTP5xxElevated` | ≥25 prod 5xx over 6h (the error-budget alarm) | warning | +| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning | +| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` from prod in 20m | critical | +| `NotifierHeartbeatMissingBeta` | same, on beta | warning | +| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical | +| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical | +| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning | +| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning | +| `AlertingWatchdog` | always — one Discord message per day | watchdog | + +`AlertingWatchdog` is the dead-man's switch and fires permanently by design. +**Its value is its absence:** nothing can report its own death, so if +`#ops-alerts` has been silent for more than a day, alerting is broken rather +than the estate healthy. + +### The strict CI check, and why + +A rule whose `condition` names a refId that does not exist is **perfectly valid +YAML, provisions without complaint, and then never fires** — a paging path that +looks healthy and reaches nobody. `observability-validate.yml` checks that every +condition names a real refId and every policy routes to a receiver that exists, +precisely because nothing else would catch it. + +The same asymmetry shows up in verification: `GET +/api/v1/provisioning/contact-points` **redacts** the webhook URL, so you cannot +confirm interpolation by reading it back. A contact point holding the literal +string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and +pages nobody. **The only proof is a message actually arriving in +`#ops-alerts`** — force one with the receivers-test endpoint (the command is in +`observability/README.md`). + +## Deploying an observability change + +> ⚠️ **Merging is not deploying — and this one has already bitten.** +> `/opt/observability` on beta and prod is a checkout of the **archived** +> `emi/thermograph-observability` repo. It can never `git pull` again; the +> content now lives here under `observability/`. A previous observability PR +> merged and had **zero live effect** for exactly this reason. + +Until someone re-points those checkouts at the monorepo, every change ships by +hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up, +`sudo cp` it into place, restart the specific container. + +Two gotchas in that sequence: + +- Use `sudo docker restart ` on **prod** — not `docker compose`, + which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to + interpolate from. Beta's stack *does* have an `.env`, so compose works there. +- **`docker restart` will not pick up a new environment variable.** Any change + that introduces one needs `docker compose up -d grafana` on beta. + +Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana. + +Full step-by-step, including rollback, is in +[`observability/README.md`](../../observability/README.md). + +## Known gaps (as documented in the repo) + +- **Alloy can silently stop tailing a container after Swarm replaces the task.** + Observed on prod 2026-07-24: `thermograph_worker` was `1/1`, healthy, writing + ~19 lines per 10 minutes to its own docker log, yet not one line reached Loki + for 13½ hours. What localises it to Alloy: prod's `tag=heartbeat` lines kept + arriving throughout, and those come from a *different* Alloy source + (`loki.source.file` on `/applogs`) than container stdout + (`loki.source.docker`). One source went deaf, the other didn't. Workaround: + `sudo docker restart alloy-alloy-1` on the node. **Not root-caused.** + `ProdWorkerContainerSilent` now catches this class of failure. +- **Some services are too quiet to alert on.** `thermograph_daemon` (23 lines / + 24h), `thermograph_autoscaler` (1) and `_autoscaler-lake` (2) have a healthy + floor indistinguishable from dead; `thermograph_lake` is bursty ETL with + legitimate idle gaps. Giving each an `audit.log_heartbeat()` line would make + all four alertable with the pattern the notifier already uses. **The single + highest-value follow-up in this domain.** +- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual* + dead-man's switch — it relies on somebody noticing the daily message stopped. + An external uptime pinger from off-estate would close this properly. +- **Beta runs a `docker-compose.override.yml` that is not in this repo**, wiring + Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked + compose file or deleted; alerting no longer depends on it. + +## When something breaks + +1. `fleet_status` — is it up at all? +2. `logs_query host=prod contains="(?i)error|traceback"` since the incident. +3. `deployed_version env=prod` — did something ship just before it started? +4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass? +5. **Write down what you found with `notes_write` *before* fixing it.** The + reconstruction is worth more than the fix, and it will be gone by tomorrow. + +You are authorised to fix production roadblocks directly — restart or roll a +wedged service, clear a full disk, kill a runaway process, bounce a crash-looping +container. What you must not do is make the fix *durable* on the box: writes to +code, docs and dashboards go through a PR, and `deploy.sh` hard-resets the +checkout anyway. The on-box action is stabilisation; the PR is the fix. + +Next: [Recipes](10-recipes.md). diff --git a/docs/onboarding/10-recipes.md b/docs/onboarding/10-recipes.md new file mode 100644 index 0000000..58766e7 --- /dev/null +++ b/docs/onboarding/10-recipes.md @@ -0,0 +1,256 @@ +# 10. Recipes + +Task-shaped walkthroughs. Each one names the files you'll touch and the traps +specific to that change. + +--- + +## Add a field to a graded API payload + +1. **Build it** in `backend/api/payloads.py` (or `content_payloads.py` for the + SSR content API). Payload assembly is pure `inputs → dict`; keep HTTP + semantics with the caller. +2. **Bump `PAYLOAD_VER`** in `api/payloads.py`. Non-negotiable — the derived + store is token-validated with no clock expiry, so without a bump the old + shape serves out of cache forever, silently. If the change is content-API + only, `CONTENT_VER` is the narrower lever. +3. **Decide whether it's breaking.** Additive → fine on `v2`. Breaking → a new + `v3` router mounted alongside `v2` re-registering only the changed handlers, + plus `API_CONTRACT_VERSION` bumped in the same PR. +4. **Check `/cell`.** If the field lands in a slice, the slice→view map in + `frontend/static/cache.js` has to track it. +5. **Consume it** in `frontend/server/internal/contentapi/types.go` and whichever + template or JS renders it. +6. **Test both sides**: `cd backend && make test`, then + `cd frontend/server && go test ./...`. +7. **Refresh fixtures if the shape moved**: `cd frontend && make capture-fixtures` + against a live backend. Those fixtures feed the **Go** tests too. + +--- + +## Add a new SSR page + +The backend owns the page's *data and metadata*; the frontend owns its *markup*. + +1. `backend/api/content_payloads.py` — a builder returning JSON-safe dicts (raw + floats, no `Markup`, no ContextVar unit rendering), including `page_title`, + `canonical_path`, `breadcrumb` and `jsonld`. Those live here, **not** in the + frontend. +2. `backend/api/content_routes.py` — route wiring plus the same derived-store / + ETag pattern the neighbouring routes use. +3. `frontend/server/internal/contentapi/` — a typed client method. +4. `frontend/server/internal/render/templates/.html.tmpl` — embedded in + the binary, so a template change needs a rebuild. +5. `frontend/server/internal/content/pages.go` — the handler; register the route. +6. If the page should be crawlable, add it to the sitemap + (`backend/api/sitemap.py` and the `/content/sitemap` payload). +7. Copy that isn't per-city data goes in `frontend/content/pages.yaml`, loaded + fail-loud at boot. + +--- + +## Change something visual + +1. Read [`frontend/DESIGN.md`](../../frontend/DESIGN.md) first. It wins over any + general design guidance. +2. Edit `frontend/static/style.css`. **Use `var(--token)`; never paste a hex.** + New surfaces take their colour from existing tokens so light mode keeps + working for free. +3. If it's a grade colour, keep the scale's order and midpoints intact — they + encode meaning, not decoration, and the charts pull from the same tokens as + the legend. +4. **Verify at 390 / 800 / 1920 / 2560 / 3840px in both light and dark:** + ```bash + cd frontend && .venv/bin/python tools/shoot.py + ``` + or drive a real browser with the Chrome DevTools MCP. Narrow while + iterating: `tools/shoot.py index --width 390 --scheme dark`. +5. Watch the hard rules: inputs at `font-size: 16px` minimum (iOS zoom), ~44px + touch targets, honour `prefers-reduced-motion`, metric order always + `Precip · High · Low`. + +--- + +## Add or rotate a secret + +```bash +sops edit infra/deploy/secrets/prod.yaml # never a plain editor +git commit && push && open a PR +``` + +Then it reaches the hosts by either route: an `infra/**` push to `main` fires +`infra-sync.yml` (re-renders `/etc/thermograph.env` on beta and prod, rolls +nothing), or the next app deploy renders it as step 2 of `deploy.sh`. + +Decide **which file** first: + +- identical on prod *and* beta, and not a "lets one environment act as another" + credential → `common.yaml`; +- per-host, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` / + `THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`; +- dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's. + +Preview with `infra/deploy/secrets/dry-run-render.sh`. Audit cross-environment +drift with the `key-gaps` skill (names only, never values). + +**Never** hand-edit `/etc/thermograph.env`, paste a plaintext secret onto a box, +or run `seed-from-live.sh`. + +If a rotated value needs containers recreated to take effect, remember +`infra-sync` rolls nothing — you need an app deploy or a by-hand +`SERVICE=all … deploy.sh`. + +--- + +## Add a recurring background job + +Ask one question first: **does it need data?** + +- **No** (a timer, a connection, an external poke) → add a `cron.Job` in + `backend/daemon/main.go`. One replica, so no election needed. Give it an env + var for its interval, defaulting sensibly and falling back to the default on a + malformed value. +- **Yes** → add an `/internal/*` route in `backend/api/internal_routes.py` and + have the daemon POST to it. Grading depends on polars and the cache; + reimplementing it out-of-process would let results drift from the API's. + +If you genuinely need an in-process background thread in the Python app instead, +**read `core/singleton.py` first**. Anything that issues upstream fetches on a +timer must run behind `claim_leader()` — N workers × M hosts multiplies quota +use N×M, and that has already caused overnight 429s in production. Things that +drain a per-worker request-fed queue (the neighbour warmer) correctly run +everywhere; so does the heartbeat, because a beat costs one local file append. + +Then give it a heartbeat (`audit.log_heartbeat()`) so observability can alert on +its death — see the "too quiet to alert on" gap in +[observability](09-observability.md). + +--- + +## Change a Grafana alert or dashboard + +**Never through the UI.** Provisioning overwrites it on the next start. + +1. Edit `observability/grafana/provisioning/alerting/*.yml` or + `observability/grafana/dashboards/*.json`. +2. If you're changing a threshold, **re-derive it against real Loki data** — the + working for the current number is in the comments beside the rule. +3. Open a PR. `observability-validate.yml` will check that every condition names + a real refId, every policy routes to a real receiver, and no literal webhook + URL is committed. +4. **Merging is not deploying.** Ship it by hand per + `observability/README.md` — and remember `docker restart` won't pick up a new + environment variable. +5. Verify by forcing a real message into `#ops-alerts`. The API redacts the + webhook, so arrival is the only proof. + +--- + +## Debug: "the site is serving stale data" + +Work the token chain, in this order: + +1. `GET /api/version` on the affected environment — is `payload_ver` what you + expect? If a payload shape changed and this didn't move, that's your answer. +2. Is the derived-store token advancing? It's built from `PAYLOAD_VER` plus the + history end date (`history_token`), so a cell whose history stopped updating + pins its token. +3. Is the *history* updating? Check `climate_sync` freshness + (`infra/ops/dbq.sh prod …`) and whether upstream is in a rate-limit cooldown + (the errors log, tagged `retry`/`error`). +4. Is it the **frontend's** TTL cache instead? `THERMOGRAPH_SSR_CACHE_TTL` + defaults to 600s, in front of the backend's own caching. +5. Is it the **browser**? `cache.js` seeds IndexedDB from the `/cell` bundle. If + ETags stopped being visible cross-origin (`expose_headers`), revalidation + silently stops. + +--- + +## Debug: "a deploy went green but nothing changed" + +Common causes, most likely first: + +1. **The tree was already identical.** A promotion between branches whose trees + match merges nothing while still firing the deploy workflows. +2. **The domain wasn't touched.** `deploy.yml` re-checks whether *this* push + touched *this* domain, and skips the leg if not. +3. **The tag is domain-keyed.** `deployed_version` showing a tag that isn't the + branch head is correct behaviour, not drift. +4. **Someone hand-edited `/opt/thermograph`.** `deploy.sh` runs + `git reset --hard` first; the edit is gone. +5. **It's an observability change.** `/opt/observability` points at the archived + repo and cannot pull. See above. + +--- + +## Promote to beta, and then to prod + +``` +dev → main your call, batched. Beta is the test environment; + not promoting is not testing. +main → release ALWAYS the owner's call. Prepare it, state what would ship, + the CI evidence and how long it soaked on beta, then recommend — + including recommending against. +``` + +Centralis's `promote(from, to)` does the mechanics and enforces the rule; +`forge_pr_merge` refuses `main → release` under any flag. Before promoting, +compare the two tips' **trees**, not their commit counts. + +--- + +## Roll back + +Redeploying the previous image tag: + +```bash +SERVICE=backend BACKEND_IMAGE_TAG=sha- \ + /opt/thermograph/infra/deploy/deploy.sh +``` + +**Check the tag actually exists first.** Both deploy scripts *end* by deleting +images outside the running pair, so beta typically holds no local rollback +target at all. `rollback_to(dry_run: true)` checks host and registry and will +tell you when it's in neither. + +--- + +## Regenerate the bundled city data + +`backend/cities.json` and `cities_flavor.json` are generated reference data, not +runtime state: + +```bash +cd backend +python gen_cities.py # the curated city set +python gen_flavor.py # the editorial blurbs +``` + +Editorial copy has its own voice guide — the **`city-copy`** skill defines what +good reads like, and **`backfill-city-copy`** drives generation at scale. Both +enforce the relative-grades rule. + +After a deploy, `warm_cities.py` pre-warms those cells so `/climate` pages serve +from cache and a crawl never bursts the archive quota. It's idempotent, paced, +and claims a flock — deploys launch it detached with no overlap check, so two +overlapping runs would otherwise double-spend quota and race the cache writes. + +--- + +## Work on a live host + +Prefer Centralis: `run_on_host`, `fleet_status`, `deployed_version`, +`logs_query`, `sql_query`. + +You may, without asking: read anything; restart or roll a wedged service; clear +a full disk; kill a runaway process; bounce a crash-looping container; run the +blessed deploy path onto an already-published image. + +You must not: hand-edit the checkout to persist a change (`git reset --hard` +eats it); hand-edit `/etc/thermograph.env` (rendered artifact); force-push; +deploy prod from an unmerged branch; `docker system prune -a` on a live box (it +deletes your rollback image — `docker image prune -f` is the safe form); or +restart the `_db` service casually. Postgres/Timescale restarts and volume +operations get confirmed with the operator first. + +Next: [Traps and stale docs](11-traps.md). diff --git a/docs/onboarding/11-traps.md b/docs/onboarding/11-traps.md new file mode 100644 index 0000000..4604b5d --- /dev/null +++ b/docs/onboarding/11-traps.md @@ -0,0 +1,279 @@ +# 11. Traps and stale docs + +Two kinds of thing here: **live traps** (correct code that will surprise you) +and **stale docs** (files in this repo that currently describe a system that no +longer exists). The second list matters as much as the first — this repo has a +lot of documentation, most of it excellent, and some of it describing the world +before a rewrite. + +Everything below was checked against the tree at the time of writing. + +--- + +## Stale docs in this repo + +### ~~`frontend/CLAUDE.md` and `frontend/README.md` describe a service that is not deployed~~ — fixed + +**Fixed.** Both now describe the Go service (`frontend/server/`), state plainly +that the Python files at that level are the superseded original, and correct +`CLAUDE.md`'s claim that `make test-unit` is *"the tier CI runs"* — it isn't. +CI builds the frontend image, and `frontend/Dockerfile`'s builder stage runs +`gofmt -l` + `go vet` + `go test ./...` before compiling, so a failing **Go** +test is what fails CI. The Python tiers are local-only. + +Still worth knowing: the superseded Python files carry **duplicate copies of +some contracts** — `api_client.py` has a third `API_VERSION` pin and +`format.py` a fourth `F_COUNTRIES`. Nothing deploys them, but if you run them +locally, keep them in step or you'll debug a phantom. + +Whether to delete them outright is an open question, not a settled one: the +Python test tiers still run, against fixtures the Go tests share. Raise it +rather than doing it in passing. + +### `infra/DEPLOY.md` describes the pre-Swarm, pre-monorepo deploy + +Three separate layers of drift: + +- It says **prod is deployed with `terraform apply`**. Prod is deployed by a + push to `release` firing `deploy.yml`, exactly like beta's `main` path — see + [07](07-ci-and-release.md). +- It says prod runs a **compose** stack. Prod runs **Swarm** + (`infra/deploy/stack/`). +- Its file table points at `/opt/thermograph/deploy/deploy.sh` and + `/opt/thermograph/docker-compose.yml`. Both moved under `infra/` at the + monorepo cutover. + +The document's own header flags the legacy `provision.sh` / systemd / venv +walkthrough as superseded — but the drift goes further than that note admits. +The **key-management** sections (Part 1) and the `/etc/hosts` registry note are +still accurate and useful. + +### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows + +Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist. The two +LAN-dev deploy workflows were deleted rather than ported at the CI +consolidation — they were documented as inert (they targeted a monorepo layout +at `~/thermograph-dev` on a box still holding a split-era checkout). **LAN dev +is a local `make dev-up` concern, not a CI environment.** + +`ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the +workflow set; there are now nine files and the deploy/build ones were collapsed. + +Everything else in `ACCESS.md` — the host table, the agent-access model, the key +rotation procedure, the Swarm/Forgejo tracks — is current and worth reading. + +### `backend/README.md` is written for the split-repo era + +It describes `thermograph-backend` as its own repo with sibling repos, uses the +retired image path `git.thermograph.org/emi/thermograph-backend/app`, and says +*"there is no Makefile in this repo yet"* — there is (`make test`, `make smoke`, +`make clean-venv`). + +The **"How it works"** section (grid → data → percentiles → caching) is one of +the best explanations of the domain model anywhere in the repo. Read it; just +mentally translate the topology. + +### `CUTOVER-NOTES.md` carries a superseded block, correctly labelled + +Its per-domain workflow names (`backend-build-push.yml`, `backend-deploy.yml`, …) +no longer exist — the six deploy workflows collapsed into `deploy.yml` and the +two build-push workflows into `build-push.yml`. The file says so in an explicit +"Superseded 2026-07-25" note, and everything it describes about *behaviour* is +unchanged. + +It also references `backend/MONOREPO-CUTOVER-PLAN.md`, which is not in the tree. + +`CUTOVER-NOTES.md` remains the source of truth for what is and isn't live. + +### The domain `CLAUDE.md` files still use repo-era wording + +They say "this repo" and link to sibling repo names. Mentally map +`thermograph-backend` → `backend/`, and fix the wording as you touch files near +it — that's the standing instruction in the root `CLAUDE.md`. + +### `observability/README.md`: merging is not deploying + +`/opt/observability` on beta and prod is a checkout of the **archived** +`emi/thermograph-observability` repo. It can never `git pull` again. A previous +observability PR merged and had **zero live effect** for exactly this reason. +The README documents the hand-ship procedure; until the checkouts are re-pointed +at the monorepo, that's the only path. Flagged again in +[09](09-observability.md). + +--- + +## Live traps + +### ~~The `docker-compose.test.yml` files pull retired image paths~~ — fixed + +**Fixed.** Both now use `emi/thermograph/backend`, the path `build-push.yml` +actually publishes. + +The frontend harness had the worse half of this: it also pinned +`THERMOGRAPH_BACKEND_TEST_TAG` to the split-era `v0.0.2-split-ci`. Rather than +swap one hardcoded pin for another, `scripts/backend-for-tests.sh` now +**derives** the tag from the checkout — `sha-<12hex of git log -1 -- backend/>`, +the same domain-keyed rule `build-push.yml` and `deploy.yml` both use — so the +harness follows the tree instead of drifting behind a pin. `docker-compose.test.yml` +requires the variable (`:?`) rather than defaulting it, so a stale pin can't +creep back in silently. + +If the derived tag has no published image (backend commits that are still +local-only), the script says so and tells you to pin one: + +```bash +THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> make backend-up +``` + +> Unrelated but adjacent: `make test-integration` against a freshly-booted +> throwaway backend currently fails 7 of 16 tests with `503`, because nothing +> in that empty database is warm. Verified identical on the old image and the +> new one, so it predates these changes and is **not** caused by them. Not +> investigated further here. + +### ~~`static/units.js`'s `F_REGIONS` is guarded by nothing~~ — fixed + +**Fixed.** `format_test.go::TestFCountriesMatchesUnitsJS` now parses +`static/units.js` and diffs it against the Go set in both directions. + +The subtlety worth knowing: the *existing* backend cross-check +(`TestFCountriesMatchesBackend`) **skips in CI**. The frontend image's build +context is `frontend/`, so `backend/` is structurally unreachable from the +builder stage — and the builder stage is the only place CI runs these tests. It +is a checkout-only guard and always was. + +The new `units.js` check does not have that problem, because `static/` *is* +inside the context: `frontend/Dockerfile` copies that one file into the builder +stage specifically so the assertion runs during the image build. Verified by +mutating `units.js` and confirming the **image build fails**, not just the local +test run. + +### The compose project name is load-bearing + +`infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose +from `infra/` derives project `infra` — a silently **new** stack with empty +volumes beside the running one. `deploy-dev.sh` exports +`COMPOSE_PROJECT_NAME=thermograph-dev` to keep LAN dev separate on purpose. +Keep both halves. + +### `deploy.sh` hard-resets the host checkout + +`git reset --hard origin/$BRANCH` runs before anything else. Uncommitted edits +in `/opt/thermograph` evaporate. `infra-sync.yml` does the same reset without +deploying. The one deliberate exception is the untracked +`deploy/.image-tags.env` — losing *that* makes the next single-service deploy +roll the sibling onto `local`. + +### The deploy scripts eat your rollback image + +Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the +running pair. On beta that succeeds, so beta typically holds **no local rollback +target at all**; on prod it usually fails only because Swarm's stopped task +containers still hold references. `docker system prune -a` is forbidden on a +live box for the same reason — but the deploy script is the thing that actually +eats the image, not prune. Verify a tag exists before promising a rollback. + +### `daemon` and `lake` are never deploy targets on their own + +They ride with `backend`. All three run the same image at the same tag — a +second binary and a second role inside one image — and the `/internal/*` +contract between daemon and web has no version negotiation. There's also a +mechanical reason: a single-service deploy runs `up -d --no-deps `, so +a service in compose but absent from `TARGETS` would simply never be created. + +### `docker compose config --images ` does not filter + +Confirmed live on Compose v5.3.1: it prints **every** service's image, one per +line, in file order. A `| head -1` therefore grabbed `db`'s image, and +`deploy.sh`'s daemon capability probe found no daemon binary in a Postgres image +and dropped the daemon from *every* backend deploy. The script now builds the +image reference from the same variables compose interpolates. + +### Prod's Loki streams carry no `service` label + +Prod runs Swarm; its `job="docker"` streams have no `service` label. A raw +`{host="prod", service="backend"}` query matches nothing and reads as "no logs" +when it means "wrong selector". Use `logs_query`'s `service` argument. Prod +container names also change on every redeploy (the task suffix) — resolve via +`docker ps --filter name=` at call time, never hardcode. + +### A Grafana alert rule can provision cleanly and never fire + +A rule whose `condition` names a non-existent refId is valid YAML, provisions +without complaint, and pages nobody. Likewise a contact point holding the +literal string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the +API — and the API **redacts** the URL, so you cannot confirm interpolation by +reading it back. The only proof is a message arriving in `#ops-alerts`. + +### Terraform has no state anywhere + +A casual `terraform apply` would attempt full re-provisioning of live hosts. +It's executable documentation until state is bootstrapped. + +### The registry is mesh-only + +`git.thermograph.org` resolves publicly to beta's IP, but beta's Caddy rejects +`/v2/*` from outside the WireGuard mesh. Any host that pulls needs +`10.10.0.2 git.thermograph.org` in `/etc/hosts`. Beta itself doesn't — it *is* +the box. + +### Forgejo runners only have the labels they registered with + +GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does not. +`runs-on: [self-hosted, thermograph-lan]` is permanently unschedulable there — +a real bug, found and fixed once. If a job silently stops appearing in the +Actions history, check this first. + +### Forgejo's `needs:` doesn't cross workflow files + +`build-push.yml` and `deploy.yml` fire from the same push with no ordering +guarantee, and a deploy racing ahead of its build and failing "not found" is +confirmed-live. That's why `deploy.sh`'s pull has a bounded ~5-minute retry. + +### `/api/v2/cell?prefetch=1` must never spend quota + +The hard guarantee: a cold cell answers `204` with no body. Anything named +"prefetch" or "warm" calls the cache-only loaders. Breaking this gets the whole +service rate-limited. + +### Background timers must go through leader election + +`singleton.claim_leader()` — flock per host, Postgres advisory lock per cluster. +A timer-driven sweep running in N uvicorn workers × M Swarm hosts multiplies +Open-Meteo quota use N×M. That is the documented cause of overnight 429/503s +after prod went to 3 workers. The heartbeat is the deliberate exception — a beat +is one local file append, so duplicates are free. + +### `THERMOGRAPH_DATA_DIR` exists because of a real collision + +The backend's Python package `data/` and its runtime data dir `/data` +collide at the same path once the code lives at the image root. Mounting a +volume there shadows and erases `data/*.py`, so `import data.climate` fails at +boot. Point the env var **outside the code tree** in a container. And never +reintroduce `__file__`-relative paths in a module — `paths.py` resolves +everything from the repo root for this reason. + +### Alloy can silently stop tailing a container + +Observed on prod, 13½ hours of `thermograph_worker` logs never reaching Loki +while the container was healthy and writing. Not root-caused. Workaround: +`sudo docker restart alloy-alloy-1`. `ProdWorkerContainerSilent` now catches the +class. + +### Nominatim is ~1 req/s, on one thread + +All reverse geocoding funnels through a single dedicated worker draining a +queue. The earlier design slept inside the caller's thread — which in the server +is one of Starlette's shared sync-threadpool threads. + +--- + +## If you fix one of these + +Docs are cheap to correct, and the root `CLAUDE.md` already asks you to fix +repo-era wording as you touch files near it. The three entries struck through +above were fixed in the same change that added this guide; the rest are still +open, and `infra/DEPLOY.md` is probably the highest-value one left — it +describes a deploy model that has since changed twice. + +Back to the [index](README.md). diff --git a/docs/onboarding/README.md b/docs/onboarding/README.md new file mode 100644 index 0000000..9b177df --- /dev/null +++ b/docs/onboarding/README.md @@ -0,0 +1,70 @@ +# Onboarding — becoming a full contributor to Thermograph + +This is the developer onboarding path for the `emi/thermograph` monorepo: what +the product is, how the code is shaped, how to run it, what will break silently +if you get it wrong, and how a change actually travels from your editor to +`thermograph.org`. + +**Scope.** This covers *working in this repo*. Cross-cutting architecture +decision records and operator runbooks live in the separate `thermograph-docs` +repo (reachable through Centralis's `docs_search`) — this set links out rather +than duplicating them. Where a fact is owned by a `CLAUDE.md`, that file stays +the source of truth and this set explains the context around it. + +## Reading order + +| # | Doc | Read it when | +|---|-----|--------------| +| 1 | [Orientation](01-orientation.md) | First. What the product actually claims, the estate, the four non-negotiable rules. | +| 2 | [Local setup](02-setup.md) | Before you touch anything. Toolchain, verified run/test recipes for every service. | +| 3 | [Repo map](03-repo-map.md) | To find things. Every domain and directory, and which files matter. | +| 4 | [Backend deep dive](04-backend.md) | Before your first backend change. Data pipeline, grading, caching, roles, daemon. | +| 5 | [Frontend deep dive](05-frontend.md) | Before your first frontend change. The Go SSR service, static assets, design system. | +| 6 | [Cross-service contracts](06-contracts.md) | **Before any change that touches both.** These break silently and in production. | +| 7 | [CI and release](07-ci-and-release.md) | Before you open a PR. Nine workflows, three branches, three environments. | +| 8 | [Infra and secrets](08-infra-secrets.md) | Before you touch deploy, compose, or a secret. | +| 9 | [Observability](09-observability.md) | When something is wrong and you need to see it. | +| 10 | [Recipes](10-recipes.md) | Task-shaped walkthroughs for the things you'll actually do. | +| 11 | [Traps and stale docs](11-traps.md) | **Skim early, re-read often.** Which docs in this repo currently lie, and why. | + +## The short version + +Thermograph grades how unusual today's weather is at any point on Earth against +~45 years of that exact location's own history. Percentiles, never thermometer +readings. + +- **`backend/`** — Python 3.12 / FastAPI. Owns all climate data, grading, the + API, accounts, notifications, plus a Go daemon (`backend/daemon/`) that owns + the Discord gateway and recurring timers. Ships as `emi/thermograph/backend`. +- **`frontend/`** — **Go** SSR service (`frontend/server/`) plus every static + asset. No climate data, no database, no compute. Ships as + `emi/thermograph/frontend`. (The Python implementation at `frontend/*.py` is + the superseded original — see [traps](11-traps.md).) +- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets + vault, Terraform, ops query tooling. +- **`observability/`** — Loki + Grafana on beta, an Alloy agent per node. + +Branches stage environments: PR → `dev` (LAN dev) → `main` (beta) → `release` +(prod). The two app domains build and deploy **independently** — that +independence is the whole reason the split-then-reunify history exists, and +[contracts](06-contracts.md) is the list of things that keep it safe. + +## Your first day + +1. Read [Orientation](01-orientation.md) and [Traps](11-traps.md). +2. Work through [Local setup](02-setup.md) until `make test` is green in both + `backend/` and `frontend/server/`. +3. Read [Repo map](03-repo-map.md) with the tree open beside it. +4. Pick something small in the domain you're least afraid of, and follow + [Recipes](10-recipes.md) end to end — including opening the PR. + +## Your first week + +- Read [Contracts](06-contracts.md) properly. Every item on that list has + already cost someone a production incident somewhere in this project's + history; that's why each one is written down. +- Get Centralis wired up (see [Local setup](02-setup.md)) and ask it + `fleet_status`, `logs_overview`, `deployed_version`. You cannot debug this + estate blind, and the boxes are not reachable from a laptop off the + WireGuard mesh. +- Read the `CLAUDE.md` for each domain. They are terse, current, and binding. diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 5542457..4804185 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -7,65 +7,117 @@ no compute — everything comes from `backend/`'s content API over HTTP. Read the root `CLAUDE.md` first for the branch model, deploy contract and image names. +## The live service is Go — `server/` + +`server/` (module `thermograph/frontend`, Go 1.26) is what builds, tests, ships +and runs. It is the only thing in the `emi/thermograph/frontend` image. + +The Python files at this level — `app.py`, `content.py`, `api_client.py`, +`format.py`, `content_loader.py`, `paths.py`, `templates/*.html.j2` — are the +**superseded original implementation**, kept as the reference the port was made +from. Nothing deploys them. Don't add features there; don't take their file +paths as current. (They still carry duplicate copies of some contracts below — +notably `api_client.py`'s `API_VERSION` pin and `format.py`'s `F_COUNTRIES`. If +you use them locally, keep them in step or you will debug a phantom.) + +Deleting them is a live question, not a settled one — the Python test tiers +still run against the fixtures the Go tests share. Raise it rather than doing it +in passing. + ## Build & verify -- `make test` — whole suite (`scripts/test.sh`). -- `make test-unit` — hermetic unit tier: SSR rendering fed committed fixtures, - no Docker. **This is the tier CI runs.** -- `make test-integration` — pulls and runs the real backend image and tests the - live contract. Local only; CI does not run it. +- `cd server && go build ./... && go vet ./... && go test ./...` — **the suite + that matters.** +- **CI runs it inside `Dockerfile`'s builder stage**, which does `gofmt -l` + + `go vet` + `go test ./...` before compiling. A failing Go test therefore fails + the image build — that is the whole frontend check. There is no separate + frontend test step in any workflow. +- `make test-unit` / `make test-integration` — the **Python** tiers. Local only; + CI runs neither. The unit tier is hermetic (SSR rendering fed committed + fixtures); the integration tier pulls and runs a real backend image. - `make backend-up` / `make backend-down` — a local backend container for dev. + The image tag is derived from this checkout (last commit touching `backend/`, + the same domain-keyed rule build-push and deploy use); override with + `THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex>`. - `make capture-fixtures` — refresh `tests/fixtures/*.json` from a live backend. + **These fixtures feed the Go tests too**, not just the Python ones. ## Running it -`THERMOGRAPH_API_BASE_INTERNAL` is **required** — `api_client.py` raises at -import if unset. +`THERMOGRAPH_API_BASE_INTERNAL` is **required** — the boot fails loudly without +it, as the Python raised at import. + +Build in `server/`, run from `frontend/`: `static/` and `content/` resolve +relative to the working directory, while templates are embedded in the binary. ```bash +cd server && go build -o thermograph-frontend . +cd .. THERMOGRAPH_API_BASE_INTERNAL=http://127.0.0.1:8137 THERMOGRAPH_BASE=/thermograph \ - .venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080 + ./server/thermograph-frontend ``` Other env: `THERMOGRAPH_BASE` (default `/thermograph`; the Dockerfile sets `/`), `THERMOGRAPH_API_VERSION` (default `v2`), `THERMOGRAPH_API_BASE_PUBLIC` (browser-facing backend origin when cross-origin; empty = same-origin, today's -default), `THERMOGRAPH_SSR_CACHE_TTL` (default 600s). +default), `THERMOGRAPH_SSR_CACHE_TTL` (default 600s), +`THERMOGRAPH_GOOGLE_VERIFY` / `THERMOGRAPH_BING_VERIFY`, `PORT` (default 8080). + +Boot is fail-loud on bad **config or content**: a missing backend URL, a garbage +TTL, or malformed `content/*.yaml` all stop the process rather than 500 on the +first request. ## Layout -- **`content.py`** — SSR routes (climate hub, city, month, records, glossary, - about, privacy) + `robots.txt` + `sitemap.xml`. Each fetches from the backend - via `api_client.py`; the backend's `content_payloads.py` owns `page_title` / - `canonical_path` / `breadcrumb` / `jsonld`, not this domain. +- **`server/internal/content/`** — the SSR page handlers (climate hub, city, + month, records, glossary, about, privacy) + `robots.txt` + `sitemap.xml`, + plus the template funcmap and SEO helpers. The backend's + `content_payloads.py` owns `page_title` / `canonical_path` / `breadcrumb` / + `jsonld`, **not this domain**. +- **`server/internal/contentapi/`** — the backend `/content/*` client: TTL + cache, bounded LRU, per-key single-flight, origin forwarding. All four + properties are load-bearing; read the file before changing caching. +- **`server/internal/render/`** + `render/templates/*.tmpl` — `html/template` + over an `embed.FS`, plus the ETag helpers. Templates are **embedded**, so a + template change needs a rebuild. +- **`server/internal/{config,format,contentdata,handlers}/`** — env parsing, + unit-aware °C/°F formatting and band names, the glossary/pages loader, and + the SPA-shell + static handlers. - **`static/*.js`** — `app.js` (map/search/results + inline SVG chart), `calendar.js`/`day.js`/`score.js`/`compare.js` (SPA shells), `account.js` (auth), `cache.js` (IndexedDB cache + `/cell` bundle prefetch), `shared.js`. - **`static/style.css`** — the single hand-written stylesheet; design tokens live here, documented in `DESIGN.md`. -- **`templates/*.html.j2`**, **`content/*.yaml`** (structured SSR copy, loaded by - `content_loader.py`). +- **`content/*.yaml`** — structured SSR copy, loaded by + `server/internal/contentdata`. ## Contracts with the backend -- **API version is pinned in exactly two places** — `api_client.py`'s - `API_VERSION` (Python) and `static/account.js`'s exported `API_VERSION` plus the - `uv(path)` helper (JS). Never hardcode `api/v2/...` anywhere else. Bump both in - one PR, and only after the target backend's `/api/version` confirms it serves - that version and its `min_frontend` doesn't exclude the one you're leaving. +- **API version is pinned in exactly two places** — `server/internal/config`'s + `THERMOGRAPH_API_VERSION` (Go) and `static/account.js`'s exported + `API_VERSION` plus the `uv(path)` helper (JS). Never hardcode `api/v2/...` + anywhere else. Bump both in one PR, and only after the target backend's + `/api/version` confirms it serves that version and its `min_frontend` doesn't + exclude the one you're leaving. - **`shared.js::pctOrd()` must mirror `backend`'s `data/grading.py::pct_ordinal()`** - in behaviour — floor into `1..99`, never 0 or 100. Every percentile on every - surface goes through one of the two; if they diverge, the same reading says two + in behaviour — floor into `1..99`, never 0 or 100 — and so must + `server/internal/format`'s `PctOrdinal`. Every percentile on every surface + goes through one of the three; if they diverge, the same reading says two different things in two places. - **`/cell` bundle + ETag** — `cache.js` fetches `/api/v2/cell` once per view-set and slices it, revalidating with `If-None-Match`. The slice→view map must track the backend's payload shape. -- **Fahrenheit country set** — `format.py::F_COUNTRIES` and `static/units.js`'s - `F_REGIONS` must stay identical to the backend's `F_COUNTRIES`. -- **Boot must survive an unreachable backend.** `content.py`'s `register()` tries - the IndexNow-key fetch once, catches failure, and falls back to a lazy - per-request lookup. Asynchronous FE/BE deploys depend on this — don't make it - fatal. +- **Fahrenheit country set** — `server/internal/format`'s `FCountries` and + `static/units.js`'s `F_REGIONS` must stay identical to the backend's + `F_COUNTRIES`. Both are now asserted by tests in + `server/internal/format/format_test.go`: the backend cross-check is a + checkout-only guard (the image build context is `frontend/`, so `backend/` is + unreachable there), while the `units.js` check runs in the image build too — + `Dockerfile` copies that one file into the builder stage for exactly that + reason. +- **Boot must survive an unreachable backend.** The IndexNow-key fetch is tried + once at boot, caught, and falls back to a lazy per-request lookup. + Asynchronous FE/BE deploys depend on this — don't make it fatal. ## Commits & PRs diff --git a/frontend/Dockerfile b/frontend/Dockerfile index 692ecb9..920c545 100644 --- a/frontend/Dockerfile +++ b/frontend/Dockerfile @@ -35,6 +35,19 @@ COPY server/ ./ COPY tests/fixtures /tests/fixtures COPY content /content +# static/units.js is copied for ONE test: internal/format's +# TestFCountriesMatchesUnitsJS, which asserts the browser's F_REGIONS still +# matches the Go/backend Fahrenheit country set. The builder stage is the only +# place CI ever executes these tests, so without this the check would skip in CI +# and only ever run on a developer's checkout. Same three-levels-up relationship +# the test expects (/src/internal/format -> /static/units.js), anchored the same +# way the two directories above are. +# +# The sibling backend cross-check cannot be wired up this way: the build context +# is frontend/, so backend/ is structurally unreachable and that test stays a +# checkout-only guard. +COPY static/units.js /static/units.js + RUN test -z "$(gofmt -l .)" && go vet ./... && go test ./... # Static binary: CGO off (no libc dependency on Alpine), -trimpath for diff --git a/frontend/README.md b/frontend/README.md index ccc40d5..c22d8e4 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -2,107 +2,92 @@ The server-rendered content pages, the interactive-tool SPA shells, and every static asset for [Thermograph](https://thermograph.org) — grades recent local -weather against ~45 years of climate history. This repo holds **no climate -data and does no compute**; it renders pages and serves the JS/CSS/image -assets that call the backend's API from the browser. It was split out of the -`emi/thermograph` monorepo (`frontend_ssr/` + `frontend/` there) so it can -build its own image and deploy independently of the backend. See -[`CLAUDE.md`](CLAUDE.md) for the full agent-facing detail on the split -topology and deploy contract, and -[`thermograph-docs`](../thermograph-docs) (sibling repo) for the -architecture decision record behind the split. +weather against ~45 years of climate history. This domain holds **no climate +data and does no compute**; it renders pages and serves the JS/CSS/image assets +that call the backend's API from the browser. It builds its own image +(`emi/thermograph/frontend`) and deploys independently of the backend. -## Layout +See [`CLAUDE.md`](CLAUDE.md) for the agent-facing detail on the deploy contract +and the cross-service contracts, and [`DESIGN.md`](DESIGN.md) for the visual +system. + +## The service is Go + +`server/` is the live implementation. The Python files at this level (`app.py`, +`content.py`, `api_client.py`, `format.py`, `content_loader.py`, `paths.py`, +`templates/*.html.j2`) are the **superseded original**, kept as the reference +the port was made from — nothing deploys them. See `server/README.md` for the +port's own notes. ``` -api_client.py HTTP client for the backend's content API (api/content_routes.py - on the backend). Fails loud at import if - THERMOGRAPH_API_BASE_INTERNAL is unset. TTL-cached (10 min - default) in front of every call. -app.py FastAPI app: /healthz, the interactive tool's SPA-shell - routes (calendar/day/score/compare/legend/alerts), and the - StaticFiles mount for everything else. Calls - content.register(app) for the SSR routes below. -content.py Server-rendered pages: climate hub, per-city, month, - records, glossary, about, privacy, robots.txt, sitemap.xml. -content_loader.py Loads content/glossary.yaml + content/pages.yaml (SSR copy), - validated fail-loud at load time. -format.py Unit-aware (°C/°F) formatting for the SSR templates, - ContextVar-scoped active unit. -paths.py Canonical filesystem locations (STATIC_DIR, TEMPLATES_DIR, - CONTENT_DIR), resolved from the repo root. -templates/ Jinja templates content.py renders (base, home, city, month, - hub, records, glossary, glossary_term, about, privacy). -static/ Every static asset: the interactive tool's JS - (app.js, calendar.js, day.js, score.js, compare.js, - account.js, cache.js, shared.js, units.js, mappicker.js, …), - the SPA-shell HTML pages, style.css (all design tokens — - see DESIGN.md), icons/favicons, manifest.webmanifest, sw.js. -content/ Structured SSR copy: glossary.yaml, pages.yaml. -tests/ pytest suite — see the KNOWN GAP note below before relying on it. -tools/ shoot.py — the make-shots visual-verify screenshot tool - (see DESIGN.md). -Dockerfile Builds this repo's own image (python:3.12-slim, - uvicorn app:app), independent of the backend's. +server/ the service (module thermograph/frontend, Go 1.26) + main.go config + mux + static + graceful shutdown + internal/config/ every env var it reads + internal/contentapi/ backend /content/* client: TTL cache, bounded LRU, + per-key single-flight, origin forwarding + internal/content/ SSR page handlers, template funcmap, SEO + internal/contentdata/ glossary.yaml / pages.yaml loader (fail-loud) + internal/format/ unit-aware °C/°F formatting + band names + internal/handlers/ SPA shells + static serving + internal/render/ html/template over embed.FS + ETag helpers + internal/render/templates/ the page templates (embedded, *.tmpl) +static/ every static asset: the interactive tool's JS + (app.js, calendar.js, day.js, score.js, compare.js, + account.js, cache.js, shared.js, units.js, + mappicker.js, …), the SPA-shell HTML pages, + style.css (all design tokens — see DESIGN.md), + icons/favicons, manifest.webmanifest, sw.js +content/ structured SSR copy: glossary.yaml, pages.yaml +tests/fixtures/ golden payloads — consumed by the GO tests as well + as the Python tiers +tests/{unit,integration}/ the Python tiers (local only; CI runs neither) +tools/shoot.py the screenshot sweep for visual verification +Dockerfile golang builder (gofmt + vet + test) → alpine runtime ``` -## How it fits the split - -Four sibling repos replace the old monorepo: - -- **`thermograph-backend`** — the FastAPI API, grading/scoring/grid compute, - and DB. This repo's only runtime dependency. -- **`thermograph-frontend`** (this repo). -- **`thermograph-infra`** — Terraform, compose files, `deploy/deploy.sh` (the - script both repos' deploy workflows SSH into), the secrets vault. -- **`thermograph-docs`** — architecture/decision docs and runbooks. - -Each app repo now builds and publishes its **own** image -(`git.thermograph.org/emi/thermograph-frontend/app`, tagged by git SHA and -semver) instead of the old shared `emi/thermograph/app` image, and deploys via -its own `.forgejo/workflows/deploy.yml` (main → beta) and `deploy-prod.yml` -(release → prod), each rolling only the `frontend` compose service on the -target VPS. +Dependencies are **stdlib plus `gopkg.in/yaml.v3`** — the committed SSR copy in +`content/*.yaml` uses block/folded scalars, so a real YAML parser is required. ## Talking to the backend -All data comes from the backend's content API over HTTP -(`api_client.py`) — this process holds nothing in-process. Two env vars -control the topology: +All data comes from the backend's content API over HTTP — this process holds +nothing. Two env vars control the topology: - **`THERMOGRAPH_API_BASE_INTERNAL`** (required) — where this process reaches the backend server-side, e.g. `http://backend:8137` in compose, or - `http://127.0.0.1:8137` for a local backend checkout. `api_client.py` raises - at import if this is unset — a missing backend URL should break boot, not - silently 500 the first request. -- **`THERMOGRAPH_API_BASE_PUBLIC`** (optional) — the backend's - browser-facing origin, used only when frontend and backend are served - cross-origin (e.g. a genuinely separate LAN-dev topology). Left empty - (the default), asset/API references stay relative and same-origin, which is - today's real deployed topology. + `http://127.0.0.1:8137` for a local backend. Boot **fails loudly** if unset: + a missing backend URL should break the boot, not silently 500 on the first + request. +- **`THERMOGRAPH_API_BASE_PUBLIC`** (optional) — the backend's browser-facing + origin, used only when frontend and backend are served cross-origin. Left + empty (the default), asset/API references stay relative and same-origin, + which is today's real deployed topology. + +`THERMOGRAPH_BASE` (default `/thermograph`) must be set identically on both +frontend and backend in every real deployment — it's how both processes agree on +the shared URL prefix (or the deployed clean-root `/`, which the Dockerfile +sets). **Same-origin cookie-auth caveat:** the interactive tool's auth (`static/account.js`) is an HttpOnly session cookie. It uses `credentials: "include"` (not the fetch default) specifically so the cookie -still rides along if this script is ever loaded cross-origin, but the backend -must actually be configured to accept credentialed cross-origin requests -(CORS + `SameSite`) for that case to work — in the default same-origin -deployment this is a non-issue. Don't assume a fully decoupled, independently- -hosted frontend "just works" for logged-in features without checking that -cookie/CORS configuration first. - -`THERMOGRAPH_BASE` (default `/thermograph`) must be set identically on both -frontend and backend in every real deployment — it's how both processes agree -on the shared URL prefix (or the deployed clean-root `/`, which the Dockerfile -sets by default). +still rides along if this script is ever loaded cross-origin — but the backend +must actually be configured to accept credentialed cross-origin requests (CORS + +`SameSite`) for that case to work. In the default same-origin deployment this is +a non-issue. Don't assume a fully decoupled, independently-hosted frontend "just +works" for logged-in features without checking that configuration first. ## Build & run +Build in `server/`, run from here — `static/` and `content/` resolve relative to +the working directory, while templates are embedded in the binary: + ```bash -python3 -m venv .venv && .venv/bin/pip install -r requirements.txt +cd server && go build -o thermograph-frontend . +cd .. THERMOGRAPH_API_BASE_INTERNAL=http://127.0.0.1:8137 \ THERMOGRAPH_BASE=/thermograph \ - .venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080 + ./server/thermograph-frontend ``` Or build the image directly: @@ -115,12 +100,37 @@ docker run -p 8080:8080 -e THERMOGRAPH_API_BASE_INTERNAL=http://backend:8137 the The image healthchecks `GET /healthz` (liveness only — it does no I/O, so it stays cheap; it does not prove the backend is reachable). +## Test + +```bash +cd server && go build ./... && go vet ./... && go test ./... +``` + +`Dockerfile`'s builder stage runs `gofmt -l` + `go vet` + `go test ./...` before +compiling, so **a failing Go test fails the image build** — and that is the +whole frontend check in CI. There is no separate frontend test step in any +workflow. + +The Python tiers are local-only: + +```bash +make test-unit # hermetic: SSR rendering fed committed fixtures +make test-integration # pulls + runs a real backend image +make backend-up # just the backend container, for dev +make capture-fixtures # refresh tests/fixtures/*.json from a live backend +``` + +`make backend-up` derives the backend image tag from this checkout (the last +commit touching `backend/` — the same domain-keyed rule build-push and deploy +use); pin one with `THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex>`. + ## More detail -- [`CLAUDE.md`](CLAUDE.md) — agent-facing instructions: deploy contract, - API-version pinning, the cross-repo `/cell`/ETag and `pctOrd()` contracts, - and the current known test-suite gap. +- [`CLAUDE.md`](CLAUDE.md) — agent-facing instructions: build/verify, layout, + API-version pinning, and the `/cell`/ETag, `pctOrd()` and Fahrenheit-set + contracts. +- [`server/README.md`](server/README.md) — the Go port's own notes. - [`DESIGN.md`](DESIGN.md) — the visual system (tokens, components, - breakpoints) and `make shots` visual verification. -- [`thermograph-docs`](../thermograph-docs) — the repo-topology decision - record this split implements. + breakpoints) and `tools/shoot.py` visual verification. +- [`docs/onboarding/`](../docs/onboarding/README.md) — the developer onboarding + path for the whole monorepo. diff --git a/frontend/docker-compose.test.yml b/frontend/docker-compose.test.yml index 83b5d33..234fa2d 100644 --- a/frontend/docker-compose.test.yml +++ b/frontend/docker-compose.test.yml @@ -20,9 +20,16 @@ services: retries: 20 backend: - # The published backend image is pulled (no local build). Default a published tag; - # override THERMOGRAPH_BACKEND_TEST_TAG to pin a specific backend build (e.g. a sha). - image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph-backend/app}:${THERMOGRAPH_BACKEND_TEST_TAG:-v0.0.2-split-ci} + # The published backend image is pulled (no local build). + # + # No default tag on purpose: scripts/backend-for-tests.sh computes one from + # THIS checkout -- sha-<12hex of `git log -1 -- backend/`>, the same + # domain-keyed rule build-push.yml and deploy.yml both use -- so the harness + # tracks the tree instead of drifting behind a hardcoded pin. (It was pinned + # to the split-era v0.0.2-split-ci, under the retired + # emi/thermograph-backend/app path, long after both were superseded.) + # Override THERMOGRAPH_BACKEND_TEST_TAG to pin a specific build. + image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${THERMOGRAPH_BACKEND_TEST_TAG:?set it, or run via scripts/backend-for-tests.sh which derives it} depends_on: db: condition: service_healthy diff --git a/frontend/scripts/backend-for-tests.sh b/frontend/scripts/backend-for-tests.sh index 0770a96..1f48ab4 100755 --- a/frontend/scripts/backend-for-tests.sh +++ b/frontend/scripts/backend-for-tests.sh @@ -7,8 +7,9 @@ # scripts/backend-for-tests.sh down # stop + remove (incl. the throwaway db volume) # scripts/backend-for-tests.sh url # print the base URL (no lifecycle change) # -# Which backend build: THERMOGRAPH_BACKEND_TEST_TAG (default a published tag; set a -# sha-<12hex> to pin one). Host port: BACKEND_TEST_HOST_PORT (default 18137). +# Which backend build: THERMOGRAPH_BACKEND_TEST_TAG. Unset, it is DERIVED from this +# checkout (see below) rather than defaulted to a hardcoded tag -- set a sha-<12hex> +# to pin one. Host port: BACKEND_TEST_HOST_PORT (default 18137). set -euo pipefail cd "$(dirname "$0")/.." @@ -17,10 +18,39 @@ export BACKEND_TEST_HOST_PORT="${BACKEND_TEST_HOST_PORT:-18137}" BASE_URL="http://127.0.0.1:${BACKEND_TEST_HOST_PORT}" COMPOSE=(docker compose -f docker-compose.test.yml) +# Derive the backend image tag the SAME way build-push.yml and deploy.yml do: +# keyed to the last commit that touched backend/, not the branch tip. That is the +# image actually published for this checkout's backend tree, so the harness follows +# the tree instead of drifting behind a pin (this file previously defaulted to the +# split-era v0.0.2-split-ci tag under a retired image path). +# +# Exported before ANY compose call -- docker-compose.test.yml requires the variable +# (`:?`), so `down` and `url` need it set too, not just `up`. +# +# `:/backend/` is a repo-ROOT-relative pathspec: a bare `backend/` would resolve +# against this script's cwd (frontend/) and silently match nothing, yielding an +# empty sha and a confusing error instead of the right tag. +if [ -z "${THERMOGRAPH_BACKEND_TEST_TAG:-}" ]; then + domain_sha="$(git log -1 --format=%H -- ':/backend/' 2>/dev/null || true)" + if [ -z "$domain_sha" ]; then + echo "!! could not derive a backend image tag from git (no backend/ history here?)." >&2 + echo " Set THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> explicitly." >&2 + exit 2 + fi + THERMOGRAPH_BACKEND_TEST_TAG="sha-${domain_sha:0:12}" +fi +export THERMOGRAPH_BACKEND_TEST_TAG + case "${1:-up}" in up) - echo "==> pulling backend image (${THERMOGRAPH_BACKEND_TEST_TAG:-v0.0.2-split-ci})" >&2 - "${COMPOSE[@]}" pull backend >&2 + echo "==> pulling backend image (${THERMOGRAPH_BACKEND_TEST_TAG})" >&2 + if ! "${COMPOSE[@]}" pull backend >&2; then + echo "!! no published backend image for ${THERMOGRAPH_BACKEND_TEST_TAG}." >&2 + echo " That tag is keyed to the last commit touching backend/ -- if those" >&2 + echo " commits are local-only, build-push.yml has not published it yet." >&2 + echo " Pin a published build: THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> $0 up" >&2 + exit 1 + fi echo "==> starting backend + db" >&2 "${COMPOSE[@]}" up -d >&2 echo "==> waiting for backend /healthz (up to 90s)" >&2 diff --git a/frontend/server/internal/format/format_test.go b/frontend/server/internal/format/format_test.go index defa4b5..699f999 100644 --- a/frontend/server/internal/format/format_test.go +++ b/frontend/server/internal/format/format_test.go @@ -45,42 +45,93 @@ func TestUnitForCountry(t *testing.T) { } } -// TestFCountriesMatchesBackend re-parses the backend's F_COUNTRIES (the -// source of truth per both CLAUDE.md files) and asserts our set is identical. -// Skips when the monorepo backend checkout isn't present. -func TestFCountriesMatchesBackend(t *testing.T) { - path := filepath.Join("..", "..", "..", "..", "backend", "api", "content_payloads.py") +// parseCountryCodes pulls a set of ISO-3166 alpha-2 codes out of `path` by +// matching `outer` (which must capture the literal's body in group 1) and then +// scanning that body for quoted codes. Returns nil when the file isn't present, +// so a caller can skip rather than fail in a checkout/build context that does +// not carry it. +func parseCountryCodes(t *testing.T, path string, outer *regexp.Regexp) map[string]bool { + t.Helper() raw, err := os.ReadFile(path) if err != nil { - t.Skipf("backend checkout not present (%v) — parity asserted only against the committed copy", err) + return nil } - m := regexp.MustCompile(`(?s)F_COUNTRIES\s*=\s*frozenset\(\{(.*?)\}\)`).FindSubmatch(raw) + m := outer.FindSubmatch(raw) if m == nil { - t.Fatalf("could not find F_COUNTRIES in %s", path) + t.Fatalf("could not find the country-set literal in %s", path) } codes := regexp.MustCompile(`"([A-Z]{2})"`).FindAllSubmatch(m[1], -1) if len(codes) == 0 { t.Fatalf("no codes parsed from %s", path) } - backend := map[string]bool{} + set := map[string]bool{} for _, c := range codes { - backend[string(c[1])] = true + set[string(c[1])] = true } - if len(backend) != len(FCountries) { - t.Errorf("F_COUNTRIES size mismatch: backend %d vs ours %d", len(backend), len(FCountries)) + return set +} + +// assertMatchesFCountries diffs a parsed copy against ours, both directions. +func assertMatchesFCountries(t *testing.T, name string, other map[string]bool) { + t.Helper() + if len(other) != len(FCountries) { + t.Errorf("size mismatch: %s %d vs ours %d", name, len(other), len(FCountries)) } - for code := range backend { + for code := range other { if !FCountries[code] { - t.Errorf("backend has %q, ours does not", code) + t.Errorf("%s has %q, ours does not", name, code) } } for code := range FCountries { - if !backend[code] { - t.Errorf("ours has %q, backend does not", code) + if !other[code] { + t.Errorf("ours has %q, %s does not", code, name) } } } +// TestFCountriesMatchesBackend re-parses the backend's F_COUNTRIES (the +// source of truth per both CLAUDE.md files) and asserts our set is identical. +// Skips when the monorepo backend checkout isn't present — notably inside +// frontend/Dockerfile's builder stage, whose build context is frontend/ only, +// so backend/ is structurally unreachable there. This check is therefore a +// local/CI-checkout guard, not an image-build one. +func TestFCountriesMatchesBackend(t *testing.T) { + path := filepath.Join("..", "..", "..", "..", "backend", "api", "content_payloads.py") + backend := parseCountryCodes(t, path, regexp.MustCompile(`(?s)F_COUNTRIES\s*=\s*frozenset\(\{(.*?)\}\)`)) + if backend == nil { + t.Skip("backend checkout not present — parity asserted only against the committed copy") + } + assertMatchesFCountries(t, "backend", backend) +} + +// TestFCountriesMatchesUnitsJS asserts the BROWSER copy — static/units.js's +// F_REGIONS — matches too. +// +// This copy was previously guarded by nothing at all, despite comments in +// backend/api/content_payloads.py, frontend/format.py and format.go each +// claiming "a test asserts all three stay identical": the only check compared +// Go against the backend's Python, and nothing read units.js. The sets happened +// to agree, held in step by convention alone. Client-side unit selection +// disagreeing with server-rendered unit selection means the same page shows °C +// in SSR and °F after hydration — a silent, per-country split. +// +// Unlike the backend check above, static/ IS inside the frontend build context, +// and frontend/Dockerfile copies units.js into the builder stage precisely so +// this runs during the image build — the only place CI executes these tests. +func TestFCountriesMatchesUnitsJS(t *testing.T) { + for _, path := range []string{ + filepath.Join("..", "..", "..", "static", "units.js"), // repo checkout + filepath.Join("/", "static", "units.js"), // Dockerfile builder stage + } { + js := parseCountryCodes(t, path, regexp.MustCompile(`(?s)F_REGIONS\s*=\s*new Set\(\[(.*?)\]\)`)) + if js != nil { + assertMatchesFCountries(t, "static/units.js", js) + return + } + } + t.Skip("static/units.js not reachable from this working directory") +} + func TestTempSpans(t *testing.T) { // Exact rendered bytes — the golden-diff contract. if got := Temp("F", fp(71.1)); got != `71°F` {