promote: main → release #132
13 changed files with 2531 additions and 0 deletions
|
|
@ -16,6 +16,13 @@ for: **per-domain images, per-domain deploys, and an async FE/BE contract**.
|
|||
`thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no
|
||||
build artifacts, different change cadence).
|
||||
|
||||
## New here?
|
||||
|
||||
[`docs/onboarding/`](docs/onboarding/README.md) is the developer onboarding
|
||||
path: orientation, verified local-setup recipes, a per-domain deep dive, the
|
||||
cross-service contracts that break silently, the release flow, and a list of
|
||||
which docs in this repo are currently stale.
|
||||
|
||||
## How CI stays decoupled
|
||||
|
||||
Every workflow in `.forgejo/workflows/` is **path-filtered to its domain**: a
|
||||
|
|
|
|||
146
docs/onboarding/01-orientation.md
Normal file
146
docs/onboarding/01-orientation.md
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
# 1. Orientation
|
||||
|
||||
## What the product actually claims
|
||||
|
||||
Thermograph answers one question: **how unusual is the weather here, right now,
|
||||
compared to this exact place's own history?**
|
||||
|
||||
That framing is not decoration — it constrains the code, the copy and the
|
||||
colour scales:
|
||||
|
||||
- A grade is a **percentile against ~45 years of that grid cell's own record**,
|
||||
not an absolute reading. 60 °F is *Above Normal* on a cool coast and *Below
|
||||
Normal* inland on the same afternoon.
|
||||
- Tier names are therefore relative words — *Near Record*, *High*, *Above
|
||||
Normal*, *Normal*, *Below Normal*, *Low*, *Near Record* — never "hot",
|
||||
"warm", "cold". This rule binds prose, alt text, commit messages and API
|
||||
fields alike.
|
||||
- Precipitation is graded on its **own** ladder, because most days are dry: a
|
||||
rainy day is ranked only among the rain days in its seasonal window, and dry
|
||||
days are coloured by dry-streak length instead.
|
||||
|
||||
Public since **2026-07-16**. Free. Run by one operator plus automation, which
|
||||
is why so much of this repo is about making irreversible things hard.
|
||||
|
||||
## The domain model, in six steps
|
||||
|
||||
1. **Grid** (`backend/data/grid.py`) — an arbitrary lat/lon snaps to a stable
|
||||
**~4 sq mi cell**. Latitude rows are ~2 miles tall; longitude spacing scales
|
||||
by `cos(latitude)` so cells stay roughly square at any latitude. The cell id
|
||||
is deterministic and is the cache key for everything downstream. Worldwide
|
||||
coverage, no gaps.
|
||||
2. **History** (`backend/data/climate.py`) — the full **1980 → present daily
|
||||
record** (max/min temp, precip, wind, gust, humidity, feels-like) for that
|
||||
cell, fetched once and stored durably. Expensive to obtain, so it is treated
|
||||
as source-of-truth data, not cache.
|
||||
3. **Recent + forecast** — a separate bundle of recent observations plus ~8
|
||||
forward days, refreshed on its own TTL. This is what "today" is graded from.
|
||||
4. **Grading** (`backend/data/grading.py`) — for a given day of year, the
|
||||
reference distribution is *every historical day within ±7 days of it*
|
||||
(a 15-day seasonal window that wraps the year end). The observed value is
|
||||
placed on that distribution as an **empirical mid-rank percentile**, then
|
||||
mapped onto `TEMP_BANDS` / `RAIN_BANDS`.
|
||||
5. **Payloads** (`backend/api/payloads.py`) — pure "inputs → response dict"
|
||||
assembly. One definition per payload shape, shared by the HTTP routes, the
|
||||
`/cell` bundle and offline tooling.
|
||||
6. **Caching** (`backend/data/store.py`) — every derived payload is stored
|
||||
against a **validity token** that encodes everything the payload depends on.
|
||||
A token mismatch is simply a miss. Freshness is driven by the token
|
||||
advancing, *never* by clock expiry — so stale data cannot be served, and one
|
||||
`PAYLOAD_VER` bump atomically orphans every pre-upgrade row.
|
||||
|
||||
If you internalise one thing: **the token is the cache-correctness mechanism,
|
||||
and `PAYLOAD_VER` is the lever that moves it.** See
|
||||
[contracts](06-contracts.md).
|
||||
|
||||
## The estate
|
||||
|
||||
Four machines on a WireGuard mesh (`10.10.0.0/24`):
|
||||
|
||||
| Host | Mesh IP | Public | Runs |
|
||||
|---|---|---|---|
|
||||
| **prod** | `10.10.0.1` | `169.58.46.181`, thermograph.org | Docker **Swarm** app stack, TimescaleDB, Postfix, backups, Centralis |
|
||||
| **beta** | `10.10.0.2` | `75.119.132.91`, beta.thermograph.org | **compose** app stack, Forgejo (git + CI + registry), Grafana + Loki |
|
||||
| **desktop** | `10.10.0.3` | — | LAN dev stack, the Forgejo Actions runner, your editor |
|
||||
| phone | — | — | alerts |
|
||||
|
||||
Two consequences you will hit within the first week:
|
||||
|
||||
- **Forgejo is mesh-only.** `git.thermograph.org` resolves publicly to beta's
|
||||
IP, but beta's Caddy rejects `/v2/*` (the registry API) from outside the mesh.
|
||||
Any host that pulls images needs `10.10.0.2 git.thermograph.org` in
|
||||
`/etc/hosts`.
|
||||
- **Prod and beta run different orchestrators.** Prod is Swarm
|
||||
(`infra/deploy/stack/`), beta and LAN dev are compose
|
||||
(`infra/docker-compose.yml`). `deploy.sh` routes between them by reading
|
||||
`/etc/thermograph/deploy-mode` on the box. Most of the time you don't care —
|
||||
but when you're reading logs or naming containers, you do
|
||||
(see [observability](09-observability.md)).
|
||||
|
||||
## How a change travels
|
||||
|
||||
```
|
||||
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
|
||||
│
|
||||
promotion PR
|
||||
▼
|
||||
main ──▶ beta.thermograph.org
|
||||
│
|
||||
promotion PR ← the owner's call
|
||||
▼
|
||||
release ──▶ thermograph.org
|
||||
```
|
||||
|
||||
`dev`, `main` and `release` are **protected** — no direct pushes, for humans or
|
||||
agents. Promotion is by pull request, continuously and per-decision; there is
|
||||
no release calendar. Full detail in [CI and release](07-ci-and-release.md).
|
||||
|
||||
## The four rules that are not negotiable
|
||||
|
||||
These come from the operator, via Centralis's own onboarding. They override
|
||||
convenience.
|
||||
|
||||
1. **Everything is a PR.** If it didn't go through the pipeline, it didn't
|
||||
happen. This includes Grafana dashboards (provisioned from repo JSON — a UI
|
||||
edit is silently overwritten) and secrets (SOPS vault only).
|
||||
2. **Boring prod.** Anything clever proves itself on `dev` or beta first.
|
||||
3. **Privacy is a feature.** No tracking cookies, no per-visitor IDs,
|
||||
allowlisted anonymous counters only (`backend/core/metrics.py`,
|
||||
`POST /api/v2/event`). Cheap to keep, expensive to retrofit.
|
||||
4. **Grades are relative.** Never describe a percentile as hot or cold.
|
||||
|
||||
### And a fifth, local to the repo
|
||||
|
||||
**Commits and PRs are concise and technical, and never mention AI, assistants
|
||||
or automated authorship.** Every domain `CLAUDE.md` says this; it applies to
|
||||
you too.
|
||||
|
||||
## Quota discipline
|
||||
|
||||
Thermograph runs on free upstream data sources. Two hard rules protect them:
|
||||
|
||||
- **Anything named "prefetch" or "warm" must never spend the Open-Meteo
|
||||
quota.** The warming paths call `climate.load_cached_history` /
|
||||
`load_cached_recent_forecast` — the cache-only loaders — and skip cells that
|
||||
aren't warm. A cell self-heals on first real request instead.
|
||||
- **Nominatim gets at most ~1 request/second.** All reverse geocoding funnels
|
||||
through a single dedicated worker thread
|
||||
(`climate._revgeo_worker`) that paces itself; never call it from a request
|
||||
thread.
|
||||
|
||||
Breaking either one gets the whole service rate-limited, which is why
|
||||
`singleton.claim_leader()` exists: a background sweep running in N uvicorn
|
||||
workers × M hosts multiplies quota use by N×M. That failure has already
|
||||
happened once, in production, at 3 workers.
|
||||
|
||||
## Where the written record lives
|
||||
|
||||
| Question | Where |
|
||||
|---|---|
|
||||
| How does this repo work? | The domain `CLAUDE.md` files, then this set |
|
||||
| Why is the architecture like this? | `thermograph-docs` (ADRs) — `docs_search` |
|
||||
| How do I operate the fleet? | `thermograph-docs` runbooks, plus the `thermograph-ops` Centralis skill |
|
||||
| What did someone find out last Tuesday? | `notes_search` |
|
||||
| What is and isn't live yet? | [`CUTOVER-NOTES.md`](../../CUTOVER-NOTES.md) at the repo root |
|
||||
|
||||
Next: [Local setup](02-setup.md).
|
||||
217
docs/onboarding/02-setup.md
Normal file
217
docs/onboarding/02-setup.md
Normal file
|
|
@ -0,0 +1,217 @@
|
|||
# 2. Local setup
|
||||
|
||||
Every command in this document was run against this checkout and produced the
|
||||
output shown. If one fails for you, the difference is your machine, not the doc.
|
||||
|
||||
## Toolchain
|
||||
|
||||
| Tool | Why | Notes |
|
||||
|---|---|---|
|
||||
| **Python 3.12** | backend runtime and test suite | Must have the `_sqlite3` extension. A pyenv 3.10 without it will fail at `conftest` import. `scripts/test.sh` looks for `python3.12` (or uses `uv`) rather than whatever `python3` is on `PATH` — deliberately. |
|
||||
| **Go 1.26** | frontend service and the backend daemon | Both are static `CGO_ENABLED=0` builds. |
|
||||
| **Docker** | images, the compose/Swarm stacks, smoke tests | Docker CLI ≥ 27 is fine. |
|
||||
| **uv** *(recommended)* | fast venv creation | `scripts/test.sh` uses it when present. |
|
||||
| **shellcheck v0.11.0** | matches CI's pin exactly | Install to `~/.local/bin/shellcheck`. A *different* version can invent new findings and fail CI on an unrelated push. |
|
||||
| **sops + age** | reading/editing the secrets vault | Only needed if you touch `infra/deploy/secrets/`. |
|
||||
| **jq** | the repo's `.claude` hooks use it | Without it the prod-guard hook fails toward *asking*, which is safe but noisy. |
|
||||
|
||||
Check what you have:
|
||||
|
||||
```bash
|
||||
which go python3.12 uv docker sops age jq shellcheck
|
||||
go version && python3.12 --version && docker --version
|
||||
```
|
||||
|
||||
## Backend
|
||||
|
||||
### Test suite (hermetic, no network, no Docker)
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
make test # or: ./scripts/test.sh
|
||||
make test ARGS='tests/data -q' # pass pytest args through
|
||||
```
|
||||
|
||||
Verified: **429 passed, 8 skipped in 11.57s** on a cold venv build.
|
||||
|
||||
The first run builds `.venv-test/` on Python 3.12 and installs
|
||||
`requirements-dev.txt`. `tests/conftest.py` is what makes the suite hermetic:
|
||||
|
||||
- no real Open-Meteo, Nominatim or GeoNames calls (the places index is marked
|
||||
as already-loaded so no background download starts);
|
||||
- a throwaway SQLite accounts DB and derived store in `/tmp`, never the repo's
|
||||
`data/`;
|
||||
- audit/error/access/activity/heartbeat log dirs redirected to `/tmp`;
|
||||
- notifier and heartbeat threads disabled;
|
||||
- `THERMOGRAPH_FRONTEND_BASE_INTERNAL` pointed at an unreachable placeholder
|
||||
(the app fails loud at import without it).
|
||||
|
||||
### Boot it locally, no Docker, no Postgres
|
||||
|
||||
The backend falls back to SQLite when `THERMOGRAPH_DATABASE_URL` isn't a
|
||||
Postgres URL, so this is enough:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
THERMOGRAPH_FRONTEND_BASE_INTERNAL=http://127.0.0.1:8080 \
|
||||
THERMOGRAPH_BASE=/thermograph \
|
||||
THERMOGRAPH_ENABLE_NOTIFIER=0 \
|
||||
THERMOGRAPH_ENABLE_HEARTBEAT=0 \
|
||||
.venv-test/bin/python -m uvicorn app:app --host 127.0.0.1 --port 8137
|
||||
```
|
||||
|
||||
Verified responses:
|
||||
|
||||
```
|
||||
GET /healthz → {"status":"ok","role":"all"}
|
||||
GET /thermograph/api/version → {"backend_version":"2","min_frontend":"1","payload_ver":"p2"}
|
||||
```
|
||||
|
||||
`app:app` is a one-line re-export shim for `web/app.py`. Keep it — systemd, CI
|
||||
and the container entrypoint all target that name.
|
||||
|
||||
Note the port: **8137** is the backend everywhere in this project (compose,
|
||||
Caddy, the smoke harness at 18137, the frontend's default internal base).
|
||||
|
||||
### Image smoke test
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
make smoke # builds the image, boots it + a throwaway TimescaleDB (tmpfs),
|
||||
# asserts /healthz and /api/version
|
||||
```
|
||||
|
||||
Uses `docker-compose.test.yml` on host port 18137 so it can't collide with a
|
||||
dev server on 8137.
|
||||
|
||||
## Frontend
|
||||
|
||||
**The live frontend is Go.** `frontend/server/` is what builds, tests, ships
|
||||
and runs. The Python files one level up (`app.py`, `content.py`,
|
||||
`api_client.py`, `format.py`) are the superseded original — see
|
||||
[traps](11-traps.md).
|
||||
|
||||
### Test
|
||||
|
||||
```bash
|
||||
cd frontend/server
|
||||
go build ./... && go vet ./... && go test ./...
|
||||
```
|
||||
|
||||
Verified: all seven packages `ok` (`config`, `content`, `contentapi`,
|
||||
`contentdata`, `format`, `handlers`, `render`).
|
||||
|
||||
These same commands run inside `frontend/Dockerfile`'s builder stage — plus a
|
||||
`gofmt -l` check that must come back empty — so **a failing Go test fails the
|
||||
image build**, which is how CI catches it. There is no separate frontend test
|
||||
step in the workflows.
|
||||
|
||||
### Run it
|
||||
|
||||
The process resolves `static/` and `content/` **relative to its working
|
||||
directory**, so build in `server/` and run from `frontend/`:
|
||||
|
||||
```bash
|
||||
cd frontend/server && go build -o thermograph-frontend .
|
||||
cd ..
|
||||
THERMOGRAPH_API_BASE_INTERNAL=http://127.0.0.1:8137 \
|
||||
THERMOGRAPH_BASE=/thermograph \
|
||||
PORT=8080 \
|
||||
./server/thermograph-frontend
|
||||
```
|
||||
|
||||
Verified: `GET /healthz` → `{"status":"ok"}`, with a structured JSON log line
|
||||
per request.
|
||||
|
||||
`THERMOGRAPH_API_BASE_INTERNAL` is **required** — boot fails loudly without it,
|
||||
by design. Optional: `THERMOGRAPH_BASE` (default `/thermograph`; the image sets
|
||||
`/`), `THERMOGRAPH_API_VERSION` (default `v2` — only ever change it per the
|
||||
[API-version contract](06-contracts.md)), `THERMOGRAPH_API_BASE_PUBLIC`,
|
||||
`THERMOGRAPH_SSR_CACHE_TTL` (seconds, default 600), `THERMOGRAPH_GOOGLE_VERIFY`,
|
||||
`THERMOGRAPH_BING_VERIFY`, `PORT` (default 8080).
|
||||
|
||||
### Frontend against a real backend container
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
make backend-up # pulls + runs the published backend image + throwaway db
|
||||
# on 127.0.0.1:18137, waits for /healthz, prints the URL
|
||||
make backend-down
|
||||
```
|
||||
|
||||
`make test-integration` runs the Python integration tier against that. CI does
|
||||
**not** run it — it needs a live backend container.
|
||||
|
||||
## The daemon
|
||||
|
||||
```bash
|
||||
cd backend/daemon
|
||||
go build ./... && go vet ./... && go test ./...
|
||||
|
||||
THERMOGRAPH_INTERNAL_TOKEN=dev-token \
|
||||
THERMOGRAPH_API_BASE_INTERNAL=http://localhost:8137 \
|
||||
go run .
|
||||
```
|
||||
|
||||
It refuses to start without `THERMOGRAPH_INTERNAL_TOKEN` — and the backend
|
||||
answers `404` on the whole `/internal/*` surface when that token is unset. Both
|
||||
ends fail closed. With Discord unconfigured it logs once and runs cron-only.
|
||||
|
||||
## The full stack, locally
|
||||
|
||||
```bash
|
||||
cd infra
|
||||
make dev-up # docker-compose.yml + docker-compose.dev.yml overlay:
|
||||
# uncapped CPU, backend published on 0.0.0.0:8137 for the LAN
|
||||
make dev-down
|
||||
```
|
||||
|
||||
The dev overlay exports `COMPOSE_PROJECT_NAME=thermograph-dev` so the LAN stack
|
||||
keeps volumes separate from anything else. **Do not remove either half of the
|
||||
project-name pinning** — `infra/docker-compose.yml` pins `name: thermograph`,
|
||||
and without it running compose from `infra/` derives the project name `infra`,
|
||||
silently creating a *new* stack beside the running one with fresh volumes.
|
||||
|
||||
Other `infra/Makefile` targets: `up`/`down` (pull + run the published images),
|
||||
`db-up`/`db-down` (just Postgres, e.g. to run the app from a venv against it),
|
||||
`om-up`/`om-down`/`om-backfill` (the self-hosted Open-Meteo overlay — the
|
||||
backfill writes ~1–1.5 TB and takes hours).
|
||||
|
||||
## Connectors (Centralis and friends)
|
||||
|
||||
The fleet is not reachable from a laptop off the WireGuard mesh, and Forgejo is
|
||||
mesh-only. **Centralis** is the control plane that fronts all of it — the app
|
||||
database, the ERA5 lake, fleet logs, Grafana, Forgejo, docs and notes, and
|
||||
Discord.
|
||||
|
||||
```bash
|
||||
claude mcp add --transport http centralis https://mcp.thermograph.org/mcp \
|
||||
--header "Authorization: Bearer $CENTRALIS_TOKEN"
|
||||
```
|
||||
|
||||
Ask the operator for a token; don't share it. Verify with *"what's running on
|
||||
prod right now?"* — it should call `fleet_status` and list the Swarm services.
|
||||
|
||||
Also worth installing locally: **Chrome DevTools MCP** (design verification
|
||||
needs a real browser on your machine) and **Figma**. Grafana's official MCP
|
||||
server is optional and read-only — but remember dashboards are provisioned from
|
||||
repo JSON, so a durable change is still a PR via `dashboard_write`.
|
||||
|
||||
Run `mcp__centralis__onboarding` for the current, authoritative connector list;
|
||||
it will be fresher than this page.
|
||||
|
||||
## Repo-local guardrails
|
||||
|
||||
`.claude/settings.json` wires three hooks that travel with the checkout:
|
||||
|
||||
| Hook | When | What |
|
||||
|---|---|---|
|
||||
| `prod-guard.sh` | before Bash / live-host MCP calls | Classifies by **allowlist**: only positively-recognised read-only commands pass; everything else asks. Beta is guarded as strictly as prod — it hosts Forgejo, Grafana *and* beta.thermograph.org. |
|
||||
| `secrets-guard.sh` | before Write/Edit | Denies any direct write to `infra/deploy/secrets/*.yaml`. Use `sops edit`. |
|
||||
| `lint-after-edit.sh` | after Write/Edit | shellchecks an edited `*.sh` and feeds findings straight back. Exits quietly if shellcheck is missing. |
|
||||
|
||||
They enforce what `CLAUDE.md` can only ask for. If you change `prod-guard.sh`'s
|
||||
classifier, re-read `.claude/hooks/README.md` first — it documents a real
|
||||
silent-total-bypass failure mode in the parsing loop.
|
||||
|
||||
Next: [Repo map](03-repo-map.md).
|
||||
189
docs/onboarding/03-repo-map.md
Normal file
189
docs/onboarding/03-repo-map.md
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
# 3. Repo map
|
||||
|
||||
The monorepo was reunified on **2026-07-22** by subtree-merging four split
|
||||
repos with full history (380 commits). The split's *properties* were kept
|
||||
deliberately: per-domain images, per-domain deploys, and an async FE/BE
|
||||
contract. That's why the layout looks like four projects sharing a root.
|
||||
|
||||
```
|
||||
thermograph/
|
||||
├── CLAUDE.md root agent instructions — branch model, deploy contract
|
||||
├── CUTOVER-NOTES.md what is and isn't live; the reunification record
|
||||
├── README.md domain table + how CI stays decoupled
|
||||
├── .forgejo/workflows/ ALL CI. Nine files. Never add workflows elsewhere.
|
||||
├── .claude/ settings.json + enforcement hooks
|
||||
├── backend/ FastAPI API + Go daemon → emi/thermograph/backend
|
||||
├── frontend/ Go SSR + static assets → emi/thermograph/frontend
|
||||
├── infra/ compose, Swarm, deploy, secrets, terraform, ops
|
||||
├── observability/ Loki + Grafana + Alloy
|
||||
└── docs/onboarding/ you are here
|
||||
```
|
||||
|
||||
> **CI lives only in root `.forgejo/workflows/`**, path-filtered per domain.
|
||||
> Forgejo only reads root workflows — a copy under a domain's own `.forgejo/`
|
||||
> is inert, and becomes a trap when someone edits it expecting an effect.
|
||||
|
||||
## `backend/` — the API service
|
||||
|
||||
Python 3.12 / FastAPI. Owns climate data, grading, the API, accounts,
|
||||
notifications, and the SSR-content JSON API the frontend renders from. Serves
|
||||
**no** HTML/CSS/JS.
|
||||
|
||||
```
|
||||
app.py one-line re-export shim for web/app.py — keeps `app:app`
|
||||
stable for systemd/CI/entrypoint. Don't change the name.
|
||||
paths.py every filesystem location, resolved from the repo root.
|
||||
NEVER reintroduce __file__-relative paths in a module.
|
||||
web/app.py (1022) the real FastAPI app: routes, CORS, ETag, middleware,
|
||||
lifespan, role gating, the frontend proxy fallback
|
||||
api/
|
||||
payloads.py (342) pure inputs → response dict builders. One definition per
|
||||
payload shape; PAYLOAD_VER lives here.
|
||||
content_payloads.py the SSR content API's builders (JSON-safe, no Markup)
|
||||
content_routes.py route wiring for /content/* + the same ETag pattern
|
||||
internal_routes.py /internal/* — the Go daemon's control surface. Fails
|
||||
closed: no token configured ⇒ the whole surface 404s.
|
||||
homepage.py (367) the precomputed "unusual right now" feed
|
||||
sitemap.py
|
||||
data/
|
||||
climate.py (1248) fetch + cache the daily record. The source ladder lives
|
||||
here. The single biggest file in the repo.
|
||||
grading.py (532) day-of-year climatology, percentiles, TEMP/RAIN_BANDS
|
||||
grid.py (92) lat/lon → stable ~4 sq mi cell id
|
||||
store.py (329) derived-payload store (SQLite/Postgres). Pure accelerator.
|
||||
climate_store.py(353) raw climate record on TimescaleDB hypertables
|
||||
era5lake.py (173) the ERA5 object-storage lake: layout, grid math, clients
|
||||
scoring.py (293) climate-shift scoring (recent years vs full baseline)
|
||||
places.py (355) typo-tolerant local place index for /suggest
|
||||
cities.py, city_events.py, meteostat.py
|
||||
core/
|
||||
metrics.py (805) since-start traffic counters, multi-worker via SQLite
|
||||
audit.py (167) JSONL audit / errors / access / activity / heartbeat
|
||||
singleton.py (129) leader election: flock (per host) or PG advisory lock
|
||||
(per cluster). Read this before adding any timer.
|
||||
accounts/ fastapi-users: models, schemas, db, users, api_accounts
|
||||
notifications/
|
||||
notify.py (542) the subscription evaluation engine (background thread)
|
||||
push.py, mailer.py, digest.py, discord.py, discord_link.py,
|
||||
discord_interactions.py
|
||||
daemon/ the Go daemon — Discord gateway + cron timers
|
||||
alembic/ Postgres migrations (run by the entrypoint on boot)
|
||||
deploy/entrypoint.sh container entrypoint: alembic migrate, then serve
|
||||
lake_app.py (204) the prod-only `lake` service (DuckDB over the bucket)
|
||||
tests/ the hermetic pytest suite
|
||||
cities.json, cities_flavor.json bundled reference data (generated)
|
||||
```
|
||||
|
||||
Standalone scripts at the root are tooling, not runtime: `warm_cities.py`,
|
||||
`indexnow.py`, `seed_era5.py`, `gen_era5_lake.py`, `gen_cities.py`,
|
||||
`gen_flavor.py`, `drift_check.py`, `migrate*.py`.
|
||||
|
||||
Deep dive: [04-backend.md](04-backend.md).
|
||||
|
||||
## `frontend/` — SSR + static assets
|
||||
|
||||
```
|
||||
server/ ★ THE LIVE SERVICE (Go 1.26, module thermograph/frontend)
|
||||
main.go config + mux + static + graceful shutdown
|
||||
internal/config/ every env var, with the same names/defaults as before
|
||||
internal/contentapi/ backend /content/* client: TTL cache, bounded LRU,
|
||||
per-key single-flight, origin forwarding
|
||||
internal/content/ page handlers, funcmap, SEO
|
||||
internal/contentdata/ glossary.yaml + pages.yaml loader (fail-loud)
|
||||
internal/format/ unit-aware °C/°F formatting + band names
|
||||
internal/handlers/ routing, SPA shells, static
|
||||
internal/render/ html/template over embed.FS + ETag helpers
|
||||
internal/render/templates/*.tmpl the embedded page templates
|
||||
static/ every static asset — app.js, cache.js, shared.js,
|
||||
account.js, calendar/day/score/compare .js + .html
|
||||
shells, style.css (all design tokens), icons, sw.js
|
||||
content/ glossary.yaml, pages.yaml — structured SSR copy
|
||||
templates/*.html.j2 Jinja templates — used by the PYTHON service only
|
||||
tests/ Python tiers: unit (hermetic, fixtures) + integration
|
||||
tests/fixtures/*.json golden payloads — ALSO consumed by the Go tests
|
||||
tools/shoot.py the `make shots` screenshot sweep
|
||||
DESIGN.md the visual system: tokens, components, breakpoints
|
||||
app.py, content.py, ⚠ the superseded Python implementation — not deployed
|
||||
api_client.py, (see traps)
|
||||
format.py, paths.py,
|
||||
content_loader.py
|
||||
```
|
||||
|
||||
Deep dive: [05-frontend.md](05-frontend.md).
|
||||
|
||||
## `infra/` — how and where images run
|
||||
|
||||
```
|
||||
docker-compose.yml the compose stack (beta + LAN dev):
|
||||
db, backend, lake, daemon, frontend.
|
||||
`name: thermograph` is PINNED — see below.
|
||||
docker-compose.dev.yml LAN overlay: uncapped, backend on 0.0.0.0:8137
|
||||
docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only)
|
||||
Makefile compose orchestration only
|
||||
deploy/
|
||||
deploy.sh ★ the single entry point for beta and prod
|
||||
render-secrets.sh renders /etc/thermograph.env from the SOPS vault
|
||||
secrets/ the vault: common.yaml, prod/beta/dev.yaml,
|
||||
centralis.prod.yaml, example.yaml
|
||||
stack/ the SWARM path (live on prod):
|
||||
thermograph-stack.yml, deploy-stack.sh,
|
||||
autoscale.sh, the LB
|
||||
swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning
|
||||
Caddyfile the reverse proxy (path-splits FE vs BE)
|
||||
provision-*.sh host bootstrap: agent access, dev LAN, mail, secrets
|
||||
db/init/ TimescaleDB init + tuning
|
||||
migrations/ hand-run SQL migrations
|
||||
ops/
|
||||
dbq.sh read-only psql into any environment's db container
|
||||
iceberg.sh duckdb over the ERA5 Iceberg lake
|
||||
terraform/ host provisioning — NO STATE IS PERSISTED
|
||||
ACCESS.md, DEPLOY.md, DEPLOY-DEV.md (partly stale — see traps)
|
||||
```
|
||||
|
||||
**Why the project name is pinned.** Compose creates volumes
|
||||
`thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stack declares
|
||||
those exact names as `external: true` at the same mount paths. Running compose
|
||||
from `infra/` without `name: thermograph` derives project `infra`, which makes a
|
||||
whole new stack with empty volumes next to the running one. `deploy-dev.sh`
|
||||
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
|
||||
keep LAN dev separate on purpose. Keep both halves.
|
||||
|
||||
Deep dive: [08-infra-secrets.md](08-infra-secrets.md).
|
||||
|
||||
## `observability/` — the logging stack
|
||||
|
||||
```
|
||||
docker-compose.yml Loki + Grafana (runs on beta)
|
||||
loki/config.yml mesh-only, filesystem storage, 30-day retention
|
||||
grafana/provisioning/ datasource + dashboard provider (auto-loaded)
|
||||
grafana/provisioning/alerting/ rules, the Discord contact point, the policy
|
||||
grafana/dashboards/*.json the fleet-logs dashboard
|
||||
alloy/config.alloy the per-node shipper
|
||||
alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host)
|
||||
caddy-grafana.conf beta's Grafana vhost, kept for reference
|
||||
```
|
||||
|
||||
No build, no deploy automation — it ships by hand. **The repo is the only
|
||||
durable path**: Grafana is provisioned from this directory at startup, so UI
|
||||
edits are overwritten. Grafana is at **`dashboard.thermograph.org`** (Google
|
||||
SSO, pre-provisioned users only) — never `grafana.thermograph.org`.
|
||||
|
||||
Deep dive: [09-observability.md](09-observability.md).
|
||||
|
||||
## The nine CI workflows
|
||||
|
||||
| Workflow | Trigger | Does |
|
||||
|---|---|---|
|
||||
| `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. |
|
||||
| `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. |
|
||||
| `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. |
|
||||
| `deploy.yml` | push to `main`/`release` touching an app domain | Branch selects environment; matrix covers services; SSHes and runs `deploy.sh`. |
|
||||
| `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards each host's checkout and re-renders secrets. Rolls **no** service. |
|
||||
| `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. |
|
||||
| `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow. Uses `PROD_SSH_*`. |
|
||||
| `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. |
|
||||
| `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. |
|
||||
|
||||
Deep dive: [07-ci-and-release.md](07-ci-and-release.md).
|
||||
|
||||
Next: [Backend deep dive](04-backend.md).
|
||||
292
docs/onboarding/04-backend.md
Normal file
292
docs/onboarding/04-backend.md
Normal file
|
|
@ -0,0 +1,292 @@
|
|||
# 4. Backend deep dive
|
||||
|
||||
FastAPI on Python 3.12, plus a Go daemon that ships in the same image. Owns
|
||||
every piece of climate data and every bit of grading logic in the project.
|
||||
|
||||
Read [`backend/CLAUDE.md`](../../backend/CLAUDE.md) alongside this — it is the
|
||||
binding, terse version.
|
||||
|
||||
## The one-process-many-roles model
|
||||
|
||||
The backend image runs **five different jobs** depending on `THERMOGRAPH_ROLE`
|
||||
and which binary compose/Swarm selects. One image means version skew between
|
||||
them is impossible.
|
||||
|
||||
| Service | How it's selected | Does |
|
||||
|---|---|---|
|
||||
| **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. |
|
||||
| **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. |
|
||||
| **all** | default | Both — the single-process dev default. |
|
||||
| **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Prod only. |
|
||||
| **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. |
|
||||
|
||||
`deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres
|
||||
volume can still be starting) and then serves. `/healthz` and `/api/version`
|
||||
are deliberately **I/O-free** so they stay cheap under tight healthcheck
|
||||
intervals; they're also deliberately **not** under `THERMOGRAPH_BASE`, so a
|
||||
healthcheck needn't know the base path.
|
||||
|
||||
## The data pipeline
|
||||
|
||||
### Where history comes from — the source ladder
|
||||
|
||||
`data/climate.py::_load_history` walks this ladder on a cache miss, and the
|
||||
order matters:
|
||||
|
||||
1. **The ERA5 lake** (`data/era5lake.py`) — our own parquet extract of the
|
||||
public ERA5 archive in Contabo object storage. True ERA5 with measured
|
||||
gusts, no third-party API in the path. Unconfigured or missing-point cells
|
||||
fall through at zero cost.
|
||||
2. **The Open-Meteo archive** — the same ERA5 family, so a backup-sourced
|
||||
record grades consistently with lake-sourced neighbours. Guarded by its own
|
||||
rate-limit cooldown.
|
||||
3. **Stale cache**, served without rewriting it (so `recent_stamp` stays old
|
||||
and the next request retries upstream first).
|
||||
4. Otherwise `WeatherUnavailable` → a clean 503.
|
||||
|
||||
**NASA POWER is retired from every serving path.** Its MERRA-2 record ran
|
||||
~1.3 °F off ERA5 and carried no gusts. `drift_check.py` still fetches it, and
|
||||
only that.
|
||||
|
||||
Every source must return a plausibly-full span (`MIN_ARCHIVE_DAYS`) before it's
|
||||
accepted and cached as complete — a short response is rejected, not stored.
|
||||
|
||||
### Where "today" comes from
|
||||
|
||||
`_load_recent_forecast` is a separate ladder:
|
||||
|
||||
1. **Open-Meteo forecast API** — recent past *and* forward days in one call,
|
||||
with gusts, ERA5-consistent with the history record. Skipped while its own
|
||||
cooldown is active.
|
||||
2. **Fallback** — an archive recent-past range plus **MET Norway** forward days.
|
||||
Costs no Open-Meteo forecast quota.
|
||||
3. Stale cache, again without rewriting.
|
||||
|
||||
Cached per cell for `FORECAST_TTL_HOURS`, so it tracks model updates without
|
||||
hourly upstream I/O. `RECENT_PAST_DAYS = 25`, `FORECAST_DAYS = 8`.
|
||||
|
||||
### Concurrency and quota guards
|
||||
|
||||
- A per-cell lock serialises archive fetches, so N concurrent first-requests for
|
||||
one cell cost one fetch.
|
||||
- Rate limits set module-level cooldowns (`_archive_cooldown_until`,
|
||||
`_forecast_cooldown_until`) rather than retrying into the wall.
|
||||
- Reverse geocoding runs on **one dedicated worker thread** draining a queue at
|
||||
~1 req/s. The earlier design slept inside the caller's thread — which, in the
|
||||
server, is one of Starlette's shared sync-threadpool threads.
|
||||
|
||||
## Storage: three tiers with three different contracts
|
||||
|
||||
| Tier | Module | Backend | Contract |
|
||||
|---|---|---|---|
|
||||
| **Raw climate record** | `data/climate_store.py` | TimescaleDB `climate_history` hypertable (durable), `climate_recent` (rewritten each refresh), `climate_sync` (freshness) | **Source of truth.** Expensive to refetch. Fail-soft: a miss degrades to *fetch-from-upstream*. Parquet files instead when `THERMOGRAPH_DATABASE_URL` isn't Postgres (dev, tests, offline tooling). |
|
||||
| **Derived payloads** | `data/store.py` | SQLite WAL, or **UNLOGGED** Postgres tables | **Pure accelerator.** Every reader falls back to recomputing from the raw record; every helper swallows its own errors. Deleting the store is a safe full reset. |
|
||||
| **Accounts** | `accounts/db.py` | Postgres (or SQLite) via SQLAlchemy | **Not regenerable.** Foreign keys enforced, errors surface rather than get swallowed, must be backed up. |
|
||||
|
||||
That third row is the important distinction: accounts, subscriptions and
|
||||
notifications have no source to recompute from, so they live under stricter
|
||||
rules than everything else in the process.
|
||||
|
||||
On Postgres, `accounts/db.py` runs **two async engines** over one primary —
|
||||
read-only (`default_transaction_read_only`) for pure-GET endpoints, read-write
|
||||
for everything else — plus a separate **sync** psycopg engine for the notifier
|
||||
thread, which has no event loop.
|
||||
|
||||
### The validity token
|
||||
|
||||
Derived rows are keyed `(kind, cell_id, request_key)` and guarded by a `token`
|
||||
that encodes everything the payload depends on:
|
||||
|
||||
```python
|
||||
PAYLOAD_VER = "p2" # api/payloads.py
|
||||
history_token(history) → f"{PAYLOAD_VER}:{hist_end(history)}"
|
||||
content_token(...) → f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date}"
|
||||
```
|
||||
|
||||
A stored token that doesn't match the caller's current token is a **miss**.
|
||||
There is no clock-based expiry anywhere. That's why one `PAYLOAD_VER` bump
|
||||
atomically orphans every pre-upgrade row — and why forgetting to bump it serves
|
||||
old-shaped JSON forever. `CONTENT_VER` is kept separate so a content-only shape
|
||||
change doesn't invalidate the whole grading cache.
|
||||
|
||||
The same token doubles as a **weak ETag**. See [contracts](06-contracts.md) for
|
||||
what that means for CORS.
|
||||
|
||||
## Grading
|
||||
|
||||
`data/grading.py` is the heart of the product:
|
||||
|
||||
- For a target day of year, the reference distribution is every historical day
|
||||
within **±`HALF_WINDOW` (7) days**, wrapping the year boundary — a 15-day
|
||||
seasonal window.
|
||||
- The observed value is placed as an **empirical mid-rank percentile**, which
|
||||
handles ties correctly (crucial: most days have zero precipitation).
|
||||
- `TEMP_BANDS` is a symmetric seven-tier ladder; `RAIN_BANDS` is separate, and
|
||||
dry days are coloured by dry-streak length instead of rank.
|
||||
- `pct_ordinal()` floors a percentile into `1..99` — never 0, never 100.
|
||||
`frontend/static/shared.js::pctOrd()` mirrors it. If they diverge, the same
|
||||
reading says two different things on two surfaces.
|
||||
|
||||
`data/scoring.py` is a different question: how far a cell's **recent years**
|
||||
have drifted from its full multi-decade baseline, per metric and per
|
||||
percentile category, expressed in unit-free percentile points. Note
|
||||
`baseline_overlaps_recent` in the payload — the baseline deliberately includes
|
||||
the recent years, which mildly and uniformly attenuates the divergence.
|
||||
|
||||
## The HTTP surface
|
||||
|
||||
`web/app.py` mounts routers so that every version stays served simultaneously:
|
||||
|
||||
```
|
||||
/api → v1 (legacy unversioned alias)
|
||||
/api/v1 → v1 geocode, grade
|
||||
/api/v2 → v2 geocode, suggest, place, grade, calendar, day,
|
||||
score, forecast, cell, metrics, event
|
||||
/api/v2 → content_routes: /content/hub, /content/sitemap,
|
||||
/content/indexnow-key, /content/home,
|
||||
/content/city/{slug}[/month/{month}|/records]
|
||||
/api/v2/auth, /users fastapi-users: login, logout, register, verify, me
|
||||
/api/v2/subscriptions, /notifications, /push/* accounts/api_accounts.py
|
||||
/api/v2/discord/* account linking (OAuth2)
|
||||
/internal/* the Go daemon's control surface — NOT under BASE
|
||||
/healthz, /api/version I/O-free
|
||||
/discord/interactions Discord slash-command webhook (Ed25519-verified)
|
||||
/digest the footer signup form
|
||||
/{path:path} catch-all proxy to the frontend ← registered LAST
|
||||
```
|
||||
|
||||
Two routes worth understanding properly:
|
||||
|
||||
**`GET /api/v2/cell`** — one bundle carrying every view's payload for a cell, so
|
||||
the client warms four views with one request. Each slice is the *exact* payload
|
||||
its per-view endpoint returns, built by the same builders under the same
|
||||
derived-store keys and tokens, paired with the ETag that endpoint would emit.
|
||||
The client seeds its per-view cache from the slices and later revalidates each
|
||||
view individually. `prefetch=1` is the warm-only mode with a hard guarantee:
|
||||
**it never spends weather-API quota** — a cold cell answers `204` with no body.
|
||||
`neighbors=1` additionally enqueues the 8 surrounding cells for background
|
||||
warming, also warm-only.
|
||||
|
||||
**The catch-all proxy** — the frontend owns every page and asset. In prod and
|
||||
beta, Caddy path-splits directly, so this proxy is never exercised. It exists as
|
||||
the fallback for environments with no Caddy in front (LAN dev, bare-metal). It
|
||||
forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute
|
||||
URLs instead of resolving to the internal hop's own address. Because it's
|
||||
registered dead last, every real backend route wins first —
|
||||
`/internal/*` in particular is registered before it, deliberately.
|
||||
|
||||
## Background work: who runs what, and where
|
||||
|
||||
This is where the project's hardest-won lessons live.
|
||||
|
||||
- **Neighbour warmer** — a per-worker daemon thread draining a request-fed
|
||||
queue. Correct to run in every worker: each drains its own queue.
|
||||
- **Subscription notifier** (`notifications/notify.py`) — a timer-driven sweep
|
||||
that issues upstream fetches independent of any request. Running it in N
|
||||
workers multiplies Open-Meteo quota use N-fold. That is not hypothetical: it
|
||||
is the documented cause of overnight 429/503s after prod went to 3 workers.
|
||||
So it runs behind `singleton.claim_leader()`:
|
||||
- `claim()` — a non-blocking `flock` on a lockfile, arbitrating workers **on
|
||||
one machine**;
|
||||
- `claim_pg()` — a Postgres advisory lock, visible to every host talking to
|
||||
the same database, arbitrating **cluster-wide** under Swarm;
|
||||
- unset (dev, tests, single worker) ⇒ always the leader.
|
||||
- **Heartbeat** — deliberately **unguarded** across every worker and replica. A
|
||||
beat is one local file append (no quota, no DB, no lock), so duplicates are
|
||||
free, and only the process actually being dead stops them. Container-log
|
||||
silence can't distinguish a live web/worker process from a dead one, so this
|
||||
is the signal observability alerts on.
|
||||
- **Discord gateway + recurring jobs** — moved *out* of this process entirely
|
||||
into the Go daemon. See below.
|
||||
|
||||
The notifier's own quiet guards: a `UNIQUE(subscription_id, event_date, metric,
|
||||
direction, kind)` row-level dedup, and a **weekly cap** — a subscription that
|
||||
notified within 7 days is skipped entirely. Archive reads come from cache; a
|
||||
freshly-subscribed cell is fetched **once**, budget-capped per pass.
|
||||
|
||||
## The Go daemon (`backend/daemon/`)
|
||||
|
||||
One binary, built into the backend image, running as exactly one replica.
|
||||
|
||||
**Why it exists:** the Discord gateway tolerates one session per token, and
|
||||
duplicated timers repeat every job. One replica makes that a *deployment fact*
|
||||
instead of a runtime election.
|
||||
|
||||
**Why it's thin:** Go owns no climate or grading logic. Grading needs polars and
|
||||
the cache, and the slash-command path deliberately shares one grade builder with
|
||||
the API so bot grades and API grades can never drift. So anything data-shaped is
|
||||
a POST back into Python:
|
||||
|
||||
| Route | Purpose |
|
||||
|---|---|
|
||||
| `POST /internal/discord/grade` | `{"query": "phoenix"}` → gateway-ready Discord message JSON, relayed verbatim |
|
||||
| `POST /internal/jobs/warm-cities` | trigger the warm-cities job |
|
||||
| `POST /internal/jobs/indexnow` | trigger IndexNow check-and-submit |
|
||||
|
||||
Every request carries `X-Thermograph-Internal-Token`. **Both ends fail closed**:
|
||||
with no `THERMOGRAPH_INTERNAL_TOKEN`, Python answers 404 across the whole
|
||||
surface and the daemon refuses to start. Caddy never routes `/internal/*`
|
||||
publicly, so the token is defence in depth, not the only control.
|
||||
|
||||
The cron half runs unconditionally (`warm-cities` every 24h, `indexnow` every
|
||||
6h, both configurable). A fatal gateway error logs loudly and leaves cron
|
||||
running — a Discord misconfiguration must not become a crash loop that also
|
||||
stops the schedule. The cron scheduler defers the first run by one interval,
|
||||
never overlaps a job with itself, and drops ticks that fire mid-run.
|
||||
|
||||
## The lake (`lake_app.py` + `data/era5lake.py`)
|
||||
|
||||
Prod-only Swarm service, same image, `THERMOGRAPH_ROLE=lake`.
|
||||
|
||||
The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`),
|
||||
extracted once from the public Earthmover ERA5 Icechunk archive by
|
||||
`gen_era5_lake.py`. Two layouts under `era5/`:
|
||||
|
||||
- `daily/tile=<ti>_<tj>/year=<Y>/month=<M>/part.parquet` — the analytical hive
|
||||
table. The location grain is a 12×12-point tile (~3°×3°) aligned to the
|
||||
source's chunk grid; a per-point-per-month grain would be ~170M objects.
|
||||
- `points/lat=<i>/lon=<j>.parquet` — the serving projection: one file per grid
|
||||
point with its full history, so the hot path is a single GET.
|
||||
- `manifest.parquet` — an index of every point file; readers prune with it and
|
||||
the extractor resumes from it.
|
||||
|
||||
Endpoints: `/healthz`, `GET /history?lat=&lon=` (the hot path — nearest point's
|
||||
history as parquet bytes, disk-cached per point), `POST /query` (SELECT-only
|
||||
DuckDB SQL over `era5_daily` and `manifest`).
|
||||
|
||||
Two operational facts: **Contabo is path-style only** with `region=default`
|
||||
(virtual-host addressing fails outright), and DuckDB's `httpfs`/`iceberg`
|
||||
extensions are baked into the image **as uid 10001**, because extensions install
|
||||
to the invoking user's `~/.duckdb` and a root-time bake left the runtime user
|
||||
unable to `LOAD` them — seen live as prod's lake `/query` returning 500.
|
||||
|
||||
Filter on partition columns (`tile`/`year`/`month`). That predicate is the
|
||||
difference between a pruned scan and reading the whole warehouse.
|
||||
|
||||
## Observability hooks inside the app
|
||||
|
||||
- `core/audit.py` writes JSONL streams: `audit` (one line per graded run, with
|
||||
per-phase timings and a `full`/`partial` run type), `errors` (upstream
|
||||
failures tagged `retry` or `error`), plus access, activity and heartbeat.
|
||||
Alloy parses these so `level`/`tag`/`phase` become Loki labels.
|
||||
- `core/metrics.py` keeps since-start counters, shared across uvicorn workers
|
||||
via `THERMOGRAPH_METRICS_DB` (a SQLite file) — unset keeps the
|
||||
zero-dependency in-memory store. Read over the token-gated
|
||||
`GET /api/v2/metrics`.
|
||||
- Every unhandled 500 gets an explicit exception handler emitting
|
||||
`tag="http.error"`. Without it the exception propagates past the middleware
|
||||
and the 500 is invisible in both the access log and metrics.
|
||||
|
||||
## Working on it
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
make test # hermetic pytest, ~12s
|
||||
make test ARGS='tests/data/test_grading.py -q'
|
||||
make smoke # build the image, boot it + a throwaway db
|
||||
```
|
||||
|
||||
CI runs the suite **inside the built image**, so the exact interpreter and
|
||||
dependencies that ship are what gets tested.
|
||||
|
||||
Next: [Frontend deep dive](05-frontend.md).
|
||||
221
docs/onboarding/05-frontend.md
Normal file
221
docs/onboarding/05-frontend.md
Normal file
|
|
@ -0,0 +1,221 @@
|
|||
# 5. Frontend deep dive
|
||||
|
||||
The **SSR + static-asset service**: server-rendered crawlable pages, the
|
||||
interactive tool's SPA shells, and every static asset. No climate data, no
|
||||
database, no compute — everything comes from the backend's content API over
|
||||
HTTP.
|
||||
|
||||
> **The live service is Go.** `frontend/server/` builds, tests, ships and runs.
|
||||
> The Python files one directory up (`app.py`, `content.py`, `api_client.py`,
|
||||
> `format.py`, `content_loader.py`, `paths.py`, `templates/*.html.j2`) are the
|
||||
> superseded original implementation, kept in-tree as the reference the port
|
||||
> was made from. `frontend/CLAUDE.md` and `frontend/README.md` still describe
|
||||
> the Python service — see [traps](11-traps.md). Everything in those docs about
|
||||
> **contracts, env vars and design** is still correct; only "which language and
|
||||
> which file" changed.
|
||||
|
||||
## Shape of the Go service
|
||||
|
||||
```
|
||||
main.go config → content client → glossary/pages → mux →
|
||||
listen → graceful shutdown
|
||||
internal/config/ every env var the service reads, with the same names,
|
||||
defaults and required/optional split as the Python
|
||||
internal/contentapi/ the backend /content/* client
|
||||
internal/content/ the SSR page handlers, template funcmap, SEO helpers
|
||||
internal/contentdata/ glossary.yaml / pages.yaml loader
|
||||
internal/format/ unit-aware °C/°F formatting + band names
|
||||
internal/handlers/ SPA shells + static serving
|
||||
internal/render/ html/template over an embed.FS + ETag helpers
|
||||
internal/render/templates/*.tmpl the page templates (embedded in the binary)
|
||||
```
|
||||
|
||||
Dependencies: **stdlib plus `gopkg.in/yaml.v3`** only. The committed SSR copy
|
||||
in `frontend/content/*.yaml` is shared with the rest of the repo and uses
|
||||
block/folded scalars, so a real YAML parser is genuinely required.
|
||||
|
||||
**Templates are embedded in the binary; `static/` and `content/` are read from
|
||||
disk relative to the working directory** (`frontend/` locally, `/app` in the
|
||||
image). That's why you build in `server/` and run from `frontend/`.
|
||||
|
||||
### Fail-loud boot
|
||||
|
||||
`main.go` exits at startup on bad configuration *or* bad content: a missing
|
||||
backend URL, a garbage TTL, malformed glossary/pages YAML. The Python raised at
|
||||
import for the same cases, and the reasoning is identical — a bad content edit
|
||||
should break the boot, not silently 500 on the first request.
|
||||
|
||||
The one deliberate exception is the IndexNow key: boot tries the key fetch once,
|
||||
catches failure, and falls back to a lazy per-request lookup. **Boot must
|
||||
survive an unreachable backend** — asynchronous FE/BE deploys depend on it.
|
||||
Don't make it fatal.
|
||||
|
||||
## Routes
|
||||
|
||||
Everything below sits under `THERMOGRAPH_BASE` (default `/thermograph`; the
|
||||
image sets `/`).
|
||||
|
||||
| Path | Kind |
|
||||
|---|---|
|
||||
| `/` | SSR homepage |
|
||||
| `/climate` | SSR climate hub |
|
||||
| `/climate/{slug}` | SSR per-city page |
|
||||
| `/climate/{slug}/records` | SSR all-time records |
|
||||
| `/climate/{slug}/{month}` | SSR per-month page |
|
||||
| `/glossary`, `/glossary/{term}` | SSR |
|
||||
| `/about`, `/privacy` | SSR |
|
||||
| `/robots.txt`, `/sitemap.xml` | generated |
|
||||
| `/{indexnow-key}.txt` | the IndexNow ownership file |
|
||||
| `/calendar`, `/day`, `/score`, `/compare`, `/legend`, `/alerts` | SPA shells |
|
||||
| `/healthz` | liveness, I/O-free (does **not** prove the backend is reachable) |
|
||||
| everything else | static assets |
|
||||
|
||||
The static mount is registered last so the explicit routes win.
|
||||
|
||||
**The backend owns the page metadata, not this service.** `page_title`,
|
||||
`canonical_path`, `breadcrumb` and `jsonld` come from
|
||||
`backend/api/content_payloads.py`. Don't recompute them here.
|
||||
|
||||
### SPA shells and `__ORIGIN__`
|
||||
|
||||
The shell HTML files in `static/` carry an `__ORIGIN__` placeholder for the
|
||||
link-preview/Open Graph tags — preview crawlers need absolute URLs and the host
|
||||
differs between LAN and prod. The substituted HTML and its ETag are **memoized
|
||||
per origin**, not recomputed per request.
|
||||
|
||||
Origin resolution prefers `X-Forwarded-Host` over `Host`. That matters when the
|
||||
request arrives through the backend's catch-all proxy fallback: without it, the
|
||||
page would advertise the internal hop's own address as canonical.
|
||||
|
||||
## The content client
|
||||
|
||||
`internal/contentapi` is a careful piece of code and worth reading in full
|
||||
before you change caching behaviour. It has four properties, all load-bearing:
|
||||
|
||||
1. **TTL cache** (`THERMOGRAPH_SSR_CACHE_TTL`, default 600s). Climate data
|
||||
changes at most hourly, so without this a page-view burst on one city costs
|
||||
one backend round trip per request. This is *in addition to* the backend's
|
||||
derived-store/ETag caching — this one saves the network hop, the backend's
|
||||
saves the recompute.
|
||||
2. **Bounded LRU** (2048 entries). `city()` and `city_records()` fold the
|
||||
browser-facing `origin` into the cache key, and `origin` is client-controlled
|
||||
via `Host`/`X-Forwarded-Host`. An unbounded map there is a cheap
|
||||
memory-exhaustion vector.
|
||||
3. **Per-key single-flight.** Without it, N concurrent requests missing the same
|
||||
key (cold start, or right after TTL expiry on a hot city) each fire their own
|
||||
backend call — a thundering herd.
|
||||
4. **Origin forwarding.** The original browser-facing origin is passed through as
|
||||
`Host` / `X-Forwarded-Proto` so the backend builds JSON-LD `url` fields
|
||||
against the public origin rather than the internal `http://backend:8137`
|
||||
address.
|
||||
|
||||
## Static assets
|
||||
|
||||
Hand-written, no build step, no bundler, no framework.
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `app.js` | the main tool: map, search, results, inline SVG chart |
|
||||
| `calendar.js` / `day.js` / `score.js` / `compare.js` | the SPA views |
|
||||
| `cache.js` | IndexedDB cache + the `/cell` bundle prefetch + `If-None-Match` revalidation |
|
||||
| `account.js` | auth; **pins `API_VERSION` and the `uv(path)` helper** |
|
||||
| `shared.js` | shared helpers; **`pctOrd()` mirrors the backend's `pct_ordinal()`** |
|
||||
| `units.js` | `F_REGIONS` — must match the backend's `F_COUNTRIES` |
|
||||
| `chart.js`, `mappicker.js`, `filtersheet.js`, `nav.js`, `push-client.js`, `ios-install.js`, `digest.js`, `climate.js`, `subscriptions.js` | the rest |
|
||||
| `style.css` | **the single hand-written stylesheet — every design token lives here** |
|
||||
| `sw.js`, `manifest.webmanifest`, icons | PWA |
|
||||
|
||||
`cache.js` is where the ETag contract actually lands: it fetches
|
||||
`/api/v2/cell` once per view-set, slices it, and later revalidates each view
|
||||
with `If-None-Match`. The slice→view map must track the backend's payload
|
||||
shape. See [contracts](06-contracts.md).
|
||||
|
||||
Static responses are served with `Cache-Control: public, max-age=300` — short
|
||||
on purpose, because asset filenames are **not** content-hashed. It's a
|
||||
revalidation window, not an immutable cache.
|
||||
|
||||
## The design system
|
||||
|
||||
[`frontend/DESIGN.md`](../../frontend/DESIGN.md) is the source of truth and
|
||||
wins over any general design guidance. The essentials:
|
||||
|
||||
- **Never hardcode a colour.** Every colour is a CSS custom property in
|
||||
`style.css`'s `:root`, with a `@media (prefers-color-scheme: light)` override
|
||||
right below. Always `var(--token)`.
|
||||
- **Grade palettes encode meaning, not decoration.** Temperature is a 9-step
|
||||
diverging, colourblind-safe scale (`--rec-cold` … `--normal` green midpoint …
|
||||
`--rec-hot`). Precipitation is `--dry` plus `--wet-1…9`. Keep the order and
|
||||
midpoints intact.
|
||||
- **Inter is not self-hosted.** It renders where the OS has it and falls back to
|
||||
`system-ui`. Adding a webfont link is a deliberate decision, not a tweak — it's
|
||||
a network dependency on every page.
|
||||
- **Inputs are `font-size: 16px` minimum.** Smaller text makes iOS zoom on
|
||||
focus. Hard rule.
|
||||
- **Mobile-first.** The base stylesheet is the phone layout; wider screens layer
|
||||
on via `min-width`. Primary breakpoint 640/641px. The content column widens in
|
||||
real steps at 1680 / 2400 / 3400px — it does *not* stay a centred 1200px strip.
|
||||
- **Metric order is always `Precip · High · Low`**, everywhere.
|
||||
- **Dark is the default**, light comes from `prefers-color-scheme`. Every change
|
||||
must look right in both. Honour `prefers-reduced-motion`.
|
||||
- Charts are bespoke inline SVG. No canvas, no charting library.
|
||||
|
||||
### Verifying a visual change
|
||||
|
||||
**Validate at 390 / 800 / 1920 / 2560 / 3840px in both light and dark.** These
|
||||
straddle every breakpoint above.
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
.venv/bin/pip install -q -r tools/requirements.txt
|
||||
.venv/bin/python -m playwright install chromium
|
||||
.venv/bin/python tools/shoot.py # the full matrix
|
||||
.venv/bin/python tools/shoot.py index --width 390 --scheme dark # while iterating
|
||||
```
|
||||
|
||||
PNGs land in `.screenshots/` (gitignored) as `{page}@{width}-{scheme}.png`.
|
||||
Point it elsewhere with `SHOTS_BASE`. Chrome DevTools MCP is the other good
|
||||
option here — a real browser on your own machine.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
cd frontend/server && go build ./... && go vet ./... && go test ./...
|
||||
```
|
||||
|
||||
The Go tests read two directories via a fixed relative path: the committed
|
||||
golden fixtures (`frontend/tests/fixtures/*.json` — the same set the Python
|
||||
golden-diff comparison used) and the SSR copy (`frontend/content/*.yaml`).
|
||||
`frontend/Dockerfile`'s builder stage copies them to container-root paths to
|
||||
preserve that relationship, and runs `gofmt -l` + `go vet` + `go test` **before**
|
||||
building. A failing Go test therefore fails the image build, which is how CI
|
||||
catches it — there is no separate frontend test step in the workflows.
|
||||
|
||||
The Python tiers still exist and still run locally:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
make test-unit # hermetic: SSR rendering fed committed fixtures
|
||||
make test-integration # pulls + runs the real backend image; local only
|
||||
make capture-fixtures # refresh tests/fixtures/*.json from a live backend
|
||||
```
|
||||
|
||||
Treat `make capture-fixtures` as the shared operation: those fixtures feed the
|
||||
**Go** tests too.
|
||||
|
||||
## The image
|
||||
|
||||
`python:3.12-slim` → gone. The frontend image is:
|
||||
|
||||
- a `golang:1.26` builder that runs `gofmt -l` / `go vet` / `go test`, then
|
||||
builds a static `CGO_ENABLED=0 -trimpath -ldflags="-s -w"` binary;
|
||||
- an `alpine:3.22` final stage with **bash** (the Swarm stack bind-mounts a
|
||||
bash entrypoint shim over the image's entrypoint) and **curl** (the
|
||||
healthcheck);
|
||||
- **uid 10001**, matching the backend image — that's the uid infra provisions
|
||||
readable secrets for. Don't change it.
|
||||
- `static/` and `content/` copied to `/app`, read-only at runtime.
|
||||
|
||||
There is no `WORKERS` knob any more: uvicorn needed a process count, the Go
|
||||
server handles concurrency in one process.
|
||||
|
||||
Next: [Cross-service contracts](06-contracts.md).
|
||||
183
docs/onboarding/06-contracts.md
Normal file
183
docs/onboarding/06-contracts.md
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
# 6. Cross-service contracts
|
||||
|
||||
**Read this before any change that touches both domains.**
|
||||
|
||||
Backend and frontend build separate images and deploy independently. That
|
||||
independence is the point of the architecture — and it means there is no build
|
||||
step, no type checker and no linker that will catch a mismatch between them. The
|
||||
items below are the seams. Each one fails *silently and in production*, which is
|
||||
why they're enumerated rather than left to judgement.
|
||||
|
||||
## 1. API version — `GET /api/version`
|
||||
|
||||
```json
|
||||
{"backend_version": "2", "min_frontend": "1", "payload_ver": "p2"}
|
||||
```
|
||||
|
||||
Driven by `API_CONTRACT_VERSION` and `MIN_SUPPORTED_FRONTEND` in
|
||||
`backend/web/app.py`.
|
||||
|
||||
**The rule for a breaking change:** ship a new `v3` router mounted **alongside**
|
||||
`v2`, re-registering only the handlers that changed, and bump the constant in
|
||||
the same PR. `/api` and `/api/v1` stay mounted as aliases. Every version stays
|
||||
served simultaneously — that is how FE and BE ship out of lockstep at all.
|
||||
|
||||
**The frontend pins the version in exactly two places:**
|
||||
|
||||
| Where | What |
|
||||
|---|---|
|
||||
| `frontend/server/internal/config` (Go) — `THERMOGRAPH_API_VERSION`, default `v2` | every server-side path builder reads it |
|
||||
| `frontend/static/account.js` (JS) — the exported `API_VERSION` and the `uv(path)` helper | every browser-side call |
|
||||
|
||||
Never hardcode `api/v2/...` anywhere else. Bump both in one PR, and **only
|
||||
after** the target backend's `/api/version` confirms it serves that version and
|
||||
its `min_frontend` doesn't exclude the one you're leaving.
|
||||
|
||||
> The Python `frontend/api_client.py` has a third pin. It isn't deployed, but if
|
||||
> you're using it locally, keep it in step or you'll debug a phantom.
|
||||
|
||||
## 2. `PAYLOAD_VER` — the cache invalidation token
|
||||
|
||||
`backend/api/payloads.py`, currently `"p2"`.
|
||||
|
||||
This is **separate from URL versioning** and does a different job: it's the
|
||||
cache/ETag invalidation token. Every derived row is stored against a validity
|
||||
token built from it, and a stored token that doesn't match the caller's current
|
||||
token is simply a miss.
|
||||
|
||||
**Bump it whenever a response payload's shape changes.** One bump atomically
|
||||
orphans every pre-upgrade cached row across every environment. Forget it and
|
||||
the backend serves old-shaped JSON out of cache indefinitely, with no error
|
||||
anywhere — the new code is live and the old data keeps flowing.
|
||||
|
||||
`CONTENT_VER` is deliberately separate so a content-only shape change doesn't
|
||||
invalidate the whole grading cache.
|
||||
|
||||
## 3. ETag / `If-None-Match` — and `expose_headers`
|
||||
|
||||
Every graded payload is cached against a validity token that doubles as a weak
|
||||
ETag. `frontend/static/cache.js` fetches `/api/v2/cell` once per view-set,
|
||||
slices it, and revalidates each view individually with `If-None-Match`.
|
||||
|
||||
**`expose_headers=["ETag"]` in the CORS middleware matters as much as
|
||||
`allow_origins`.** Browsers hide every response header except a fixed "simple"
|
||||
allowlist from cross-origin JS. Without `ETag` exposed, `cache.js`'s
|
||||
`res.headers.get("ETag")` silently returns `null`, 304 revalidation quietly
|
||||
stops working, and **there is no console error**. You'd see it only as a traffic
|
||||
bill.
|
||||
|
||||
CORS is off by default (`THERMOGRAPH_CORS_ORIGINS` unset), which is today's real
|
||||
same-origin topology. Never set it to `*` — Starlette itself refuses a
|
||||
credentialed wildcard.
|
||||
|
||||
Don't change the ETag derivation or that header list without checking
|
||||
`cache.js`. And when the payload shape changes, the **slice → view map** in
|
||||
`cache.js` has to track it.
|
||||
|
||||
## 4. `pct_ordinal()` — mirrored in four places
|
||||
|
||||
Floor a percentile into `1..99`. **Never 0, never 100.**
|
||||
|
||||
| Copy | File | Guarded by |
|
||||
|---|---|---|
|
||||
| canonical | `backend/data/grading.py::pct_ordinal` | its own unit tests |
|
||||
| Go SSR | `frontend/server/internal/format/format.go::PctOrdinal` | `format_test.go::TestPctOrdinal` (table-driven, not a cross-check) |
|
||||
| browser | `frontend/static/shared.js::pctOrd` | nothing automated |
|
||||
| Python SSR *(not deployed)* | `frontend/format.py::pct_ordinal` | the Python unit tier |
|
||||
|
||||
Every percentile on every surface goes through one of these. If they diverge,
|
||||
the same reading says two different things in two places. The backend copy is
|
||||
canonical; the others follow it.
|
||||
|
||||
`TEMP_BANDS` / `RAIN_BANDS` in `grading.py` are the source of truth for tier
|
||||
names and thresholds. Changing them without the frontend produces tiers drawn
|
||||
in colours that disagree with the labels the API returns.
|
||||
|
||||
## 5. The Fahrenheit country set
|
||||
|
||||
Fourteen codes: `US PR GU VI AS MP UM BS BZ KY PW FM MH LR`.
|
||||
|
||||
| Copy | File |
|
||||
|---|---|
|
||||
| canonical | `backend/api/content_payloads.py::F_COUNTRIES` |
|
||||
| Go SSR | `frontend/server/internal/format/format.go::FCountries` |
|
||||
| browser | `frontend/static/units.js::F_REGIONS` |
|
||||
| Python SSR *(not deployed)* | `frontend/format.py::F_COUNTRIES` |
|
||||
|
||||
`format_test.go::TestFCountriesMatchesBackend` re-parses the backend's Python
|
||||
source and asserts the Go copy matches — a genuinely good guard.
|
||||
|
||||
> ⚠️ **Verified gap.** Several source comments claim "a test asserts all three
|
||||
> stay identical." As of this writing that is not true: no test reads
|
||||
> `static/units.js`. All four copies currently *are* identical (checked), but
|
||||
> the browser copy is held in step by convention alone. If you touch this set,
|
||||
> update `units.js` by hand and diff it yourself. Extending the Go test to
|
||||
> parse `units.js` too would close this properly and is a good first
|
||||
> contribution.
|
||||
|
||||
## 6. Backend boot must survive an unreachable frontend, and vice versa
|
||||
|
||||
Both directions are load-bearing for asynchronous deploys:
|
||||
|
||||
- **Frontend → backend:** boot tries the IndexNow-key fetch once, catches
|
||||
failure, and falls back to a lazy per-request lookup. Don't make it fatal.
|
||||
- **Backend → frontend:** `THERMOGRAPH_FRONTEND_BASE_INTERNAL` is required at
|
||||
import (fails loud), but it's only *used* by the catch-all proxy fallback —
|
||||
an unreachable value doesn't stop the backend serving its own routes. That's
|
||||
why the test suite and the smoke harness both point it at
|
||||
`http://127.0.0.1:1`.
|
||||
|
||||
`/healthz` on both services is liveness only. Neither proves the other is
|
||||
reachable, deliberately — readiness is the ingress health-gate's job.
|
||||
|
||||
## 7. `THERMOGRAPH_BASE` must match on both processes
|
||||
|
||||
Default `/thermograph`; both images set `/`. Both processes must be configured
|
||||
**identically** in every real deployment — it's how they agree on the shared URL
|
||||
prefix. The Go frontend's content client reads it too, so a sub-path deployment
|
||||
works rather than only ever working at the root.
|
||||
|
||||
## 8. Image tags are keyed to the domain, not the branch tip
|
||||
|
||||
Both `build-push.yml` and `deploy.yml` compute the tag as
|
||||
`sha-` + the first 12 hex of `git log -1 --format=%H -- <domain>/`.
|
||||
|
||||
Two consequences:
|
||||
|
||||
- An infra-only commit at the branch tip cannot send a deploy chasing an image
|
||||
no build produced.
|
||||
- `deployed_version` can legitimately show a tag that is **not** the head of the
|
||||
branch. That's correct, not drift.
|
||||
|
||||
`fetch-depth: 0` in the deploy workflow exists for exactly this: in a
|
||||
path-filtered monorepo the tip is often another domain's commit, and a depth-1
|
||||
clone can't see past it.
|
||||
|
||||
## 9. Cookie auth is same-origin today
|
||||
|
||||
The interactive tool's auth is an HttpOnly session cookie. `account.js` uses
|
||||
`credentials: "include"` (not the fetch default) so the cookie still rides along
|
||||
if the script is ever loaded cross-origin — but the backend must actually be
|
||||
configured to accept credentialed cross-origin requests (CORS +
|
||||
`SameSite`) for that to work. In the default same-origin deployment it's a
|
||||
non-issue. **Don't assume a fully decoupled, independently-hosted frontend
|
||||
"just works" for logged-in features** without checking that configuration.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
Before opening a PR that touches both domains:
|
||||
|
||||
- [ ] Did any response payload shape change? → bump `PAYLOAD_VER`.
|
||||
- [ ] Did any route or payload break compatibility? → new `vN` router
|
||||
alongside, bump `API_CONTRACT_VERSION`, check `min_frontend`.
|
||||
- [ ] Did I change the API version pin? → both Go config **and**
|
||||
`static/account.js`, in one PR, after checking the target backend.
|
||||
- [ ] Did I touch tier names, thresholds, or percentile rendering? → check all
|
||||
four `pct_ordinal` copies and the colour tokens.
|
||||
- [ ] Did I touch the Fahrenheit set? → four copies, and `units.js` is
|
||||
unguarded.
|
||||
- [ ] Did I change the `/cell` bundle shape? → the slice→view map in
|
||||
`cache.js`.
|
||||
- [ ] Did I change ETag derivation or CORS headers? → re-read `cache.js`.
|
||||
|
||||
Next: [CI and release](07-ci-and-release.md).
|
||||
229
docs/onboarding/07-ci-and-release.md
Normal file
229
docs/onboarding/07-ci-and-release.md
Normal file
|
|
@ -0,0 +1,229 @@
|
|||
# 7. CI and release
|
||||
|
||||
## The branch model
|
||||
|
||||
```
|
||||
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
|
||||
│
|
||||
promotion PR
|
||||
▼
|
||||
main ──▶ beta.thermograph.org
|
||||
│
|
||||
promotion PR ← the owner's call, always
|
||||
▼
|
||||
release ──▶ thermograph.org
|
||||
```
|
||||
|
||||
`dev`, `main` and `release` are protected — direct pushes are blocked for
|
||||
everyone. Forgejo has **no auto-merge-on-green**, so merges are explicit: either
|
||||
click "Auto merge when checks succeed" on a PR, or merge via the API once CI is
|
||||
green.
|
||||
|
||||
Promotion is continuous and per-decision. There is no release calendar. Size a
|
||||
promotion to whatever has actually soaked on the environment below.
|
||||
|
||||
### Who may merge what
|
||||
|
||||
| Hop | Whose call | Norm |
|
||||
|---|---|---|
|
||||
| anything → `dev` | yours | Merge it. An open PR into `dev` is unfinished work, not caution. |
|
||||
| `dev` → `main` (beta) | yours, batched | Promote when a batch is coherent enough to be worth *testing*. Beta is the test environment; not promoting is not testing. |
|
||||
| `main` → `release` (prod) | **the owner's** | Prepare it, recommend for or against, and ask. Never unilaterally, never by another route. |
|
||||
|
||||
Centralis's `promote(from, to)` implements exactly this, and `forge_pr_merge`
|
||||
enforces it with no back door: it merges into `dev` and `main`, refuses
|
||||
`main → release` under any flag, and requires `confirm_protected_base: true`
|
||||
for the separate hotfix path.
|
||||
|
||||
### Two things about branch divergence
|
||||
|
||||
**Commit counts are not deliverable content.** Before promoting, compare the two
|
||||
tips' *trees*. Identical trees mean the same changes are already on the target
|
||||
under different SHAs — promoting there merges nothing while still firing the
|
||||
target's deploy workflows, including `infra-sync` re-rendering
|
||||
`/etc/thermograph.env` on prod and beta. The ahead/behind numbers will look like
|
||||
real work. `promote` refuses this outright.
|
||||
|
||||
**Expect `dev`, `main` and `release` to be mutually divergent, and expect
|
||||
`--ff-only` to fail.** Every promotion merges the source into the target,
|
||||
creating a merge commit *on the target* that the source never receives — so each
|
||||
promotion widens the gap it just closed. That is accumulated design, not drift.
|
||||
(Reconciling merges back into `dev` do happen; `be4aa94` in the history is one.)
|
||||
|
||||
## The nine workflows
|
||||
|
||||
All in root `.forgejo/workflows/`. **Never add a workflow under a domain's own
|
||||
`.forgejo/`** — Forgejo only reads the root ones, so a copy there is inert and
|
||||
becomes a trap for whoever edits it next.
|
||||
|
||||
### `pr-build.yml` — the required check
|
||||
|
||||
One workflow, one stable required-check name (**`gate`**), any number of
|
||||
domains:
|
||||
|
||||
1. a `changes` job diffs the PR against its base (needs `fetch-depth: 0` — the
|
||||
base commit isn't reachable from a depth-1 clone);
|
||||
2. per-domain jobs run only for domains the PR touches;
|
||||
3. `gate` reduces their results into the single status branch protection
|
||||
requires.
|
||||
|
||||
**Deliberately not path-filtered at the workflow level.** A path-filtered
|
||||
required check that never fires leaves a PR stuck on *"expected — waiting for
|
||||
status"* forever, so auto-merge never proceeds. An always-running gate whose
|
||||
domain jobs skip cleanly gives the same compute savings without the deadlock.
|
||||
This is the one intentionally *coupled* piece of the CI design.
|
||||
|
||||
### `build.yml` — the reusable build check
|
||||
|
||||
Called via `uses:` with a `domain` input. The build **context is that domain's
|
||||
subdirectory**, so each Dockerfile sees exactly the tree it saw when its domain
|
||||
was a repo root.
|
||||
|
||||
For **backend** it then runs the hermetic pytest suite *inside the image it just
|
||||
built* — the exact interpreter and dependencies that ship. Three flags there are
|
||||
load-bearing: `-u 0` (the image's default user can't write to site-packages),
|
||||
`--entrypoint sh` (otherwise the test command is swallowed as entrypoint args
|
||||
and the container just boots the server forever), and `unset THERMOGRAPH_BASE`
|
||||
(the image bakes `/`, which moves every route off the `/thermograph` prefix the
|
||||
tests are written against).
|
||||
|
||||
For **frontend** there's no separate test step: it's a Go binary with no
|
||||
interpreter in the runtime image, and `gofmt`/`vet`/`go test` already run in the
|
||||
Dockerfile's builder stage — so a failing Go test fails the Build step above.
|
||||
|
||||
It is deliberately **not** a live boot+healthz check. That was tried in the split
|
||||
repos and repeatedly hit the runner's own environment quirks (bridge-IP
|
||||
reachability, non-unique `$$`, squatted host ports) without ever going reliably
|
||||
green. Image boot is verified by hand instead — `make smoke`, or `docker run` +
|
||||
alembic + `/healthz` 200 + a real `/api/v2/place` 200.
|
||||
|
||||
### `build-push.yml` — images
|
||||
|
||||
Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
|
||||
`frontend/**`, and on `v*.*.*` tags. Publishes
|
||||
`git.thermograph.org/emi/thermograph/{backend,frontend}:sha-<12hex>`.
|
||||
|
||||
- **Serialised** (`max-parallel: 1`): one physical runner, and two whole image
|
||||
builds racing each other buys nothing.
|
||||
- **Semver exception:** a `v*.*.*` tag push carries no paths context, so a
|
||||
release tag builds **both** images. That's intended — a release names the
|
||||
whole set, not whichever domain moved last.
|
||||
- The registry is mesh-only, so the runner host needs
|
||||
`10.10.0.2 git.thermograph.org` in `/etc/hosts`.
|
||||
|
||||
### `deploy.yml` — one file, two environments, two services
|
||||
|
||||
Formerly six near-identical files. Branch selects the environment
|
||||
(`main → beta`, `release → prod`); a matrix covers the services.
|
||||
|
||||
The style is **deliberately boring**: no dynamic `fromJSON` matrix, no
|
||||
`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act
|
||||
runner may evaluate differently, and the failure mode here is "production does
|
||||
not deploy" or, worse, "deploys with an empty SSH host". So the two environments
|
||||
get two explicit, mutually exclusive steps and the tag is computed in shell.
|
||||
|
||||
Preserved from the originals, all load-bearing:
|
||||
|
||||
- `fetch-depth: 0` — the tag is keyed to the last commit that touched *that
|
||||
domain*, not the branch tip.
|
||||
- 12-hex truncation matching `build-push.yml` exactly.
|
||||
- Per-service, per-environment concurrency with **`cancel-in-progress: false`** —
|
||||
a half-finished deploy must never be cancelled by a newer one.
|
||||
- **Separate `PROD_SSH_*` credentials**, so a beta credential leak cannot reach
|
||||
prod.
|
||||
- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's
|
||||
default action registry.
|
||||
|
||||
There's also a per-leg refinement: the workflow-level `paths` filter only says
|
||||
*backend or frontend moved*, so each leg re-checks whether **this** push touched
|
||||
**this** domain. Without it a backend-only push would also roll the frontend,
|
||||
losing the independent-deploy property the whole split exists for. When there's
|
||||
no usable base SHA (first push, force push, `workflow_dispatch`) it deploys
|
||||
rather than skips — rolling onto the tag already running is a no-op, whereas
|
||||
skipping silently strands a change.
|
||||
|
||||
### `infra-sync.yml` — infra's own pipeline
|
||||
|
||||
Push to `main` touching `infra/**` → SSH to beta **and** prod, fast-forward
|
||||
`/opt/thermograph`, re-render `/etc/thermograph.env` from the vault.
|
||||
|
||||
**Rolls no service.** Image tags are the app domains' axis, not infra's. A
|
||||
compose change that must recreate containers takes effect on the next app
|
||||
deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh`.
|
||||
|
||||
Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release`
|
||||
maps to LAN→beta→prod via image tags); **infra is not** — both hosts track infra
|
||||
via `main`. Prod's *app images* are staged by `release`, but its checkout
|
||||
follows `main`.
|
||||
|
||||
### `observability-validate.yml`
|
||||
|
||||
Parses every artifact that ships to a node: both compose files, all dashboard
|
||||
JSON, the Loki and provisioning YAML, and the Alloy config — checked with the
|
||||
**pinned `alloy` binary (v1.9.1, matching what the fleet runs)**, downloaded
|
||||
from its GitHub release, no Docker CLI needed.
|
||||
|
||||
Alerting gets a stricter third step: every rule's `condition` must name a refId
|
||||
that exists, every policy must route to a receiver that exists, and a literal
|
||||
Discord webhook URL in the repo is a hard failure. **A rule pointing at a
|
||||
missing refId is valid YAML, provisions cleanly, and then never fires** — that
|
||||
silent no-op is exactly what this check exists to prevent.
|
||||
|
||||
### `ops-cron.yml` — the prod backup
|
||||
|
||||
Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into prod
|
||||
and run inside the already-running stack.
|
||||
|
||||
**It uses `PROD_SSH_*`, not `SSH_*`.** An earlier revision reused `SSH_*` — so
|
||||
the "prod" backup was silently dumping *beta*, and prod had no backup at all. If
|
||||
you touch this file, verify a dump actually lands in
|
||||
`agent@prod:~/thermograph-backups/`.
|
||||
|
||||
### `secrets-guard.yml` and `shell-lint.yml` — the backstops
|
||||
|
||||
Both run on **every** PR and push, deliberately un-path-filtered: *a backstop
|
||||
that only runs when you expect it to isn't a backstop.*
|
||||
|
||||
- `secrets-guard` fails if any `infra/deploy/secrets/*.yaml` lacks both a
|
||||
`sops:` block and `ENC[AES256_GCM` values. ~1 second.
|
||||
- `shell-lint` runs shellcheck over every `*.sh` found by `find` (not a list —
|
||||
so a new script is covered the moment it lands). Pinned to **v0.11.0 plus a
|
||||
sha256**, installed from the official static release rather than `apt-get`:
|
||||
the distro version drifts with the job image, and a drifted shellcheck grows
|
||||
new warnings that fail CI on an unrelated push. Bump the version and the hash
|
||||
together, and expect to fix new findings in the same PR.
|
||||
|
||||
Keep the tree at **zero shellcheck findings**. These scripts run as root over
|
||||
SSH against live hosts with no test suite in front of them; this is the only
|
||||
guard.
|
||||
|
||||
## Runner facts
|
||||
|
||||
- Jobs run on the `docker` label — the LAN box's Forgejo Actions runner.
|
||||
- Job containers get the **host's docker.sock** automounted
|
||||
(Docker-outside-of-Docker, no privileged mode). The job image
|
||||
(`node:20-bookworm`) ships no docker CLI, which is why several workflows
|
||||
`apt-get install docker.io` first.
|
||||
- Forgejo's runner has **only the labels it was explicitly registered with**.
|
||||
GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does
|
||||
not. A `runs-on: [self-hosted, thermograph-lan]` array is permanently
|
||||
unschedulable there — a real bug that was found and fixed once, and worth
|
||||
recognising if a job ever silently stops appearing in the Actions history.
|
||||
|
||||
## Opening a PR
|
||||
|
||||
```bash
|
||||
git switch -c fix/short-description # branch off dev
|
||||
# ... work ...
|
||||
cd backend && make test # or: cd frontend/server && go test ./...
|
||||
git commit # concise, technical, no AI mentions
|
||||
git push -u origin fix/short-description
|
||||
```
|
||||
|
||||
Then open the PR into `dev` (Centralis's `forge_pr_open`, or the Forgejo web
|
||||
UI). Wait for `gate`. Merge — squash is the configured style.
|
||||
|
||||
If your change touches shell, expect `shell-lint` to have an opinion. If it
|
||||
touches `infra/deploy/secrets/`, `secrets-guard` will too.
|
||||
|
||||
Next: [Infra and secrets](08-infra-secrets.md).
|
||||
254
docs/onboarding/08-infra-secrets.md
Normal file
254
docs/onboarding/08-infra-secrets.md
Normal file
|
|
@ -0,0 +1,254 @@
|
|||
# 8. Infra and secrets
|
||||
|
||||
`infra/` owns **how and where the already-built images run**. It never builds
|
||||
app source. Hosts' `/opt/thermograph` is a checkout of this whole monorepo.
|
||||
|
||||
Read [`infra/CLAUDE.md`](../../infra/CLAUDE.md) alongside this.
|
||||
|
||||
## The machines
|
||||
|
||||
| Host | Public | Mesh | Orchestrator | Login |
|
||||
|---|---|---|---|---|
|
||||
| **prod** | `169.58.46.181` / thermograph.org | `10.10.0.1` | Docker **Swarm** | `agent`, passwordless sudo |
|
||||
| **beta** | `75.119.132.91` / beta.thermograph.org | `10.10.0.2` | **compose** | `agent`, passwordless sudo |
|
||||
| **desktop** | — | `10.10.0.3` | compose (LAN dev) | it's your box |
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod
|
||||
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta
|
||||
```
|
||||
|
||||
Beta also hosts **Forgejo** (git + CI + registry) and **Grafana + Loki**, which
|
||||
is why beta is guarded as strictly as prod: a destructive command there takes
|
||||
out git, CI and the registry at once. The LAN dev box is deliberately
|
||||
unguarded.
|
||||
|
||||
Every root-effective command on the VPSes is logged by `auditd`
|
||||
(`ausearch -k agentcmd`). That's a feature — fixes are attributable.
|
||||
|
||||
Prefer Centralis for routine work (`run_on_host`, `fleet_status`,
|
||||
`deployed_version`) over raw SSH.
|
||||
|
||||
## The two orchestrators
|
||||
|
||||
Which path a host takes is decided by **`/etc/thermograph/deploy-mode`**: the
|
||||
string `stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything
|
||||
else is compose. The workflows never need to know which mode a host runs.
|
||||
|
||||
| | compose (beta, LAN dev) | Swarm (prod) |
|
||||
|---|---|---|
|
||||
| File | `infra/docker-compose.yml` | `infra/deploy/stack/thermograph-stack.yml` |
|
||||
| Services | `db`, `backend`, `lake`, `daemon`, `frontend` | `db`, `web`, `worker`, `lake`, `daemon`, `frontend`, `autoscaler`, `autoscaler-lake` |
|
||||
| Rolling | `up -d --no-deps <targets>` | start-first, health-gated, auto-rollback |
|
||||
| Tag file | `deploy/.image-tags.env` | `deploy/.stack-image-tags.env` |
|
||||
|
||||
Note prod splits `backend` into **`web`** and **`worker`** (the
|
||||
`THERMOGRAPH_ROLE` split from [04](04-backend.md)); compose runs one `backend`
|
||||
service in role `all`.
|
||||
|
||||
`STACK_TEST=1` rehearses the entire Swarm deploy under stack name
|
||||
`thermograph-test` on throwaway volumes and ports `18137`/`18080`. Leftover
|
||||
`thermograph-test_*` containers in logs are rehearsal residue — ignore them.
|
||||
|
||||
### The volume/project-name coupling
|
||||
|
||||
Compose creates `thermograph_pgdata` / `_appdata` / `_applogs`; the Swarm stack
|
||||
declares those exact names as `external: true` at the same mount paths. That
|
||||
coupling is why `name: thermograph` is pinned in the compose file — running
|
||||
compose from `infra/` without it derives project `infra`, silently creating a
|
||||
whole new stack with empty volumes beside the running one. `deploy-dev.sh`
|
||||
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
|
||||
keep LAN dev separate on purpose. **Keep both halves.**
|
||||
|
||||
## `deploy/deploy.sh` — read this before you touch a deploy
|
||||
|
||||
Single entry point for beta and prod. Contract:
|
||||
|
||||
```bash
|
||||
SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
|
||||
SERVICE=frontend FRONTEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
|
||||
SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> …/deploy.sh
|
||||
```
|
||||
|
||||
What it does, in order, and why each step is the way it is:
|
||||
|
||||
1. **`flock` on `deploy/.deploy.lock`**, with a self re-exec so the lock spans
|
||||
the whole run. Backend and frontend deploys can fire for the same push
|
||||
seconds apart and both SSH into one checkout; concurrent runs race the git
|
||||
reset, the compose project and the tag file. `-w 600` bounds the wait.
|
||||
2. **Render secrets** from the SOPS vault into `/etc/thermograph.env`, then
|
||||
source it, so a by-hand run interpolates the same as the systemd unit does.
|
||||
Guarded on the helper's existence so the very deploy that *introduces*
|
||||
`render-secrets.sh` is safe.
|
||||
3. **`git reset --hard origin/$BRANCH`** on the checkout root (`BRANCH` defaults
|
||||
to `main`). ⚠️ **Uncommitted edits in `/opt/thermograph` evaporate here.**
|
||||
The one exception is `deploy/.image-tags.env`, untracked on purpose.
|
||||
4. **Read `.image-tags.env`** so a single-service roll re-renders compose with
|
||||
*both* services' real tags and never accidentally recreates or downgrades the
|
||||
sibling.
|
||||
5. **Registry login** only if `REGISTRY_TOKEN` is set. The CI paths don't pass
|
||||
one — the host is already `docker login`ed — and an unconditional login with
|
||||
an empty token would abort under `set -e`.
|
||||
6. **Pull, with a bounded ~5-minute retry.** `build-push.yml` is a *separate*
|
||||
workflow triggered by the same push, and Forgejo's `needs:` only orders jobs
|
||||
within one file. A deploy racing ahead of its build and failing "not found"
|
||||
is confirmed-live behaviour; the retry covers a normal build and still fails
|
||||
loudly on a genuine problem.
|
||||
7. **Daemon capability probe.** Infra tracks `main` while image tags are
|
||||
env-staged, so a host can legitimately be asked to roll a backend image older
|
||||
than the compose file — one built before the daemon binary existed. The
|
||||
script `docker run`s the image to test for `/usr/local/bin/thermograph-daemon`
|
||||
and drops `daemon` from this run if it's absent, rather than leaving a
|
||||
container crash-looping on a missing binary.
|
||||
8. **Roll.** `TARGETS` are `backend → (backend, lake, daemon)`,
|
||||
`frontend → (frontend)`, `all → everything`. **`daemon` and `lake` ride with
|
||||
`backend` and are never targets on their own** — all three run the same image
|
||||
at the same tag, and rolling one without the others is exactly the version
|
||||
skew the `/internal/*` contract has no negotiation for. Single-service rolls
|
||||
use `--no-deps`; a full `all` uses `--remove-orphans` (a renamed-away
|
||||
service's old container would otherwise squat its port — confirmed live).
|
||||
9. **Write `.image-tags.env`** *after* `up`, so a failed pull never records a
|
||||
tag that isn't running.
|
||||
10. **Health check** (backend 8137, frontend 8080), then detached city warming.
|
||||
|
||||
### Rollback
|
||||
|
||||
Rollback is redeploying the previous image tag. But **do not treat image
|
||||
retention as a rollback guarantee**: both `deploy.sh` and `deploy-stack.sh`
|
||||
*end* by deleting images outside the running pair. On beta that succeeds, so
|
||||
beta typically holds **no local rollback target at all**; on prod it usually
|
||||
fails only because Swarm's stopped task containers still hold references, which
|
||||
is why prod keeps a handful. Verify the target tag is actually present before
|
||||
promising a rollback — Centralis's `rollback_to(dry_run: true)` checks both host
|
||||
and registry.
|
||||
|
||||
`docker system prune -a` on a live box is forbidden for the same reason.
|
||||
`docker image prune -f` (dangling only) is the safe form.
|
||||
|
||||
## Secrets — the SOPS + age vault
|
||||
|
||||
**The single source of truth is `infra/deploy/secrets/*.yaml`**, committed
|
||||
encrypted (values only — keys stay readable so diffs mean something) and
|
||||
rendered into `/etc/thermograph.env` at deploy time by
|
||||
`deploy/render-secrets.sh`.
|
||||
|
||||
**Cycling a key is: edit → commit → deploy.** No SSH, no hand-edited root file,
|
||||
no per-host duplication.
|
||||
|
||||
| File | Holds |
|
||||
|---|---|
|
||||
| `common.yaml` | the 16 values identical on prod **and** beta — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config |
|
||||
| `prod.yaml` | prod's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL`, Discord + mail credentials that exist nowhere else |
|
||||
| `beta.yaml` | beta's own — the three held-back credentials, sizing, base URL |
|
||||
| `dev.yaml` | the LAN box's own, **self-contained**, 12 values, no production credential |
|
||||
| `centralis.prod.yaml` | Centralis's nine variables → `/etc/centralis.env`, its own renderer |
|
||||
| `example.yaml` | format reference / CI fixture (fake values) |
|
||||
| `../../.sops.yaml` | which age recipients files encrypt to (plaintext config) |
|
||||
|
||||
The renderer concatenates `common.yaml` then `<env>.yaml`, so a **host value
|
||||
wins** (last occurrence of a duplicate key). `<env>` comes from
|
||||
`/etc/thermograph/secrets-env` on the box.
|
||||
|
||||
### Two design decisions worth understanding
|
||||
|
||||
**Three credentials are deliberately *not* in `common.yaml`** —
|
||||
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_DATABASE_URL`.
|
||||
They hold identical values on prod and beta today, so by the mechanical rule
|
||||
they belong there. They're kept per-host anyway because these are the
|
||||
credentials that let one environment *act as* another: with them, a foothold on
|
||||
beta (the more exposed box — public Forgejo and Grafana) is a foothold on prod's
|
||||
database and prod's session signing. They match because beta was seeded from
|
||||
prod, not because the two are meant to be one system. Keeping them per-host
|
||||
costs one extra line and buys the ability to diverge.
|
||||
|
||||
**`dev` renders `dev.yaml` alone** — `deploy-dev.sh` exports
|
||||
`THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of `common.yaml`'s sixteen values
|
||||
are live production credentials, including a **read-write** object-storage
|
||||
keypair on the bucket that also holds prod's database backups, and the VAPID
|
||||
private key that signs push to real subscribers. The render is plaintext
|
||||
concatenation, so layering it would put all of those into every container in the
|
||||
dev stack — on the operator's desktop, which is also the CI runner whose
|
||||
`docker`-labelled jobs get the host socket, running unreviewed `dev` code. An
|
||||
override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the
|
||||
production value is still physically a line in the file.
|
||||
|
||||
Dev works fine without them: no S3 keys means the `lake` service answers 503 and
|
||||
history falls through to the Open-Meteo archive (an accelerator, never a
|
||||
dependency); VAPID and IndexNow both self-generate and persist to the `appdata`
|
||||
volume; with no metrics token `/api/v2/metrics` is direct-loopback-only, which
|
||||
is right on a LAN box. One `common.yaml` value is simply *wrong* for dev —
|
||||
`THERMOGRAPH_COOKIE_SECURE=1` silently breaks login over plain HTTP.
|
||||
|
||||
### Working with the vault
|
||||
|
||||
```bash
|
||||
sops edit infra/deploy/secrets/beta.yaml # never a plain editor
|
||||
infra/deploy/secrets/dry-run-render.sh # preview a render
|
||||
```
|
||||
|
||||
- **Never hand-edit `/etc/thermograph.env`** — it's a rendered artifact,
|
||||
overwritten on the next deploy.
|
||||
- **Never paste a plaintext secret onto a box.**
|
||||
- `secrets-guard` CI rejects a plaintext commit; the repo's `secrets-guard.sh`
|
||||
hook blocks a direct Write/Edit to those files locally.
|
||||
- `deploy/secrets/seed-from-live.sh` reads production secrets and is **not**
|
||||
for an agent to run.
|
||||
- The **`key-gaps`** skill audits which keys are missing or drifting across
|
||||
environments. It reads key *names* only, never values.
|
||||
|
||||
There's a cautionary tale in `deploy/secrets/README.md` worth reading: Centralis
|
||||
was hand-configured until a hand-edit wrote `CENTRALIS_TOKENS` as bare JSON into
|
||||
a file that gets `source`d, bash stripped the quotes, and Centralis — correctly
|
||||
failing closed on malformed JSON — dropped to a single identity. Nothing logged
|
||||
an error. It looked exactly like the file had never been written. That's why
|
||||
`centralis.prod.yaml` exists: the only durable fix was *a human stops writing
|
||||
the file*.
|
||||
|
||||
## Terraform
|
||||
|
||||
`infra/terraform/` provisions and configures hosts, SSH-driven.
|
||||
|
||||
> **Never run `terraform apply` casually.** No tfstate is persisted anywhere, so
|
||||
> an apply would attempt full re-provisioning of live hosts. Terraform here is
|
||||
> **executable documentation** until state is bootstrapped.
|
||||
|
||||
## Ops query tooling
|
||||
|
||||
Read-only, uniform across environments:
|
||||
|
||||
```bash
|
||||
infra/ops/dbq.sh dev "select count(*) from climate_history"
|
||||
infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history"
|
||||
infra/ops/iceberg.sh prod -c "select count(*) from era5_daily"
|
||||
```
|
||||
|
||||
Both exec **into the container**. No database is exposed over TCP — each listens
|
||||
only on its private docker network, and prod's is a Swarm **overlay the prod
|
||||
host itself cannot route to**, so `ssh -L` works for beta and is *impossible*
|
||||
for prod. Exec works identically everywhere with no ports, no tunnels, no infra
|
||||
changes. The prod container name is a Swarm task name that changes on every
|
||||
redeploy, so it's resolved at call time via `docker ps --filter name=`, never
|
||||
hardcoded.
|
||||
|
||||
Queries connect as **`thermograph_ro`** — `NOSUPERUSER`, granted only
|
||||
`pg_read_all_data`. Read-only is enforced by Postgres, not by convention:
|
||||
|
||||
```
|
||||
$ infra/ops/dbq.sh prod -c "create table t(i int)"
|
||||
ERROR: permission denied for schema public
|
||||
```
|
||||
|
||||
Centralis's `sql_query` does the same thing with the same role. `write: true`
|
||||
escalates to the app's superuser and takes effect immediately with no
|
||||
confirmation — on prod, know what you are doing.
|
||||
|
||||
## Backups
|
||||
|
||||
The nightly `ops-cron.yml` `pg_dump` into `agent@prod:~/thermograph-backups/`
|
||||
is it. Known gaps, stated plainly: **a single copy on the same box as the
|
||||
database, no offsite yet**, and restores must handle TimescaleDB's
|
||||
`continuous_agg` circular-FK warning (`--disable-triggers`). The `backups/`
|
||||
prefix in the object-storage bucket belongs to those jobs — don't write outside
|
||||
the lake's own prefix.
|
||||
|
||||
Next: [Observability](09-observability.md).
|
||||
199
docs/onboarding/09-observability.md
Normal file
199
docs/onboarding/09-observability.md
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
# 9. Observability
|
||||
|
||||
Fleet-wide log aggregation: **Loki + Grafana on beta**, fed by a **Grafana
|
||||
Alloy** agent on every node, all over the WireGuard mesh.
|
||||
|
||||
```
|
||||
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3)
|
||||
┌────────────┐ ┌──────────────────┐ ┌────────────┐
|
||||
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│
|
||||
└────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘
|
||||
docker+caddy+app └──────────────────┘ docker+caddy+app
|
||||
```
|
||||
|
||||
- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
|
||||
`10.10.0.2:3100`. Never public.
|
||||
- **Grafana** — the UI, fronted by beta's Caddy at
|
||||
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
|
||||
`allow_sign_up=false`, plus a break-glass local admin). Use that hostname
|
||||
everywhere; **never** `grafana.thermograph.org`.
|
||||
- **Alloy** — on each node, shipping three sources: every Docker container's
|
||||
stdout/stderr, Caddy's host access logs, and the app's structured JSON logs
|
||||
(`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` become
|
||||
labels). Every line is tagged `host = prod|beta|dev`.
|
||||
|
||||
There is **no build and no deploy automation** for this domain — it ships by
|
||||
hand. But `observability-validate.yml` parses every artifact that ships to a
|
||||
node, so a malformed dashboard or an unparseable Alloy config fails in CI
|
||||
instead of on the host.
|
||||
|
||||
## The rule that matters most
|
||||
|
||||
**The repo is the only durable path.** Dashboards and alerting are provisioned
|
||||
from `observability/grafana/provisioning/` at startup, so **edits made through
|
||||
Grafana's UI or API are overwritten on the next provision**. A change goes
|
||||
through a PR. Centralis's `dashboard_write` opens one for you.
|
||||
|
||||
## Reading the fleet
|
||||
|
||||
Start wide, then narrow:
|
||||
|
||||
1. `fleet_status` — HTTP reachability plus what's running on each host.
|
||||
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
|
||||
3. `logs_query service=… host=…` — the actual lines.
|
||||
|
||||
### ⚠ The prod trap
|
||||
|
||||
Prod runs a Swarm stack, and its `job="docker"` streams carry **no `service`
|
||||
label**. A raw `{host="prod", service="backend"}` query matches nothing and
|
||||
reads as *"no logs"* when it actually means *"wrong selector"*.
|
||||
|
||||
Use Centralis's `logs_query` `service` argument, which rewrites it to a
|
||||
container name-regex with the churning task suffix wildcarded. Prod service
|
||||
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
|
||||
`_lake`.
|
||||
|
||||
Related: **prod container names change on every redeploy** (the Swarm task
|
||||
suffix). Never hardcode one; resolve via `docker ps --filter name=` at call
|
||||
time.
|
||||
|
||||
## The dashboard
|
||||
|
||||
`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log
|
||||
volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a
|
||||
per-node notifier-liveness indicator, recent app errors, and a live
|
||||
all-container tail.
|
||||
|
||||
## Alerting
|
||||
|
||||
Provisioned from `grafana/provisioning/alerting/`, mounted read-only into
|
||||
Grafana.
|
||||
|
||||
**Every rule is LogQL** — there is no Prometheus anywhere in the fleet. Loki is
|
||||
the only datasource, so every signal is derived from log lines. Thresholds were
|
||||
backtested against real Loki data and the working is written out in the comments
|
||||
beside each rule; **re-derive before changing a number rather than guessing**.
|
||||
|
||||
### Alerts go to Discord `#ops-alerts`, never email
|
||||
|
||||
Three reasons, all learned the hard way:
|
||||
|
||||
- Grafana's factory default pointed at the literal string
|
||||
`<example@email.com>` — a paging path that had never delivered a message to
|
||||
anyone.
|
||||
- Beta's Grafana relays SMTP through **prod's** Postfix. Email alerts therefore
|
||||
travel through the box most likely to be on fire, and vanish exactly when they
|
||||
matter.
|
||||
- Discord is off-estate, works when prod is dead, and reaches a phone.
|
||||
|
||||
`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
|
||||
subscription output — real deliveries to real subscribers — so routing ops noise
|
||||
there would corrupt the evidence.
|
||||
|
||||
The webhook URL is a secret and lives only in beta's gitignored `.env` as
|
||||
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
|
||||
webhook URL committed to the repo is a **hard CI failure**.
|
||||
|
||||
### The twelve rules
|
||||
|
||||
| Rule | Fires when | Severity |
|
||||
|---|---|---|
|
||||
| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the pipeline is broken | critical |
|
||||
| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
|
||||
| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
|
||||
| `ProdHTTP5xxElevated` | ≥25 prod 5xx over 6h (the error-budget alarm) | warning |
|
||||
| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
|
||||
| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` from prod in 20m | critical |
|
||||
| `NotifierHeartbeatMissingBeta` | same, on beta | warning |
|
||||
| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
|
||||
| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
|
||||
| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
|
||||
| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
|
||||
| `AlertingWatchdog` | always — one Discord message per day | watchdog |
|
||||
|
||||
`AlertingWatchdog` is the dead-man's switch and fires permanently by design.
|
||||
**Its value is its absence:** nothing can report its own death, so if
|
||||
`#ops-alerts` has been silent for more than a day, alerting is broken rather
|
||||
than the estate healthy.
|
||||
|
||||
### The strict CI check, and why
|
||||
|
||||
A rule whose `condition` names a refId that does not exist is **perfectly valid
|
||||
YAML, provisions without complaint, and then never fires** — a paging path that
|
||||
looks healthy and reaches nobody. `observability-validate.yml` checks that every
|
||||
condition names a real refId and every policy routes to a receiver that exists,
|
||||
precisely because nothing else would catch it.
|
||||
|
||||
The same asymmetry shows up in verification: `GET
|
||||
/api/v1/provisioning/contact-points` **redacts** the webhook URL, so you cannot
|
||||
confirm interpolation by reading it back. A contact point holding the literal
|
||||
string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and
|
||||
pages nobody. **The only proof is a message actually arriving in
|
||||
`#ops-alerts`** — force one with the receivers-test endpoint (the command is in
|
||||
`observability/README.md`).
|
||||
|
||||
## Deploying an observability change
|
||||
|
||||
> ⚠️ **Merging is not deploying — and this one has already bitten.**
|
||||
> `/opt/observability` on beta and prod is a checkout of the **archived**
|
||||
> `emi/thermograph-observability` repo. It can never `git pull` again; the
|
||||
> content now lives here under `observability/`. A previous observability PR
|
||||
> merged and had **zero live effect** for exactly this reason.
|
||||
|
||||
Until someone re-points those checkouts at the monorepo, every change ships by
|
||||
hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up,
|
||||
`sudo cp` it into place, restart the specific container.
|
||||
|
||||
Two gotchas in that sequence:
|
||||
|
||||
- Use `sudo docker restart <container>` on **prod** — not `docker compose`,
|
||||
which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
|
||||
interpolate from. Beta's stack *does* have an `.env`, so compose works there.
|
||||
- **`docker restart` will not pick up a new environment variable.** Any change
|
||||
that introduces one needs `docker compose up -d grafana` on beta.
|
||||
|
||||
Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana.
|
||||
|
||||
Full step-by-step, including rollback, is in
|
||||
[`observability/README.md`](../../observability/README.md).
|
||||
|
||||
## Known gaps (as documented in the repo)
|
||||
|
||||
- **Alloy can silently stop tailing a container after Swarm replaces the task.**
|
||||
Observed on prod 2026-07-24: `thermograph_worker` was `1/1`, healthy, writing
|
||||
~19 lines per 10 minutes to its own docker log, yet not one line reached Loki
|
||||
for 13½ hours. What localises it to Alloy: prod's `tag=heartbeat` lines kept
|
||||
arriving throughout, and those come from a *different* Alloy source
|
||||
(`loki.source.file` on `/applogs`) than container stdout
|
||||
(`loki.source.docker`). One source went deaf, the other didn't. Workaround:
|
||||
`sudo docker restart alloy-alloy-1` on the node. **Not root-caused.**
|
||||
`ProdWorkerContainerSilent` now catches this class of failure.
|
||||
- **Some services are too quiet to alert on.** `thermograph_daemon` (23 lines /
|
||||
24h), `thermograph_autoscaler` (1) and `_autoscaler-lake` (2) have a healthy
|
||||
floor indistinguishable from dead; `thermograph_lake` is bursty ETL with
|
||||
legitimate idle gaps. Giving each an `audit.log_heartbeat()` line would make
|
||||
all four alertable with the pattern the notifier already uses. **The single
|
||||
highest-value follow-up in this domain.**
|
||||
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
|
||||
dead-man's switch — it relies on somebody noticing the daily message stopped.
|
||||
An external uptime pinger from off-estate would close this properly.
|
||||
- **Beta runs a `docker-compose.override.yml` that is not in this repo**, wiring
|
||||
Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked
|
||||
compose file or deleted; alerting no longer depends on it.
|
||||
|
||||
## When something breaks
|
||||
|
||||
1. `fleet_status` — is it up at all?
|
||||
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
|
||||
3. `deployed_version env=prod` — did something ship just before it started?
|
||||
4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass?
|
||||
5. **Write down what you found with `notes_write` *before* fixing it.** The
|
||||
reconstruction is worth more than the fix, and it will be gone by tomorrow.
|
||||
|
||||
You are authorised to fix production roadblocks directly — restart or roll a
|
||||
wedged service, clear a full disk, kill a runaway process, bounce a crash-looping
|
||||
container. What you must not do is make the fix *durable* on the box: writes to
|
||||
code, docs and dashboards go through a PR, and `deploy.sh` hard-resets the
|
||||
checkout anyway. The on-box action is stabilisation; the PR is the fix.
|
||||
|
||||
Next: [Recipes](10-recipes.md).
|
||||
256
docs/onboarding/10-recipes.md
Normal file
256
docs/onboarding/10-recipes.md
Normal file
|
|
@ -0,0 +1,256 @@
|
|||
# 10. Recipes
|
||||
|
||||
Task-shaped walkthroughs. Each one names the files you'll touch and the traps
|
||||
specific to that change.
|
||||
|
||||
---
|
||||
|
||||
## Add a field to a graded API payload
|
||||
|
||||
1. **Build it** in `backend/api/payloads.py` (or `content_payloads.py` for the
|
||||
SSR content API). Payload assembly is pure `inputs → dict`; keep HTTP
|
||||
semantics with the caller.
|
||||
2. **Bump `PAYLOAD_VER`** in `api/payloads.py`. Non-negotiable — the derived
|
||||
store is token-validated with no clock expiry, so without a bump the old
|
||||
shape serves out of cache forever, silently. If the change is content-API
|
||||
only, `CONTENT_VER` is the narrower lever.
|
||||
3. **Decide whether it's breaking.** Additive → fine on `v2`. Breaking → a new
|
||||
`v3` router mounted alongside `v2` re-registering only the changed handlers,
|
||||
plus `API_CONTRACT_VERSION` bumped in the same PR.
|
||||
4. **Check `/cell`.** If the field lands in a slice, the slice→view map in
|
||||
`frontend/static/cache.js` has to track it.
|
||||
5. **Consume it** in `frontend/server/internal/contentapi/types.go` and whichever
|
||||
template or JS renders it.
|
||||
6. **Test both sides**: `cd backend && make test`, then
|
||||
`cd frontend/server && go test ./...`.
|
||||
7. **Refresh fixtures if the shape moved**: `cd frontend && make capture-fixtures`
|
||||
against a live backend. Those fixtures feed the **Go** tests too.
|
||||
|
||||
---
|
||||
|
||||
## Add a new SSR page
|
||||
|
||||
The backend owns the page's *data and metadata*; the frontend owns its *markup*.
|
||||
|
||||
1. `backend/api/content_payloads.py` — a builder returning JSON-safe dicts (raw
|
||||
floats, no `Markup`, no ContextVar unit rendering), including `page_title`,
|
||||
`canonical_path`, `breadcrumb` and `jsonld`. Those live here, **not** in the
|
||||
frontend.
|
||||
2. `backend/api/content_routes.py` — route wiring plus the same derived-store /
|
||||
ETag pattern the neighbouring routes use.
|
||||
3. `frontend/server/internal/contentapi/` — a typed client method.
|
||||
4. `frontend/server/internal/render/templates/<page>.html.tmpl` — embedded in
|
||||
the binary, so a template change needs a rebuild.
|
||||
5. `frontend/server/internal/content/pages.go` — the handler; register the route.
|
||||
6. If the page should be crawlable, add it to the sitemap
|
||||
(`backend/api/sitemap.py` and the `/content/sitemap` payload).
|
||||
7. Copy that isn't per-city data goes in `frontend/content/pages.yaml`, loaded
|
||||
fail-loud at boot.
|
||||
|
||||
---
|
||||
|
||||
## Change something visual
|
||||
|
||||
1. Read [`frontend/DESIGN.md`](../../frontend/DESIGN.md) first. It wins over any
|
||||
general design guidance.
|
||||
2. Edit `frontend/static/style.css`. **Use `var(--token)`; never paste a hex.**
|
||||
New surfaces take their colour from existing tokens so light mode keeps
|
||||
working for free.
|
||||
3. If it's a grade colour, keep the scale's order and midpoints intact — they
|
||||
encode meaning, not decoration, and the charts pull from the same tokens as
|
||||
the legend.
|
||||
4. **Verify at 390 / 800 / 1920 / 2560 / 3840px in both light and dark:**
|
||||
```bash
|
||||
cd frontend && .venv/bin/python tools/shoot.py
|
||||
```
|
||||
or drive a real browser with the Chrome DevTools MCP. Narrow while
|
||||
iterating: `tools/shoot.py index --width 390 --scheme dark`.
|
||||
5. Watch the hard rules: inputs at `font-size: 16px` minimum (iOS zoom), ~44px
|
||||
touch targets, honour `prefers-reduced-motion`, metric order always
|
||||
`Precip · High · Low`.
|
||||
|
||||
---
|
||||
|
||||
## Add or rotate a secret
|
||||
|
||||
```bash
|
||||
sops edit infra/deploy/secrets/prod.yaml # never a plain editor
|
||||
git commit && push && open a PR
|
||||
```
|
||||
|
||||
Then it reaches the hosts by either route: an `infra/**` push to `main` fires
|
||||
`infra-sync.yml` (re-renders `/etc/thermograph.env` on beta and prod, rolls
|
||||
nothing), or the next app deploy renders it as step 2 of `deploy.sh`.
|
||||
|
||||
Decide **which file** first:
|
||||
|
||||
- identical on prod *and* beta, and not a "lets one environment act as another"
|
||||
credential → `common.yaml`;
|
||||
- per-host, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` /
|
||||
`THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`;
|
||||
- dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's.
|
||||
|
||||
Preview with `infra/deploy/secrets/dry-run-render.sh`. Audit cross-environment
|
||||
drift with the `key-gaps` skill (names only, never values).
|
||||
|
||||
**Never** hand-edit `/etc/thermograph.env`, paste a plaintext secret onto a box,
|
||||
or run `seed-from-live.sh`.
|
||||
|
||||
If a rotated value needs containers recreated to take effect, remember
|
||||
`infra-sync` rolls nothing — you need an app deploy or a by-hand
|
||||
`SERVICE=all … deploy.sh`.
|
||||
|
||||
---
|
||||
|
||||
## Add a recurring background job
|
||||
|
||||
Ask one question first: **does it need data?**
|
||||
|
||||
- **No** (a timer, a connection, an external poke) → add a `cron.Job` in
|
||||
`backend/daemon/main.go`. One replica, so no election needed. Give it an env
|
||||
var for its interval, defaulting sensibly and falling back to the default on a
|
||||
malformed value.
|
||||
- **Yes** → add an `/internal/*` route in `backend/api/internal_routes.py` and
|
||||
have the daemon POST to it. Grading depends on polars and the cache;
|
||||
reimplementing it out-of-process would let results drift from the API's.
|
||||
|
||||
If you genuinely need an in-process background thread in the Python app instead,
|
||||
**read `core/singleton.py` first**. Anything that issues upstream fetches on a
|
||||
timer must run behind `claim_leader()` — N workers × M hosts multiplies quota
|
||||
use N×M, and that has already caused overnight 429s in production. Things that
|
||||
drain a per-worker request-fed queue (the neighbour warmer) correctly run
|
||||
everywhere; so does the heartbeat, because a beat costs one local file append.
|
||||
|
||||
Then give it a heartbeat (`audit.log_heartbeat()`) so observability can alert on
|
||||
its death — see the "too quiet to alert on" gap in
|
||||
[observability](09-observability.md).
|
||||
|
||||
---
|
||||
|
||||
## Change a Grafana alert or dashboard
|
||||
|
||||
**Never through the UI.** Provisioning overwrites it on the next start.
|
||||
|
||||
1. Edit `observability/grafana/provisioning/alerting/*.yml` or
|
||||
`observability/grafana/dashboards/*.json`.
|
||||
2. If you're changing a threshold, **re-derive it against real Loki data** — the
|
||||
working for the current number is in the comments beside the rule.
|
||||
3. Open a PR. `observability-validate.yml` will check that every condition names
|
||||
a real refId, every policy routes to a real receiver, and no literal webhook
|
||||
URL is committed.
|
||||
4. **Merging is not deploying.** Ship it by hand per
|
||||
`observability/README.md` — and remember `docker restart` won't pick up a new
|
||||
environment variable.
|
||||
5. Verify by forcing a real message into `#ops-alerts`. The API redacts the
|
||||
webhook, so arrival is the only proof.
|
||||
|
||||
---
|
||||
|
||||
## Debug: "the site is serving stale data"
|
||||
|
||||
Work the token chain, in this order:
|
||||
|
||||
1. `GET /api/version` on the affected environment — is `payload_ver` what you
|
||||
expect? If a payload shape changed and this didn't move, that's your answer.
|
||||
2. Is the derived-store token advancing? It's built from `PAYLOAD_VER` plus the
|
||||
history end date (`history_token`), so a cell whose history stopped updating
|
||||
pins its token.
|
||||
3. Is the *history* updating? Check `climate_sync` freshness
|
||||
(`infra/ops/dbq.sh prod …`) and whether upstream is in a rate-limit cooldown
|
||||
(the errors log, tagged `retry`/`error`).
|
||||
4. Is it the **frontend's** TTL cache instead? `THERMOGRAPH_SSR_CACHE_TTL`
|
||||
defaults to 600s, in front of the backend's own caching.
|
||||
5. Is it the **browser**? `cache.js` seeds IndexedDB from the `/cell` bundle. If
|
||||
ETags stopped being visible cross-origin (`expose_headers`), revalidation
|
||||
silently stops.
|
||||
|
||||
---
|
||||
|
||||
## Debug: "a deploy went green but nothing changed"
|
||||
|
||||
Common causes, most likely first:
|
||||
|
||||
1. **The tree was already identical.** A promotion between branches whose trees
|
||||
match merges nothing while still firing the deploy workflows.
|
||||
2. **The domain wasn't touched.** `deploy.yml` re-checks whether *this* push
|
||||
touched *this* domain, and skips the leg if not.
|
||||
3. **The tag is domain-keyed.** `deployed_version` showing a tag that isn't the
|
||||
branch head is correct behaviour, not drift.
|
||||
4. **Someone hand-edited `/opt/thermograph`.** `deploy.sh` runs
|
||||
`git reset --hard` first; the edit is gone.
|
||||
5. **It's an observability change.** `/opt/observability` points at the archived
|
||||
repo and cannot pull. See above.
|
||||
|
||||
---
|
||||
|
||||
## Promote to beta, and then to prod
|
||||
|
||||
```
|
||||
dev → main your call, batched. Beta is the test environment;
|
||||
not promoting is not testing.
|
||||
main → release ALWAYS the owner's call. Prepare it, state what would ship,
|
||||
the CI evidence and how long it soaked on beta, then recommend —
|
||||
including recommending against.
|
||||
```
|
||||
|
||||
Centralis's `promote(from, to)` does the mechanics and enforces the rule;
|
||||
`forge_pr_merge` refuses `main → release` under any flag. Before promoting,
|
||||
compare the two tips' **trees**, not their commit counts.
|
||||
|
||||
---
|
||||
|
||||
## Roll back
|
||||
|
||||
Redeploying the previous image tag:
|
||||
|
||||
```bash
|
||||
SERVICE=backend BACKEND_IMAGE_TAG=sha-<previous12hex> \
|
||||
/opt/thermograph/infra/deploy/deploy.sh
|
||||
```
|
||||
|
||||
**Check the tag actually exists first.** Both deploy scripts *end* by deleting
|
||||
images outside the running pair, so beta typically holds no local rollback
|
||||
target at all. `rollback_to(dry_run: true)` checks host and registry and will
|
||||
tell you when it's in neither.
|
||||
|
||||
---
|
||||
|
||||
## Regenerate the bundled city data
|
||||
|
||||
`backend/cities.json` and `cities_flavor.json` are generated reference data, not
|
||||
runtime state:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python gen_cities.py # the curated city set
|
||||
python gen_flavor.py # the editorial blurbs
|
||||
```
|
||||
|
||||
Editorial copy has its own voice guide — the **`city-copy`** skill defines what
|
||||
good reads like, and **`backfill-city-copy`** drives generation at scale. Both
|
||||
enforce the relative-grades rule.
|
||||
|
||||
After a deploy, `warm_cities.py` pre-warms those cells so `/climate` pages serve
|
||||
from cache and a crawl never bursts the archive quota. It's idempotent, paced,
|
||||
and claims a flock — deploys launch it detached with no overlap check, so two
|
||||
overlapping runs would otherwise double-spend quota and race the cache writes.
|
||||
|
||||
---
|
||||
|
||||
## Work on a live host
|
||||
|
||||
Prefer Centralis: `run_on_host`, `fleet_status`, `deployed_version`,
|
||||
`logs_query`, `sql_query`.
|
||||
|
||||
You may, without asking: read anything; restart or roll a wedged service; clear
|
||||
a full disk; kill a runaway process; bounce a crash-looping container; run the
|
||||
blessed deploy path onto an already-published image.
|
||||
|
||||
You must not: hand-edit the checkout to persist a change (`git reset --hard`
|
||||
eats it); hand-edit `/etc/thermograph.env` (rendered artifact); force-push;
|
||||
deploy prod from an unmerged branch; `docker system prune -a` on a live box (it
|
||||
deletes your rollback image — `docker image prune -f` is the safe form); or
|
||||
restart the `_db` service casually. Postgres/Timescale restarts and volume
|
||||
operations get confirmed with the operator first.
|
||||
|
||||
Next: [Traps and stale docs](11-traps.md).
|
||||
268
docs/onboarding/11-traps.md
Normal file
268
docs/onboarding/11-traps.md
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
# 11. Traps and stale docs
|
||||
|
||||
Two kinds of thing here: **live traps** (correct code that will surprise you)
|
||||
and **stale docs** (files in this repo that currently describe a system that no
|
||||
longer exists). The second list matters as much as the first — this repo has a
|
||||
lot of documentation, most of it excellent, and some of it describing the world
|
||||
before a rewrite.
|
||||
|
||||
Everything below was checked against the tree at the time of writing.
|
||||
|
||||
---
|
||||
|
||||
## Stale docs in this repo
|
||||
|
||||
### `frontend/CLAUDE.md` and `frontend/README.md` describe a service that is not deployed
|
||||
|
||||
**The live frontend is Go** (`frontend/server/`, module `thermograph/frontend`).
|
||||
Both documents describe the Python FastAPI/Jinja implementation
|
||||
(`frontend/app.py`, `content.py`, `api_client.py`, `format.py`,
|
||||
`templates/*.html.j2`), which was superseded by the Go port and is kept in-tree
|
||||
as the reference the port was made from.
|
||||
|
||||
What is still **correct** in them: every contract (API-version pinning,
|
||||
`pctOrd`, the `/cell` + ETag relationship, the Fahrenheit set, boot-must-survive-
|
||||
an-unreachable-backend), every environment variable name and default, and all of
|
||||
`DESIGN.md`.
|
||||
|
||||
What is **wrong**: the language, the file paths, and — notably —
|
||||
`frontend/CLAUDE.md`'s claim that `make test-unit` is *"the tier CI runs."* It
|
||||
isn't any more. CI builds the frontend image, and `frontend/Dockerfile`'s
|
||||
builder stage runs `gofmt -l` + `go vet` + `go test ./...` before compiling, so
|
||||
a failing **Go** test is what fails CI. The Python tiers are local-only now.
|
||||
|
||||
Also note `api_client.py` carries a **third** copy of the `API_VERSION` pin.
|
||||
It isn't deployed, but keep it in step or you'll debug a phantom locally.
|
||||
|
||||
### `infra/DEPLOY.md` describes the pre-Swarm, pre-monorepo deploy
|
||||
|
||||
Three separate layers of drift:
|
||||
|
||||
- It says **prod is deployed with `terraform apply`**. Prod is deployed by a
|
||||
push to `release` firing `deploy.yml`, exactly like beta's `main` path — see
|
||||
[07](07-ci-and-release.md).
|
||||
- It says prod runs a **compose** stack. Prod runs **Swarm**
|
||||
(`infra/deploy/stack/`).
|
||||
- Its file table points at `/opt/thermograph/deploy/deploy.sh` and
|
||||
`/opt/thermograph/docker-compose.yml`. Both moved under `infra/` at the
|
||||
monorepo cutover.
|
||||
|
||||
The document's own header flags the legacy `provision.sh` / systemd / venv
|
||||
walkthrough as superseded — but the drift goes further than that note admits.
|
||||
The **key-management** sections (Part 1) and the `/etc/hosts` registry note are
|
||||
still accurate and useful.
|
||||
|
||||
### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows
|
||||
|
||||
Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist. The two
|
||||
LAN-dev deploy workflows were deleted rather than ported at the CI
|
||||
consolidation — they were documented as inert (they targeted a monorepo layout
|
||||
at `~/thermograph-dev` on a box still holding a split-era checkout). **LAN dev
|
||||
is a local `make dev-up` concern, not a CI environment.**
|
||||
|
||||
`ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the
|
||||
workflow set; there are now nine files and the deploy/build ones were collapsed.
|
||||
|
||||
Everything else in `ACCESS.md` — the host table, the agent-access model, the key
|
||||
rotation procedure, the Swarm/Forgejo tracks — is current and worth reading.
|
||||
|
||||
### `backend/README.md` is written for the split-repo era
|
||||
|
||||
It describes `thermograph-backend` as its own repo with sibling repos, uses the
|
||||
retired image path `git.thermograph.org/emi/thermograph-backend/app`, and says
|
||||
*"there is no Makefile in this repo yet"* — there is (`make test`, `make smoke`,
|
||||
`make clean-venv`).
|
||||
|
||||
The **"How it works"** section (grid → data → percentiles → caching) is one of
|
||||
the best explanations of the domain model anywhere in the repo. Read it; just
|
||||
mentally translate the topology.
|
||||
|
||||
### `CUTOVER-NOTES.md` carries a superseded block, correctly labelled
|
||||
|
||||
Its per-domain workflow names (`backend-build-push.yml`, `backend-deploy.yml`, …)
|
||||
no longer exist — the six deploy workflows collapsed into `deploy.yml` and the
|
||||
two build-push workflows into `build-push.yml`. The file says so in an explicit
|
||||
"Superseded 2026-07-25" note, and everything it describes about *behaviour* is
|
||||
unchanged.
|
||||
|
||||
It also references `backend/MONOREPO-CUTOVER-PLAN.md`, which is not in the tree.
|
||||
|
||||
`CUTOVER-NOTES.md` remains the source of truth for what is and isn't live.
|
||||
|
||||
### The domain `CLAUDE.md` files still use repo-era wording
|
||||
|
||||
They say "this repo" and link to sibling repo names. Mentally map
|
||||
`thermograph-backend` → `backend/`, and fix the wording as you touch files near
|
||||
it — that's the standing instruction in the root `CLAUDE.md`.
|
||||
|
||||
### `observability/README.md`: merging is not deploying
|
||||
|
||||
`/opt/observability` on beta and prod is a checkout of the **archived**
|
||||
`emi/thermograph-observability` repo. It can never `git pull` again. A previous
|
||||
observability PR merged and had **zero live effect** for exactly this reason.
|
||||
The README documents the hand-ship procedure; until the checkouts are re-pointed
|
||||
at the monorepo, that's the only path. Flagged again in
|
||||
[09](09-observability.md).
|
||||
|
||||
---
|
||||
|
||||
## Live traps
|
||||
|
||||
### The `docker-compose.test.yml` files still pull retired image paths
|
||||
|
||||
Both `backend/docker-compose.test.yml` and `frontend/docker-compose.test.yml`
|
||||
default `BACKEND_IMAGE_PATH` to **`emi/thermograph-backend/app`** — the
|
||||
split-era path. The current image is `emi/thermograph/backend`.
|
||||
|
||||
- `backend/make smoke` is fine: it builds `:local` and never pulls.
|
||||
- `frontend/make backend-up` **does** pull, and pins
|
||||
`THERMOGRAPH_BACKEND_TEST_TAG` to `v0.0.2-split-ci`. The old packages were
|
||||
deliberately left in the registry for rollback and GC'd only after burn-in, so
|
||||
this may work, may serve a very old backend, or may 404 depending on when you
|
||||
read this. If `make backend-up` fails or behaves oddly, set the path and tag
|
||||
explicitly:
|
||||
```bash
|
||||
BACKEND_IMAGE_PATH=emi/thermograph/backend \
|
||||
THERMOGRAPH_BACKEND_TEST_TAG=sha-<12hex> make backend-up
|
||||
```
|
||||
|
||||
### `static/units.js`'s `F_REGIONS` is guarded by nothing
|
||||
|
||||
Source comments in three files claim *"a test asserts all three stay
|
||||
identical."* The only automated assertion is
|
||||
`format_test.go::TestFCountriesMatchesBackend`, which re-parses the backend's
|
||||
Python and compares the **Go** copy. Nothing reads `units.js`.
|
||||
|
||||
All four copies currently *are* identical (14 codes, checked). But the browser
|
||||
copy is held in step by convention alone. Extending the Go test to parse
|
||||
`units.js` too is a good first contribution.
|
||||
|
||||
### The compose project name is load-bearing
|
||||
|
||||
`infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose
|
||||
from `infra/` derives project `infra` — a silently **new** stack with empty
|
||||
volumes beside the running one. `deploy-dev.sh` exports
|
||||
`COMPOSE_PROJECT_NAME=thermograph-dev` to keep LAN dev separate on purpose.
|
||||
Keep both halves.
|
||||
|
||||
### `deploy.sh` hard-resets the host checkout
|
||||
|
||||
`git reset --hard origin/$BRANCH` runs before anything else. Uncommitted edits
|
||||
in `/opt/thermograph` evaporate. `infra-sync.yml` does the same reset without
|
||||
deploying. The one deliberate exception is the untracked
|
||||
`deploy/.image-tags.env` — losing *that* makes the next single-service deploy
|
||||
roll the sibling onto `local`.
|
||||
|
||||
### The deploy scripts eat your rollback image
|
||||
|
||||
Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the
|
||||
running pair. On beta that succeeds, so beta typically holds **no local rollback
|
||||
target at all**; on prod it usually fails only because Swarm's stopped task
|
||||
containers still hold references. `docker system prune -a` is forbidden on a
|
||||
live box for the same reason — but the deploy script is the thing that actually
|
||||
eats the image, not prune. Verify a tag exists before promising a rollback.
|
||||
|
||||
### `daemon` and `lake` are never deploy targets on their own
|
||||
|
||||
They ride with `backend`. All three run the same image at the same tag — a
|
||||
second binary and a second role inside one image — and the `/internal/*`
|
||||
contract between daemon and web has no version negotiation. There's also a
|
||||
mechanical reason: a single-service deploy runs `up -d --no-deps <targets>`, so
|
||||
a service in compose but absent from `TARGETS` would simply never be created.
|
||||
|
||||
### `docker compose config --images <service>` does not filter
|
||||
|
||||
Confirmed live on Compose v5.3.1: it prints **every** service's image, one per
|
||||
line, in file order. A `| head -1` therefore grabbed `db`'s image, and
|
||||
`deploy.sh`'s daemon capability probe found no daemon binary in a Postgres image
|
||||
and dropped the daemon from *every* backend deploy. The script now builds the
|
||||
image reference from the same variables compose interpolates.
|
||||
|
||||
### Prod's Loki streams carry no `service` label
|
||||
|
||||
Prod runs Swarm; its `job="docker"` streams have no `service` label. A raw
|
||||
`{host="prod", service="backend"}` query matches nothing and reads as "no logs"
|
||||
when it means "wrong selector". Use `logs_query`'s `service` argument. Prod
|
||||
container names also change on every redeploy (the task suffix) — resolve via
|
||||
`docker ps --filter name=` at call time, never hardcode.
|
||||
|
||||
### A Grafana alert rule can provision cleanly and never fire
|
||||
|
||||
A rule whose `condition` names a non-existent refId is valid YAML, provisions
|
||||
without complaint, and pages nobody. Likewise a contact point holding the
|
||||
literal string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the
|
||||
API — and the API **redacts** the URL, so you cannot confirm interpolation by
|
||||
reading it back. The only proof is a message arriving in `#ops-alerts`.
|
||||
|
||||
### Terraform has no state anywhere
|
||||
|
||||
A casual `terraform apply` would attempt full re-provisioning of live hosts.
|
||||
It's executable documentation until state is bootstrapped.
|
||||
|
||||
### The registry is mesh-only
|
||||
|
||||
`git.thermograph.org` resolves publicly to beta's IP, but beta's Caddy rejects
|
||||
`/v2/*` from outside the WireGuard mesh. Any host that pulls needs
|
||||
`10.10.0.2 git.thermograph.org` in `/etc/hosts`. Beta itself doesn't — it *is*
|
||||
the box.
|
||||
|
||||
### Forgejo runners only have the labels they registered with
|
||||
|
||||
GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does not.
|
||||
`runs-on: [self-hosted, thermograph-lan]` is permanently unschedulable there —
|
||||
a real bug, found and fixed once. If a job silently stops appearing in the
|
||||
Actions history, check this first.
|
||||
|
||||
### Forgejo's `needs:` doesn't cross workflow files
|
||||
|
||||
`build-push.yml` and `deploy.yml` fire from the same push with no ordering
|
||||
guarantee, and a deploy racing ahead of its build and failing "not found" is
|
||||
confirmed-live. That's why `deploy.sh`'s pull has a bounded ~5-minute retry.
|
||||
|
||||
### `/api/v2/cell?prefetch=1` must never spend quota
|
||||
|
||||
The hard guarantee: a cold cell answers `204` with no body. Anything named
|
||||
"prefetch" or "warm" calls the cache-only loaders. Breaking this gets the whole
|
||||
service rate-limited.
|
||||
|
||||
### Background timers must go through leader election
|
||||
|
||||
`singleton.claim_leader()` — flock per host, Postgres advisory lock per cluster.
|
||||
A timer-driven sweep running in N uvicorn workers × M Swarm hosts multiplies
|
||||
Open-Meteo quota use N×M. That is the documented cause of overnight 429/503s
|
||||
after prod went to 3 workers. The heartbeat is the deliberate exception — a beat
|
||||
is one local file append, so duplicates are free.
|
||||
|
||||
### `THERMOGRAPH_DATA_DIR` exists because of a real collision
|
||||
|
||||
The backend's Python package `data/` and its runtime data dir `<root>/data`
|
||||
collide at the same path once the code lives at the image root. Mounting a
|
||||
volume there shadows and erases `data/*.py`, so `import data.climate` fails at
|
||||
boot. Point the env var **outside the code tree** in a container. And never
|
||||
reintroduce `__file__`-relative paths in a module — `paths.py` resolves
|
||||
everything from the repo root for this reason.
|
||||
|
||||
### Alloy can silently stop tailing a container
|
||||
|
||||
Observed on prod, 13½ hours of `thermograph_worker` logs never reaching Loki
|
||||
while the container was healthy and writing. Not root-caused. Workaround:
|
||||
`sudo docker restart alloy-alloy-1`. `ProdWorkerContainerSilent` now catches the
|
||||
class.
|
||||
|
||||
### Nominatim is ~1 req/s, on one thread
|
||||
|
||||
All reverse geocoding funnels through a single dedicated worker draining a
|
||||
queue. The earlier design slept inside the caller's thread — which in the server
|
||||
is one of Starlette's shared sync-threadpool threads.
|
||||
|
||||
---
|
||||
|
||||
## If you fix one of these
|
||||
|
||||
Docs are cheap to correct and the root `CLAUDE.md` already asks you to fix
|
||||
repo-era wording as you touch files near it. If you're in `frontend/` anyway,
|
||||
bringing `frontend/CLAUDE.md` in line with the Go service would be the single
|
||||
highest-value documentation change available — it's the file an agent reads
|
||||
*before* touching that domain.
|
||||
|
||||
Back to the [index](README.md).
|
||||
70
docs/onboarding/README.md
Normal file
70
docs/onboarding/README.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# Onboarding — becoming a full contributor to Thermograph
|
||||
|
||||
This is the developer onboarding path for the `emi/thermograph` monorepo: what
|
||||
the product is, how the code is shaped, how to run it, what will break silently
|
||||
if you get it wrong, and how a change actually travels from your editor to
|
||||
`thermograph.org`.
|
||||
|
||||
**Scope.** This covers *working in this repo*. Cross-cutting architecture
|
||||
decision records and operator runbooks live in the separate `thermograph-docs`
|
||||
repo (reachable through Centralis's `docs_search`) — this set links out rather
|
||||
than duplicating them. Where a fact is owned by a `CLAUDE.md`, that file stays
|
||||
the source of truth and this set explains the context around it.
|
||||
|
||||
## Reading order
|
||||
|
||||
| # | Doc | Read it when |
|
||||
|---|-----|--------------|
|
||||
| 1 | [Orientation](01-orientation.md) | First. What the product actually claims, the estate, the four non-negotiable rules. |
|
||||
| 2 | [Local setup](02-setup.md) | Before you touch anything. Toolchain, verified run/test recipes for every service. |
|
||||
| 3 | [Repo map](03-repo-map.md) | To find things. Every domain and directory, and which files matter. |
|
||||
| 4 | [Backend deep dive](04-backend.md) | Before your first backend change. Data pipeline, grading, caching, roles, daemon. |
|
||||
| 5 | [Frontend deep dive](05-frontend.md) | Before your first frontend change. The Go SSR service, static assets, design system. |
|
||||
| 6 | [Cross-service contracts](06-contracts.md) | **Before any change that touches both.** These break silently and in production. |
|
||||
| 7 | [CI and release](07-ci-and-release.md) | Before you open a PR. Nine workflows, three branches, three environments. |
|
||||
| 8 | [Infra and secrets](08-infra-secrets.md) | Before you touch deploy, compose, or a secret. |
|
||||
| 9 | [Observability](09-observability.md) | When something is wrong and you need to see it. |
|
||||
| 10 | [Recipes](10-recipes.md) | Task-shaped walkthroughs for the things you'll actually do. |
|
||||
| 11 | [Traps and stale docs](11-traps.md) | **Skim early, re-read often.** Which docs in this repo currently lie, and why. |
|
||||
|
||||
## The short version
|
||||
|
||||
Thermograph grades how unusual today's weather is at any point on Earth against
|
||||
~45 years of that exact location's own history. Percentiles, never thermometer
|
||||
readings.
|
||||
|
||||
- **`backend/`** — Python 3.12 / FastAPI. Owns all climate data, grading, the
|
||||
API, accounts, notifications, plus a Go daemon (`backend/daemon/`) that owns
|
||||
the Discord gateway and recurring timers. Ships as `emi/thermograph/backend`.
|
||||
- **`frontend/`** — **Go** SSR service (`frontend/server/`) plus every static
|
||||
asset. No climate data, no database, no compute. Ships as
|
||||
`emi/thermograph/frontend`. (The Python implementation at `frontend/*.py` is
|
||||
the superseded original — see [traps](11-traps.md).)
|
||||
- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets
|
||||
vault, Terraform, ops query tooling.
|
||||
- **`observability/`** — Loki + Grafana on beta, an Alloy agent per node.
|
||||
|
||||
Branches stage environments: PR → `dev` (LAN dev) → `main` (beta) → `release`
|
||||
(prod). The two app domains build and deploy **independently** — that
|
||||
independence is the whole reason the split-then-reunify history exists, and
|
||||
[contracts](06-contracts.md) is the list of things that keep it safe.
|
||||
|
||||
## Your first day
|
||||
|
||||
1. Read [Orientation](01-orientation.md) and [Traps](11-traps.md).
|
||||
2. Work through [Local setup](02-setup.md) until `make test` is green in both
|
||||
`backend/` and `frontend/server/`.
|
||||
3. Read [Repo map](03-repo-map.md) with the tree open beside it.
|
||||
4. Pick something small in the domain you're least afraid of, and follow
|
||||
[Recipes](10-recipes.md) end to end — including opening the PR.
|
||||
|
||||
## Your first week
|
||||
|
||||
- Read [Contracts](06-contracts.md) properly. Every item on that list has
|
||||
already cost someone a production incident somewhere in this project's
|
||||
history; that's why each one is written down.
|
||||
- Get Centralis wired up (see [Local setup](02-setup.md)) and ask it
|
||||
`fleet_status`, `logs_overview`, `deployed_version`. You cannot debug this
|
||||
estate blind, and the boxes are not reachable from a laptop off the
|
||||
WireGuard mesh.
|
||||
- Read the `CLAUDE.md` for each domain. They are terse, current, and binding.
|
||||
Loading…
Reference in a new issue