63e1967bbb
The SELECT guard was a denylist of DDL/DML verbs. DuckDB's file readers and
setting introspection are plain SELECTs, so it passed all of these:
SELECT * FROM read_csv_auto('/etc/passwd') -> 200, file contents
SELECT * FROM glob('/etc/*') -> 200, directory listing
SELECT current_setting('s3_secret_access_key') -> 200, the bucket credentials
The last one is the worst: bucket mode installed the S3 credentials with
SET s3_secret_access_key, and settings are readable back out. So one
unauthenticated POST returned the ERA5 bucket key in plaintext -- precisely what
the module docstring says this service exists to avoid ("without shipping S3
credentials to every web task"). /query had no authentication of any kind, while
the comparable /internal/* router has gated its whole surface since it landed.
Three changes, outermost first:
- /query now requires the shared internal token, reusing internal_routes'
dependency. Nothing calls it: the only in-repo caller of this service is
era5lake.py hitting /history, and Centralis reaches the lake through its own
DuckDB rather than this endpoint. /history is deliberately left open for now --
gating it means threading the token through the web->lake hot path.
- Credentials are installed as a DuckDB SECRET instead of SET s3_*. Secret values
are opaque: current_setting() returns NULL and duckdb_secrets() lists the name
and scope but no key material.
- Bucket mode disables LocalFileSystem after the views are created (they resolve
lazily, so ordering matters) and locks the configuration. Local mode cannot do
this -- its views read local parquet -- so the function denylist stays as
defence in depth for dev and tests.
The new guard tests assert the *guard* refused, not merely that the request
failed: several of these shapes (read_parquet on a text file, a bare
'/etc/passwd') error inside DuckDB anyway, so a status-only assertion would keep
passing with the guard deleted. That is the trap the original four-case test fell
into -- it tested negatives shaped like the regex that had been written.
|
||
|---|---|---|
| .. | ||
| accounts | ||
| alembic | ||
| api | ||
| core | ||
| daemon | ||
| data | ||
| deploy | ||
| notifications | ||
| scripts | ||
| tests | ||
| web | ||
| .dockerignore | ||
| .gitignore | ||
| alembic.ini | ||
| app.py | ||
| cities.json | ||
| cities_flavor.json | ||
| CLAUDE.md | ||
| docker-compose.test.yml | ||
| Dockerfile | ||
| drift_check.py | ||
| gen_cities.py | ||
| gen_era5_lake.py | ||
| gen_flavor.py | ||
| indexnow.py | ||
| lake_app.py | ||
| Makefile | ||
| migrate.py | ||
| migrate_accounts_to_pg.py | ||
| migrate_cache_to_pg.py | ||
| paths.py | ||
| README.md | ||
| requirements-dev.txt | ||
| requirements-seed.txt | ||
| requirements.txt | ||
| seed_era5.py | ||
| warm_cities.py | ||
thermograph-backend
The Thermograph API service: grades recent local weather against ~45 years of
climate history, and hosts the accounts, notification, and SSR-content
back-end that the rest of the split Thermograph stack (thermograph-frontend)
talks to over HTTP. Split from the emi/thermograph monorepo — this repo owns
the API/DB/accounts/notifications layer only; it renders no HTML/CSS/JS of its
own.
See CLAUDE.md for the full split topology, deploy flow, and
API version contract, and thermograph-docs (a sibling
repo) for cross-cutting architecture decisions and operator runbooks.
How it works
- Grid — a lat/lon is snapped to a stable ~4 sq mi cell
(
data/grid.py); longitude spacing is scaled bycos(latitude)so cells stay roughly square at any latitude. The cell id is the cache key, so the same spot always resolves to the same data. - Data (on-demand, cached to parquet) — the first request for a cell
fetches the full 1980–present daily record (max/min temp, precip) from
the free Open-Meteo archive (ERA5) and writes it
to
data/cache/<cell_id>.parquet(zstd, ~200 KB for 45 years). Later requests read the parquet directly. Recent days come from Open-Meteo's forecast API (past_days), so history and grading share one source. - Percentiles & grading (
data/grading.py) — each day is graded against every historical day within ±7 days of it (a 15-day seasonal window, wrapping year-end), as an empirical mid-rank percentile. Temperature uses a symmetric tier ladder (TEMP_BANDS: Near Record / High / Above Normal / Normal / Below Normal / Low / Near Record); precipitation is graded separately (RAIN_BANDS) since most days are dry — a rainy day is ranked only among rain days in its window, dry days are colored by dry-streak length instead. - Caching & ETags — every derived payload (grade/calendar/day/SSR
content) is cached in SQLite (
data/store.py) under(kind, cell_id, key), validated by a token that only advances when the cell's history actually changes. That token doubles as a weak ETag, so an unchanged request costs a 304 with no payload rebuild (api/payloads.py,web/app.py).
Layout
accounts/ fastapi-users models/schemas/db + api_accounts routes
alembic/ Postgres schema migrations (alembic upgrade head on boot)
api/ versioned payload builders (payloads.py, content_payloads.py)
+ route wiring (content_routes.py, sitemap.py, homepage.py)
core/ metrics, audit/access logging, a singleton helper
data/ grid snapping, climate fetch/cache, grading/scoring,
places/cities, the derived-payload store
notifications/ push (VAPID), email, Discord bot (interactions + linking),
monthly digest, the in-process scheduler (city warming, IndexNow)
web/app.py the FastAPI app (routes, CORS, ETag/versioning, middleware)
deploy/ container entrypoint.sh (alembic migrate, then serve)
app.py shim re-exporting web.app:app — keeps the launch target
`app:app` stable regardless of internal package layout
scripts/ one-off admin scripts (Discord slash-command registration)
tests/ pytest suite, hermetic (see tests/conftest.py)
cities.json,
cities_flavor.json bundled reference data (generated by gen_cities.py /
gen_flavor.py), not runtime state
How it fits the split
thermograph-frontendcalls this service'sGET /api/v2/...endpoints (grade, geocode, calendar) and the SSR content endpoints under/content/...; it negotiates compatibility viaGET /api/version.thermograph-infraowns the deploy/Compose/Terraform layer — this repo only builds and publishes its own container image (git.thermograph.org/emi/thermograph-backend/app) and hands infra a tag to roll out (SERVICE=backend+BACKEND_IMAGE_TAGinto infra'sdeploy/deploy.sh).thermograph-docsholds the cross-repo architecture/runbook docs; this README only covers what's local to this service.
Build & run
docker build -t thermograph-backend .
docker run -p 8137:8137 --env-file .env thermograph-backend
The image runs deploy/entrypoint.sh: alembic upgrade head against
THERMOGRAPH_DATABASE_URL (retried, since a fresh Postgres volume can still be
starting up), then uvicorn app:app on $PORT (default 8137) with
$WORKERS workers (default 4). /healthz is an I/O-free liveness probe.
For local development without a container, see the "Run / test locally"
section of CLAUDE.md — there is no Makefile in this repo yet,
so it's a plain venv + pytest/uvicorn invocation.
Notifications: Discord slash commands
notifications/discord_interactions.py answers Discord's HTTP Interactions
endpoint (/discord/interactions) for the /grade <city> slash command.
Registering (or updating) the command definition with Discord's REST API is a
one-off admin action, not part of the running app:
THERMOGRAPH_DISCORD_APP_ID=... THERMOGRAPH_DISCORD_BOT_TOKEN=... \
python3 scripts/register_discord_commands.py
Global command changes can take up to an hour to propagate.