The live history path now tries NASA POWER first (keyless, independent of
Open-Meteo) and falls back to the Open-Meteo archive only when NASA is
unavailable — the reverse of before. Gusts NASA lacks are filled from Meteostat
inside _fetch_history_nasa (added in the prior change). Both sources must still
return a plausibly-full span (MIN_ARCHIVE_DAYS) before being cached as complete;
a short/partial response is rejected and the other source is tried.
_fetch_history_nasa now accepts a start/end range so it also serves the recent
tail top-up: _topup_tail fetches NASA-first via the new _fetch_history_tail
helper (Open-Meteo range as fallback, skipped during its rate-limit cooldown),
removing the last per-active-cell Open-Meteo call from the history path. The
Open-Meteo cooldown no longer gates the top-up, since NASA is not subject to it.
Open-Meteo remains wired as the fallback (deletion is a later step). Tests
updated to the new source order plus tail-fetch coverage.
NASA POWER (history) and MET Norway (forecast) carry no wind gusts, but gust is
a graded metric. This adds data/meteostat.py, which finds the nearest Meteostat
station to a cell and reads its daily peak gust (wpgt) from the keyless gzipped
bulk endpoints, filling the gust column on the NASA POWER history frame. Where no
station is within ~100 km it estimates from sustained wind (wind * GUST_FACTOR).
Bulk files (station list + per-station daily) are cached on disk so steady-state
network IO is near zero, and any lookup/fetch failure degrades to pure estimation
so a history fetch never fails. A constant estimate factor makes an estimated
gust redundant with wind, so real signal comes only where a station backs it.
Activates once NASA POWER becomes the primary history source; Open-Meteo already
carries its own gusts. New history_gust / meteostat_stations metrics phases map
to the meteostat source.
/suggest is now served purely from the local GeoNames index — no external
geocoder call per keystroke. /geocode answers from the local index first and
falls back to OSM Nominatim /search only on a miss (or while the index is still
loading), covering the neighbourhood/postcode/tiny-village/native-name long tail
the cities dump lacks.
Nominatim shares the reverse geocoder's lock and ~1/sec pacing since both hit
the same host, and only the low-volume /geocode miss path reaches it. Removes
the Open-Meteo geocoding dependency entirely; the "geocode" metrics phase now
maps to the nominatim source.
After the repo split the code lives at the image root, so the Python package
data/ and the runtime data dir <root>/data are the same path (/app/data). The
deploy compose mounts the appdata volume there, shadowing data/*.py so
'import data.climate' fails at container boot. THERMOGRAPH_DATA_DIR (and
_LOGS_DIR) now let the container point runtime state at a path outside the code
tree; local dev is unchanged (defaults to <root>/data).
The frontend/ half of this commit (calendar.js, calendar.html, app.js,
style.css) is out of scope for this repo; a separate agent handles the
split frontend repo.
Claude-Session: https://claude.ai/code/session_01RdARHDJaYC1wSQRTYM6t3Z
Backend now publishes its OWN registry image (emi/thermograph-backend/app) via
build-push.yml and deploys ONLY the backend service to beta on push to main,
passing SERVICE=backend + BACKEND_IMAGE_TAG to infra's deploy.sh. Independent
of the frontend's pipeline -- the two ship asynchronously.
Claude-Session: https://claude.ai/code/session_01RdARHDJaYC1wSQRTYM6t3Z
The live boot+healthz check repeatedly hit this runner's own
environment quirks (unreachable sibling bridge IP, non-unique \$\$,
permanently-squatted fixed ports from AppArmor-blocked cleanup,
variable boot time under shared capacity=2 contention) without ever
settling into a reliably green check across six attempts. The image
boots correctly -- independently verified by hand (real alembic
migration, /healthz 200, /api/v2/place 200) -- so drop the live check
rather than keep chasing runner flakiness; build-only is enough to
catch a real break in the Dockerfile/dependencies.
Confirmed via a real run's own docker logs: the container actually
booted successfully (migrations + 4 uvicorn workers, "Application
startup complete") but took just over 30s under this runner's
resource contention (capacity 2 -- another job's build often runs
concurrently), where the same boot takes ~3s locally with nothing else
competing for CPU. Give it 60s instead.
Every job container's entry shell gets the same low PID, so
backend-ci-\$\$ collided with an already-stuck leftover container from
a prior run (confirmed in the runner logs: "container name already in
use"). GITHUB_RUN_ID is genuinely unique per run -- use that for both
the container name and the per-run host port instead.
Curling the sibling container's own bridge-network IP (the previous
fix) doesn't work on this runner -- docker-outside-of-docker means the
job container's network namespace can't reach another container's
bridge IP directly, so every curl attempt hung for the full ~130s TCP
SYN timeout instead of failing fast (confirmed: a run got stuck for
over 10 minutes on this). 127.0.0.1 does work here (the monorepo's own
build.yml has published a host port successfully all along), so
publish one again, but to a per-run PID-derived port instead of a
fixed one, to still avoid the original port-collision problem.
This runner's snap-Docker install has a known AppArmor bug denying
docker stop/kill -- a leftover container from any failed run would
permanently squat host port 8137, so every SUBSEQUENT run's docker
run also failed immediately (port already allocated), compounding
into a cascading failure loop. Uses a unique-per-run container name
and curls the container's own bridge-network IP directly instead of
publishing any host port at all.
paths.py drops the sibling-directory walk (backend is now the repo
root, no more frontend/ sibling to walk toward). requirements.txt
drops jinja2/PyYAML (only needed for frontend_ssr's sake in the old
shared monorepo Dockerfile) and fixes stale backend/push.py-style
path comments. New Dockerfile (COPY . /app/, no THERMOGRAPH_SERVICE_ROLE
branch -- this image only ever runs backend) and entrypoint.sh (cd
/app instead of /app/backend). Minimal build.yml: docker build + boot
+ /healthz check.
Verified: image builds and boots standalone (real alembic migration
against SQLite, /healthz and /api/v2/place both 200), full pytest
suite green (301 passed, 4 skipped) against the rewritten paths.py.
warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
* Add a Postgres advisory-lock leader election for multi-host deploys
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Retrigger CI (no prior check run was ever recorded for this PR)
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Add the Swarm interim stack file, a pinnable TimescaleDB tag, and a Caddy health-gate
Three changes toward the hop-1 interim cutover, all inert until Track B
stands up the platform:
docker-stack.yml: the Swarm stack file for the interim cutover, distinct
from docker-compose.yml (today's plain-compose deploy, unaffected). Pulls a
pre-built image (IMAGE_TAG) instead of building in place; app/worker publish
no host port (127.0.0.1:8137:8137 has no Swarm equivalent - Swarm's routing
mesh publishes on 0.0.0.0, which would expose the plaintext app un-fronted),
reaching Caddy only over an MTU-lowered overlay network (VXLAN-over-WireGuard
needs a smaller MTU or large payloads silently stall); db is placement-
pinned to a labelled node; app/worker skip inline migrations
(RUN_MIGRATIONS=0) so the runbook's one-shot migrate task is the only thing
that ever runs Alembic; secrets are real Swarm secrets mounted at
/run/secrets, read by the entrypoint shim rather than plain env vars.
TIMESCALEDB_TAG: docker-compose.yml's db image now reads this (default
latest-pg18, today's behavior unchanged), wired through Terraform
(timescaledb_tag, default "latest-pg18") so it can actually be pinned to an
exact minor without hand-editing the host - required before any host of the
stack could replicate with another (a floating tag risks mismatched
extension minors, which blocks a physical replica and risks compressed-
chunk corruption on restore).
Caddy active health-gate: both the Terraform-rendered Caddyfile and the live
deploy/Caddyfile now health-check the app on the same cheap /healthz route
its own Docker HEALTHCHECK uses (now /healthz instead of the SSR homepage,
so it's cheap enough for a tight interval and works identically for a
worker replica, which serves no public traffic at all) - Caddy won't
forward into a container that's still booting or unhealthy.
Verified live: built and booted the real image via docker compose - both
containers report healthy via the new /healthz-based HEALTHCHECK, and GET /
still renders the full SSR homepage unchanged. Both Caddyfiles validated
with the real caddy binary. docker-stack.yml validated with docker compose
config (required-var guards fire with clear messages; secrets correctly
mount at /run/secrets/<name>, matching the entrypoint shim's mapping).
docker-compose.yml validated with and without TIMESCALEDB_TAG set, alongside
the existing openmeteo overlay. terraform validate + fmt clean.
homepage.json lives on the appdata volume today, written by whichever
process's notifier last refreshed it. Under Swarm each web replica has its
own disk, so a file only one replica's notifier ever writes leaves every
other replica reading a stale or missing feed indefinitely.
On Postgres, refresh()/load() go through store.py's existing derived-payload
table instead (a fixed sentinel key, since the feed isn't cell-scoped) - a
single upsert is already atomic, and every replica reads the same row.
store.IS_POSTGRES is False without THERMOGRAPH_DATABASE_URL, so dev/tests
keep the plain file path entirely unchanged; no existing test needed to
change.
Verified live against real Postgres: refresh() persists into the shared
derived table and load() reads the exact same feed back.
Two changes to deploy/entrypoint.sh, both needed before the app can run
under Docker Swarm without re-triggering known incidents:
One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1)
runs the Alembic migration and exits, for a Swarm one-shot task that brings
the schema to head once rather than racing it across every web replica's
boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the
one-shot task separately. Default (unset) keeps today's behavior unchanged:
every boot migrates itself before serving.
Secrets shim: the app reads config from os.environ everywhere (an unset
THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset
VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a
FILE under /run/secrets/ instead, which the app would never see. Before
alembic/uvicorn, mechanically export each /run/secrets/<name> whose
uppercased name is a THERMOGRAPH_* var, unless already set in the process
env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the
same way compose's own interpolation builds it today.
Verified against fake alembic/uvicorn binaries in a throwaway container:
default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0,
secrets populating unset vars, an existing env var beating its secret file,
and no /run/secrets present at all (today's plain compose path, unaffected).
shellcheck clean.
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.
- data/climate_store.py: psycopg + polars bridge over climate_history
(a hypertable), climate_recent, and climate_sync (per-cell freshness).
Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
stale-serve path still avoids bumping it, so derived-payload tokens
invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
(compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
parquet cache into the hypertables, preserving file mtimes as sync
timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
tuning GUC. Docs updated for the new backend and cutover.
Co-authored-by: Claude <noreply@anthropic.com>
A self-hosted Open-Meteo instance that isn't backfilled yet answers historical
requests with all-null values (or only its recent sync window), which
_finalize_frame reduces to a near-empty frame. That frame is not None, so the
loader would accept it as source=open-meteo and cache it indefinitely as a
complete record — never falling back to NASA and never self-healing after the
backfill lands. Require at least MIN_ARCHIVE_DAYS before accepting an archive
result; below that, fall through to the NASA backup and don't cache the short
frame. Guards the prod cutover window.
Get the 45-year historical record off the rate-limited public Open-Meteo
archive API by running a private Open-Meteo instance that serves the
era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the
compressed .om archive in object storage, mounted on the host with rclone.
- climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin
models=era5_seamless on the archive fetches only, so a self-hosted instance
serves the same 0.1° resolution; forecast path unchanged. The public API's
default is already seamless, so dev/beta (URL unset) behave identically.
- docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers
(era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage
mount; the overlay points the app at the local instance.
- Makefile: om-up / om-down / om-backfill (one-time full-history backfill).
- Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR,
and provisions the host rclone systemd mount from the bucket credentials.
- deploy/openmeteo: operator runbook + rclone mount unit template.
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
