Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.
- data/climate_store.py: psycopg + polars bridge over climate_history
(a hypertable), climate_recent, and climate_sync (per-cell freshness).
Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
stale-serve path still avoids bumping it, so derived-payload tokens
invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
(compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
parquet cache into the hypertables, preserving file mtimes as sync
timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
tuning GUC. Docs updated for the new backend and cutover.
Co-authored-by: Claude <noreply@anthropic.com>
A self-hosted Open-Meteo instance that isn't backfilled yet answers historical
requests with all-null values (or only its recent sync window), which
_finalize_frame reduces to a near-empty frame. That frame is not None, so the
loader would accept it as source=open-meteo and cache it indefinitely as a
complete record — never falling back to NASA and never self-healing after the
backfill lands. Require at least MIN_ARCHIVE_DAYS before accepting an archive
result; below that, fall through to the NASA backup and don't cache the short
frame. Guards the prod cutover window.
Get the 45-year historical record off the rate-limited public Open-Meteo
archive API by running a private Open-Meteo instance that serves the
era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the
compressed .om archive in object storage, mounted on the host with rclone.
- climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin
models=era5_seamless on the archive fetches only, so a self-hosted instance
serves the same 0.1° resolution; forecast path unchanged. The public API's
default is already seamless, so dev/beta (URL unset) behave identically.
- docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers
(era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage
mount; the overlay points the app at the local instance.
- Makefile: om-up / om-down / om-backfill (one-time full-history backfill).
- Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR,
and provisions the host rclone systemd mount from the bucket credentials.
- deploy/openmeteo: operator runbook + rclone mount unit template.
* Centralize filesystem paths in a single module
Add paths.py, which resolves the repo root once and derives the cache,
accounts DB, logs, templates, frontend and bundled-city-data locations
from it. Replace the 13 per-module `dirname(__file__)/..` anchors with
references to it, so a module's location no longer determines where the
app reads its data. Env overrides (accounts DB, VAPID, IndexNow) are
unchanged; every resolved path is byte-identical to before.
Groundwork for moving modules into packages without re-pointing paths.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
* Split the backend into domain packages
Group the flat backend modules into packages that mirror their concerns:
data/ climate, grading, scoring, grid, places, cities,
city_events, store
web/ app, views, homepage, content, schemas
notifications/ notify, digest, push, mailer, discord,
discord_interactions, discord_link
accounts/ models, users, api_accounts, db
core/ metrics, singleton, audit
Intra-project imports are rewritten to the package-qualified form. The
entry scripts (indexnow, warm_cities, migrate, gen_cities, gen_flavor)
and paths.py stay at the backend/ root, and backend/app.py becomes a
shim re-exporting web.app:app so the launch target stays `app:app` —
run.sh, the systemd units, and CI need no change.
Verified: full suite (318) passes, `uvicorn app:app` boots and serves
the home/SEO/static/API surfaces, and every root script imports clean.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
