Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.
- data/climate_store.py: psycopg + polars bridge over climate_history
(a hypertable), climate_recent, and climate_sync (per-cell freshness).
Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
stale-serve path still avoids bumping it, so derived-payload tokens
invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
(compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
parquet cache into the hypertables, preserving file mtimes as sync
timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
tuning GUC. Docs updated for the new backend and cutover.
Co-authored-by: Claude <noreply@anthropic.com>
Two prod-readiness hardening changes:
DB tuning scales with the container budget. Replace the fixed 8 GB
20-tuning.sql with 20-tuning.sh, which derives shared_buffers (25%),
effective_cache_size (75%), work_mem, maintenance_work_mem and
duckdb.max_memory (50%) from the DB_MEMORY the compose db service now passes in.
The ratios reproduce the historical 8 GB tuning exactly and scale linearly, so
the 48 GB prod box (db_memory 16g) gets shared_buffers 4 GB / duckdb 8 GB with
no separate edit. Beta/local (8g default) are unchanged. Docs that told
operators to raise the tuning by hand are updated.
Boot ordering for the self-hosted archive. On an openmeteo host, install a
docker.service drop-in (Wants/After rclone-om.service) so Docker starts after
the object-storage mount is ready on every boot — the restart-policy containers
never bind an empty mount point. rclone-om is Type=notify, so After waits for
the mount to actually be ready. Cleaned up when openmeteo is toggled off.
Get the 45-year historical record off the rate-limited public Open-Meteo
archive API by running a private Open-Meteo instance that serves the
era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the
compressed .om archive in object storage, mounted on the host with rclone.
- climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin
models=era5_seamless on the archive fetches only, so a self-hosted instance
serves the same 0.1° resolution; forecast path unchanged. The public API's
default is already seamless, so dev/beta (URL unset) behave identically.
- docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers
(era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage
mount; the overlay points the app at the local instance.
- Makefile: om-up / om-down / om-backfill (one-time full-history backfill).
- Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR,
and provisions the host rclone systemd mount from the bucket credentials.
- deploy/openmeteo: operator runbook + rclone mount unit template.
Terraform config under terraform/ manages the two existing VPS hosts and hands the
app to docker-compose, with local state:
- prod: the new 48GB/12-core VPS (release branch, thermograph.org), sized larger.
- beta: the old VPS 75.119.132.91 (main branch, testing tier), no public domain.
- The LAN dev box stays on deploy/deploy-dev.sh (dev branch) — out of Terraform.
A reusable module (modules/thermograph-host) SSHes each host to install docker/
compose/ufw (+ Caddy when a domain is set), sync the checkout to the host's branch,
render /etc/thermograph.env from Terraform variables (secrets pushed via provisioner
content, never on local disk), `docker compose up -d`, and health-check. Named
volumes are preserved on re-apply, so the Postgres data is never recreated.
Container resources are now env-driven in docker-compose.yml (APP_CPUS/DB_CPUS/
DB_MEMORY/WORKERS) with unchanged defaults, so Terraform can size each host.
