thermograph/infra/docker-compose.openmeteo.yml

73 lines
3.3 KiB
YAML
Raw Normal View History

# Self-hosted Open-Meteo overlay — serves the ERA5 archive locally so the app is
# off the rate-limited public archive API. Enable it only on the self-hosting host:
#
# docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml up -d
#
# The hourly .om data (copernicus_era5_land 0.1° + copernicus_era5 0.25°) lives in
# OBJECT STORAGE, surfaced on the host as a local directory by an rclone FUSE mount
# (a host systemd unit — see deploy/openmeteo/README.md). OM_DATA_DIR points the
# containers at that mount; it defaults to ./data/om-archive so a local smoke test
# works against a plain directory. Object storage is never bind-mounted into the app
# — only Open-Meteo reads it; the app just talks HTTP to open-meteo-api and keeps its
Move the climate record from parquet to TimescaleDB hypertables (#227) Replace the per-cell parquet cache with TimescaleDB hypertables as the production backend for the raw daily climate record, and drop pg_duckdb. Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a Postgres URL (dev, tests, offline tooling), the same dialect switch the accounts DB and derived store already use, so CI stays Postgres-free. - data/climate_store.py: psycopg + polars bridge over climate_history (a hypertable), climate_recent, and climate_sync (per-cell freshness). Reads via pl.read_database, writes via COPY + ON CONFLICT upsert; fail-soft to a cache miss so a DB hiccup degrades to upstream refetch. - data/climate.py: route every cache/mtime touchpoint through a backend dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the stale-serve path still avoids bumping it, so derived-payload tokens invalidate on exactly the same events as before. - alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema (compression policy on year-old chunks), guarded to no-op off Postgres. - migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the parquet cache into the hypertables, preserving file mtimes as sync timestamps so recent_stamp is unchanged across cutover. - db image -> stock timescale/timescaledb:latest-pg18; drop the custom pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb tuning GUC. Docs updated for the new backend and cutover. Co-authored-by: Claude <noreply@anthropic.com>
2026-07-20 20:15:55 +00:00
# own daily-per-cell climate record in the TimescaleDB `db` service.
#
# One-time backfill (writes ~11.5 TB of .om to object storage — run once before the
# app is flipped over): `make om-backfill`.
services:
# Serves /v1/archive on the compose network. No host port: only `backend`
# reaches it, as http://open-meteo-api:8080. Reads .om from the object-storage
# mount, on demand for any point, returning JSON byte-identical to the public
# archive API.
open-meteo-api:
image: ${OM_IMAGE:-ghcr.io/open-meteo/open-meteo}
command: ["serve", "--env", "production", "--hostname", "0.0.0.0", "--port", "8080"]
volumes:
- ${OM_DATA_DIR:-./data/om-archive}:/app/data
restart: unless-stopped
# Rolling keep-current worker for ERA5-Land (0.1°): the surface variables it
# carries. --past-days 14 re-syncs the recent tail every day, appending new days and
# absorbing ERA5T→final corrections. dew_point_2m → relative_humidity and
# shortwave_radiation → apparent_temperature are derived server-side at query time.
open-meteo-sync-land:
image: ${OM_IMAGE:-ghcr.io/open-meteo/open-meteo}
command:
- "sync"
- "copernicus_era5_land"
- "temperature_2m,dew_point_2m,precipitation,shortwave_radiation,wind_u_component_10m,wind_v_component_10m"
- "--past-days"
- "${OM_SYNC_PAST_DAYS:-14}"
- "--repeat-interval"
- "1440"
volumes:
- ${OM_DATA_DIR:-./data/om-archive}:/app/data
restart: unless-stopped
# Rolling keep-current worker for ERA5 (0.25°): wind gusts — absent from ERA5-Land —
# plus the seamless-blend fallback over water/coastline where ERA5-Land has no data.
open-meteo-sync-era5:
image: ${OM_IMAGE:-ghcr.io/open-meteo/open-meteo}
command:
- "sync"
- "copernicus_era5"
- "wind_gusts_10m,temperature_2m,dew_point_2m,precipitation,shortwave_radiation,wind_u_component_10m,wind_v_component_10m"
- "--past-days"
- "${OM_SYNC_PAST_DAYS:-14}"
- "--repeat-interval"
- "1440"
volumes:
- ${OM_DATA_DIR:-./data/om-archive}:/app/data
restart: unless-stopped
# Point backend's historical fetches at the local instance (frontend never
# fetches climate data itself). Set here (not in the base file) so it
# applies only when this overlay is active — and so an unset var never
# reaches the app as an empty string, which would defeat climate.py's default.
backend:
depends_on:
open-meteo-api:
condition: service_started
environment:
THERMOGRAPH_ARCHIVE_URL: http://open-meteo-api:8080/v1/archive