thermograph/infra/deploy/openmeteo/README.md

Self-hosted Open-Meteo (ERA5 archive)

Operator runbook for running a private Open-Meteo instance that serves the ERA5 historical archive to Thermograph, with the .om data held in object storage and surfaced on the host through an rclone FUSE mount.

1. What this is and why

Thermograph reads daily historical weather from an ERA5 archive. Off the shelf that means the public Open-Meteo archive API, which is rate-limited and not something to lean on for a production workload. This overlay runs our own Open-Meteo instance instead:

• open-meteo-api serves era5_seamless locally (internal to the compose network). The app points at it via THERMOGRAPH_ARCHIVE_URL.
• Two sync workers (open-meteo-sync-land, open-meteo-sync-era5) pull .om files from Open-Meteo's free AWS Open-Data bucket (no API key, no rate limit) and write them into the archive.

era5_seamless is a blend: 0.1° ERA5-Land for temperature, precipitation, humidity, and wind, plus 0.25° ERA5 for wind gusts (which ERA5-Land does not carry) and as the over-water fallback. The 0.1° resolution is a hard requirement for city-level accuracy.

The archive is ~1–1.5 TB of .om. That does not fit on the host's 400 GB disk, so it lives in an object-storage bucket and is mounted read/write via rclone. The host disk holds only the bounded rclone VFS cache, the app's own parquet cache, and Postgres — never a full copy. The app never reads object storage directly; it only talks to open-meteo-api, which reads the mount.

2. Object storage prerequisites

Provision a bucket of ~2 TB (holds the ~1–1.5 TB archive with headroom):

• Co-located with the VPS and low- or no-egress — e.g. Cloudflare R2, or same-provider object storage in the VPS's region.
• Co-location and low egress matter because the mount serves per-request range reads: every archive query pulls byte ranges out of many .om files. Cross-region or metered egress turns each read into latency and cost. Keep the bucket next to the compute and on a plan that does not bill egress.

You'll need S3-compatible credentials (access key id + secret) and the bucket's S3 endpoint.

3. Host rclone mount setup

Install rclone:

curl https://rclone.org/install.sh | sudo bash

Create /etc/rclone/rclone.conf with an S3-compatible remote. Use real values for your provider; never commit real secrets:

[om-archive]
type = s3
provider = Cloudflare
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
access_key_id = REPLACE_WITH_ACCESS_KEY_ID
secret_access_key = REPLACE_WITH_SECRET_ACCESS_KEY

Install and enable the mount unit (see rclone-mount.service.example):

sudo install -m0644 rclone-mount.service.example /etc/systemd/system/rclone-om.service
# edit BUCKET_NAME in the unit first; see the unit's header comments
sudo systemctl daemon-reload
sudo systemctl enable --now rclone-om

Verify the mount:

mountpoint -q /mnt/om-archive && echo mounted
ls /mnt/om-archive

The unit runs with --vfs-cache-mode full and a bounded --vfs-cache-max-size (e.g. 80G). Full VFS cache mode keeps hot cells on local disk after first read so repeat range reads don't go back to the bucket, and the size cap keeps that cache inside the 400 GB disk budget by evicting cold data.

Order Docker after the mount (reboots). So the restart: unless-stopped containers never start against an empty mount point, make Docker wait for the mount (rclone-om is Type=notify, so this waits until the mount is actually ready). Terraform installs this automatically; for a manual setup:

sudo install -d /etc/systemd/system/docker.service.d
printf '[Unit]\nWants=rclone-om.service\nAfter=rclone-om.service\n' \
  | sudo tee /etc/systemd/system/docker.service.d/10-wait-rclone.conf
sudo systemctl daemon-reload

(If the mount ever drops and remounts while the containers are running, the existing bind won't see the new mount — restart the Open-Meteo containers to re-bind. The app stays safe either way: it rejects a short/empty archive and falls back to NASA rather than caching a gap. See make om-up.)

4. Point the overlay at the mount

OM_DATA_DIR is read from the environment at docker compose time; in prod it lives in /etc/thermograph.env (which the systemd unit sources). Set it to the mount:

# /etc/thermograph.env
OM_DATA_DIR=/mnt/om-archive

All three services bind-mount ${OM_DATA_DIR} to /app/data, so with this set the archive reads and writes go to object storage.

5. One-time backfill

The sync workers only maintain a rolling recent window. To populate full history, run the backfill once:

make om-backfill

This runs each dataset's sync ... --past-days 17000 once via docker compose run --rm --no-deps. It writes the full ~1–1.5 TB of .om to object storage, is hours-long, and should be watched against the 2 TB budget. Run it before flipping the app over — if the app is pointed at an empty instance it falls back to NASA POWER, so bring the archive up to full history first.

For a smoke test, shorten the window with OM_BACKFILL_DAYS:

make om-backfill OM_BACKFILL_DAYS=30

6. Bring the overlay up

make om-up

This is docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml up -d --build. The overlay sets THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive automatically.

Smoke test the internal API and confirm every daily field is present and non-null. open-meteo-api has no published host port, so either run the curl from inside the compose network, or temporarily publish the port:

docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  exec app curl "http://open-meteo-api:8080/v1/archive?latitude=47.6&longitude=-122.3&start_date=2026-06-01&end_date=2026-06-10&daily=temperature_2m_max,temperature_2m_min,precipitation_sum,wind_speed_10m_max,wind_gusts_10m_max,apparent_temperature_max,apparent_temperature_min,relative_humidity_2m_mean&models=era5_seamless&temperature_unit=fahrenheit&wind_speed_unit=mph&precipitation_unit=inch"

If you've temporarily published the port instead, the same query works against http://127.0.0.1:8080/.... Check that each daily array is present and free of nulls across the date range.

7. Keeping current

The two sync workers re-sync --past-days 14 every 1440 minutes (daily). If a worker stalls, the recent tail of history goes stale — the last couple of weeks stop updating. (The separate forecast path is unaffected; this only touches the historical archive.)

Check the workers:

docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  logs open-meteo-sync-land
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  logs open-meteo-sync-era5

8. Attribution

The ERA5 and ERA5-Land data is CC-BY-4.0 (Copernicus/ECMWF), sourced via Open-Meteo. The Open-Meteo software is AGPLv3. The app already surfaces this credit; keep it in place.

9. Not on dev/beta

This overlay runs only for prod specifically — not for every environment on prod's host. Beta now shares vps2 with prod, but that doesn't extend the self-hosted archive to it: beta is its own Swarm stack (thermograph-beta-stack.yml), which never sets THERMOGRAPH_ARCHIVE_URL, so it reaches the public Open-Meteo archive API like dev does. Dev leaves it unset for the same reason on vps1. Do not bring this overlay up for beta or dev, and don't assume co-residency with prod on vps2 changes that.