# Self-hosted Open-Meteo (ERA5 archive) Operator runbook for running a private Open-Meteo instance that serves the ERA5 historical archive to Thermograph, with the `.om` data held in object storage and surfaced on the host through an rclone FUSE mount. ## 1. What this is and why Thermograph reads daily historical weather from an ERA5 archive. Off the shelf that means the public Open-Meteo archive API, which is rate-limited and not something to lean on for a production workload. This overlay runs our own Open-Meteo instance instead: - `open-meteo-api` serves `era5_seamless` locally (internal to the compose network). The app points at it via `THERMOGRAPH_ARCHIVE_URL`. - Two sync workers (`open-meteo-sync-land`, `open-meteo-sync-era5`) pull `.om` files from Open-Meteo's free AWS Open-Data bucket (no API key, no rate limit) and write them into the archive. `era5_seamless` is a blend: 0.1° ERA5-Land for temperature, precipitation, humidity, and wind, plus 0.25° ERA5 for wind gusts (which ERA5-Land does not carry) and as the over-water fallback. The 0.1° resolution is a hard requirement for city-level accuracy. The archive is ~1–1.5 TB of `.om`. That does not fit on the host's 400 GB disk, so it lives in an object-storage bucket and is mounted read/write via rclone. The host disk holds only the bounded rclone VFS cache, the app's own parquet cache, and Postgres — never a full copy. The app never reads object storage directly; it only talks to `open-meteo-api`, which reads the mount. ## 2. Object storage prerequisites Provision a bucket of ~2 TB (holds the ~1–1.5 TB archive with headroom): - **Co-located with the VPS** and **low- or no-egress** — e.g. Cloudflare R2, or same-provider object storage in the VPS's region. - Co-location and low egress matter because the mount serves per-request **range reads**: every archive query pulls byte ranges out of many `.om` files. Cross-region or metered egress turns each read into latency and cost. Keep the bucket next to the compute and on a plan that does not bill egress. You'll need S3-compatible credentials (access key id + secret) and the bucket's S3 endpoint. ## 3. Host rclone mount setup Install rclone: ```sh curl https://rclone.org/install.sh | sudo bash ``` Create `/etc/rclone/rclone.conf` with an S3-compatible remote. Use real values for your provider; **never commit real secrets**: ```ini [om-archive] type = s3 provider = Cloudflare endpoint = https://.r2.cloudflarestorage.com access_key_id = REPLACE_WITH_ACCESS_KEY_ID secret_access_key = REPLACE_WITH_SECRET_ACCESS_KEY ``` Install and enable the mount unit (see `rclone-mount.service.example`): ```sh sudo install -m0644 rclone-mount.service.example /etc/systemd/system/rclone-om.service # edit BUCKET_NAME in the unit first; see the unit's header comments sudo systemctl daemon-reload sudo systemctl enable --now rclone-om ``` Verify the mount: ```sh mountpoint -q /mnt/om-archive && echo mounted ls /mnt/om-archive ``` The unit runs with `--vfs-cache-mode full` and a bounded `--vfs-cache-max-size` (e.g. `80G`). Full VFS cache mode keeps hot cells on local disk after first read so repeat range reads don't go back to the bucket, and the size cap keeps that cache inside the 400 GB disk budget by evicting cold data. **Order Docker after the mount (reboots).** So the `restart: unless-stopped` containers never start against an empty mount point, make Docker wait for the mount (`rclone-om` is `Type=notify`, so this waits until the mount is actually ready). Terraform installs this automatically; for a manual setup: ```sh sudo install -d /etc/systemd/system/docker.service.d printf '[Unit]\nWants=rclone-om.service\nAfter=rclone-om.service\n' \ | sudo tee /etc/systemd/system/docker.service.d/10-wait-rclone.conf sudo systemctl daemon-reload ``` (If the mount ever drops and remounts *while* the containers are running, the existing bind won't see the new mount — restart the Open-Meteo containers to re-bind. The app stays safe either way: it rejects a short/empty archive and falls back to NASA rather than caching a gap. See `make om-up`.) ## 4. Point the overlay at the mount `OM_DATA_DIR` is read from the environment at `docker compose` time; in prod it lives in `/etc/thermograph.env` (which the systemd unit sources). Set it to the mount: ```sh # /etc/thermograph.env OM_DATA_DIR=/mnt/om-archive ``` All three services bind-mount `${OM_DATA_DIR}` to `/app/data`, so with this set the archive reads and writes go to object storage. ## 5. One-time backfill The sync workers only maintain a rolling recent window. To populate full history, run the backfill once: ```sh make om-backfill ``` This runs each dataset's `sync ... --past-days 17000` once via `docker compose run --rm --no-deps`. It writes the full ~1–1.5 TB of `.om` **to object storage**, is **hours-long**, and should be watched against the 2 TB budget. Run it **before** flipping the app over — if the app is pointed at an empty instance it falls back to NASA POWER, so bring the archive up to full history first. For a smoke test, shorten the window with `OM_BACKFILL_DAYS`: ```sh make om-backfill OM_BACKFILL_DAYS=30 ``` ## 6. Bring the overlay up ```sh make om-up ``` This is `docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml up -d --build`. The overlay sets `THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive` automatically. Smoke test the internal API and confirm every daily field is present and non-null. `open-meteo-api` has **no published host port**, so either run the curl from inside the compose network, or temporarily publish the port: ```sh docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \ exec app curl "http://open-meteo-api:8080/v1/archive?latitude=47.6&longitude=-122.3&start_date=2026-06-01&end_date=2026-06-10&daily=temperature_2m_max,temperature_2m_min,precipitation_sum,wind_speed_10m_max,wind_gusts_10m_max,apparent_temperature_max,apparent_temperature_min,relative_humidity_2m_mean&models=era5_seamless&temperature_unit=fahrenheit&wind_speed_unit=mph&precipitation_unit=inch" ``` If you've temporarily published the port instead, the same query works against `http://127.0.0.1:8080/...`. Check that each `daily` array is present and free of nulls across the date range. ## 7. Keeping current The two sync workers re-sync `--past-days 14` every 1440 minutes (daily). If a worker stalls, the recent tail of history goes stale — the last couple of weeks stop updating. (The separate forecast path is unaffected; this only touches the historical archive.) Check the workers: ```sh docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \ logs open-meteo-sync-land docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \ logs open-meteo-sync-era5 ``` ## 8. Attribution The ERA5 and ERA5-Land data is CC-BY-4.0 (Copernicus/ECMWF), sourced via Open-Meteo. The Open-Meteo software is AGPLv3. The app already surfaces this credit; keep it in place. ## 9. Not on dev/beta This overlay runs only for prod specifically — not for every environment on prod's host. Beta now shares vps2 with prod, but that doesn't extend the self-hosted archive to it: beta is its own Swarm stack (`thermograph-beta-stack.yml`), which never sets `THERMOGRAPH_ARCHIVE_URL`, so it reaches the public Open-Meteo archive API like dev does. Dev leaves it unset for the same reason on vps1. Do not bring this overlay up for beta or dev, and don't assume co-residency with prod on vps2 changes that.