a26ca72834
Get the 45-year historical record off the rate-limited public Open-Meteo archive API by running a private Open-Meteo instance that serves the era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the compressed .om archive in object storage, mounted on the host with rclone. - climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin models=era5_seamless on the archive fetches only, so a self-hosted instance serves the same 0.1° resolution; forecast path unchanged. The public API's default is already seamless, so dev/beta (URL unset) behave identically. - docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers (era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage mount; the overlay points the app at the local instance. - Makefile: om-up / om-down / om-backfill (one-time full-history backfill). - Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR, and provisions the host rclone systemd mount from the bucket credentials. - deploy/openmeteo: operator runbook + rclone mount unit template.
172 lines
6.3 KiB
Markdown
172 lines
6.3 KiB
Markdown
# Self-hosted Open-Meteo (ERA5 archive)
|
||
|
||
Operator runbook for running a private Open-Meteo instance that serves the
|
||
ERA5 historical archive to Thermograph, with the `.om` data held in object
|
||
storage and surfaced on the host through an rclone FUSE mount.
|
||
|
||
## 1. What this is and why
|
||
|
||
Thermograph reads daily historical weather from an ERA5 archive. Off the
|
||
shelf that means the public Open-Meteo archive API, which is rate-limited and
|
||
not something to lean on for a production workload. This overlay runs our own
|
||
Open-Meteo instance instead:
|
||
|
||
- `open-meteo-api` serves `era5_seamless` locally (internal to the compose
|
||
network). The app points at it via `THERMOGRAPH_ARCHIVE_URL`.
|
||
- Two sync workers (`open-meteo-sync-land`, `open-meteo-sync-era5`) pull `.om`
|
||
files from Open-Meteo's free AWS Open-Data bucket (no API key, no rate
|
||
limit) and write them into the archive.
|
||
|
||
`era5_seamless` is a blend: 0.1° ERA5-Land for temperature, precipitation,
|
||
humidity, and wind, plus 0.25° ERA5 for wind gusts (which ERA5-Land does not
|
||
carry) and as the over-water fallback. The 0.1° resolution is a hard
|
||
requirement for city-level accuracy.
|
||
|
||
The archive is ~1–1.5 TB of `.om`. That does not fit on the host's 400 GB
|
||
disk, so it lives in an object-storage bucket and is mounted read/write via
|
||
rclone. The host disk holds only the bounded rclone VFS cache, the app's own
|
||
parquet cache, and Postgres — never a full copy. The app never reads object
|
||
storage directly; it only talks to `open-meteo-api`, which reads the mount.
|
||
|
||
## 2. Object storage prerequisites
|
||
|
||
Provision a bucket of ~2 TB (holds the ~1–1.5 TB archive with headroom):
|
||
|
||
- **Co-located with the VPS** and **low- or no-egress** — e.g. Cloudflare R2,
|
||
or same-provider object storage in the VPS's region.
|
||
- Co-location and low egress matter because the mount serves per-request
|
||
**range reads**: every archive query pulls byte ranges out of many `.om`
|
||
files. Cross-region or metered egress turns each read into latency and cost.
|
||
Keep the bucket next to the compute and on a plan that does not bill egress.
|
||
|
||
You'll need S3-compatible credentials (access key id + secret) and the
|
||
bucket's S3 endpoint.
|
||
|
||
## 3. Host rclone mount setup
|
||
|
||
Install rclone:
|
||
|
||
```sh
|
||
curl https://rclone.org/install.sh | sudo bash
|
||
```
|
||
|
||
Create `/etc/rclone/rclone.conf` with an S3-compatible remote. Use real
|
||
values for your provider; **never commit real secrets**:
|
||
|
||
```ini
|
||
[om-archive]
|
||
type = s3
|
||
provider = Cloudflare
|
||
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
|
||
access_key_id = REPLACE_WITH_ACCESS_KEY_ID
|
||
secret_access_key = REPLACE_WITH_SECRET_ACCESS_KEY
|
||
```
|
||
|
||
Install and enable the mount unit (see `rclone-mount.service.example`):
|
||
|
||
```sh
|
||
sudo install -m0644 rclone-mount.service.example /etc/systemd/system/rclone-om.service
|
||
# edit BUCKET_NAME in the unit first; see the unit's header comments
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl enable --now rclone-om
|
||
```
|
||
|
||
Verify the mount:
|
||
|
||
```sh
|
||
mountpoint -q /mnt/om-archive && echo mounted
|
||
ls /mnt/om-archive
|
||
```
|
||
|
||
The unit runs with `--vfs-cache-mode full` and a bounded
|
||
`--vfs-cache-max-size` (e.g. `80G`). Full VFS cache mode keeps hot cells on
|
||
local disk after first read so repeat range reads don't go back to the bucket,
|
||
and the size cap keeps that cache inside the 400 GB disk budget by evicting
|
||
cold data.
|
||
|
||
## 4. Point the overlay at the mount
|
||
|
||
`OM_DATA_DIR` is read from the environment at `docker compose` time; in prod
|
||
it lives in `/etc/thermograph.env` (which the systemd unit sources). Set it to
|
||
the mount:
|
||
|
||
```sh
|
||
# /etc/thermograph.env
|
||
OM_DATA_DIR=/mnt/om-archive
|
||
```
|
||
|
||
All three services bind-mount `${OM_DATA_DIR}` to `/app/data`, so with this
|
||
set the archive reads and writes go to object storage.
|
||
|
||
## 5. One-time backfill
|
||
|
||
The sync workers only maintain a rolling recent window. To populate full
|
||
history, run the backfill once:
|
||
|
||
```sh
|
||
make om-backfill
|
||
```
|
||
|
||
This runs each dataset's `sync ... --past-days 17000` once via
|
||
`docker compose run --rm --no-deps`. It writes the full ~1–1.5 TB of `.om`
|
||
**to object storage**, is **hours-long**, and should be watched against the
|
||
2 TB budget. Run it **before** flipping the app over — if the app is pointed
|
||
at an empty instance it falls back to NASA POWER, so bring the archive up to
|
||
full history first.
|
||
|
||
For a smoke test, shorten the window with `OM_BACKFILL_DAYS`:
|
||
|
||
```sh
|
||
make om-backfill OM_BACKFILL_DAYS=30
|
||
```
|
||
|
||
## 6. Bring the overlay up
|
||
|
||
```sh
|
||
make om-up
|
||
```
|
||
|
||
This is `docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml
|
||
up -d --build`. The overlay sets `THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive`
|
||
automatically.
|
||
|
||
Smoke test the internal API and confirm every daily field is present and
|
||
non-null. `open-meteo-api` has **no published host port**, so either run the
|
||
curl from inside the compose network, or temporarily publish the port:
|
||
|
||
```sh
|
||
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
|
||
exec app curl "http://open-meteo-api:8080/v1/archive?latitude=47.6&longitude=-122.3&start_date=2026-06-01&end_date=2026-06-10&daily=temperature_2m_max,temperature_2m_min,precipitation_sum,wind_speed_10m_max,wind_gusts_10m_max,apparent_temperature_max,apparent_temperature_min,relative_humidity_2m_mean&models=era5_seamless&temperature_unit=fahrenheit&wind_speed_unit=mph&precipitation_unit=inch"
|
||
```
|
||
|
||
If you've temporarily published the port instead, the same query works
|
||
against `http://127.0.0.1:8080/...`. Check that each `daily` array is present
|
||
and free of nulls across the date range.
|
||
|
||
## 7. Keeping current
|
||
|
||
The two sync workers re-sync `--past-days 14` every 1440 minutes (daily). If a
|
||
worker stalls, the recent tail of history goes stale — the last couple of
|
||
weeks stop updating. (The separate forecast path is unaffected; this only
|
||
touches the historical archive.)
|
||
|
||
Check the workers:
|
||
|
||
```sh
|
||
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
|
||
logs open-meteo-sync-land
|
||
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
|
||
logs open-meteo-sync-era5
|
||
```
|
||
|
||
## 8. Attribution
|
||
|
||
The ERA5 and ERA5-Land data is CC-BY-4.0 (Copernicus/ECMWF), sourced via
|
||
Open-Meteo. The Open-Meteo software is AGPLv3. The app already surfaces this
|
||
credit; keep it in place.
|
||
|
||
## 9. Not on dev/beta
|
||
|
||
This overlay runs only on the self-hosting host (prod). Dev and beta leave
|
||
`THERMOGRAPH_ARCHIVE_URL` unset and use the public Open-Meteo archive API — do
|
||
not bring the overlay up there.
|