thermograph/deploy/openmeteo/README.md

# Self-hosted Open-Meteo (ERA5 archive)

Operator runbook for running a private Open-Meteo instance that serves the
ERA5 historical archive to Thermograph, with the `.om` data held in object
storage and surfaced on the host through an rclone FUSE mount.

## 1. What this is and why

Thermograph reads daily historical weather from an ERA5 archive. Off the
shelf that means the public Open-Meteo archive API, which is rate-limited and
not something to lean on for a production workload. This overlay runs our own
Open-Meteo instance instead:

- `open-meteo-api` serves `era5_seamless` locally (internal to the compose
  network). The app points at it via `THERMOGRAPH_ARCHIVE_URL`.
- Two sync workers (`open-meteo-sync-land`, `open-meteo-sync-era5`) pull `.om`
  files from Open-Meteo's free AWS Open-Data bucket (no API key, no rate
  limit) and write them into the archive.

`era5_seamless` is a blend: 0.1° ERA5-Land for temperature, precipitation,
humidity, and wind, plus 0.25° ERA5 for wind gusts (which ERA5-Land does not
carry) and as the over-water fallback. The 0.1° resolution is a hard
requirement for city-level accuracy.

The archive is ~1–1.5 TB of `.om`. That does not fit on the host's 400 GB
disk, so it lives in an object-storage bucket and is mounted read/write via
rclone. The host disk holds only the bounded rclone VFS cache, the app's own
parquet cache, and Postgres — never a full copy. The app never reads object
storage directly; it only talks to `open-meteo-api`, which reads the mount.

## 2. Object storage prerequisites

Provision a bucket of ~2 TB (holds the ~1–1.5 TB archive with headroom):

- **Co-located with the VPS** and **low- or no-egress** — e.g. Cloudflare R2,
  or same-provider object storage in the VPS's region.
- Co-location and low egress matter because the mount serves per-request
  **range reads**: every archive query pulls byte ranges out of many `.om`
  files. Cross-region or metered egress turns each read into latency and cost.
  Keep the bucket next to the compute and on a plan that does not bill egress.

You'll need S3-compatible credentials (access key id + secret) and the
bucket's S3 endpoint.

## 3. Host rclone mount setup

Install rclone:

```sh
curl https://rclone.org/install.sh | sudo bash
```

Create `/etc/rclone/rclone.conf` with an S3-compatible remote. Use real
values for your provider; **never commit real secrets**:

```ini
[om-archive]
type = s3
provider = Cloudflare
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
access_key_id = REPLACE_WITH_ACCESS_KEY_ID
secret_access_key = REPLACE_WITH_SECRET_ACCESS_KEY
```

Install and enable the mount unit (see `rclone-mount.service.example`):

```sh
sudo install -m0644 rclone-mount.service.example /etc/systemd/system/rclone-om.service
# edit BUCKET_NAME in the unit first; see the unit's header comments
sudo systemctl daemon-reload
sudo systemctl enable --now rclone-om
```

Verify the mount:

```sh
mountpoint -q /mnt/om-archive && echo mounted
ls /mnt/om-archive
```

The unit runs with `--vfs-cache-mode full` and a bounded
`--vfs-cache-max-size` (e.g. `80G`). Full VFS cache mode keeps hot cells on
local disk after first read so repeat range reads don't go back to the bucket,
and the size cap keeps that cache inside the 400 GB disk budget by evicting
cold data.

## 4. Point the overlay at the mount

`OM_DATA_DIR` is read from the environment at `docker compose` time; in prod
it lives in `/etc/thermograph.env` (which the systemd unit sources). Set it to
the mount:

```sh
# /etc/thermograph.env
OM_DATA_DIR=/mnt/om-archive
```

All three services bind-mount `${OM_DATA_DIR}` to `/app/data`, so with this
set the archive reads and writes go to object storage.

## 5. One-time backfill

The sync workers only maintain a rolling recent window. To populate full
history, run the backfill once:

```sh
make om-backfill
```

This runs each dataset's `sync ... --past-days 17000` once via
`docker compose run --rm --no-deps`. It writes the full ~1–1.5 TB of `.om`
**to object storage**, is **hours-long**, and should be watched against the
2 TB budget. Run it **before** flipping the app over — if the app is pointed
at an empty instance it falls back to NASA POWER, so bring the archive up to
full history first.

For a smoke test, shorten the window with `OM_BACKFILL_DAYS`:

```sh
make om-backfill OM_BACKFILL_DAYS=30
```

## 6. Bring the overlay up

```sh
make om-up
```

This is `docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml
up -d --build`. The overlay sets `THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive`
automatically.

Smoke test the internal API and confirm every daily field is present and
non-null. `open-meteo-api` has **no published host port**, so either run the
curl from inside the compose network, or temporarily publish the port:

```sh
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  exec app curl "http://open-meteo-api:8080/v1/archive?latitude=47.6&longitude=-122.3&start_date=2026-06-01&end_date=2026-06-10&daily=temperature_2m_max,temperature_2m_min,precipitation_sum,wind_speed_10m_max,wind_gusts_10m_max,apparent_temperature_max,apparent_temperature_min,relative_humidity_2m_mean&models=era5_seamless&temperature_unit=fahrenheit&wind_speed_unit=mph&precipitation_unit=inch"
```

If you've temporarily published the port instead, the same query works
against `http://127.0.0.1:8080/...`. Check that each `daily` array is present
and free of nulls across the date range.

## 7. Keeping current

The two sync workers re-sync `--past-days 14` every 1440 minutes (daily). If a
worker stalls, the recent tail of history goes stale — the last couple of
weeks stop updating. (The separate forecast path is unaffected; this only
touches the historical archive.)

Check the workers:

```sh
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  logs open-meteo-sync-land
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  logs open-meteo-sync-era5
```

## 8. Attribution

The ERA5 and ERA5-Land data is CC-BY-4.0 (Copernicus/ECMWF), sourced via
Open-Meteo. The Open-Meteo software is AGPLv3. The app already surfaces this
credit; keep it in place.

## 9. Not on dev/beta

This overlay runs only on the self-hosting host (prod). Dev and beta leave
`THERMOGRAPH_ARCHIVE_URL` unset and use the public Open-Meteo archive API — do
not bring the overlay up there.