thermograph/infra/deploy/openmeteo/README.md

# Self-hosted Open-Meteo (ERA5 archive)

Operator runbook for running a private Open-Meteo instance that serves the
ERA5 historical archive to Thermograph, with the `.om` data held in object
storage and surfaced on the host through an rclone FUSE mount.

## 1. What this is and why

Thermograph reads daily historical weather from an ERA5 archive. Off the
shelf that means the public Open-Meteo archive API, which is rate-limited and
not something to lean on for a production workload. This overlay runs our own
Open-Meteo instance instead:

- `open-meteo-api` serves `era5_seamless` locally (internal to the compose
  network). The app points at it via `THERMOGRAPH_ARCHIVE_URL`.
- Two sync workers (`open-meteo-sync-land`, `open-meteo-sync-era5`) pull `.om`
  files from Open-Meteo's free AWS Open-Data bucket (no API key, no rate
  limit) and write them into the archive.

`era5_seamless` is a blend: 0.1° ERA5-Land for temperature, precipitation,
humidity, and wind, plus 0.25° ERA5 for wind gusts (which ERA5-Land does not
carry) and as the over-water fallback. The 0.1° resolution is a hard
requirement for city-level accuracy.

The archive is ~1–1.5 TB of `.om`. That does not fit on the host's 400 GB
disk, so it lives in an object-storage bucket and is mounted read/write via
rclone. The host disk holds only the bounded rclone VFS cache, the app's own
parquet cache, and Postgres — never a full copy. The app never reads object
storage directly; it only talks to `open-meteo-api`, which reads the mount.

## 2. Object storage prerequisites

Provision a bucket of ~2 TB (holds the ~1–1.5 TB archive with headroom):

- **Co-located with the VPS** and **low- or no-egress** — e.g. Cloudflare R2,
  or same-provider object storage in the VPS's region.
- Co-location and low egress matter because the mount serves per-request
  **range reads**: every archive query pulls byte ranges out of many `.om`
  files. Cross-region or metered egress turns each read into latency and cost.
  Keep the bucket next to the compute and on a plan that does not bill egress.

You'll need S3-compatible credentials (access key id + secret) and the
bucket's S3 endpoint.

## 3. Host rclone mount setup

Install rclone:

```sh
curl https://rclone.org/install.sh | sudo bash
```

Create `/etc/rclone/rclone.conf` with an S3-compatible remote. Use real
values for your provider; **never commit real secrets**:

```ini
[om-archive]
type = s3
provider = Cloudflare
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
access_key_id = REPLACE_WITH_ACCESS_KEY_ID
secret_access_key = REPLACE_WITH_SECRET_ACCESS_KEY
```

Install and enable the mount unit (see `rclone-mount.service.example`):

```sh
sudo install -m0644 rclone-mount.service.example /etc/systemd/system/rclone-om.service
# edit BUCKET_NAME in the unit first; see the unit's header comments
sudo systemctl daemon-reload
sudo systemctl enable --now rclone-om
```

Verify the mount:

```sh
mountpoint -q /mnt/om-archive && echo mounted
ls /mnt/om-archive
```

The unit runs with `--vfs-cache-mode full` and a bounded
`--vfs-cache-max-size` (e.g. `80G`). Full VFS cache mode keeps hot cells on
local disk after first read so repeat range reads don't go back to the bucket,
and the size cap keeps that cache inside the 400 GB disk budget by evicting
cold data.

**Order Docker after the mount (reboots).** So the `restart: unless-stopped`
containers never start against an empty mount point, make Docker wait for the
mount (`rclone-om` is `Type=notify`, so this waits until the mount is actually
ready). Terraform installs this automatically; for a manual setup:

```sh
sudo install -d /etc/systemd/system/docker.service.d
printf '[Unit]\nWants=rclone-om.service\nAfter=rclone-om.service\n' \
  | sudo tee /etc/systemd/system/docker.service.d/10-wait-rclone.conf
sudo systemctl daemon-reload
```

(If the mount ever drops and remounts *while* the containers are running, the
existing bind won't see the new mount — restart the Open-Meteo containers to
re-bind. The app stays safe either way: it rejects a short/empty archive and
falls back to NASA rather than caching a gap. See `make om-up`.)

## 4. Point the overlay at the mount

`OM_DATA_DIR` is read from the environment at `docker compose` time; in prod
it lives in `/etc/thermograph.env` (which the systemd unit sources). Set it to
the mount:

```sh
# /etc/thermograph.env
OM_DATA_DIR=/mnt/om-archive
```

All three services bind-mount `${OM_DATA_DIR}` to `/app/data`, so with this
set the archive reads and writes go to object storage.

## 5. One-time backfill

The sync workers only maintain a rolling recent window. To populate full
history, run the backfill once:

```sh
make om-backfill
```

This runs each dataset's `sync ... --past-days 17000` once via
`docker compose run --rm --no-deps`. It writes the full ~1–1.5 TB of `.om`
**to object storage**, is **hours-long**, and should be watched against the
2 TB budget. Run it **before** flipping the app over — if the app is pointed
at an empty instance it falls back to NASA POWER, so bring the archive up to
full history first.

For a smoke test, shorten the window with `OM_BACKFILL_DAYS`:

```sh
make om-backfill OM_BACKFILL_DAYS=30
```

## 6. Bring the overlay up

```sh
make om-up
```

This is `docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml
up -d --build`. The overlay sets `THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive`
automatically.

Smoke test the internal API and confirm every daily field is present and
non-null. `open-meteo-api` has **no published host port**, so either run the
curl from inside the compose network, or temporarily publish the port:

```sh
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  exec app curl "http://open-meteo-api:8080/v1/archive?latitude=47.6&longitude=-122.3&start_date=2026-06-01&end_date=2026-06-10&daily=temperature_2m_max,temperature_2m_min,precipitation_sum,wind_speed_10m_max,wind_gusts_10m_max,apparent_temperature_max,apparent_temperature_min,relative_humidity_2m_mean&models=era5_seamless&temperature_unit=fahrenheit&wind_speed_unit=mph&precipitation_unit=inch"
```

If you've temporarily published the port instead, the same query works
against `http://127.0.0.1:8080/...`. Check that each `daily` array is present
and free of nulls across the date range.

## 7. Keeping current

The two sync workers re-sync `--past-days 14` every 1440 minutes (daily). If a
worker stalls, the recent tail of history goes stale — the last couple of
weeks stop updating. (The separate forecast path is unaffected; this only
touches the historical archive.)

Check the workers:

```sh
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  logs open-meteo-sync-land
docker compose -f docker-compose.yml -f docker-compose.openmeteo.yml \
  logs open-meteo-sync-era5
```

## 8. Attribution

The ERA5 and ERA5-Land data is CC-BY-4.0 (Copernicus/ECMWF), sourced via
Open-Meteo. The Open-Meteo software is AGPLv3. The app already surfaces this
credit; keep it in place.

## 9. Not on dev/beta

This overlay runs only for prod specifically — not for every environment on
prod's host. Beta now shares vps2 with prod, but that doesn't extend the
self-hosted archive to it: beta is its own Swarm stack
(`thermograph-beta-stack.yml`), which never sets `THERMOGRAPH_ARCHIVE_URL`, so
it reaches the public Open-Meteo archive API like dev does. Dev leaves it
unset for the same reason on vps1. Do not bring this overlay up for beta or
dev, and don't assume co-residency with prod on vps2 changes that.