ERA5 lake: bucket-hosted history primary + SQL indexer service

admin_emi commented

2026-07-23 21:17:58 +00:00

Owner

Full-period (1940+) ERA5 into the era5-thermograph bucket, partitioned tile/year/month with per-point serving files and a manifest; prod-only DuckDB lake service (1-2 replicas, autoscaled) in front; history chain becomes era5-lake -> NASA -> OM. Rehearsed end-to-end against the live Earthmover store: 86y tile extracted in 39s, /history 11ms cold / 3ms warm, pruned SQL 79ms. 385 tests pass. Blocked on bucket creds in the prod vault before extraction can run - the chain falls through to NASA until then.

admin_emi added 1 commit 2026-07-23 21:18:02 +00:00

ERA5 lake: bucket-hosted history primary + SQL indexer service

PR build (required check) / changes (pull_request) Successful in 12s

Details

secrets-guard / encrypted (pull_request) Successful in 11s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / build-backend (pull_request) Successful in 1m40s

Details

PR build (required check) / gate (pull_request) Successful in 4s

Details

0f34594ce0

Move climate history to ERA5 served from our own object storage, with a
prod-only query service in front:

- data/era5lake.py: lake layout (per-point whole-record 1940+ serving files,
  a tile/year/month hive table, an Iceberg-style manifest) plus grid math and
  the read client (local dir -> lake service -> bucket).
- gen_era5_lake.py: tile-aligned extractor from the Earthmover Icechunk ERA5
  archive (one 86-year pull covers all 144 points of a chunk tile; resumable
  from the manifest; --cities / --tiles / --land).
- lake_app.py + THERMOGRAPH_ROLE=lake: the lake service. /history serves a
  point's parquet off a disk cache (11ms cold / 3ms warm in rehearsal);
  /query runs SELECT-only SQL on DuckDB over the hive table (79ms pruned
  aggregate, 88ms full scan of a 4.5M-row tile). httpfs is baked at image
  build.
- climate.py: history chain is now era5-lake -> NASA POWER -> Open-Meteo; the
  lake slice starts at START_DATE so grading windows are unchanged, and an
  unconfigured lake costs nothing (beta/LAN unchanged).
- stack: lake service (1..2 replicas behind the VIP, own cache volume) plus a
  second autoscaler instance (autoscale.sh gains TARGET_SERVICE); web/worker
  get THERMOGRAPH_LAKE_URL.
- seed_era5.py: constants corrected against the live store (icechunkV2,
  single/temporal, valid_time, ECMWF short names, pcodec).

Bucket creds (THERMOGRAPH_LAKE_S3_ACCESS_KEY/_SECRET_KEY) go in the prod
vault; until they land the lake stays healthy and everything falls through to
NASA exactly as before.

admin_emi merged commit 370a4ffdc1 into dev

2026-07-23 21:20:37 +00:00

admin_emi referenced this pull request from a commit

2026-07-23 21:20:38 +00:00

ERA5 lake: bucket-hosted history primary + SQL indexer service (#15)

admin_emi deleted branch feat/era5-lake

2026-07-23 21:20:38 +00:00

admin_emi referenced this pull request from a commit

2026-08-01 16:02:10 +00:00

forgejo: serve dev.jinemi.com alongside git.thermograph.org

admin_emi referenced this pull request

2026-08-01 16:16:42 +00:00

forgejo: serve dev.jinemi.com alongside git.thermograph.org #148

ERA5 lake: bucket-hosted history primary + SQL indexer service #15