Commit graph

2 commits

Author SHA1 Message Date
Emi Griffith
f87e22be51 Add infra/lake-iceberg: Iceberg conversion for the ERA5 lake
All checks were successful
PR build (required check) / changes (pull_request) Successful in 9s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 44s
PR build (required check) / gate (pull_request) Successful in 2s
Registers the hive part files under era5/daily into an Apache Iceberg v2
table at iceberg/era5_daily via pyiceberg add_files -- the metadata points
at the existing parquet in place, no data rewrite. Incremental: each run
diffs era5/manifest.parquet (the completeness signal while extraction is
running) against the thermograph.synced-tiles table property, updated in
the same commit as the files, so runs are idempotent and resume at batch
boundaries. Partitioned by truncate[12](lat_idx), truncate[12](lon_idx),
year(date), month(date) -- order-preserving transforms add_files derives
from footer stats; queries prune on plain column predicates. Local sqlite
catalog that re-registers from the latest metadata JSON if lost;
version-hint.text refreshed per run so DuckDB can iceberg_scan the bare
table root. Per-batch retries with backoff for Contabo SLOW_DOWN and
slow-transfer timeouts. Slim pinned one-shot image; local-filesystem
pytest suite (no network).
2026-07-23 15:49:33 -07:00
Emi Griffith
ae1d9bb534 Subtree-merge thermograph-infra (origin/main) into infra/
git-subtree-dir: infra
git-subtree-mainline: d6df04eab2
git-subtree-split: 99b4b3f78d
2026-07-22 22:01:11 -07:00