f87e22be51
All checks were successful
PR build (required check) / changes (pull_request) Successful in 9s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 44s
PR build (required check) / gate (pull_request) Successful in 2s
Registers the hive part files under era5/daily into an Apache Iceberg v2 table at iceberg/era5_daily via pyiceberg add_files -- the metadata points at the existing parquet in place, no data rewrite. Incremental: each run diffs era5/manifest.parquet (the completeness signal while extraction is running) against the thermograph.synced-tiles table property, updated in the same commit as the files, so runs are idempotent and resume at batch boundaries. Partitioned by truncate[12](lat_idx), truncate[12](lon_idx), year(date), month(date) -- order-preserving transforms add_files derives from footer stats; queries prune on plain column predicates. Local sqlite catalog that re-registers from the latest metadata JSON if lost; version-hint.text refreshed per run so DuckDB can iceberg_scan the bare table root. Per-batch retries with backoff for Contabo SLOW_DOWN and slow-transfer timeouts. Slim pinned one-shot image; local-filesystem pytest suite (no network).
20 lines
802 B
Docker
20 lines
802 B
Docker
# One-shot Iceberg sync for the ERA5 lake: registers new hive tiles from the
|
|
# era5-thermograph bucket into the iceberg/era5_daily table (see
|
|
# sync_iceberg.py). Everything ships as manylinux wheels — no toolchain.
|
|
FROM python:3.12-slim
|
|
|
|
# The pyiceberg extras pull sqlalchemy (sqlite catalog), pyarrow (FileIO +
|
|
# parquet footers) and s3fs (bucket listing/manifest reads).
|
|
COPY requirements.txt /tmp/requirements.txt
|
|
RUN pip install --no-cache-dir -r /tmp/requirements.txt
|
|
|
|
WORKDIR /app
|
|
COPY sync_iceberg.py .
|
|
|
|
# The sqlite catalog lives on a volume so runs share it; losing it is fine
|
|
# (the next run re-registers the table from its latest metadata JSON).
|
|
ENV THERMOGRAPH_ICEBERG_CATALOG_DB=/state/iceberg-catalog.db \
|
|
PYTHONUNBUFFERED=1
|
|
VOLUME /state
|
|
|
|
ENTRYPOINT ["python", "sync_iceberg.py"]
|