ERA5 Iceberg lake is unqueryable in both prod and beta - "is the ERA5 ingest current?" cannot be answered by any means #56

New issue

Closed

opened 2026-07-24 19:54:20 +00:00 by admin_emi · 0 comments

admin_emi commented

2026-07-24 19:54:20 +00:00

Owner

Summary

Neither environment can query the ERA5 Iceberg lake, for two different reasons. The practical consequence is that there is currently no way to verify whether the ERA5 ingest is up to date.

Reproduction (2026-07-24 ~19:40 UTC)

prod:

lake_tables(env: "prod")
-> lake query failed on prod:
   bash: line 7: THERMOGRAPH_LAKE_S3_ACCESS_KEY: lake S3 access key not present on this host

beta:

lake_tables(env: "beta")
-> lake query failed on beta:
   Unable to find image 'thermograph/duckdb-lake:latest' locally
   docker: Error response from daemon: pull access denied for thermograph/duckdb-lake,
   repository does not exist or may require 'docker login'

Both reproduce consistently.

Two independent faults

prod - the lake S3 access key is not present on the host, so the query container cannot authenticate to the bucket at all.
beta - the query path references thermograph/duckdb-lake:latest, an unqualified Docker Hub name that does not exist. Every other image in the estate is pulled from git.thermograph.org/emi/thermograph/*, so this reads as a missing registry prefix rather than a missing build.

For contrast, the lake services themselves are running normally in both environments (deployed_version):

prod: thermograph_lake     git.thermograph.org/emi/thermograph/backend:sha-0f1b3a168c65
beta: thermograph-lake-1   git.thermograph.org/emi/thermograph/backend:sha-0f1b3a168c65

So the ingest may well be perfectly healthy - which is exactly the problem: it is unverifiable either way.

Relationship to PR #44

PR #44 (Lake /query reads era5_daily through Iceberg metadata) is already closed and merged, promoted to main in #45 and to release in #46. It changed backend/Dockerfile and backend/lake_app.py so bucket mode iceberg_scans the newest metadata JSON instead of binding the hive-glob view. It does not address either failure here - neither a missing host credential nor an unresolvable image name. #44 is not the pending fix for this issue.

Worth checking whether beta's thermograph/duckdb-lake reference was introduced by #44's Dockerfile change ("iceberg extension baked as runtime user"), since the timing lines up closely.

Impact

The ~45 years of ERA5 history is the entire basis of comparison for every grade the product issues. There is currently no read path to confirm it is current, in either environment, by any available tool. A silent ingest stall would look identical to a healthy lake from the outside.

Suggested action

prod: provision THERMOGRAPH_LAKE_S3_ACCESS_KEY (and its secret) onto the host the query path runs on.
beta: qualify the image reference to the internal registry, or build/publish duckdb-lake where beta can pull it.
Once queryable, add a freshness check on era5_daily (max partition date) so ingest staleness is detectable without a manual query - same gap as the climate_sync staleness issue.

## Summary Neither environment can query the ERA5 Iceberg lake, for two *different* reasons. The practical consequence is that there is currently no way to verify whether the ERA5 ingest is up to date. ## Reproduction (2026-07-24 ~19:40 UTC) **prod:** ``` lake_tables(env: "prod") -> lake query failed on prod: bash: line 7: THERMOGRAPH_LAKE_S3_ACCESS_KEY: lake S3 access key not present on this host ``` **beta:** ``` lake_tables(env: "beta") -> lake query failed on beta: Unable to find image 'thermograph/duckdb-lake:latest' locally docker: Error response from daemon: pull access denied for thermograph/duckdb-lake, repository does not exist or may require 'docker login' ``` Both reproduce consistently. ## Two independent faults - **prod** - the lake S3 access key is not present on the host, so the query container cannot authenticate to the bucket at all. - **beta** - the query path references `thermograph/duckdb-lake:latest`, an unqualified Docker Hub name that does not exist. Every other image in the estate is pulled from `git.thermograph.org/emi/thermograph/*`, so this reads as a missing registry prefix rather than a missing build. For contrast, the lake *services* themselves are running normally in both environments (`deployed_version`): ``` prod: thermograph_lake git.thermograph.org/emi/thermograph/backend:sha-0f1b3a168c65 beta: thermograph-lake-1 git.thermograph.org/emi/thermograph/backend:sha-0f1b3a168c65 ``` So the ingest may well be perfectly healthy - which is exactly the problem: it is unverifiable either way. ## Relationship to PR #44 PR #44 (`Lake /query reads era5_daily through Iceberg metadata`) is **already closed and merged**, promoted to main in #45 and to release in #46. It changed `backend/Dockerfile` and `backend/lake_app.py` so bucket mode `iceberg_scan`s the newest metadata JSON instead of binding the hive-glob view. **It does not address either failure here** - neither a missing host credential nor an unresolvable image name. #44 is not the pending fix for this issue. Worth checking whether beta's `thermograph/duckdb-lake` reference was introduced by #44's Dockerfile change ("iceberg extension baked as runtime user"), since the timing lines up closely. ## Impact The ~45 years of ERA5 history is the entire basis of comparison for every grade the product issues. There is currently no read path to confirm it is current, in either environment, by any available tool. A silent ingest stall would look identical to a healthy lake from the outside. ## Suggested action - prod: provision `THERMOGRAPH_LAKE_S3_ACCESS_KEY` (and its secret) onto the host the query path runs on. - beta: qualify the image reference to the internal registry, or build/publish `duckdb-lake` where beta can pull it. - Once queryable, add a freshness check on `era5_daily` (max partition date) so ingest staleness is detectable without a manual query - same gap as the climate_sync staleness issue.