Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions

Lake /query reads era5_daily through Iceberg metadata #44

Merged
admin_emi merged 1 commit from fix/lake-query-iceberg into dev 2026-07-24 19:09:23 +00:00
Conversation 0 Commits 1 Files changed 2 +44 -25
admin_emi commented 2026-07-24 19:07:58 +00:00
Owner
Copy link

Recovered commit: the hive-glob view LISTs ~1k objects/tile at bind (times out at 390 tiles, seen on prod); bucket mode now iceberg_scans the newest metadata JSON; iceberg extension baked as runtime user; local mode (tests) unchanged.

Recovered commit: the hive-glob view LISTs ~1k objects/tile at bind (times out at 390 tiles, seen on prod); bucket mode now iceberg_scans the newest metadata JSON; iceberg extension baked as runtime user; local mode (tests) unchanged.
admin_emi added 1 commit 2026-07-24 19:08:00 +00:00
Lake /query reads era5_daily through Iceberg metadata
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
Details
PR build (required check) / changes (pull_request) Successful in 8s
Details
shell-lint / shellcheck (pull_request) Successful in 7s
Details
PR build (required check) / build-frontend (pull_request) Has been skipped
Details
PR build (required check) / validate-observability (pull_request) Has been skipped
Details
PR build (required check) / build-backend (pull_request) Successful in 52s
Details
PR build (required check) / gate (pull_request) Successful in 2s
Details
38b11635a6 
The hive-glob view must LIST every object under era5/daily/ at view bind —
~1k objects per tile, which crossed the request timeout once the lake hit
390 tiles (seen live: prod /query timing out). Bucket mode now resolves the
newest Iceberg metadata JSON (one single-page LIST) and iceberg_scans it, so
the file list comes from table metadata; local mode (tests, small trees)
keeps the plain glob. The iceberg extension is baked next to httpfs, as the
runtime user.
admin_emi merged commit 0f1b3a168c into dev 2026-07-24 19:09:23 +00:00
admin_emi deleted branch fix/lake-query-iceberg 2026-07-24 19:09:23 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
Compat/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#44
No description provided.
Delete branch "fix/lake-query-iceberg"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?