Parallelize lake tile uploads #17

Merged
admin_emi merged 1 commit from fix/lake-parallel-puts into dev 2026-07-23 21:33:01 +00:00

1 commit

Author SHA1 Message Date
Emi Griffith
c38b42e0f1 Parallelize lake tile uploads
All checks were successful
PR build (required check) / changes (pull_request) Successful in 10s
secrets-guard / encrypted (pull_request) Successful in 8s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 56s
PR build (required check) / gate (pull_request) Successful in 2s
A tile is ~1.2k small parquet objects; sequential PUTs to the bucket are
round-trip-bound (measured 2.5 obj/s, ~8 min per tile — ~78 h for the city
stage). A 16-thread pool per tile turns that into well under a minute; boto3
clients are thread-safe and failures surface via future.result().
2026-07-23 14:30:57 -07:00