f87e22be51
All checks were successful
PR build (required check) / changes (pull_request) Successful in 9s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 44s
PR build (required check) / gate (pull_request) Successful in 2s
Registers the hive part files under era5/daily into an Apache Iceberg v2 table at iceberg/era5_daily via pyiceberg add_files -- the metadata points at the existing parquet in place, no data rewrite. Incremental: each run diffs era5/manifest.parquet (the completeness signal while extraction is running) against the thermograph.synced-tiles table property, updated in the same commit as the files, so runs are idempotent and resume at batch boundaries. Partitioned by truncate[12](lat_idx), truncate[12](lon_idx), year(date), month(date) -- order-preserving transforms add_files derives from footer stats; queries prune on plain column predicates. Local sqlite catalog that re-registers from the latest metadata JSON if lost; version-hint.text refreshed per run so DuckDB can iceberg_scan the bare table root. Per-batch retries with backoff for Contabo SLOW_DOWN and slow-transfer timeouts. Slim pinned one-shot image; local-filesystem pytest suite (no network).
29 lines
868 B
Text
29 lines
868 B
Text
# Terraform state/vars are also ignored inside terraform/.gitignore; repeated here
|
|
# in case a stray file ever lands at repo root.
|
|
*.tfstate
|
|
*.tfstate.*
|
|
.terraform/
|
|
*.tfvars
|
|
!*.tfvars.example
|
|
crash.log
|
|
crash.*.log
|
|
|
|
# The SOPS+age private key must never be committed (deploy/secrets/*.yaml, the
|
|
# encrypted values, are meant to be committed).
|
|
age.key
|
|
*.age.key
|
|
|
|
.DS_Store
|
|
|
|
# lake-iceberg's pytest suite runs in place.
|
|
__pycache__/
|
|
*.pyc
|
|
|
|
# Host-side deploy state (deploy.sh): the live per-service image tags and the
|
|
# cross-repo deploy lock. Untracked on purpose -- they must survive the
|
|
# `git reset --hard` at the top of every deploy (deploy.sh's comments already
|
|
# assumed .image-tags.env was ignored; make it actually true so a stray
|
|
# `git clean` can't destroy the record of what's running).
|
|
deploy/.image-tags.env
|
|
deploy/.deploy.lock
|
|
deploy/.stack-image-tags.env
|