0f34594ce0
All checks were successful
PR build (required check) / changes (pull_request) Successful in 12s
secrets-guard / encrypted (pull_request) Successful in 11s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 1m40s
PR build (required check) / gate (pull_request) Successful in 4s
Move climate history to ERA5 served from our own object storage, with a prod-only query service in front: - data/era5lake.py: lake layout (per-point whole-record 1940+ serving files, a tile/year/month hive table, an Iceberg-style manifest) plus grid math and the read client (local dir -> lake service -> bucket). - gen_era5_lake.py: tile-aligned extractor from the Earthmover Icechunk ERA5 archive (one 86-year pull covers all 144 points of a chunk tile; resumable from the manifest; --cities / --tiles / --land). - lake_app.py + THERMOGRAPH_ROLE=lake: the lake service. /history serves a point's parquet off a disk cache (11ms cold / 3ms warm in rehearsal); /query runs SELECT-only SQL on DuckDB over the hive table (79ms pruned aggregate, 88ms full scan of a 4.5M-row tile). httpfs is baked at image build. - climate.py: history chain is now era5-lake -> NASA POWER -> Open-Meteo; the lake slice starts at START_DATE so grading windows are unchanged, and an unconfigured lake costs nothing (beta/LAN unchanged). - stack: lake service (1..2 replicas behind the VIP, own cache volume) plus a second autoscaler instance (autoscale.sh gains TARGET_SERVICE); web/worker get THERMOGRAPH_LAKE_URL. - seed_era5.py: constants corrected against the live store (icechunkV2, single/temporal, valid_time, ECMWF short names, pcodec). Bucket creds (THERMOGRAPH_LAKE_S3_ACCESS_KEY/_SECRET_KEY) go in the prod vault; until they land the lake stays healthy and everything falls through to NASA exactly as before.
95 lines
3.3 KiB
Bash
Executable file
95 lines
3.3 KiB
Bash
Executable file
#!/bin/sh
|
|
# Autoscaler for the stack's `web` service: scale replicas between
|
|
# MIN_REPLICAS and MAX_REPLICAS on sustained per-task CPU.
|
|
#
|
|
# Runs as a Swarm service on the manager with the docker socket mounted (see
|
|
# thermograph-stack.yml). Every web task is placed on this node today, so
|
|
# node-local `docker stats` sees them all — when a second app node exists,
|
|
# this needs a per-node reader or a metrics-based signal instead; that's the
|
|
# documented upgrade path, not a today problem.
|
|
#
|
|
# Semantics (deliberately boring):
|
|
# - Sample avg CPU% per web task every POLL_SECONDS (docker stats CPUPerc:
|
|
# 100 = one full host core).
|
|
# - UP_SAMPLES consecutive samples above SCALE_UP_CPU -> scale +1.
|
|
# - DOWN_SAMPLES consecutive samples below SCALE_DOWN_CPU -> scale -1.
|
|
# (Down is ~7x slower than up on defaults: flap-averse by construction.)
|
|
# - COOLDOWN_SECONDS after any change: samples are ignored entirely.
|
|
# - Clamped to [MIN_REPLICAS, MAX_REPLICAS]; scaling waits for convergence
|
|
# (--detach=false), so a stuck rollout blocks further changes rather than
|
|
# stacking them.
|
|
set -eu
|
|
|
|
STACK_NAME="${STACK_NAME:-thermograph}"
|
|
# Which stack service to scale (the stack runs one autoscaler instance per
|
|
# scaled service — web and lake today; same loop, different bounds).
|
|
SERVICE="${STACK_NAME}_${TARGET_SERVICE:-web}"
|
|
MIN="${MIN_REPLICAS:-1}"
|
|
MAX="${MAX_REPLICAS:-3}"
|
|
UP_AT="${SCALE_UP_CPU:-220}"
|
|
DOWN_AT="${SCALE_DOWN_CPU:-60}"
|
|
POLL="${POLL_SECONDS:-15}"
|
|
UP_N="${UP_SAMPLES:-3}"
|
|
DOWN_N="${DOWN_SAMPLES:-20}"
|
|
COOLDOWN="${COOLDOWN_SECONDS:-180}"
|
|
|
|
up_hits=0
|
|
down_hits=0
|
|
last_change=0
|
|
|
|
log() { echo "[autoscale] $(date -u +%H:%M:%S) $*"; }
|
|
|
|
replicas() {
|
|
docker service inspect "$SERVICE" \
|
|
--format '{{.Spec.Mode.Replicated.Replicas}}' 2>/dev/null || echo ""
|
|
}
|
|
|
|
avg_cpu() {
|
|
# Mean CPUPerc across this node's web tasks, as an integer percent.
|
|
docker stats --no-stream --format '{{.Name}} {{.CPUPerc}}' 2>/dev/null \
|
|
| awk -v svc="$SERVICE" '
|
|
index($1, svc".") == 1 {
|
|
gsub(/%/, "", $2); sum += $2; n++
|
|
}
|
|
END { if (n > 0) printf "%d", sum / n; else print "" }'
|
|
}
|
|
|
|
log "watching $SERVICE: min=$MIN max=$MAX up>@${UP_AT}%x${UP_N} down<@${DOWN_AT}%x${DOWN_N} poll=${POLL}s cooldown=${COOLDOWN}s"
|
|
|
|
while :; do
|
|
sleep "$POLL"
|
|
|
|
now=$(date +%s)
|
|
if [ $((now - last_change)) -lt "$COOLDOWN" ]; then
|
|
continue
|
|
fi
|
|
|
|
cur=$(replicas)
|
|
[ -n "$cur" ] || { log "service $SERVICE not found; waiting"; continue; }
|
|
cpu=$(avg_cpu)
|
|
[ -n "$cpu" ] || continue # no running tasks visible this sample
|
|
|
|
if [ "$cpu" -gt "$UP_AT" ]; then
|
|
up_hits=$((up_hits + 1)); down_hits=0
|
|
elif [ "$cpu" -lt "$DOWN_AT" ]; then
|
|
down_hits=$((down_hits + 1)); up_hits=0
|
|
else
|
|
up_hits=0; down_hits=0
|
|
fi
|
|
|
|
if [ "$up_hits" -ge "$UP_N" ] && [ "$cur" -lt "$MAX" ]; then
|
|
target=$((cur + 1))
|
|
log "avg cpu ${cpu}% > ${UP_AT}% x${UP_N}: scaling $cur -> $target"
|
|
if docker service scale --detach=false "$SERVICE=$target"; then
|
|
last_change=$(date +%s)
|
|
fi
|
|
up_hits=0; down_hits=0
|
|
elif [ "$down_hits" -ge "$DOWN_N" ] && [ "$cur" -gt "$MIN" ]; then
|
|
target=$((cur - 1))
|
|
log "avg cpu ${cpu}% < ${DOWN_AT}% x${DOWN_N}: scaling $cur -> $target"
|
|
if docker service scale --detach=false "$SERVICE=$target"; then
|
|
last_change=$(date +%s)
|
|
fi
|
|
up_hits=0; down_hits=0
|
|
fi
|
|
done
|