Add cheap, stable content-page cache token #47

Merged

admin_emi merged 1 commit from fix/content-token into dev

2026-07-24 19:30:39 +00:00

admin_emi commented

2026-07-24 19:16:39 +00:00

Owner

Problem

Content SEO pages (/climate/<city>[/month|/records]) key their derived-payload cache validity on history_token(history) = PAYLOAD_VER:hist_end, computed from the fully-loaded ~45-year archive. Two costs fall out of that:

Every hourly tail top-up advances hist_end (an intra-day timestamp change), so the entire content cache for a cell invalidates ~1-2x/day, forcing expensive recomputes.
Computing the token at all requires loading the full history first — so even a cache hit pays the full-history load.

Change (Fix 1 of 3)

Adds a new token that content routes and the warm job (other agents' scope) will use instead of history_token:

payloads.CONTENT_VER = "c1" — a separate content-shape version, so a content-only change need not orphan every other cache kind.
payloads.content_token(cell_id) -> str returning f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date or 'none'}".
climate_store.history_max_date(cell_id) -> str | None — indexed MAX(date) over the (cell_id, date) primary key (confirmed in migration 0002_climate_hypertables), never a full-history load.
climate.history_max_date(cell_id) -> str | None — backend dispatcher: Postgres path delegates to the store; the dev/parquet path does a single-column scan_parquet(...).select(col("date").max()), not a full frame load.

The token keys on the archive's newest date, so it survives intra-day top-ups and turns over only when the last archived day genuinely advances (~1x/day) — keeping content pages ≤1 day stale, acceptable for SEO. Fail-soft throughout: a store/DB error buckets to 'none' rather than raising.

Scope kept to backend/api/payloads.py, backend/data/climate_store.py, backend/data/climate.py (+ tests). content_routes.py / warm_cities.py / scheduler.py untouched — owned by the other fixes.

Tests

New unit tests: parquet + gated-Postgres history_max_date round-trips (including cell isolation and tail-topup advance), and content_token stability (same max date -> identical token across simulated top-ups; changed max date -> new token) plus fail-soft bucketing to 'none'. Suite green (the 2 unrelated test_lake_app failures are a missing duckdb in the dev venv, pre-existing).

## Problem Content SEO pages (`/climate/<city>[/month|/records]`) key their derived-payload cache validity on `history_token(history) = PAYLOAD_VER:hist_end`, computed from the fully-loaded ~45-year archive. Two costs fall out of that: - Every hourly tail top-up advances `hist_end` (an intra-day timestamp change), so the entire content cache for a cell invalidates ~1-2x/day, forcing expensive recomputes. - Computing the token at all requires loading the full history first — so even a cache **hit** pays the full-history load. ## Change (Fix 1 of 3) Adds a new token that content routes and the warm job (other agents' scope) will use instead of `history_token`: - `payloads.CONTENT_VER = "c1"` — a separate content-shape version, so a content-only change need not orphan every other cache kind. - `payloads.content_token(cell_id) -> str` returning `f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date or 'none'}"`. - `climate_store.history_max_date(cell_id) -> str | None` — indexed `MAX(date)` over the `(cell_id, date)` primary key (confirmed in migration `0002_climate_hypertables`), never a full-history load. - `climate.history_max_date(cell_id) -> str | None` — backend dispatcher: Postgres path delegates to the store; the dev/parquet path does a single-column `scan_parquet(...).select(col("date").max())`, not a full frame load. The token keys on the archive's newest **date**, so it survives intra-day top-ups and turns over only when the last archived day genuinely advances (~1x/day) — keeping content pages ≤1 day stale, acceptable for SEO. Fail-soft throughout: a store/DB error buckets to `'none'` rather than raising. Scope kept to `backend/api/payloads.py`, `backend/data/climate_store.py`, `backend/data/climate.py` (+ tests). `content_routes.py` / `warm_cities.py` / `scheduler.py` untouched — owned by the other fixes. ## Tests New unit tests: parquet + gated-Postgres `history_max_date` round-trips (including cell isolation and tail-topup advance), and `content_token` stability (same max date -> identical token across simulated top-ups; changed max date -> new token) plus fail-soft bucketing to `'none'`. Suite green (the 2 unrelated `test_lake_app` failures are a missing `duckdb` in the dev venv, pre-existing).

admin_emi added 1 commit 2026-07-24 19:16:42 +00:00

Add cheap, stable content-page cache token

PR build (required check) / changes (pull_request) Successful in 8s

Details

secrets-guard / encrypted (pull_request) Successful in 6s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 10s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / build-backend (pull_request) Successful in 1m2s

Details

PR build (required check) / gate (pull_request) Successful in 3s

Details

85810c21c5

Content SEO pages keyed derived-payload cache validity on
history_token = PAYLOAD_VER:hist_end, computed from the fully-loaded
~45-year archive. Every hourly tail top-up advanced hist_end and
invalidated the whole content cache for a cell, and computing the token
at all required loading the full history first — so even cache hits paid
the full load.

Add content_token(cell_id) = PAYLOAD_VER:CONTENT_VER:max_date, keyed on
the archive's newest DATE read cheaply without loading history:
climate_store.history_max_date does an indexed MAX(date) over the
(cell_id, date) PK on Postgres; climate.history_max_date dispatches to it
or to a single-column scan of the cached parquet on the dev backend. The
token survives intra-day top-ups and turns over only when the last
archived day advances (~1x/day), keeping content pages <=1 day stale.
Fail-soft: a store/DB error buckets to 'none' rather than raising.

CONTENT_VER ("c1") is a separate content-shape version so a content-only
change need not orphan every other kind's cache.