Check derived store before loading history on content routes #48

Merged

admin_emi merged 2 commits from fix/content-store-first into dev

2026-07-24 19:30:58 +00:00

admin_emi commented

2026-07-24 19:23:20 +00:00

Owner

Problem

The three SEO content handlers in backend/api/content_routes.py (city / month / records) resolved the city by loading its full ~45-year archive up front, only then checking the derived-store cache. So even a cache HIT paid the full-history load — the 2–6s content-page latency seen on prod.

Fix (Fix #2 of 3)

Restructure so the cheap path runs first:

cities.get(slug) -> 404 (dict lookup)
grid.snap(lat, lon) (pure, no history)
payloads.content_token(cell["id"]) — the cheap indexed MAX(date) token from Fix #1, no history load
Same cache key/kind/token as before

The expensive work — _load_history(cell) (raises 503 while warming), plus climate.get_recent_forecast for the city handler — moves into the build() closure that _cached only invokes on a miss. A store hit, or a 304, now returns replayed bytes without ever touching the archive. Cache keys, kinds, and ETag behavior are unchanged.

_resolve_city (its only callers were these three handlers) is replaced by _city_cell (cheap) + _load_history (expensive, miss-only).

Tests

New backend/tests/api/test_content_routes.py mirrors the web/test_api.py client pattern with call counters on climate.load_cached_history / get_history and a stubbed history_max_date. Core regression guard: a first request loads history once, a second identical request is served from the store and loads history zero more times; 304 revalidation loads nothing; plus city/month/records payload shapes and unknown-slug / unknown-month 404s.

backend/tests full suite: 394 passed, 8 skipped.

Stacks on #47 (Fix #1, fix/content-token); this branch is based on it, so review/merge #47 first.

## Problem The three SEO content handlers in `backend/api/content_routes.py` (city / month / records) resolved the city by loading its full ~45-year archive up front, only *then* checking the derived-store cache. So even a cache HIT paid the full-history load — the 2–6s content-page latency seen on prod. ## Fix (Fix #2 of 3) Restructure so the cheap path runs first: 1. `cities.get(slug)` -> 404 (dict lookup) 2. `grid.snap(lat, lon)` (pure, no history) 3. `payloads.content_token(cell["id"])` — the cheap indexed `MAX(date)` token from Fix #1, no history load 4. Same cache key/kind/token as before The expensive work — `_load_history(cell)` (raises 503 while warming), plus `climate.get_recent_forecast` for the city handler — moves into the `build()` closure that `_cached` only invokes on a miss. A store hit, or a 304, now returns replayed bytes without ever touching the archive. Cache keys, kinds, and ETag behavior are unchanged. `_resolve_city` (its only callers were these three handlers) is replaced by `_city_cell` (cheap) + `_load_history` (expensive, miss-only). ## Tests New `backend/tests/api/test_content_routes.py` mirrors the `web/test_api.py` client pattern with call counters on `climate.load_cached_history` / `get_history` and a stubbed `history_max_date`. Core regression guard: a first request loads history once, a second identical request is served from the store and loads history **zero** more times; 304 revalidation loads nothing; plus city/month/records payload shapes and unknown-slug / unknown-month 404s. `backend/tests` full suite: 394 passed, 8 skipped. Stacks on #47 (Fix #1, `fix/content-token`); this branch is based on it, so review/merge #47 first.

admin_emi added 2 commits 2026-07-24 19:23:21 +00:00

Add cheap, stable content-page cache token

PR build (required check) / changes (pull_request) Successful in 8s

Details

secrets-guard / encrypted (pull_request) Successful in 6s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 10s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / build-backend (pull_request) Successful in 1m2s

Details

PR build (required check) / gate (pull_request) Successful in 3s

Details

85810c21c5

Content SEO pages keyed derived-payload cache validity on
history_token = PAYLOAD_VER:hist_end, computed from the fully-loaded
~45-year archive. Every hourly tail top-up advanced hist_end and
invalidated the whole content cache for a cell, and computing the token
at all required loading the full history first — so even cache hits paid
the full load.

Add content_token(cell_id) = PAYLOAD_VER:CONTENT_VER:max_date, keyed on
the archive's newest DATE read cheaply without loading history:
climate_store.history_max_date does an indexed MAX(date) over the
(cell_id, date) PK on Postgres; climate.history_max_date dispatches to it
or to a single-column scan of the cached parquet on the dev backend. The
token survives intra-day top-ups and turns over only when the last
archived day advances (~1x/day), keeping content pages <=1 day stale.
Fail-soft: a store/DB error buckets to 'none' rather than raising.

CONTENT_VER ("c1") is a separate content-shape version so a content-only
change need not orphan every other kind's cache.

Check derived store before loading history on content routes

PR build (required check) / changes (pull_request) Successful in 8s

Details

secrets-guard / encrypted (pull_request) Successful in 6s

Details

shell-lint / shellcheck (pull_request) Successful in 10s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / build-backend (pull_request) Successful in 52s

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

9d41236ec5

The three SEO content handlers (city / month / records) resolved the city by loading its full ~45-year archive up front, only then checking the derived-store cache, so even a cache HIT paid the full-history load (2-6s on prod).

Restructure so the cheap path runs first: city lookup + grid snap + the cheap content_token (indexed MAX(date), no history load) produce the cache key/ETag, and the expensive load (history, recent forecast) moves into the build() closure that _cached only calls on a miss. A store hit, or a 304, now returns without ever touching the archive. Cache keys/kinds/tokens are unchanged.

Add route tests asserting a second identical request loads history zero more times, that 304 revalidation loads nothing, plus payload/404 coverage.