thermograph/backend/data/cities.py

105 lines
3.5 KiB
Python
Raw Normal View History

SEO: crawlable programmatic climate pages + technical hygiene (#96) * SEO: generate curated city set for crawlable climate pages gen_cities.py reuses the GeoNames index places.py already parses to select the top ~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when it repeats the city name), and writes committed backend/cities.json. cities.py loads it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping for the upcoming hub + sitemap. * SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene - content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt (disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates the home/static pages plus every city, month, and records URL from cities.py). Registered on the app before the StaticFiles mount so the routes win. - templates/base.html.j2: shared layout with unique title/description, self- referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate link) and a footer link graph. - Give each existing page a unique <meta description> (were 5x identical) and a self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page. - Pin jinja2. * SEO: server-rendered per-city climate page (/climate/{slug}) The keystone crawlable page: for a city it snaps to the grid cell, loads the archive (fetching once if missing, self-healing), and renders as real HTML — a 'how today compares' block (grade + percentile per metric from grade_day, tinted by tier), a monthly normals table (climatology at each month's 15th, shown in °F and °C), all-time records (new grading.all_time_records helper), a breadcrumb, Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into the interactive tool + month/records pages. Content-page CSS added to style.css (renamed the table class to avoid colliding with the app's .normals flex row). * SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages Month pages render the exact-month long-tail ('average weather in {city} in {month}') with that month's average high/low, typical p10-p90 range, month-specific records, and prev/next month links. Records pages show all-time record highs/lows per metric with dates (grading.all_time_records). Shared _resolve_city helper; the literal /records route is registered before the {month} param and month names are validated (unknown month -> 404). * SEO: climate hub, weather glossary, and about/methodology pages - /climate: crawlable directory of all ~500 cities grouped by country — the internal-link graph that lets search engines discover every city page. - /glossary + /glossary/{term}: plain-language definitions (climate normal, percentile, temperature anomaly, feels-like, heat index, wind chill, humidity, reanalysis) with DefinedTerm JSON-LD and cross-links into the tool. - /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window, percentile grading) for E-E-A-T. All linked from the shared footer. * SEO: archive warmer, content-page tests, and deploy docs - warm_cities.py: paced, idempotent offline warmer that pre-fetches each city cell's archive so /climate pages serve from cache and a crawl can't burst the archive quota (pages self-heal if hit before warming). - tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap enumerating city/month/records URLs, and that a rendered city page carries the stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/ about routing and 404s. - DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
"""Access to the curated city set (backend/cities.json) that gets crawlable
climate pages. Loaded once, lazily; regenerate the JSON with gen_cities.py."""
import json
import os
Split the backend into domain packages (#217) * Centralize filesystem paths in a single module Add paths.py, which resolves the repo root once and derives the cache, accounts DB, logs, templates, frontend and bundled-city-data locations from it. Replace the 13 per-module `dirname(__file__)/..` anchors with references to it, so a module's location no longer determines where the app reads its data. Env overrides (accounts DB, VAPID, IndexNow) are unchanged; every resolved path is byte-identical to before. Groundwork for moving modules into packages without re-pointing paths. Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62 * Split the backend into domain packages Group the flat backend modules into packages that mirror their concerns: data/ climate, grading, scoring, grid, places, cities, city_events, store web/ app, views, homepage, content, schemas notifications/ notify, digest, push, mailer, discord, discord_interactions, discord_link accounts/ models, users, api_accounts, db core/ metrics, singleton, audit Intra-project imports are rewritten to the package-qualified form. The entry scripts (indexnow, warm_cities, migrate, gen_cities, gen_flavor) and paths.py stay at the backend/ root, and backend/app.py becomes a shim re-exporting web.app:app so the launch target stays `app:app` — run.sh, the systemd units, and CI need no change. Verified: full suite (318) passes, `uvicorn app:app` boots and serves the home/SEO/static/API surfaces, and every root script imports clean. Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
2026-07-20 05:31:03 +00:00
import paths
_PATH = paths.CITIES_JSON
_FLAVOR_PATH = paths.CITIES_FLAVOR_JSON
SEO: crawlable programmatic climate pages + technical hygiene (#96) * SEO: generate curated city set for crawlable climate pages gen_cities.py reuses the GeoNames index places.py already parses to select the top ~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when it repeats the city name), and writes committed backend/cities.json. cities.py loads it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping for the upcoming hub + sitemap. * SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene - content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt (disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates the home/static pages plus every city, month, and records URL from cities.py). Registered on the app before the StaticFiles mount so the routes win. - templates/base.html.j2: shared layout with unique title/description, self- referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate link) and a footer link graph. - Give each existing page a unique <meta description> (were 5x identical) and a self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page. - Pin jinja2. * SEO: server-rendered per-city climate page (/climate/{slug}) The keystone crawlable page: for a city it snaps to the grid cell, loads the archive (fetching once if missing, self-healing), and renders as real HTML — a 'how today compares' block (grade + percentile per metric from grade_day, tinted by tier), a monthly normals table (climatology at each month's 15th, shown in °F and °C), all-time records (new grading.all_time_records helper), a breadcrumb, Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into the interactive tool + month/records pages. Content-page CSS added to style.css (renamed the table class to avoid colliding with the app's .normals flex row). * SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages Month pages render the exact-month long-tail ('average weather in {city} in {month}') with that month's average high/low, typical p10-p90 range, month-specific records, and prev/next month links. Records pages show all-time record highs/lows per metric with dates (grading.all_time_records). Shared _resolve_city helper; the literal /records route is registered before the {month} param and month names are validated (unknown month -> 404). * SEO: climate hub, weather glossary, and about/methodology pages - /climate: crawlable directory of all ~500 cities grouped by country — the internal-link graph that lets search engines discover every city page. - /glossary + /glossary/{term}: plain-language definitions (climate normal, percentile, temperature anomaly, feels-like, heat index, wind chill, humidity, reanalysis) with DefinedTerm JSON-LD and cross-links into the tool. - /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window, percentile grading) for E-E-A-T. All linked from the shared footer. * SEO: archive warmer, content-page tests, and deploy docs - warm_cities.py: paced, idempotent offline warmer that pre-fetches each city cell's archive so /climate pages serve from cache and a crawl can't burst the archive quota (pages self-heal if hit before warming). - tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap enumerating city/month/records URLs, and that a rendered city page carries the stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/ about routing and 404s. - DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
_CITIES: list[dict] | None = None
_BY_SLUG: dict[str, dict] | None = None
_FLAVOR: dict[str, dict] | None = None
Lead climate page titles with the city and the payload (#188) Search Console's first 48 hours showed city-month pages drawing impressions at positions 44-77 and converting almost none of them: the title template spent its width on region and country ("Average weather in London, England, United Kingdom in July"), so Google cut it before the month a searcher was looking for. Titles now lead with the bare city name and put the subject next, which keeps both inside the first ~40 characters: London in July: normal weather & records London climate: daily normals, records & how unusual it is now London weather records: hottest & coldest days since 1980 Add cities.title_name() for the short form. 968 of the 1000 cities have a unique name and render bare; the 32 that don't are qualified with their country code ("London, GB"). Five names — Columbus, Arlington, Aurora, Glendale, Gorakhpur — recur *within* one country, where the country code collides too and would hand two different pages the same title, so those fall back to admin1 ("Columbus, Ohio"). A test asserts every rendered title is unique across the set. Meta descriptions now lead with the page's actual numbers rather than restating the template ("London averages highs of 73°F in July and lows of 38°F in January."), capped at the ~155 characters a SERP shows. The records title derives its year from the loaded history rather than a fixed date, so it can't contradict the page. H1s, URLs, breadcrumbs and JSON-LD are unchanged; og:title and og:description follow <title>/<meta> automatically through base.html.j2. Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
2026-07-19 18:17:13 +00:00
_DUPES: dict[str, dict[str, int]] | None = None
SEO: crawlable programmatic climate pages + technical hygiene (#96) * SEO: generate curated city set for crawlable climate pages gen_cities.py reuses the GeoNames index places.py already parses to select the top ~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when it repeats the city name), and writes committed backend/cities.json. cities.py loads it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping for the upcoming hub + sitemap. * SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene - content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt (disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates the home/static pages plus every city, month, and records URL from cities.py). Registered on the app before the StaticFiles mount so the routes win. - templates/base.html.j2: shared layout with unique title/description, self- referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate link) and a footer link graph. - Give each existing page a unique <meta description> (were 5x identical) and a self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page. - Pin jinja2. * SEO: server-rendered per-city climate page (/climate/{slug}) The keystone crawlable page: for a city it snaps to the grid cell, loads the archive (fetching once if missing, self-healing), and renders as real HTML — a 'how today compares' block (grade + percentile per metric from grade_day, tinted by tier), a monthly normals table (climatology at each month's 15th, shown in °F and °C), all-time records (new grading.all_time_records helper), a breadcrumb, Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into the interactive tool + month/records pages. Content-page CSS added to style.css (renamed the table class to avoid colliding with the app's .normals flex row). * SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages Month pages render the exact-month long-tail ('average weather in {city} in {month}') with that month's average high/low, typical p10-p90 range, month-specific records, and prev/next month links. Records pages show all-time record highs/lows per metric with dates (grading.all_time_records). Shared _resolve_city helper; the literal /records route is registered before the {month} param and month names are validated (unknown month -> 404). * SEO: climate hub, weather glossary, and about/methodology pages - /climate: crawlable directory of all ~500 cities grouped by country — the internal-link graph that lets search engines discover every city page. - /glossary + /glossary/{term}: plain-language definitions (climate normal, percentile, temperature anomaly, feels-like, heat index, wind chill, humidity, reanalysis) with DefinedTerm JSON-LD and cross-links into the tool. - /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window, percentile grading) for E-E-A-T. All linked from the shared footer. * SEO: archive warmer, content-page tests, and deploy docs - warm_cities.py: paced, idempotent offline warmer that pre-fetches each city cell's archive so /climate pages serve from cache and a crawl can't burst the archive quota (pages self-heal if hit before warming). - tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap enumerating city/month/records URLs, and that a rendered city page carries the stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/ about routing and 404s. - DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
def _load() -> list[dict]:
global _CITIES, _BY_SLUG
if _CITIES is None:
with open(_PATH, encoding="utf-8") as f:
_CITIES = json.load(f)
_BY_SLUG = {c["slug"]: c for c in _CITIES}
return _CITIES
def all_cities() -> list[dict]:
return _load()
def all_slugs() -> list[str]:
return [c["slug"] for c in _load()]
def get(slug: str) -> dict | None:
"""The city for a slug, or None (→ 404)."""
_load()
return _BY_SLUG.get(slug)
def flavor(slug: str) -> dict | None:
"""A city's descriptive blurb {extract, url, title} from cities_flavor.json, or
None when we have no confident match (the page renders fine without it)."""
global _FLAVOR
if _FLAVOR is None:
try:
with open(_FLAVOR_PATH, encoding="utf-8") as f:
_FLAVOR = json.load(f)
except (OSError, ValueError):
_FLAVOR = {}
return _FLAVOR.get(slug)
Lead climate page titles with the city and the payload (#188) Search Console's first 48 hours showed city-month pages drawing impressions at positions 44-77 and converting almost none of them: the title template spent its width on region and country ("Average weather in London, England, United Kingdom in July"), so Google cut it before the month a searcher was looking for. Titles now lead with the bare city name and put the subject next, which keeps both inside the first ~40 characters: London in July: normal weather & records London climate: daily normals, records & how unusual it is now London weather records: hottest & coldest days since 1980 Add cities.title_name() for the short form. 968 of the 1000 cities have a unique name and render bare; the 32 that don't are qualified with their country code ("London, GB"). Five names — Columbus, Arlington, Aurora, Glendale, Gorakhpur — recur *within* one country, where the country code collides too and would hand two different pages the same title, so those fall back to admin1 ("Columbus, Ohio"). A test asserts every rendered title is unique across the set. Meta descriptions now lead with the page's actual numbers rather than restating the template ("London averages highs of 73°F in July and lows of 38°F in January."), capped at the ~155 characters a SERP shows. The records title derives its year from the loaded history rather than a fixed date, so it can't contradict the page. H1s, URLs, breadcrumbs and JSON-LD are unchanged; og:title and og:description follow <title>/<meta> automatically through base.html.j2. Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
2026-07-19 18:17:13 +00:00
def title_name(city: dict) -> str:
"""The shortest label that still names this city unambiguously — for page
titles, where display_name()'s region + country spend the width Google gives
us before it ever reaches the payload ("… in July").
'Seattle' for the 968 cities whose name is unique; 'London, GB' when the name
recurs in another country; 'Columbus, Ohio' when it recurs *within* one
country, where the country code would collide too and hand two different
pages the same title.
"""
name = city["name"]
dupes = _dupe_names().get(name)
if not dupes:
return name
if dupes.get(city.get("country_code")) == 1:
return f"{name}, {city['country_code']}"
return f"{name}, {city['admin1']}" if city.get("admin1") else name
def _dupe_names() -> dict[str, dict[str, int]]:
"""{name: {country_code: how many cities share both}} for duplicated names
only. Built once alongside the city list."""
global _DUPES
if _DUPES is None:
counts: dict[str, dict[str, int]] = {}
for c in _load():
counts.setdefault(c["name"], {})
cc = c.get("country_code")
counts[c["name"]][cc] = counts[c["name"]].get(cc, 0) + 1
_DUPES = {n: by_cc for n, by_cc in counts.items() if sum(by_cc.values()) > 1}
return _DUPES
SEO: crawlable programmatic climate pages + technical hygiene (#96) * SEO: generate curated city set for crawlable climate pages gen_cities.py reuses the GeoNames index places.py already parses to select the top ~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when it repeats the city name), and writes committed backend/cities.json. cities.py loads it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping for the upcoming hub + sitemap. * SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene - content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt (disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates the home/static pages plus every city, month, and records URL from cities.py). Registered on the app before the StaticFiles mount so the routes win. - templates/base.html.j2: shared layout with unique title/description, self- referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate link) and a footer link graph. - Give each existing page a unique <meta description> (were 5x identical) and a self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page. - Pin jinja2. * SEO: server-rendered per-city climate page (/climate/{slug}) The keystone crawlable page: for a city it snaps to the grid cell, loads the archive (fetching once if missing, self-healing), and renders as real HTML — a 'how today compares' block (grade + percentile per metric from grade_day, tinted by tier), a monthly normals table (climatology at each month's 15th, shown in °F and °C), all-time records (new grading.all_time_records helper), a breadcrumb, Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into the interactive tool + month/records pages. Content-page CSS added to style.css (renamed the table class to avoid colliding with the app's .normals flex row). * SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages Month pages render the exact-month long-tail ('average weather in {city} in {month}') with that month's average high/low, typical p10-p90 range, month-specific records, and prev/next month links. Records pages show all-time record highs/lows per metric with dates (grading.all_time_records). Shared _resolve_city helper; the literal /records route is registered before the {month} param and month names are validated (unknown month -> 404). * SEO: climate hub, weather glossary, and about/methodology pages - /climate: crawlable directory of all ~500 cities grouped by country — the internal-link graph that lets search engines discover every city page. - /glossary + /glossary/{term}: plain-language definitions (climate normal, percentile, temperature anomaly, feels-like, heat index, wind chill, humidity, reanalysis) with DefinedTerm JSON-LD and cross-links into the tool. - /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window, percentile grading) for E-E-A-T. All linked from the shared footer. * SEO: archive warmer, content-page tests, and deploy docs - warm_cities.py: paced, idempotent offline warmer that pre-fetches each city cell's archive so /climate pages serve from cache and a crawl can't burst the archive quota (pages self-heal if hit before warming). - tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap enumerating city/month/records URLs, and that a rendered city page carries the stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/ about routing and 404s. - DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
def display_name(city: dict) -> str:
"""Human label: 'Seattle, Washington, United States' (drops repeated admin1)."""
parts = [city["name"]]
if city.get("admin1") and city["admin1"] != city["name"]:
parts.append(city["admin1"])
if city.get("country"):
parts.append(city["country"])
return ", ".join(parts)
def by_country() -> dict[str, list[dict]]:
"""Cities grouped by country, both the countries and the cities within each
ordered alphabetically for the /climate hub's crawlable link graph + search."""
SEO: crawlable programmatic climate pages + technical hygiene (#96) * SEO: generate curated city set for crawlable climate pages gen_cities.py reuses the GeoNames index places.py already parses to select the top ~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when it repeats the city name), and writes committed backend/cities.json. cities.py loads it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping for the upcoming hub + sitemap. * SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene - content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt (disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates the home/static pages plus every city, month, and records URL from cities.py). Registered on the app before the StaticFiles mount so the routes win. - templates/base.html.j2: shared layout with unique title/description, self- referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate link) and a footer link graph. - Give each existing page a unique <meta description> (were 5x identical) and a self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page. - Pin jinja2. * SEO: server-rendered per-city climate page (/climate/{slug}) The keystone crawlable page: for a city it snaps to the grid cell, loads the archive (fetching once if missing, self-healing), and renders as real HTML — a 'how today compares' block (grade + percentile per metric from grade_day, tinted by tier), a monthly normals table (climatology at each month's 15th, shown in °F and °C), all-time records (new grading.all_time_records helper), a breadcrumb, Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into the interactive tool + month/records pages. Content-page CSS added to style.css (renamed the table class to avoid colliding with the app's .normals flex row). * SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages Month pages render the exact-month long-tail ('average weather in {city} in {month}') with that month's average high/low, typical p10-p90 range, month-specific records, and prev/next month links. Records pages show all-time record highs/lows per metric with dates (grading.all_time_records). Shared _resolve_city helper; the literal /records route is registered before the {month} param and month names are validated (unknown month -> 404). * SEO: climate hub, weather glossary, and about/methodology pages - /climate: crawlable directory of all ~500 cities grouped by country — the internal-link graph that lets search engines discover every city page. - /glossary + /glossary/{term}: plain-language definitions (climate normal, percentile, temperature anomaly, feels-like, heat index, wind chill, humidity, reanalysis) with DefinedTerm JSON-LD and cross-links into the tool. - /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window, percentile grading) for E-E-A-T. All linked from the shared footer. * SEO: archive warmer, content-page tests, and deploy docs - warm_cities.py: paced, idempotent offline warmer that pre-fetches each city cell's archive so /climate pages serve from cache and a crawl can't burst the archive quota (pages self-heal if hit before warming). - tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap enumerating city/month/records URLs, and that a rendered city page carries the stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/ about routing and 404s. - DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
groups: dict[str, list[dict]] = {}
for c in _load():
key = c.get("country") or c.get("country_code") or "Other"
groups.setdefault(key, []).append(c)
for v in groups.values():
v.sort(key=lambda x: x["name"].lower())
return dict(sorted(groups.items(), key=lambda kv: kv[0].lower()))