Commit graph

21 commits

Author SHA1 Message Date
Emi Griffith
669e52a0d9 Merge remote-tracking branch 'forgejo/qa/dry-trace' into qa/frontend-qa-batch
All checks were successful
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 58s
PR build (required check) / build-backend (pull_request) Successful in 1m16s
PR build (required check) / gate (pull_request) Successful in 2s
# Conflicts:
#	frontend/static/day.js
2026-07-24 16:10:39 -07:00
Emi Griffith
45f4d18a4e Merge remote-tracking branch 'forgejo/qa/https-origin-live' into qa/frontend-qa-batch 2026-07-24 16:09:44 -07:00
Emi Griffith
71d1ee95db Merge remote-tracking branch 'forgejo/qa/vapid-rotation' into qa/frontend-qa-batch 2026-07-24 16:09:44 -07:00
Emi Griffith
9e93d1bfdf Merge remote-tracking branch 'forgejo/qa/recentday-gate' into qa/frontend-qa-batch 2026-07-24 16:09:44 -07:00
Emi Griffith
fedd18a74b Make trace-rain days read consistently across every surface
Since the dry/rain grading split moved to precip > 0, a sub-0.01" reanalysis
"trace" day is graded as a rain tier but its depth rounds to 0.00" / 0 mm, and
several surfaces still treated it as dry — a day would read "0.0" · Light rain,
N days since rain" at once. Align every consumer of a graded precip day with the
> 0 split so a trace day reads as the (very light) rain it was graded to be.

- Dry streak: dry_streaks and longest_dry_streak reset on any rain (> 0), not
  the 0.01" rain-frequency line, so a trace day breaks the streak and its
  "days since rain" no longer keeps climbing. (The rain_freq climatology stat
  keeps the 0.01" measurable-rain convention.)
- Display: new fmtPrecipTier() prints "trace" for a rain-tier day whose depth
  rounds to zero, rather than a bone-dry "0.00". Used on the calendar tooltip,
  the day-page observation + ladder marker, and the recent/forecast table.
- weatherType() decides wet/dry from the grade class when it has it (a rain tier
  means it rained even at trace depth), falling back to depth > 0; callers on the
  calendar and day page pass the class.
- Recent-table and chart precip dots key their dry-vs-rain rendering on the grade
  class instead of value > 0, so a trace day tints as rain, not dry.

Rain chart fan: the precipitation fan still used the pre-split colour map,
painting the whole 90-99 rain-day-percentile region one shade and 75-90 a tier
too dark. _band_stats emits a p95 mark (additive; unused by the temperature fan)
and RAIN_FAN remaps to the eight tiers -- 95-99 Severe, 90-95 Very Heavy, 60-90
Heavy -- with a p95 fallback in pget so an older cached payload still renders.
2026-07-24 16:07:14 -07:00
Emi Griffith
72e141afd3 Emit https origin for the public host in canonical/og/sitemap URLs
The SSR frontend's _origin() (canonical, og:url, og:image, robots, sitemap
<loc>) and the backend content API's _origin() (the jsonld url folded into
each payload) both trusted x-forwarded-proto / request.url.scheme. Behind
Caddy — which terminates TLS and reverse-proxies plain HTTP — those read
"http", so on the HTTPS-only public site every absolute URL emitted http://,
which 308-redirects to https://: canonicals self-conflict and the sitemap
lists redirecting URLs.

Take the scheme from the configured public origin (THERMOGRAPH_BASE_URL, set
per-host in the deploy env) when the request arrives on that host, so prod and
beta emit https. localhost and LAN dev never match the public host and keep the
observed scheme, so plain-HTTP development is unchanged; the frontend keeps its
X-Forwarded-Host precedence for the proxy-fallback path.

Tests assert canonical/og:url/og:image, robots Sitemap and every sitemap <loc>
are https for the public host (and via X-Forwarded-Host), that a LAN host stays
http, and that the backend payload's jsonld url is likewise https.
2026-07-24 16:04:42 -07:00
Emi Griffith
078aaab759 Drop the in-progress local day from the Open-Meteo recent/forecast fallback
Open-Meteo's daily high/low for the current day aggregates only the hours
elapsed so far, so grading it reads a still-unfolding day as complete and
produces spurious extremes — a cool morning served as a 1st-percentile
record-low high. The MET Norway primary already guards this with its
diurnal-coverage gate; the Open-Meteo fallback had no equivalent, so a
fleet-wide failover onto it exposed the bug.

Use the utc_offset_seconds Open-Meteo reports for a timezone=auto request to
identify the cell's local today and exclude it from the bundle. Past days are
complete and future days are whole-day forecasts, so only today is dropped;
the day lands in the record once it is over. When no offset is reported the
guard is skipped rather than guessing a date.
2026-07-24 16:01:08 -07:00
Emi Griffith
35cf1036d4 push: re-subscribe on VAPID key rotation and prune dead 401/403 rows
After a VAPID keypair rotation, subscribers minted under the old key
silently stopped receiving notifications while the UI still reported
"on", and the dead rows were never cleaned up.

Frontend (enable): a browser holding an existing PushSubscription never
re-subscribed, so it kept using the old applicationServerKey. Now the
current server VAPID key is always fetched and compared against the
subscription's baked-in key; on a mismatch the stale subscription is
unsubscribed and re-created with the new key. The matching-key path is
unchanged.

Backend (send): a rotated key makes the push service reject delivery
with 401/403, which returned "error" and left the row in place forever.
Treat 401/403 as permanently dead alongside 404/410 so the caller prunes
the row. Genuinely transient failures (rate limits, 5xx) still return
"error" and keep the row.

Also correct the default VAPID contact to mailto:admin@thermograph.org
(the .app domain was a typo); the env override is unchanged.

Extends tests/notifications/test_push.py with the send() status mapping
and the contact default.
2026-07-24 15:48:25 -07:00
Emi Griffith
ca589545a6 Return 422 for malformed date query param in grade/day endpoints
api_grade and api_day parsed the `date` query param with an unguarded
datetime.date.fromisoformat, so a malformed or non-calendar value
(notadate, 2026-13-40, 2026-02-30) raised ValueError and surfaced as a
500. Route the parse through a _parse_target_date helper that maps the
ValueError to HTTPException(422); absent/empty and valid dates behave
exactly as before. Add a route-level test asserting 422 on bad dates.
2026-07-24 15:43:35 -07:00
emi
248eeee110 Gate MET forecast days on diurnal sample coverage (#63)
All checks were successful
secrets-guard / encrypted (push) Successful in 10s
shell-lint / shellcheck (push) Successful in 12s
secrets-guard / encrypted (pull_request) Successful in 10s
PR build (required check) / changes (pull_request) Successful in 14s
shell-lint / shellcheck (pull_request) Successful in 11s
PR build (required check) / validate-observability (pull_request) Has been skipped
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m41s
Deploy backend to LAN dev server / build (push) Successful in 1m48s
PR build (required check) / build-frontend (pull_request) Successful in 1m23s
PR build (required check) / build-backend (pull_request) Successful in 1m38s
PR build (required check) / gate (pull_request) Successful in 3s
Deploy backend to LAN dev server / deploy (push) Successful in 33s
2026-07-24 22:23:51 +00:00
emi
cda6e75b38 data/climate: fix geocode_nominatim's NameError on every call (P0, live) (#51)
All checks were successful
secrets-guard / encrypted (push) Successful in 14s
shell-lint / shellcheck (push) Successful in 13s
PR build (required check) / changes (pull_request) Successful in 10s
secrets-guard / encrypted (pull_request) Successful in 10s
shell-lint / shellcheck (pull_request) Successful in 10s
PR build (required check) / validate-observability (pull_request) Has been skipped
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m24s
Deploy backend to LAN dev server / build (push) Successful in 1m49s
PR build (required check) / build-frontend (pull_request) Successful in 1m21s
PR build (required check) / build-backend (pull_request) Successful in 1m35s
PR build (required check) / gate (pull_request) Successful in 3s
Deploy backend to LAN dev server / deploy (push) Successful in 22s
2026-07-24 19:45:49 +00:00
emi
bf0aaf1ec0 Pre-warm the SEO content derived-store off-request (#49)
All checks were successful
secrets-guard / encrypted (push) Successful in 10s
shell-lint / shellcheck (push) Successful in 13s
Deploy backend to LAN dev server / build (push) Successful in 1m40s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m47s
Deploy backend to LAN dev server / deploy (push) Successful in 30s
After a cell's archive advances, the first content request recomputes a 45yr payload cold. Add warm_cities.warm_content — idempotent, cache-only, bounded — that pre-populates the content derived-store under the same (kind,key,content_token) the routes read, wired into the leader-gated notifier loop (50 cities/30-min tick) so the ~1000-city set stays warm within hours of each daily advance.
2026-07-24 19:31:08 +00:00
emi
6fef57f05f Check derived store before loading history on content routes (#48)
Some checks failed
Deploy backend to LAN dev server / deploy (push) Blocked by required conditions
Build + push backend image (Forgejo registry) / build-push (push) Has been cancelled
Deploy backend to LAN dev server / build (push) Has been cancelled
shell-lint / shellcheck (push) Has been cancelled
secrets-guard / encrypted (push) Has been cancelled
The three content handlers loaded the full ~45yr archive before checking the cache, so even hits paid the load. Restructure to resolve cell + compute content_token cheaply first, check the store, and load history only inside the miss-only build() closure. Cache hits and 304s now return without touching the archive. Keys/kinds/ETag unchanged.
2026-07-24 19:30:57 +00:00
emi
a4fa72639f Add cheap, stable content-page cache token (#47)
Some checks failed
Deploy backend to LAN dev server / deploy (push) Blocked by required conditions
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 7s
Build + push backend image (Forgejo registry) / build-push (push) Has been cancelled
Deploy backend to LAN dev server / build (push) Has been cancelled
Content pages keyed cache validity on the full-history hist_end, invalidating ~1-2x/day and paying a 45yr load even to compute the token. Add content_token(cell_id) = PAYLOAD_VER:CONTENT_VER:max_date, backed by an indexed MAX(date) (climate_store.history_max_date / climate.history_max_date), so it survives intra-day top-ups and needs no full-history load.
2026-07-24 19:30:38 +00:00
emi
4ff113f2cf Stop logging /healthz + internal SSR hop, truncate logged IPs (#35)
Some checks failed
secrets-guard / encrypted (push) Has been cancelled
shell-lint / shellcheck (push) Has been cancelled
Deploy backend to LAN dev server / build (push) Successful in 1m22s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m24s
Deploy backend to LAN dev server / deploy (push) Successful in 19s
2026-07-24 19:28:47 +00:00
Emi Griffith
30b0c70bbb Reconcile: merge main (shellcheck guard, Go daemon) into dev
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / changes (pull_request) Successful in 8s
shell-lint / shellcheck (pull_request) Successful in 9s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m9s
PR build (required check) / build-backend (pull_request) Successful in 1m44s
PR build (required check) / gate (pull_request) Successful in 2s
main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:

- compose + Swarm stack: lake AND daemon are sibling services; backend env
  carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
  daemon), duckdb stays (the lake role's engine).

Merged tree: backend suite green, both compose configs validate.
2026-07-23 17:06:52 -07:00
emi
6396d553d2 Own /state in the image; lake cache degrades instead of failing (#25)
All checks were successful
secrets-guard / encrypted (push) Successful in 12s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 57s
Deploy backend to LAN dev server / build (push) Successful in 1m13s
Deploy backend to LAN dev server / deploy (push) Successful in 18s
2026-07-23 22:55:41 +00:00
emi
2d3f37c474 daemon: move the Discord gateway and scheduler out of the web process into Go (#21)
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 8s
Sync infra to hosts / sync-prod (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 8s
shell-lint / shellcheck (push) Successful in 10s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m14s
Deploy backend to beta VPS / deploy (push) Successful in 2m4s
The gateway bot and APScheduler were long-lived stateful I/O loops running
inside the async web app under a leader election. They move into a single Go
binary that owns ONLY that I/O -- websocket, RESUME, heartbeat, backoff, timers.

It owns no grading logic. Anything needing data calls back over a new
internal-only surface (/internal/discord/grade, /internal/jobs/*). Grading
depends on polars and the parquet cache; reimplementing it in Go would let the
bot's grades drift from the API's. The grade route returns gateway-ready JSON
and Go relays the bytes verbatim.

The binary ships in the backend image and runs as a second compose service off
the same tag, so the two ends of the /internal/* contract can never skew.
deploy.sh rolls daemon alongside backend -- without that the service would never
be created, since a single-service deploy uses --no-deps. It also probes the
image first and skips the daemon when rolling a tag that predates the binary:
infra tracks main while image tags are env-staged, so a host can legitimately be
asked to roll an older backend image, and creating the service anyway would
leave a container crash-looping on a missing binary.

replicas: 1 with order: stop-first replaces the leader election -- Discord
permits one gateway connection per bot token.

THERMOGRAPH_INTERNAL_TOKEN is optional: both ends derive it from
THERMOGRAPH_AUTH_SECRET via HMAC under a domain-separation label, so this needs
no new vault entry. The derivation is pinned to a shared cross-language test
vector asserted on both sides, so drift fails CI instead of 401ing every call.
Fail closed when neither secret is set.

Improvements over the Python: a close intended for RESUME uses 4000 rather than
1000 (Discord invalidates a session closed 1000, so the old default defeated its
own resume); MESSAGE_CREATE runs on a bounded worker pool; and a malformed HELLO
returns an error rather than a clean reconnect, which would otherwise reset
backoff and hot-loop against the gateway.

365 Python tests pass; Go build/vet/test -race clean; shellcheck 0 findings.
2026-07-23 22:49:54 +00:00
emi
370a4ffdc1 ERA5 lake: bucket-hosted history primary + SQL indexer service (#15)
All checks were successful
secrets-guard / encrypted (push) Successful in 10s
Deploy backend to LAN dev server / build (push) Successful in 1m13s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m39s
Deploy backend to LAN dev server / deploy (push) Successful in 34s
2026-07-23 21:20:35 +00:00
emi
790b6bf0dc Migrate off the Open-Meteo weather API (Phases 0-4)
All checks were successful
secrets-guard / encrypted (push) Successful in 8s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m4s
Deploy backend to LAN dev server / build (push) Successful in 1m12s
Deploy backend to LAN dev server / deploy (push) Successful in 17s
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 54s
PR build (required check) / gate (pull_request) Successful in 3s
Steady-state usage of the Open-Meteo API is removed; it remains only as a dormant fallback. Geocoding -> local GeoNames + Nominatim; wind gusts -> Meteostat; history -> NASA POWER (curated cells seeded with keyless ERA5); recent+forecast -> NASA range + MET Norway. Plus a drift-check tool comparing NASA vs Open-Meteo. All keyless, no new infra. Backend suite green in-image (gate).
2026-07-23 14:34:51 +00:00
Emi Griffith
a4be7066e5 Subtree-merge thermograph-backend (origin/main) into backend/
git-subtree-dir: backend
git-subtree-mainline: 6723fc0326
git-subtree-split: 83c2e05b96
2026-07-22 22:01:11 -07:00