web/worker: add a process-level liveness heartbeat #80

Merged

admin_emi merged 18 commits from fix/web-worker-heartbeat into dev 2026-07-25 04:13:50 +00:00

Conversation 0 Commits 18 Files changed 52 +6671 -55

admin_emi commented 2026-07-25 03:53:08 +00:00

Owner

Copy link

ProdWebContainerSilent and ProdWorkerContainerSilent fired repeatedly for ~7 hours on 2026-07-24/25 against a healthy prod. Root cause: both key on Docker container-stdout volume, which isn't a valid liveness signal for either role — web's real access logging lives in the file-tailed app-json source, not stdout, and worker's stdout is unconditionally dropped by Alloy's own relabel rule (its stdout is pure /healthz-poll noise by design). ProdWorkerContainerSilent in particular can never receive data, redeploy or not.

observability/README.md's "Known gaps" section already names the fix: give each near-silent-stdout daemon a real audit.log_heartbeat() beat, same pattern the subscription notifier already uses.

What this does

• backend/web/app.py: new _heartbeat_loop/_start_heartbeat, mirroring notify.py's own _STOP/loop/start() shape. Wired into _lifespan() next to the existing notifier gate. Tags the beat daemon=ROLE (web/worker/all), gated by THERMOGRAPH_ENABLE_HEARTBEAT (default on).
• Deliberately unguarded across every uvicorn worker process and every autoscaled web replica — a beat is a single local file append (no upstream quota, no DB, no lock), unlike the notifier's leader-election, which exists specifically to stop N workers hammering Open-Meteo/DB N-fold.
• worker gets its own beat rather than relying solely on the notifier's existing daemon="subscription-notifier" heartbeat, because the notifier thread only starts if it wins leader election — during that gap the container can be up and healthy with no notifier beat yet. The two together also give a differential signal: both silent → container down; only the notifier one silent → notifier/leader-election specifically wedged.
• Test gating: THERMOGRAPH_ENABLE_HEARTBEAT=0 added to tests/conftest.py and docker-compose.test.yml, next to the existing notifier gate.
• New tests/web/test_heartbeat.py.

Not in this PR

The corresponding observability/grafana/provisioning/alerting/rules-prod.yml change (new ProdWebHeartbeatMissing/ProdWorkerHeartbeatMissing rules, pausing the two broken ones) ships separately, after confirming these beats are actually flowing on beta/prod — deploying the new rules before beats exist would make them immediately and correctly fire on "no data."

Tests

pytest tests: 426 passed, 8 skipped, 2 pre-existing unrelated failures (missing duckdb, not touched by this change).

`ProdWebContainerSilent` and `ProdWorkerContainerSilent` fired repeatedly for ~7 hours on 2026-07-24/25 against a healthy prod. Root cause: both key on Docker container-stdout volume, which isn't a valid liveness signal for either role — `web`'s real access logging lives in the file-tailed `app-json` source, not stdout, and `worker`'s stdout is unconditionally dropped by Alloy's own relabel rule (its stdout is pure `/healthz`-poll noise by design). `ProdWorkerContainerSilent` in particular can never receive data, redeploy or not. `observability/README.md`'s "Known gaps" section already names the fix: give each near-silent-stdout daemon a real `audit.log_heartbeat()` beat, same pattern the subscription notifier already uses. ## What this does - `backend/web/app.py`: new `_heartbeat_loop`/`_start_heartbeat`, mirroring `notify.py`'s own `_STOP`/loop/`start()` shape. Wired into `_lifespan()` next to the existing notifier gate. Tags the beat `daemon=ROLE` (`web`/`worker`/`all`), gated by `THERMOGRAPH_ENABLE_HEARTBEAT` (default on). - Deliberately unguarded across every uvicorn worker process and every autoscaled `web` replica — a beat is a single local file append (no upstream quota, no DB, no lock), unlike the notifier's leader-election, which exists specifically to stop N workers hammering Open-Meteo/DB N-fold. - `worker` gets its own beat rather than relying solely on the notifier's existing `daemon="subscription-notifier"` heartbeat, because the notifier thread only starts if it wins leader election — during that gap the container can be up and healthy with no notifier beat yet. The two together also give a differential signal: both silent → container down; only the notifier one silent → notifier/leader-election specifically wedged. - Test gating: `THERMOGRAPH_ENABLE_HEARTBEAT=0` added to `tests/conftest.py` and `docker-compose.test.yml`, next to the existing notifier gate. - New `tests/web/test_heartbeat.py`. ## Not in this PR The corresponding `observability/grafana/provisioning/alerting/rules-prod.yml` change (new `ProdWebHeartbeatMissing`/`ProdWorkerHeartbeatMissing` rules, pausing the two broken ones) ships separately, after confirming these beats are actually flowing on beta/prod — deploying the new rules before beats exist would make them immediately and correctly fire on "no data." ## Tests `pytest tests`: 426 passed, 8 skipped, 2 pre-existing unrelated failures (missing `duckdb`, not touched by this change).

admin_emi added 18 commits 2026-07-25 03:53:09 +00:00

Merge pull request 'Promote dev to main (deploy beta): ERA5 lake stack' (#26) from dev into main 755886e244

frontend: rewrite the SSR content service in Go (#28) ... 92e74c585a

Ports frontend/ (Jinja2/FastAPI, ~1180 LOC) to Go with html/template. No
climate math, no DB, no auth -- every route fetches from the backend's
/content/* API.

Verified with a golden-HTML diff, not just unit tests: both the Python
original and the Go rewrite were run against the same committed fixtures
and every route compared byte-for-byte, confirmed programmatically. That
process caught defects unit tests alone missed, since map[string]any has
no compile-time field check:

- Render-context keys were snake_case throughout while the templates read
  PascalCase fields. A missing map key doesn't error, it silently renders
  empty -- title, meta description, canonical URL, OpenGraph tags, and the
  homepage's entire ranked list were blank on every page despite every
  route returning 200. Fixed by renaming every key to match each
  template's own documented field contract, and passing API structs
  straight through wherever their fields already matched (removes a whole
  layer of future drift risk).
- Three pages 500'd: ToolHref needed a composed href, not a bare
  "lat,lon" fragment; the records table needed the raw API struct.
- JSON-LD was double-encoded: <script type="application/ld+json"> is
  JAVASCRIPT context to html/template's escaper regardless of the
  script's type attribute, so template.HTML gets re-escaped as a quoted
  JS string. Needed template.JS. The glossary term page's JSON-LD was
  never built at all -- added.
- html/template silently strips literal HTML and JS comments from parsed
  output (verified in isolation) -- both need a FuncMap function
  returning template.HTML/template.JS to survive.

Packaging: 187MB -> 22.6MB. Two defects caught before reaching a host: the
Swarm stack's entrypoint override with no explicit command drops the
image's CMD entirely (every deploy would have exited 127), and
COPY --chown by name fails under the classic Docker builder on Alpine.
Both fixed.

go build/vet/test -race clean; docker build passes its embedded test step
under both BuildKit and the classic builder; shellcheck 0 findings.

Merge pull request 'Promote dev to main: lake extension bake + daemon healthcheck fixes' (#31) from dev into main b83bfdbd67

deploy.sh: fix the daemon-binary probe, which always dropped daemon (#33) ... 979653f407

docker compose config --images daemon does not filter to the named service
on this host's Compose v5.3.1 -- it prints every service's image, one per
line, in file order, so `| head -1` was silently grabbing db's image
(timescaledb) instead of daemon's. The probe then always found no
/usr/local/bin/thermograph-daemon in a Postgres image and dropped daemon
from every backend deploy, regardless of what the real backend image
contained.

Fixed by building the image reference directly from the same vars
docker-compose.yml's daemon.image: already interpolates, instead of going
through docker compose config at all.

Reproduced against beta directly: the old sequence selected the wrong
image; the new construction resolves correctly and the binary probe
passes.

frontend: fetch City() concurrently with the page's primary API call (#37) ... da82abda27

MonthPage and RecordsPage each made two backend calls sequentially
(CityMonth/CityRecords, then City) where the second never depended on the
first's result. fetchWithCity launches both concurrently via goroutines
and a WaitGroup.

Verified live against a stub with an injected 400ms delay on both
endpoints: city page (1 call) and month/records pages (2 calls each) all
cost ~0.404s now, not double for the two-call pages.

Error priority preserved exactly: primary's error wins even when City
also fails, matching the old sequential code. One trade-off: City() is
now always launched even on a request about to 404 from primary, costing
one extra cheap lookup on that rare path.

Tests include a deterministic concurrency proof via rendezvous channels
(the old sequential code would deadlock this test, not just run it
slower). Full suite green under -race -count=2.

infra/forgejo: add resource limits and a leaner CI job image (#38) dcf15ea572

Merge pull request 'Promote dev to main: daemon roll + Iceberg-backed lake query' (#41) from dev into main 88f5d0cce0

infra/forgejo: raise LAN runner concurrency to 8 (#43) 81c579ddcb

Merge pull request 'Promote dev to main: Iceberg-backed lake query' (#45) from dev into main 6f5682b7d7

Merge pull request 'Promote dev to main: geocode P0 fix + ops tooling + privacy logging' (#52) from dev into main 0935eea394

Merge pull request 'Promote dev to main: off-box encrypted backups' (#57) from dev into main 60d5af41dd

Merge pull request 'Promote dev to main: first alerting + Postfix supervision' (#60) from dev into main 633c6a5ace

Merge pull request 'Promote dev to main: MET forecast coverage gate' (#64) from dev into main 150029075e

Promote dev → main (frontend QA batch → beta) (#71) ca84e0ce95

Merge pull request 'Promote dev to main: NASA retirement' (#73) from dev into main f596fa87d8

Merge pull request 'Promote dev to main: vault consolidation + renderer fail-closed' (#76) from dev into main 9350045866

Merge pull request 'Promote dev to main: dev + Centralis vaults' (#78) from dev into main 6c6842ab9e

web/worker: add a process-level liveness heartbeat ... d60e6c0abe

ProdWebContainerSilent and ProdWorkerContainerSilent alert on Docker
container-stdout volume, which isn't a valid liveness signal for either
role: web's real access logging lives in the file-tailed app-json
source, not stdout, and worker's stdout is unconditionally dropped by
Alloy's own relabel rule. Both fired for hours against a healthy prod.

Give web and worker a real audit.log_heartbeat() beat, same pattern the
subscription notifier already uses, tagged daemon=ROLE and unguarded
across every uvicorn worker/replica (a beat is a single local file
append, nothing like the notifier's leader-election cost). Observability
alert rules to consume this land separately.

admin_emi merged commit a4ecb51401 into dev 2026-07-25 04:13:50 +00:00

admin_emi deleted branch fix/web-worker-heartbeat 2026-07-25 04:13:51 +00:00

admin_emi referenced this pull request from a commit 2026-07-25 04:13:51 +00:00

web/worker: add a process-level liveness heartbeat (#80)

admin_emi referenced this pull request 2026-07-25 04:16:23 +00:00

promote: dev → main #83

admin_emi referenced this pull request 2026-07-25 17:15:04 +00:00

promote: main → release #93

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

Compat/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Compat/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#80

No description provided.