web/worker: add a process-level liveness heartbeat #80

Merged
admin_emi merged 18 commits from fix/web-worker-heartbeat into dev 2026-07-25 04:13:50 +00:00

18 commits

Author SHA1 Message Date
Emi Griffith
d60e6c0abe web/worker: add a process-level liveness heartbeat
All checks were successful
PR build (required check) / changes (pull_request) Successful in 10s
secrets-guard / encrypted (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 10s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m11s
PR build (required check) / build-backend (pull_request) Successful in 1m41s
PR build (required check) / gate (pull_request) Successful in 3s
ProdWebContainerSilent and ProdWorkerContainerSilent alert on Docker
container-stdout volume, which isn't a valid liveness signal for either
role: web's real access logging lives in the file-tailed app-json
source, not stdout, and worker's stdout is unconditionally dropped by
Alloy's own relabel rule. Both fired for hours against a healthy prod.

Give web and worker a real audit.log_heartbeat() beat, same pattern the
subscription notifier already uses, tagged daemon=ROLE and unguarded
across every uvicorn worker/replica (a beat is a single local file
append, nothing like the notifier's leader-election cost). Observability
alert rules to consume this land separately.
2026-07-24 20:52:41 -07:00
emi
6c6842ab9e Merge pull request 'Promote dev to main: dev + Centralis vaults' (#78) from dev into main
All checks were successful
Sync infra to hosts / sync-prod (push) Successful in 5s
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 9s
Sync infra to hosts / sync-beta (push) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 6s
shell-lint / shellcheck (pull_request) Successful in 8s
2026-07-25 01:16:04 +00:00
emi
9350045866 Merge pull request 'Promote dev to main: vault consolidation + renderer fail-closed' (#76) from dev into main
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 7s
Sync infra to hosts / sync-prod (push) Successful in 6s
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 7s
2026-07-25 00:47:04 +00:00
emi
f596fa87d8 Merge pull request 'Promote dev to main: NASA retirement' (#73) from dev into main
All checks were successful
secrets-guard / encrypted (push) Successful in 12s
shell-lint / shellcheck (push) Successful in 11s
secrets-guard / encrypted (pull_request) Successful in 12s
shell-lint / shellcheck (pull_request) Successful in 10s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m54s
Deploy backend to beta VPS / deploy (push) Successful in 2m24s
2026-07-24 23:59:00 +00:00
emi
ca84e0ce95 Promote dev → main (frontend QA batch → beta) (#71)
Some checks failed
Deploy frontend to beta VPS / deploy (push) Failing after 8s
Sync infra to hosts / sync-beta (push) Successful in 7s
Deploy backend to beta VPS / deploy (push) Failing after 13s
secrets-guard / encrypted (push) Successful in 12s
Sync infra to hosts / sync-prod (push) Successful in 14s
shell-lint / shellcheck (push) Successful in 12s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 57s
Build + push frontend image (Forgejo registry) / build-push (push) Successful in 1m2s
2026-07-24 23:20:16 +00:00
emi
150029075e Merge pull request 'Promote dev to main: MET forecast coverage gate' (#64) from dev into main
All checks were successful
secrets-guard / encrypted (push) Successful in 9s
shell-lint / shellcheck (push) Successful in 16s
secrets-guard / encrypted (pull_request) Successful in 11s
shell-lint / shellcheck (pull_request) Successful in 13s
Deploy backend to beta VPS / deploy (push) Successful in 51s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m22s
2026-07-24 22:26:20 +00:00
emi
633c6a5ace Merge pull request 'Promote dev to main: first alerting + Postfix supervision' (#60) from dev into main
Some checks failed
Sync infra to hosts / sync-prod (push) Failing after 7s
Sync infra to hosts / sync-beta (push) Successful in 13s
secrets-guard / encrypted (push) Successful in 8s
shell-lint / shellcheck (push) Successful in 13s
Validate observability stack / validate (push) Successful in 17s
secrets-guard / encrypted (pull_request) Successful in 6s
shell-lint / shellcheck (pull_request) Successful in 6s
2026-07-24 20:49:08 +00:00
emi
60d5af41dd Merge pull request 'Promote dev to main: off-box encrypted backups' (#57) from dev into main
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 6s
secrets-guard / encrypted (push) Successful in 5s
Sync infra to hosts / sync-prod (push) Successful in 8s
shell-lint / shellcheck (push) Successful in 7s
2026-07-24 20:10:12 +00:00
emi
0935eea394 Merge pull request 'Promote dev to main: geocode P0 fix + ops tooling + privacy logging' (#52) from dev into main
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 8s
secrets-guard / encrypted (push) Successful in 12s
shell-lint / shellcheck (push) Successful in 10s
Sync infra to hosts / sync-prod (push) Successful in 16s
Deploy backend to beta VPS / deploy (push) Successful in 38s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 52s
secrets-guard / encrypted (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 10s
2026-07-24 19:48:39 +00:00
emi
6f5682b7d7 Merge pull request 'Promote dev to main: Iceberg-backed lake query' (#45) from dev into main
All checks were successful
secrets-guard / encrypted (push) Successful in 8s
shell-lint / shellcheck (push) Successful in 8s
secrets-guard / encrypted (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 10s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 41s
Deploy backend to beta VPS / deploy (push) Successful in 47s
2026-07-24 19:11:50 +00:00
emi
81c579ddcb infra/forgejo: raise LAN runner concurrency to 8 (#43)
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 8s
Sync infra to hosts / sync-prod (push) Successful in 10s
secrets-guard / encrypted (push) Successful in 8s
shell-lint / shellcheck (push) Successful in 8s
2026-07-24 19:05:21 +00:00
emi
88f5d0cce0 Merge pull request 'Promote dev to main: daemon roll + Iceberg-backed lake query' (#41) from dev into main
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 6s
Sync infra to hosts / sync-prod (push) Successful in 11s
secrets-guard / encrypted (push) Successful in 9s
Validate observability stack / validate (push) Successful in 14s
shell-lint / shellcheck (push) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 8s
2026-07-24 19:01:52 +00:00
emi
dcf15ea572 infra/forgejo: add resource limits and a leaner CI job image (#38)
Some checks failed
Sync infra to hosts / sync-beta (push) Successful in 7s
Sync infra to hosts / sync-prod (push) Failing after 8s
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 8s
2026-07-24 18:56:46 +00:00
emi
da82abda27 frontend: fetch City() concurrently with the page's primary API call (#37)
All checks were successful
secrets-guard / encrypted (push) Successful in 8s
shell-lint / shellcheck (push) Successful in 14s
Build + push frontend image (Forgejo registry) / build-push (push) Successful in 42s
Deploy frontend to beta VPS / deploy (push) Successful in 1m5s
MonthPage and RecordsPage each made two backend calls sequentially
(CityMonth/CityRecords, then City) where the second never depended on the
first's result. fetchWithCity launches both concurrently via goroutines
and a WaitGroup.

Verified live against a stub with an injected 400ms delay on both
endpoints: city page (1 call) and month/records pages (2 calls each) all
cost ~0.404s now, not double for the two-call pages.

Error priority preserved exactly: primary's error wins even when City
also fails, matching the old sequential code. One trade-off: City() is
now always launched even on a request about to 404 from primary, costing
one extra cheap lookup on that rare path.

Tests include a deterministic concurrency proof via rendezvous channels
(the old sequential code would deadlock this test, not just run it
slower). Full suite green under -race -count=2.
2026-07-24 06:25:06 +00:00
emi
979653f407 deploy.sh: fix the daemon-binary probe, which always dropped daemon (#33)
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 5s
Sync infra to hosts / sync-prod (push) Successful in 4s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 7s
docker compose config --images daemon does not filter to the named service
on this host's Compose v5.3.1 -- it prints every service's image, one per
line, in file order, so `| head -1` was silently grabbing db's image
(timescaledb) instead of daemon's. The probe then always found no
/usr/local/bin/thermograph-daemon in a Postgres image and dropped daemon
from every backend deploy, regardless of what the real backend image
contained.

Fixed by building the image reference directly from the same vars
docker-compose.yml's daemon.image: already interpolates, instead of going
through docker compose config at all.

Reproduced against beta directly: the old sequence selected the wrong
image; the new construction resolves correctly and the binary probe
passes.
2026-07-24 04:03:26 +00:00
emi
b83bfdbd67 Merge pull request 'Promote dev to main: lake extension bake + daemon healthcheck fixes' (#31) from dev into main
All checks were successful
Deploy backend to beta VPS / deploy (push) Successful in 42s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 44s
Sync infra to hosts / sync-beta (push) Successful in 5s
Sync infra to hosts / sync-prod (push) Successful in 6s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 8s
secrets-guard / encrypted (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 9s
2026-07-24 03:50:21 +00:00
emi
92e74c585a frontend: rewrite the SSR content service in Go (#28)
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 13s
Sync infra to hosts / sync-prod (push) Successful in 12s
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 8s
Build + push frontend image (Forgejo registry) / build-push (push) Successful in 53s
Deploy frontend to beta VPS / deploy (push) Successful in 1m16s
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 6s
Ports frontend/ (Jinja2/FastAPI, ~1180 LOC) to Go with html/template. No
climate math, no DB, no auth -- every route fetches from the backend's
/content/* API.

Verified with a golden-HTML diff, not just unit tests: both the Python
original and the Go rewrite were run against the same committed fixtures
and every route compared byte-for-byte, confirmed programmatically. That
process caught defects unit tests alone missed, since map[string]any has
no compile-time field check:

- Render-context keys were snake_case throughout while the templates read
  PascalCase fields. A missing map key doesn't error, it silently renders
  empty -- title, meta description, canonical URL, OpenGraph tags, and the
  homepage's entire ranked list were blank on every page despite every
  route returning 200. Fixed by renaming every key to match each
  template's own documented field contract, and passing API structs
  straight through wherever their fields already matched (removes a whole
  layer of future drift risk).
- Three pages 500'd: ToolHref needed a composed href, not a bare
  "lat,lon" fragment; the records table needed the raw API struct.
- JSON-LD was double-encoded: <script type="application/ld+json"> is
  JAVASCRIPT context to html/template's escaper regardless of the
  script's type attribute, so template.HTML gets re-escaped as a quoted
  JS string. Needed template.JS. The glossary term page's JSON-LD was
  never built at all -- added.
- html/template silently strips literal HTML and JS comments from parsed
  output (verified in isolation) -- both need a FuncMap function
  returning template.HTML/template.JS to survive.

Packaging: 187MB -> 22.6MB. Two defects caught before reaching a host: the
Swarm stack's entrypoint override with no explicit command drops the
image's CMD entirely (every deploy would have exited 127), and
COPY --chown by name fails under the classic Docker builder on Alpine.
Both fixed.

go build/vet/test -race clean; docker build passes its embedded test step
under both BuildKit and the classic builder; shellcheck 0 findings.
2026-07-24 00:53:48 +00:00
emi
755886e244 Merge pull request 'Promote dev to main (deploy beta): ERA5 lake stack' (#26) from dev into main
All checks were successful
Build + push backend image (Forgejo registry) / build-push (push) Successful in 28s
Sync infra to hosts / sync-beta (push) Successful in 9s
Sync infra to hosts / sync-prod (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 10s
Deploy backend to beta VPS / deploy (push) Successful in 50s
2026-07-24 00:12:32 +00:00