ba11423fd9
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / changes (pull_request) Successful in 9s
PR build (required check) / validate-observability (pull_request) Successful in 23s
PR build (required check) / gate (pull_request) Successful in 2s
Three of the prod liveness rules were measuring streams that no longer say
anything about the thing they claim to watch.
ProdWorkerContainerSilent was unsatisfiable by construction: prod's alloy
config drops the thermograph_worker container on purpose, so the stream it
queried never reaches Loki and the rule returned zero regardless of worker
health. Firing continuously since 2026-07-24T21:05Z.
ProdWebContainerSilent measured a floor that stopped existing when the app
moved request logging off stdout onto app-json; a web container now emits
~15 uvicorn banner lines for its whole lifetime. It cleared only for the few
minutes after each deploy/autoscale event, producing 20+ false pages in a day.
NotifierHeartbeatMissing{Prod,Beta} matched a bare tag=heartbeat, which was
unambiguous until _start_heartbeat shipped at 2026-07-25T17:24Z and every
uvicorn process began beating under the same tag. The rule that had just
caught a real 17.5h notifier outage would have gone silent on the next one.
Both liveness rules now read the app's own heartbeat filtered by daemon, and
the notifier rules are pinned to daemon=subscription-notifier. db and frontend
keep their container-log rules, which remain valid. Queries verified against
live Loki: web 16, worker 4, notifier 2 per 20m window; beta notifier never
below 1 across 37 consecutive buckets.
|
||
|---|---|---|
| .. | ||
| dashboards | ||
| provisioning | ||