alerting: base web/worker liveness on the app heartbeat, not container stdout

Three of the prod liveness rules were measuring streams that no longer say
anything about the thing they claim to watch.

ProdWorkerContainerSilent was unsatisfiable by construction: prod's alloy
config drops the thermograph_worker container on purpose, so the stream it
queried never reaches Loki and the rule returned zero regardless of worker
health. Firing continuously since 2026-07-24T21:05Z.

ProdWebContainerSilent measured a floor that stopped existing when the app
moved request logging off stdout onto app-json; a web container now emits
~15 uvicorn banner lines for its whole lifetime. It cleared only for the few
minutes after each deploy/autoscale event, producing 20+ false pages in a day.

NotifierHeartbeatMissing{Prod,Beta} matched a bare tag=heartbeat, which was
unambiguous until _start_heartbeat shipped at 2026-07-25T17:24Z and every
uvicorn process began beating under the same tag. The rule that had just
caught a real 17.5h notifier outage would have gone silent on the next one.

Both liveness rules now read the app's own heartbeat filtered by daemon, and
the notifier rules are pinned to daemon=subscription-notifier. db and frontend
keep their container-log rules, which remain valid. Queries verified against
live Loki: web 16, worker 4, notifier 2 per 20m window; beta notifier never
below 1 across 37 consecutive buckets.

observability/grafana/provisioning/alerting/rules-prod.yml

@@ -400,6 +400,24 @@ groups:
       # dev is deliberately excluded: the LAN box had zero heartbeats for the
       # first 10.5h of the backtest window because it was simply off. Alerting
       # on dev would mean an alert that is firing more often than not.
+      #
+      # MUST stay filtered to daemon="subscription-notifier". The selector was
+      # originally bare `tag="heartbeat"`, which was unambiguous while the
+      # notifier thread was the only thing that ever beat. It stopped being
+      # unambiguous on 2026-07-25T17:24Z, when web/app.py's _start_heartbeat
+      # shipped: every uvicorn process in every replica now beats every 300s
+      # under the same tag, tagged daemon=web / daemon=worker. Measured
+      # immediately after that deploy, a 20m window held 16 web beats and 4
+      # worker beats against 2 notifier beats — so the bare selector clears the
+      # `< 1` threshold on web traffic alone and can no longer observe the
+      # notifier at all.
+      #
+      # That matters because this rule had just caught a real 17.5h outage
+      # (notifier silent 2026-07-24T23:49Z -> 2026-07-25T17:25Z, zero
+      # subscription notifications delivered). Left bare, it would have gone
+      # quiet on the next occurrence instead of paging. `daemon` is a JSON field
+      # and not a stream label, hence the `| json |` parse rather than a label
+      # matcher.
       - uid: tg_notifier_heartbeat_prod
         title: NotifierHeartbeatMissingProd
         condition: C
@@ -427,7 +445,7 @@ groups:
             model:
               refId: A
               datasource: { type: loki, uid: loki }
-              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} [20m])) or vector(0)'
+              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
               queryType: instant
               editorMode: code
               intervalMs: 1000
@@ -480,7 +498,7 @@ groups:
             model:
               refId: A
               datasource: { type: loki, uid: loki }
-              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} [20m])) or vector(0)'
+              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
               queryType: instant
               editorMode: code
               intervalMs: 1000
@@ -506,16 +524,31 @@ groups:
                 - evaluator: { type: lt, params: [1] }
 
   # ===========================================================================
-  # Per-container liveness, inferred from log volume. The fleet collects no
+  # Per-service liveness. The fleet collects no container metrics, so liveness
-  # container metrics, so "did this service stop producing any output at all" is
+  # has to be inferred from "is this thing still producing output at all". Two
-  # the only death certificate available — and it turns out to be a good one,
+  # different sources are used here, and picking the wrong one is how this group
-  # because every service below logs on a floor that never approaches zero.
+  # generated a day of false pages:
   #
-  # Caveat worth knowing before you trust these: a container can be running and
+  #   * db and frontend infer it from CONTAINER LOG VOLUME. Valid for these two:
-  # logging while its lines never reach Loki, if the node's Alloy agent stops
+  #     both log to stdout on a floor that never approaches zero (db a metronomic
-  # tailing it. That is a real observed failure on prod (see README). These
+  #     54-69 lines/2h, frontend ~66/30m), and prod's Alloy ships both.
-  # rules cannot tell the two apart — they say "no logs", which is the honest
+  #
-  # claim. Either way something needs looking at.
+  #   * web and worker infer it from the APP'S OWN HEARTBEAT (tag=heartbeat in
+  #     the app-json file source, filtered by the daemon field). Container log
+  #     volume is NOT a valid liveness signal for either: web's request logging
+  #     moved off stdout onto app-json, and worker's stdout is dropped outright
+  #     by a rule in prod's alloy config. Both containers can be perfectly
+  #     healthy while their docker streams sit at exactly zero forever.
+  #
+  # Before adding a container-log-silence rule for a new service, confirm the
+  # service actually logs to stdout AND that prod's discovery.relabel "containers"
+  # block does not drop it. A rule against a dropped stream is unsatisfiable and
+  # will page forever.
+  #
+  # Caveat that still applies to the db/frontend rules: a container can be
+  # running and logging while its lines never reach Loki, if the node's Alloy
+  # agent stops tailing it. Those rules cannot tell the two apart — they say
+  # "no logs", which is the honest claim. Either way something needs looking at.
   # ===========================================================================
   - orgId: 1
     name: prod-services
@@ -524,24 +557,43 @@ groups:
     rules:
 
       # -- 8 -----------------------------------------------------------------
-      # DATA: 15-minute windows over 24h ranged 54 -> 954 lines. The minimum, 54,
+      # Web-tier liveness, from the app's own heartbeat rather than container
-      # occurred during the outage; the quietest healthy window was ~150. A
+      # stdout.
-      # 30-minute window has never been below ~110. Zero means the container is
+      #
-      # gone. This is the app itself, so it pages.
+      # This rule used to count `{job="docker", container=~"thermograph_web.+"}`
+      # over 30m against a measured floor of ~110 lines. That floor was real when
+      # it was written (the backtest saw 3.8k-10.5k lines per 6h), but the app
+      # moved its request logging off stdout onto the app-json file source, and
+      # from 2026-07-25T00:02Z a web container emits ~15 uvicorn banner lines for
+      # its entire lifetime and nothing after. The old rule then sat firing
+      # permanently on a perfectly healthy tier, clearing only for the few
+      # minutes after each deploy or autoscale event produced fresh banners —
+      # 20+ pages in a day, every one of them false. See web/app.py's
+      # _start_heartbeat docstring, which states outright that stdout silence
+      # cannot distinguish a live web process from a dead one and that this
+      # heartbeat is the signal to alert on instead.
+      #
+      # DATA: _start_heartbeat beats every 300s from every uvicorn process in
+      # every replica, unguarded by the leader election, so a 4-process replica
+      # yields ~16 beats per 20m (measured: exactly 16). Zero over 20m means no
+      # web process is alive anywhere. This is the app itself, so it pages.
       - uid: tg_prod_web_silent
-        title: ProdWebContainerSilent
+        title: ProdWebHeartbeatMissing
         condition: C
         for: 5m
         noDataState: OK
         execErrState: OK
         annotations:
-          summary: 'prod: thermograph_web has logged nothing for 30 minutes'
+          summary: 'prod: no web heartbeat for 20 minutes'
           description: >-
-            The main app container has produced zero log lines in 30 minutes.
+            No thermograph_web process on prod has emitted a liveness heartbeat
-            Healthy floor is ~110 lines per 30m and it never dipped below 54 even
+            in 20 minutes. Every uvicorn process beats every 5 minutes
-            mid-outage, so this is a container that is gone, wedged, or no longer
+            independently of the notifier leader election, so a healthy 4-process
-            being tailed by prod's Alloy agent. Run `docker service ps
+            replica produces ~16 beats per window — zero means no web process is
-            thermograph_web` on prod.
+            alive anywhere, not merely quiet. Run `docker service ps
+            thermograph_web` on prod. Note this is the app's own heartbeat, not
+            container stdout: web's stdout is silent by design and is not a
+            liveness signal.
         labels:
           severity: critical
           host: prod
@@ -549,12 +601,12 @@ groups:
         isPaused: false
         data:
           - refId: A
-            relativeTimeRange: { from: 1800, to: 0 }
+            relativeTimeRange: { from: 1200, to: 0 }
             datasourceUid: loki
             model:
               refId: A
               datasource: { type: loki, uid: loki }
-              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_web.+"} [30m])) or vector(0)'
+              expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="web" [20m])) or vector(0)'
               queryType: instant
               editorMode: code
               intervalMs: 1000
@@ -689,42 +741,50 @@ groups:
                 - evaluator: { type: lt, params: [1] }
 
       # -- 11 ----------------------------------------------------------------
-      # DATA: a dead-steady 121 lines/hour (its own healthcheck polling) for the
+      # Worker liveness, from the app's own heartbeat rather than container
-      # first 14h of the backtest, then nothing at all.
+      # stdout.
       #
-      # HEADS UP: THIS RULE WILL BE FIRING THE MOMENT YOU DEPLOY IT, and it is
+      # The previous rule counted `{job="docker", container=~"thermograph_worker.+"}`
-      # right to. As of 2026-07-24 19:45Z the prod worker container is running
+      # and was UNSATISFIABLE BY CONSTRUCTION — it could never return anything but
-      # and healthy and is writing ~19 lines per 10 minutes to its own docker
+      # zero, no matter how healthy the worker was. prod's Alloy config drops that
-      # log — but not one of those lines has reached Loki since 04:43Z. Prod's
+      # container on purpose, in discovery.relabel "containers":
-      # Alloy agent stopped tailing the container when Swarm replaced the task.
-      # Nobody knew, because nothing was watching. See README "Known gaps".
       #
-      # WHY THIS IS A WARNING AND NotifierHeartbeatMissingProd IS CRITICAL: the
+      #   regex  = "(alloy|autoscaler|thermograph-test_.*|thermograph-lb|thermograph_worker).*"
-      # two draw on *different* Alloy sources — this one on loki.source.docker
+      #   action = "drop"
-      # (the container's stdout), the heartbeat on loki.source.file tailing
+      #
-      # /applogs. That independence is the diagnostic. Right now prod's
+      # with the stated reasoning that worker stdout was 100% /healthz poll noise
-      # heartbeats are arriving in every single 20-minute window while its
+      # and the app's own access/*.jsonl is a strict superset of it. So the stream
-      # worker docker stream has been empty for 14 hours, which localises the
+      # this rule queried is discarded at the agent and never reaches Loki.
-      # fault to Alloy's docker tail rather than to the worker: the process is
+      #
-      # doing its job, we just cannot see it breathe. Both rules firing together
+      # The old comment here read the resulting permanent zero as "Alloy stopped
-      # means the worker really is down.
+      # tailing the container after Swarm replaced the task" and pointed the
+      # runbook at restarting alloy-alloy-1. That diagnosis was wrong, and the
+      # fix it recommended could not have worked: a restarted Alloy re-reads the
+      # same config and drops the same container again. The rule fired
+      # continuously from 2026-07-24T21:05Z with no underlying fault.
+      #
+      # DATA: _start_heartbeat beats every 300s from the worker process
+      # (measured: exactly 4 per 20m). Unlike the notifier below it is not gated
+      # on the leader election, so it reports the process being alive even when
+      # this worker is not the elected notifier — which is the distinction the
+      # old rule was trying and failing to draw.
       - uid: tg_prod_worker_silent
-        title: ProdWorkerContainerSilent
+        title: ProdWorkerHeartbeatMissing
         condition: C
         for: 10m
         noDataState: OK
         execErrState: OK
         annotations:
-          summary: 'prod: thermograph_worker has logged nothing for 2 hours'
+          summary: 'prod: no worker heartbeat for 20 minutes'
           description: >-
-            The worker container has produced zero log lines in 2 hours against
+            The thermograph_worker process on prod has not emitted a liveness
-            a healthy floor of ~242 per 2h. Two different faults look identical
+            heartbeat in 20 minutes (healthy is ~4 per window, one every 5
-            here: the worker is actually dead, or prod's Alloy agent has stopped
+            minutes). This beat is independent of the notifier leader election,
-            tailing it (a real, observed failure — Alloy can miss a container
+            so it means the worker process itself is gone or wedged, not merely
-            after Swarm replaces the task). Check both: `docker service ps
+            that it lost the election. Run `docker service ps thermograph_worker`
-            thermograph_worker` on prod, and whether `docker logs` on the live
+            on prod. Do not look at the worker's container logs — prod's Alloy
-            container is producing lines that Loki does not have. Restarting
+            config drops that stream deliberately, so it is always empty and is
-            alloy-alloy-1 on prod fixes the second case.
+            not evidence of anything.
         labels:
           severity: warning
           host: prod
@@ -732,12 +792,12 @@ groups:
         isPaused: false
         data:
           - refId: A
-            relativeTimeRange: { from: 7200, to: 0 }
+            relativeTimeRange: { from: 1200, to: 0 }
             datasourceUid: loki
             model:
               refId: A
               datasource: { type: loki, uid: loki }
-              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_worker.+"} [2h])) or vector(0)'
+              expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="worker" [20m])) or vector(0)'
               queryType: instant
               editorMode: code
               intervalMs: 1000