# Thermograph alert rules — the estate's first. All Loki/LogQL, because there is
# no Prometheus anywhere in the fleet: Loki is the only datasource, so every
# signal here is derived from log lines rather than metrics.
#
# ---------------------------------------------------------------------------
# HOW THE THRESHOLDS WERE CHOSEN
# ---------------------------------------------------------------------------
# Every number below was backtested against the real 24h ending 2026-07-24
# ~19:45Z. That window contains a genuine ~20-minute prod outage starting
# 2026-07-24 04:20Z, which is the event these rules must catch:
#
#   10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown):
#     2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1        (total 86)
#              ^^^^^^  the outage: 64% and 56% of all requests 5xx
#   All five /healthz failures in the whole day fell inside those two buckets.
#   Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic —
#   which is why absolute counts beat error *ratios* here: at 7 req/min a single
#   bad request is 1.4% and a ratio alert would scream all night.
#
# The design goal was: fire on the outage, stay silent for the other 142 of 144
# buckets, and still surface the slow drip of ~20 scattered errors that made up
# the rest of the day's 86 (that is what ProdHTTP5xxElevated is for).
#
# ---------------------------------------------------------------------------
# TWO CONVENTIONS WORTH KNOWING
# ---------------------------------------------------------------------------
# 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a
#    naive "count < 1" silence alert goes to NoData instead of firing, which is
#    the exact failure mode of every homemade log alert ever written. Appending
#    `or vector(0)` makes an empty result evaluate to a real 0. Verified working
#    on Loki 3.5.1.
# 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service`
#    label — only `container`, which includes a per-task suffix that changes on
#    every redeploy (thermograph_web.1.<taskid>). So prod services are matched as
#    container=~"thermograph_<name>.+" with job="docker". Do not use `service=`
#    here; it only works on the compose hosts (beta, dev).
#
# ---------------------------------------------------------------------------
# NOT ALERTABLE, ON PURPOSE
# ---------------------------------------------------------------------------
# thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and
# thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their
# healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL
# with long legitimate idle gaps. Catching those needs either a heartbeat line
# (extend backend/core/audit.py log_heartbeat(), as the notifier already does)
# or container metrics, which the fleet does not collect. Tracked as a gap.
#
# Error budget / failure isolation: only ProdEdgeDark escalates on NoData or
# datasource errors, so a Loki or WireGuard blip produces ONE page rather than
# twelve. Every other rule degrades to OK if the query cannot run.
apiVersion: 1

groups:
  # ===========================================================================
  # Is the product up? Evaluated every minute; all of these page.
  # ===========================================================================
  - orgId: 1
    name: prod-availability
    folder: Alerts
    interval: 1m
    rules:

      # -- 1 -----------------------------------------------------------------
      # The top-level "is anything alive" check, and deliberately the only rule
      # that also owns NoData/error: Caddy fronts every request to
      # thermograph.org, so zero access-log lines means the site is dark, the
      # box is gone, Alloy died, or the mesh dropped. Any of those is a page.
      #
      # DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35,
      # and never once reached zero — including *during* the outage (72), when
      # Caddy was happily logging 503s. Zero is unambiguous.
      - uid: tg_prod_edge_dark
        title: ProdEdgeDark
        condition: C
        for: 5m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'prod: no Caddy access-log lines for 15 minutes'
          description: >-
            Nothing has reached thermograph.org's reverse proxy in 15 minutes
            (healthy floor is 35 lines/15m, and it stayed at 72 even during the
            04:20Z outage). Either prod is down, Caddy is down, prod's Alloy
            agent stopped shipping, or the WireGuard mesh to beta is broken.
            This rule also fires on Loki NoData/errors, so it doubles as the
            "we have lost observability" alarm — if it is the only thing firing,
            suspect the pipeline before suspecting the app.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 900, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 2 -----------------------------------------------------------------
      # /healthz is the app's own statement that it is well. When it 503s, it is
      # not an opinion.
      #
      # DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour,
      # so this endpoint is polled sparsely and irregularly. That is why `for`
      # is 0m: waiting for a second consecutive bad sample could take 25
      # minutes. All 5 failures landed inside the two outage buckets and nowhere
      # else in the day, so one failure is already a real signal, not noise.
      - uid: tg_prod_healthz_failing
        title: ProdHealthzFailing
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: /healthz returned a non-200'
          description: >-
            The app's own liveness endpoint failed at the edge in the last 10
            minutes. Over the backtested 24h this happened 5 times, all of them
            inside the 04:20Z outage — /healthz failing has a 100% correlation
            with prod actually being broken. Check `docker service ps
            thermograph_web` on prod first.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }

      # -- 3 -----------------------------------------------------------------
      # The acute "users are seeing errors right now" page.
      #
      # DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and
      # the largest *benign* bucket had 5. The outage produced 45 and 22.
      # Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst
      # benign bucket and 4.5x below the incident peak — it fires on the outage
      # and on nothing else in the entire day. It catches 67 of that day's 86
      # errors as a single incident; the remaining scattered ~20 are picked up
      # by ProdHTTP5xxElevated below rather than by pushing this threshold down
      # into the noise.
      - uid: tg_prod_5xx_burst
        title: ProdHTTP5xxBurst
        condition: C
        for: 2m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 10+ HTTP 5xx in the last 10 minutes'
          description: >-
            Prod is serving server errors at an outage-grade rate. Baseline is
            zero in 90% of 10-minute windows and never above 5; the reference
            incident hit 45. Look at the "Thermograph — Traffic & Domains"
            dashboard, panel "HTTP errors by status", and at
            {job="app-json", host="prod", filename=~".+/errors/.+"} for the
            exceptions behind them.
        labels:
          severity: critical
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [9] }

  # ===========================================================================
  # Slow-burn error signals. Five-minute evaluation is plenty; these are tickets,
  # not pages.
  # ===========================================================================
  - orgId: 1
    name: prod-errors
    folder: Alerts
    interval: 5m
    rules:

      # -- 4 -----------------------------------------------------------------
      # The rule that would have broken the silence on "84 5xx in 24h and nobody
      # was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation.
      #
      # DATA: 6-hour rolling windows sampled hourly across the backtest read
      # 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11.
      # The largest window containing NO incident was 11. Threshold 25
      # (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any
      # day that looks like this one. Deliberately a long window: it will stay
      # firing for ~6h after an incident, which is the point. That is the daily
      # error budget talking, not a flap.
      - uid: tg_prod_5xx_elevated
        title: ProdHTTP5xxElevated
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours'
          description: >-
            Sustained server errors. This is the error-budget alarm, not an
            outage alarm — it catches the scattered drip that never trips the
            burst rule but still added up to 86 errors (1.7% of requests) on
            2026-07-24. Expect it to stay firing for up to 6h after an incident
            clears; that is the window emptying, not a new fault.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 21600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [24] }

      # -- 5 -----------------------------------------------------------------
      # A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees
      # 502/503 (the app is unreachable), while tag="http.error" is the app
      # catching an unhandled exception and still answering. A bad deploy shows
      # up here first, and can do so without moving the Caddy numbers at all.
      #
      # DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour.
      # Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak:
      # silent today, loud on an exception storm.
      - uid: tg_prod_unhandled_exceptions
        title: ProdUnhandledExceptions
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 3+ unhandled exceptions in 30 minutes'
          description: >-
            The app is throwing. Normal is under 2 per DAY. A cluster this tight
            almost always means a regression just shipped — check the app.start
            lines on the "App Activity & Delivery" dashboard for a recent deploy,
            then {job="app-json", host="prod", filename=~".+/errors/.+"} |
            tag="http.error" for the traceback and route.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [2] }

  # ===========================================================================
  # The subscription notifier — the thing that actually delivers the product.
  # ===========================================================================
  - orgId: 1
    name: notifier
    folder: Alerts
    interval: 5m
    rules:

      # -- 6 -----------------------------------------------------------------
      # backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it
      # stops, subscribers stop getting weather alerts and absolutely nothing
      # else in this file would notice — the site stays up and serves 200s while
      # the product quietly stops working.
      #
      # DATA: prod produced 1-2 beats in every single 20-minute bucket across
      # the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval
      # means two consecutive empty evaluations, so roughly 25 minutes of real
      # silence before it pages — about one and a half missed beats.
      #
      # dev is deliberately excluded: the LAN box had zero heartbeats for the
      # first 10.5h of the backtest window because it was simply off. Alerting
      # on dev would mean an alert that is firing more often than not.
      #
      # MUST stay filtered to daemon="subscription-notifier". The selector was
      # originally bare `tag="heartbeat"`, which was unambiguous while the
      # notifier thread was the only thing that ever beat. It stopped being
      # unambiguous on 2026-07-25T17:24Z, when web/app.py's _start_heartbeat
      # shipped: every uvicorn process in every replica now beats every 300s
      # under the same tag, tagged daemon=web / daemon=worker. Measured
      # immediately after that deploy, a 20m window held 16 web beats and 4
      # worker beats against 2 notifier beats — so the bare selector clears the
      # `< 1` threshold on web traffic alone and can no longer observe the
      # notifier at all.
      #
      # That matters because this rule had just caught a real 17.5h outage
      # (notifier silent 2026-07-24T23:49Z -> 2026-07-25T17:25Z, zero
      # subscription notifications delivered). Left bare, it would have gone
      # quiet on the next occurrence instead of paging. `daemon` is a JSON field
      # and not a stream label, hence the `| json |` parse rather than a label
      # matcher.
      - uid: tg_notifier_heartbeat_prod
        title: NotifierHeartbeatMissingProd
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no notifier heartbeat for 20 minutes'
          description: >-
            The subscription notifier has gone quiet on prod. Expected cadence is
            ~15 minutes and it did not miss a single 20-minute window over the
            backtested day. Subscribers are not being notified right now, even
            though the website is probably still serving fine. Check the
            thermograph_worker service and the "Notifier alive?" panel on the
            Fleet Logs dashboard.
        labels:
          severity: critical
          host: prod
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 7 -----------------------------------------------------------------
      # Same signal on beta. Beta was equally gap-free over the backtest (1-2
      # beats every 20m, no empty buckets), but beta is UAT — worth knowing,
      # not worth a page.
      - uid: tg_notifier_heartbeat_beta
        title: NotifierHeartbeatMissingBeta
        condition: C
        for: 15m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'beta: no notifier heartbeat for 20 minutes'
          description: >-
            The notifier has gone quiet on beta (UAT). Same signal as the prod
            rule but a longer `for`, because beta gets redeployed all day and a
            short gap is usually just a release.
        labels:
          severity: warning
          host: beta
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

  # ===========================================================================
  # Per-service liveness. The fleet collects no container metrics, so liveness
  # has to be inferred from "is this thing still producing output at all". Two
  # different sources are used here, and picking the wrong one is how this group
  # generated a day of false pages:
  #
  #   * db and frontend infer it from CONTAINER LOG VOLUME. Valid for these two:
  #     both log to stdout on a floor that never approaches zero (db a metronomic
  #     54-69 lines/2h, frontend ~66/30m), and prod's Alloy ships both.
  #
  #   * web and worker infer it from the APP'S OWN HEARTBEAT (tag=heartbeat in
  #     the app-json file source, filtered by the daemon field). Container log
  #     volume is NOT a valid liveness signal for either: web's request logging
  #     moved off stdout onto app-json, and worker's stdout is dropped outright
  #     by a rule in prod's alloy config. Both containers can be perfectly
  #     healthy while their docker streams sit at exactly zero forever.
  #
  # Before adding a container-log-silence rule for a new service, confirm the
  # service actually logs to stdout AND that prod's discovery.relabel "containers"
  # block does not drop it. A rule against a dropped stream is unsatisfiable and
  # will page forever.
  #
  # Caveat that still applies to the db/frontend rules: a container can be
  # running and logging while its lines never reach Loki, if the node's Alloy
  # agent stops tailing it. Those rules cannot tell the two apart — they say
  # "no logs", which is the honest claim. Either way something needs looking at.
  # ===========================================================================
  - orgId: 1
    name: prod-services
    folder: Alerts
    interval: 5m
    rules:

      # -- 8 -----------------------------------------------------------------
      # Web-tier liveness, from the app's own heartbeat rather than container
      # stdout.
      #
      # This rule used to count `{job="docker", container=~"thermograph_web.+"}`
      # over 30m against a measured floor of ~110 lines. That floor was real when
      # it was written (the backtest saw 3.8k-10.5k lines per 6h), but the app
      # moved its request logging off stdout onto the app-json file source, and
      # from 2026-07-25T00:02Z a web container emits ~15 uvicorn banner lines for
      # its entire lifetime and nothing after. The old rule then sat firing
      # permanently on a perfectly healthy tier, clearing only for the few
      # minutes after each deploy or autoscale event produced fresh banners —
      # 20+ pages in a day, every one of them false. See web/app.py's
      # _start_heartbeat docstring, which states outright that stdout silence
      # cannot distinguish a live web process from a dead one and that this
      # heartbeat is the signal to alert on instead.
      #
      # DATA: _start_heartbeat beats every 300s from every uvicorn process in
      # every replica, unguarded by the leader election, so a 4-process replica
      # yields ~16 beats per 20m (measured: exactly 16). Zero over 20m means no
      # web process is alive anywhere. This is the app itself, so it pages.
      - uid: tg_prod_web_silent
        title: ProdWebHeartbeatMissing
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no web heartbeat for 20 minutes'
          description: >-
            No thermograph_web process on prod has emitted a liveness heartbeat
            in 20 minutes. Every uvicorn process beats every 5 minutes
            independently of the notifier leader election, so a healthy 4-process
            replica produces ~16 beats per window — zero means no web process is
            alive anywhere, not merely quiet. Run `docker service ps
            thermograph_web` on prod. Note this is the app's own heartbeat, not
            container stdout: web's stdout is silent by design and is not a
            liveness signal.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="web" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 9 -----------------------------------------------------------------
      # DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69
      # lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a
      # metronomic ~30 lines/hour. Zero over two hours is as close to a
      # definitive "Postgres is not there" as logs can give.
      - uid: tg_prod_db_silent
        title: ProdDbContainerSilent
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_db has logged nothing for 2 hours'
          description: >-
            The production TimescaleDB container has produced zero log lines in
            2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no
            observed gaps, so zero is not quiet — it is absent. Everything else
            in the app depends on this.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 7200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 10 ----------------------------------------------------------------
      # DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than
      # critical because Caddy serves the static build and ProdEdgeDark /
      # ProdHTTP5xxBurst will catch it first if users are actually affected.
      - uid: tg_prod_frontend_silent
        title: ProdFrontendContainerSilent
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_frontend has logged nothing for 30 minutes'
          description: >-
            The frontend container has produced zero log lines in 30 minutes
            (healthy floor ~66 per 30m). If users were visibly affected you would
            expect ProdHTTP5xxBurst alongside this; on its own it more often
            means a stuck redeploy or a lost Alloy tail.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 11 ----------------------------------------------------------------
      # Worker liveness, from the app's own heartbeat rather than container
      # stdout.
      #
      # The previous rule counted `{job="docker", container=~"thermograph_worker.+"}`
      # and was UNSATISFIABLE BY CONSTRUCTION — it could never return anything but
      # zero, no matter how healthy the worker was. prod's Alloy config drops that
      # container on purpose, in discovery.relabel "containers":
      #
      #   regex  = "(alloy|autoscaler|thermograph-test_.*|thermograph-lb|thermograph_worker).*"
      #   action = "drop"
      #
      # with the stated reasoning that worker stdout was 100% /healthz poll noise
      # and the app's own access/*.jsonl is a strict superset of it. So the stream
      # this rule queried is discarded at the agent and never reaches Loki.
      #
      # The old comment here read the resulting permanent zero as "Alloy stopped
      # tailing the container after Swarm replaced the task" and pointed the
      # runbook at restarting alloy-alloy-1. That diagnosis was wrong, and the
      # fix it recommended could not have worked: a restarted Alloy re-reads the
      # same config and drops the same container again. The rule fired
      # continuously from 2026-07-24T21:05Z with no underlying fault.
      #
      # DATA: _start_heartbeat beats every 300s from the worker process
      # (measured: exactly 4 per 20m). Unlike the notifier below it is not gated
      # on the leader election, so it reports the process being alive even when
      # this worker is not the elected notifier — which is the distinction the
      # old rule was trying and failing to draw.
      - uid: tg_prod_worker_silent
        title: ProdWorkerHeartbeatMissing
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no worker heartbeat for 20 minutes'
          description: >-
            The thermograph_worker process on prod has not emitted a liveness
            heartbeat in 20 minutes (healthy is ~4 per window, one every 5
            minutes). This beat is independent of the notifier leader election,
            so it means the worker process itself is gone or wedged, not merely
            that it lost the election. Run `docker service ps thermograph_worker`
            on prod. Do not look at the worker's container logs — prod's Alloy
            config drops that stream deliberately, so it is always empty and is
            not evidence of anything.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="worker" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

  # ===========================================================================
  # The dead-man's switch.
  # ===========================================================================
  - orgId: 1
    name: watchdog
    folder: Alerts
    interval: 5m
    rules:

      # -- 12 ----------------------------------------------------------------
      # Nothing in this file can tell you that Grafana itself has stopped, or
      # that beta is down, or that the Discord webhook was revoked — a monitoring
      # system cannot report its own death. So this rule fires permanently and
      # by design, throttled by the notification policy to exactly one Discord
      # message per 24 hours.
      #
      # HOW TO USE IT: if #ops-alerts has been silent for more than a day, the
      # alerting pipeline is broken, not the estate healthy. That one daily
      # message is the only evidence you will ever get that the other eleven
      # rules are still capable of reaching you.
      #
      # To silence it (and lose that guarantee), set isPaused: true.
      - uid: tg_alerting_watchdog
        title: AlertingWatchdog
        condition: C
        for: 0m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)'
          description: >-
            This is the dead-man's switch and it is supposed to be firing. It
            proves Grafana on beta is evaluating rules, Loki is answering, and
            the Discord webhook still works. Seeing it once a day is correct.
            NOT seeing it for over a day means alerting itself is down and no
            other rule can reach you — check observability-grafana-1 on beta.
        labels:
          severity: watchdog
          host: beta
          class: meta
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'vector(1)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }