# Thermograph alert rules — the estate's first. All Loki/LogQL, because there is
# no Prometheus anywhere in the fleet: Loki is the only datasource, so every
# signal here is derived from log lines rather than metrics.
#
# ---------------------------------------------------------------------------
# HOW THE THRESHOLDS WERE CHOSEN
# ---------------------------------------------------------------------------
# Every number below was backtested against the real 24h ending 2026-07-24
# ~19:45Z. That window contains a genuine ~20-minute prod outage starting
# 2026-07-24 04:20Z, which is the event these rules must catch:
#
#   10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown):
#     2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1        (total 86)
#              ^^^^^^  the outage: 64% and 56% of all requests 5xx
#   All five /healthz failures in the whole day fell inside those two buckets.
#   Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic —
#   which is why absolute counts beat error *ratios* here: at 7 req/min a single
#   bad request is 1.4% and a ratio alert would scream all night.
#
# The design goal was: fire on the outage, stay silent for the other 142 of 144
# buckets, and still surface the slow drip of ~20 scattered errors that made up
# the rest of the day's 86 (that is what ProdHTTP5xxElevated is for).
#
# ---------------------------------------------------------------------------
# TWO CONVENTIONS WORTH KNOWING
# ---------------------------------------------------------------------------
# 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a
#    naive "count < 1" silence alert goes to NoData instead of firing, which is
#    the exact failure mode of every homemade log alert ever written. Appending
#    `or vector(0)` makes an empty result evaluate to a real 0. Verified working
#    on Loki 3.5.1.
# 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service`
#    label — only `container`, which includes a per-task suffix that changes on
#    every redeploy (thermograph_web.1.<taskid>). So prod services are matched as
#    container=~"thermograph_<name>.+" with job="docker". Do not use `service=`
#    here; it only works on the compose hosts (beta, dev).
#
# ---------------------------------------------------------------------------
# NOT ALERTABLE, ON PURPOSE
# ---------------------------------------------------------------------------
# thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and
# thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their
# healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL
# with long legitimate idle gaps. Catching those needs either a heartbeat line
# (extend backend/core/audit.py log_heartbeat(), as the notifier already does)
# or container metrics, which the fleet does not collect. Tracked as a gap.
#
# Error budget / failure isolation: only ProdEdgeDark escalates on NoData or
# datasource errors, so a Loki or WireGuard blip produces ONE page rather than
# twelve. Every other rule degrades to OK if the query cannot run.
apiVersion: 1

groups:
  # ===========================================================================
  # Is the product up? Evaluated every minute; all of these page.
  # ===========================================================================
  - orgId: 1
    name: prod-availability
    folder: Alerts
    interval: 1m
    rules:

      # -- 1 -----------------------------------------------------------------
      # The top-level "is anything alive" check, and deliberately the only rule
      # that also owns NoData/error: Caddy fronts every request to
      # thermograph.org, so zero access-log lines means the site is dark, the
      # box is gone, Alloy died, or the mesh dropped. Any of those is a page.
      #
      # DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35,
      # and never once reached zero — including *during* the outage (72), when
      # Caddy was happily logging 503s. Zero is unambiguous.
      - uid: tg_prod_edge_dark
        title: ProdEdgeDark
        condition: C
        for: 5m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'prod: no Caddy access-log lines for 15 minutes'
          description: >-
            Nothing has reached thermograph.org's reverse proxy in 15 minutes
            (healthy floor is 35 lines/15m, and it stayed at 72 even during the
            04:20Z outage). Either prod is down, Caddy is down, prod's Alloy
            agent stopped shipping, or the WireGuard mesh to beta is broken.
            This rule also fires on Loki NoData/errors, so it doubles as the
            "we have lost observability" alarm — if it is the only thing firing,
            suspect the pipeline before suspecting the app.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 900, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 2 -----------------------------------------------------------------
      # /healthz is the app's own statement that it is well. When it 503s, it is
      # not an opinion.
      #
      # DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour,
      # so this endpoint is polled sparsely and irregularly. That is why `for`
      # is 0m: waiting for a second consecutive bad sample could take 25
      # minutes. All 5 failures landed inside the two outage buckets and nowhere
      # else in the day, so one failure is already a real signal, not noise.
      - uid: tg_prod_healthz_failing
        title: ProdHealthzFailing
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: /healthz returned a non-200'
          description: >-
            The app's own liveness endpoint failed at the edge in the last 10
            minutes. Over the backtested 24h this happened 5 times, all of them
            inside the 04:20Z outage — /healthz failing has a 100% correlation
            with prod actually being broken. Check `docker service ps
            thermograph_web` on prod first.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }

      # -- 3 -----------------------------------------------------------------
      # The acute "users are seeing errors right now" page.
      #
      # DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and
      # the largest *benign* bucket had 5. The outage produced 45 and 22.
      # Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst
      # benign bucket and 4.5x below the incident peak — it fires on the outage
      # and on nothing else in the entire day. It catches 67 of that day's 86
      # errors as a single incident; the remaining scattered ~20 are picked up
      # by ProdHTTP5xxElevated below rather than by pushing this threshold down
      # into the noise.
      - uid: tg_prod_5xx_burst
        title: ProdHTTP5xxBurst
        condition: C
        for: 2m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 10+ HTTP 5xx in the last 10 minutes'
          description: >-
            Prod is serving server errors at an outage-grade rate. Baseline is
            zero in 90% of 10-minute windows and never above 5; the reference
            incident hit 45. Look at the "Thermograph — Traffic & Domains"
            dashboard, panel "HTTP errors by status", and at
            {job="app-json", host="prod", filename=~".+/errors/.+"} for the
            exceptions behind them.
        labels:
          severity: critical
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [9] }

  # ===========================================================================
  # Slow-burn error signals. Five-minute evaluation is plenty; these are tickets,
  # not pages.
  # ===========================================================================
  - orgId: 1
    name: prod-errors
    folder: Alerts
    interval: 5m
    rules:

      # -- 4 -----------------------------------------------------------------
      # The rule that would have broken the silence on "84 5xx in 24h and nobody
      # was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation.
      #
      # DATA: 6-hour rolling windows sampled hourly across the backtest read
      # 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11.
      # The largest window containing NO incident was 11. Threshold 25
      # (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any
      # day that looks like this one. Deliberately a long window: it will stay
      # firing for ~6h after an incident, which is the point. That is the daily
      # error budget talking, not a flap.
      - uid: tg_prod_5xx_elevated
        title: ProdHTTP5xxElevated
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours'
          description: >-
            Sustained server errors. This is the error-budget alarm, not an
            outage alarm — it catches the scattered drip that never trips the
            burst rule but still added up to 86 errors (1.7% of requests) on
            2026-07-24. Expect it to stay firing for up to 6h after an incident
            clears; that is the window emptying, not a new fault.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 21600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [24] }

      # -- 5 -----------------------------------------------------------------
      # A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees
      # 502/503 (the app is unreachable), while tag="http.error" is the app
      # catching an unhandled exception and still answering. A bad deploy shows
      # up here first, and can do so without moving the Caddy numbers at all.
      #
      # DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour.
      # Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak:
      # silent today, loud on an exception storm.
      - uid: tg_prod_unhandled_exceptions
        title: ProdUnhandledExceptions
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 3+ unhandled exceptions in 30 minutes'
          description: >-
            The app is throwing. Normal is under 2 per DAY. A cluster this tight
            almost always means a regression just shipped — check the app.start
            lines on the "App Activity & Delivery" dashboard for a recent deploy,
            then {job="app-json", host="prod", filename=~".+/errors/.+"} |
            tag="http.error" for the traceback and route.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [2] }

  # ===========================================================================
  # The subscription notifier — the thing that actually delivers the product.
  # ===========================================================================
  - orgId: 1
    name: notifier
    folder: Alerts
    interval: 5m
    rules:

      # -- 6 -----------------------------------------------------------------
      # backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it
      # stops, subscribers stop getting weather alerts and absolutely nothing
      # else in this file would notice — the site stays up and serves 200s while
      # the product quietly stops working.
      #
      # DATA: prod produced 1-2 beats in every single 20-minute bucket across
      # the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval
      # means two consecutive empty evaluations, so roughly 25 minutes of real
      # silence before it pages — about one and a half missed beats.
      #
      # dev is deliberately excluded: the LAN box had zero heartbeats for the
      # first 10.5h of the backtest window because it was simply off. Alerting
      # on dev would mean an alert that is firing more often than not.
      - uid: tg_notifier_heartbeat_prod
        title: NotifierHeartbeatMissingProd
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no notifier heartbeat for 20 minutes'
          description: >-
            The subscription notifier has gone quiet on prod. Expected cadence is
            ~15 minutes and it did not miss a single 20-minute window over the
            backtested day. Subscribers are not being notified right now, even
            though the website is probably still serving fine. Check the
            thermograph_worker service and the "Notifier alive?" panel on the
            Fleet Logs dashboard.
        labels:
          severity: critical
          host: prod
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 7 -----------------------------------------------------------------
      # Same signal on beta. Beta was equally gap-free over the backtest (1-2
      # beats every 20m, no empty buckets), but beta is UAT — worth knowing,
      # not worth a page.
      - uid: tg_notifier_heartbeat_beta
        title: NotifierHeartbeatMissingBeta
        condition: C
        for: 15m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'beta: no notifier heartbeat for 20 minutes'
          description: >-
            The notifier has gone quiet on beta (UAT). Same signal as the prod
            rule but a longer `for`, because beta gets redeployed all day and a
            short gap is usually just a release.
        labels:
          severity: warning
          host: beta
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

  # ===========================================================================
  # Per-container liveness, inferred from log volume. The fleet collects no
  # container metrics, so "did this service stop producing any output at all" is
  # the only death certificate available — and it turns out to be a good one,
  # because every service below logs on a floor that never approaches zero.
  #
  # Caveat worth knowing before you trust these: a container can be running and
  # logging while its lines never reach Loki, if the node's Alloy agent stops
  # tailing it. That is a real observed failure on prod (see README). These
  # rules cannot tell the two apart — they say "no logs", which is the honest
  # claim. Either way something needs looking at.
  # ===========================================================================
  - orgId: 1
    name: prod-services
    folder: Alerts
    interval: 5m
    rules:

      # -- 8 -----------------------------------------------------------------
      # DATA: 15-minute windows over 24h ranged 54 -> 954 lines. The minimum, 54,
      # occurred during the outage; the quietest healthy window was ~150. A
      # 30-minute window has never been below ~110. Zero means the container is
      # gone. This is the app itself, so it pages.
      - uid: tg_prod_web_silent
        title: ProdWebContainerSilent
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_web has logged nothing for 30 minutes'
          description: >-
            The main app container has produced zero log lines in 30 minutes.
            Healthy floor is ~110 lines per 30m and it never dipped below 54 even
            mid-outage, so this is a container that is gone, wedged, or no longer
            being tailed by prod's Alloy agent. Run `docker service ps
            thermograph_web` on prod.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_web.+"} [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 9 -----------------------------------------------------------------
      # DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69
      # lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a
      # metronomic ~30 lines/hour. Zero over two hours is as close to a
      # definitive "Postgres is not there" as logs can give.
      - uid: tg_prod_db_silent
        title: ProdDbContainerSilent
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_db has logged nothing for 2 hours'
          description: >-
            The production TimescaleDB container has produced zero log lines in
            2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no
            observed gaps, so zero is not quiet — it is absent. Everything else
            in the app depends on this.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 7200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 10 ----------------------------------------------------------------
      # DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than
      # critical because Caddy serves the static build and ProdEdgeDark /
      # ProdHTTP5xxBurst will catch it first if users are actually affected.
      - uid: tg_prod_frontend_silent
        title: ProdFrontendContainerSilent
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_frontend has logged nothing for 30 minutes'
          description: >-
            The frontend container has produced zero log lines in 30 minutes
            (healthy floor ~66 per 30m). If users were visibly affected you would
            expect ProdHTTP5xxBurst alongside this; on its own it more often
            means a stuck redeploy or a lost Alloy tail.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 11 ----------------------------------------------------------------
      # DATA: a dead-steady 121 lines/hour (its own healthcheck polling) for the
      # first 14h of the backtest, then nothing at all.
      #
      # HEADS UP: THIS RULE WILL BE FIRING THE MOMENT YOU DEPLOY IT, and it is
      # right to. As of 2026-07-24 19:45Z the prod worker container is running
      # and healthy and is writing ~19 lines per 10 minutes to its own docker
      # log — but not one of those lines has reached Loki since 04:43Z. Prod's
      # Alloy agent stopped tailing the container when Swarm replaced the task.
      # Nobody knew, because nothing was watching. See README "Known gaps".
      #
      # WHY THIS IS A WARNING AND NotifierHeartbeatMissingProd IS CRITICAL: the
      # two draw on *different* Alloy sources — this one on loki.source.docker
      # (the container's stdout), the heartbeat on loki.source.file tailing
      # /applogs. That independence is the diagnostic. Right now prod's
      # heartbeats are arriving in every single 20-minute window while its
      # worker docker stream has been empty for 14 hours, which localises the
      # fault to Alloy's docker tail rather than to the worker: the process is
      # doing its job, we just cannot see it breathe. Both rules firing together
      # means the worker really is down.
      - uid: tg_prod_worker_silent
        title: ProdWorkerContainerSilent
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_worker has logged nothing for 2 hours'
          description: >-
            The worker container has produced zero log lines in 2 hours against
            a healthy floor of ~242 per 2h. Two different faults look identical
            here: the worker is actually dead, or prod's Alloy agent has stopped
            tailing it (a real, observed failure — Alloy can miss a container
            after Swarm replaces the task). Check both: `docker service ps
            thermograph_worker` on prod, and whether `docker logs` on the live
            container is producing lines that Loki does not have. Restarting
            alloy-alloy-1 on prod fixes the second case.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 7200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_worker.+"} [2h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

  # ===========================================================================
  # The dead-man's switch.
  # ===========================================================================
  - orgId: 1
    name: watchdog
    folder: Alerts
    interval: 5m
    rules:

      # -- 12 ----------------------------------------------------------------
      # Nothing in this file can tell you that Grafana itself has stopped, or
      # that beta is down, or that the Discord webhook was revoked — a monitoring
      # system cannot report its own death. So this rule fires permanently and
      # by design, throttled by the notification policy to exactly one Discord
      # message per 24 hours.
      #
      # HOW TO USE IT: if #ops-alerts has been silent for more than a day, the
      # alerting pipeline is broken, not the estate healthy. That one daily
      # message is the only evidence you will ever get that the other eleven
      # rules are still capable of reaching you.
      #
      # To silence it (and lose that guarantee), set isPaused: true.
      - uid: tg_alerting_watchdog
        title: AlertingWatchdog
        condition: C
        for: 0m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)'
          description: >-
            This is the dead-man's switch and it is supposed to be firing. It
            proves Grafana on beta is evaluating rules, Loki is answering, and
            the Discord webhook still works. Seeing it once a day is correct.
            NOT seeing it for over a day means alerting itself is down and no
            other rule can reach you — check observability-grafana-1 on beta.
        labels:
          severity: watchdog
          host: beta
          class: meta
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'vector(1)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }
