thermograph/observability/grafana/provisioning/alerting/rules-prod.yml

# Thermograph alert rules — the estate's first. All Loki/LogQL, because there is
# no Prometheus anywhere in the fleet: Loki is the only datasource, so every
# signal here is derived from log lines rather than metrics.
#
# ---------------------------------------------------------------------------
# HOW THE THRESHOLDS WERE CHOSEN
# ---------------------------------------------------------------------------
# Every number below was backtested against the real 24h ending 2026-07-24
# ~19:45Z. That window contains a genuine ~20-minute prod outage starting
# 2026-07-24 04:20Z, which is the event these rules must catch:
#
#   10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown):
#     2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1        (total 86)
#              ^^^^^^  the outage: 64% and 56% of all requests 5xx
#   All five /healthz failures in the whole day fell inside those two buckets.
#   Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic —
#   which is why absolute counts beat error *ratios* here: at 7 req/min a single
#   bad request is 1.4% and a ratio alert would scream all night.
#
# The design goal was: fire on the outage, stay silent for the other 142 of 144
# buckets, and still surface the slow drip of ~20 scattered errors that made up
# the rest of the day's 86 (that is what ProdHTTP5xxElevated is for).
#
# ---------------------------------------------------------------------------
# TWO CONVENTIONS WORTH KNOWING
# ---------------------------------------------------------------------------
# 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a
#    naive "count < 1" silence alert goes to NoData instead of firing, which is
#    the exact failure mode of every homemade log alert ever written. Appending
#    `or vector(0)` makes an empty result evaluate to a real 0. Verified working
#    on Loki 3.5.1.
# 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service`
#    label — only `container`, which includes a per-task suffix that changes on
#    every redeploy (thermograph_web.1.<taskid>). So prod services are matched as
#    container=~"thermograph_<name>.+" with job="docker". Do not use `service=`
#    here; it only works on dev, now the ONLY compose host left — beta is Swarm
#    too (stack `thermograph-beta`, since both prod and beta run on vps2), so a
#    beta-targeted rule needs its own regex against beta's prefixed container
#    names (thermograph-beta_beta-web.1.<taskid> etc.), not a bare `service=`
#    match either.
#
# ---------------------------------------------------------------------------
# NOT ALERTABLE, ON PURPOSE
# ---------------------------------------------------------------------------
# thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and
# thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their
# healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL
# with long legitimate idle gaps. Catching those needs either a heartbeat line
# (extend backend/core/audit.py log_heartbeat(), as the notifier already does)
# or container metrics, which the fleet does not collect. Tracked as a gap.
#
# Error budget / failure isolation: only ProdEdgeDark escalates on NoData or
# datasource errors, so a Loki or WireGuard blip produces ONE page rather than
# twelve. Every other rule degrades to OK if the query cannot run.
apiVersion: 1

groups:
  # ===========================================================================
  # Is the product up? Evaluated every minute; all of these page.
  # ===========================================================================
  - orgId: 1
    name: prod-availability
    folder: Alerts
    interval: 1m
    rules:

      # -- 1 -----------------------------------------------------------------
      # The top-level "is anything alive" check, and deliberately the only rule
      # that also owns NoData/error: Caddy fronts every request to
      # thermograph.org, so zero access-log lines means the site is dark, the
      # box is gone, Alloy died, or the mesh dropped. Any of those is a page.
      #
      # DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35,
      # and never once reached zero — including *during* the outage (72), when
      # Caddy was happily logging 503s. Zero is unambiguous.
      #
      # The description names vps1, not "beta": the mesh path that matters here
      # is the one to the MONITORING host, which is vps1. Beta is on vps2 and
      # has nothing to do with whether these logs arrive.
      - uid: tg_prod_edge_dark
        title: ProdEdgeDark
        condition: C
        for: 5m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'prod: no Caddy access-log lines for 15 minutes'
          description: >-
            Nothing has reached thermograph.org's reverse proxy in 15 minutes
            (healthy floor is 35 lines/15m, and it stayed at 72 even during the
            04:20Z outage). Either prod is down, Caddy is down, prod's Alloy
            agent stopped shipping, or the WireGuard mesh to vps1 (where Loki
            runs) is broken.
            This rule also fires on Loki NoData/errors, so it doubles as the
            "we have lost observability" alarm — if it is the only thing firing,
            suspect the pipeline before suspecting the app.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 900, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 2 -----------------------------------------------------------------
      # /healthz is the app's own statement that it is well. When it 503s, it is
      # not an opinion.
      #
      # DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour,
      # so this endpoint is polled sparsely and irregularly. That is why `for`
      # is 0m: waiting for a second consecutive bad sample could take 25
      # minutes. All 5 failures landed inside the two outage buckets and nowhere
      # else in the day, so one failure is already a real signal, not noise.
      - uid: tg_prod_healthz_failing
        title: ProdHealthzFailing
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: /healthz returned a non-200'
          description: >-
            The app's own liveness endpoint failed at the edge in the last 10
            minutes. Over the backtested 24h this happened 5 times, all of them
            inside the 04:20Z outage — /healthz failing has a 100% correlation
            with prod actually being broken. Check `docker service ps
            thermograph_web` on prod first.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }

      # -- 3 -----------------------------------------------------------------
      # The acute "users are seeing errors right now" page.
      #
      # DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and
      # the largest *benign* bucket had 5. The outage produced 45 and 22.
      # Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst
      # benign bucket and 4.5x below the incident peak — it fires on the outage
      # and on nothing else in the entire day. It catches 67 of that day's 86
      # errors as a single incident; the remaining scattered ~20 are picked up
      # by ProdHTTP5xxElevated below rather than by pushing this threshold down
      # into the noise.
      - uid: tg_prod_5xx_burst
        title: ProdHTTP5xxBurst
        condition: C
        for: 2m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 10+ HTTP 5xx in the last 10 minutes'
          description: >-
            Prod is serving server errors at an outage-grade rate. Baseline is
            zero in 90% of 10-minute windows and never above 5; the reference
            incident hit 45. Look at the "Thermograph — Traffic & Domains"
            dashboard, panel "HTTP errors by status", and at
            {job="app-json", host="prod", filename=~".+/errors/.+"} for the
            exceptions behind them.
        labels:
          severity: critical
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [9] }

  # ===========================================================================
  # Slow-burn error signals. Five-minute evaluation is plenty; these are tickets,
  # not pages.
  # ===========================================================================
  - orgId: 1
    name: prod-errors
    folder: Alerts
    interval: 5m
    rules:

      # -- 4 -----------------------------------------------------------------
      # The rule that would have broken the silence on "84 5xx in 24h and nobody
      # was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation.
      #
      # DATA: 6-hour rolling windows sampled hourly across the backtest read
      # 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11.
      # The largest window containing NO incident was 11. Threshold 25
      # (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any
      # day that looks like this one. Deliberately a long window: it will stay
      # firing for ~6h after an incident, which is the point. That is the daily
      # error budget talking, not a flap.
      - uid: tg_prod_5xx_elevated
        title: ProdHTTP5xxElevated
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours'
          description: >-
            Sustained server errors. This is the error-budget alarm, not an
            outage alarm — it catches the scattered drip that never trips the
            burst rule but still added up to 86 errors (1.7% of requests) on
            2026-07-24. Expect it to stay firing for up to 6h after an incident
            clears; that is the window emptying, not a new fault.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 21600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [24] }

      # -- 5 -----------------------------------------------------------------
      # A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees
      # 502/503 (the app is unreachable), while tag="http.error" is the app
      # catching an unhandled exception and still answering. A bad deploy shows
      # up here first, and can do so without moving the Caddy numbers at all.
      #
      # DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour.
      # Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak:
      # silent today, loud on an exception storm.
      - uid: tg_prod_unhandled_exceptions
        title: ProdUnhandledExceptions
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 3+ unhandled exceptions in 30 minutes'
          description: >-
            The app is throwing. Normal is under 2 per DAY. A cluster this tight
            almost always means a regression just shipped — check the app.start
            lines on the "App Activity & Delivery" dashboard for a recent deploy,
            then {job="app-json", host="prod", filename=~".+/errors/.+"} |
            tag="http.error" for the traceback and route.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [2] }

  # ===========================================================================
  # The subscription notifier — the thing that actually delivers the product.
  # ===========================================================================
  - orgId: 1
    name: notifier
    folder: Alerts
    interval: 5m
    rules:

      # -- 6 -----------------------------------------------------------------
      # backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it
      # stops, subscribers stop getting weather alerts and absolutely nothing
      # else in this file would notice — the site stays up and serves 200s while
      # the product quietly stops working.
      #
      # DATA: prod produced 1-2 beats in every single 20-minute bucket across
      # the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval
      # means two consecutive empty evaluations, so roughly 25 minutes of real
      # silence before it pages — about one and a half missed beats.
      #
      # dev is deliberately excluded: the dev box (mesh-only, on vps1) had zero
      # heartbeats for the first 10.5h of the backtest window because it was
      # simply off. Alerting on dev would mean an alert that is firing more
      # often than not.
      #
      # MUST stay filtered to daemon="subscription-notifier". The selector was
      # originally bare `tag="heartbeat"`, which was unambiguous while the
      # notifier thread was the only thing that ever beat. It stopped being
      # unambiguous on 2026-07-25T17:24Z, when web/app.py's _start_heartbeat
      # shipped: every uvicorn process in every replica now beats every 300s
      # under the same tag, tagged daemon=web / daemon=worker. Measured
      # immediately after that deploy, a 20m window held 16 web beats and 4
      # worker beats against 2 notifier beats — so the bare selector clears the
      # `< 1` threshold on web traffic alone and can no longer observe the
      # notifier at all.
      #
      # That matters because this rule had just caught a real 17.5h outage
      # (notifier silent 2026-07-24T23:49Z -> 2026-07-25T17:25Z, zero
      # subscription notifications delivered). Left bare, it would have gone
      # quiet on the next occurrence instead of paging. `daemon` is a JSON field
      # and not a stream label, hence the `| json |` parse rather than a label
      # matcher.
      - uid: tg_notifier_heartbeat_prod
        title: NotifierHeartbeatMissingProd
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no notifier heartbeat for 20 minutes'
          description: >-
            The subscription notifier has gone quiet on prod. Expected cadence is
            ~15 minutes and it did not miss a single 20-minute window over the
            backtested day. Subscribers are not being notified right now, even
            though the website is probably still serving fine. Check the
            thermograph_worker service and the "Notifier alive?" panel on the
            Fleet Logs dashboard.
        labels:
          severity: critical
          host: prod
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 7 -----------------------------------------------------------------
      # Same signal on beta. Beta was equally gap-free over the backtest (1-2
      # beats every 20m, no empty buckets), but beta is UAT — worth knowing,
      # not worth a page.
      - uid: tg_notifier_heartbeat_beta
        title: NotifierHeartbeatMissingBeta
        condition: C
        for: 15m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'beta: no notifier heartbeat for 20 minutes'
          description: >-
            The notifier has gone quiet on beta (UAT). Same signal as the prod
            rule but a longer `for`, because beta gets redeployed all day and a
            short gap is usually just a release.
        labels:
          severity: warning
          host: beta
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

  # ===========================================================================
  # Per-service liveness. The fleet collects no container metrics, so liveness
  # has to be inferred from "is this thing still producing output at all". Two
  # different sources are used here, and picking the wrong one is how this group
  # generated a day of false pages:
  #
  #   * db and frontend infer it from CONTAINER LOG VOLUME. Valid for these two:
  #     both log to stdout on a floor that never approaches zero (db a metronomic
  #     54-69 lines/2h, frontend ~66/30m), and prod's Alloy ships both.
  #
  #   * web and worker infer it from the APP'S OWN HEARTBEAT (tag=heartbeat in
  #     the app-json file source, filtered by the daemon field). Container log
  #     volume is NOT a valid liveness signal for either: web's request logging
  #     moved off stdout onto app-json, and worker's stdout is dropped outright
  #     by a rule in prod's alloy config. Both containers can be perfectly
  #     healthy while their docker streams sit at exactly zero forever.
  #
  # Before adding a container-log-silence rule for a new service, confirm the
  # service actually logs to stdout AND that prod's discovery.relabel "containers"
  # block does not drop it. A rule against a dropped stream is unsatisfiable and
  # will page forever.
  #
  # Caveat that still applies to the db/frontend rules: a container can be
  # running and logging while its lines never reach Loki, if the node's Alloy
  # agent stops tailing it. Those rules cannot tell the two apart — they say
  # "no logs", which is the honest claim. Either way something needs looking at.
  # ===========================================================================
  - orgId: 1
    name: prod-services
    folder: Alerts
    interval: 5m
    rules:

      # -- 8 -----------------------------------------------------------------
      # Web-tier liveness, from the app's own heartbeat rather than container
      # stdout.
      #
      # This rule used to count `{job="docker", container=~"thermograph_web.+"}`
      # over 30m against a measured floor of ~110 lines. That floor was real when
      # it was written (the backtest saw 3.8k-10.5k lines per 6h), but the app
      # moved its request logging off stdout onto the app-json file source, and
      # from 2026-07-25T00:02Z a web container emits ~15 uvicorn banner lines for
      # its entire lifetime and nothing after. The old rule then sat firing
      # permanently on a perfectly healthy tier, clearing only for the few
      # minutes after each deploy or autoscale event produced fresh banners —
      # 20+ pages in a day, every one of them false. See web/app.py's
      # _start_heartbeat docstring, which states outright that stdout silence
      # cannot distinguish a live web process from a dead one and that this
      # heartbeat is the signal to alert on instead.
      #
      # DATA: _start_heartbeat beats every 300s from every uvicorn process in
      # every replica, unguarded by the leader election, so a 4-process replica
      # yields ~16 beats per 20m (measured: exactly 16). Zero over 20m means no
      # web process is alive anywhere. This is the app itself, so it pages.
      - uid: tg_prod_web_silent
        title: ProdWebHeartbeatMissing
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no web heartbeat for 20 minutes'
          description: >-
            No thermograph_web process on prod has emitted a liveness heartbeat
            in 20 minutes. Every uvicorn process beats every 5 minutes
            independently of the notifier leader election, so a healthy 4-process
            replica produces ~16 beats per window — zero means no web process is
            alive anywhere, not merely quiet. Run `docker service ps
            thermograph_web` on prod. Note this is the app's own heartbeat, not
            container stdout: web's stdout is silent by design and is not a
            liveness signal.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="web" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 9 -----------------------------------------------------------------
      # DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69
      # lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a
      # metronomic ~30 lines/hour. Zero over two hours is as close to a
      # definitive "Postgres is not there" as logs can give.
      - uid: tg_prod_db_silent
        title: ProdDbContainerSilent
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_db has logged nothing for 2 hours'
          description: >-
            The production TimescaleDB container has produced zero log lines in
            2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no
            observed gaps, so zero is not quiet — it is absent. Everything else
            in the app depends on this.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 7200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 10 ----------------------------------------------------------------
      # DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than
      # critical because Caddy serves the static build and ProdEdgeDark /
      # ProdHTTP5xxBurst will catch it first if users are actually affected.
      - uid: tg_prod_frontend_silent
        title: ProdFrontendContainerSilent
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_frontend has logged nothing for 30 minutes'
          description: >-
            The frontend container has produced zero log lines in 30 minutes
            (healthy floor ~66 per 30m). If users were visibly affected you would
            expect ProdHTTP5xxBurst alongside this; on its own it more often
            means a stuck redeploy or a lost Alloy tail.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

      # -- 11 ----------------------------------------------------------------
      # Worker liveness, from the app's own heartbeat rather than container
      # stdout.
      #
      # The previous rule counted `{job="docker", container=~"thermograph_worker.+"}`
      # and was UNSATISFIABLE BY CONSTRUCTION — it could never return anything but
      # zero, no matter how healthy the worker was. prod's Alloy config drops that
      # container on purpose, in discovery.relabel "containers":
      #
      #   regex  = "(alloy|autoscaler|thermograph-test_.*|thermograph-lb|thermograph_worker).*"
      #   action = "drop"
      #
      # with the stated reasoning that worker stdout was 100% /healthz poll noise
      # and the app's own access/*.jsonl is a strict superset of it. So the stream
      # this rule queried is discarded at the agent and never reaches Loki.
      #
      # The old comment here read the resulting permanent zero as "Alloy stopped
      # tailing the container after Swarm replaced the task" and pointed the
      # runbook at restarting alloy-alloy-1. That diagnosis was wrong, and the
      # fix it recommended could not have worked: a restarted Alloy re-reads the
      # same config and drops the same container again. The rule fired
      # continuously from 2026-07-24T21:05Z with no underlying fault.
      #
      # DATA: _start_heartbeat beats every 300s from the worker process
      # (measured: exactly 4 per 20m). Unlike the notifier below it is not gated
      # on the leader election, so it reports the process being alive even when
      # this worker is not the elected notifier — which is the distinction the
      # old rule was trying and failing to draw.
      - uid: tg_prod_worker_silent
        title: ProdWorkerHeartbeatMissing
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no worker heartbeat for 20 minutes'
          description: >-
            The thermograph_worker process on prod has not emitted a liveness
            heartbeat in 20 minutes (healthy is ~4 per window, one every 5
            minutes). This beat is independent of the notifier leader election,
            so it means the worker process itself is gone or wedged, not merely
            that it lost the election. Run `docker service ps thermograph_worker`
            on prod. Do not look at the worker's container logs — prod's Alloy
            config drops that stream deliberately, so it is always empty and is
            not evidence of anything.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="worker" [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }

  # ===========================================================================
  # The dead-man's switch.
  # ===========================================================================
  - orgId: 1
    name: watchdog
    folder: Alerts
    interval: 5m
    rules:

      # -- 12 ----------------------------------------------------------------
      # Nothing in this file can tell you that Grafana itself has stopped, or
      # that vps1 (the monitoring host) is down, or that the Discord webhook was
      # revoked — a monitoring system cannot report its own death. So this rule
      # fires permanently and by design, throttled by the notification policy
      # to exactly one Discord message per 24 hours.
      #
      # HOW TO USE IT: if #ops-alerts has been silent for more than a day, the
      # alerting pipeline is broken, not the estate healthy. That one daily
      # message is the only evidence you will ever get that the other eleven
      # rules are still capable of reaching you.
      #
      # To silence it (and lose that guarantee), set isPaused: true.
      #
      - uid: tg_alerting_watchdog
        title: AlertingWatchdog
        condition: C
        for: 0m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)'
          description: >-
            This is the dead-man's switch and it is supposed to be firing. It
            proves Grafana on vps1 is evaluating rules, Loki is answering, and
            the Discord webhook still works. Seeing it once a day is correct.
            NOT seeing it for over a day means alerting itself is down and no
            other rule can reach you — check observability-grafana-1 on vps1.
        labels:
          severity: watchdog
          # `node`, not `host`: every other rule's `host` names an ENVIRONMENT
          # (prod/beta/dev), and this rule is about the alerting pipeline
          # itself, which is a machine. It used to be filed `host: beta` back
          # when the monitoring box and the beta environment were the same
          # machine — they are not any more, and that label would now read as a
          # claim about beta. Routing is unaffected: notification-policies.yml
          # matches on `severity` only and uses `host` just for grouping.
          node: vps1
          class: meta
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'vector(1)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }