# Thermograph alert rules — the estate's first. All Loki/LogQL, because there is # no Prometheus anywhere in the fleet: Loki is the only datasource, so every # signal here is derived from log lines rather than metrics. # # --------------------------------------------------------------------------- # HOW THE THRESHOLDS WERE CHOSEN # --------------------------------------------------------------------------- # Every number below was backtested against the real 24h ending 2026-07-24 # ~19:45Z. That window contains a genuine ~20-minute prod outage starting # 2026-07-24 04:20Z, which is the event these rules must catch: # # 10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown): # 2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1 (total 86) # ^^^^^^ the outage: 64% and 56% of all requests 5xx # All five /healthz failures in the whole day fell inside those two buckets. # Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic — # which is why absolute counts beat error *ratios* here: at 7 req/min a single # bad request is 1.4% and a ratio alert would scream all night. # # The design goal was: fire on the outage, stay silent for the other 142 of 144 # buckets, and still surface the slow drip of ~20 scattered errors that made up # the rest of the day's 86 (that is what ProdHTTP5xxElevated is for). # # --------------------------------------------------------------------------- # TWO CONVENTIONS WORTH KNOWING # --------------------------------------------------------------------------- # 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a # naive "count < 1" silence alert goes to NoData instead of firing, which is # the exact failure mode of every homemade log alert ever written. Appending # `or vector(0)` makes an empty result evaluate to a real 0. Verified working # on Loki 3.5.1. # 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service` # label — only `container`, which includes a per-task suffix that changes on # every redeploy (thermograph_web.1.). So prod services are matched as # container=~"thermograph_.+" with job="docker". Do not use `service=` # here; it only works on the compose hosts (beta, dev). # # --------------------------------------------------------------------------- # NOT ALERTABLE, ON PURPOSE # --------------------------------------------------------------------------- # thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and # thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their # healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL # with long legitimate idle gaps. Catching those needs either a heartbeat line # (extend backend/core/audit.py log_heartbeat(), as the notifier already does) # or container metrics, which the fleet does not collect. Tracked as a gap. # # Error budget / failure isolation: only ProdEdgeDark escalates on NoData or # datasource errors, so a Loki or WireGuard blip produces ONE page rather than # twelve. Every other rule degrades to OK if the query cannot run. apiVersion: 1 groups: # =========================================================================== # Is the product up? Evaluated every minute; all of these page. # =========================================================================== - orgId: 1 name: prod-availability folder: Alerts interval: 1m rules: # -- 1 ----------------------------------------------------------------- # The top-level "is anything alive" check, and deliberately the only rule # that also owns NoData/error: Caddy fronts every request to # thermograph.org, so zero access-log lines means the site is dark, the # box is gone, Alloy died, or the mesh dropped. Any of those is a page. # # DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35, # and never once reached zero — including *during* the outage (72), when # Caddy was happily logging 503s. Zero is unambiguous. - uid: tg_prod_edge_dark title: ProdEdgeDark condition: C for: 5m noDataState: Alerting execErrState: Alerting annotations: summary: 'prod: no Caddy access-log lines for 15 minutes' description: >- Nothing has reached thermograph.org's reverse proxy in 15 minutes (healthy floor is 35 lines/15m, and it stayed at 72 even during the 04:20Z outage). Either prod is down, Caddy is down, prod's Alloy agent stopped shipping, or the WireGuard mesh to beta is broken. This rule also fires on Loki NoData/errors, so it doubles as the "we have lost observability" alarm — if it is the only thing firing, suspect the pipeline before suspecting the app. labels: severity: critical host: prod class: availability isPaused: false data: - refId: A relativeTimeRange: { from: 900, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: lt, params: [1] } # -- 2 ----------------------------------------------------------------- # /healthz is the app's own statement that it is well. When it 503s, it is # not an opinion. # # DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour, # so this endpoint is polled sparsely and irregularly. That is why `for` # is 0m: waiting for a second consecutive bad sample could take 25 # minutes. All 5 failures landed inside the two outage buckets and nowhere # else in the day, so one failure is already a real signal, not noise. - uid: tg_prod_healthz_failing title: ProdHealthzFailing condition: C for: 0m noDataState: OK execErrState: OK annotations: summary: 'prod: /healthz returned a non-200' description: >- The app's own liveness endpoint failed at the edge in the last 10 minutes. Over the backtested 24h this happened 5 times, all of them inside the 04:20Z outage — /healthz failing has a 100% correlation with prod actually being broken. Check `docker service ps thermograph_web` on prod first. labels: severity: critical host: prod class: availability isPaused: false data: - refId: A relativeTimeRange: { from: 600, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: gt, params: [0] } # -- 3 ----------------------------------------------------------------- # The acute "users are seeing errors right now" page. # # DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and # the largest *benign* bucket had 5. The outage produced 45 and 22. # Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst # benign bucket and 4.5x below the incident peak — it fires on the outage # and on nothing else in the entire day. It catches 67 of that day's 86 # errors as a single incident; the remaining scattered ~20 are picked up # by ProdHTTP5xxElevated below rather than by pushing this threshold down # into the noise. - uid: tg_prod_5xx_burst title: ProdHTTP5xxBurst condition: C for: 2m noDataState: OK execErrState: OK annotations: summary: 'prod: 10+ HTTP 5xx in the last 10 minutes' description: >- Prod is serving server errors at an outage-grade rate. Baseline is zero in 90% of 10-minute windows and never above 5; the reference incident hit 45. Look at the "Thermograph — Traffic & Domains" dashboard, panel "HTTP errors by status", and at {job="app-json", host="prod", filename=~".+/errors/.+"} for the exceptions behind them. labels: severity: critical host: prod class: errors isPaused: false data: - refId: A relativeTimeRange: { from: 600, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: gt, params: [9] } # =========================================================================== # Slow-burn error signals. Five-minute evaluation is plenty; these are tickets, # not pages. # =========================================================================== - orgId: 1 name: prod-errors folder: Alerts interval: 5m rules: # -- 4 ----------------------------------------------------------------- # The rule that would have broken the silence on "84 5xx in 24h and nobody # was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation. # # DATA: 6-hour rolling windows sampled hourly across the backtest read # 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11. # The largest window containing NO incident was 11. Threshold 25 # (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any # day that looks like this one. Deliberately a long window: it will stay # firing for ~6h after an incident, which is the point. That is the daily # error budget talking, not a flap. - uid: tg_prod_5xx_elevated title: ProdHTTP5xxElevated condition: C for: 10m noDataState: OK execErrState: OK annotations: summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours' description: >- Sustained server errors. This is the error-budget alarm, not an outage alarm — it catches the scattered drip that never trips the burst rule but still added up to 86 errors (1.7% of requests) on 2026-07-24. Expect it to stay firing for up to 6h after an incident clears; that is the window emptying, not a new fault. labels: severity: warning host: prod class: errors isPaused: false data: - refId: A relativeTimeRange: { from: 21600, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: gt, params: [24] } # -- 5 ----------------------------------------------------------------- # A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees # 502/503 (the app is unreachable), while tag="http.error" is the app # catching an unhandled exception and still answering. A bad deploy shows # up here first, and can do so without moving the Caddy numbers at all. # # DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour. # Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak: # silent today, loud on an exception storm. - uid: tg_prod_unhandled_exceptions title: ProdUnhandledExceptions condition: C for: 0m noDataState: OK execErrState: OK annotations: summary: 'prod: 3+ unhandled exceptions in 30 minutes' description: >- The app is throwing. Normal is under 2 per DAY. A cluster this tight almost always means a regression just shipped — check the app.start lines on the "App Activity & Delivery" dashboard for a recent deploy, then {job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" for the traceback and route. labels: severity: warning host: prod class: errors isPaused: false data: - refId: A relativeTimeRange: { from: 1800, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: gt, params: [2] } # =========================================================================== # The subscription notifier — the thing that actually delivers the product. # =========================================================================== - orgId: 1 name: notifier folder: Alerts interval: 5m rules: # -- 6 ----------------------------------------------------------------- # backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it # stops, subscribers stop getting weather alerts and absolutely nothing # else in this file would notice — the site stays up and serves 200s while # the product quietly stops working. # # DATA: prod produced 1-2 beats in every single 20-minute bucket across # the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval # means two consecutive empty evaluations, so roughly 25 minutes of real # silence before it pages — about one and a half missed beats. # # dev is deliberately excluded: the LAN box had zero heartbeats for the # first 10.5h of the backtest window because it was simply off. Alerting # on dev would mean an alert that is firing more often than not. - uid: tg_notifier_heartbeat_prod title: NotifierHeartbeatMissingProd condition: C for: 5m noDataState: OK execErrState: OK annotations: summary: 'prod: no notifier heartbeat for 20 minutes' description: >- The subscription notifier has gone quiet on prod. Expected cadence is ~15 minutes and it did not miss a single 20-minute window over the backtested day. Subscribers are not being notified right now, even though the website is probably still serving fine. Check the thermograph_worker service and the "Notifier alive?" panel on the Fleet Logs dashboard. labels: severity: critical host: prod class: product isPaused: false data: - refId: A relativeTimeRange: { from: 1200, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} [20m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: lt, params: [1] } # -- 7 ----------------------------------------------------------------- # Same signal on beta. Beta was equally gap-free over the backtest (1-2 # beats every 20m, no empty buckets), but beta is UAT — worth knowing, # not worth a page. - uid: tg_notifier_heartbeat_beta title: NotifierHeartbeatMissingBeta condition: C for: 15m noDataState: OK execErrState: OK annotations: summary: 'beta: no notifier heartbeat for 20 minutes' description: >- The notifier has gone quiet on beta (UAT). Same signal as the prod rule but a longer `for`, because beta gets redeployed all day and a short gap is usually just a release. labels: severity: warning host: beta class: product isPaused: false data: - refId: A relativeTimeRange: { from: 1200, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} [20m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: lt, params: [1] } # =========================================================================== # Per-container liveness, inferred from log volume. The fleet collects no # container metrics, so "did this service stop producing any output at all" is # the only death certificate available — and it turns out to be a good one, # because every service below logs on a floor that never approaches zero. # # Caveat worth knowing before you trust these: a container can be running and # logging while its lines never reach Loki, if the node's Alloy agent stops # tailing it. That is a real observed failure on prod (see README). These # rules cannot tell the two apart — they say "no logs", which is the honest # claim. Either way something needs looking at. # =========================================================================== - orgId: 1 name: prod-services folder: Alerts interval: 5m rules: # -- 8 ----------------------------------------------------------------- # DATA: 15-minute windows over 24h ranged 54 -> 954 lines. The minimum, 54, # occurred during the outage; the quietest healthy window was ~150. A # 30-minute window has never been below ~110. Zero means the container is # gone. This is the app itself, so it pages. - uid: tg_prod_web_silent title: ProdWebContainerSilent condition: C for: 5m noDataState: OK execErrState: OK annotations: summary: 'prod: thermograph_web has logged nothing for 30 minutes' description: >- The main app container has produced zero log lines in 30 minutes. Healthy floor is ~110 lines per 30m and it never dipped below 54 even mid-outage, so this is a container that is gone, wedged, or no longer being tailed by prod's Alloy agent. Run `docker service ps thermograph_web` on prod. labels: severity: critical host: prod class: liveness isPaused: false data: - refId: A relativeTimeRange: { from: 1800, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_web.+"} [30m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: lt, params: [1] } # -- 9 ----------------------------------------------------------------- # DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69 # lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a # metronomic ~30 lines/hour. Zero over two hours is as close to a # definitive "Postgres is not there" as logs can give. - uid: tg_prod_db_silent title: ProdDbContainerSilent condition: C for: 10m noDataState: OK execErrState: OK annotations: summary: 'prod: thermograph_db has logged nothing for 2 hours' description: >- The production TimescaleDB container has produced zero log lines in 2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no observed gaps, so zero is not quiet — it is absent. Everything else in the app depends on this. labels: severity: critical host: prod class: liveness isPaused: false data: - refId: A relativeTimeRange: { from: 7200, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: lt, params: [1] } # -- 10 ---------------------------------------------------------------- # DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than # critical because Caddy serves the static build and ProdEdgeDark / # ProdHTTP5xxBurst will catch it first if users are actually affected. - uid: tg_prod_frontend_silent title: ProdFrontendContainerSilent condition: C for: 5m noDataState: OK execErrState: OK annotations: summary: 'prod: thermograph_frontend has logged nothing for 30 minutes' description: >- The frontend container has produced zero log lines in 30 minutes (healthy floor ~66 per 30m). If users were visibly affected you would expect ProdHTTP5xxBurst alongside this; on its own it more often means a stuck redeploy or a lost Alloy tail. labels: severity: warning host: prod class: liveness isPaused: false data: - refId: A relativeTimeRange: { from: 1800, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: lt, params: [1] } # -- 11 ---------------------------------------------------------------- # DATA: a dead-steady 121 lines/hour (its own healthcheck polling) for the # first 14h of the backtest, then nothing at all. # # HEADS UP: THIS RULE WILL BE FIRING THE MOMENT YOU DEPLOY IT, and it is # right to. As of 2026-07-24 19:45Z the prod worker container is running # and healthy and is writing ~19 lines per 10 minutes to its own docker # log — but not one of those lines has reached Loki since 04:43Z. Prod's # Alloy agent stopped tailing the container when Swarm replaced the task. # Nobody knew, because nothing was watching. See README "Known gaps". # # WHY THIS IS A WARNING AND NotifierHeartbeatMissingProd IS CRITICAL: the # two draw on *different* Alloy sources — this one on loki.source.docker # (the container's stdout), the heartbeat on loki.source.file tailing # /applogs. That independence is the diagnostic. Right now prod's # heartbeats are arriving in every single 20-minute window while its # worker docker stream has been empty for 14 hours, which localises the # fault to Alloy's docker tail rather than to the worker: the process is # doing its job, we just cannot see it breathe. Both rules firing together # means the worker really is down. - uid: tg_prod_worker_silent title: ProdWorkerContainerSilent condition: C for: 10m noDataState: OK execErrState: OK annotations: summary: 'prod: thermograph_worker has logged nothing for 2 hours' description: >- The worker container has produced zero log lines in 2 hours against a healthy floor of ~242 per 2h. Two different faults look identical here: the worker is actually dead, or prod's Alloy agent has stopped tailing it (a real, observed failure — Alloy can miss a container after Swarm replaces the task). Check both: `docker service ps thermograph_worker` on prod, and whether `docker logs` on the live container is producing lines that Loki does not have. Restarting alloy-alloy-1 on prod fixes the second case. labels: severity: warning host: prod class: liveness isPaused: false data: - refId: A relativeTimeRange: { from: 7200, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_worker.+"} [2h])) or vector(0)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: lt, params: [1] } # =========================================================================== # The dead-man's switch. # =========================================================================== - orgId: 1 name: watchdog folder: Alerts interval: 5m rules: # -- 12 ---------------------------------------------------------------- # Nothing in this file can tell you that Grafana itself has stopped, or # that beta is down, or that the Discord webhook was revoked — a monitoring # system cannot report its own death. So this rule fires permanently and # by design, throttled by the notification policy to exactly one Discord # message per 24 hours. # # HOW TO USE IT: if #ops-alerts has been silent for more than a day, the # alerting pipeline is broken, not the estate healthy. That one daily # message is the only evidence you will ever get that the other eleven # rules are still capable of reaching you. # # To silence it (and lose that guarantee), set isPaused: true. - uid: tg_alerting_watchdog title: AlertingWatchdog condition: C for: 0m noDataState: Alerting execErrState: Alerting annotations: summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)' description: >- This is the dead-man's switch and it is supposed to be firing. It proves Grafana on beta is evaluating rules, Loki is answering, and the Discord webhook still works. Seeing it once a day is correct. NOT seeing it for over a day means alerting itself is down and no other rule can reach you — check observability-grafana-1 on beta. labels: severity: watchdog host: beta class: meta isPaused: false data: - refId: A relativeTimeRange: { from: 600, to: 0 } datasourceUid: loki model: refId: A datasource: { type: loki, uid: loki } expr: 'vector(1)' queryType: instant editorMode: code intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: B type: reduce reducer: last expression: A datasource: { type: __expr__, uid: __expr__ } - refId: C relativeTimeRange: { from: 0, to: 0 } datasourceUid: __expr__ model: refId: C type: threshold expression: B datasource: { type: __expr__, uid: __expr__ } conditions: - evaluator: { type: gt, params: [0] }