Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 10 Projects Releases Packages Wiki Activity Actions 1
thermograph/observability/grafana/provisioning/alerting/rules-prod.yml
Emi Griffith e4693dce58
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 8s
Details
shell-lint / shellcheck (pull_request) Successful in 10s
Details
PR build (required check) / changes (pull_request) Successful in 16s
Details
PR build (required check) / validate-observability (pull_request) Successful in 18s
Details
PR build (required check) / gate (pull_request) Successful in 2s
Details
PR build (required check) / build-backend (pull_request) Has been skipped
Details
PR build (required check) / build-frontend (pull_request) Has been skipped
Details
observability: add the estate's first alerting; supervise Postfix 
The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.

Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
  only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
  prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
  one bad request is 1.4% and a ratio alert would scream all night. The 5xx
  burst rule fires on the outage's 45 and 22 buckets and on nothing else in
  the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
  #weather-events, #announcements and #prod are product surfaces that notify
  real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
  Grafana against beta's live Loki and a real alert arrived in Discord. This
  matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
  for the URL — a contact point holding an uninterpolated env var looks
  perfectly healthy and pages nobody. The only proof is a message arriving.

CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.

Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
  started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
  docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
  Stock postfix@.service is ordered only After=network-online.target and ships
  no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
  address actually exists, which absorbs the transient case inside a single
  start attempt. That makes bounded retry correct: 5 attempts in 600s, then
  failed — a genuinely broken config reaches a visible failed state in ~100s
  instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
  "failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
  mesh listeners that do not depend on Docker at all.

Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.

DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.

Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 13:19:19 -07:00

836 lines
35 KiB
YAML
Raw Blame History

# Thermograph alert rules — the estate's first. All Loki/LogQL, because there is
# no Prometheus anywhere in the fleet: Loki is the only datasource, so every
# signal here is derived from log lines rather than metrics.
#
# ---------------------------------------------------------------------------
# HOW THE THRESHOLDS WERE CHOSEN
# ---------------------------------------------------------------------------
# Every number below was backtested against the real 24h ending 2026-07-24
# ~19:45Z. That window contains a genuine ~20-minute prod outage starting
# 2026-07-24 04:20Z, which is the event these rules must catch:
#
# 10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown):
# 2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1 (total 86)
# ^^^^^^ the outage: 64% and 56% of all requests 5xx
# All five /healthz failures in the whole day fell inside those two buckets.
# Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic —
# which is why absolute counts beat error *ratios* here: at 7 req/min a single
# bad request is 1.4% and a ratio alert would scream all night.
#
# The design goal was: fire on the outage, stay silent for the other 142 of 144
# buckets, and still surface the slow drip of ~20 scattered errors that made up
# the rest of the day's 86 (that is what ProdHTTP5xxElevated is for).
#
# ---------------------------------------------------------------------------
# TWO CONVENTIONS WORTH KNOWING
# ---------------------------------------------------------------------------
# 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a
# naive "count < 1" silence alert goes to NoData instead of firing, which is
# the exact failure mode of every homemade log alert ever written. Appending
# `or vector(0)` makes an empty result evaluate to a real 0. Verified working
# on Loki 3.5.1.
# 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service`
# label — only `container`, which includes a per-task suffix that changes on
# every redeploy (thermograph_web.1.<taskid>). So prod services are matched as
# container=~"thermograph_<name>.+" with job="docker". Do not use `service=`
# here; it only works on the compose hosts (beta, dev).
#
# ---------------------------------------------------------------------------
# NOT ALERTABLE, ON PURPOSE
# ---------------------------------------------------------------------------
# thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and
# thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their
# healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL
# with long legitimate idle gaps. Catching those needs either a heartbeat line
# (extend backend/core/audit.py log_heartbeat(), as the notifier already does)
# or container metrics, which the fleet does not collect. Tracked as a gap.
#
# Error budget / failure isolation: only ProdEdgeDark escalates on NoData or
# datasource errors, so a Loki or WireGuard blip produces ONE page rather than
# twelve. Every other rule degrades to OK if the query cannot run.
apiVersion: 1
groups:
# ===========================================================================
# Is the product up? Evaluated every minute; all of these page.
# ===========================================================================
- orgId: 1
name: prod-availability
folder: Alerts
interval: 1m
rules:
# -- 1 -----------------------------------------------------------------
# The top-level "is anything alive" check, and deliberately the only rule
# that also owns NoData/error: Caddy fronts every request to
# thermograph.org, so zero access-log lines means the site is dark, the
# box is gone, Alloy died, or the mesh dropped. Any of those is a page.
#
# DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35,
# and never once reached zero — including *during* the outage (72), when
# Caddy was happily logging 503s. Zero is unambiguous.
- uid: tg_prod_edge_dark
title: ProdEdgeDark
condition: C
for: 5m
noDataState: Alerting
execErrState: Alerting
annotations:
summary: 'prod: no Caddy access-log lines for 15 minutes'
description: >-
Nothing has reached thermograph.org's reverse proxy in 15 minutes
(healthy floor is 35 lines/15m, and it stayed at 72 even during the
04:20Z outage). Either prod is down, Caddy is down, prod's Alloy
agent stopped shipping, or the WireGuard mesh to beta is broken.
This rule also fires on Loki NoData/errors, so it doubles as the
"we have lost observability" alarm — if it is the only thing firing,
suspect the pipeline before suspecting the app.
labels:
severity: critical
host: prod
class: availability
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 900, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: lt, params: [1] }
# -- 2 -----------------------------------------------------------------
# /healthz is the app's own statement that it is well. When it 503s, it is
# not an opinion.
#
# DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour,
# so this endpoint is polled sparsely and irregularly. That is why `for`
# is 0m: waiting for a second consecutive bad sample could take 25
# minutes. All 5 failures landed inside the two outage buckets and nowhere
# else in the day, so one failure is already a real signal, not noise.
- uid: tg_prod_healthz_failing
title: ProdHealthzFailing
condition: C
for: 0m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: /healthz returned a non-200'
description: >-
The app's own liveness endpoint failed at the edge in the last 10
minutes. Over the backtested 24h this happened 5 times, all of them
inside the 04:20Z outage — /healthz failing has a 100% correlation
with prod actually being broken. Check `docker service ps
thermograph_web` on prod first.
labels:
severity: critical
host: prod
class: availability
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 600, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: gt, params: [0] }
# -- 3 -----------------------------------------------------------------
# The acute "users are seeing errors right now" page.
#
# DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and
# the largest *benign* bucket had 5. The outage produced 45 and 22.
# Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst
# benign bucket and 4.5x below the incident peak — it fires on the outage
# and on nothing else in the entire day. It catches 67 of that day's 86
# errors as a single incident; the remaining scattered ~20 are picked up
# by ProdHTTP5xxElevated below rather than by pushing this threshold down
# into the noise.
- uid: tg_prod_5xx_burst
title: ProdHTTP5xxBurst
condition: C
for: 2m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: 10+ HTTP 5xx in the last 10 minutes'
description: >-
Prod is serving server errors at an outage-grade rate. Baseline is
zero in 90% of 10-minute windows and never above 5; the reference
incident hit 45. Look at the "Thermograph — Traffic & Domains"
dashboard, panel "HTTP errors by status", and at
{job="app-json", host="prod", filename=~".+/errors/.+"} for the
exceptions behind them.
labels:
severity: critical
host: prod
class: errors
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 600, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: gt, params: [9] }
# ===========================================================================
# Slow-burn error signals. Five-minute evaluation is plenty; these are tickets,
# not pages.
# ===========================================================================
- orgId: 1
name: prod-errors
folder: Alerts
interval: 5m
rules:
# -- 4 -----------------------------------------------------------------
# The rule that would have broken the silence on "84 5xx in 24h and nobody
# was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation.
#
# DATA: 6-hour rolling windows sampled hourly across the backtest read
# 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11.
# The largest window containing NO incident was 11. Threshold 25
# (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any
# day that looks like this one. Deliberately a long window: it will stay
# firing for ~6h after an incident, which is the point. That is the daily
# error budget talking, not a flap.
- uid: tg_prod_5xx_elevated
title: ProdHTTP5xxElevated
condition: C
for: 10m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours'
description: >-
Sustained server errors. This is the error-budget alarm, not an
outage alarm — it catches the scattered drip that never trips the
burst rule but still added up to 86 errors (1.7% of requests) on
2026-07-24. Expect it to stay firing for up to 6h after an incident
clears; that is the window emptying, not a new fault.
labels:
severity: warning
host: prod
class: errors
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 21600, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: gt, params: [24] }
# -- 5 -----------------------------------------------------------------
# A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees
# 502/503 (the app is unreachable), while tag="http.error" is the app
# catching an unhandled exception and still answering. A bad deploy shows
# up here first, and can do so without moving the Caddy numbers at all.
#
# DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour.
# Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak:
# silent today, loud on an exception storm.
- uid: tg_prod_unhandled_exceptions
title: ProdUnhandledExceptions
condition: C
for: 0m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: 3+ unhandled exceptions in 30 minutes'
description: >-
The app is throwing. Normal is under 2 per DAY. A cluster this tight
almost always means a regression just shipped — check the app.start
lines on the "App Activity & Delivery" dashboard for a recent deploy,
then {job="app-json", host="prod", filename=~".+/errors/.+"} |
tag="http.error" for the traceback and route.
labels:
severity: warning
host: prod
class: errors
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 1800, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: gt, params: [2] }
# ===========================================================================
# The subscription notifier — the thing that actually delivers the product.
# ===========================================================================
- orgId: 1
name: notifier
folder: Alerts
interval: 5m
rules:
# -- 6 -----------------------------------------------------------------
# backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it
# stops, subscribers stop getting weather alerts and absolutely nothing
# else in this file would notice — the site stays up and serves 200s while
# the product quietly stops working.
#
# DATA: prod produced 1-2 beats in every single 20-minute bucket across
# the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval
# means two consecutive empty evaluations, so roughly 25 minutes of real
# silence before it pages — about one and a half missed beats.
#
# dev is deliberately excluded: the LAN box had zero heartbeats for the
# first 10.5h of the backtest window because it was simply off. Alerting
# on dev would mean an alert that is firing more often than not.
- uid: tg_notifier_heartbeat_prod
title: NotifierHeartbeatMissingProd
condition: C
for: 5m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: no notifier heartbeat for 20 minutes'
description: >-
The subscription notifier has gone quiet on prod. Expected cadence is
~15 minutes and it did not miss a single 20-minute window over the
backtested day. Subscribers are not being notified right now, even
though the website is probably still serving fine. Check the
thermograph_worker service and the "Notifier alive?" panel on the
Fleet Logs dashboard.
labels:
severity: critical
host: prod
class: product
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 1200, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} [20m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: lt, params: [1] }
# -- 7 -----------------------------------------------------------------
# Same signal on beta. Beta was equally gap-free over the backtest (1-2
# beats every 20m, no empty buckets), but beta is UAT — worth knowing,
# not worth a page.
- uid: tg_notifier_heartbeat_beta
title: NotifierHeartbeatMissingBeta
condition: C
for: 15m
noDataState: OK
execErrState: OK
annotations:
summary: 'beta: no notifier heartbeat for 20 minutes'
description: >-
The notifier has gone quiet on beta (UAT). Same signal as the prod
rule but a longer `for`, because beta gets redeployed all day and a
short gap is usually just a release.
labels:
severity: warning
host: beta
class: product
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 1200, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} [20m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: lt, params: [1] }
# ===========================================================================
# Per-container liveness, inferred from log volume. The fleet collects no
# container metrics, so "did this service stop producing any output at all" is
# the only death certificate available — and it turns out to be a good one,
# because every service below logs on a floor that never approaches zero.
#
# Caveat worth knowing before you trust these: a container can be running and
# logging while its lines never reach Loki, if the node's Alloy agent stops
# tailing it. That is a real observed failure on prod (see README). These
# rules cannot tell the two apart — they say "no logs", which is the honest
# claim. Either way something needs looking at.
# ===========================================================================
- orgId: 1
name: prod-services
folder: Alerts
interval: 5m
rules:
# -- 8 -----------------------------------------------------------------
# DATA: 15-minute windows over 24h ranged 54 -> 954 lines. The minimum, 54,
# occurred during the outage; the quietest healthy window was ~150. A
# 30-minute window has never been below ~110. Zero means the container is
# gone. This is the app itself, so it pages.
- uid: tg_prod_web_silent
title: ProdWebContainerSilent
condition: C
for: 5m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: thermograph_web has logged nothing for 30 minutes'
description: >-
The main app container has produced zero log lines in 30 minutes.
Healthy floor is ~110 lines per 30m and it never dipped below 54 even
mid-outage, so this is a container that is gone, wedged, or no longer
being tailed by prod's Alloy agent. Run `docker service ps
thermograph_web` on prod.
labels:
severity: critical
host: prod
class: liveness
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 1800, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_web.+"} [30m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: lt, params: [1] }
# -- 9 -----------------------------------------------------------------
# DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69
# lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a
# metronomic ~30 lines/hour. Zero over two hours is as close to a
# definitive "Postgres is not there" as logs can give.
- uid: tg_prod_db_silent
title: ProdDbContainerSilent
condition: C
for: 10m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: thermograph_db has logged nothing for 2 hours'
description: >-
The production TimescaleDB container has produced zero log lines in
2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no
observed gaps, so zero is not quiet — it is absent. Everything else
in the app depends on this.
labels:
severity: critical
host: prod
class: liveness
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 7200, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: lt, params: [1] }
# -- 10 ----------------------------------------------------------------
# DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than
# critical because Caddy serves the static build and ProdEdgeDark /
# ProdHTTP5xxBurst will catch it first if users are actually affected.
- uid: tg_prod_frontend_silent
title: ProdFrontendContainerSilent
condition: C
for: 5m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: thermograph_frontend has logged nothing for 30 minutes'
description: >-
The frontend container has produced zero log lines in 30 minutes
(healthy floor ~66 per 30m). If users were visibly affected you would
expect ProdHTTP5xxBurst alongside this; on its own it more often
means a stuck redeploy or a lost Alloy tail.
labels:
severity: warning
host: prod
class: liveness
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 1800, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: lt, params: [1] }
# -- 11 ----------------------------------------------------------------
# DATA: a dead-steady 121 lines/hour (its own healthcheck polling) for the
# first 14h of the backtest, then nothing at all.
#
# HEADS UP: THIS RULE WILL BE FIRING THE MOMENT YOU DEPLOY IT, and it is
# right to. As of 2026-07-24 19:45Z the prod worker container is running
# and healthy and is writing ~19 lines per 10 minutes to its own docker
# log — but not one of those lines has reached Loki since 04:43Z. Prod's
# Alloy agent stopped tailing the container when Swarm replaced the task.
# Nobody knew, because nothing was watching. See README "Known gaps".
#
# WHY THIS IS A WARNING AND NotifierHeartbeatMissingProd IS CRITICAL: the
# two draw on *different* Alloy sources — this one on loki.source.docker
# (the container's stdout), the heartbeat on loki.source.file tailing
# /applogs. That independence is the diagnostic. Right now prod's
# heartbeats are arriving in every single 20-minute window while its
# worker docker stream has been empty for 14 hours, which localises the
# fault to Alloy's docker tail rather than to the worker: the process is
# doing its job, we just cannot see it breathe. Both rules firing together
# means the worker really is down.
- uid: tg_prod_worker_silent
title: ProdWorkerContainerSilent
condition: C
for: 10m
noDataState: OK
execErrState: OK
annotations:
summary: 'prod: thermograph_worker has logged nothing for 2 hours'
description: >-
The worker container has produced zero log lines in 2 hours against
a healthy floor of ~242 per 2h. Two different faults look identical
here: the worker is actually dead, or prod's Alloy agent has stopped
tailing it (a real, observed failure — Alloy can miss a container
after Swarm replaces the task). Check both: `docker service ps
thermograph_worker` on prod, and whether `docker logs` on the live
container is producing lines that Loki does not have. Restarting
alloy-alloy-1 on prod fixes the second case.
labels:
severity: warning
host: prod
class: liveness
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 7200, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_worker.+"} [2h])) or vector(0)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: lt, params: [1] }
# ===========================================================================
# The dead-man's switch.
# ===========================================================================
- orgId: 1
name: watchdog
folder: Alerts
interval: 5m
rules:
# -- 12 ----------------------------------------------------------------
# Nothing in this file can tell you that Grafana itself has stopped, or
# that beta is down, or that the Discord webhook was revoked — a monitoring
# system cannot report its own death. So this rule fires permanently and
# by design, throttled by the notification policy to exactly one Discord
# message per 24 hours.
#
# HOW TO USE IT: if #ops-alerts has been silent for more than a day, the
# alerting pipeline is broken, not the estate healthy. That one daily
# message is the only evidence you will ever get that the other eleven
# rules are still capable of reaching you.
#
# To silence it (and lose that guarantee), set isPaused: true.
- uid: tg_alerting_watchdog
title: AlertingWatchdog
condition: C
for: 0m
noDataState: Alerting
execErrState: Alerting
annotations:
summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)'
description: >-
This is the dead-man's switch and it is supposed to be firing. It
proves Grafana on beta is evaluating rules, Loki is answering, and
the Discord webhook still works. Seeing it once a day is correct.
NOT seeing it for over a day means alerting itself is down and no
other rule can reach you — check observability-grafana-1 on beta.
labels:
severity: watchdog
host: beta
class: meta
isPaused: false
data:
- refId: A
relativeTimeRange: { from: 600, to: 0 }
datasourceUid: loki
model:
refId: A
datasource: { type: loki, uid: loki }
expr: 'vector(1)'
queryType: instant
editorMode: code
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource: { type: __expr__, uid: __expr__ }
- refId: C
relativeTimeRange: { from: 0, to: 0 }
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
datasource: { type: __expr__, uid: __expr__ }
conditions:
- evaluator: { type: gt, params: [0] }
Reference in a new issue View git blame Copy permalink