1a56a63800
All checks were successful
shell-lint / shellcheck (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 5s
Validate observability stack / validate (push) Successful in 12s
secrets-guard / encrypted (pull_request) Successful in 7s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / changes (pull_request) Successful in 8s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
896 lines
38 KiB
YAML
896 lines
38 KiB
YAML
# Thermograph alert rules — the estate's first. All Loki/LogQL, because there is
|
|
# no Prometheus anywhere in the fleet: Loki is the only datasource, so every
|
|
# signal here is derived from log lines rather than metrics.
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# HOW THE THRESHOLDS WERE CHOSEN
|
|
# ---------------------------------------------------------------------------
|
|
# Every number below was backtested against the real 24h ending 2026-07-24
|
|
# ~19:45Z. That window contains a genuine ~20-minute prod outage starting
|
|
# 2026-07-24 04:20Z, which is the event these rules must catch:
|
|
#
|
|
# 10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown):
|
|
# 2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1 (total 86)
|
|
# ^^^^^^ the outage: 64% and 56% of all requests 5xx
|
|
# All five /healthz failures in the whole day fell inside those two buckets.
|
|
# Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic —
|
|
# which is why absolute counts beat error *ratios* here: at 7 req/min a single
|
|
# bad request is 1.4% and a ratio alert would scream all night.
|
|
#
|
|
# The design goal was: fire on the outage, stay silent for the other 142 of 144
|
|
# buckets, and still surface the slow drip of ~20 scattered errors that made up
|
|
# the rest of the day's 86 (that is what ProdHTTP5xxElevated is for).
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# TWO CONVENTIONS WORTH KNOWING
|
|
# ---------------------------------------------------------------------------
|
|
# 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a
|
|
# naive "count < 1" silence alert goes to NoData instead of firing, which is
|
|
# the exact failure mode of every homemade log alert ever written. Appending
|
|
# `or vector(0)` makes an empty result evaluate to a real 0. Verified working
|
|
# on Loki 3.5.1.
|
|
# 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service`
|
|
# label — only `container`, which includes a per-task suffix that changes on
|
|
# every redeploy (thermograph_web.1.<taskid>). So prod services are matched as
|
|
# container=~"thermograph_<name>.+" with job="docker". Do not use `service=`
|
|
# here; it only works on the compose hosts (beta, dev).
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# NOT ALERTABLE, ON PURPOSE
|
|
# ---------------------------------------------------------------------------
|
|
# thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and
|
|
# thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their
|
|
# healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL
|
|
# with long legitimate idle gaps. Catching those needs either a heartbeat line
|
|
# (extend backend/core/audit.py log_heartbeat(), as the notifier already does)
|
|
# or container metrics, which the fleet does not collect. Tracked as a gap.
|
|
#
|
|
# Error budget / failure isolation: only ProdEdgeDark escalates on NoData or
|
|
# datasource errors, so a Loki or WireGuard blip produces ONE page rather than
|
|
# twelve. Every other rule degrades to OK if the query cannot run.
|
|
apiVersion: 1
|
|
|
|
groups:
|
|
# ===========================================================================
|
|
# Is the product up? Evaluated every minute; all of these page.
|
|
# ===========================================================================
|
|
- orgId: 1
|
|
name: prod-availability
|
|
folder: Alerts
|
|
interval: 1m
|
|
rules:
|
|
|
|
# -- 1 -----------------------------------------------------------------
|
|
# The top-level "is anything alive" check, and deliberately the only rule
|
|
# that also owns NoData/error: Caddy fronts every request to
|
|
# thermograph.org, so zero access-log lines means the site is dark, the
|
|
# box is gone, Alloy died, or the mesh dropped. Any of those is a page.
|
|
#
|
|
# DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35,
|
|
# and never once reached zero — including *during* the outage (72), when
|
|
# Caddy was happily logging 503s. Zero is unambiguous.
|
|
- uid: tg_prod_edge_dark
|
|
title: ProdEdgeDark
|
|
condition: C
|
|
for: 5m
|
|
noDataState: Alerting
|
|
execErrState: Alerting
|
|
annotations:
|
|
summary: 'prod: no Caddy access-log lines for 15 minutes'
|
|
description: >-
|
|
Nothing has reached thermograph.org's reverse proxy in 15 minutes
|
|
(healthy floor is 35 lines/15m, and it stayed at 72 even during the
|
|
04:20Z outage). Either prod is down, Caddy is down, prod's Alloy
|
|
agent stopped shipping, or the WireGuard mesh to beta is broken.
|
|
This rule also fires on Loki NoData/errors, so it doubles as the
|
|
"we have lost observability" alarm — if it is the only thing firing,
|
|
suspect the pipeline before suspecting the app.
|
|
labels:
|
|
severity: critical
|
|
host: prod
|
|
class: availability
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 900, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
|
|
# -- 2 -----------------------------------------------------------------
|
|
# /healthz is the app's own statement that it is well. When it 503s, it is
|
|
# not an opinion.
|
|
#
|
|
# DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour,
|
|
# so this endpoint is polled sparsely and irregularly. That is why `for`
|
|
# is 0m: waiting for a second consecutive bad sample could take 25
|
|
# minutes. All 5 failures landed inside the two outage buckets and nowhere
|
|
# else in the day, so one failure is already a real signal, not noise.
|
|
- uid: tg_prod_healthz_failing
|
|
title: ProdHealthzFailing
|
|
condition: C
|
|
for: 0m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: /healthz returned a non-200'
|
|
description: >-
|
|
The app's own liveness endpoint failed at the edge in the last 10
|
|
minutes. Over the backtested 24h this happened 5 times, all of them
|
|
inside the 04:20Z outage — /healthz failing has a 100% correlation
|
|
with prod actually being broken. Check `docker service ps
|
|
thermograph_web` on prod first.
|
|
labels:
|
|
severity: critical
|
|
host: prod
|
|
class: availability
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 600, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: gt, params: [0] }
|
|
|
|
# -- 3 -----------------------------------------------------------------
|
|
# The acute "users are seeing errors right now" page.
|
|
#
|
|
# DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and
|
|
# the largest *benign* bucket had 5. The outage produced 45 and 22.
|
|
# Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst
|
|
# benign bucket and 4.5x below the incident peak — it fires on the outage
|
|
# and on nothing else in the entire day. It catches 67 of that day's 86
|
|
# errors as a single incident; the remaining scattered ~20 are picked up
|
|
# by ProdHTTP5xxElevated below rather than by pushing this threshold down
|
|
# into the noise.
|
|
- uid: tg_prod_5xx_burst
|
|
title: ProdHTTP5xxBurst
|
|
condition: C
|
|
for: 2m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: 10+ HTTP 5xx in the last 10 minutes'
|
|
description: >-
|
|
Prod is serving server errors at an outage-grade rate. Baseline is
|
|
zero in 90% of 10-minute windows and never above 5; the reference
|
|
incident hit 45. Look at the "Thermograph — Traffic & Domains"
|
|
dashboard, panel "HTTP errors by status", and at
|
|
{job="app-json", host="prod", filename=~".+/errors/.+"} for the
|
|
exceptions behind them.
|
|
labels:
|
|
severity: critical
|
|
host: prod
|
|
class: errors
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 600, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: gt, params: [9] }
|
|
|
|
# ===========================================================================
|
|
# Slow-burn error signals. Five-minute evaluation is plenty; these are tickets,
|
|
# not pages.
|
|
# ===========================================================================
|
|
- orgId: 1
|
|
name: prod-errors
|
|
folder: Alerts
|
|
interval: 5m
|
|
rules:
|
|
|
|
# -- 4 -----------------------------------------------------------------
|
|
# The rule that would have broken the silence on "84 5xx in 24h and nobody
|
|
# was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation.
|
|
#
|
|
# DATA: 6-hour rolling windows sampled hourly across the backtest read
|
|
# 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11.
|
|
# The largest window containing NO incident was 11. Threshold 25
|
|
# (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any
|
|
# day that looks like this one. Deliberately a long window: it will stay
|
|
# firing for ~6h after an incident, which is the point. That is the daily
|
|
# error budget talking, not a flap.
|
|
- uid: tg_prod_5xx_elevated
|
|
title: ProdHTTP5xxElevated
|
|
condition: C
|
|
for: 10m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours'
|
|
description: >-
|
|
Sustained server errors. This is the error-budget alarm, not an
|
|
outage alarm — it catches the scattered drip that never trips the
|
|
burst rule but still added up to 86 errors (1.7% of requests) on
|
|
2026-07-24. Expect it to stay firing for up to 6h after an incident
|
|
clears; that is the window emptying, not a new fault.
|
|
labels:
|
|
severity: warning
|
|
host: prod
|
|
class: errors
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 21600, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: gt, params: [24] }
|
|
|
|
# -- 5 -----------------------------------------------------------------
|
|
# A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees
|
|
# 502/503 (the app is unreachable), while tag="http.error" is the app
|
|
# catching an unhandled exception and still answering. A bad deploy shows
|
|
# up here first, and can do so without moving the Caddy numbers at all.
|
|
#
|
|
# DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour.
|
|
# Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak:
|
|
# silent today, loud on an exception storm.
|
|
- uid: tg_prod_unhandled_exceptions
|
|
title: ProdUnhandledExceptions
|
|
condition: C
|
|
for: 0m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: 3+ unhandled exceptions in 30 minutes'
|
|
description: >-
|
|
The app is throwing. Normal is under 2 per DAY. A cluster this tight
|
|
almost always means a regression just shipped — check the app.start
|
|
lines on the "App Activity & Delivery" dashboard for a recent deploy,
|
|
then {job="app-json", host="prod", filename=~".+/errors/.+"} |
|
|
tag="http.error" for the traceback and route.
|
|
labels:
|
|
severity: warning
|
|
host: prod
|
|
class: errors
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 1800, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: gt, params: [2] }
|
|
|
|
# ===========================================================================
|
|
# The subscription notifier — the thing that actually delivers the product.
|
|
# ===========================================================================
|
|
- orgId: 1
|
|
name: notifier
|
|
folder: Alerts
|
|
interval: 5m
|
|
rules:
|
|
|
|
# -- 6 -----------------------------------------------------------------
|
|
# backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it
|
|
# stops, subscribers stop getting weather alerts and absolutely nothing
|
|
# else in this file would notice — the site stays up and serves 200s while
|
|
# the product quietly stops working.
|
|
#
|
|
# DATA: prod produced 1-2 beats in every single 20-minute bucket across
|
|
# the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval
|
|
# means two consecutive empty evaluations, so roughly 25 minutes of real
|
|
# silence before it pages — about one and a half missed beats.
|
|
#
|
|
# dev is deliberately excluded: the LAN box had zero heartbeats for the
|
|
# first 10.5h of the backtest window because it was simply off. Alerting
|
|
# on dev would mean an alert that is firing more often than not.
|
|
#
|
|
# MUST stay filtered to daemon="subscription-notifier". The selector was
|
|
# originally bare `tag="heartbeat"`, which was unambiguous while the
|
|
# notifier thread was the only thing that ever beat. It stopped being
|
|
# unambiguous on 2026-07-25T17:24Z, when web/app.py's _start_heartbeat
|
|
# shipped: every uvicorn process in every replica now beats every 300s
|
|
# under the same tag, tagged daemon=web / daemon=worker. Measured
|
|
# immediately after that deploy, a 20m window held 16 web beats and 4
|
|
# worker beats against 2 notifier beats — so the bare selector clears the
|
|
# `< 1` threshold on web traffic alone and can no longer observe the
|
|
# notifier at all.
|
|
#
|
|
# That matters because this rule had just caught a real 17.5h outage
|
|
# (notifier silent 2026-07-24T23:49Z -> 2026-07-25T17:25Z, zero
|
|
# subscription notifications delivered). Left bare, it would have gone
|
|
# quiet on the next occurrence instead of paging. `daemon` is a JSON field
|
|
# and not a stream label, hence the `| json |` parse rather than a label
|
|
# matcher.
|
|
- uid: tg_notifier_heartbeat_prod
|
|
title: NotifierHeartbeatMissingProd
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: no notifier heartbeat for 20 minutes'
|
|
description: >-
|
|
The subscription notifier has gone quiet on prod. Expected cadence is
|
|
~15 minutes and it did not miss a single 20-minute window over the
|
|
backtested day. Subscribers are not being notified right now, even
|
|
though the website is probably still serving fine. Check the
|
|
thermograph_worker service and the "Notifier alive?" panel on the
|
|
Fleet Logs dashboard.
|
|
labels:
|
|
severity: critical
|
|
host: prod
|
|
class: product
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 1200, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
|
|
# -- 7 -----------------------------------------------------------------
|
|
# Same signal on beta. Beta was equally gap-free over the backtest (1-2
|
|
# beats every 20m, no empty buckets), but beta is UAT — worth knowing,
|
|
# not worth a page.
|
|
- uid: tg_notifier_heartbeat_beta
|
|
title: NotifierHeartbeatMissingBeta
|
|
condition: C
|
|
for: 15m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'beta: no notifier heartbeat for 20 minutes'
|
|
description: >-
|
|
The notifier has gone quiet on beta (UAT). Same signal as the prod
|
|
rule but a longer `for`, because beta gets redeployed all day and a
|
|
short gap is usually just a release.
|
|
labels:
|
|
severity: warning
|
|
host: beta
|
|
class: product
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 1200, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} | json | daemon="subscription-notifier" [20m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
|
|
# ===========================================================================
|
|
# Per-service liveness. The fleet collects no container metrics, so liveness
|
|
# has to be inferred from "is this thing still producing output at all". Two
|
|
# different sources are used here, and picking the wrong one is how this group
|
|
# generated a day of false pages:
|
|
#
|
|
# * db and frontend infer it from CONTAINER LOG VOLUME. Valid for these two:
|
|
# both log to stdout on a floor that never approaches zero (db a metronomic
|
|
# 54-69 lines/2h, frontend ~66/30m), and prod's Alloy ships both.
|
|
#
|
|
# * web and worker infer it from the APP'S OWN HEARTBEAT (tag=heartbeat in
|
|
# the app-json file source, filtered by the daemon field). Container log
|
|
# volume is NOT a valid liveness signal for either: web's request logging
|
|
# moved off stdout onto app-json, and worker's stdout is dropped outright
|
|
# by a rule in prod's alloy config. Both containers can be perfectly
|
|
# healthy while their docker streams sit at exactly zero forever.
|
|
#
|
|
# Before adding a container-log-silence rule for a new service, confirm the
|
|
# service actually logs to stdout AND that prod's discovery.relabel "containers"
|
|
# block does not drop it. A rule against a dropped stream is unsatisfiable and
|
|
# will page forever.
|
|
#
|
|
# Caveat that still applies to the db/frontend rules: a container can be
|
|
# running and logging while its lines never reach Loki, if the node's Alloy
|
|
# agent stops tailing it. Those rules cannot tell the two apart — they say
|
|
# "no logs", which is the honest claim. Either way something needs looking at.
|
|
# ===========================================================================
|
|
- orgId: 1
|
|
name: prod-services
|
|
folder: Alerts
|
|
interval: 5m
|
|
rules:
|
|
|
|
# -- 8 -----------------------------------------------------------------
|
|
# Web-tier liveness, from the app's own heartbeat rather than container
|
|
# stdout.
|
|
#
|
|
# This rule used to count `{job="docker", container=~"thermograph_web.+"}`
|
|
# over 30m against a measured floor of ~110 lines. That floor was real when
|
|
# it was written (the backtest saw 3.8k-10.5k lines per 6h), but the app
|
|
# moved its request logging off stdout onto the app-json file source, and
|
|
# from 2026-07-25T00:02Z a web container emits ~15 uvicorn banner lines for
|
|
# its entire lifetime and nothing after. The old rule then sat firing
|
|
# permanently on a perfectly healthy tier, clearing only for the few
|
|
# minutes after each deploy or autoscale event produced fresh banners —
|
|
# 20+ pages in a day, every one of them false. See web/app.py's
|
|
# _start_heartbeat docstring, which states outright that stdout silence
|
|
# cannot distinguish a live web process from a dead one and that this
|
|
# heartbeat is the signal to alert on instead.
|
|
#
|
|
# DATA: _start_heartbeat beats every 300s from every uvicorn process in
|
|
# every replica, unguarded by the leader election, so a 4-process replica
|
|
# yields ~16 beats per 20m (measured: exactly 16). Zero over 20m means no
|
|
# web process is alive anywhere. This is the app itself, so it pages.
|
|
- uid: tg_prod_web_silent
|
|
title: ProdWebHeartbeatMissing
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: no web heartbeat for 20 minutes'
|
|
description: >-
|
|
No thermograph_web process on prod has emitted a liveness heartbeat
|
|
in 20 minutes. Every uvicorn process beats every 5 minutes
|
|
independently of the notifier leader election, so a healthy 4-process
|
|
replica produces ~16 beats per window — zero means no web process is
|
|
alive anywhere, not merely quiet. Run `docker service ps
|
|
thermograph_web` on prod. Note this is the app's own heartbeat, not
|
|
container stdout: web's stdout is silent by design and is not a
|
|
liveness signal.
|
|
labels:
|
|
severity: critical
|
|
host: prod
|
|
class: liveness
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 1200, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="web" [20m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
|
|
# -- 9 -----------------------------------------------------------------
|
|
# DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69
|
|
# lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a
|
|
# metronomic ~30 lines/hour. Zero over two hours is as close to a
|
|
# definitive "Postgres is not there" as logs can give.
|
|
- uid: tg_prod_db_silent
|
|
title: ProdDbContainerSilent
|
|
condition: C
|
|
for: 10m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: thermograph_db has logged nothing for 2 hours'
|
|
description: >-
|
|
The production TimescaleDB container has produced zero log lines in
|
|
2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no
|
|
observed gaps, so zero is not quiet — it is absent. Everything else
|
|
in the app depends on this.
|
|
labels:
|
|
severity: critical
|
|
host: prod
|
|
class: liveness
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 7200, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
|
|
# -- 10 ----------------------------------------------------------------
|
|
# DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than
|
|
# critical because Caddy serves the static build and ProdEdgeDark /
|
|
# ProdHTTP5xxBurst will catch it first if users are actually affected.
|
|
- uid: tg_prod_frontend_silent
|
|
title: ProdFrontendContainerSilent
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: thermograph_frontend has logged nothing for 30 minutes'
|
|
description: >-
|
|
The frontend container has produced zero log lines in 30 minutes
|
|
(healthy floor ~66 per 30m). If users were visibly affected you would
|
|
expect ProdHTTP5xxBurst alongside this; on its own it more often
|
|
means a stuck redeploy or a lost Alloy tail.
|
|
labels:
|
|
severity: warning
|
|
host: prod
|
|
class: liveness
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 1800, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
|
|
# -- 11 ----------------------------------------------------------------
|
|
# Worker liveness, from the app's own heartbeat rather than container
|
|
# stdout.
|
|
#
|
|
# The previous rule counted `{job="docker", container=~"thermograph_worker.+"}`
|
|
# and was UNSATISFIABLE BY CONSTRUCTION — it could never return anything but
|
|
# zero, no matter how healthy the worker was. prod's Alloy config drops that
|
|
# container on purpose, in discovery.relabel "containers":
|
|
#
|
|
# regex = "(alloy|autoscaler|thermograph-test_.*|thermograph-lb|thermograph_worker).*"
|
|
# action = "drop"
|
|
#
|
|
# with the stated reasoning that worker stdout was 100% /healthz poll noise
|
|
# and the app's own access/*.jsonl is a strict superset of it. So the stream
|
|
# this rule queried is discarded at the agent and never reaches Loki.
|
|
#
|
|
# The old comment here read the resulting permanent zero as "Alloy stopped
|
|
# tailing the container after Swarm replaced the task" and pointed the
|
|
# runbook at restarting alloy-alloy-1. That diagnosis was wrong, and the
|
|
# fix it recommended could not have worked: a restarted Alloy re-reads the
|
|
# same config and drops the same container again. The rule fired
|
|
# continuously from 2026-07-24T21:05Z with no underlying fault.
|
|
#
|
|
# DATA: _start_heartbeat beats every 300s from the worker process
|
|
# (measured: exactly 4 per 20m). Unlike the notifier below it is not gated
|
|
# on the leader election, so it reports the process being alive even when
|
|
# this worker is not the elected notifier — which is the distinction the
|
|
# old rule was trying and failing to draw.
|
|
- uid: tg_prod_worker_silent
|
|
title: ProdWorkerHeartbeatMissing
|
|
condition: C
|
|
for: 10m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
annotations:
|
|
summary: 'prod: no worker heartbeat for 20 minutes'
|
|
description: >-
|
|
The thermograph_worker process on prod has not emitted a liveness
|
|
heartbeat in 20 minutes (healthy is ~4 per window, one every 5
|
|
minutes). This beat is independent of the notifier leader election,
|
|
so it means the worker process itself is gone or wedged, not merely
|
|
that it lost the election. Run `docker service ps thermograph_worker`
|
|
on prod. Do not look at the worker's container logs — prod's Alloy
|
|
config drops that stream deliberately, so it is always empty and is
|
|
not evidence of anything.
|
|
labels:
|
|
severity: warning
|
|
host: prod
|
|
class: liveness
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 1200, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'sum(count_over_time({job="app-json", host="prod", tag="heartbeat"} | json | daemon="worker" [20m])) or vector(0)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
|
|
# ===========================================================================
|
|
# The dead-man's switch.
|
|
# ===========================================================================
|
|
- orgId: 1
|
|
name: watchdog
|
|
folder: Alerts
|
|
interval: 5m
|
|
rules:
|
|
|
|
# -- 12 ----------------------------------------------------------------
|
|
# Nothing in this file can tell you that Grafana itself has stopped, or
|
|
# that beta is down, or that the Discord webhook was revoked — a monitoring
|
|
# system cannot report its own death. So this rule fires permanently and
|
|
# by design, throttled by the notification policy to exactly one Discord
|
|
# message per 24 hours.
|
|
#
|
|
# HOW TO USE IT: if #ops-alerts has been silent for more than a day, the
|
|
# alerting pipeline is broken, not the estate healthy. That one daily
|
|
# message is the only evidence you will ever get that the other eleven
|
|
# rules are still capable of reaching you.
|
|
#
|
|
# To silence it (and lose that guarantee), set isPaused: true.
|
|
- uid: tg_alerting_watchdog
|
|
title: AlertingWatchdog
|
|
condition: C
|
|
for: 0m
|
|
noDataState: Alerting
|
|
execErrState: Alerting
|
|
annotations:
|
|
summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)'
|
|
description: >-
|
|
This is the dead-man's switch and it is supposed to be firing. It
|
|
proves Grafana on beta is evaluating rules, Loki is answering, and
|
|
the Discord webhook still works. Seeing it once a day is correct.
|
|
NOT seeing it for over a day means alerting itself is down and no
|
|
other rule can reach you — check observability-grafana-1 on beta.
|
|
labels:
|
|
severity: watchdog
|
|
host: beta
|
|
class: meta
|
|
isPaused: false
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 600, to: 0 }
|
|
datasourceUid: loki
|
|
model:
|
|
refId: A
|
|
datasource: { type: loki, uid: loki }
|
|
expr: 'vector(1)'
|
|
queryType: instant
|
|
editorMode: code
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
- refId: B
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: B
|
|
type: reduce
|
|
reducer: last
|
|
expression: A
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
- refId: C
|
|
relativeTimeRange: { from: 0, to: 0 }
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: B
|
|
datasource: { type: __expr__, uid: __expr__ }
|
|
conditions:
|
|
- evaluator: { type: gt, params: [0] }
|