thermograph/observability/README.md

Thermograph — observability

Fleet-wide log aggregation for Thermograph: a central Grafana + Loki stack, fed by a Grafana Alloy agent on every node, all over the private WireGuard mesh. This replaces the old in-repo scripts/dashboard.py (an SSH-tailed, single-host, metrics-endpoint view) with a persistent, queryable, fleet-wide log store and UI.

  prod (10.10.0.1)          beta (10.10.0.2)            desktop / LAN dev (10.10.0.3)
  ┌────────────┐            ┌──────────────────┐        ┌────────────┐
  │ Alloy agent│──wg0──┐    │ Loki  ◀── Alloy  │   ┌──wg0│ Alloy agent│
  └────────────┘       └───▶│ Grafana (Caddy)  │◀──┘    └────────────┘
   docker+caddy+app         └──────────────────┘         docker+caddy+app
     logs                    central store + UI            logs

• Loki stores logs (filesystem, 30-day retention). Listens on the mesh IP 10.10.0.2:3100 — never public.
• Grafana is the UI, fronted by beta's Caddy at dashboard.thermograph.org, login via Google SSO (with a break-glass local admin); Loki is not exposed.
• Alloy runs on each node and ships three sources to Loki: every Docker container's stdout/stderr, Caddy's host access logs, and the app's structured JSON logs (errors/access/audit *.jsonl, parsed so level/tag/phase become labels). Every line is tagged with its node (host = prod|beta|dev).

Layout

Path	What
docker-compose.yml	the central Loki + Grafana stack (runs on beta)
loki/config.yml	Loki config (filesystem, retention)
grafana/provisioning/	auto-wired Loki datasource + dashboard provider
grafana/provisioning/alerting/	contact point, notification policy and alert rules — see Alerting
grafana/dashboards/thermograph-logs.json	the fleet-logs dashboard
alloy/config.alloy	the per-node collector config
alloy/docker-compose.agent.yml	runs the Alloy agent on a node
caddy-grafana.conf	beta Caddy site block for the Grafana UI

Deploy

⚠️ Merging is not deploying

/opt/observability on beta (and prod) is a checkout of the archived emi/thermograph-observability repo. It can never git pull again — the content now lives here, in the mono repo, under observability/. A previous observability PR merged and had zero live effect for exactly this reason.

Until someone re-points those checkouts at the mono repo, every change here ships by hand: back up the live file with a UTC-timestamped suffix, scp the new one up, sudo cp it into place, restart the specific container. Use sudo docker restart <container> — not docker compose, which on prod demands a GF_SECURITY_ADMIN_PASSWORD it has no .env to interpolate from. (Beta's stack does have an .env, so compose works there — and is required when a change adds a new environment variable, since docker restart alone will not pick one up.)

1. Central stack (on beta)

# on beta (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000

Expose the Grafana UI: point dashboard.thermograph.org at beta, append caddy-grafana.conf to /etc/caddy/Caddyfile, systemctl reload caddy.

Google SSO (Grafana + Forgejo)

Both the dashboard and Forgejo log in with Google. Create one Google Cloud OAuth 2.0 Client (type: Web application) and register both redirect URIs:

• Grafana: https://dashboard.thermograph.org/login/google
• Forgejo: https://git.thermograph.org/user/oauth2/google/callback

Grafana: put the client id/secret in .env, set OAUTH_ENABLED=true, and docker compose up -d. Login is locked to pre-provisioned accounts (allow_sign_up=false), so provision your email as an admin once:

# on beta, after the stack is up (uses the break-glass admin creds from .env):
curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \
  -X POST http://127.0.0.1:3000/api/admin/users \
  -d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}'
# then make that user a Grafana admin (org role Admin + server admin) via the UI
# or /api/org/users; the Google login matches on email.

Forgejo: add the Google auth source (once, as a Forgejo admin):

docker exec -u git <forgejo-container> forgejo admin auth add-oauth \
  --name google --provider openidConnect \
  --key "<CLIENT_ID>" --secret "<CLIENT_SECRET>" \
  --auto-discover-url https://accounts.google.com/.well-known/openid-configuration

2. Alloy agent (on every node — prod, beta, desktop)

cd thermograph-observability/alloy
# Every node (beta included) pushes to Loki over the mesh IP — a containerized
# agent can't reach the host's 127.0.0.1, but it can reach beta's wg0 address:
#   beta:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=beta
#   prod:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=prod
#   desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=dev
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d

On the LAN dev node the app's log volume is thermograph-dev_applogs (not thermograph_applogs) — edit the two references in docker-compose.agent.yml there. The mesh must already be up (see the main repo's deploy/swarm/ / INFRA.md); Swarm is not required — the agents talk plain HTTP over wg0.

Verify

# From any mesh node, confirm Loki is receiving from every host:
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.

The dashboard

Thermograph — Fleet Logs (auto-provisioned), with a Node selector: log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container tail. Edit it in Grafana and re-export the JSON here to version a change.

Alerting

Provisioned from grafana/provisioning/alerting/, mounted read-only into Grafana by the compose file, so the repo is the only durable path — as with the dashboards, edits made in the Grafana UI are overwritten on the next provision.

Every rule is LogQL, because there is no Prometheus in the fleet: Loki is the only datasource, so every signal is derived from log lines.

Where alerts go: Discord, not email

#ops-alerts (Owners category, private to Emi and Jin) via Grafana's native Discord contact point. Deliberately not email:

• Grafana's factory default pointed at the literal string <example@email.com> — a paging path that had never delivered a message to anyone.
• beta's Grafana relays SMTP through prod's Postfix at 10.10.0.1:25 (see docker-compose.override.yml on beta). Email alerts therefore travel through the box most likely to be on fire, and vanish exactly when they matter.
• Discord is off-estate, works when prod is dead, and reaches a phone.

#ops-alerts is a new dedicated channel, not one of the existing ones: the #dev / #uat / #prod channels are the product's alert-subscription output (real deliveries — routing ops noise there would corrupt the evidence), and #notes / #staff-chat are for humans.

The webhook URL is a secret and is not in this repo: it comes from DISCORD_ALERT_WEBHOOK_URL in beta's gitignored .env, which compose passes to Grafana and which Grafana interpolates when reading the provisioning file. Grafana refuses to start if it is unset.

The rules

Rule	Fires when	Severity
ProdEdgeDark	zero prod Caddy log lines in 15m — site dark, or the log pipeline is broken	critical
ProdHealthzFailing	any non-200 on /healthz in 10m	critical
ProdHTTP5xxBurst	≥10 prod 5xx in 10m	critical
ProdHTTP5xxElevated	≥25 prod 5xx accumulated over 6h (the error-budget alarm)	warning
ProdUnhandledExceptions	≥3 app tag=http.error in 30m	warning
NotifierHeartbeatMissingProd	no tag=heartbeat line from prod in 20m	critical
NotifierHeartbeatMissingBeta	same, on beta	warning
ProdWebContainerSilent	thermograph_web logged nothing in 30m	critical
ProdDbContainerSilent	thermograph_db logged nothing in 2h	critical
ProdFrontendContainerSilent	thermograph_frontend logged nothing in 30m	warning
ProdWorkerContainerSilent	thermograph_worker logged nothing in 2h	warning
AlertingWatchdog	always — one Discord message per day	watchdog

Every threshold was backtested against real Loki data; the working is written out in the comments of rules-prod.yml, next to the rule it justifies. Read those before changing a number.

AlertingWatchdog is the dead-man's switch and fires permanently by design. Its value is its absence: nothing can report its own death, so if #ops-alerts has been silent for more than a day, alerting is broken rather than the estate healthy. Set isPaused: true on it to opt out of that guarantee.

Deploying an alerting change to beta

Grafana already mounts ./grafana/provisioning read-only, so new files under alerting/ need no compose change — but the first deploy does, because it introduces DISCORD_ALERT_WEBHOOK_URL, and docker restart will not pick up a new environment variable.

# run from the mono-repo root
BETA="agent@75.119.132.91"
SSH="ssh -i ~/.ssh/thermograph_agent_ed25519"
TS=$(date -u +%Y%m%dT%H%M%SZ)

# 1. put the secret in beta's .env (once), then confirm it took:
$SSH $BETA "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING"

# 2. back up the live compose file, stage the new files, move them into place:
$SSH $BETA "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \
            && rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting"
scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $BETA:/tmp/obs-stage/
scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $BETA:/tmp/obs-stage/alerting/
$SSH $BETA "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \
            && sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \
            && sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/"

# 3. recreate Grafana so it reads the new env var AND the new provisioning.
#    `docker restart` is NOT enough here — it will not pick up a new env var.
$SSH $BETA "cd /opt/observability && docker compose up -d grafana"

Subsequent rule-only changes are just step 2 plus sudo docker restart observability-grafana-1 — no compose needed, because no new environment variable is involved.

Rollback: the provisioning files are additive. To back the whole thing out, sudo rm -rf /opt/observability/grafana/provisioning/alerting, restore docker-compose.yml.<timestamp>, and docker compose up -d grafana. Grafana falls back to its default (email → <example@email.com>) root policy — i.e. no alerting, which is where this started.

Alerting lives on beta only — prod and dev run Alloy agents, not Grafana, so nothing in alerting/ ever ships to them.

Verify alerting

# on beta, after deploying and restarting Grafana:
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-)
# 1. all 12 rules loaded and healthy (health should be "ok", never "error"):
curl -s -u admin:"$GFP" localhost:3000/api/prometheus/grafana/api/v1/rules \
  | python3 -c 'import json,sys; [print(r["name"], r["state"], r["health"]) for g in json.load(sys.stdin)["data"]["groups"] for r in g["rules"]]'
# 2. the root policy points at Discord, NOT grafana-default-email:
curl -s -u admin:"$GFP" localhost:3000/api/v1/provisioning/policies
# 3. force a real message into #ops-alerts (does not persist anything):
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' \
  -X POST localhost:3000/api/alertmanager/grafana/config/api/v1/receivers/test \
  -d '{"receivers":[{"name":"t","grafana_managed_receiver_configs":[{"name":"t","type":"discord","settings":{"url":"'"$WH"'"}}]}]}'
#    -> {"receivers":[{... "status":"ok"}]} and a message in #ops-alerts.

The webhook URL is redacted ([REDACTED]) by GET /api/v1/provisioning/contact-points, so you cannot confirm interpolation by reading it back — the only proof is a message actually arriving in #ops-alerts. A contact point holding the literal string $DISCORD_ALERT_WEBHOOK_URL looks completely healthy from the API and pages nobody.

The leftover default contact point

Grafana's built-in grafana-default-email → <example@email.com> receiver is created by Grafana's default Alertmanager config, not by provisioning, and its uid is the empty string. deleteContactPoints matches on uid, so provisioning cannot remove it (DELETE /api/v1/provisioning/contact-points/ with an empty uid returns 404 — verified).

Provisioning the notification policy already makes it inert: nothing routes there any more, so it cannot receive anything. Actually deleting the row is a one-time manual step, and only works after this config is live (Grafana refuses to delete a receiver a policy still references). Either use the UI — Alerting → Contact points → grafana-default-email → Delete — or:

# on beta. Reads the running Alertmanager config, drops the one receiver,
# writes it back. Verified against Grafana 11.6.1 (returns 202).
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts
curl -s -u admin:"$GFP" $AM > /tmp/am.json
python3 -c "
import json; d=json.load(open('/tmp/am.json')); c=d['alertmanager_config']
c['receivers']=[r for r in c['receivers'] if r['name']!='grafana-default-email']
json.dump(d, open('/tmp/am2.json','w'))"
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' -X POST --data @/tmp/am2.json $AM
rm -f /tmp/am.json /tmp/am2.json

This is cosmetic — leaving it costs nothing but a confusing row in the UI.

Notes / follow-ups

• Loki is mesh-only by design; only Grafana (with auth) is public.
• Alloy reads the Docker socket read-only to discover containers.
• Retention is 30 days on beta's disk — bump loki/config.yml if you want more.
• The app emits a tag=heartbeat line for the subscription notifier every ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it. Extend the same audit.log_heartbeat() to any future background daemon.

Known gaps

• Alloy can silently stop tailing a container after Swarm replaces the task. Observed on prod on 2026-07-24: thermograph_worker was 1/1, healthy, and writing ~19 lines per 10 minutes to its own docker log, yet not one line had reached Loki since 04:43Z — a 13½-hour hole nobody could see, because nothing was watching. ProdWorkerContainerSilent now catches exactly this. What localises it to Alloy rather than the app: prod's tag=heartbeat lines kept arriving in every 20-minute window throughout, and those come from a different Alloy source (loki.source.file on /applogs) than the container stdout (loki.source.docker). One source went deaf, the other did not. The workaround is sudo docker restart alloy-alloy-1 on the affected node; the fix is probably an explicit refresh_interval on discovery.docker plus a look at whether loki.source.docker re-resolves targets. Not root-caused.
• Some services are too quiet to alert on. thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and thermograph_autoscaler-lake (2) have a healthy floor indistinguishable from dead, and thermograph_lake is bursty ETL with legitimate idle gaps. Giving each one an audit.log_heartbeat() line would make all four alertable with the same pattern the notifier already uses. That is the single highest-value follow-up here.
• Grafana cannot alert on its own death — that is what AlertingWatchdog is for, but it is a manual dead-man's switch: it relies on somebody noticing that the daily message stopped. A real external watchdog (an uptime pinger hitting a Grafana endpoint from off-estate) would close this properly.
• beta runs a docker-compose.override.yml that is not in this repo, wiring Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked compose file or deleted; alerting no longer depends on it.