thermograph/observability/README.md

Thermograph — observability

Fleet-wide log aggregation for Thermograph: a central Grafana + Loki stack, fed by a Grafana Alloy agent on every node, all over the private WireGuard mesh. This replaces the old in-repo scripts/dashboard.py (an SSH-tailed, single-host, metrics-endpoint view) with a persistent, queryable, fleet-wide log store and UI.

  vps1 (10.10.0.2)                  vps2 (10.10.0.1)                desktop (10.10.0.3)
  ┌───────────────────────┐         ┌───────────────────────┐       ┌──────────────┐
  │ Loki  ◀───── Alloy    │◀──wg0───┤ Alloy agent           │◀─wg0──┤ Alloy agent  │
  │ Grafana (Caddy)       │         │ prod + beta (Swarm)   │       │ (no app      │
  │ dev (compose)         │◀──wg0───┤ Postfix, backups      │       │  stack)      │
  │ Forgejo, portfolio    │         └───────────────────────┘       └──────────────┘
  └───────────────────────┘          docker+caddy+app logs           ships no app logs
   central store + UI + dev

• vps1 is the monitoring host: Loki, Grafana, Forgejo, the portfolio site, and the dev environment (its own compose project, mesh-only, no public DNS). It does not run the beta environment — that now lives on vps2 — so any older doc that says "beta" meaning this box is stale and needs reading as vps1.
• vps2 runs prod and beta as two separate Docker Swarm stacks (thermograph, thermograph-beta), plus Postfix and backups. One shared TimescaleDB serves both, on separate roles/databases.
• desktop hosts no Thermograph environment now (AI-model hosting + flex Swarm-worker capacity instead), so it ships no app logs.
• Loki stores logs (filesystem, 30-day retention). Listens on the mesh IP 10.10.0.2:3100 — never public.
• Grafana is the UI, fronted by vps1's Caddy at dashboard.thermograph.org, login via Google SSO (with a break-glass local admin); Loki is not exposed.
• Alloy runs on each node and ships three sources to Loki: every Docker container's stdout/stderr, Caddy's host access logs, and the app's structured JSON logs (errors/access/audit *.jsonl, parsed so level/tag/phase become labels). Every line now carries two distinct labels: host (prod|beta|dev) is the ENVIRONMENT, derived per source (a container's stack prefix, the Caddy access-log filename, or which applogs volume a JSON line came out of) — because vps2 alone carries two environments, and one node-wide value would have stamped every beta container and log line as prod. node (vps1|vps2|desktop) is the MACHINE, stamped externally on everything an agent ships. See alloy/config.alloy for exactly how host is derived per source.

Layout

Path	What
docker-compose.yml	the central Loki + Grafana stack (runs on vps1)
loki/config.yml	Loki config (filesystem, retention)
grafana/provisioning/	auto-wired Loki datasource + dashboard provider
grafana/provisioning/alerting/	contact point, notification policy and alert rules — see Alerting
grafana/dashboards/thermograph-logs.json	the fleet-logs dashboard
alloy/config.alloy	the per-node collector config
alloy/docker-compose.agent.yml	runs the Alloy agent on a node
alloy/docker-compose.agent.beta.yml	overlay adding beta's applogs mount — vps2 only
caddy-grafana.conf	vps1 Caddy site block for the Grafana UI

Deploy

⚠️ Merging is not deploying

/opt/observability on vps1 (and vps2) is a checkout of the archived emi/thermograph-observability repo. It can never git pull again — the content now lives here, in the mono repo, under observability/. A previous observability PR merged and had zero live effect for exactly this reason.

Until someone re-points those checkouts at the mono repo, every change here ships by hand: back up the live file with a UTC-timestamped suffix, scp the new one up, sudo cp it into place, restart the specific container. Use sudo docker restart <container> — not docker compose, which on vps2 demands a GF_SECURITY_ADMIN_PASSWORD it has no .env to interpolate from (vps2 only runs the Alloy agent — there's no Grafana there to configure). (vps1's stack does have an .env, so compose works there — and is required when a change adds a new environment variable, since docker restart alone will not pick one up.)

1. Central stack (on vps1)

# on vps1 (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000

Expose the Grafana UI: point dashboard.thermograph.org at vps1, append caddy-grafana.conf to /etc/caddy/Caddyfile, systemctl reload caddy.

Google SSO (Grafana + Forgejo)

Both the dashboard and Forgejo log in with Google. Create one Google Cloud OAuth 2.0 Client (type: Web application) and register both redirect URIs:

• Grafana: https://dashboard.thermograph.org/login/google
• Forgejo: https://git.thermograph.org/user/oauth2/google/callback

Grafana: put the client id/secret in .env, set OAUTH_ENABLED=true, and docker compose up -d. Login is locked to pre-provisioned accounts (allow_sign_up=false), so provision your email as an admin once:

# on vps1, after the stack is up (uses the break-glass admin creds from .env):
curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \
  -X POST http://127.0.0.1:3000/api/admin/users \
  -d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}'
# then make that user a Grafana admin (org role Admin + server admin) via the UI
# or /api/org/users; the Google login matches on email.

Forgejo: add the Google auth source (once, as a Forgejo admin):

docker exec -u git <forgejo-container> forgejo admin auth add-oauth \
  --name google --provider openidConnect \
  --key "<CLIENT_ID>" --secret "<CLIENT_SECRET>" \
  --auto-discover-url https://accounts.google.com/.well-known/openid-configuration

2. Alloy agent (on vps1 and vps2)

ALLOY_NODE names the MACHINE now, not the environment — vps2 alone runs two environments (prod and beta), so a node name can no longer stand in for a host value. ALLOY_ENV supplies the default environment label for anything on that node not attributable to a specific stack (Forgejo, Grafana, the portfolio site). APPLOGS_VOLUME is the node's primary applogs volume; vps2 additionally needs BETA_APPLOGS_VOLUME and a second compose file, since it carries both environments' structured logs side by side.

Every node pushes to Loki over the mesh IP — a containerized agent can't reach the host's 127.0.0.1, but it can reach vps1's wg0 address at 10.10.0.2:3100.

cd thermograph-observability/alloy

# vps1 (dev + Forgejo + the monitoring stack itself):
ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d

# vps2 (prod + beta) — note the SECOND file, adding beta's log volume:
ALLOY_NODE=vps2 ALLOY_ENV=prod APPLOGS_VOLUME=thermograph_applogs \
BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d

The mesh must already be up (see the main repo's deploy/swarm/ / INFRA.md); Swarm is not required for the agent itself — it talks plain HTTP over wg0.

Verify

# From any mesh node, confirm Loki is receiving from every environment
# (host = prod|beta|dev; a separate node/vps1|vps2|desktop label exists too):
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.

The dashboard

Thermograph — Fleet Logs (auto-provisioned), with a Node selector: log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container tail. Edit it in Grafana and re-export the JSON here to version a change.

Alerting

Provisioned from grafana/provisioning/alerting/, mounted read-only into Grafana by the compose file, so the repo is the only durable path — as with the dashboards, edits made in the Grafana UI are overwritten on the next provision.

Every rule is LogQL, because there is no Prometheus in the fleet: Loki is the only datasource, so every signal is derived from log lines.

Where alerts go: Discord, not email

#ops-alerts (Owners category, private to Emi and Jin) via Grafana's native Discord contact point. Deliberately not email:

• Grafana's factory default pointed at the literal string <example@email.com> — a paging path that had never delivered a message to anyone.
• vps1's Grafana relays SMTP through vps2's Postfix at 10.10.0.1:25 (see docker-compose.override.yml on vps1). Email alerts therefore travel through the box most likely to be on fire, and vanish exactly when they matter.
• Discord is off-estate, works when prod is dead, and reaches a phone.

#ops-alerts is a new dedicated channel, not one of the existing ones: the #dev / #uat / #prod channels are the product's alert-subscription output (real deliveries — routing ops noise there would corrupt the evidence), and #notes / #staff-chat are for humans.

The webhook URL is a secret and is not in this repo: it comes from DISCORD_ALERT_WEBHOOK_URL in vps1's gitignored .env, which compose passes to Grafana and which Grafana interpolates when reading the provisioning file. Grafana refuses to start if it is unset.

The rules

Rule	Fires when	Severity
ProdEdgeDark	zero prod Caddy log lines in 15m — site dark, or the log pipeline is broken	critical
ProdHealthzFailing	any non-200 on /healthz in 10m	critical
ProdHTTP5xxBurst	≥10 prod 5xx in 10m	critical
ProdHTTP5xxElevated	≥25 prod 5xx accumulated over 6h (the error-budget alarm)	warning
ProdUnhandledExceptions	≥3 app tag=http.error in 30m	warning
NotifierHeartbeatMissingProd	no tag=heartbeat line from prod in 20m	critical
NotifierHeartbeatMissingBeta	same, on beta	warning
ProdWebContainerSilent	thermograph_web logged nothing in 30m	critical
ProdDbContainerSilent	thermograph_db logged nothing in 2h	critical
ProdFrontendContainerSilent	thermograph_frontend logged nothing in 30m	warning
ProdWorkerContainerSilent	thermograph_worker logged nothing in 2h	warning
AlertingWatchdog	always — one Discord message per day	watchdog

Every threshold was backtested against real Loki data; the working is written out in the comments of rules-prod.yml, next to the rule it justifies. Read those before changing a number.

AlertingWatchdog is the dead-man's switch and fires permanently by design. Its value is its absence: nothing can report its own death, so if #ops-alerts has been silent for more than a day, alerting is broken rather than the estate healthy. Set isPaused: true on it to opt out of that guarantee.

Deploying an alerting change to vps1

Grafana already mounts ./grafana/provisioning read-only, so new files under alerting/ need no compose change — but the first deploy does, because it introduces DISCORD_ALERT_WEBHOOK_URL, and docker restart will not pick up a new environment variable.

# run from the mono-repo root
VPS1="agent@75.119.132.91"  # the monitoring host — NOT the beta environment,
                             # which now runs on vps2; don't repoint this at it.
SSH="ssh -i ~/.ssh/thermograph_agent_ed25519"
TS=$(date -u +%Y%m%dT%H%M%SZ)

# 1. put the secret in vps1's .env (once), then confirm it took:
$SSH $VPS1 "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING"

# 2. back up the live compose file, stage the new files, move them into place:
$SSH $VPS1 "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \
            && rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting"
scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $VPS1:/tmp/obs-stage/
scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $VPS1:/tmp/obs-stage/alerting/
$SSH $VPS1 "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \
            && sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \
            && sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/"

# 3. recreate Grafana so it reads the new env var AND the new provisioning.
#    `docker restart` is NOT enough here — it will not pick up a new env var.
$SSH $VPS1 "cd /opt/observability && docker compose up -d grafana"

Subsequent rule-only changes are just step 2 plus sudo docker restart observability-grafana-1 — no compose needed, because no new environment variable is involved.

Rollback: the provisioning files are additive. To back the whole thing out, sudo rm -rf /opt/observability/grafana/provisioning/alerting, restore docker-compose.yml.<timestamp>, and docker compose up -d grafana. Grafana falls back to its default (email → <example@email.com>) root policy — i.e. no alerting, which is where this started.

Alerting lives on vps1 only — vps2 (prod + beta) and dev only run Alloy agents, not Grafana, so nothing in alerting/ ever ships to them, even though dev happens to share a box with vps1's Grafana.

Verify alerting

# on vps1, after deploying and restarting Grafana:
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-)
# 1. all 12 rules loaded and healthy (health should be "ok", never "error"):
curl -s -u admin:"$GFP" localhost:3000/api/prometheus/grafana/api/v1/rules \
  | python3 -c 'import json,sys; [print(r["name"], r["state"], r["health"]) for g in json.load(sys.stdin)["data"]["groups"] for r in g["rules"]]'
# 2. the root policy points at Discord, NOT grafana-default-email:
curl -s -u admin:"$GFP" localhost:3000/api/v1/provisioning/policies
# 3. force a real message into #ops-alerts (does not persist anything):
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' \
  -X POST localhost:3000/api/alertmanager/grafana/config/api/v1/receivers/test \
  -d '{"receivers":[{"name":"t","grafana_managed_receiver_configs":[{"name":"t","type":"discord","settings":{"url":"'"$WH"'"}}]}]}'
#    -> {"receivers":[{... "status":"ok"}]} and a message in #ops-alerts.

The webhook URL is redacted ([REDACTED]) by GET /api/v1/provisioning/contact-points, so you cannot confirm interpolation by reading it back — the only proof is a message actually arriving in #ops-alerts. A contact point holding the literal string $DISCORD_ALERT_WEBHOOK_URL looks completely healthy from the API and pages nobody.

The leftover default contact point

Grafana's built-in grafana-default-email → <example@email.com> receiver is created by Grafana's default Alertmanager config, not by provisioning, and its uid is the empty string. deleteContactPoints matches on uid, so provisioning cannot remove it (DELETE /api/v1/provisioning/contact-points/ with an empty uid returns 404 — verified).

Provisioning the notification policy already makes it inert: nothing routes there any more, so it cannot receive anything. Actually deleting the row is a one-time manual step, and only works after this config is live (Grafana refuses to delete a receiver a policy still references). Either use the UI — Alerting → Contact points → grafana-default-email → Delete — or:

# on vps1. Reads the running Alertmanager config, drops the one receiver,
# writes it back. Verified against Grafana 11.6.1 (returns 202).
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts
curl -s -u admin:"$GFP" $AM > /tmp/am.json
python3 -c "
import json; d=json.load(open('/tmp/am.json')); c=d['alertmanager_config']
c['receivers']=[r for r in c['receivers'] if r['name']!='grafana-default-email']
json.dump(d, open('/tmp/am2.json','w'))"
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' -X POST --data @/tmp/am2.json $AM
rm -f /tmp/am.json /tmp/am2.json

This is cosmetic — leaving it costs nothing but a confusing row in the UI.

Notes / follow-ups

• Loki is mesh-only by design; only Grafana (with auth) is public.
• Alloy reads the Docker socket read-only to discover containers.
• Retention is 30 days on vps1's disk — bump loki/config.yml if you want more.
• The app emits a tag=heartbeat line for the subscription notifier every ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it. Extend the same audit.log_heartbeat() to any future background daemon.

Known gaps

• Alloy can silently stop tailing a container after Swarm replaces the task. Observed on prod on 2026-07-24: thermograph_worker was 1/1, healthy, and writing ~19 lines per 10 minutes to its own docker log, yet not one line had reached Loki since 04:43Z — a 13½-hour hole nobody could see, because nothing was watching. ProdWorkerContainerSilent now catches exactly this. What localises it to Alloy rather than the app: prod's tag=heartbeat lines kept arriving in every 20-minute window throughout, and those come from a different Alloy source (loki.source.file on /applogs) than the container stdout (loki.source.docker). One source went deaf, the other did not. The workaround is sudo docker restart alloy-alloy-1 on the affected node; the fix is probably an explicit refresh_interval on discovery.docker plus a look at whether loki.source.docker re-resolves targets. Not root-caused.
• Some services are too quiet to alert on. thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and thermograph_autoscaler-lake (2) have a healthy floor indistinguishable from dead, and thermograph_lake is bursty ETL with legitimate idle gaps. Giving each one an audit.log_heartbeat() line would make all four alertable with the same pattern the notifier already uses. That is the single highest-value follow-up here.
• Grafana cannot alert on its own death — that is what AlertingWatchdog is for, but it is a manual dead-man's switch: it relies on somebody noticing that the daily message stopped. A real external watchdog (an uptime pinger hitting a Grafana endpoint from off-estate) would close this properly.
• vps1 runs a docker-compose.override.yml that is not in this repo, wiring Grafana's SMTP to vps2's Postfix. It should be mirrored into the tracked compose file or deleted; alerting no longer depends on it.