thermograph/docs/onboarding/09-observability.md

9. Observability

Fleet-wide log aggregation: Loki + Grafana on vps1, fed by a Grafana Alloy agent on every node, all over the WireGuard mesh. vps2 runs two environments (beta, prod) behind that one Alloy agent; vps1 runs Alloy alongside the things it's shipping logs from (Forgejo, Grafana itself, dev).

  vps2 (10.10.0.1)          vps1 (10.10.0.2)          desktop (10.10.0.3)
  prod + beta               Loki, Grafana, dev
  ┌────────────┐           ┌──────────────────┐      ┌────────────┐
  │ Alloy agent│──wg0──┐   │ Loki  ◀── Alloy  │  ┌───│ Alloy agent│
  └────────────┘       └──▶│ Grafana (Caddy)  │◀─┘   └────────────┘
   docker+caddy+app        └──────────────────┘       docker (AI models)

• Loki — filesystem storage, 30-day retention, listening on the mesh IP 10.10.0.2:3100. Never public.
• Grafana — the UI, fronted by vps1's Caddy at dashboard.thermograph.org (Google SSO, pre-provisioned accounts only, allow_sign_up=false, plus a break-glass local admin). Use that hostname everywhere; never grafana.thermograph.org.
• Alloy — one agent per host, not per environment: vps2's single agent ships logs for both beta and prod, vps1's ships Forgejo/Grafana/dev, and the desktop's ships whatever runs there now. Each ships three sources: every Docker container's stdout/stderr, Caddy's host access logs, and the app's structured JSON logs (errors/access/audit *.jsonl, parsed so level/tag/phase become labels). Every app log line is tagged host = prod|beta|dev — that label is the environment, not the box, which matters on vps2 where one Alloy agent ships lines carrying two different host values.

There is no build and no deploy automation for this domain — it ships by hand. But observability-validate.yml parses every artifact that ships to a node, so a malformed dashboard or an unparseable Alloy config fails in CI instead of on the host.

The rule that matters most

The repo is the only durable path. Dashboards and alerting are provisioned from observability/grafana/provisioning/ at startup, so edits made through Grafana's UI or API are overwritten on the next provision. A change goes through a PR. Centralis's dashboard_write opens one for you.

Reading the fleet

Start wide, then narrow:

1. fleet_status — HTTP reachability plus what's running on each host.
2. logs_overview since="1h" — where the volume is, and what has gone quiet.
3. logs_query service=… host=… — the actual lines.

⚠ The prod (and now beta) trap

Prod and beta both run Swarm stacks — co-resident on vps2 — and their job="docker" streams carry no service label. A raw {host="prod", service="backend"} query matches nothing and reads as "no logs" when it actually means "wrong selector". This used to be a prod-only trap when beta ran compose; it now applies to both environments on vps2, and dev (compose, on vps1) is the one environment this does not affect.

Use Centralis's logs_query service argument, which rewrites it to a container name-regex with the churning task suffix wildcarded. Prod service names are thermograph_web, _worker, _frontend, _db, _autoscaler, _lake; beta's are the same names prefixed beta- (beta-web, beta-worker, …) with no _db or autoscaler equivalents.

Related: prod and beta container names both change on every redeploy (the Swarm task suffix). Never hardcode one; resolve via docker ps --filter name= at call time.

The dashboard

Thermograph — Fleet Logs (auto-provisioned), with a Node selector: log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a per-node notifier-liveness indicator, recent app errors, and a live all-container tail.

Alerting

Provisioned from grafana/provisioning/alerting/, mounted read-only into Grafana.

Every rule is LogQL — there is no Prometheus anywhere in the fleet. Loki is the only datasource, so every signal is derived from log lines. Thresholds were backtested against real Loki data and the working is written out in the comments beside each rule; re-derive before changing a number rather than guessing.

Alerts go to Discord #ops-alerts, never email

Three reasons, all learned the hard way:

• Grafana's factory default pointed at the literal string <example@email.com> — a paging path that had never delivered a message to anyone.
• vps1's Grafana relays SMTP through vps2's Postfix (prod's mail service). Email alerts therefore travel through the box most likely to be on fire, and vanish exactly when they matter.
• Discord is off-estate, works when prod is dead, and reaches a phone.

#ops-alerts is dedicated. #dev / #uat / #prod are the product's alert subscription output — real deliveries to real subscribers — so routing ops noise there would corrupt the evidence.

The webhook URL is a secret and lives only in vps1's gitignored .env as DISCORD_ALERT_WEBHOOK_URL. Grafana refuses to start if it's unset. A literal webhook URL committed to the repo is a hard CI failure.

The twelve rules

Rule	Fires when	Severity
ProdEdgeDark	zero prod Caddy log lines in 15m — site dark, or the pipeline is broken	critical
ProdHealthzFailing	any non-200 on /healthz in 10m	critical
ProdHTTP5xxBurst	≥10 prod 5xx in 10m	critical
ProdHTTP5xxElevated	≥25 prod 5xx over 6h (the error-budget alarm)	warning
ProdUnhandledExceptions	≥3 app tag=http.error in 30m	warning
NotifierHeartbeatMissingProd	no tag=heartbeat from prod in 20m	critical
NotifierHeartbeatMissingBeta	same, on beta	warning
ProdWebContainerSilent	thermograph_web logged nothing in 30m	critical
ProdDbContainerSilent	thermograph_db logged nothing in 2h	critical
ProdFrontendContainerSilent	thermograph_frontend logged nothing in 30m	warning
ProdWorkerContainerSilent	thermograph_worker logged nothing in 2h	warning
AlertingWatchdog	always — one Discord message per day	watchdog

AlertingWatchdog is the dead-man's switch and fires permanently by design. Its value is its absence: nothing can report its own death, so if #ops-alerts has been silent for more than a day, alerting is broken rather than the estate healthy.

The strict CI check, and why

A rule whose condition names a refId that does not exist is perfectly valid YAML, provisions without complaint, and then never fires — a paging path that looks healthy and reaches nobody. observability-validate.yml checks that every condition names a real refId and every policy routes to a receiver that exists, precisely because nothing else would catch it.

The same asymmetry shows up in verification: GET /api/v1/provisioning/contact-points redacts the webhook URL, so you cannot confirm interpolation by reading it back. A contact point holding the literal string $DISCORD_ALERT_WEBHOOK_URL looks completely healthy from the API and pages nobody. The only proof is a message actually arriving in #ops-alerts — force one with the receivers-test endpoint (the command is in observability/README.md).

Deploying an observability change

⚠️ Merging is not deploying — and this one has already bitten. /opt/observability on vps1 and vps2 is a checkout of the archived admin_emi/thermograph-observability repo. It can never git pull again; the content now lives here under observability/. A previous observability PR merged and had zero live effect for exactly this reason.

Until someone re-points those checkouts at the monorepo, every change ships by hand: back up the live file with a UTC-timestamped suffix, scp the new one up, sudo cp it into place, restart the specific container.

Two gotchas in that sequence:

• Use sudo docker restart <container> on vps2 — not docker compose, which there demands a GF_SECURITY_ADMIN_PASSWORD it has no .env to interpolate from. vps1's stack does have an .env, so compose works there.
• docker restart will not pick up a new environment variable. Any change that introduces one needs docker compose up -d grafana on vps1.

Alerting lives on vps1 only — vps2 (prod and beta alike) and dev run Alloy agents, not Grafana.

Full step-by-step, including rollback, is in observability/README.md.

Known gaps (as documented in the repo)

• Alloy can silently stop tailing a container after Swarm replaces the task. Observed on prod 2026-07-24: thermograph_worker was 1/1, healthy, writing ~19 lines per 10 minutes to its own docker log, yet not one line reached Loki for 13½ hours. What localises it to Alloy: prod's tag=heartbeat lines kept arriving throughout, and those come from a different Alloy source (loki.source.file on /applogs) than container stdout (loki.source.docker). One source went deaf, the other didn't. Workaround: sudo docker restart alloy-alloy-1 on the node. Not root-caused. ProdWorkerContainerSilent now catches this class of failure.
• Some services are too quiet to alert on. thermograph_daemon (23 lines / 24h), thermograph_autoscaler (1) and _autoscaler-lake (2) have a healthy floor indistinguishable from dead; thermograph_lake is bursty ETL with legitimate idle gaps. Giving each an audit.log_heartbeat() line would make all four alertable with the pattern the notifier already uses. The single highest-value follow-up in this domain.
• Grafana cannot alert on its own death. AlertingWatchdog is a manual dead-man's switch — it relies on somebody noticing the daily message stopped. An external uptime pinger from off-estate would close this properly.
• vps1 runs a docker-compose.override.yml that is not in this repo, wiring Grafana's SMTP to vps2's Postfix (prod's). It should be mirrored into the tracked compose file or deleted; alerting no longer depends on it.

When something breaks

1. fleet_status — is it up at all?
2. logs_query host=prod contains="(?i)error|traceback" since the incident.
3. deployed_version env=prod — did something ship just before it started?
4. forge_prs / forge_ci — what landed recently, and did CI actually pass?
5. Write down what you found with notes_write before fixing it. The reconstruction is worth more than the fix, and it will be gone by tomorrow.

You are authorised to fix production roadblocks directly — restart or roll a wedged service, clear a full disk, kill a runaway process, bounce a crash-looping container. What you must not do is make the fix durable on the box: writes to code, docs and dashboards go through a PR, and deploy.sh hard-resets the checkout anyway. The on-box action is stabilisation; the PR is the fix.

Next: Recipes.