thermograph/observability/README.md

# Thermograph — observability

Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central
**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all
over the private WireGuard mesh. This replaces the old in-repo
`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a
persistent, queryable, fleet-wide log store and UI.

```
  vps1 (10.10.0.2)                  vps2 (10.10.0.1)                desktop (10.10.0.3)
  ┌───────────────────────┐         ┌───────────────────────┐       ┌──────────────┐
  │ Loki  ◀───── Alloy    │◀──wg0───┤ Alloy agent           │◀─wg0──┤ Alloy agent  │
  │ Grafana (Caddy)       │         │ prod + beta (Swarm)   │       │ (no app      │
  │ dev (compose)         │◀──wg0───┤ Postfix, backups      │       │  stack)      │
  │ Forgejo, portfolio    │         └───────────────────────┘       └──────────────┘
  └───────────────────────┘          docker+caddy+app logs           ships no app logs
   central store + UI + dev
```

- **vps1** is the monitoring host: Loki, Grafana, Forgejo, the portfolio site,
  and the **dev** environment (its own compose project, mesh-only, no public
  DNS). It does **not** run the beta *environment* — that now lives on vps2 — so
  any older doc that says "beta" meaning *this box* is stale and needs reading
  as vps1.
- **vps2** runs **prod and beta** as two separate Docker Swarm stacks
  (`thermograph`, `thermograph-beta`), plus Postfix and backups. One shared
  TimescaleDB serves both, on separate roles/databases.
- **desktop** hosts no Thermograph environment now (AI-model hosting + flex
  Swarm-worker capacity instead), so it ships no app logs.
- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
  `10.10.0.2:3100` — never public.
- **Grafana** is the UI, fronted by vps1's Caddy at `dashboard.thermograph.org`,
  login via **Google SSO** (with a break-glass local admin); Loki is not exposed.
- **Alloy** runs on each node and ships three sources to Loki: every Docker
  container's stdout/stderr, Caddy's host access logs, and the app's structured
  JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
  become labels). Every line now carries two distinct labels: `host`
  (`prod`|`beta`|`dev`) is the ENVIRONMENT, derived *per source* (a container's
  stack prefix, the Caddy access-log filename, or which `applogs` volume a JSON
  line came out of) — because vps2 alone carries two environments, and one
  node-wide value would have stamped every beta container and log line as
  `prod`. `node` (`vps1`|`vps2`|`desktop`) is the MACHINE, stamped externally on
  everything an agent ships. See `alloy/config.alloy` for exactly how `host` is
  derived per source.

## Layout

| Path | What |
|------|------|
| `docker-compose.yml` | the central Loki + Grafana stack (runs on vps1) |
| `loki/config.yml` | Loki config (filesystem, retention) |
| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
| `grafana/provisioning/alerting/` | contact point, notification policy and alert rules — see [Alerting](#alerting) |
| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
| `alloy/config.alloy` | the per-node collector config |
| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
| `alloy/docker-compose.agent.beta.yml` | overlay adding beta's `applogs` mount — vps2 only |
| `caddy-grafana.conf` | vps1 Caddy site block for the Grafana UI |

## Deploy

> ### ⚠️ Merging is not deploying
>
> `/opt/observability` on vps1 (and vps2) is a checkout of the **archived**
> `emi/thermograph-observability` repo. It can never `git pull` again — the
> content now lives here, in the mono repo, under `observability/`. A previous
> observability PR merged and had **zero live effect** for exactly this reason.
>
> Until someone re-points those checkouts at the mono repo, every change here
> ships by hand: back up the live file with a UTC-timestamped suffix, `scp` the
> new one up, `sudo cp` it into place, restart the specific container. Use
> `sudo docker restart <container>` — **not `docker compose`**, which on vps2
> demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to interpolate from
> (vps2 only runs the Alloy agent — there's no Grafana there to configure).
> (vps1's stack does have an `.env`, so compose works there — and is required
> when a change adds a new environment variable, since `docker restart` alone
> will not pick one up.)

### 1. Central stack (on vps1)

```bash
# on vps1 (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
```

Expose the Grafana UI: point `dashboard.thermograph.org` at vps1, append
`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.

### Google SSO (Grafana + Forgejo)

Both the dashboard and Forgejo log in with Google. Create **one** Google Cloud
OAuth 2.0 Client (type: *Web application*) and register both redirect URIs:

- Grafana: `https://dashboard.thermograph.org/login/google`
- Forgejo: `https://git.thermograph.org/user/oauth2/google/callback`

**Grafana:** put the client id/secret in `.env`, set `OAUTH_ENABLED=true`, and
`docker compose up -d`. Login is locked to pre-provisioned accounts
(`allow_sign_up=false`), so provision your email as an admin once:

```bash
# on vps1, after the stack is up (uses the break-glass admin creds from .env):
curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \
  -X POST http://127.0.0.1:3000/api/admin/users \
  -d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}'
# then make that user a Grafana admin (org role Admin + server admin) via the UI
# or /api/org/users; the Google login matches on email.
```

**Forgejo:** add the Google auth source (once, as a Forgejo admin):

```bash
docker exec -u git <forgejo-container> forgejo admin auth add-oauth \
  --name google --provider openidConnect \
  --key "<CLIENT_ID>" --secret "<CLIENT_SECRET>" \
  --auto-discover-url https://accounts.google.com/.well-known/openid-configuration
```

### 2. Alloy agent (on vps1 and vps2)

`ALLOY_NODE` names the MACHINE now, not the environment — vps2 alone runs two
environments (prod and beta), so a node name can no longer stand in for a
`host` value. `ALLOY_ENV` supplies the default environment label for anything
on that node not attributable to a specific stack (Forgejo, Grafana, the
portfolio site). `APPLOGS_VOLUME` is the node's primary `applogs` volume; vps2
additionally needs `BETA_APPLOGS_VOLUME` and a second compose file, since it
carries both environments' structured logs side by side.

Every node pushes to Loki over the mesh IP — a containerized agent can't reach
the host's `127.0.0.1`, but it can reach vps1's wg0 address at
`10.10.0.2:3100`.

```bash
cd thermograph-observability/alloy

# vps1 (dev + Forgejo + the monitoring stack itself):
ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d

# vps2 (prod + beta) — note the SECOND file, adding beta's log volume:
ALLOY_NODE=vps2 ALLOY_ENV=prod APPLOGS_VOLUME=thermograph_applogs \
BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
```

The mesh must already be up (see the main repo's `deploy/swarm/` / `INFRA.md`);
Swarm is **not** required for the agent itself — it talks plain HTTP over wg0.

<!-- TODO(cutover): desktop no longer runs a Thermograph app stack and ships no
     app logs, but it isn't clear whether it still runs an Alloy agent at all
     (for its AI-model / flex-worker container logs) or has been dropped from
     the fleet entirely. `ALLOY_NODE=desktop` is still a valid value in
     alloy/config.alloy and docker-compose.agent.yml, but no desktop deploy
     command is given here — confirm and fill in once decided. -->

## Verify

```bash
# From any mesh node, confirm Loki is receiving from every environment
# (host = prod|beta|dev; a separate node/vps1|vps2|desktop label exists too):
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
```

## The dashboard

`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector:
log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container
tail. Edit it in Grafana and re-export the JSON here to version a change.

<!-- TODO(cutover): the dashboard JSON isn't in this pass's scope, but check
     whether its "Node" selector still means `node` (the machine) now that the
     label has that meaning, or whether some panels actually want to filter by
     `host` (the environment) instead — vps2 mixes prod and beta under one
     node value, so a machine-only selector can no longer isolate one
     environment there. -->

## Alerting

Provisioned from `grafana/provisioning/alerting/`, mounted read-only into Grafana
by the compose file, so **the repo is the only durable path** — as with the
dashboards, edits made in the Grafana UI are overwritten on the next provision.

Every rule is **LogQL**, because there is no Prometheus in the fleet: Loki is the
only datasource, so every signal is derived from log lines.

### Where alerts go: Discord, not email

`#ops-alerts` (Owners category, private to Emi and Jin) via Grafana's native
Discord contact point. Deliberately **not email**:

- Grafana's factory default pointed at the literal string `<example@email.com>` —
  a paging path that had never delivered a message to anyone.
- vps1's Grafana relays SMTP through **vps2's** Postfix at `10.10.0.1:25` (see
  `docker-compose.override.yml` on vps1). Email alerts therefore travel through
  the box most likely to be on fire, and vanish exactly when they matter.
- Discord is off-estate, works when prod is dead, and reaches a phone.

`#ops-alerts` is a new dedicated channel, not one of the existing ones: the
`#dev` / `#uat` / `#prod` channels are the *product's* alert-subscription output
(real deliveries — routing ops noise there would corrupt the evidence), and
`#notes` / `#staff-chat` are for humans.

The webhook URL is a secret and is **not in this repo**: it comes from
`DISCORD_ALERT_WEBHOOK_URL` in vps1's gitignored `.env`, which compose passes to
Grafana and which Grafana interpolates when reading the provisioning file.
Grafana refuses to start if it is unset.

### The rules

| Rule | Fires when | Severity |
|------|-----------|----------|
| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the log pipeline is broken | critical |
| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
| `ProdHTTP5xxElevated` | ≥25 prod 5xx accumulated over 6h (the error-budget alarm) | warning |
| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` line from prod in 20m | critical |
| `NotifierHeartbeatMissingBeta` | same, on beta | warning |
| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
| `AlertingWatchdog` | always — one Discord message per day | watchdog |

Every threshold was backtested against real Loki data; the working is written out
in the comments of `rules-prod.yml`, next to the rule it justifies. Read those
before changing a number.

`AlertingWatchdog` is the dead-man's switch and fires permanently by design. **Its
value is its absence**: nothing can report its own death, so if `#ops-alerts` has
been silent for more than a day, alerting is broken rather than the estate
healthy. Set `isPaused: true` on it to opt out of that guarantee.

### Deploying an alerting change to vps1

Grafana already mounts `./grafana/provisioning` read-only, so new files under
`alerting/` need no compose change — but the **first** deploy does, because it
introduces `DISCORD_ALERT_WEBHOOK_URL`, and `docker restart` will not pick up a
new environment variable.

```bash
# run from the mono-repo root
VPS1="agent@75.119.132.91"  # the monitoring host — NOT the beta environment,
                             # which now runs on vps2; don't repoint this at it.
SSH="ssh -i ~/.ssh/thermograph_agent_ed25519"
TS=$(date -u +%Y%m%dT%H%M%SZ)

# 1. put the secret in vps1's .env (once), then confirm it took:
$SSH $VPS1 "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING"

# 2. back up the live compose file, stage the new files, move them into place:
$SSH $VPS1 "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \
            && rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting"
scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $VPS1:/tmp/obs-stage/
scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $VPS1:/tmp/obs-stage/alerting/
$SSH $VPS1 "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \
            && sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \
            && sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/"

# 3. recreate Grafana so it reads the new env var AND the new provisioning.
#    `docker restart` is NOT enough here — it will not pick up a new env var.
$SSH $VPS1 "cd /opt/observability && docker compose up -d grafana"
```

Subsequent rule-only changes are just step 2 plus
`sudo docker restart observability-grafana-1` — no compose needed, because no
new environment variable is involved.

**Rollback:** the provisioning files are additive. To back the whole thing out,
`sudo rm -rf /opt/observability/grafana/provisioning/alerting`, restore
`docker-compose.yml.<timestamp>`, and `docker compose up -d grafana`. Grafana
falls back to its default (email → `<example@email.com>`) root policy — i.e. no
alerting, which is where this started.

Alerting lives on **vps1 only** — vps2 (prod + beta) and dev only run Alloy
agents, not Grafana, so nothing in `alerting/` ever ships to them, even though
dev happens to share a box with vps1's Grafana.

### Verify alerting

```bash
# on vps1, after deploying and restarting Grafana:
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-)
# 1. all 12 rules loaded and healthy (health should be "ok", never "error"):
curl -s -u admin:"$GFP" localhost:3000/api/prometheus/grafana/api/v1/rules \
  | python3 -c 'import json,sys; [print(r["name"], r["state"], r["health"]) for g in json.load(sys.stdin)["data"]["groups"] for r in g["rules"]]'
# 2. the root policy points at Discord, NOT grafana-default-email:
curl -s -u admin:"$GFP" localhost:3000/api/v1/provisioning/policies
# 3. force a real message into #ops-alerts (does not persist anything):
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' \
  -X POST localhost:3000/api/alertmanager/grafana/config/api/v1/receivers/test \
  -d '{"receivers":[{"name":"t","grafana_managed_receiver_configs":[{"name":"t","type":"discord","settings":{"url":"'"$WH"'"}}]}]}'
#    -> {"receivers":[{... "status":"ok"}]} and a message in #ops-alerts.
```

The webhook URL is redacted (`[REDACTED]`) by
`GET /api/v1/provisioning/contact-points`, so you **cannot** confirm
interpolation by reading it back — the only proof is a message actually arriving
in `#ops-alerts`. A contact point holding the literal string
`$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and pages
nobody.

### The leftover default contact point

Grafana's built-in `grafana-default-email` → `<example@email.com>` receiver is
created by Grafana's *default Alertmanager config*, not by provisioning, and its
uid is the empty string. `deleteContactPoints` matches on uid, so **provisioning
cannot remove it** (`DELETE /api/v1/provisioning/contact-points/` with an empty
uid returns 404 — verified).

Provisioning the notification policy already makes it inert: nothing routes there
any more, so it cannot receive anything. Actually deleting the row is a one-time
manual step, and only works *after* this config is live (Grafana refuses to
delete a receiver a policy still references). Either use the UI — Alerting →
Contact points → `grafana-default-email` → Delete — or:

```bash
# on vps1. Reads the running Alertmanager config, drops the one receiver,
# writes it back. Verified against Grafana 11.6.1 (returns 202).
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts
curl -s -u admin:"$GFP" $AM > /tmp/am.json
python3 -c "
import json; d=json.load(open('/tmp/am.json')); c=d['alertmanager_config']
c['receivers']=[r for r in c['receivers'] if r['name']!='grafana-default-email']
json.dump(d, open('/tmp/am2.json','w'))"
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' -X POST --data @/tmp/am2.json $AM
rm -f /tmp/am.json /tmp/am2.json
```

This is cosmetic — leaving it costs nothing but a confusing row in the UI.

## Notes / follow-ups

- **Loki is mesh-only** by design; only Grafana (with auth) is public.
- Alloy reads the Docker socket read-only to discover containers.
- Retention is 30 days on vps1's disk — bump `loki/config.yml` if you want more.
- The app emits a `tag=heartbeat` line for the subscription notifier every
  ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it.
  Extend the same `audit.log_heartbeat()` to any future background daemon.

### Known gaps

- **Alloy can silently stop tailing a container after Swarm replaces the task.**
  Observed on prod on 2026-07-24: `thermograph_worker` was `1/1`, healthy, and
  writing ~19 lines per 10 minutes to its own docker log, yet not one line had
  reached Loki since 04:43Z — a 13½-hour hole nobody could see, because nothing
  was watching. `ProdWorkerContainerSilent` now catches exactly this.
  What localises it to Alloy rather than the app: prod's `tag=heartbeat` lines
  kept arriving in *every* 20-minute window throughout, and those come from a
  different Alloy source (`loki.source.file` on `/applogs`) than the container
  stdout (`loki.source.docker`). One source went deaf, the other did not.
  The workaround is `sudo docker restart alloy-alloy-1` on the affected node;
  the fix is probably an explicit `refresh_interval` on `discovery.docker` plus
  a look at whether `loki.source.docker` re-resolves targets. Not root-caused.
- **Some services are too quiet to alert on.** `thermograph_daemon` (23 log
  lines/24h), `thermograph_autoscaler` (1) and `thermograph_autoscaler-lake` (2)
  have a healthy floor indistinguishable from dead, and `thermograph_lake` is
  bursty ETL with legitimate idle gaps. Giving each one an
  `audit.log_heartbeat()` line would make all four alertable with the same
  pattern the notifier already uses. That is the single highest-value follow-up
  here.
- **Grafana cannot alert on its own death** — that is what `AlertingWatchdog`
  is for, but it is a *manual* dead-man's switch: it relies on somebody noticing
  that the daily message stopped. A real external watchdog (an uptime pinger
  hitting a Grafana endpoint from off-estate) would close this properly.
- **vps1 runs a `docker-compose.override.yml` that is not in this repo**, wiring
  Grafana's SMTP to vps2's Postfix. It should be mirrored into the tracked
  compose file or deleted; alerting no longer depends on it.