thermograph/observability/README.md

# Thermograph — observability

Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central
**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all
over the private WireGuard mesh. This replaces the old in-repo
`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a
persistent, queryable, fleet-wide log store and UI.

```
  prod (10.10.0.1)          beta (10.10.0.2)            desktop / LAN dev (10.10.0.3)
  ┌────────────┐            ┌──────────────────┐        ┌────────────┐
  │ Alloy agent│──wg0──┐    │ Loki  ◀── Alloy  │   ┌──wg0│ Alloy agent│
  └────────────┘       └───▶│ Grafana (Caddy)  │◀──┘    └────────────┘
   docker+caddy+app         └──────────────────┘         docker+caddy+app
     logs                    central store + UI            logs
```

- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
  `10.10.0.2:3100` — never public.
- **Grafana** is the UI, fronted by beta's Caddy at `dashboard.thermograph.org`,
  login via **Google SSO** (with a break-glass local admin); Loki is not exposed.
- **Alloy** runs on each node and ships three sources to Loki: every Docker
  container's stdout/stderr, Caddy's host access logs, and the app's structured
  JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
  become labels). Every line is tagged with its node (`host = prod|beta|dev`).

## Layout

| Path | What |
|------|------|
| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) |
| `loki/config.yml` | Loki config (filesystem, retention) |
| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
| `grafana/provisioning/alerting/` | contact point, notification policy and alert rules — see [Alerting](#alerting) |
| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
| `alloy/config.alloy` | the per-node collector config |
| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI |

## Deploy

> ### ⚠️ Merging is not deploying
>
> `/opt/observability` on beta (and prod) is a checkout of the **archived**
> `emi/thermograph-observability` repo. It can never `git pull` again — the
> content now lives here, in the mono repo, under `observability/`. A previous
> observability PR merged and had **zero live effect** for exactly this reason.
>
> Until someone re-points those checkouts at the mono repo, every change here
> ships by hand: back up the live file with a UTC-timestamped suffix, `scp` the
> new one up, `sudo cp` it into place, restart the specific container. Use
> `sudo docker restart <container>` — **not `docker compose`**, which on prod
> demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to interpolate from.
> (Beta's stack does have an `.env`, so compose works there — and is required
> when a change adds a new environment variable, since `docker restart` alone
> will not pick one up.)

### 1. Central stack (on beta)

```bash
# on beta (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
```

Expose the Grafana UI: point `dashboard.thermograph.org` at beta, append
`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.

### Google SSO (Grafana + Forgejo)

Both the dashboard and Forgejo log in with Google. Create **one** Google Cloud
OAuth 2.0 Client (type: *Web application*) and register both redirect URIs:

- Grafana: `https://dashboard.thermograph.org/login/google`
- Forgejo: `https://git.thermograph.org/user/oauth2/google/callback`

**Grafana:** put the client id/secret in `.env`, set `OAUTH_ENABLED=true`, and
`docker compose up -d`. Login is locked to pre-provisioned accounts
(`allow_sign_up=false`), so provision your email as an admin once:

```bash
# on beta, after the stack is up (uses the break-glass admin creds from .env):
curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \
  -X POST http://127.0.0.1:3000/api/admin/users \
  -d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}'
# then make that user a Grafana admin (org role Admin + server admin) via the UI
# or /api/org/users; the Google login matches on email.
```

**Forgejo:** add the Google auth source (once, as a Forgejo admin):

```bash
docker exec -u git <forgejo-container> forgejo admin auth add-oauth \
  --name google --provider openidConnect \
  --key "<CLIENT_ID>" --secret "<CLIENT_SECRET>" \
  --auto-discover-url https://accounts.google.com/.well-known/openid-configuration
```

### 2. Alloy agent (on every node — prod, beta, desktop)

```bash
cd thermograph-observability/alloy
# Every node (beta included) pushes to Loki over the mesh IP — a containerized
# agent can't reach the host's 127.0.0.1, but it can reach beta's wg0 address:
#   beta:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=beta
#   prod:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=prod
#   desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=dev
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d
```

On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not
`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml`
there. The mesh must already be up (see the main repo's `deploy/swarm/` /
`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0.

## Verify

```bash
# From any mesh node, confirm Loki is receiving from every host:
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
```

## The dashboard

`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector:
log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container
tail. Edit it in Grafana and re-export the JSON here to version a change.

## Alerting

Provisioned from `grafana/provisioning/alerting/`, mounted read-only into Grafana
by the compose file, so **the repo is the only durable path** — as with the
dashboards, edits made in the Grafana UI are overwritten on the next provision.

Every rule is **LogQL**, because there is no Prometheus in the fleet: Loki is the
only datasource, so every signal is derived from log lines.

### Where alerts go: Discord, not email

`#ops-alerts` (Owners category, private to Emi and Jin) via Grafana's native
Discord contact point. Deliberately **not email**:

- Grafana's factory default pointed at the literal string `<example@email.com>` —
  a paging path that had never delivered a message to anyone.
- beta's Grafana relays SMTP through **prod's** Postfix at `10.10.0.1:25` (see
  `docker-compose.override.yml` on beta). Email alerts therefore travel through
  the box most likely to be on fire, and vanish exactly when they matter.
- Discord is off-estate, works when prod is dead, and reaches a phone.

`#ops-alerts` is a new dedicated channel, not one of the existing ones: the
`#dev` / `#uat` / `#prod` channels are the *product's* alert-subscription output
(real deliveries — routing ops noise there would corrupt the evidence), and
`#notes` / `#staff-chat` are for humans.

The webhook URL is a secret and is **not in this repo**: it comes from
`DISCORD_ALERT_WEBHOOK_URL` in beta's gitignored `.env`, which compose passes to
Grafana and which Grafana interpolates when reading the provisioning file.
Grafana refuses to start if it is unset.

### The rules

| Rule | Fires when | Severity |
|------|-----------|----------|
| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the log pipeline is broken | critical |
| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
| `ProdHTTP5xxElevated` | ≥25 prod 5xx accumulated over 6h (the error-budget alarm) | warning |
| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` line from prod in 20m | critical |
| `NotifierHeartbeatMissingBeta` | same, on beta | warning |
| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
| `AlertingWatchdog` | always — one Discord message per day | watchdog |

Every threshold was backtested against real Loki data; the working is written out
in the comments of `rules-prod.yml`, next to the rule it justifies. Read those
before changing a number.

`AlertingWatchdog` is the dead-man's switch and fires permanently by design. **Its
value is its absence**: nothing can report its own death, so if `#ops-alerts` has
been silent for more than a day, alerting is broken rather than the estate
healthy. Set `isPaused: true` on it to opt out of that guarantee.

### Deploying an alerting change to beta

Grafana already mounts `./grafana/provisioning` read-only, so new files under
`alerting/` need no compose change — but the **first** deploy does, because it
introduces `DISCORD_ALERT_WEBHOOK_URL`, and `docker restart` will not pick up a
new environment variable.

```bash
# run from the mono-repo root
BETA="agent@75.119.132.91"
SSH="ssh -i ~/.ssh/thermograph_agent_ed25519"
TS=$(date -u +%Y%m%dT%H%M%SZ)

# 1. put the secret in beta's .env (once), then confirm it took:
$SSH $BETA "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING"

# 2. back up the live compose file, stage the new files, move them into place:
$SSH $BETA "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \
            && rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting"
scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $BETA:/tmp/obs-stage/
scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $BETA:/tmp/obs-stage/alerting/
$SSH $BETA "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \
            && sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \
            && sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/"

# 3. recreate Grafana so it reads the new env var AND the new provisioning.
#    `docker restart` is NOT enough here — it will not pick up a new env var.
$SSH $BETA "cd /opt/observability && docker compose up -d grafana"
```

Subsequent rule-only changes are just step 2 plus
`sudo docker restart observability-grafana-1` — no compose needed, because no
new environment variable is involved.

**Rollback:** the provisioning files are additive. To back the whole thing out,
`sudo rm -rf /opt/observability/grafana/provisioning/alerting`, restore
`docker-compose.yml.<timestamp>`, and `docker compose up -d grafana`. Grafana
falls back to its default (email → `<example@email.com>`) root policy — i.e. no
alerting, which is where this started.

Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana, so
nothing in `alerting/` ever ships to them.

### Verify alerting

```bash
# on beta, after deploying and restarting Grafana:
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-)
# 1. all 12 rules loaded and healthy (health should be "ok", never "error"):
curl -s -u admin:"$GFP" localhost:3000/api/prometheus/grafana/api/v1/rules \
  | python3 -c 'import json,sys; [print(r["name"], r["state"], r["health"]) for g in json.load(sys.stdin)["data"]["groups"] for r in g["rules"]]'
# 2. the root policy points at Discord, NOT grafana-default-email:
curl -s -u admin:"$GFP" localhost:3000/api/v1/provisioning/policies
# 3. force a real message into #ops-alerts (does not persist anything):
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' \
  -X POST localhost:3000/api/alertmanager/grafana/config/api/v1/receivers/test \
  -d '{"receivers":[{"name":"t","grafana_managed_receiver_configs":[{"name":"t","type":"discord","settings":{"url":"'"$WH"'"}}]}]}'
#    -> {"receivers":[{... "status":"ok"}]} and a message in #ops-alerts.
```

The webhook URL is redacted (`[REDACTED]`) by
`GET /api/v1/provisioning/contact-points`, so you **cannot** confirm
interpolation by reading it back — the only proof is a message actually arriving
in `#ops-alerts`. A contact point holding the literal string
`$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and pages
nobody.

### The leftover default contact point

Grafana's built-in `grafana-default-email` → `<example@email.com>` receiver is
created by Grafana's *default Alertmanager config*, not by provisioning, and its
uid is the empty string. `deleteContactPoints` matches on uid, so **provisioning
cannot remove it** (`DELETE /api/v1/provisioning/contact-points/` with an empty
uid returns 404 — verified).

Provisioning the notification policy already makes it inert: nothing routes there
any more, so it cannot receive anything. Actually deleting the row is a one-time
manual step, and only works *after* this config is live (Grafana refuses to
delete a receiver a policy still references). Either use the UI — Alerting →
Contact points → `grafana-default-email` → Delete — or:

```bash
# on beta. Reads the running Alertmanager config, drops the one receiver,
# writes it back. Verified against Grafana 11.6.1 (returns 202).
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts
curl -s -u admin:"$GFP" $AM > /tmp/am.json
python3 -c "
import json; d=json.load(open('/tmp/am.json')); c=d['alertmanager_config']
c['receivers']=[r for r in c['receivers'] if r['name']!='grafana-default-email']
json.dump(d, open('/tmp/am2.json','w'))"
curl -s -u admin:"$GFP" -H 'Content-Type: application/json' -X POST --data @/tmp/am2.json $AM
rm -f /tmp/am.json /tmp/am2.json
```

This is cosmetic — leaving it costs nothing but a confusing row in the UI.

## Notes / follow-ups

- **Loki is mesh-only** by design; only Grafana (with auth) is public.
- Alloy reads the Docker socket read-only to discover containers.
- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more.
- The app emits a `tag=heartbeat` line for the subscription notifier every
  ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it.
  Extend the same `audit.log_heartbeat()` to any future background daemon.

### Known gaps

- **Alloy can silently stop tailing a container after Swarm replaces the task.**
  Observed on prod on 2026-07-24: `thermograph_worker` was `1/1`, healthy, and
  writing ~19 lines per 10 minutes to its own docker log, yet not one line had
  reached Loki since 04:43Z — a 13½-hour hole nobody could see, because nothing
  was watching. `ProdWorkerContainerSilent` now catches exactly this.
  What localises it to Alloy rather than the app: prod's `tag=heartbeat` lines
  kept arriving in *every* 20-minute window throughout, and those come from a
  different Alloy source (`loki.source.file` on `/applogs`) than the container
  stdout (`loki.source.docker`). One source went deaf, the other did not.
  The workaround is `sudo docker restart alloy-alloy-1` on the affected node;
  the fix is probably an explicit `refresh_interval` on `discovery.docker` plus
  a look at whether `loki.source.docker` re-resolves targets. Not root-caused.
- **Some services are too quiet to alert on.** `thermograph_daemon` (23 log
  lines/24h), `thermograph_autoscaler` (1) and `thermograph_autoscaler-lake` (2)
  have a healthy floor indistinguishable from dead, and `thermograph_lake` is
  bursty ETL with legitimate idle gaps. Giving each one an
  `audit.log_heartbeat()` line would make all four alertable with the same
  pattern the notifier already uses. That is the single highest-value follow-up
  here.
- **Grafana cannot alert on its own death** — that is what `AlertingWatchdog`
  is for, but it is a *manual* dead-man's switch: it relies on somebody noticing
  that the daily message stopped. A real external watchdog (an uptime pinger
  hitting a Grafana endpoint from off-estate) would close this properly.
- **beta runs a `docker-compose.override.yml` that is not in this repo**, wiring
  Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked
  compose file or deleted; alerting no longer depends on it.