# Thermograph — observability

Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central
**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all
over the private WireGuard mesh. This replaces the old in-repo
`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a
persistent, queryable, fleet-wide log store and UI.

```
  prod (10.10.0.1)          beta (10.10.0.2)            desktop / LAN dev (10.10.0.3)
  ┌────────────┐            ┌──────────────────┐        ┌────────────┐
  │ Alloy agent│──wg0──┐    │ Loki  ◀── Alloy  │   ┌──wg0│ Alloy agent│
  └────────────┘       └───▶│ Grafana (Caddy)  │◀──┘    └────────────┘
   docker+caddy+app         └──────────────────┘         docker+caddy+app
     logs                    central store + UI            logs
```

- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
  `10.10.0.2:3100` — never public.
- **Grafana** is the UI, fronted by beta's Caddy at `grafana.thermograph.org`
  (its own login; Loki is not exposed).
- **Alloy** runs on each node and ships three sources to Loki: every Docker
  container's stdout/stderr, Caddy's host access logs, and the app's structured
  JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
  become labels). Every line is tagged with its node (`host = prod|beta|dev`).

## Layout

| Path | What |
|------|------|
| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) |
| `loki/config.yml` | Loki config (filesystem, retention) |
| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
| `alloy/config.alloy` | the per-node collector config |
| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI |

## Deploy

### 1. Central stack (on beta)

```bash
# on beta (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
```

Expose the Grafana UI: point `grafana.thermograph.org` at beta, append
`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.

### 2. Alloy agent (on every node — prod, beta, desktop)

```bash
cd thermograph-observability/alloy
# beta pushes to local Loki; prod & desktop push over the mesh:
#   beta:    LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push  ALLOY_NODE=beta
#   prod:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=prod
#   desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=dev
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d
```

On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not
`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml`
there. The mesh must already be up (see the main repo's `deploy/swarm/` /
`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0.

## Verify

```bash
# From any mesh node, confirm Loki is receiving from every host:
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
```

## The dashboard

`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector:
log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
count, a notifier-liveness log panel, recent app errors, and a live all-container
tail. Edit it in Grafana and re-export the JSON here to version a change.

## Notes / follow-ups

- **Loki is mesh-only** by design; only Grafana (with auth) is public.
- Alloy reads the Docker socket read-only to discover containers.
- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more.
- Possible next step: have the app emit a periodic `notifier heartbeat` log line
  so notifier liveness is a first-class log-derived signal, not just container
  chatter.