thermograph/README.md

Thermograph — observability

Fleet-wide log aggregation for Thermograph: a central Grafana + Loki stack, fed by a Grafana Alloy agent on every node, all over the private WireGuard mesh. This replaces the old in-repo scripts/dashboard.py (an SSH-tailed, single-host, metrics-endpoint view) with a persistent, queryable, fleet-wide log store and UI.

  prod (10.10.0.1)          beta (10.10.0.2)            desktop / LAN dev (10.10.0.3)
  ┌────────────┐            ┌──────────────────┐        ┌────────────┐
  │ Alloy agent│──wg0──┐    │ Loki  ◀── Alloy  │   ┌──wg0│ Alloy agent│
  └────────────┘       └───▶│ Grafana (Caddy)  │◀──┘    └────────────┘
   docker+caddy+app         └──────────────────┘         docker+caddy+app
     logs                    central store + UI            logs

• Loki stores logs (filesystem, 30-day retention). Listens on the mesh IP 10.10.0.2:3100 — never public.
• Grafana is the UI, fronted by beta's Caddy at grafana.thermograph.org (its own login; Loki is not exposed).
• Alloy runs on each node and ships three sources to Loki: every Docker container's stdout/stderr, Caddy's host access logs, and the app's structured JSON logs (errors/access/audit *.jsonl, parsed so level/tag/phase become labels). Every line is tagged with its node (host = prod|beta|dev).

Layout

Path	What
docker-compose.yml	the central Loki + Grafana stack (runs on beta)
loki/config.yml	Loki config (filesystem, retention)
grafana/provisioning/	auto-wired Loki datasource + dashboard provider
grafana/dashboards/thermograph-logs.json	the fleet-logs dashboard
alloy/config.alloy	the per-node collector config
alloy/docker-compose.agent.yml	runs the Alloy agent on a node
caddy-grafana.conf	beta Caddy site block for the Grafana UI

Deploy

1. Central stack (on beta)

# on beta (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000

Expose the Grafana UI: point grafana.thermograph.org at beta, append caddy-grafana.conf to /etc/caddy/Caddyfile, systemctl reload caddy.

2. Alloy agent (on every node — prod, beta, desktop)

cd thermograph-observability/alloy
# beta pushes to local Loki; prod & desktop push over the mesh:
#   beta:    LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push  ALLOY_NODE=beta
#   prod:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=prod
#   desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=dev
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d

On the LAN dev node the app's log volume is thermograph-dev_applogs (not thermograph_applogs) — edit the two references in docker-compose.agent.yml there. The mesh must already be up (see the main repo's deploy/swarm/ / INFRA.md); Swarm is not required — the agents talk plain HTTP over wg0.

Verify

# From any mesh node, confirm Loki is receiving from every host:
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.

The dashboard

Thermograph — Fleet Logs (auto-provisioned), with a Node selector: log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a notifier-liveness log panel, recent app errors, and a live all-container tail. Edit it in Grafana and re-export the JSON here to version a change.

Notes / follow-ups

• Loki is mesh-only by design; only Grafana (with auth) is public.
• Alloy reads the Docker socket read-only to discover containers.
• Retention is 30 days on beta's disk — bump loki/config.yml if you want more.
• Possible next step: have the app emit a periodic notifier heartbeat log line so notifier liveness is a first-class log-derived signal, not just container chatter.