thermograph/observability

Thermograph — observability

Fleet-wide log aggregation for Thermograph: a central Grafana + Loki stack, fed by a Grafana Alloy agent on every node, all over the private WireGuard mesh. This replaces the old in-repo scripts/dashboard.py (an SSH-tailed, single-host, metrics-endpoint view) with a persistent, queryable, fleet-wide log store and UI.

  prod (10.10.0.1)          beta (10.10.0.2)            desktop / LAN dev (10.10.0.3)
  ┌────────────┐            ┌──────────────────┐        ┌────────────┐
  │ Alloy agent│──wg0──┐    │ Loki  ◀── Alloy  │   ┌──wg0│ Alloy agent│
  └────────────┘       └───▶│ Grafana (Caddy)  │◀──┘    └────────────┘
   docker+caddy+app         └──────────────────┘         docker+caddy+app
     logs                    central store + UI            logs

• Loki stores logs (filesystem, 30-day retention). Listens on the mesh IP 10.10.0.2:3100 — never public.
• Grafana is the UI, fronted by beta's Caddy at dashboard.thermograph.org, login via Google SSO (with a break-glass local admin); Loki is not exposed.
• Alloy runs on each node and ships three sources to Loki: every Docker container's stdout/stderr, Caddy's host access logs, and the app's structured JSON logs (errors/access/audit *.jsonl, parsed so level/tag/phase become labels). Every line is tagged with its node (host = prod|beta|dev).

Layout

Path	What
docker-compose.yml	the central Loki + Grafana stack (runs on beta)
loki/config.yml	Loki config (filesystem, retention)
grafana/provisioning/	auto-wired Loki datasource + dashboard provider
grafana/dashboards/thermograph-logs.json	the fleet-logs dashboard
alloy/config.alloy	the per-node collector config
alloy/docker-compose.agent.yml	runs the Alloy agent on a node
caddy-grafana.conf	beta Caddy site block for the Grafana UI

Deploy

1. Central stack (on beta)

# on beta (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000

Expose the Grafana UI: point dashboard.thermograph.org at beta, append caddy-grafana.conf to /etc/caddy/Caddyfile, systemctl reload caddy.

Google SSO (Grafana + Forgejo)

Both the dashboard and Forgejo log in with Google. Create one Google Cloud OAuth 2.0 Client (type: Web application) and register both redirect URIs:

• Grafana: https://dashboard.thermograph.org/login/google
• Forgejo: https://git.thermograph.org/user/oauth2/google/callback

Grafana: put the client id/secret in .env, set OAUTH_ENABLED=true, and docker compose up -d. Login is locked to pre-provisioned accounts (allow_sign_up=false), so provision your email as an admin once:

# on beta, after the stack is up (uses the break-glass admin creds from .env):
curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \
  -X POST http://127.0.0.1:3000/api/admin/users \
  -d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}'
# then make that user a Grafana admin (org role Admin + server admin) via the UI
# or /api/org/users; the Google login matches on email.

Forgejo: add the Google auth source (once, as a Forgejo admin):

docker exec -u git <forgejo-container> forgejo admin auth add-oauth \
  --name google --provider openidConnect \
  --key "<CLIENT_ID>" --secret "<CLIENT_SECRET>" \
  --auto-discover-url https://accounts.google.com/.well-known/openid-configuration

2. Alloy agent (on every node — prod, beta, desktop)

cd thermograph-observability/alloy
# Every node (beta included) pushes to Loki over the mesh IP — a containerized
# agent can't reach the host's 127.0.0.1, but it can reach beta's wg0 address:
#   beta:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=beta
#   prod:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=prod
#   desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=dev
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d

On the LAN dev node the app's log volume is thermograph-dev_applogs (not thermograph_applogs) — edit the two references in docker-compose.agent.yml there. The mesh must already be up (see the main repo's deploy/swarm/ / INFRA.md); Swarm is not required — the agents talk plain HTTP over wg0.

Verify

# From any mesh node, confirm Loki is receiving from every host:
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.

The dashboard

Thermograph — Fleet Logs (auto-provisioned), with a Node selector: log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container tail. Edit it in Grafana and re-export the JSON here to version a change.

Notes / follow-ups

• Loki is mesh-only by design; only Grafana (with auth) is public.
• Alloy reads the Docker socket read-only to discover containers.
• Retention is 30 days on beta's disk — bump loki/config.yml if you want more.
• The app emits a tag=heartbeat line for the subscription notifier every ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it. Extend the same audit.log_heartbeat() to any future background daemon.