Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE
|
|
|
|
|
# node — the monitoring host, `beta` (75.119.132.91 / mesh 10.10.0.2) — because
|
|
|
|
|
# beta is an always-on VPS with a public Caddy already fronting it. Every node's
|
|
|
|
|
# Alloy agent (see alloy/) ships logs here; agents on prod and the desktop reach
|
|
|
|
|
# Loki over the WireGuard mesh, so Loki must listen on the mesh interface.
|
|
|
|
|
#
|
|
|
|
|
# Deploy (on beta): docker compose up -d
|
2026-07-22 18:47:21 +00:00
|
|
|
# Grafana is proxied by beta's Caddy at dashboard.thermograph.org (see README);
|
Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0.
|
|
|
|
|
|
|
|
|
|
services:
|
|
|
|
|
loki:
|
|
|
|
|
image: grafana/loki:3.5.1
|
|
|
|
|
command: -config.file=/etc/loki/config.yml
|
|
|
|
|
volumes:
|
|
|
|
|
- ./loki/config.yml:/etc/loki/config.yml:ro
|
|
|
|
|
- loki_data:/loki
|
|
|
|
|
ports:
|
|
|
|
|
# Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here
|
|
|
|
|
# over wg0. Never published on the public interface.
|
|
|
|
|
- "10.10.0.2:3100:3100"
|
|
|
|
|
# Also localhost, so the beta-local Alloy agent and curl checks can reach it.
|
|
|
|
|
- "127.0.0.1:3100:3100"
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
|
|
|
|
grafana:
|
|
|
|
|
image: grafana/grafana:11.6.1
|
|
|
|
|
depends_on: [loki]
|
|
|
|
|
environment:
|
2026-07-21 16:18:29 +00:00
|
|
|
# Primary login is Google SSO (below). The admin user is kept as a
|
|
|
|
|
# break-glass local fallback; its password comes from the host env
|
|
|
|
|
# (.env — see .env.example), never baked into the compose file.
|
Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin}
|
|
|
|
|
GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD}
|
|
|
|
|
GF_USERS_ALLOW_SIGN_UP: "false"
|
|
|
|
|
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
|
|
|
|
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
|
|
|
|
# Served behind Caddy at this external URL (sub-path-safe cookie/redirects).
|
2026-07-21 16:18:29 +00:00
|
|
|
GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-dashboard.thermograph.org}/"
|
|
|
|
|
# --- Google SSO (OIDC) -----------------------------------------------------
|
|
|
|
|
# Enabled once GOOGLE_CLIENT_ID/SECRET are set in .env and OAUTH_ENABLED=true.
|
|
|
|
|
# allow_sign_up is FALSE: a Google login only succeeds if a Grafana user with
|
|
|
|
|
# that email already exists — so this dashboard is locked to pre-provisioned
|
|
|
|
|
# accounts (see README), not "any Google user". Redirect URI to register in
|
|
|
|
|
# Google Cloud: https://dashboard.thermograph.org/login/google
|
|
|
|
|
GF_AUTH_GOOGLE_ENABLED: ${OAUTH_ENABLED:-false}
|
|
|
|
|
GF_AUTH_GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID:-}
|
|
|
|
|
GF_AUTH_GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET:-}
|
|
|
|
|
GF_AUTH_GOOGLE_SCOPES: "openid email profile"
|
|
|
|
|
GF_AUTH_GOOGLE_AUTH_URL: "https://accounts.google.com/o/oauth2/v2/auth"
|
|
|
|
|
GF_AUTH_GOOGLE_TOKEN_URL: "https://oauth2.googleapis.com/token"
|
|
|
|
|
GF_AUTH_GOOGLE_API_URL: "https://openidconnect.googleapis.com/v1/userinfo"
|
|
|
|
|
GF_AUTH_GOOGLE_ALLOW_SIGN_UP: "false"
|
2026-07-21 20:24:37 +00:00
|
|
|
# Match a Google login to a PRE-PROVISIONED user by email (the admin created
|
|
|
|
|
# via the API has no prior Google-auth linkage). Without this, Grafana 9.3+
|
|
|
|
|
# won't look up by email and instead tries to auto-create the user — which
|
|
|
|
|
# allow_sign_up=false then rejects ("signup is disabled"). Safe here: Google
|
|
|
|
|
# verifies email ownership and it's the only OAuth provider, so there's no
|
|
|
|
|
# cross-provider takeover vector the "insecure" name warns about.
|
|
|
|
|
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true"
|
Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
volumes:
|
|
|
|
|
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
|
|
|
|
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
|
|
|
|
|
- grafana_data:/var/lib/grafana
|
|
|
|
|
ports:
|
|
|
|
|
# Host-local; beta's Caddy reverse-proxies to it. Not public directly.
|
|
|
|
|
- "127.0.0.1:3000:3000"
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
|
|
|
|
volumes:
|
|
|
|
|
loki_data: {}
|
|
|
|
|
grafana_data: {}
|