thermograph/observability/docker-compose.yml

84 lines
4.5 KiB
YAML
Raw Permalink Normal View History

# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE
# node — the monitoring host, `vps1` (75.119.132.91 / mesh 10.10.0.2) — because
# vps1 is an always-on VPS with a public Caddy already fronting it. This is NOT
# where the beta *environment* lives (that's vps2 now, alongside prod, as two
# Swarm stacks) — don't let "beta" in older docs send you looking for Grafana
# there. Every node's Alloy agent (see alloy/) ships logs here; agents on vps2
# and desktop reach Loki over the WireGuard mesh, so Loki must listen on the
# mesh interface.
#
# Deploy (on vps1): docker compose up -d
# Grafana is proxied by vps1's Caddy at dashboard.thermograph.org (see README);
# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0.
services:
loki:
image: grafana/loki:3.5.1
command: -config.file=/etc/loki/config.yml
volumes:
- ./loki/config.yml:/etc/loki/config.yml:ro
- loki_data:/loki
ports:
# Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here
# over wg0. Never published on the public interface.
- "10.10.0.2:3100:3100"
# Also localhost, so the vps1-local Alloy agent and curl checks can reach it.
- "127.0.0.1:3100:3100"
restart: unless-stopped
grafana:
image: grafana/grafana:11.6.1
depends_on: [loki]
environment:
# Primary login is Google SSO (below). The admin user is kept as a
# break-glass local fallback; its password comes from the host env
# (.env — see .env.example), never baked into the compose file.
GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin}
GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD}
GF_USERS_ALLOW_SIGN_UP: "false"
GF_ANALYTICS_REPORTING_ENABLED: "false"
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
# Served behind Caddy at this external URL (sub-path-safe cookie/redirects).
GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-dashboard.thermograph.org}/"
# --- Google SSO (OIDC) -----------------------------------------------------
# Enabled once GOOGLE_CLIENT_ID/SECRET are set in .env and OAUTH_ENABLED=true.
# allow_sign_up is FALSE: a Google login only succeeds if a Grafana user with
# that email already exists — so this dashboard is locked to pre-provisioned
# accounts (see README), not "any Google user". Redirect URI to register in
# Google Cloud: https://dashboard.thermograph.org/login/google
GF_AUTH_GOOGLE_ENABLED: ${OAUTH_ENABLED:-false}
GF_AUTH_GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID:-}
GF_AUTH_GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET:-}
GF_AUTH_GOOGLE_SCOPES: "openid email profile"
GF_AUTH_GOOGLE_AUTH_URL: "https://accounts.google.com/o/oauth2/v2/auth"
GF_AUTH_GOOGLE_TOKEN_URL: "https://oauth2.googleapis.com/token"
GF_AUTH_GOOGLE_API_URL: "https://openidconnect.googleapis.com/v1/userinfo"
GF_AUTH_GOOGLE_ALLOW_SIGN_UP: "false"
# Match a Google login to a PRE-PROVISIONED user by email (the admin created
# via the API has no prior Google-auth linkage). Without this, Grafana 9.3+
# won't look up by email and instead tries to auto-create the user — which
# allow_sign_up=false then rejects ("signup is disabled"). Safe here: Google
# verifies email ownership and it's the only OAuth provider, so there's no
# cross-provider takeover vector the "insecure" name warns about.
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true"
observability: add the estate's first alerting; supervise Postfix The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 20:19:19 +00:00
# --- Alerting --------------------------------------------------------------
# The Discord webhook that grafana/provisioning/alerting/contact-points.yml
# interpolates as $DISCORD_ALERT_WEBHOOK_URL. It is a secret (holding it is
# enough to post in the channel), so it lives only in vps1's .env.
observability: add the estate's first alerting; supervise Postfix The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 20:19:19 +00:00
# Required, not defaulted: a Grafana that comes up with an empty webhook
# looks perfectly healthy and pages nobody, which is the failure mode this
# whole config exists to end. Better to refuse to start.
DISCORD_ALERT_WEBHOOK_URL: ${DISCORD_ALERT_WEBHOOK_URL:?set DISCORD_ALERT_WEBHOOK_URL — see .env.example}
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana_data:/var/lib/grafana
ports:
# Host-local; vps1's Caddy reverse-proxies to it. Not public directly.
- "127.0.0.1:3000:3000"
restart: unless-stopped
volumes:
loki_data: {}
grafana_data: {}