e4693dce58
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 8s
shell-lint / shellcheck (pull_request) Successful in 10s
PR build (required check) / changes (pull_request) Successful in 16s
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
80 lines
4.3 KiB
YAML
80 lines
4.3 KiB
YAML
# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE
|
|
# node — the monitoring host, `beta` (75.119.132.91 / mesh 10.10.0.2) — because
|
|
# beta is an always-on VPS with a public Caddy already fronting it. Every node's
|
|
# Alloy agent (see alloy/) ships logs here; agents on prod and the desktop reach
|
|
# Loki over the WireGuard mesh, so Loki must listen on the mesh interface.
|
|
#
|
|
# Deploy (on beta): docker compose up -d
|
|
# Grafana is proxied by beta's Caddy at dashboard.thermograph.org (see README);
|
|
# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0.
|
|
|
|
services:
|
|
loki:
|
|
image: grafana/loki:3.5.1
|
|
command: -config.file=/etc/loki/config.yml
|
|
volumes:
|
|
- ./loki/config.yml:/etc/loki/config.yml:ro
|
|
- loki_data:/loki
|
|
ports:
|
|
# Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here
|
|
# over wg0. Never published on the public interface.
|
|
- "10.10.0.2:3100:3100"
|
|
# Also localhost, so the beta-local Alloy agent and curl checks can reach it.
|
|
- "127.0.0.1:3100:3100"
|
|
restart: unless-stopped
|
|
|
|
grafana:
|
|
image: grafana/grafana:11.6.1
|
|
depends_on: [loki]
|
|
environment:
|
|
# Primary login is Google SSO (below). The admin user is kept as a
|
|
# break-glass local fallback; its password comes from the host env
|
|
# (.env — see .env.example), never baked into the compose file.
|
|
GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin}
|
|
GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD}
|
|
GF_USERS_ALLOW_SIGN_UP: "false"
|
|
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
|
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
|
# Served behind Caddy at this external URL (sub-path-safe cookie/redirects).
|
|
GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-dashboard.thermograph.org}/"
|
|
# --- Google SSO (OIDC) -----------------------------------------------------
|
|
# Enabled once GOOGLE_CLIENT_ID/SECRET are set in .env and OAUTH_ENABLED=true.
|
|
# allow_sign_up is FALSE: a Google login only succeeds if a Grafana user with
|
|
# that email already exists — so this dashboard is locked to pre-provisioned
|
|
# accounts (see README), not "any Google user". Redirect URI to register in
|
|
# Google Cloud: https://dashboard.thermograph.org/login/google
|
|
GF_AUTH_GOOGLE_ENABLED: ${OAUTH_ENABLED:-false}
|
|
GF_AUTH_GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID:-}
|
|
GF_AUTH_GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET:-}
|
|
GF_AUTH_GOOGLE_SCOPES: "openid email profile"
|
|
GF_AUTH_GOOGLE_AUTH_URL: "https://accounts.google.com/o/oauth2/v2/auth"
|
|
GF_AUTH_GOOGLE_TOKEN_URL: "https://oauth2.googleapis.com/token"
|
|
GF_AUTH_GOOGLE_API_URL: "https://openidconnect.googleapis.com/v1/userinfo"
|
|
GF_AUTH_GOOGLE_ALLOW_SIGN_UP: "false"
|
|
# Match a Google login to a PRE-PROVISIONED user by email (the admin created
|
|
# via the API has no prior Google-auth linkage). Without this, Grafana 9.3+
|
|
# won't look up by email and instead tries to auto-create the user — which
|
|
# allow_sign_up=false then rejects ("signup is disabled"). Safe here: Google
|
|
# verifies email ownership and it's the only OAuth provider, so there's no
|
|
# cross-provider takeover vector the "insecure" name warns about.
|
|
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true"
|
|
# --- Alerting --------------------------------------------------------------
|
|
# The Discord webhook that grafana/provisioning/alerting/contact-points.yml
|
|
# interpolates as $DISCORD_ALERT_WEBHOOK_URL. It is a secret (holding it is
|
|
# enough to post in the channel), so it lives only in beta's .env.
|
|
# Required, not defaulted: a Grafana that comes up with an empty webhook
|
|
# looks perfectly healthy and pages nobody, which is the failure mode this
|
|
# whole config exists to end. Better to refuse to start.
|
|
DISCORD_ALERT_WEBHOOK_URL: ${DISCORD_ALERT_WEBHOOK_URL:?set DISCORD_ALERT_WEBHOOK_URL — see .env.example}
|
|
volumes:
|
|
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
|
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
|
|
- grafana_data:/var/lib/grafana
|
|
ports:
|
|
# Host-local; beta's Caddy reverse-proxies to it. Not public directly.
|
|
- "127.0.0.1:3000:3000"
|
|
restart: unless-stopped
|
|
|
|
volumes:
|
|
loki_data: {}
|
|
grafana_data: {}
|