2026-07-26 06:56:38 +00:00
|
|
|
# Copy to .env on the monitoring host (vps1) before `docker compose up`.
|
Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
# .env is gitignored — never commit real secrets.
|
2026-07-26 06:56:38 +00:00
|
|
|
#
|
|
|
|
|
# vps1 is NOT the beta environment — that runs on vps2 now, as its own Swarm
|
|
|
|
|
# stack. Don't confuse the two when reading older docs that say "beta" here.
|
Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
|
2026-07-26 06:56:38 +00:00
|
|
|
# The public hostname vps1's Caddy serves Grafana at (see caddy-grafana.conf).
|
2026-07-21 16:18:29 +00:00
|
|
|
GRAFANA_DOMAIN=dashboard.thermograph.org
|
|
|
|
|
|
|
|
|
|
# Break-glass local admin (Google SSO below is the primary login). Keep this
|
|
|
|
|
# strong; it's the fallback if OAuth ever breaks.
|
Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
GF_ADMIN_USER=admin
|
|
|
|
|
GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong
|
|
|
|
|
|
2026-07-21 16:18:29 +00:00
|
|
|
# --- Google SSO ------------------------------------------------------------------
|
|
|
|
|
# Create a Google Cloud "OAuth 2.0 Client ID" (type: Web application) and set:
|
|
|
|
|
# Authorized redirect URI: https://dashboard.thermograph.org/login/google
|
|
|
|
|
# Then paste the client id/secret here and flip OAUTH_ENABLED=true.
|
|
|
|
|
# Login is locked to pre-provisioned accounts (allow_sign_up=false) — provision
|
|
|
|
|
# your email as an admin (see README) so only you can sign in.
|
|
|
|
|
OAUTH_ENABLED=false
|
|
|
|
|
GOOGLE_CLIENT_ID=
|
|
|
|
|
GOOGLE_CLIENT_SECRET=
|
observability: add the estate's first alerting; supervise Postfix
The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.
Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
one bad request is 1.4% and a ratio alert would scream all night. The 5xx
burst rule fires on the outage's 45 and 22 buckets and on nothing else in
the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
#weather-events, #announcements and #prod are product surfaces that notify
real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
Grafana against beta's live Loki and a real alert arrived in Discord. This
matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
for the URL — a contact point holding an uninterpolated env var looks
perfectly healthy and pages nobody. The only proof is a message arriving.
CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.
Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
Stock postfix@.service is ordered only After=network-online.target and ships
no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
address actually exists, which absorbs the transient case inside a single
start attempt. That makes bounded retry correct: 5 attempts in 600s, then
failed — a genuinely broken config reaches a visible failed state in ~100s
instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
"failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
mesh listeners that do not depend on Docker at all.
Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.
DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.
Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 20:19:19 +00:00
|
|
|
|
|
|
|
|
# --- Alerting (Discord) ----------------------------------------------------------
|
|
|
|
|
# Where every alert in grafana/provisioning/alerting/ is delivered: a webhook on
|
|
|
|
|
# the private #ops-alerts channel (Owners category) of the Thermograph.org
|
2026-07-26 06:56:38 +00:00
|
|
|
# Discord. NOT email — vps1's Grafana relays SMTP through vps2's Postfix, so
|
|
|
|
|
# email alerts die exactly when prod (on vps2) does.
|
observability: add the estate's first alerting; supervise Postfix
The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.
Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
one bad request is 1.4% and a ratio alert would scream all night. The 5xx
burst rule fires on the outage's 45 and 22 buckets and on nothing else in
the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
#weather-events, #announcements and #prod are product surfaces that notify
real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
Grafana against beta's live Loki and a real alert arrived in Discord. This
matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
for the URL — a contact point holding an uninterpolated env var looks
perfectly healthy and pages nobody. The only proof is a message arriving.
CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.
Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
Stock postfix@.service is ordered only After=network-online.target and ships
no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
address actually exists, which absorbs the transient case inside a single
start attempt. That makes bounded retry correct: 5 attempts in 600s, then
failed — a genuinely broken config reaches a visible failed state in ~100s
instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
"failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
mesh listeners that do not depend on Docker at all.
Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.
DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.
Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 20:19:19 +00:00
|
|
|
#
|
|
|
|
|
# To mint one: Discord -> #ops-alerts -> Edit Channel -> Integrations ->
|
|
|
|
|
# Webhooks -> New Webhook -> Copy Webhook URL. Treat it like a password; anyone
|
|
|
|
|
# with it can post to the channel. Grafana refuses to start without it.
|
|
|
|
|
DISCORD_ALERT_WEBHOOK_URL=https://discord.com/api/webhooks/<id>/<token>
|
