e4693dce58
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 8s
shell-lint / shellcheck (pull_request) Successful in 10s
PR build (required check) / changes (pull_request) Successful in 16s
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
220 lines
13 KiB
Text
220 lines
13 KiB
Text
# This is the REFERENCE for what /etc/thermograph.env contains — the full set of
|
|
# vars the app reads. On a host wired for the SOPS vault (an age key +
|
|
# /etc/thermograph/secrets-env), you do NOT hand-edit /etc/thermograph.env: it is
|
|
# rendered at deploy from the encrypted source of truth in deploy/secrets/*.yaml
|
|
# (see deploy/secrets/README.md). To change a secret there, `sops edit` + commit +
|
|
# deploy — not a host edit. (Legacy: Terraform can also render this file from tfvars,
|
|
# and a from-scratch host can start by copying this example and editing it.)
|
|
#
|
|
# Sourced by deploy/deploy.sh (and any `docker compose` invocation) so compose can
|
|
# interpolate it, AND loaded into the app container (env_file in docker-compose.yml).
|
|
# Anything secret the app needs — Postgres password, VAPID keys, auth secret —
|
|
# belongs here.
|
|
|
|
# Port uvicorn binds inside the container. The compose stack publishes it on the
|
|
# host loopback (127.0.0.1:8137) for Caddy to proxy to. Keep 8137.
|
|
PORT=8137
|
|
|
|
# --- Registry pull (deploy.sh / Terraform) ---------------------------------------
|
|
# deploy.sh logs in to git.thermograph.org's registry and `docker compose pull`s
|
|
# the image build-push.yml already pushed for the deployed commit (repo-split
|
|
# Stage 6), instead of building in place. A personal access token with at least
|
|
# read:package scope -- the same REGISTRY_TOKEN secret build-push.yml uses (that
|
|
# one needs write:package too, to push; this only needs to pull).
|
|
REGISTRY_TOKEN=
|
|
|
|
# --- PostgreSQL (docker-compose stack) ------------------------------------------
|
|
# The app and Postgres run as a docker-compose stack (see docker-compose.yml).
|
|
# POSTGRES_PASSWORD is the database password: compose uses it to initialize the
|
|
# postgres container AND to build the app's THERMOGRAPH_DATABASE_URL. It MUST be
|
|
# set here (the systemd unit sources this file so `docker compose up` can
|
|
# interpolate it). Change it from the default before the first `up`.
|
|
POSTGRES_PASSWORD=change-me
|
|
|
|
# The app's compose service already builds THERMOGRAPH_DATABASE_URL from
|
|
# POSTGRES_PASSWORD, so you normally DON'T need this. It's here for reference and
|
|
# for running the app outside compose against the same DB (keep the password in
|
|
# sync with POSTGRES_PASSWORD above).
|
|
#THERMOGRAPH_DATABASE_URL=postgresql+asyncpg://thermograph:change-me@db:5432/thermograph
|
|
|
|
# --- Historical archive (self-hosted Open-Meteo) --------------------------------
|
|
# Where the app fetches its 45-year daily history. Unset (default) → the public
|
|
# Open-Meteo archive API (rate-limited). On the self-hosting host, the
|
|
# docker-compose.openmeteo.yml overlay sets this to the internal service URL for
|
|
# you, so you normally DON'T set it here. Only pin it to run the app outside that
|
|
# overlay against a reachable Open-Meteo instance. Leave it UNSET rather than empty:
|
|
# an empty value is honored as-is and would break the fallback to the public API.
|
|
#THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive
|
|
|
|
# Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 in prod;
|
|
# leave unset only for plain-HTTP LAN dev (a Secure cookie is never sent over HTTP).
|
|
THERMOGRAPH_COOKIE_SECURE=1
|
|
|
|
# Pin these too (see their own sections below), so container restarts don't rotate
|
|
# them: THERMOGRAPH_AUTH_SECRET (else every emailed confirm/reset link breaks on
|
|
# restart) and THERMOGRAPH_VAPID_PRIVATE_KEY / _PUBLIC_KEY (else every existing push
|
|
# subscription silently stops delivering). The data dir persists on the appdata
|
|
# volume, but pinning here is the safe default.
|
|
|
|
# Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch
|
|
# (e.g. a cache-miss weather lookup) from blocking every other request — the cause of
|
|
# past brief outages. Prod runs 4 (the compose app service also defaults to WORKERS=4);
|
|
# leave unset (defaults to 1) on a small box. Workers elect one leader for the
|
|
# subscription notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, set by the compose
|
|
# app service to /app/data/notifier.lock) so its timer-driven upstream sweep runs once,
|
|
# not once per worker. ~200 MB RAM per worker.
|
|
WORKERS=4
|
|
|
|
# THERMOGRAPH_SINGLETON_LOCK arbitrates workers on ONE host. Under multi-host Swarm,
|
|
# each host would independently elect its own leader — multiplying the Open-Meteo
|
|
# quota use N-fold again. Set THERMOGRAPH_SINGLETON_PG=1 (with THERMOGRAPH_DATABASE_URL
|
|
# pointing at Postgres) to switch to a cluster-wide Postgres advisory lock instead, so
|
|
# exactly one host — not one per host — runs the notifier. Leave unset on a single-host
|
|
# deploy (today's default); the flock above is sufficient there.
|
|
#THERMOGRAPH_SINGLETON_PG=1
|
|
|
|
# Which duties this process performs. Every replica runs the same image; ROLE just
|
|
# decides whether it's allowed to own the notifier once it wins the leader election
|
|
# above — this is what lets the web tier scale to N stateless replicas under Swarm
|
|
# without also scaling notifier instances, while a single worker replica owns it.
|
|
# all -> both (default; today's single-process behavior, unchanged)
|
|
# web -> never runs the notifier, even if it would win leader election
|
|
# worker -> runs the notifier if it wins leader election
|
|
# Read once at process start; changing it needs a restart, not a live toggle.
|
|
#THERMOGRAPH_ROLE=all
|
|
|
|
# The worker's own recurring jobs (city warming, IndexNow), gated by the same
|
|
# leader election as the notifier above. Hours between runs; both are cheap
|
|
# no-op skips when there's nothing to do, so the defaults rarely need changing.
|
|
#THERMOGRAPH_WARM_CITIES_INTERVAL_HOURS=24
|
|
#THERMOGRAPH_INDEXNOW_INTERVAL_HOURS=6
|
|
|
|
# Base path the app is served under.
|
|
# / -> app at the domain root (Thermograph owns the whole domain —
|
|
# this is what the Caddyfile expects: thermograph.org proxies "/")
|
|
# /thermograph -> app under a sub-path (share the host with other apps, e.g. a
|
|
# portfolio at the root)
|
|
THERMOGRAPH_BASE=/
|
|
|
|
# --- SEO: search-engine verification + IndexNow ---------------------------------
|
|
# Ownership-verification tokens, rendered as <meta> tags in every page's <head>.
|
|
# Google Search Console → add property https://thermograph.org → "HTML tag" method
|
|
# → paste just the content="…" value below. (Or verify via DNS TXT and skip this.)
|
|
#THERMOGRAPH_GOOGLE_VERIFY=
|
|
# Bing Webmaster Tools → add site → "HTML Meta Tag" (msvalidate.01) → paste the
|
|
# content value. (Bing can also import verification from Google Search Console.)
|
|
#THERMOGRAPH_BING_VERIFY=
|
|
|
|
# IndexNow key (Bing/DuckDuckGo/Yandex instant re-crawl). Auto-generated to
|
|
# data/indexnow_key.txt on first use; set here to pin a specific key.
|
|
#THERMOGRAPH_INDEXNOW_KEY=
|
|
# Public site URL for IndexNow. The deploy hook auto-pings IndexNow after a
|
|
# successful deploy, but only when the URL set changed (a new/removed city), so
|
|
# code-only deploys don't resubmit. `make indexnow` forces a full submit.
|
|
THERMOGRAPH_BASE_URL=https://thermograph.org
|
|
|
|
# --- Web Push (VAPID) -----------------------------------------------------------
|
|
# Keys that sign push notifications. If unset, the app generates a pair into
|
|
# data/vapid.json on first run — fine as long as that file PERSISTS (it lives in the
|
|
# writable data dir and survives deploys). PIN them here to be safe: if the keys ever
|
|
# change, every existing browser subscription silently stops receiving (the push
|
|
# service rejects with 401/403), and users must toggle alerts off/on to re-subscribe.
|
|
# Generate a pair: cd backend && ../.venv/bin/python -c "import push,json; k=push._generate(); print('PRIVATE=',k['private_key']); print('PUBLIC=',k['public_key'])"
|
|
#THERMOGRAPH_VAPID_PRIVATE_KEY=
|
|
#THERMOGRAPH_VAPID_PUBLIC_KEY=
|
|
# Contact (mailto: or https URL) sent to push services in the VAPID claim.
|
|
#THERMOGRAPH_VAPID_CONTACT=mailto:you@example.com
|
|
|
|
# --- Outbound email --------------------------------------------------------------
|
|
# Delivery goes through the host's Postfix null client (deploy/provision-mail.sh).
|
|
# The app runs in a container, so it can't reach the host's loopback — it speaks
|
|
# plain SMTP to the compose bridge's gateway (172.19.0.1, pinned in
|
|
# docker-compose.yml), where Postfix listens and relays out. Switching between
|
|
# "direct to MX" and "relay through a provider" is a Postfix change, no redeploy.
|
|
# Stack-mode hosts (prod) override this per-service to the docker_gwbridge
|
|
# gateway (172.18.0.1) in deploy/stack/thermograph-stack.yml -- overlay tasks
|
|
# can't reach a compose bridge gateway; leave this file's value as the compose
|
|
# default.
|
|
#
|
|
# WARNING -- this host's rendered value must name an address Postfix ACTUALLY
|
|
# LISTENS ON (`postconf -h inet_interfaces`), or mail fails at send time with
|
|
# connection-refused. prod's SOPS-rendered /etc/thermograph.env currently says
|
|
# 172.19.0.1, which is the *compose* bridge and is NOT in prod's
|
|
# inet_interfaces. It is inert today only because prod runs in stack mode, where
|
|
# thermograph-stack.yml sets THERMOGRAPH_SMTP_HOST per-service and nothing on
|
|
# the box reads this variable from here. It becomes live breakage the moment
|
|
# prod falls back to compose mode. Fix it in deploy/secrets/prod.yaml (sops)
|
|
# rather than on the box; 10.10.0.1 (the mesh address) is the durable choice,
|
|
# since it depends on wg0 rather than on Docker.
|
|
#
|
|
# Backends: console (log it, send nothing — the default, right for dev),
|
|
# smtp (actually send), disabled (drop silently).
|
|
# Leave unset until Postfix is provisioned: signups are still collected either way.
|
|
# THERMOGRAPH_MAIL_FROM is left unset here on purpose — the app default
|
|
# (Thermograph <no-reply@thermograph.org>) is correct, and its "<>" would need
|
|
# escaping in this shell-sourced file. Override only if the address differs.
|
|
#THERMOGRAPH_MAIL_BACKEND=smtp
|
|
#THERMOGRAPH_SMTP_HOST=172.19.0.1
|
|
#THERMOGRAPH_SMTP_PORT=25
|
|
# Only needed if talking to a remote SMTP server directly instead of local Postfix.
|
|
#THERMOGRAPH_SMTP_USER=
|
|
#THERMOGRAPH_SMTP_PASSWORD=
|
|
#THERMOGRAPH_SMTP_STARTTLS=1
|
|
#THERMOGRAPH_MAIL_FROM=Thermograph <no-reply@thermograph.org>
|
|
#THERMOGRAPH_MAIL_REPLY_TO=
|
|
|
|
# Signing secret for email confirmation / password-reset tokens. MUST be set to a
|
|
# fixed value before any such link is mailed: it defaults to a per-boot random
|
|
# value, which would invalidate every outstanding link on each restart.
|
|
# generate with: python -c "import secrets; print(secrets.token_urlsafe(48))"
|
|
#THERMOGRAPH_AUTH_SECRET=
|
|
|
|
# --- Discord ---------------------------------------------------------------------
|
|
# Incoming webhook URL for the daily "most unusual right now" post. Create it in the
|
|
# Discord server: Channel → Edit → Integrations → Webhooks → New Webhook → Copy URL.
|
|
# The URL IS the credential — anyone who has it can post as the webhook, so keep it
|
|
# here and never in the repo. Unset => the daily post is disabled (no-op).
|
|
# The post rides the notifier daemon (leader-only), once per day after the feed
|
|
# refresh, so it needs THERMOGRAPH_ENABLE_NOTIFIER on (the default).
|
|
#THERMOGRAPH_DISCORD_WEBHOOK=
|
|
# Slash commands (/grade) are served over Discord's HTTP interactions endpoint by
|
|
# the app itself (no bot process). Set the portal's "Interactions Endpoint URL" to
|
|
# https://thermograph.org/discord/interactions. These come from the Developer Portal:
|
|
# - PUBLIC_KEY: General Information -> Public Key (used to verify every request).
|
|
# - APP_ID / BOT_TOKEN: only needed to (re)register the commands, via
|
|
# scripts/register_discord_commands.py. The bot token is a credential.
|
|
#THERMOGRAPH_DISCORD_PUBLIC_KEY=
|
|
#THERMOGRAPH_DISCORD_APP_ID=
|
|
#THERMOGRAPH_DISCORD_BOT_TOKEN=
|
|
# Channel IDs (numeric snowflakes) for the bot to post into with the BOT_TOKEN above.
|
|
# Subscription test feed: this deployment mirrors every notification the notifier
|
|
# creates into this channel, so a run can be watched live. Set it per environment;
|
|
# leave unset to disable. In the Thermograph.org server the hidden channels are:
|
|
# dev = 1529274513009934336
|
|
# uat = 1529274514066636861
|
|
# prod = 1529274515127799940
|
|
#THERMOGRAPH_DISCORD_SUBSCRIPTION_CHANNEL=
|
|
# Notable-weather feed: prod broadcasts the daily "most unusual right now" digest
|
|
# into this channel via the bot (independent of the WEBHOOK above — either can run
|
|
# alone). Leave unset to disable. In the Thermograph.org server:
|
|
# weather-events = 1529274516746932307
|
|
#THERMOGRAPH_DISCORD_WEATHER_CHANNEL=
|
|
# Discord gateway bot (@mention/DM grading): runs INSIDE the backend leader
|
|
# process (same singleton election as the notifier — see web/app.py's lifespan),
|
|
# riding its event loop; no separate bot process. Opt-in: any truthy value here
|
|
# starts it, provided BOT_TOKEN above is also set (flag alone is a silent no-op).
|
|
# Discord allows ONE gateway connection per bot token, so enable this in exactly
|
|
# one environment — prod, the only vault with the token.
|
|
#THERMOGRAPH_DISCORD_BOT=
|
|
# Account linking (OAuth2 "identify"): lets a signed-in user connect their Discord
|
|
# account, storing their Discord user id for DM alerts. CLIENT_ID is the same App
|
|
# ID above. The CLIENT_SECRET is from OAuth2 -> Client Secret (a credential). In the
|
|
# portal, add the redirect: https://thermograph.org/api/v2/discord/link/callback
|
|
#THERMOGRAPH_DISCORD_CLIENT_SECRET=
|
|
# Gateway bot (opt-in): holds a live websocket so the bot replies to messages that
|
|
# @mention it (or DM it) with a city grade — e.g. "@Thermograph Phoenix". Reuses
|
|
# THERMOGRAPH_DISCORD_BOT_TOKEN above. Runs on the single notifier leader only, so
|
|
# it needs THERMOGRAPH_ENABLE_NOTIFIER on (the default). No privileged intent
|
|
# required — Discord delivers content for mentions/DMs. Unset/0 => no gateway
|
|
# connection. (Code: thermograph-backend notifications/discord_bot.py.)
|
|
#THERMOGRAPH_DISCORD_BOT=1
|