# Deploying Thermograph to prod (and beta) on vps2 > **Prod and beta both run as Docker Swarm stacks on vps2** (`169.58.46.181`, > mesh `10.10.0.1`) — two separate stacks, two separate checkouts > (`/opt/thermograph` and `/opt/thermograph-beta`), sharing one host because a > beta green light is meant to be evidence about prod (same orchestrator, same > Postgres build, same Caddy, same mesh position). `deploy/env-topology.sh` is > the single source of truth for which environment lives where; every path > below sources it rather than hardcoding a path or a port. The **manual > walkthrough below (`provision.sh`, `thermograph.service`, the venv) is > legacy** — kept as a from-scratch, no-orchestrator reference; prod and beta's > actual process model is a Swarm stack (`deploy/stack/`), and dev's is plain > compose on vps1 (see `DEPLOY-DEV.md`), so the systemd/venv specifics below no > longer match how any live environment actually runs. Pipeline, both environments: **push to `main` (beta) or `release` (prod) on Forgejo → `build-push.yml` builds + pushes the image, tagged by SHA → the single `Deploy` workflow (`.forgejo/workflows/deploy.yml`) SSHes into **vps2** with `THERMOGRAPH_ENV=beta` or `THERMOGRAPH_ENV=prod` → `deploy/deploy.sh` resolves the right checkout/branch/stack from `env-topology.sh`, `git reset`s it, renders that environment's secrets, and execs `deploy/stack/deploy-stack.sh` (schema migration runs as a one-shot task before the roll) → each environment's own loopback Caddy LB fronts it with TLS.** Credentials are keyed by **host** (`VPS2_SSH_*`), not by environment — vps2 answers to both prod and beta, so the environment is passed as data (`THERMOGRAPH_ENV`), not baked into which secret is used. (GitHub is retired/archived — Forgejo, self-hosted at `git.thermograph.org` on **vps1** over the WireGuard mesh, is now the sole git host and CI.) Files that make this work: | File | Where it lives on vps2 | Purpose | |------|------------------------|---------| | `.forgejo/workflows/deploy.yml` | Forgejo | CI job that SSHes in and runs the deploy script, for all three environments | | `.forgejo/workflows/build-push.yml` | Forgejo | builds + pushes the SHA-tagged image `deploy.sh` pulls | | `deploy/env-topology.sh` | `/opt/thermograph{,-beta}/infra/deploy/env-topology.sh` | resolves `THERMOGRAPH_ENV` to a checkout, branch, stack name, ports, DB role | | `deploy/deploy.sh` | `/opt/thermograph{,-beta}/infra/deploy/deploy.sh` | `git reset` + render secrets + routes to the Swarm-stack deploy | | `deploy/stack/thermograph-stack.yml` / `thermograph-beta-stack.yml` | same | prod's / beta's Swarm stack (db, web, worker, lake, daemon, frontend; beta has no `db` of its own) | | `deploy/thermograph.env.example` | `/etc/thermograph.env` (prod) / `/etc/thermograph-beta.env` (beta) | Postgres password, VAPID/auth secrets, `WORKERS`, sizing, base path | | `deploy/stack/lb/Caddyfile` / `Caddyfile.beta` | each stack's loopback LB container | reverse proxy from the host's real Caddy into the Swarm overlay | | `terraform/` | run from your machine | provisions the box (host-level only; does not know about the two-environment split — see `ACCESS.md`) | | `deploy/provision.sh`, `deploy/thermograph.service` | *(legacy)* | pre-compose venv+systemd bootstrap — superseded by the Swarm stack | Each app serves on **loopback only** (prod `127.0.0.1:8137`/`:8080`, beta `127.0.0.1:8237`/`:8180`); the host's real Caddy is the only thing exposed to the internet (ports 80/443), reverse-proxying into whichever loopback LB matches the domain requested. Each environment's own checkout keeps its own gitignored cache under `data/cache/`, so a `git pull` in one never touches the other's. **Any deploy host needs a `/etc/hosts` entry for the registry.** `git.thermograph.org`'s public DNS resolves to **vps1's** public IP, and vps1's own Caddy rejects `/v2/*` (the registry API) from anything outside the WireGuard mesh (10.10.0.0/24) — confirmed live: `docker login`/`pull` from vps2 failed with a 403 until adding `10.10.0.2 git.thermograph.org` to `/etc/hosts` (10.10.0.2 is vps1's mesh IP; every deploy host is already on the mesh, this is purely a DNS-routing gap, not a connectivity one). vps1 itself doesn't need this — it's the box Forgejo runs on, so `git.thermograph.org` just resolves to itself either way. `build-push.yml`'s own header comment documents the same requirement for the CI runner host. **Current production layout**: prod's Caddy config owns **`thermograph.org`** at its root (`THERMOGRAPH_BASE=/`, so uvicorn serves `/`, `/calendar`, `/api/v2/…` with no prefix) and beta's owns **`beta.thermograph.org`** the same way. **`emigriffith.dev`** is a separate static portfolio site, served from **vps1**, not vps2 — it shares no host, checkout or Caddy config with either app environment. `emigriffith.dev/thermograph*` permanently redirects to `thermograph.org` (prefix stripped). > **Note:** `deploy.sh` (the CI deploy) only pulls code, reinstalls deps, and > restarts the app service — it does **not** touch Caddy or `/etc/thermograph.env`. > After changing `deploy/Caddyfile` or the env, copy it onto the VPS and reload > Caddy / restart the service by hand (see Part 2 step 4 and Part 3). --- ## Part 1 — Keys: how to manage them You need **two** separate SSH keypairs. Don't reuse one for both. ### Key A — your login key (you ↔ VPS), you probably already have this This is how *you* SSH in to run `provision.sh`. Nothing to set up here beyond your existing access. ### Key B — the CI deploy key (Forgejo Actions ↔ VPS), you create this A dedicated key whose **private half** lives in a Forgejo Actions secret and whose **public half** is authorized on the VPS's deploy user. It should only be able to log in as `deploy` and run the deploy script. Generate it on your machine (no passphrase — CI can't type one): ```bash ssh-keygen -t ed25519 -C "thermograph-ci" -f ~/.ssh/thermograph_ci -N "" ``` That makes two files: - `~/.ssh/thermograph_ci` → **private** key → goes into the Forgejo Actions secret `SSH_KEY` - `~/.ssh/thermograph_ci.pub` → **public** key → goes on the VPS Install the **public** key on the VPS (as the `deploy` user): ```bash ssh-copy-id -i ~/.ssh/thermograph_ci.pub deploy@YOUR_VPS_IP # or manually: append the .pub line to /home/deploy/.ssh/authorized_keys ``` Put the **private** key into Forgejo → repo **Settings → Actions → Secrets → Add Secret**. Set all of these: | Secret | Value | |--------|-------| | `SSH_KEY` | full contents of `~/.ssh/thermograph_ci` (the private key, incl. BEGIN/END lines) | | `SSH_HOST` | VPS IP or hostname | | `SSH_USER` | `deploy` | | `SSH_PORT` | `22` (or your custom SSH port) | No `tea` CLI is installed for a command-line alternative; use the web UI, or the REST API with a personal access token: `POST /api/v1/repos/{owner}/{repo}/actions/secrets/{name}` (body `{"data": "..."}`). **Why I can't do this for you:** Forgejo secrets are write-only and require your Forgejo auth; the VPS's `authorized_keys` requires your SSH access. Both are credentials only you should hold. ### Deploy key for cloning a private repo (optional) If the Forgejo repo is **private**, the VPS also needs to *pull* from it. Two options: - Make `provision.sh`'s `REPO_URL` an **HTTPS** URL with an embedded personal access token (what the live VPS checkout actually uses today — see its `origin` remote), or - Add the `deploy` user's own key (`ssh-keygen` on the VPS) as a **read-only Deploy Key** in the repo (Settings → Deploy keys). This is separate from Key B. If the repo is **public**, skip this — `git pull` needs no auth. --- ## Part 2 — Install (once, on the VPS) 1. **Create the Forgejo repo and push** (this project isn't versioned yet): ```bash cd ~/Code/Thermograph git init && git add -A && git commit -m "Initial commit" # create an empty private repo on the Forgejo instance first (web UI or # POST /api/v1/user/repos), then: git remote add origin https:///OWNER/thermograph.git git push -u origin main ``` 2. **Point DNS** for each domain you serve (`thermograph.org` for the app, `emigriffith.dev` for the portfolio) at the VPS IP with an `A` record. Let's Encrypt won't issue a cert until the name resolves to the box. 3. **Provision the box.** Copy `deploy/provision.sh` over (or clone the repo) and edit `REPO_URL` at the top, then: ```bash REPO_URL="https:///OWNER/thermograph.git" bash deploy/provision.sh ``` It installs Python + git + Caddy, creates the `deploy` user, clones to `/opt/thermograph`, installs the systemd unit + env file, grants the deploy user a password-less `systemctl restart thermograph` (so CI can restart it), and starts the service. 4. **Install the Caddy config** and reload. `deploy/Caddyfile` already carries the real domains (`thermograph.org` + `emigriffith.dev`); edit them if yours differ, then copy it into place, validate, and reload: ```bash sudo cp /opt/thermograph/deploy/Caddyfile /etc/caddy/Caddyfile sudo caddy validate --config /etc/caddy/Caddyfile # syntax check before reload sudo systemctl reload caddy # Caddy fetches each domain's TLS cert automatically ``` Do this again any time `deploy/Caddyfile` changes — the CI deploy does not ship it. Set the app's base path to match in `/etc/thermograph.env` (`THERMOGRAPH_BASE=/` for the app-at-root layout), then `sudo systemctl restart thermograph`. 5. **Install Key B** (Part 1) so CI can log in. 6. **Verify:** open `https://YOUR_DOMAIN/` — you should get a clean padlock. --- ## Part 3 — Day-to-day - **Deploy:** just `git push` to `main`. Actions runs `deploy.sh` and health-checks the service. Watch it under the repo's **Actions** tab on Forgejo (or query `/api/v1/repos/{owner}/{repo}/actions/tasks`). - **Manual deploy / rollback:** `ssh deploy@vps '/opt/thermograph/deploy/deploy.sh'` (set `BRANCH=` or check out a tag first to roll back). - **Logs:** `journalctl -u thermograph -f` (app), `/var/log/caddy/thermograph.log` (proxy). - **Restart:** `sudo systemctl restart thermograph`. - **Change port / base path:** edit `/etc/thermograph.env`, then `sudo systemctl restart thermograph`. Changing the base path usually pairs with a Caddy change (proxy target / redirects) — update `/etc/caddy/Caddyfile` and reload Caddy too. - **Change routing / domains:** edit `/etc/caddy/Caddyfile` (or copy the repo's), `sudo caddy validate --config /etc/caddy/Caddyfile`, then `sudo systemctl reload caddy`. The CI deploy never ships the Caddyfile, so this is always by hand. ## Security notes - App is bound to `127.0.0.1` — not reachable except through Caddy. - Firewall: allow only `22`, `80`, `443` (e.g. `ufw allow 22,80,443/tcp`). - The CI sudoers rule is scoped to exactly `systemctl restart/status thermograph`. - Rotate Key B by regenerating it and updating the `SSH_KEY` secret + `authorized_keys`. ## Accounts & notifications Accounts, subscriptions, and notifications are **authoritative and not regenerable** — back them up. **Prod, beta and dev all keep them in Postgres/TimescaleDB** — prod and beta on the one shared instance on vps2 (as the `thermograph` and `thermograph_beta` databases, separate roles, `CONNECT` revoked from `PUBLIC` — see `deploy/db/provision-env-db.sh`), dev in its own `db` container on vps1. SQLite (`data/accounts.sqlite`) is only a fallback for running the app **outside** any of these stacks (a bare venv, the test suite) when `THERMOGRAPH_DATABASE_URL` isn't a Postgres URL — it's not what any live environment actually uses. - **Back up the Postgres data** (e.g. `pg_dump` the relevant database) as part of each environment's backup routine — losing it wipes all accounts and alerts for that environment. (Only a bare-venv/offline run backs up `data/accounts.sqlite` and its `-wal`/`-shm` sidecars instead.) - **Multiple workers, one notifier (leader election).** Each environment runs several uvicorn workers (`WORKERS`, sized per environment — see `deploy/secrets/{prod,beta,dev}.yaml` and each stack file's replica/worker counts). The subscription evaluator and the recurring scheduler must run in exactly one, so the workers elect a single leader — a host-local `flock` (`THERMOGRAPH_SINGLETON_LOCK`, set by the compose/stack app service to `/app/data/notifier.lock`), or a cluster-wide Postgres advisory lock (`THERMOGRAPH_SINGLETON_PG=1`) if the notifier must be single across *replicas on multiple hosts* — this is what prod's Swarm stack uses, since `web` scales to N replicas under the autoscaler. See `backend/core/singleton.py`. `THERMOGRAPH_ROLE` (`web`/`worker`/`all`, default `all`) further restricts which processes may own it, so a stateless web tier can scale without scaling notifiers. - **Env vars** (all optional, sensible defaults): - `THERMOGRAPH_COOKIE_SECURE` — set to `1` when serving over HTTPS (the VPS/TLS deploy) so the session cookie is `Secure`. Leave unset for plain-HTTP LAN, where a `Secure` cookie would never be sent. - `THERMOGRAPH_SESSION_TTL_DAYS` — login/cookie lifetime (default 30). - `THERMOGRAPH_NOTIFY_INTERVAL` — seconds between evaluation passes (default 900). - `THERMOGRAPH_NOTIFY_ARCHIVE_FETCHES` — max missing-archive fetches per pass (default 4). A newly-subscribed cell with no cached ~45-year archive is fetched once (then only read); this caps how many such fetches one pass may do so a burst of new subscriptions can't exhaust the archive quota. - `THERMOGRAPH_ENABLE_NOTIFIER` — set to `0` to disable the background evaluator. - `THERMOGRAPH_AUTH_SECRET` — signing secret for (future) email reset/verify tokens; unused today, a per-boot random value is used if unset. ### Web Push (VAPID) Push notifications are signed with a VAPID keypair. If `THERMOGRAPH_VAPID_PRIVATE_KEY` / `THERMOGRAPH_VAPID_PUBLIC_KEY` are unset, the app generates a pair into **`data/vapid.json`** on first run. That's fine **only if the file persists** — the key the browser subscribed with must keep matching the key the server signs with. If the keys ever change (regenerated `data/vapid.json`, a non-persistent data dir), every existing subscription silently stops delivering: the push service returns 401/403 and the app records it (`logs/errors/*.jsonl`, tagged `"phase":"push"`) but the `/push/test` call still returns 202 with `sent:0, failed:N`. To be safe, **pin the keys** in `/etc/thermograph.env` (see `deploy/thermograph.env.example` for the generate command). After changing keys, users must toggle alerts off/on to re-subscribe. Diagnose a "test says sent but nothing arrives": check the `/push/test` response (`failed:N` ⇒ delivery rejected) or `tail logs/errors/*.jsonl | grep push`, and `docker compose logs backend | grep -i push` for the exact status code. ## Homepage "unusual right now" feed `backend/api/homepage.py` sweeps the tracked cities, grades each one's latest observation from the parquet cache, and writes the ranked result to `data/homepage.json` for the homepage template. It runs on the notifier's timer (hourly guard) and at the tail of `warm_cities.py`, so a deploy repopulates it. **It spends a small amount of forecast quota, and has to.** Nothing else keeps the recent/forecast cache current for the tracked cities: `warm_cities.py` skips any city whose archive is already cached, so it never re-fetches their forecast bundle, and the notifier only touches cells somebody has subscribed to. Reading that cache without refreshing it made the strip present days-old readings as "right now". Each pass therefore tops up the **stalest** cells first, capped: - `THERMOGRAPH_HOMEPAGE_REFRESH` — recent/forecast fetches per pass (default 40, one per cell). At the hourly cadence that is ~960/day. Set `0` to disable, in which case the strip only ranks cities something else happened to refresh. - `THERMOGRAPH_HOMEPAGE_CITIES` — how many cities the strip ranks over (default 250). The strip shows ~12 cards; ranking over a smaller set that is genuinely fresh beats ranking over every cached city when most of those are stale. Readings older than a day are dropped rather than ranked, so an empty strip means the cache went cold — not that the weather is unremarkable. ## Outbound email The app never talks to a mail provider directly. It speaks plain SMTP to **Postfix running as a send-only null client on `127.0.0.1:25`** (`deploy/provision-mail.sh`, run once as root). `backend/mailer.py` is the only code that sends, over stdlib `smtplib` — no new dependency. Why route through a local MTA rather than a provider's API: - **Delivery policy stays swappable.** Direct-to-MX or relayed through a transactional provider is a Postfix setting; the app's config is just "localhost:25" either way, so switching needs no code change or redeploy. - **Postfix queues and retries.** A handler hands the message off in microseconds; a slow or briefly-down upstream can't stall a request or lose a signup. - **Nothing new is exposed.** `inet_interfaces = loopback-only`, so the box accepts mail from itself and nothing else. A loopback socket is not a filesystem write, so the hardened unit (`ProtectSystem=full`, `ReadWritePaths=…`) needs no change. **Default is safe:** `THERMOGRAPH_MAIL_BACKEND` defaults to `console` — it logs the message and sends nothing — so dev and the test suite exercise the whole signup path with no mail server and no risk of mailing a real person. Prod opts in with `=smtp`; beta stays on `console` too (see `deploy/stack/thermograph-beta-stack.yml`'s header for why beta must never hold live mail or Discord credentials). ### Postfix unit topology — and the check that lies Debian/Ubuntu ships Postfix as **three** systemd units, and only one of them does anything: | Unit | What it is | Does it bind sockets? | |---|---|---| | `postfix.service` | umbrella wrapper: `Type=oneshot`, `ExecStart=/bin/true`, `RemainAfterExit=yes` | **No.** Reports `active (exited)` forever. | | `postfix@.service` | the template all instances are built from | n/a | | `postfix@-.service` | the instance for `/etc/postfix` — the real MTA | **Yes.** This is the one that matters. | > **`systemctl is-active postfix` is not a health check.** It returned `active` > for the entire 13-hour mail outage of 2026-07-24 while `postfix@-.service` was > `failed` and nothing was listening on `:25`. Reproduced deliberately during > supervision testing: with the instance in `failed` and zero listeners, > `is-active postfix` still said `active`. **Always query `postfix@-.service`.** Two consequences worth internalising: - Restart the **instance** (`systemctl restart 'postfix@-'`), not the umbrella, for a `main.cf` change to take effect. - **Postfix's own tools lie in the one state you most need to detect.** Any command that *resolves* `inet_interfaces` — `postmulti`, `postqueue`, `postfix` — fatals when one of the listed addresses is missing from the host. A checker built on them returns empty and reads as "nothing wrong". Use plain `postconf -c /etc/postfix -h inet_interfaces`, which only reads `main.cf`. ### Supervision (`deploy/provision-mail-supervision.sh`) Installed by `provision-mail.sh`, and safe to run on its own — it touches nothing in `main.cf`. Everything it writes lives outside `/opt/thermograph`, so `deploy.sh`'s `git reset --hard` cannot erase it; re-running the script is what makes a rebuilt box match. - **Drop-in** `/etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf` (a drop-in, never an edit to the packaged unit, which apt replaces): orders Postfix `After=docker.service wg-quick@wg0.service`, adds `Restart=on-failure` / `RestartSec=15s`, and bounds retries at `StartLimitBurst=5` / `StartLimitIntervalSec=600`. - **Start gate** `/usr/local/sbin/postfix-wait-interfaces` — an `ExecStartPre` that blocks (≤60s) until every address in `inet_interfaces` actually exists. Ordering makes the boot race unlikely; this makes it deterministic, and it is what makes a *bounded* start limit safe: a late interface costs seconds inside one attempt instead of burning the retry budget. - **Health check** `/usr/local/sbin/postfix-health` — the single definition of "mail works" (see below). - **Watchdog** `/usr/local/sbin/postfix-watchdog` + `postfix-watchdog.timer` — every 5 minutes. `Restart=` gives up after 5 attempts *by design*, so that a genuinely broken config lands in `failed` loudly instead of re-fataling every 15s forever; the watchdog is what stops "failed" meaning "dead until a human notices". It runs `reset-failed` (clearing the start-limit counter, which otherwise makes plain `systemctl start` refuse) and retries, forever, at a cadence 20× slower than `RestartSec`. Net effect: **bounded fast retry for transients, slow unbounded retry for everything else**, worst-case recovery ~5 minutes instead of 13 hours. `sudo touch /etc/postfix/maintenance` stops the watchdog fighting a deliberate `systemctl stop postfix@-`. Remove it when done. ### Monitoring specification For the Grafana/Discord alerting and any future `mail_health` tool. Alerts must **not** route over this box's own SMTP — prod's MTA cannot be the transport for "prod's MTA is down". `postfix-health` is the contract. Exit `0` healthy, `1` unhealthy, and one logfmt line on stdout (LogQL: `| logfmt`): ``` mail_health status=ok unit=postfix@-.service unit_state=active bound=3/3 \ banner=ok queue=2 queue_real=0 oldest_deferred_s=0 reason=none ``` It asserts, in order: the **instance** unit is active; every address in `inet_interfaces` is really bound on `:25`; none of them is **missing from the host** (the latent state that stays invisible until the next reboot and then kills all mail); a live `220` greeting comes back, not merely an open socket; **no public address is bound**; and queue depth/age with null-sender bounces excluded (they are undeliverable by design — this host accepts no inbound mail — and would otherwise hold an alert open forever). Alert on: | Severity | Condition | Why | |---|---|---| | **Page** | `mail_health status=fail` twice consecutively (≥10 min) | mail is down | | **Page** | any `postfix_watchdog action=RECOVERY_FAILED` | self-heal exhausted | | **Page** | **no `postfix_watchdog` line at all for 15 min** | see below | | Warn | `status=fail` where `reason` contains only `address-missing-from-host` | still serving, but the next reboot kills all mail | | Warn | `queue_real > 0` and `oldest_deferred_s > 3600` | mail accepted but not leaving | The **heartbeat-absence** rule is the one that is easy to leave out and the one that would have caught this outage. Every other check is an *active probe* and therefore only reports while something is alive to run it; a dead box, dead timer, or dead log shipper produces silence, and silence renders identically to health on a dashboard. Alert on the absence of the 5-minute heartbeat, or the monitoring inherits the same blind spot the umbrella unit had. > **Blocker for the alerting agent:** none of this reaches Loki today. Alloy > ships Docker container stdout, Caddy files and the app's JSONL — **not the > systemd journal** — and Postfix runs on the host. Until > `observability/alloy/config.alloy` gains a `loki.source.journal` (plus > `/var/log/journal` and `/etc/machine-id` mounted read-only in > `alloy/docker-compose.agent.yml`), no Postfix or watchdog line is queryable > and none of the rules above can fire. That change is untested — validate it > before relying on it. Until then the check is still reachable synchronously > via Centralis `run_on_host` running `postfix-health`, which is also the > natural implementation of a `mail_health` tool. ### Deliverability (do this before mailing real subscribers) Mail from a bare VPS IP is very often junked no matter how Postfix is configured, because the IP has no sending reputation. Either: - **Relay through a provider** (recommended) — `RELAYHOST` + credentials in `provision-mail.sh`. They handle SPF/DKIM alignment and reputation. - **Direct to MX** — then you own the DNS work: an **SPF** TXT record, **DKIM** (opendkim) with the public key published, a **DMARC** TXT record, and a **PTR / reverse-DNS** record on the VPS IP pointing at `mail.thermograph.org`. Missing rDNS alone is enough for Gmail and Outlook to junk everything. Check placement with , which scores all four at once. - **Env vars** (all in `deploy/thermograph.env.example`): `THERMOGRAPH_MAIL_BACKEND` (`console` | `smtp` | `disabled`), `THERMOGRAPH_SMTP_HOST` / `_PORT` / `_USER` / `_PASSWORD` / `_STARTTLS`, `THERMOGRAPH_MAIL_FROM`, `THERMOGRAPH_MAIL_REPLY_TO`. - **`THERMOGRAPH_AUTH_SECRET` must be set to a fixed value** before any confirmation or password-reset link is mailed. It currently defaults to a per-boot random value, so every outstanding link would break on restart. - **Digest signups** land in the `pending_digest` table in the accounts DB (Postgres on every environment; authoritative, back it up). The form ships ahead of delivery on purpose: addresses are collected now and confirmed once SMTP is live. - **Logs:** `journalctl -u postfix -f`; queue with `mailq`. ## SEO content pages Crawlable, server-rendered pages (Jinja2) sit alongside the interactive tool and link into it: `/climate` (hub), `/climate/` (per-city), `/climate//`, `/climate//records`, `/glossary`, `/about`, plus `/robots.txt` and `/sitemap.xml`. The routable city set is `backend/cities.json` (~1000 metros: top ~500 global + ~250 core-English + ~250 extended-English/high-proficiency), regenerated with `python gen_cities.py [n_global] [n_english] [n_extended]`. - A **values filter** in `gen_cities.py` drops cities in countries that criminalize LGBTQ+ people (plus China/Pakistan/Indonesia by choice) via `EXCLUDE_CC`, except a hand-kept list of notable/tourist/high-English hubs in `KEEP_SLUGS` (Lagos, Nairobi, Cairo, Dubai, KL, Shanghai×5-from-China, Karachi, Jakarta, …). Excluded slots are backfilled from the next-ranked non-excluded cities, so the total stays ~1000. Edit either set and regenerate to change the policy. - **Per-city blurbs** live in `backend/cities_flavor.json` (a short Wikipedia summary per city, CC BY-SA, attributed on the page). `python gen_flavor.py` fetches Wikipedia's free REST summary API and validates each match by coordinates; it's **incremental** by default (only fetches cities missing a blurb, and prunes ones no longer in cities.json) — pass `--full` to rebuild. ~94% of cities get a blurb; the rest render without one. - **Notable weather events** are a small **hand-curated** list in `backend/city_events.py` (`{slug: {text, url}}`) — one verified iconic event per city (Katrina/New Orleans, Harvey/Houston, the 1952 Great Smog/London, …). Auto-sourcing these from search proved unreliable (wrong matches), so they're edited by hand; a city not in the list simply shows its flavor blurb. Add entries freely — a test checks the slugs are valid. - **Archive warming runs automatically on every deploy of prod and dev — not beta.** `deploy/stack/deploy-stack.sh` (prod) and `deploy.sh`'s compose path (dev, via `deploy-dev.sh`) launch `warm_cities.py` detached in the background after the health check, so the ~750 city pages serve from cache and a crawl can't burst the archive API quota. Beta deliberately skips this (`TG_POST_DEPLOY=0` in `env-topology.sh`) — it would spend the shared upstream archive quota filling a cache only a rehearsal environment reads. Warming is idempotent (skips already-cached cells), so only the first deploy does the full ~25-min warm; later deploys just top up new cities. A page also self-heals (fetches its archive once) if hit before warming finishes. To run it by hand: `cd backend && python warm_cities.py --pace 2`. Logs land at `logs/warm-cities.log` inside each environment's own backend container (dev is a normal container stack on vps1 now, not a bare systemd unit; there's no `journalctl --user` path any more). - **Submit the sitemap** in Google Search Console: `https://thermograph.org/sitemap.xml`. - `jinja2` is a new dependency (already in `requirements.txt`). ## Monitoring Fleet-wide logs and dashboards live in a **separate project** — a central Grafana + Loki stack, hosted on **vps1** alongside Forgejo, fed by a Grafana Alloy agent on every node over the WireGuard mesh (`thermograph-observability` on Forgejo; UI at **`https://dashboard.thermograph.org`**, Google SSO). It ingests every container's stdout, Caddy's access logs, and the app's structured JSON logs (`logs/{errors,access,audit}/*.jsonl`) — from prod and beta on vps2 as well as dev on vps1, each distinguishable by its `host`/service labels. This replaced the old SSH-tailed `scripts/dashboard.py`. The app still exposes raw counters at the gated `GET /api/v2/metrics` route (loopback-only; refuses any request carrying a proxy `X-Forwarded-*` header, so it's never reachable through Caddy). It reports inbound requests per endpoint, outbound calls per upstream source, and background-daemon heartbeats (e.g. the subscription notifier). Read it over an SSH tunnel with `THERMOGRAPH_METRICS_TOKEN` (set in `/etc/thermograph.env`) as the `token=` query param — handy for a quick check without opening Grafana.