thermograph/infra/DEPLOY.md

# Deploying Thermograph to a prod VPS

> **Prod and beta are now provisioned by Terraform** (`terraform/README.md`) and
> run as a **`docker compose` stack** (backend + frontend + Postgres/TimescaleDB;
> repo-split Stage 4 split the single "app" service in two), not the
> manual venv + systemd model this document originally described. Current shape:
> prod = new box `169.58.46.181`, branch `release`, deployed with `terraform
> apply`; beta = old box `75.119.132.91`, branch `main`, deployed when `main` is
> pushed (`.forgejo/workflows/deploy.yml` SSHes in and runs `deploy/deploy.sh`,
> which `docker compose pull`s the image `build-push.yml` already pushed for
> that commit and `up`s it -- repo-split Stage 6's registry-pull cutover;
> `deploy.sh` no longer builds in place). The **manual walkthrough below
> (`provision.sh`, `thermograph.service`, the venv) is legacy** — kept as a
> from-scratch, no-Terraform reference; the process model is compose, so the
> systemd/venv specifics no longer match how prod/beta actually run.

Pipeline (beta, on `main`): **push to `main` on Forgejo → `build-push.yml` builds
+ pushes the image, tagged by SHA → Forgejo Actions SSHes to beta →
`deploy/deploy.sh` (`git reset` + `docker login` + `docker compose pull && up`,
schema migration runs in the app entrypoint) → Caddy fronts it with Let's Encrypt
TLS.** Prod (on `release`) is deployed with `terraform apply`, not a push
trigger — its `remote-exec` provisioner does the same pull-instead-of-build.
(GitHub is retired/archived — Forgejo, self-hosted at `git.thermograph.org` /
`10.10.0.2:3080` over the WireGuard mesh, is now the sole git host and CI.)

Files that make this work:

| File | Where it lives in prod | Purpose |
|------|------------------------|---------|
| `.forgejo/workflows/deploy.yml` | Forgejo | CI job that SSHes in and runs the deploy script (beta, on `main`) |
| `.forgejo/workflows/build-push.yml` | Forgejo | builds + pushes the SHA-tagged image `deploy.sh`/Terraform pull |
| `deploy/deploy.sh` | `/opt/thermograph/deploy/deploy.sh` | `git reset` + `docker login` + `docker compose pull && up` + health check + warm |
| `docker-compose.yml` | `/opt/thermograph/docker-compose.yml` | app + Postgres/TimescaleDB services (the process model) |
| `deploy/thermograph.env.example` | `/etc/thermograph.env` | Postgres password, VAPID/auth secrets, `WORKERS`, sizing, base path |
| `deploy/Caddyfile` | `/etc/caddy/Caddyfile` | reverse proxy + automatic HTTPS |
| `terraform/` | run from your machine | provisions the box + brings the stack up (replaces `provision.sh`) |
| `deploy/provision.sh`, `deploy/thermograph.service` | *(legacy)* | pre-compose venv+systemd bootstrap — superseded by Terraform |

The app serves on **loopback only**; Caddy is the only thing exposed to the
internet (ports 80/443). The parquet cache in `data/cache/` lives inside the
`/opt/thermograph` checkout and is gitignored, so `git pull` never touches it.

**Any deploy host needs a `/etc/hosts` entry for the registry.**
`git.thermograph.org`'s public DNS resolves to beta's public IP, and beta's
own Caddy rejects `/v2/*` (the registry API) from anything outside the
WireGuard mesh (10.10.0.0/24) — confirmed live: `docker login`/`pull` from
prod failed with a 403 until adding `10.10.0.2 git.thermograph.org` to
`/etc/hosts` (10.10.0.2 is beta's mesh IP; every deploy host is already on
the mesh, this is purely a DNS-routing gap, not a connectivity one). Beta
itself doesn't need this — it's the box Forgejo runs on, so `git.thermograph.org`
just resolves to itself either way. `build-push.yml`'s own header comment
documents the same requirement for the CI runner host.

**Current production layout** (`deploy/Caddyfile`): the app owns
**`thermograph.org`** at its root (`THERMOGRAPH_BASE=/`, so uvicorn serves `/`,
`/calendar`, `/api/v2/…` with no prefix), and **`emigriffith.dev`** serves a
static portfolio at its root with `emigriffith.dev/thermograph*` permanently
redirecting to `thermograph.org` (prefix stripped). Both domains' `A` records
point at the same VPS; Caddy provisions a separate cert for each.

> **Note:** `deploy.sh` (the CI deploy) only pulls code, reinstalls deps, and
> restarts the app service — it does **not** touch Caddy or `/etc/thermograph.env`.
> After changing `deploy/Caddyfile` or the env, copy it onto the VPS and reload
> Caddy / restart the service by hand (see Part 2 step 4 and Part 3).

---

## Part 1 — Keys: how to manage them

You need **two** separate SSH keypairs. Don't reuse one for both.

### Key A — your login key (you ↔ VPS), you probably already have this
This is how *you* SSH in to run `provision.sh`. Nothing to set up here beyond
your existing access.

### Key B — the CI deploy key (Forgejo Actions ↔ VPS), you create this
A dedicated key whose **private half** lives in a Forgejo Actions secret and
whose **public half** is authorized on the VPS's deploy user. It should only
be able to log in as `deploy` and run the deploy script.

Generate it on your machine (no passphrase — CI can't type one):

```bash
ssh-keygen -t ed25519 -C "thermograph-ci" -f ~/.ssh/thermograph_ci -N ""
```

That makes two files:
- `~/.ssh/thermograph_ci`      → **private** key → goes into the Forgejo Actions secret `SSH_KEY`
- `~/.ssh/thermograph_ci.pub`  → **public** key → goes on the VPS

Install the **public** key on the VPS (as the `deploy` user):

```bash
ssh-copy-id -i ~/.ssh/thermograph_ci.pub deploy@YOUR_VPS_IP
# or manually: append the .pub line to /home/deploy/.ssh/authorized_keys
```

Put the **private** key into Forgejo → repo **Settings → Actions → Secrets →
Add Secret**. Set all of these:

| Secret | Value |
|--------|-------|
| `SSH_KEY` | full contents of `~/.ssh/thermograph_ci` (the private key, incl. BEGIN/END lines) |
| `SSH_HOST` | VPS IP or hostname |
| `SSH_USER` | `deploy` |
| `SSH_PORT` | `22` (or your custom SSH port) |

No `tea` CLI is installed for a command-line alternative; use the web UI, or
the REST API with a personal access token:
`POST /api/v1/repos/{owner}/{repo}/actions/secrets/{name}` (body `{"data": "..."}`).

**Why I can't do this for you:** Forgejo secrets are write-only and require
your Forgejo auth; the VPS's `authorized_keys` requires your SSH access. Both
are credentials only you should hold.

### Deploy key for cloning a private repo (optional)
If the Forgejo repo is **private**, the VPS also needs to *pull* from it. Two
options:
- Make `provision.sh`'s `REPO_URL` an **HTTPS** URL with an embedded personal
  access token (what the live VPS checkout actually uses today — see its
  `origin` remote), or
- Add the `deploy` user's own key (`ssh-keygen` on the VPS) as a **read-only
  Deploy Key** in the repo (Settings → Deploy keys). This is separate from Key B.

If the repo is **public**, skip this — `git pull` needs no auth.

---

## Part 2 — Install (once, on the VPS)

1. **Create the Forgejo repo and push** (this project isn't versioned yet):

   ```bash
   cd ~/Code/Thermograph
   git init && git add -A && git commit -m "Initial commit"
   # create an empty private repo on the Forgejo instance first (web UI or
   # POST /api/v1/user/repos), then:
   git remote add origin https://<forgejo-host>/OWNER/thermograph.git
   git push -u origin main
   ```

2. **Point DNS** for each domain you serve (`thermograph.org` for the app,
   `emigriffith.dev` for the portfolio) at the VPS IP with an `A` record. Let's
   Encrypt won't issue a cert until the name resolves to the box.

3. **Provision the box.** Copy `deploy/provision.sh` over (or clone the repo)
   and edit `REPO_URL` at the top, then:

   ```bash
   REPO_URL="https://<forgejo-host>/OWNER/thermograph.git" bash deploy/provision.sh
   ```

   It installs Python + git + Caddy, creates the `deploy` user, clones to
   `/opt/thermograph`, installs the systemd unit + env file, grants the deploy
   user a password-less `systemctl restart thermograph` (so CI can restart it),
   and starts the service.

4. **Install the Caddy config** and reload. `deploy/Caddyfile` already carries the
   real domains (`thermograph.org` + `emigriffith.dev`); edit them if yours differ,
   then copy it into place, validate, and reload:

   ```bash
   sudo cp /opt/thermograph/deploy/Caddyfile /etc/caddy/Caddyfile
   sudo caddy validate --config /etc/caddy/Caddyfile   # syntax check before reload
   sudo systemctl reload caddy      # Caddy fetches each domain's TLS cert automatically
   ```

   Do this again any time `deploy/Caddyfile` changes — the CI deploy does not ship
   it. Set the app's base path to match in `/etc/thermograph.env`
   (`THERMOGRAPH_BASE=/` for the app-at-root layout), then `sudo systemctl restart
   thermograph`.

5. **Install Key B** (Part 1) so CI can log in.

6. **Verify:** open `https://YOUR_DOMAIN/` — you should get a clean padlock.

---

## Part 3 — Day-to-day

- **Deploy:** just `git push` to `main`. Actions runs `deploy.sh` and
  health-checks the service. Watch it under the repo's **Actions** tab on
  Forgejo (or query `/api/v1/repos/{owner}/{repo}/actions/tasks`).
- **Manual deploy / rollback:** `ssh deploy@vps '/opt/thermograph/deploy/deploy.sh'`
  (set `BRANCH=` or check out a tag first to roll back).
- **Logs:** `journalctl -u thermograph -f` (app), `/var/log/caddy/thermograph.log` (proxy).
- **Restart:** `sudo systemctl restart thermograph`.
- **Change port / base path:** edit `/etc/thermograph.env`, then `sudo systemctl
  restart thermograph`. Changing the base path usually pairs with a Caddy change
  (proxy target / redirects) — update `/etc/caddy/Caddyfile` and reload Caddy too.
- **Change routing / domains:** edit `/etc/caddy/Caddyfile` (or copy the repo's),
  `sudo caddy validate --config /etc/caddy/Caddyfile`, then `sudo systemctl reload
  caddy`. The CI deploy never ships the Caddyfile, so this is always by hand.

## Security notes
- App is bound to `127.0.0.1` — not reachable except through Caddy.
- Firewall: allow only `22`, `80`, `443` (e.g. `ufw allow 22,80,443/tcp`).
- The CI sudoers rule is scoped to exactly `systemctl restart/status thermograph`.
- Rotate Key B by regenerating it and updating the `SSH_KEY` secret +
  `authorized_keys`.

## Accounts & notifications

Accounts, subscriptions, and notifications are **authoritative and not
regenerable** — back them up. On **prod/beta they live in Postgres/TimescaleDB**
(the compose `db` service, on the `pgdata` volume) alongside the climate record;
on **LAN dev** they fall back to SQLite (`data/accounts.sqlite`) when
`THERMOGRAPH_DATABASE_URL` isn't a Postgres URL.

- **Back up the Postgres volume** (e.g. `pg_dump` the `thermograph` database) as
  part of the VPS backup routine — losing it wipes all accounts and alerts. (On
  a SQLite LAN dev box, back up `data/accounts.sqlite` and its `-wal`/`-shm`
  sidecars instead.)
- **Multiple workers, one notifier (leader election).** Prod runs several uvicorn
  workers (`WORKERS`, currently 8 on prod / 4 on beta). The subscription
  evaluator and the recurring scheduler must run in exactly one, so the workers
  elect a single leader — a host-local `flock` (`THERMOGRAPH_SINGLETON_LOCK`,
  set by the compose app service to `/app/data/notifier.lock`), or a cluster-wide
  Postgres advisory lock (`THERMOGRAPH_SINGLETON_PG=1`) if the notifier must be
  single across *hosts*. See `backend/core/singleton.py`. `THERMOGRAPH_ROLE`
  (`web`/`worker`/`all`, default `all`) further restricts which processes may own
  it, so a stateless web tier can scale without scaling notifiers.
- **Env vars** (all optional, sensible defaults):
  - `THERMOGRAPH_COOKIE_SECURE` — set to `1` when serving over HTTPS (the VPS/TLS
    deploy) so the session cookie is `Secure`. Leave unset for plain-HTTP LAN, where
    a `Secure` cookie would never be sent.
  - `THERMOGRAPH_SESSION_TTL_DAYS` — login/cookie lifetime (default 30).
  - `THERMOGRAPH_NOTIFY_INTERVAL` — seconds between evaluation passes (default 900).
  - `THERMOGRAPH_NOTIFY_ARCHIVE_FETCHES` — max missing-archive fetches per pass
    (default 4). A newly-subscribed cell with no cached ~45-year archive is fetched
    once (then only read); this caps how many such fetches one pass may do so a burst
    of new subscriptions can't exhaust the archive quota.
  - `THERMOGRAPH_ENABLE_NOTIFIER` — set to `0` to disable the background evaluator.
  - `THERMOGRAPH_AUTH_SECRET` — signing secret for (future) email reset/verify
    tokens; unused today, a per-boot random value is used if unset.

### Web Push (VAPID)

Push notifications are signed with a VAPID keypair. If `THERMOGRAPH_VAPID_PRIVATE_KEY`
/ `THERMOGRAPH_VAPID_PUBLIC_KEY` are unset, the app generates a pair into
**`data/vapid.json`** on first run. That's fine **only if the file persists** — the
key the browser subscribed with must keep matching the key the server signs with. If
the keys ever change (regenerated `data/vapid.json`, a non-persistent data dir), every
existing subscription silently stops delivering: the push service returns 401/403 and
the app records it (`logs/errors/*.jsonl`, tagged `"phase":"push"`) but the `/push/test`
call still returns 202 with `sent:0, failed:N`. To be safe, **pin the keys** in
`/etc/thermograph.env` (see `deploy/thermograph.env.example` for the generate command).
After changing keys, users must toggle alerts off/on to re-subscribe.

Diagnose a "test says sent but nothing arrives": check the `/push/test` response
(`failed:N` ⇒ delivery rejected) or `tail logs/errors/*.jsonl | grep push`, and
`docker compose logs backend | grep -i push` for the exact status code.

## Homepage "unusual right now" feed

`backend/api/homepage.py` sweeps the tracked cities, grades each one's latest
observation from the parquet cache, and writes the ranked result to
`data/homepage.json` for the homepage template. It runs on the notifier's timer
(hourly guard) and at the tail of `warm_cities.py`, so a deploy repopulates it.

**It spends a small amount of forecast quota, and has to.** Nothing else keeps
the recent/forecast cache current for the tracked cities: `warm_cities.py` skips
any city whose archive is already cached, so it never re-fetches their forecast
bundle, and the notifier only touches cells somebody has subscribed to. Reading
that cache without refreshing it made the strip present days-old readings as
"right now". Each pass therefore tops up the **stalest** cells first, capped:

- `THERMOGRAPH_HOMEPAGE_REFRESH` — recent/forecast fetches per pass (default 40,
  one per cell). At the hourly cadence that is ~960/day. Set `0` to disable, in
  which case the strip only ranks cities something else happened to refresh.
- `THERMOGRAPH_HOMEPAGE_CITIES` — how many cities the strip ranks over (default
  250). The strip shows ~12 cards; ranking over a smaller set that is genuinely
  fresh beats ranking over every cached city when most of those are stale.

Readings older than a day are dropped rather than ranked, so an empty strip means
the cache went cold — not that the weather is unremarkable.

## Outbound email

The app never talks to a mail provider directly. It speaks plain SMTP to
**Postfix running as a send-only null client on `127.0.0.1:25`**
(`deploy/provision-mail.sh`, run once as root). `backend/mailer.py` is the only
code that sends, over stdlib `smtplib` — no new dependency.

Why route through a local MTA rather than a provider's API:

- **Delivery policy stays swappable.** Direct-to-MX or relayed through a
  transactional provider is a Postfix setting; the app's config is just
  "localhost:25" either way, so switching needs no code change or redeploy.
- **Postfix queues and retries.** A handler hands the message off in
  microseconds; a slow or briefly-down upstream can't stall a request or lose a
  signup.
- **Nothing new is exposed.** `inet_interfaces = loopback-only`, so the box
  accepts mail from itself and nothing else. A loopback socket is not a
  filesystem write, so the hardened unit (`ProtectSystem=full`,
  `ReadWritePaths=…`) needs no change.

**Default is safe:** `THERMOGRAPH_MAIL_BACKEND` defaults to `console` — it logs
the message and sends nothing — so LAN dev and the test suite exercise the whole
signup path with no mail server and no risk of mailing a real person. Production
opts in with `=smtp`.

### Deliverability (do this before mailing real subscribers)

Mail from a bare VPS IP is very often junked no matter how Postfix is configured,
because the IP has no sending reputation. Either:

- **Relay through a provider** (recommended) — `RELAYHOST` + credentials in
  `provision-mail.sh`. They handle SPF/DKIM alignment and reputation.
- **Direct to MX** — then you own the DNS work: an **SPF** TXT record, **DKIM**
  (opendkim) with the public key published, a **DMARC** TXT record, and a
  **PTR / reverse-DNS** record on the VPS IP pointing at `mail.thermograph.org`.
  Missing rDNS alone is enough for Gmail and Outlook to junk everything.

Check placement with <https://www.mail-tester.com>, which scores all four at once.

- **Env vars** (all in `deploy/thermograph.env.example`):
  `THERMOGRAPH_MAIL_BACKEND` (`console` | `smtp` | `disabled`),
  `THERMOGRAPH_SMTP_HOST` / `_PORT` / `_USER` / `_PASSWORD` / `_STARTTLS`,
  `THERMOGRAPH_MAIL_FROM`, `THERMOGRAPH_MAIL_REPLY_TO`.
- **`THERMOGRAPH_AUTH_SECRET` must be set to a fixed value** before any
  confirmation or password-reset link is mailed. It currently defaults to a
  per-boot random value, so every outstanding link would break on restart.
- **Digest signups** land in the `pending_digest` table in the accounts DB
  (Postgres on prod/beta; authoritative, back it up). The form ships ahead of
  delivery on purpose:
  addresses are collected now and confirmed once SMTP is live.
- **Logs:** `journalctl -u postfix -f`; queue with `mailq`.

## SEO content pages

Crawlable, server-rendered pages (Jinja2) sit alongside the interactive tool and
link into it: `/climate` (hub), `/climate/<slug>` (per-city), `/climate/<slug>/<month>`,
`/climate/<slug>/records`, `/glossary`, `/about`, plus `/robots.txt` and
`/sitemap.xml`. The routable city set is `backend/cities.json` (~1000 metros: top
~500 global + ~250 core-English + ~250 extended-English/high-proficiency),
regenerated with `python gen_cities.py [n_global] [n_english] [n_extended]`.

- A **values filter** in `gen_cities.py` drops cities in countries that criminalize
  LGBTQ+ people (plus China/Pakistan/Indonesia by choice) via `EXCLUDE_CC`, except a
  hand-kept list of notable/tourist/high-English hubs in `KEEP_SLUGS` (Lagos, Nairobi,
  Cairo, Dubai, KL, Shanghai×5-from-China, Karachi, Jakarta, …). Excluded slots are
  backfilled from the next-ranked non-excluded cities, so the total stays ~1000. Edit
  either set and regenerate to change the policy.

- **Per-city blurbs** live in `backend/cities_flavor.json` (a short Wikipedia summary
  per city, CC BY-SA, attributed on the page). `python gen_flavor.py` fetches
  Wikipedia's free REST summary API and validates each match by coordinates; it's
  **incremental** by default (only fetches cities missing a blurb, and prunes ones no
  longer in cities.json) — pass `--full` to rebuild. ~94% of cities get a blurb; the
  rest render without one.
- **Notable weather events** are a small **hand-curated** list in `backend/city_events.py`
  (`{slug: {text, url}}`) — one verified iconic event per city (Katrina/New Orleans,
  Harvey/Houston, the 1952 Great Smog/London, …). Auto-sourcing these from search
  proved unreliable (wrong matches), so they're edited by hand; a city not in the list
  simply shows its flavor blurb. Add entries freely — a test checks the slugs are valid.

- **Archive warming runs automatically on every deploy.** `deploy.sh` (prod) and
  `deploy-dev.sh` (dev) launch `warm_cities.py` detached in the background after the
  health check, so the ~750 city pages serve from cache and a crawl can't burst the
  archive API quota. It's idempotent (skips already-cached cells), so only the first
  deploy does the full ~25-min warm; later deploys just top up new cities. A page
  also self-heals (fetches its archive once) if hit before warming finishes. To run
  it by hand: `cd backend && python warm_cities.py --pace 2`. Logs: prod
  `logs/warm-cities.log`; dev `journalctl --user -u thermograph-warm-cities`.
- **Submit the sitemap** in Google Search Console: `https://thermograph.org/sitemap.xml`.
- `jinja2` is a new dependency (already in `requirements.txt`).

## Monitoring

Fleet-wide logs and dashboards live in a **separate project** — a central
Grafana + Loki stack fed by a Grafana Alloy agent on every node, over the
WireGuard mesh (`thermograph-observability` on Forgejo; UI at
**`https://dashboard.thermograph.org`**, Google SSO). It ingests every
container's stdout, Caddy's access logs, and the app's structured JSON logs
(`logs/{errors,access,audit}/*.jsonl`). This replaced the old SSH-tailed
`scripts/dashboard.py`.

The app still exposes raw counters at the gated `GET /api/v2/metrics` route
(loopback-only; refuses any request carrying a proxy `X-Forwarded-*` header, so
it's never reachable through Caddy). It reports inbound requests per endpoint,
outbound calls per upstream source, and background-daemon heartbeats
(e.g. the subscription notifier). Read it over an SSH tunnel with
`THERMOGRAPH_METRICS_TOKEN` (set in `/etc/thermograph.env`) as the `token=`
query param — handy for a quick check without opening Grafana.