diff --git a/deploy/POSTGRES-MIGRATION.md b/deploy/POSTGRES-MIGRATION.md new file mode 100644 index 0000000..26e25d3 --- /dev/null +++ b/deploy/POSTGRES-MIGRATION.md @@ -0,0 +1,73 @@ +# Cutover: SQLite → PostgreSQL 18 (containerized stack) + +The app now runs as a docker-compose stack (`app` + `db`) and standardizes on +PostgreSQL. Locally you need nothing but Docker; `make up` builds and starts both. +This doc is the **one-time production cutover** from the old bare-systemd/SQLite +deploy to the compose stack, migrating the authoritative accounts data. + +## What moved + +- **accounts** (users/subscriptions/notifications/…): SQLite → Postgres **durable** + tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB. +- **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres + **UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to + migrate — the cache rebuilds from parquet, metrics start empty. +- The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg` + URL; unset ⇒ the old SQLite behavior (this is how the **test suite** stays on + SQLite — no Postgres needed in CI). +- `db` runs **pg_duckdb** (`deploy/db/Dockerfile.db`, genuine PG18): the app's + parquet cache is mounted read-only at `/parquet`, so you can query it directly, + e.g. `SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r`. + +## Connection model + +Per worker (4 workers): a read-write asyncpg engine + a read-only asyncpg engine +(`default_transaction_read_only=on`, used by the pure-GET endpoints), plus one sync +psycopg engine for the notifier thread. Single Postgres primary — the RO engine is +a guardrail, not a replica. + +## Prod cutover steps (maintenance window) + +Prereq: the deploy user can run `docker`. Secrets in `/etc/thermograph.env` must +include `POSTGRES_PASSWORD`, and pinned `THERMOGRAPH_AUTH_SECRET`, +`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY`, `THERMOGRAPH_COOKIE_SECURE=1`. + +1. **Ship the stack, don't cut over yet.** Deploy the branch; `docker compose build`. + Bring up only the DB: `docker compose up -d db`. Add a nightly `pg_dump` backup. +2. **Freeze + back up.** Stop the old app (hard stop — no writes). Back up the live + `data/accounts.sqlite` + `-wal`/`-shm` (this is the rollback artifact). +3. **Build the schema.** `docker compose run --rm app alembic upgrade head` + (or let the `app` entrypoint do it on first start — but do the data copy before + real traffic). +4. **Copy the accounts data:** + ``` + docker compose run --rm \ + -e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:$POSTGRES_PASSWORD@db:5432/thermograph" \ + -v /opt/thermograph/data/accounts.sqlite:/src.sqlite:ro \ + app python migrate_accounts_to_pg.py --sqlite /src.sqlite + ``` + It copies user → subscription/push_subscription/pending_digest → notification, + **preserving PKs**, **skips access_token** (everyone re-logins once), and + **resets the integer-PK sequences** (skipping that = a runtime PK collision). + Verify the printed per-table counts against the old DB. +5. **Start serving:** `docker compose up -d` (Caddy already proxies `127.0.0.1:8137`). +6. **Smoke test:** login, list/create/delete a subscription, one notifier tick, a + cached calendar/day request (derived store on PG), the metrics dashboard. +7. Keep the frozen `accounts.sqlite` as rollback for a few days; keep PG backed up. + Enable pg_duckdb on the (already-initialized) volume once, by hand: + `docker compose exec db psql -U thermograph -d thermograph -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'` + +## Rollback + +Redeploy the previous SQLite release (or point `THERMOGRAPH_DATABASE_URL` back to +unset/SQLite and restart). **Clean only during/immediately after the window** — once +real users write to Postgres, those writes are lost on rollback (the copy is +one-way), so keep the window short and writes frozen during the copy. + +## Notes + +- The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres + (they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB). +- The PG18 image's data dir is the versioned `…/18/docker` subdir; the compose + volume is mounted at the parent `/var/lib/postgresql` so it persists without the + initdb permission issue that pinning `PGDATA` to the mountpoint can trigger. diff --git a/deploy/db/Dockerfile.db b/deploy/db/Dockerfile.db new file mode 100644 index 0000000..d9951b4 --- /dev/null +++ b/deploy/db/Dockerfile.db @@ -0,0 +1,25 @@ +# Thermograph database image: PostgreSQL 18 with pg_duckdb. +# +# pg_duckdb embeds DuckDB inside Postgres, which lets the DB container read the +# app's Parquet climate cache directly for ad-hoc analytics: +# +# SELECT * FROM read_parquet('/parquet/1026_-2857.parquet'); +# +# Why FROM the official pg_duckdb image instead of `FROM postgres:18` + compile: +# pg_duckdb links a full DuckDB build, so compiling it from source in this +# Dockerfile would mean pulling the DuckDB toolchain and a long, fragile build. +# The maintainers (duckdb/pg_duckdb, MotherDuck) publish an official PG18 image +# that is genuine PostgreSQL 18.1 on Debian 12 bookworm -- the exact same base +# as the official `postgres:18` image, using the standard postgres +# docker-entrypoint.sh. So POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB / +# PGDATA / /docker-entrypoint-initdb.d / pg_isready all behave identically to +# `postgres:18`; this is a drop-in replacement for the compose `db` service. +# +# Pinned to a specific patch tag (not 18-main) for reproducible builds. +FROM pgduckdb/pgduckdb:18-v1.1.1 + +# Bake the parquet init script so the image enables the extension on first init +# even without the compose bind mount. It is CREATE EXTENSION IF NOT EXISTS, so +# it is idempotent with the base image's own 0001-install-pg_duckdb.sql. +# Build context is the repo root (see deploy/db/README.md for the compose snippet). +COPY deploy/db/init/ /docker-entrypoint-initdb.d/ diff --git a/deploy/db/README.md b/deploy/db/README.md new file mode 100644 index 0000000..905ab54 --- /dev/null +++ b/deploy/db/README.md @@ -0,0 +1,171 @@ +# Thermograph DB image — Postgres 18 + parquet reads + +The `db` service can read the app's Parquet climate cache +(`data/cache/*.parquet`) directly from SQL, for ad-hoc analytics, via +**pg_duckdb** — DuckDB embedded inside Postgres. + +```sql +SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin +FROM read_parquet('/parquet/1026_-2857.parquet') r +ORDER BY r['date'] LIMIT 5; +``` + +## Chosen extension: pg_duckdb — and why + +| Option | PG18? | Fit | Verdict | +| --- | --- | --- | --- | +| **pg_duckdb** (duckdb / MotherDuck) | **Yes** — official image `pgduckdb/pgduckdb:18-v1.1.1` is genuine PG18.1 | `read_parquet('…')` in plain SQL; globs, `union_by_name`, full DuckDB analytics engine | **Chosen** | +| pg_parquet (Crunchy Data) | Yes (14–18) | `COPY … TO/FROM '…' (format 'parquet')` — import/export, not a query engine | Viable, but COPY-oriented; no standalone official image (ships via Crunchy Bridge/CPK), so it'd need a Rust/pgrx source build | +| parquet_fdw | No prebuilt PG18 support; low activity | Foreign tables over parquet | Rejected — oldest, weakest PG18 story | + +pg_duckdb wins for the stated goal (ad-hoc analytics): it exposes DuckDB's +`read_parquet` directly in SQL, so you query cache files like tables — no +import step, no foreign-table DDL, and you get aggregation/joins/globs across +many cells at once. + +## PG-version reality for PG18 (empirically verified 2026-07-19) + +**No version delta.** PG18 support is real, not a fallback. The pulled image +reports: + +``` +PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu +pg_extension: pg_duckdb 1.1.0 +shared_preload_libraries: pg_duckdb +``` + +The image is built on Debian 12 bookworm — the **same base as the official +`postgres:18` image** — and uses the standard `docker-entrypoint.sh`. So +`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` / `PGDATA` / +`/docker-entrypoint-initdb.d` / `pg_isready` all behave exactly as with +`postgres:18`. It is a drop-in replacement for the `db` service; nothing else +in the stack changes. + +We `FROM` the official pg_duckdb image (pinned to `18-v1.1.1`, not `18-main`) +rather than `FROM postgres:18` + compile, because pg_duckdb links a full DuckDB +build — compiling from source in the Dockerfile means the DuckDB toolchain and +a long, fragile build for no benefit over the maintainers' official PG18 image. + +## Files here + +- **`Dockerfile.db`** — `FROM pgduckdb/pgduckdb:18-v1.1.1`, plus `COPY` of the + init script so the image enables the extension on first init even without the + compose bind mount. +- **`init/10-parquet.sql`** — `CREATE EXTENSION IF NOT EXISTS pg_duckdb;` (runs + from `/docker-entrypoint-initdb.d` on first cluster init). + +## Compose snippet to merge into the `db` service + +Replace `image: postgres:18` with the `build:` block; add the read-only parquet +bind and the init mount. Everything else in the `db` service stays as-is. + +```yaml + db: + # image: postgres:18 # <- remove; build the parquet-capable image + build: + context: . + dockerfile: deploy/db/Dockerfile.db + environment: + POSTGRES_USER: thermograph + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD} + POSTGRES_DB: thermograph + PGDATA: /var/lib/postgresql/data + volumes: + - pgdata:/var/lib/postgresql/data + - ./data/cache:/parquet:ro # read-only parquet cache + - ./deploy/db/init:/docker-entrypoint-initdb.d + # healthcheck / cpus / deploy / restart: unchanged +``` + +Notes: +- The bind mount at `/docker-entrypoint-initdb.d` **replaces** the base image's + own init scripts (its `0001-install-pg_duckdb.sql` and the MotherDuck-only + `0002-enable-md-pg_duckdb.sql`). That's intended: our `10-parquet.sql` still + runs `CREATE EXTENSION`, and we don't use MotherDuck. If you prefer to keep + the image's baked scripts, drop the `:/docker-entrypoint-initdb.d` line — the + `Dockerfile` already bakes `10-parquet.sql` in. +- `:ro` keeps the DB from ever mutating the app's cache. In prod the app writes + the cache to the `appdata` volume; point this bind at wherever that lives on + the host if you want the DB to see the live cache rather than the repo copy. +- Init scripts only run when `PGDATA` is empty. On an **existing** database, + enable it once by hand: + ``` + docker compose exec db psql -U thermograph -d thermograph \ + -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;' + ``` + +## Usage — real cache file at `/parquet/…` + +Cache columns: `date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`. +pg_duckdb ≥ 0.3 uses the `r['colname']` subscript syntax (not +`AS t(col type, …)`). + +```sql +-- one cell, first rows +SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, r['precip'] AS precip +FROM read_parquet('/parquet/1026_-2857.parquet') r +ORDER BY r['date'] LIMIT 5; + +-- per-year analytics over one cell +SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr, + ROUND(AVG(r['tmax'])::numeric, 1) AS avg_tmax, + MAX(r['tmax']) AS record_high +FROM read_parquet('/parquet/1026_-2857.parquet') r +WHERE r['date'] >= '2020-01-01' +GROUP BY yr ORDER BY yr; + +-- glob across every cached cell (union_by_name handles the _rf / _forecast +-- files whose column sets differ) +SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r; +``` + +## Proof of work (verified 2026-07-19) + +Built `deploy/db/Dockerfile.db`, ran a throwaway container with the real +`data/cache` bind-mounted read-only at `/parquet`, then: + +``` +$ psql -c "SELECT version();" + PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu ... + +$ psql -c "SELECT extname, extversion FROM pg_extension WHERE extname='pg_duckdb';" + extname | extversion +-----------+------------ + pg_duckdb | 1.1.0 + +$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/1026_-2857.parquet') r;" + row_count +----------- + 16982 + +$ psql -c "SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, + r['precip'] AS precip, r['humid'] AS humid + FROM read_parquet('/parquet/1026_-2857.parquet') r + ORDER BY r['date'] LIMIT 5;" + date | tmax | tmin | precip | humid +---------------------+------+------+--------+------- + 1980-01-01 00:00:00 | 56.2 | 34.7 | 0 | 73 + 1980-01-02 00:00:00 | 63.8 | 38 | 0 | 79 + 1980-01-03 00:00:00 | 60.1 | 46.1 | 0.315 | 83 + 1980-01-04 00:00:00 | 51.7 | 40 | 0 | 68 + 1980-01-05 00:00:00 | 56.5 | 33.7 | 0 | 78 + +$ psql -c "SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr, + ROUND(AVG(r['tmax'])::numeric,1) AS avg_tmax, MAX(r['tmax']) AS record_high + FROM read_parquet('/parquet/1026_-2857.parquet') r + WHERE r['date'] >= '2020-01-01' GROUP BY yr ORDER BY yr;" + yr | avg_tmax | record_high +------+----------+------------- + 2020 | 78.8 | 99 + 2021 | 77.0 | 93 + 2022 | 79.2 | 101.4 + 2023 | 80.8 | 107.1 + 2024 | 79.8 | 97.5 + 2025 | 80.0 | 99.4 + 2026 | 77.8 | 95.6 + +$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;" + count +-------- + 525421 -- rows across all 45 cached cells +``` diff --git a/deploy/db/init/10-parquet.sql b/deploy/db/init/10-parquet.sql new file mode 100644 index 0000000..d9e16be --- /dev/null +++ b/deploy/db/init/10-parquet.sql @@ -0,0 +1,15 @@ +-- Enable pg_duckdb so the database can read the app's Parquet climate cache +-- directly (ad-hoc analytics via DuckDB's read_parquet / read_csv / etc.). +-- +-- This runs once, on first cluster init (empty PGDATA), from +-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data +-- on a persistent named volume, init scripts do NOT re-run on an existing +-- database -- to enable pg_duckdb on a DB that was created before this image, +-- run it by hand: +-- +-- docker compose exec db psql -U thermograph -d thermograph \ +-- -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;' +-- +-- pg_duckdb requires shared_preload_libraries='pg_duckdb'; the base image's +-- postgresql.conf.sample already sets it, so it is active on fresh init. +CREATE EXTENSION IF NOT EXISTS pg_duckdb; diff --git a/deploy/deploy.sh b/deploy/deploy.sh index 53d9515..b63bdac 100755 --- a/deploy/deploy.sh +++ b/deploy/deploy.sh @@ -1,66 +1,58 @@ #!/usr/bin/env bash -# Pull the latest code and restart Thermograph. Run on the VPS — the GitHub -# Actions workflow invokes this over SSH, and you can run it by hand too. +# Pull the latest code and roll the docker-compose stack (app + PostgreSQL). Run +# on the VPS — the GitHub Actions workflow invokes this over SSH, and you can run +# it by hand too. # # ssh deploy@vps '/opt/thermograph/deploy/deploy.sh' set -euo pipefail APP_DIR="${APP_DIR:-/opt/thermograph}" BRANCH="${BRANCH:-main}" +HEALTH_PORT="${HEALTH_PORT:-8137}" cd "$APP_DIR" +# Secrets (POSTGRES_PASSWORD, VAPID keys, AUTH_SECRET, ...) drive compose +# interpolation and are also loaded into the app container via env_file. Source +# them here so a by-hand run interpolates the same as the systemd unit does. +set -a; . /etc/thermograph.env 2>/dev/null || true; set +a + # Pre-warm the ~750 city-page archives so /climate pages serve from cache and a -# search-engine crawl never bursts the archive API quota. Detached + backgrounded -# so it never blocks the deploy or health check; idempotent (skips already-cached -# cells), so it's cheap on every deploy after the first full warm. +# search-engine crawl never bursts the archive API quota. Detached inside the app +# container (compose exec -d), idempotent (skips already-cached cells), so it +# never blocks the deploy or health check and is cheap on every deploy after the +# first full warm. warm_city_archives() { - mkdir -p "$APP_DIR/logs" - echo "==> Warming city-page archives in the background (logs/warm-cities.log)" - setsid nohup bash -c "cd '$APP_DIR/backend' && exec '$APP_DIR/.venv/bin/python' warm_cities.py --pace 2" \ - "$APP_DIR/logs/warm-cities.log" 2>&1 & + echo "==> Warming city-page archives in the background (app:/app/logs/warm-cities.log)" + docker compose exec -d app sh -c \ + 'python warm_cities.py --pace 2 >> /app/logs/warm-cities.log 2>&1' || true } # Notify IndexNow (Bing / DuckDuckGo / Yandex) of the site's URLs, but only when -# the set of pages actually changed (a new/removed city) — code-only deploys skip, -# so we don't re-blast ~14k URLs every push. Best-effort: never fails the deploy. -# Sourcing the env matches the key the running service serves at /{key}.txt. +# the set of pages actually changed (a new/removed city) — code-only deploys skip. +# Best-effort: never fails the deploy. ping_indexnow() { echo "==> Pinging IndexNow (only if the URL set changed)" - ( set -a; . /etc/thermograph.env 2>/dev/null || true; set +a - base="${THERMOGRAPH_BASE_URL:-https://thermograph.org}" - cd "$APP_DIR/backend" && "$APP_DIR/.venv/bin/python" indexnow.py --if-changed "$base" - ) || echo "!! IndexNow ping failed (non-fatal)" >&2 + local base="${THERMOGRAPH_BASE_URL:-https://thermograph.org}" + docker compose exec -T app python indexnow.py --if-changed "$base" \ + || echo "!! IndexNow ping failed (non-fatal)" >&2 } echo "==> Fetching $BRANCH" git fetch --prune origin "$BRANCH" git reset --hard "origin/$BRANCH" -echo "==> Installing dependencies" -if [ ! -d .venv ]; then - python3 -m venv .venv -fi -.venv/bin/pip install --upgrade pip -q -.venv/bin/pip install -r backend/requirements.txt -q +echo "==> Building images" +docker compose build -# Apply any pending accounts-DB schema migrations before the new code starts, so -# it never queries a column an older database lacks. Idempotent and tracked, so -# every deploy can run it; sourcing the env picks up a THERMOGRAPH_ACCOUNTS_DB -# override if one is set. Runs as the deploy user, who owns the data dir. -echo "==> Applying database migrations" -( set -a; . /etc/thermograph.env 2>/dev/null || true; set +a - "$APP_DIR/.venv/bin/python" "$APP_DIR/deploy/migrate-db.py" -) - -echo "==> Restarting service" -sudo systemctl restart thermograph +# Schema migrations run inside the app container's entrypoint (alembic upgrade +# head) before uvicorn starts, so there's no separate migrate step or service +# restart here — compose owns the process model. +echo "==> Starting stack" +docker compose up -d echo "==> Health check" -PORT="$(sed -n 's/^PORT=//p' /etc/thermograph.env)"; PORT="${PORT:-8137}" -BASE="$(sed -n 's/^THERMOGRAPH_BASE=//p' /etc/thermograph.env)"; BASE="${BASE:-/}" -# Normalize: no double slash, allow root. -url="http://127.0.0.1:${PORT}${BASE%/}/" -for i in $(seq 1 15); do +url="http://127.0.0.1:${HEALTH_PORT}/" +for i in $(seq 1 30); do if curl -fsS -o /dev/null "$url"; then echo "==> OK: $url is serving" warm_city_archives @@ -70,5 +62,6 @@ for i in $(seq 1 15); do sleep 1 done echo "!! Health check failed for $url" >&2 -sudo systemctl status thermograph --no-pager -l || true +docker compose ps || true +docker compose logs --tail=50 app || true exit 1 diff --git a/deploy/thermograph.env.example b/deploy/thermograph.env.example index de7e1ee..d2b4f9e 100644 --- a/deploy/thermograph.env.example +++ b/deploy/thermograph.env.example @@ -1,17 +1,44 @@ # Copy to /etc/thermograph.env on the VPS and edit. -# Read by the systemd unit (EnvironmentFile). +# Read by the systemd unit (EnvironmentFile) so `docker compose up` can interpolate +# it, AND loaded into the app container (env_file in docker-compose.yml). Anything +# secret the app needs — Postgres password, VAPID keys, auth secret — belongs here. -# Port uvicorn binds on loopback. Caddy proxies to this. Keep 8137 unless it clashes. +# Port uvicorn binds inside the container. The compose stack publishes it on the +# host loopback (127.0.0.1:8137) for Caddy to proxy to. Keep 8137. PORT=8137 +# --- PostgreSQL (docker-compose stack) ------------------------------------------ +# The app and Postgres run as a docker-compose stack (see docker-compose.yml). +# POSTGRES_PASSWORD is the database password: compose uses it to initialize the +# postgres container AND to build the app's THERMOGRAPH_DATABASE_URL. It MUST be +# set here (the systemd unit sources this file so `docker compose up` can +# interpolate it). Change it from the default before the first `up`. +POSTGRES_PASSWORD=change-me + +# The app's compose service already builds THERMOGRAPH_DATABASE_URL from +# POSTGRES_PASSWORD, so you normally DON'T need this. It's here for reference and +# for running the app outside compose against the same DB (keep the password in +# sync with POSTGRES_PASSWORD above). +#THERMOGRAPH_DATABASE_URL=postgresql+asyncpg://thermograph:change-me@db:5432/thermograph + +# Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 in prod; +# leave unset only for plain-HTTP LAN dev (a Secure cookie is never sent over HTTP). +THERMOGRAPH_COOKIE_SECURE=1 + +# Pin these too (see their own sections below), so container restarts don't rotate +# them: THERMOGRAPH_AUTH_SECRET (else every emailed confirm/reset link breaks on +# restart) and THERMOGRAPH_VAPID_PRIVATE_KEY / _PUBLIC_KEY (else every existing push +# subscription silently stops delivering). The data dir persists on the appdata +# volume, but pinning here is the safe default. + # Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch # (e.g. a cache-miss weather lookup) from blocking every other request — the cause of -# past brief outages. Prod runs 3; leave unset (defaults to 1) on a small box. Workers -# share one metrics store (THERMOGRAPH_METRICS_DB, defaulted in the systemd unit) so the -# ops dashboard still sees the whole picture, and elect one leader for the subscription -# notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, also defaulted in the unit) so its -# timer-driven upstream sweep runs once, not once per worker. ~200 MB RAM per worker. -WORKERS=3 +# past brief outages. Prod runs 4 (the compose app service also defaults to WORKERS=4); +# leave unset (defaults to 1) on a small box. Workers elect one leader for the +# subscription notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, set by the compose +# app service to /app/data/notifier.lock) so its timer-driven upstream sweep runs once, +# not once per worker. ~200 MB RAM per worker. +WORKERS=4 # Base path the app is served under. # / -> app at the domain root (Thermograph owns the whole domain — diff --git a/deploy/thermograph.service b/deploy/thermograph.service index 58c1bd4..e305489 100644 --- a/deploy/thermograph.service +++ b/deploy/thermograph.service @@ -1,41 +1,22 @@ [Unit] -Description=Thermograph (FastAPI/uvicorn) -After=network-online.target +Description=Thermograph docker-compose stack (app + PostgreSQL) +# Docker must be up, and the network online, before compose can pull/interpolate. +After=docker.service network-online.target Wants=network-online.target +Requires=docker.service [Service] -Type=exec -User=deploy -Group=deploy -# The repo checkout. uvicorn is run from backend/ so `app:app` resolves. -WorkingDirectory=/opt/thermograph/backend -# Defaults; /etc/thermograph.env (below) overrides. WORKERS is how many uvicorn -# worker processes to run — prod sets WORKERS=3 there; a single-worker box can leave -# it. Multiple workers each keep their own in-process state, so metrics go to a shared -# SQLite DB (THERMOGRAPH_METRICS_DB) that every worker tallies into — otherwise the ops -# dashboard, polling one random worker, would see only a fraction of the traffic. -Environment=WORKERS=1 -Environment=THERMOGRAPH_METRICS_DB=/opt/thermograph/data/metrics.db -# Elect one worker to run the subscription notifier (its timer-driven upstream sweep -# must run once across the deploy, not once per worker — else it multiplies the -# Open-Meteo quota use). Workers race for this lockfile; the winner runs the notifier. -Environment=THERMOGRAPH_SINGLETON_LOCK=/opt/thermograph/data/notifier.lock -# THERMOGRAPH_BASE, PORT, WORKERS, etc. See deploy/thermograph.env.example. -EnvironmentFile=/etc/thermograph.env -# Clear the shared metrics DB on each (re)start so the dashboard's "since start" tallies -# reflect the running workers, matching how the old in-process counters behaved. -ExecStartPre=/usr/bin/rm -f /opt/thermograph/data/metrics.db /opt/thermograph/data/metrics.db-wal /opt/thermograph/data/metrics.db-shm -# Bind loopback only — Caddy terminates TLS and reverse-proxies to us. -ExecStart=/opt/thermograph/.venv/bin/uvicorn app:app --host 127.0.0.1 --port ${PORT} --workers ${WORKERS} -Restart=on-failure -RestartSec=2 -# Hardening -NoNewPrivileges=true -PrivateTmp=true -ProtectSystem=full -ProtectHome=read-only -# The parquet cache must stay writable across deploys. -ReadWritePaths=/opt/thermograph/data /opt/thermograph/logs +# A thin manager for the compose stack: `up -d` starts it and returns, the unit +# stays "active" (RemainAfterExit) so `systemctl restart thermograph` re-runs it. +# Compose owns container ordering and restarts via depends_on + restart policies. +Type=oneshot +RemainAfterExit=yes +WorkingDirectory=/opt/thermograph +# POSTGRES_PASSWORD (and the other secrets) are needed at `docker compose` +# interpolation time. The leading `-` makes a missing file non-fatal. +EnvironmentFile=-/etc/thermograph.env +ExecStart=/usr/bin/docker compose up -d +ExecStop=/usr/bin/docker compose down [Install] WantedBy=multi-user.target diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..bfae512 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,86 @@ +# Thermograph production stack: the FastAPI app plus its PostgreSQL 18 database. +# +# docker compose up -d --build # or: make up +# +# POSTGRES_PASSWORD must be set at `docker compose` time — compose reads it from +# the repo-root .env for local runs (copy .env.example -> .env), and in prod the +# systemd unit's EnvironmentFile=/etc/thermograph.env puts it in the environment +# so `docker compose up` can interpolate it. It is used BOTH to initialize the db +# container and to build the app's THERMOGRAPH_DATABASE_URL below. + +services: + db: + # PostgreSQL 18 + pg_duckdb (deploy/db/Dockerfile.db, FROM pgduckdb/pgduckdb:18-…, + # which IS PostgreSQL 18). Lets the DB read the app's parquet climate cache + # directly, e.g. SELECT * FROM read_parquet('/parquet/cache/*.parquet'). The init + # script CREATE EXTENSIONs pg_duckdb on a fresh volume. + build: + context: . + dockerfile: deploy/db/Dockerfile.db + environment: + POSTGRES_USER: thermograph + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD} + POSTGRES_DB: thermograph + volumes: + # Mount the volume at the PARENT of the data dir and let the PG18 image use + # its own versioned subdir (/var/lib/postgresql/18/docker). Pinning PGDATA + # directly at the mountpoint trips an initdb chmod on some Docker setups; the + # whole tree still persists on the named volume this way. + - pgdata:/var/lib/postgresql + # Read the app's live parquet cache (the same appdata volume the app writes, + # mounted read-only) — query it under /parquet/cache/*.parquet. + - appdata:/parquet:ro + - ./deploy/db/init:/docker-entrypoint-initdb.d + healthcheck: + test: ["CMD-SHELL", "pg_isready -U thermograph -d thermograph"] + interval: 5s + timeout: 5s + retries: 10 + # Cap the DB at 2 CPUs. Compose v2 honors the top-level `cpus:`; the + # deploy.resources block is the Swarm-style equivalent, kept for parity. + cpus: 2.0 + deploy: + resources: + limits: + cpus: "2.0" + restart: unless-stopped + # No host port on purpose: the app reaches Postgres as db:5432 on the + # compose network. Nothing outside the stack should touch the database. + + app: + build: . + depends_on: + db: + condition: service_healthy + environment: + # Built from POSTGRES_PASSWORD; this `environment` value wins over anything + # in env_file, so the URL always matches the db container's password. + THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph + THERMOGRAPH_BASE: / + PORT: 8137 + WORKERS: 4 + # One worker wins this lock and runs the subscription notifier / homepage + # sweep; it lives on the appdata volume so it's shared across workers. + THERMOGRAPH_SINGLETON_LOCK: /app/data/notifier.lock + # Prod secrets live in /etc/thermograph.env: POSTGRES_PASSWORD, + # THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY, + # THERMOGRAPH_COOKIE_SECURE=1, mail/Discord keys, ... (see + # deploy/thermograph.env.example). `required: false` so local `docker compose + # up` works without that file — it reads POSTGRES_PASSWORD from repo-root .env. + env_file: + - path: /etc/thermograph.env + required: false + volumes: + # Parquet cache, notifier.lock, homepage.json, vapid.json persist here. + - appdata:/app/data + - applogs:/app/logs + cpus: 4.0 + ports: + # Loopback only — host Caddy terminates TLS and reverse-proxies to this. + - "127.0.0.1:8137:8137" + restart: unless-stopped + +volumes: + pgdata: {} + appdata: {} + applogs: {}